WO2013106993A1 - Procédé et dispositif d'extension de capacité et procédé et dispositif d'accès aux données - Google Patents

Procédé et dispositif d'extension de capacité et procédé et dispositif d'accès aux données Download PDF

Info

Publication number
WO2013106993A1
WO2013106993A1 PCT/CN2012/070464 CN2012070464W WO2013106993A1 WO 2013106993 A1 WO2013106993 A1 WO 2013106993A1 CN 2012070464 W CN2012070464 W CN 2012070464W WO 2013106993 A1 WO2013106993 A1 WO 2013106993A1
Authority
WO
WIPO (PCT)
Prior art keywords
node
virtual
physical
data
nodes
Prior art date
Application number
PCT/CN2012/070464
Other languages
English (en)
Chinese (zh)
Inventor
印雨
潘孝忠
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to PCT/CN2012/070464 priority Critical patent/WO2013106993A1/fr
Priority to CN201280000231.1A priority patent/CN103348653B/zh
Publication of WO2013106993A1 publication Critical patent/WO2013106993A1/fr

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/104—Peer-to-peer [P2P] networks

Definitions

  • P2P (Pear to Pear) storage devices based on DHT (Distributed Hash Table) technology have high scalability, support large-scale data storage, and can better cope with human data in the era of network information explosion. Ultra-large storage and management challenges. However, with the growth of storage data, the capacity of storage devices needs to be expanded to meet the growing demand for data storage.
  • DHT Distributed Hash Table
  • the storage device When the storage device is initialized, the storage device is allocated a specified number of virtual nodes in advance, and a mapping relationship between the physical node and the virtual node is established, so that there are multiple virtual nodes on one physical node.
  • the storage device When the storage device is expanded, that is, when the physical node increases, the virtual node is dynamically migrated to the new physical node. For example, before the capacity expansion, each physical node assumes 2N virtual nodes. After the capacity expansion, the physical nodes are doubled. If the load is balanced, each physical node generally assumes N virtual nodes. During the system expansion process, the total number of virtual nodes is always constant.
  • the total number of virtual nodes is preset and remains constant during the system expansion process.
  • a large number of virtual nodes are usually set in advance. Before the system is expanded, the number of virtual nodes corresponding to each physical node is large. When the number of data copies is constant, the physical nodes correspond. The more virtual nodes, the worse the reliability of the system; and, although the total number of pre-set virtual nodes is relatively large, as the system continues to expand, it will eventually expand to the point where no virtual nodes can be assigned to new physical nodes, making the system unable to Capacity expansion, system scalability is poor. Summary of the invention
  • an embodiment of the present invention provides a method and a storage device for capacity expansion.
  • the technical solution is as follows:
  • a method of expanding a volume comprising:
  • n is less than the preset value
  • the data managed by the virtual node and the new virtual node obtained by the splitting is determined, and the data that is not managed by the virtual node and the new virtual node that is split is deleted.
  • a determining module configured to determine the number of virtual nodes according to the number of physical nodes before the expansion, so that the physical nodes before each expansion correspond to n virtual nodes, where n is less than a preset value;
  • the splitting module is used to expand the physical node when the system is expanded, and some physical nodes appear corresponding to one virtual node.
  • the virtual node before the expansion is split, and a new virtual node is generated, so that the client is based on the split virtual node.
  • Quantity redefining the virtual node of the data map;
  • a migration module configured to migrate part of the virtual node to the newly added physical node
  • the deletion module is configured to determine data managed by the virtual node and the new virtual node that is split by the virtual node, and delete the data that is not managed by the virtual node and the new virtual node that is split by the virtual node. Based on the foregoing expansion method, a method for accessing data is also provided, where the method includes:
  • a client device, the client device includes:
  • An access module configured to determine a virtual node of the data mapping to be accessed, determining, according to the first correspondence, a first physical node where the data is located, and requesting access data from the first physical node;
  • the update module is further configured to: when the virtual node migrates, update a correspondence between the physical node and the virtual node to obtain a second correspondence;
  • the access module is further configured to determine, according to the second correspondence, a second physical node where the data is located, and request access data from the second physical node.
  • the virtual node is dynamically migrated by the virtual node splitting during the capacity expansion, which avoids the reliability problem caused by pre-setting a large number of virtual nodes, and improves the storage device.
  • the reliability while avoiding the problem of non-expansion caused by the fact that no virtual nodes can be allocated, improves the scalability of the storage device.
  • a corresponding access data scheme is provided.
  • Embodiment 1 is a flowchart of a method for expanding capacity provided by Embodiment 1 of the present invention
  • FIG. 4 is a schematic diagram of a partition of a storage device after splitting according to Embodiment 2 of the present invention.
  • this embodiment provides a method for expanding a capacity, which may be performed by a storage device, where the method includes:
  • a physical node When a system is expanded, a physical node is newly added, and a part of the physical node corresponds to a virtual node.
  • a new virtual node is generated, and the client re-determines the virtual node of the data mapping according to the number of the virtual nodes after the splitting;
  • each virtual node splits a new virtual node Pi splits P (i+N), that is, P0 splits P10, PI splits Pl l, and so on, P9 splits P19, and the total number of virtual nodes after splitting is 2N 20.
  • the virtual node corresponding to each physical node is doubled. Take Figure 3 as an example. After the split, each physical node corresponds to four virtual nodes. The corresponding relationship between the physical node and the virtual node after splitting is as follows:
  • the virtual nodes corresponding to each physical node before the splitting are evenly distributed, the virtual nodes corresponding to each physical node are still balanced after the splitting. Therefore, the read and write requests to the client do not cause data hotspot problems.
  • Hash represents a hash operation
  • key represents the identity of the data
  • mod represents a modulo operation
  • N is the number of virtual nodes before the split.
  • 2N represents the number of virtual nodes after the split.
  • the migration policy is not specifically limited, and any strategy for enabling the virtual nodes corresponding to the physical nodes to be substantially averaged is adapted to the embodiment. For example, calculate the average of the number of virtual nodes corresponding to each physical node, and the physical section The virtual node corresponding to the virtual node number exceeding the average value is migrated to the physical node whose corresponding virtual node number is smaller than the average value.
  • any strategy for enabling the virtual nodes corresponding to the physical nodes to be substantially averaged is adapted to the embodiment. For example, calculate the average of the number of virtual nodes corresponding to each physical node, and the physical section The virtual node corresponding to the virtual node number exceeding the average value is migrated to the physical node whose corresponding virtual node number is smaller than the average value.
  • two new physical nodes are added, and a total of seven physical nodes are added.
  • each physical node corresponds to an average of 20/7 «2. 85 logics.
  • Nodes that is, some physical nodes correspond to three virtual nodes, and some physical nodes correspond to two virtual nodes.
  • a total of five virtual nodes are migrated to the newly added physical node, and one physical node and one virtual node are migrated.
  • the corresponding relationship can be: Physical node 1: P0, P5, P10
  • the client device re-determines the virtual node of the data mapping to be written, determines a physical node where the data is located according to the first correspondence, and requests the physical node to write data.
  • step 202 For the virtual node splitting process, refer to step 202, and details are not described herein again.
  • the physical node 6 redirects the request to the physical node 1 after receiving the request from the client device to read the data.

Abstract

Des modes de réalisation de la présente invention concernent le domaine technique de la mémorisation de données, et fournissent un procédé et un dispositif d'extension de capacité et un procédé et un dispositif d'accès aux données. Le procédé comprend les étapes consistant à : déterminer le nombre de nœuds virtuels en fonction du nombre de nœuds physiques avant l'extension de la capacité, de sorte que chaque nœud physique avant l'extension de capacité correspond à n nœuds virtuels, n étant inférieur à une valeur prédéfinie ; lorsque des nœuds physiques sont ajoutés au cours de l'extension de capacité du système et qu'une partie des nœuds physiques correspondent chacun à un nœud virtuel, les nœuds virtuels avant l'extension de capacité sont divisés pour générer de nouveaux nœuds virtuels de sorte que le client re-détermine les nœuds virtuels pour le mappage de données selon le nombre des nœuds virtuels après la division ; faire migrer une partie des nœuds virtuels vers le nouveaux nœuds physiques supplémentaires ; et déterminer les données gérées par les nœuds virtuels et par les nœuds virtuels nouvellement obtenus après division, et respectivement supprimer des données non gérées par les nœuds virtuels et les nœuds virtuels récemment obtenus après division. Au moyen de la solution ci-dessus, la présente invention améliore la fiabilité et l'extensibilité d'un dispositif de stockage.
PCT/CN2012/070464 2012-01-17 2012-01-17 Procédé et dispositif d'extension de capacité et procédé et dispositif d'accès aux données WO2013106993A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/CN2012/070464 WO2013106993A1 (fr) 2012-01-17 2012-01-17 Procédé et dispositif d'extension de capacité et procédé et dispositif d'accès aux données
CN201280000231.1A CN103348653B (zh) 2012-01-17 2012-01-17 扩容的方法和设备、以及访问数据的方法和设备

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2012/070464 WO2013106993A1 (fr) 2012-01-17 2012-01-17 Procédé et dispositif d'extension de capacité et procédé et dispositif d'accès aux données

Publications (1)

Publication Number Publication Date
WO2013106993A1 true WO2013106993A1 (fr) 2013-07-25

Family

ID=48798484

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2012/070464 WO2013106993A1 (fr) 2012-01-17 2012-01-17 Procédé et dispositif d'extension de capacité et procédé et dispositif d'accès aux données

Country Status (2)

Country Link
CN (1) CN103348653B (fr)
WO (1) WO2013106993A1 (fr)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104219163A (zh) * 2014-08-28 2014-12-17 杭州天宽科技有限公司 一种基于动态副本法和虚拟节点法的节点动态前移的负载均衡方法
CN106843745A (zh) * 2015-12-03 2017-06-13 南京中兴新软件有限责任公司 容量扩展方法及装置
CN107943412A (zh) * 2016-10-12 2018-04-20 阿里巴巴集团控股有限公司 一种分区分裂、删除分区中数据文件的方法、装置及系统
CN109428913A (zh) * 2017-08-25 2019-03-05 浙江宇视科技有限公司 一种存储扩容方法及装置

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106874218B (zh) * 2017-01-12 2019-12-03 深圳市汉云科技有限公司 一种映射地址空间的分配方法及其装置

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6667957B1 (en) * 1998-03-14 2003-12-23 University Of Maryland Adaptive routing method for a dynamic network
CN1988518A (zh) * 2006-12-13 2007-06-27 北京交通大学 一种对等网搜索方法
CN101098271A (zh) * 2006-06-29 2008-01-02 腾讯科技（深圳）有限公司 一种可扩展的p2p流媒体系统
WO2009027612A2 (fr) * 2007-08-24 2009-03-05 Thomson Licensing Procede de constitution d'un environnement virtuel de reseau p2p de type delaunay, par groupage dynamique de noeuds, et dispositif d'aide et equipement associe
US7577134B2 (en) * 2005-08-19 2009-08-18 Brocade Communications Systems, Inc. Port expander for fibre channel fabrics in storage area networks

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102063473B (zh) * 2010-12-21 2012-10-10 南京壹进制信息技术有限公司 一种可在线扩展存储空间的方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6667957B1 (en) * 1998-03-14 2003-12-23 University Of Maryland Adaptive routing method for a dynamic network
US7577134B2 (en) * 2005-08-19 2009-08-18 Brocade Communications Systems, Inc. Port expander for fibre channel fabrics in storage area networks
CN101098271A (zh) * 2006-06-29 2008-01-02 腾讯科技（深圳）有限公司 一种可扩展的p2p流媒体系统
CN1988518A (zh) * 2006-12-13 2007-06-27 北京交通大学 一种对等网搜索方法
WO2009027612A2 (fr) * 2007-08-24 2009-03-05 Thomson Licensing Procede de constitution d'un environnement virtuel de reseau p2p de type delaunay, par groupage dynamique de noeuds, et dispositif d'aide et equipement associe

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104219163A (zh) * 2014-08-28 2014-12-17 杭州天宽科技有限公司 一种基于动态副本法和虚拟节点法的节点动态前移的负载均衡方法
CN104219163B (zh) * 2014-08-28 2016-08-17 杭州天宽科技有限公司 一种基于动态副本法和虚拟节点法的节点动态前移的负载均衡方法
CN106843745A (zh) * 2015-12-03 2017-06-13 南京中兴新软件有限责任公司 容量扩展方法及装置
CN107943412A (zh) * 2016-10-12 2018-04-20 阿里巴巴集团控股有限公司 一种分区分裂、删除分区中数据文件的方法、装置及系统
CN107943412B (zh) * 2016-10-12 2021-10-22 阿里巴巴集团控股有限公司 一种分区分裂、删除分区中数据文件的方法、装置及系统
CN109428913A (zh) * 2017-08-25 2019-03-05 浙江宇视科技有限公司 一种存储扩容方法及装置
CN109428913B (zh) * 2017-08-25 2021-06-01 浙江宇视科技有限公司 一种存储扩容方法及装置

Also Published As

Publication number Publication date
CN103348653B (zh) 2016-01-20
CN103348653A (zh) 2013-10-09

Similar Documents

Publication Publication Date Title
US10896102B2 (en) Implementing secure communication in a distributed computing system
US11288286B2 (en) Storage system with data consistency checking in synchronous replication using active snapshot set
US11055014B2 (en) Storage system providing automatic configuration updates for remote storage objects in a replication process
US10642694B2 (en) Monitoring containers in a distributed computing system
US10824512B2 (en) Managing journaling resources with copies stored in multiple locations
US9904599B2 (en) Method, device, and system for data reconstruction
EP3502877B1 (fr) Procédé et dispositif de chargement de données pour machines virtuelles
WO2020204882A1 (fr) Système de stockage activé par instantané mettant en œuvre un algorithme pour la lecture efficace de données à partir d'instantanés stockés
CN110795206B (zh) 用于促进集群级缓存和内存空间的系统和方法
WO2021057956A1 (fr) Procédé, appareil et système permettant de gérer un fichier d'image miroir, et dispositif informatique et support de stockage
US20130332418A1 (en) Method of managing data in asymmetric cluster file system
US10235282B2 (en) Computer system, computer, and method to manage allocation of virtual and physical memory areas
CN109933312B (zh) 一种有效降低容器化关系型数据库i/o消耗的方法
CN112799841B (zh) 一种数据对象存储管理的方法和装置
TWI617924B (zh) 記憶體資料分版技術
WO2013106993A1 (fr) Procédé et dispositif d'extension de capacité et procédé et dispositif d'accès aux données
US10877684B2 (en) Changing a distributed storage volume from non-replicated to replicated
JP2014534482A (ja) ストレージシステム、管理計算機、ストレージ装置及びデータ管理方法
US20230273899A1 (en) Data migration in a distributive file system
JP6233403B2 (ja) ストレージシステム、ストレージ装置、ストレージ装置の制御方法及び制御プログラム、管理装置、管理装置の制御方法及び制御プログラム
CN111444157A (zh) 分布式文件系统及数据访问方法
CN117032586A (zh) 具有写入均衡的闪存寄存器
US11216204B2 (en) Degraded redundant metadata, DRuM, technique
US11256434B2 (en) Data de-duplication
US20200034475A1 (en) Relocation Of A Primary Copy Of A Replicated Volume

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 12866088

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 12866088

Country of ref document: EP

Kind code of ref document: A1