CN103929500A - Method for data fragmentation of distributed storage system - Google Patents
Method for data fragmentation of distributed storage system Download PDFInfo
- Publication number
- CN103929500A CN103929500A CN201410189121.5A CN201410189121A CN103929500A CN 103929500 A CN103929500 A CN 103929500A CN 201410189121 A CN201410189121 A CN 201410189121A CN 103929500 A CN103929500 A CN 103929500A
- Authority
- CN
- China
- Prior art keywords
- node
- data
- physical
- space
- virtual disk
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method for data fragmentation of a distributed storage system. The method comprises the steps that a physical node state is monitored through a cluster controller and the correctness of data transmission is guaranteed; physical spaces of storage nodes are managed in a unified mode through the consistent Hash algorithm, so that a logic storage space is formed; the reliability of the system is guaranteed in multi-transcript fragment mode. According to the method, the storage capacity is lager and flexible and the storage capacity virtualized by a host computer depends on resources of a native machine. Due to the limitation on hardware design of the host computer, the storage capacity cannot be infinitely increased; the storage capacity cannot be limited to the storage capacity of the host computer after the distributed system is adopted, storage resources can be increased or reduced at will according to the requirement of a client, and the storage capacity and the flexibility of the system are improved to the greatest extent.
Description
Technical field
The present invention relates to the technical field of data storage, particularly a kind of data fragmentation method of distributed memory system.
Background technology
A complete storage system at least will possess memory node and control node, memory node is responsible for storing documents, control node as file index, and be responsible for the equilibrium of capacity and load between monitoring memory node, these 2 parts just form a storage system altogether.
Traditional storage system adopts serial frame more, and memory node and control node are placed on an equipment, and memory capacity distribution is subject to physical hard disk restriction, in the time that we will dispose new memory device dilatation.Our this memory device that must be perfectly clear is any model, what interface and host-host protocol, must know clearly in storage system and have how many disks, be respectively what model, much capacity, must know between memory device and server, to adopt which type of connection cable.In order to guarantee data security and the continuity of business, we also need to set up corresponding data backup system and disaster tolerance system.In addition it is also necessary, memory device being carried out to regular condition monitoring, maintenance, software and hardware renewal and upgrading.
There is the problems such as purchase cost is high, system lock, maintainability is poor, autgmentability is low, availability is limited in heritage storage system.As carried out System Expansion at needs or changing when hardware, conventionally need to stop serve and carry out a series of complex operations could be successfully; Meanwhile, due to the limitation on framework, it is to be solved that conventional store still has several important technical problems to have, and such as Single Point of Faliure problem, brain fragmentation problem etc. make client's data have the risk of losing or being interrupted service.
Exist above shortcoming to be mainly because the limitation on traditional storage system framework, it is by memory node and control node deployment on same equipment, provide stores service by software and hardware device binding, software has departed from intrinsic hardware and cannot on other memory devices, move, large to hardware device dependence, do not accomplish separation of software and hardware.
Summary of the invention
The shortcoming that the object of the invention is to overcome prior art, with not enough, provides a kind of data fragmentation method of distributed memory system.
In order to achieve the above object, the present invention is by the following technical solutions:
A data fragmentation method for distributed memory system, comprises the steps:
S1, adopt cluster controller to manage multiple nodes, then the physical store resource of each node is gathered, be synchronized to and in each node memory, carry out record;
S2, every physical node add after cluster, can calculate the memory space of this node, calculate afterwards all node storage summations, this node storage capacity is obtained to weighted value divided by the capacity of whole nodes, be multiplied by acquiescence dummy node number with this weighted value again, obtain the dummy node actual number of this node, then by this node mapping to the dummy node that generates corresponding number in dummy node list;
S3, the allocation space request of system obtains the real space of physical node according to the consistency hash algorithm of load balancing part, be specially dummy node that each node is corresponding as the element in Hash ring, cryptographic Hash is calculated in each request, then search node on Hash ring in order, can carry out actual use to memory space like this, first each read-write requests can be undertaken by virtual disk, virtual disk provides read-write requests location-based service by metadata is positioned on concrete physical data object, each physical object is present on different physical hosts, can ensure by there being the copy of multiple data objects the reliability of data,
Preferably, each data slice is cut into multiple objects, and data object can create again multiple copies, data to read process as follows:
(1) user specifies the parameters such as the virtual disk title that need to read, original position, length;
(2) system is calculated virtual numbering according to virtual disk title, and searches in object indexing, if there is no directly unsuccessfully returns;
(3) compare by this virtual disk numbering calculating object numbering, and by calculating cryptographic Hash and dummy node numbered list, from dummy node numbered list, obtain the set of fixed joint given figure object dummy node, and by this compound mapping to physical node;
(4) if node is identical with the machine, directly obtains data object and return from the machine, if not in the machine, delivering to message to destination node, after destination node return data, returning results step by step;
(5), if destination node reads failure, reading out data from other alternative node cluster, is then synchronized to damage node, returns step by step afterwards.
Preferably, the ablation process of data is:
(1) user specifies the parameters such as the virtual disk title that need to write, original position, length;
(2) system is calculated virtual disk numbering according to virtual disk title, and searches in object indexing, if there is no directly unsuccessfully returns;
(3) compare by this virtual disk numbering calculating object numbering, and by calculating cryptographic Hash and dummy node numbered list, from dummy node numbered list, obtain the set of fixed joint given figure object dummy node, and by this compound mapping to physical node;
(4) system sends write request to the node in node listing one by one, all returns successfully to return to afterwards user and write result, and flow process finishes.
Preferably, this storage system comprises cluster controller, allocation of space module and space management module, and described cluster controller, allocation of space module and space management sequence of modules connect; This system is by the correctness of cluster controller monitors physical node state and guarantee transfer of data; By consistency hash algorithm, the physical space of each memory node is carried out to unified management, composition logical memory space; Ensure the reliability of system by many copies slicing mode.
Preferably, described cluster controller, for managing multiple physical nodes, then gathers the physical store resource of each physical node, is synchronized to and in each node memory, carries out record;
Described allocation of space module, obtain the real space of physical node according to the consistency hash algorithm of load balancing part, can carry out actual use to memory space like this, first each read-write requests can be undertaken by virtual disk, virtual disk provides read-write requests location-based service by metadata is positioned on concrete physical data object, each physical object is present on different physical hosts, can ensure by there being the copy of multiple data objects the reliability of data;
Described space management module, space management module is the administration module that dummy block will storage subsystem externally provides space, for realizing according to the establishment of Virtual Space, read-write.
Preferably, the exchanges data of described space management module stores is made up of three elements: side-play amount, length, data, and side-play amount is the position for this virtual disk, the offset location of this data block of data manipulation pointed, according to the data length of specifying, do that data write or read operation.
The present invention has following advantage and effect with respect to prior art:
1, memory capacity of the present invention is larger and have elasticity, the memory capacity of mainframe virtualization depends on the resource of the machine, but a main frame is because the restriction memory capacity in hardware designs can infinitely not increase, and adopt memory capacity after this distributed system to be no longer subject to the restriction of main frame memory capacity, can be along with client's requirement increases arbitrarily storage resources or reduces storage resources, maximized storage capacity and the elasticity that has improved system.
2, the present invention has higher availability mainframe virtualization technology and can cause the system machine of delaying by inevitable hardware fault, because hardware is along with the passing of time and time, and inevitably can be because old or line fault cause break in service.But this distributed storage virtualization technology, can avoid damaging due to local hardware device the access facility of the system that affects, and therefore has higher availability.
Brief description of the drawings
Fig. 1 is the control structure figure of distributed virtualization technology of the present invention.
Fig. 2 is logical space of the present invention and data block mapping relations figure.
Fig. 3 is logical space of the present invention and data block mapping flow chart.
Embodiment
Below in conjunction with embodiment and accompanying drawing, the present invention is described in further detail, but embodiments of the present invention are not limited to this.
Embodiment
For the virtualized shortcoming of host stores, adopt distributed Intel Virtualization Technology can solve existing problem, distributed virtualization technology adopts cluster controller to manage multiple nodes, again the physical store resource of each node is gathered, be synchronized to and in each node memory, carry out record, independent mainframe virtualization can be distributed unitedly and use thus, and physical space is become to a flexible space in logic.Can by disperse in logic, the resource virtualizing of physics becomes complete a, logical resource, use storage resources by user according to the size of any (system is also conditional with realization), the design of space management is mainly that physical resource is become to logical resource.In internal memory, record the space of each node; by have living space be added as complete logical space; use unified space that memory function is externally provided; each service request is all obtained key assignments by Hash calculation; on Hash ring, turn clockwise and find corresponding physical node, this physical node is carried out to the read-write of data.Relative theory is asked for an interview Fig. 1.
Every physical node adds after cluster, can calculate the memory space of this node, calculate afterwards all node storage summations same, that this node obtains weighted value divided by the capacity of whole nodes, be multiplied by acquiescence dummy node number with this weighted value again, obtain the dummy node actual number of this node, then by this node mapping to the dummy node that generates corresponding number in dummy node list.
The allocation space request of system obtains the real space of physical node according to the consistency hash algorithm of load balancing part, can carry out actual use to memory space like this.First each read-write requests can be undertaken by virtual disk, virtual disk provides read-write requests location-based service by metadata is positioned on concrete physical data object, each physical object is present on different physical hosts, can ensure by there being the copy of multiple data objects the reliability of data.
The main practical function of space management module has the establishment of Virtual Space, read-write etc.Space management module is the administration module that dummy block will storage subsystem externally provides space, and the exchanges data of piece storage is made up of three elements: side-play amount, length, data.Side-play amount is the position for this virtual disk, and the offset location of this data block of data manipulation pointed, according to the data length of specifying, does that data write or read operation.
A complete data space splits into the data block of fixed size in space management module, and each read-write requests is continuous data, and the position of calculating every byte data according to the multiple of 2M corresponds to respectively and in data block, carries out read-write operation; Relative theory is shown in Figure 2, Fig. 2 logical space and data block mapping relations.
As shown in Figure 3, each data slice can be cut into multiple data objects, and data object can create again multiple copies, so the read-write of data need to be considered simultaneously concurrent application scenarios.
The process that data read is as follows:
(1) user specifies the parameters such as the virtual disk title that need to read, original position, length;
(2) system is calculated virtual disk numbering according to virtual disk title, and searches in object indexing, if there is no directly unsuccessfully returns;
(3) compare by this virtual disk numbering calculating object numbering, and by calculating cryptographic Hash and dummy node numbered list, from dummy node numbered list, obtain the set of fixed joint given figure object dummy node, and by this compound mapping to physical node;
(4) if node is identical with the machine, directly obtains data object and return from the machine, if not in the machine, delivering to message to destination node, after destination node return data, returning results step by step.
(5), if destination node reads failure, reading out data from other alternative node cluster, is then synchronized to damage node, returns step by step afterwards.
Can see by above (5) article, in the time that certain segment data object in system is lost, can not affect the success of the read operation of data, system can be synchronized to from the position reading out data of complete node the node of obliterated data, is the time but the process of this reparation can take certain response.
Data write from read different, reading out data requires as much as possible to read fast data, and data write and will accomplish strong consistency, while all writing the destination node at each object number place, just can return successfully by data, other side and data read similar, and idiographic flow is as follows:
(1) user specifies the parameters such as the virtual disk title that need to write, original position, length;
(2) system is calculated virtual disk numbering according to virtual disk title, and searches in object indexing, if there is no directly unsuccessfully returns;
(3) compare by this virtual disk numbering calculating object numbering, and by calculating cryptographic Hash and dummy node numbered list, from dummy node numbered list, obtain the set of fixed joint given figure object dummy node, and by this compound mapping to physical node;
(4) system sends write request to the node in node listing one by one, all returns successfully to return to afterwards user and write result, and flow process finishes.
Above-described embodiment is preferably execution mode of the present invention; but embodiments of the present invention are not restricted to the described embodiments; other any do not deviate from change, the modification done under Spirit Essence of the present invention and principle, substitutes, combination, simplify; all should be equivalent substitute mode, within being included in protection scope of the present invention.
Claims (6)
1. a data fragmentation method for distributed memory system, is characterized in that, comprises the steps:
S1, adopt cluster controller to manage multiple nodes, then the physical store resource of each node is gathered, be synchronized to and in each node memory, carry out record;
S2, every physical node add after cluster, can calculate the memory space of this node, calculate afterwards all node storage summations, this node storage capacity is obtained to weighted value divided by the capacity of whole nodes, be multiplied by acquiescence dummy node number with this weighted value again, obtain the dummy node actual number of this node, then by this node mapping to the dummy node that generates corresponding number in dummy node list;
S3, the allocation space request of system obtains the real space of physical node according to the consistency hash algorithm of load balancing part, be specially dummy node that each node is corresponding as the element in Hash ring, cryptographic Hash is calculated in each request, then search node on Hash ring in order, can carry out actual use to memory space like this, first each read-write requests can be undertaken by virtual disk, virtual disk provides read-write requests location-based service by metadata is positioned on concrete physical data object, each physical object is present on different physical hosts, can ensure by there being the copy of multiple data objects the reliability of data.
2. distributed memory system data fragmentation method according to claim 1, is characterized in that, each data slice is cut into multiple objects, and data object can create again multiple copies, data to read process as follows:
(1) user specifies virtual disk title, original position, the length parameter that need to read;
(2) system is calculated virtual numbering according to virtual disk title, and searches in object indexing, if there is no directly unsuccessfully returns;
(3) compare by this virtual disk numbering calculating object numbering, and by calculating cryptographic Hash and dummy node numbered list, from dummy node numbered list, obtain the set of fixed joint given figure object dummy node, and by this compound mapping to physical node;
(4) if node is identical with the machine, directly obtains data object and return from the machine, if not in the machine, delivering to message to destination node, after destination node return data, returning results step by step;
(5), if destination node reads failure, reading out data from other alternative node cluster, is then synchronized to damage node, returns step by step afterwards.
3. distributed memory system data fragmentation method according to claim 1, is characterized in that, the ablation process of data is:
(1) user specifies virtual disk title, original position, the length parameter that need to write;
(2) system is calculated virtual disk numbering according to virtual disk title, and searches in object indexing, if there is no directly unsuccessfully returns;
(3) compare by this virtual disk numbering calculating object numbering, and by calculating cryptographic Hash and dummy node numbered list, from dummy node numbered list, obtain the set of fixed joint given figure object dummy node, and by this compound mapping to physical node;
(4) system sends write request to the node in node listing one by one, all returns successfully to return to afterwards user and write result, and flow process finishes.
4. distributed memory system data fragmentation method according to claim 1, it is characterized in that, this storage system comprises cluster controller, allocation of space module and space management module, and described cluster controller, allocation of space module and space management sequence of modules connect; This system is by the correctness of cluster controller monitors physical node state and guarantee transfer of data; By consistency hash algorithm, the physical space of each memory node is carried out to unified management, composition logical memory space; Ensure the reliability of system by many copies slicing mode.
5. distributed memory system data fragmentation method according to claim 4, it is characterized in that, described cluster controller, for managing multiple physical nodes, again the physical store resource of each physical node is gathered, be synchronized to and in each node memory, carry out record;
Described allocation of space module, obtain the real space of physical node according to the consistency hash algorithm of load balancing part, can carry out actual use to memory space like this, first each read-write requests can be undertaken by virtual disk, virtual disk provides read-write requests location-based service by metadata is positioned on concrete physical data object, each physical object is present on different physical hosts, can ensure by there being the copy of multiple data objects the reliability of data;
Described space management module, space management module is the administration module that dummy block will storage subsystem externally provides space, for realizing according to the establishment of Virtual Space, read-write.
6. distributed memory system data fragmentation method according to claim 4, it is characterized in that, the exchanges data of described space management module stores is made up of three elements: side-play amount, length, data, side-play amount is the position for this virtual disk, the offset location of this data block of data manipulation pointed, according to the data length of specifying, do that data write or read operation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410189121.5A CN103929500A (en) | 2014-05-06 | 2014-05-06 | Method for data fragmentation of distributed storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410189121.5A CN103929500A (en) | 2014-05-06 | 2014-05-06 | Method for data fragmentation of distributed storage system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103929500A true CN103929500A (en) | 2014-07-16 |
Family
ID=51147572
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410189121.5A Pending CN103929500A (en) | 2014-05-06 | 2014-05-06 | Method for data fragmentation of distributed storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103929500A (en) |
Cited By (44)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104270458A (en) * | 2014-10-17 | 2015-01-07 | 浪潮(北京)电子信息产业有限公司 | Method and device for achieving data distribution management of cloud storage system |
| CN105989140A (en) * | 2015-02-27 | 2016-10-05 | 阿里巴巴集团控股有限公司 | Data block processing method and equipment |
| CN106503010A (en) * | 2015-09-07 | 2017-03-15 | 北京国双科技有限公司 | A kind of method and device of database change write subregion |
| CN106776001A (en) * | 2016-11-14 | 2017-05-31 | 天津南大通用数据技术股份有限公司 | The location mode and device of a kind of distributed experiment ＆ measurement system data |
| CN106844487A (en) * | 2016-12-23 | 2017-06-13 | 航天星图科技（北京）有限公司 | A kind of distributed memory system replicated based on storehouse |
| CN106936899A (en) * | 2017-02-25 | 2017-07-07 | 九次方大数据信息集团有限公司 | The collocation method of distributed statistical analysis system and distributed statistical analysis system |
| CN106980540A (en) * | 2017-03-07 | 2017-07-25 | 清华大学 | A kind of computational methods of distributed Multidimensional Discrete data |
| CN106991170A (en) * | 2017-04-01 | 2017-07-28 | 广东浪潮大数据研究有限公司 | A kind of method and apparatus of distributed document capacity equilibrium |
| CN107018185A (en) * | 2017-03-28 | 2017-08-04 | 清华大学 | The synchronous method and device of cloud storage system |
| CN107154966A (en) * | 2017-04-24 | 2017-09-12 | 浙江宇视科技有限公司 | A kind of video data wiring method and device |
| CN107168649A (en) * | 2017-05-05 | 2017-09-15 | 南京城市职业学院 | The method and device of data distribution in a kind of distributed memory system |
| CN107256132A (en) * | 2017-06-21 | 2017-10-17 | 桂林电子科技大学 | A kind of isomery Redis cluster-based storage distribution methods based on performance test |
| CN107273040A (en) * | 2016-04-08 | 2017-10-20 | 北京优朋普乐科技有限公司 | data cache method and device |
| CN107346307A (en) * | 2016-05-04 | 2017-11-14 | 北京京东尚科信息技术有限公司 | Distributed cache system and method |
| CN107357522A (en) * | 2014-11-05 | 2017-11-17 | 华为技术有限公司 | Data processing method and device |
| CN107577434A (en) * | 2017-09-14 | 2018-01-12 | 郑州云海信息技术有限公司 | Capacity acquiring method, device, equipment and the readable storage medium storing program for executing of memory node |
| CN107608632A (en) * | 2017-09-21 | 2018-01-19 | 郑州云海信息技术有限公司 | A kind of communication means, the apparatus and system of distributed storage cluster |
| CN107623719A (en) * | 2017-08-17 | 2018-01-23 | 深圳市优品壹电子有限公司 | Data capture method and device |
| CN107634853A (en) * | 2017-08-28 | 2018-01-26 | 深圳市云舒网络技术有限公司 | A kind of customizable distributed caching method based on object storage |
| CN107656697A (en) * | 2016-07-26 | 2018-02-02 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus of operation data on a storage medium |
| CN107948233A (en) * | 2016-10-13 | 2018-04-20 | 华为技术有限公司 | The method of processing write requests or read request, interchanger, control node |
| CN108052655A (en) * | 2017-12-28 | 2018-05-18 | 新华三技术有限公司 | Data write and read method |
| CN109165262A (en) * | 2018-10-16 | 2019-01-08 | 成都索贝数码科技股份有限公司 | Fragmentation clustering system and fragmentation method of relational large table |
| CN109327539A (en) * | 2018-11-15 | 2019-02-12 | 上海天玑数据技术有限公司 | A kind of distributed block storage system and its data routing method |
| CN109416620A (en) * | 2016-05-04 | 2019-03-01 | 净睿存储股份有限公司 | Storage cluster |
| CN109635948A (en) * | 2018-12-19 | 2019-04-16 | 北京达佳互联信息技术有限公司 | On-line training method, apparatus, system and computer readable storage medium |
| CN109726600A (en) * | 2017-10-31 | 2019-05-07 | 伊姆西Ip控股有限责任公司 | The system and method for data protection are provided for super fusion infrastructure |
| CN110046160A (en) * | 2019-03-15 | 2019-07-23 | 中国科学院计算技术研究所 | A kind of consistency Hash storage system construction method based on band |
| CN110169040A (en) * | 2018-07-10 | 2019-08-23 | 深圳花儿数据技术有限公司 | Distributed data storage method and system based on multilayer consistency Hash |
| CN110321374A (en) * | 2018-10-23 | 2019-10-11 | 开采夫(杭州)科技有限公司 | Standard file I/O operations system and method based on distributed network |
| CN110324429A (en) * | 2019-07-10 | 2019-10-11 | 中国工商银行股份有限公司 | Backup method and back-up device based on Distributed Storage |
| CN110334091A (en) * | 2019-05-09 | 2019-10-15 | 重庆天蓬网络有限公司 | A kind of data fragmentation distributed approach, system, medium and electronic equipment |
| CN111090687A (en) * | 2019-12-24 | 2020-05-01 | 腾讯科技（深圳）有限公司 | Data processing method, device and system and computer readable storage medium |
| CN111125253A (en) * | 2019-12-22 | 2020-05-08 | 北京浪潮数据技术有限公司 | Data synchronization method, device, equipment and storage medium |
| CN111147546A (en) * | 2019-11-29 | 2020-05-12 | 中科院计算技术研究所大数据研究院 | Method and system for processing edge cluster resources |
| CN111309262A (en) * | 2020-02-16 | 2020-06-19 | 西安奥卡云数据科技有限公司 | Distributed storage cache reading and writing method |
| CN111756828A (en) * | 2020-06-19 | 2020-10-09 | 广东浪潮大数据研究有限公司 | Data storage method, device and equipment |
| CN112083892A (en) * | 2020-09-25 | 2020-12-15 | 上海依图网络科技有限公司 | Data storage method, device, equipment and medium |
| WO2021003935A1 (en) * | 2019-07-11 | 2021-01-14 | 平安科技（深圳）有限公司 | Data cluster storage method and apparatus, and computer device |
| CN112383628A (en) * | 2020-11-16 | 2021-02-19 | 北京中电兴发科技有限公司 | Storage gateway resource allocation method based on streaming storage |
| CN112422611A (en) * | 2020-09-11 | 2021-02-26 | 深圳市证通电子股份有限公司 | Virtual bucket storage processing method and system based on distributed object storage |
| CN113688115A (en) * | 2021-08-29 | 2021-11-23 | 中盾创新档案管理(北京)有限公司 | File big data distributed storage system based on Hadoop |
| CN114647388A (en) * | 2022-05-24 | 2022-06-21 | 杭州优云科技有限公司 | High-performance distributed block storage system and management method |
| CN114776176A (en) * | 2022-05-24 | 2022-07-22 | 中欣链证数字科技有限公司 | Precious metal storage method, device, equipment and system |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101984632A (en) * | 2010-11-15 | 2011-03-09 | 中兴通讯股份有限公司 | Load distributing method, device and server in distributed cache system |
| CN101997918A (en) * | 2010-11-11 | 2011-03-30 | 清华大学 | Method for allocating mass storage resources according to needs in heterogeneous SAN (Storage Area Network) environment |
| CN102457571A (en) * | 2011-09-15 | 2012-05-16 | 中标软件有限公司 | Method for uniformly distributing data in cloud storage |
| CN102999571A (en) * | 2012-11-09 | 2013-03-27 | 深圳市宜搜科技发展有限公司 | Realizing method for multiple nodes of single computer in cluster |
| CN103124299A (en) * | 2013-03-21 | 2013-05-29 | 杭州电子科技大学 | Distributed block-level storage system in heterogeneous environment |
| CN103229151A (en) * | 2012-12-27 | 2013-07-31 | 华为技术有限公司 | Partition extension method and device |
-
2014
- 2014-05-06 CN CN201410189121.5A patent/CN103929500A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101997918A (en) * | 2010-11-11 | 2011-03-30 | 清华大学 | Method for allocating mass storage resources according to needs in heterogeneous SAN (Storage Area Network) environment |
| CN101984632A (en) * | 2010-11-15 | 2011-03-09 | 中兴通讯股份有限公司 | Load distributing method, device and server in distributed cache system |
| CN102457571A (en) * | 2011-09-15 | 2012-05-16 | 中标软件有限公司 | Method for uniformly distributing data in cloud storage |
| CN102999571A (en) * | 2012-11-09 | 2013-03-27 | 深圳市宜搜科技发展有限公司 | Realizing method for multiple nodes of single computer in cluster |
| CN103229151A (en) * | 2012-12-27 | 2013-07-31 | 华为技术有限公司 | Partition extension method and device |
| CN103124299A (en) * | 2013-03-21 | 2013-05-29 | 杭州电子科技大学 | Distributed block-level storage system in heterogeneous environment |
Cited By (68)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104270458A (en) * | 2014-10-17 | 2015-01-07 | 浪潮(北京)电子信息产业有限公司 | Method and device for achieving data distribution management of cloud storage system |
| CN107357522B (en) * | 2014-11-05 | 2019-11-15 | 华为技术有限公司 | Data processing method and device |
| US10628050B2 (en) | 2014-11-05 | 2020-04-21 | Huawei Technologies Co., Ltd. | Data processing method and apparatus |
| CN107357522A (en) * | 2014-11-05 | 2017-11-17 | 华为技术有限公司 | Data processing method and device |
| CN105989140B (en) * | 2015-02-27 | 2019-09-03 | 阿里巴巴集团控股有限公司 | A kind of data block processing method and equipment |
| CN105989140A (en) * | 2015-02-27 | 2016-10-05 | 阿里巴巴集团控股有限公司 | Data block processing method and equipment |
| CN106503010A (en) * | 2015-09-07 | 2017-03-15 | 北京国双科技有限公司 | A kind of method and device of database change write subregion |
| CN107273040A (en) * | 2016-04-08 | 2017-10-20 | 北京优朋普乐科技有限公司 | data cache method and device |
| CN107346307A (en) * | 2016-05-04 | 2017-11-14 | 北京京东尚科信息技术有限公司 | Distributed cache system and method |
| CN109416620A (en) * | 2016-05-04 | 2019-03-01 | 净睿存储股份有限公司 | Storage cluster |
| CN109416620B (en) * | 2016-05-04 | 2022-05-10 | 净睿存储股份有限公司 | Storage cluster |
| CN107656697A (en) * | 2016-07-26 | 2018-02-02 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus of operation data on a storage medium |
| CN107948233A (en) * | 2016-10-13 | 2018-04-20 | 华为技术有限公司 | The method of processing write requests or read request, interchanger, control node |
| CN107948233B (en) * | 2016-10-13 | 2021-01-08 | 华为技术有限公司 | Method for processing write request or read request, switch and control node |
| CN106776001A (en) * | 2016-11-14 | 2017-05-31 | 天津南大通用数据技术股份有限公司 | The location mode and device of a kind of distributed experiment ＆ measurement system data |
| CN106844487A (en) * | 2016-12-23 | 2017-06-13 | 航天星图科技（北京）有限公司 | A kind of distributed memory system replicated based on storehouse |
| CN106936899A (en) * | 2017-02-25 | 2017-07-07 | 九次方大数据信息集团有限公司 | The collocation method of distributed statistical analysis system and distributed statistical analysis system |
| CN106980540A (en) * | 2017-03-07 | 2017-07-25 | 清华大学 | A kind of computational methods of distributed Multidimensional Discrete data |
| CN107018185A (en) * | 2017-03-28 | 2017-08-04 | 清华大学 | The synchronous method and device of cloud storage system |
| CN106991170A (en) * | 2017-04-01 | 2017-07-28 | 广东浪潮大数据研究有限公司 | A kind of method and apparatus of distributed document capacity equilibrium |
| CN107154966A (en) * | 2017-04-24 | 2017-09-12 | 浙江宇视科技有限公司 | A kind of video data wiring method and device |
| CN107168649A (en) * | 2017-05-05 | 2017-09-15 | 南京城市职业学院 | The method and device of data distribution in a kind of distributed memory system |
| CN107168649B (en) * | 2017-05-05 | 2019-12-17 | 南京城市职业学院 | method and device for data distribution in distributed storage system |
| CN107256132B (en) * | 2017-06-21 | 2019-12-10 | 桂林电子科技大学 | Performance test-based heterogeneous Redis cluster storage allocation method |
| CN107256132A (en) * | 2017-06-21 | 2017-10-17 | 桂林电子科技大学 | A kind of isomery Redis cluster-based storage distribution methods based on performance test |
| CN107623719A (en) * | 2017-08-17 | 2018-01-23 | 深圳市优品壹电子有限公司 | Data capture method and device |
| CN107634853A (en) * | 2017-08-28 | 2018-01-26 | 深圳市云舒网络技术有限公司 | A kind of customizable distributed caching method based on object storage |
| CN107577434A (en) * | 2017-09-14 | 2018-01-12 | 郑州云海信息技术有限公司 | Capacity acquiring method, device, equipment and the readable storage medium storing program for executing of memory node |
| CN107608632B (en) * | 2017-09-21 | 2021-02-02 | 苏州浪潮智能科技有限公司 | Communication method, device and system of distributed storage cluster |
| CN107608632A (en) * | 2017-09-21 | 2018-01-19 | 郑州云海信息技术有限公司 | A kind of communication means, the apparatus and system of distributed storage cluster |
| CN109726600A (en) * | 2017-10-31 | 2019-05-07 | 伊姆西Ip控股有限责任公司 | The system and method for data protection are provided for super fusion infrastructure |
| CN109726600B (en) * | 2017-10-31 | 2023-07-14 | 伊姆西Ip控股有限责任公司 | System and method for providing data protection for super fusion infrastructure |
| CN108052655B (en) * | 2017-12-28 | 2020-11-10 | 新华三技术有限公司 | Data writing and reading method |
| CN108052655A (en) * | 2017-12-28 | 2018-05-18 | 新华三技术有限公司 | Data write and read method |
| CN110169040A (en) * | 2018-07-10 | 2019-08-23 | 深圳花儿数据技术有限公司 | Distributed data storage method and system based on multilayer consistency Hash |
| US11461203B2 (en) | 2018-07-10 | 2022-10-04 | Here Data Technology | Systems and methods of handling node failure in a distributed data storage using multi-layer consistent hashing |
| CN110169040B (en) * | 2018-07-10 | 2021-09-28 | 深圳花儿数据技术有限公司 | Distributed data storage method and system based on multilayer consistent hash |
| WO2020010503A1 (en) * | 2018-07-10 | 2020-01-16 | 深圳花儿数据技术有限公司 | Multi-layer consistent hashing-based distributed data storage method and system |
| CN109165262B (en) * | 2018-10-16 | 2022-05-10 | 成都索贝数码科技股份有限公司 | Fragmentation clustering system and fragmentation method of relational large table |
| CN109165262A (en) * | 2018-10-16 | 2019-01-08 | 成都索贝数码科技股份有限公司 | Fragmentation clustering system and fragmentation method of relational large table |
| CN110321374A (en) * | 2018-10-23 | 2019-10-11 | 开采夫(杭州)科技有限公司 | Standard file I/O operations system and method based on distributed network |
| CN110321374B (en) * | 2018-10-23 | 2022-03-25 | 开采夫(杭州)科技有限公司 | Standard file IO operating system and method based on distributed network |
| CN109327539A (en) * | 2018-11-15 | 2019-02-12 | 上海天玑数据技术有限公司 | A kind of distributed block storage system and its data routing method |
| CN109635948A (en) * | 2018-12-19 | 2019-04-16 | 北京达佳互联信息技术有限公司 | On-line training method, apparatus, system and computer readable storage medium |
| CN110046160A (en) * | 2019-03-15 | 2019-07-23 | 中国科学院计算技术研究所 | A kind of consistency Hash storage system construction method based on band |
| CN110334091A (en) * | 2019-05-09 | 2019-10-15 | 重庆天蓬网络有限公司 | A kind of data fragmentation distributed approach, system, medium and electronic equipment |
| CN110324429A (en) * | 2019-07-10 | 2019-10-11 | 中国工商银行股份有限公司 | Backup method and back-up device based on Distributed Storage |
| WO2021003935A1 (en) * | 2019-07-11 | 2021-01-14 | 平安科技（深圳）有限公司 | Data cluster storage method and apparatus, and computer device |
| CN111147546A (en) * | 2019-11-29 | 2020-05-12 | 中科院计算技术研究所大数据研究院 | Method and system for processing edge cluster resources |
| CN111147546B (en) * | 2019-11-29 | 2021-05-14 | 中科院计算技术研究所大数据研究院 | Method and system for processing edge cluster resources |
| CN111125253A (en) * | 2019-12-22 | 2020-05-08 | 北京浪潮数据技术有限公司 | Data synchronization method, device, equipment and storage medium |
| CN111090687B (en) * | 2019-12-24 | 2023-03-10 | 腾讯科技（深圳）有限公司 | Data processing method, device and system and computer readable storage medium |
| CN111090687A (en) * | 2019-12-24 | 2020-05-01 | 腾讯科技（深圳）有限公司 | Data processing method, device and system and computer readable storage medium |
| CN111309262B (en) * | 2020-02-16 | 2021-01-29 | 西安奥卡云数据科技有限公司 | Distributed storage cache reading and writing method |
| CN111309262A (en) * | 2020-02-16 | 2020-06-19 | 西安奥卡云数据科技有限公司 | Distributed storage cache reading and writing method |
| CN111756828A (en) * | 2020-06-19 | 2020-10-09 | 广东浪潮大数据研究有限公司 | Data storage method, device and equipment |
| CN111756828B (en) * | 2020-06-19 | 2023-07-14 | 广东浪潮大数据研究有限公司 | Data storage method, device and equipment |
| CN112422611B (en) * | 2020-09-11 | 2023-04-18 | 深圳市证通电子股份有限公司 | Virtual bucket storage processing method and system based on distributed object storage |
| CN112422611A (en) * | 2020-09-11 | 2021-02-26 | 深圳市证通电子股份有限公司 | Virtual bucket storage processing method and system based on distributed object storage |
| CN112083892A (en) * | 2020-09-25 | 2020-12-15 | 上海依图网络科技有限公司 | Data storage method, device, equipment and medium |
| CN112083892B (en) * | 2020-09-25 | 2021-05-18 | 上海依图网络科技有限公司 | Data storage method, device, equipment and medium |
| CN112383628A (en) * | 2020-11-16 | 2021-02-19 | 北京中电兴发科技有限公司 | Storage gateway resource allocation method based on streaming storage |
| CN113688115A (en) * | 2021-08-29 | 2021-11-23 | 中盾创新档案管理(北京)有限公司 | File big data distributed storage system based on Hadoop |
| CN113688115B (en) * | 2021-08-29 | 2024-02-20 | 中盾创新数字科技(北京)有限公司 | Archive big data distributed storage system based on Hadoop |
| CN114647388A (en) * | 2022-05-24 | 2022-06-21 | 杭州优云科技有限公司 | High-performance distributed block storage system and management method |
| CN114776176A (en) * | 2022-05-24 | 2022-07-22 | 中欣链证数字科技有限公司 | Precious metal storage method, device, equipment and system |
| CN114647388B (en) * | 2022-05-24 | 2022-08-12 | 杭州优云科技有限公司 | Distributed block storage system and management method |
| CN114776176B (en) * | 2022-05-24 | 2024-05-14 | 北京金玖银玖数字科技有限公司 | Noble metal storage method, device, equipment and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103929500A (en) | Method for data fragmentation of distributed storage system | |
| US10956601B2 (en) | Fully managed account level blob data encryption in a distributed storage environment | |
| CN111031096B (en) | Distributed storage system construction method based on mimicry defense | |
| US9424151B2 (en) | Disk failure recovery for virtual disk with policies | |
| US10659225B2 (en) | Encrypting existing live unencrypted data using age-based garbage collection | |
| US20160004450A1 (en) | Storage system with virtual disks | |
| US20160004449A1 (en) | Storage system with virtual disks | |
| US20160004480A1 (en) | Storage system with virtual disks | |
| US20160004451A1 (en) | Storage system with virtual disks | |
| US20160004603A1 (en) | Storage system with virtual disks | |
| US20160004466A1 (en) | Convergence of multiple application protocols onto a single storage platform | |
| CN106446159B (en) | A kind of method of storage file, the first virtual machine and name node | |
| US11836513B2 (en) | Transitioning volumes between storage virtual machines | |
| CN103763383A (en) | Integrated cloud storage system and storage method thereof | |
| CN103455577A (en) | Multi-backup nearby storage and reading method and system of cloud host mirror image file | |
| CN103942112A (en) | Magnetic disk fault-tolerance method, device and system | |
| WO2016004120A2 (en) | Storage system with virtual disks | |
| US9984139B1 (en) | Publish session framework for datastore operation records | |
| CN103037004A (en) | Implement method and device of cloud storage system operation | |
| CN104735110A (en) | Metadata management method and system | |
| US11675503B1 (en) | Role-based data access | |
| CN105404565B (en) | A kind of dual-active data guard method and device | |
| US20200265008A1 (en) | Converting storage objects between formats in a copy-free transition | |
| US20140380091A1 (en) | Information processing apparatus, computer-readable recording medium having stored program for controlling information processing apparatus, and method for controlling information processing apparatus | |
| US11079960B2 (en) | Object storage system with priority meta object replication |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20140716 |