CN102970349A - Distributed hash table (DHT) network storage load balancing method - Google Patents
Distributed hash table (DHT) network storage load balancing method Download PDFInfo
- Publication number
- CN102970349A CN102970349A CN2012104349737A CN201210434973A CN102970349A CN 102970349 A CN102970349 A CN 102970349A CN 2012104349737 A CN2012104349737 A CN 2012104349737A CN 201210434973 A CN201210434973 A CN 201210434973A CN 102970349 A CN102970349 A CN 102970349A
- Authority
- CN
- China
- Prior art keywords
- key value
- data
- stored
- store
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Landscapes
- Computer And Data Communications (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a distributed hash table (DHT) network storage load balancing method which is characterized in that key values corresponding to nodes in a DHT network and with the data quantity of data stored in the key values exceeding the preset threshold value are split, the data stored in the key values are dispersed to a plurality of sub key values with preset quantity and related to the stored data, and addresses of the plurality of key values are only stored in the sub key values until the data quantity sum of all data stored in each sub key value does not exceed the preset threshold value. By adopting the DHT network storage load balancing method, generation of storage hot spots can be avoided, network node resources are saved, happening of network bottleneck caused by access of the key values with excessive large data storage quantity is effectively avoided, and network load balancing is achieved.
Description
Technical field
The present invention relates to computer network field, relate in particular to a kind of memory load equalization methods of DHT network.Background technology
DHT (Distributed Hash Table) be again distributed hash table.The DHT network is a kind of distributed structured P 2 P network, without any need for server, to be stored in dispersedly on a plurality of nodes according to procotol by key value institute uniquely identified information, can effectively avoid the paralysis of the whole network that server failure brings.The DHT network has decentralization, can expand, the characteristics such as reliable.Well-known DHT agreement and realization have CAN, Chord, Pastry and Tapestry.
DHT network utilisation hash function with key value the data message of unique sign be evenly distributed to a plurality of DHT nodes.But because storage and the enquiry frequency of the data message that each key value identifies in the DHT network is different, the network focus that causes forms network bottleneck thus so can occur that data message that certain key value identifies is accessed frequently.And the data size size of the data message that each key value identifies is also different, if the data size of the data message that certain key value identifies is excessive, can cause mass data at same key value, forms data storage hot issue.Storage and the too frequent network bottleneck that causes of enquiry frequency by a key value can be eased by caching mechanism, and because the data size of the data message that key value identifies is excessive, cause mass data at same key value, and then the data that form storage hot issue, not only consume the storage resources of network node, also can cause the access for this key value to produce network bottleneck.
Therefore, how to solve because the excessive data storage hot issue that causes of the data size of the data message that identifies of key value and for the network bottleneck problem that the access of this key value produces becomes a problem to be solved.
Summary of the invention
Because the defects of prior art, technical problem to be solved by this invention provide a kind of can with
A large amount of different pieces of informations that a key value in the DHT network is stored are distributed to a plurality of sub-key value associated with it, thus avoid storing focus generation, realize the memory load equalization methods of a kind of DHT network of Network Load Balance.
For achieving the above object, the invention provides a kind of memory load equalization methods of DHT network, in the method, if the data volume summation of all data that the corresponding key value of the node in the DHT network is stored surpasses a certain predetermined threshold values M, the data of then described key value being stored are distributed to a plurality of sub-key value that predetermined number associated with it is C, in described key value, only preserve the address of described a plurality of sub-key values, until the data volume summation of all data that each sub-key value is stored is no more than described predetermined threshold values M, specifically comprise the steps:
Step 1), key value to data to be stored is inquired about, be empty or store data if Query Result is the data volume of storing in the described key value, then carry out step 2, if Query Result is the address that stores C key value in the described key value, then from a described C key value, determine at random a key value, and with the key value of definite key value as data to be stored, enter step 1;
Step 2), data to be stored is stored in the described key value, judge whether the data volume summation of all data of storing in the described key value surpasses described reservation threshold M, if surpass described reservation threshold M, then enter step 3, if be no more than described reservation threshold M, then finish storage;
Step 3), described key value is split into the key value of C next stage, C is predetermined number.
Further, the described key value that key value is split into C next stage of described step 3 is specially:
Step 3.1), produce at random C new key value, and one by one each the new key value that produces is inquired about, the data of guaranteeing to store in each new key value are for empty, if the data that certain new key value is stored are not for empty, then regenerate until the new key value number that produces arrives C, wherein C is predetermined number, with C new key value the producing next stage key value as key value to be divided;
Step 3.2), all data that key value to be divided is stored store in C the next stage key value dispersedly, store the address of C next stage key value into key value to be divided.
Further, described predetermined number C is 3, and the threshold values of predetermined data volume summation is 4.
Beneficial effect of the present invention is: the memory load equalization methods of this DHT network divides by the key value that the corresponding storage data volume of the node in DHT network summation is surpassed reservation threshold, its data of storing are distributed to a plurality of sub-key values, thereby avoid storing the generation of focus, save network node resource, effectively prevent owing to the anti-generation of asking the network bottleneck that causes to the excessive key value of data memory space, realize Network Load Balance.
Be described further below with reference to the technique effect of accompanying drawing to design of the present invention, concrete structure and generation, to understand fully purpose of the present invention, feature and effect.
Description of drawings
Fig. 1 is the process chart of the memory load equalization methods of DHT network of the present invention;
Fig. 2 is the key value division flow chart of the memory load equalization methods of DHT network of the present invention;
Embodiment
Specify embodiments of the invention below in conjunction with accompanying drawing.
A kind of memory load equalization methods of DHT network, the method is carried out the key value division for the key value that the corresponding storage data volume of DHT network node summation surpasses reservation threshold, it is split into the new key value of predetermined number, and store dispersedly all data of storing in this key value into each new key value, and the address of only storing all new key values in this key value, so that this key value can be associated with each new key value that its division produces.Until the data volume summation that each key value is stored is no more than predetermined threshold values M.
As shown in Figure 1, in the memory load equalization methods of DHT network of the present invention, when storing data into key value, at first, inquire about the content of storing in the key value of data to be stored, if the content of storing is the address of C key value, then from the address of C key value, select an address, find the corresponding key value in this address, with the key value of this key value as data to be stored, restart this storing process.If the content of storing is data or memory contents is empty, then data to be stored is stored on this key value, and judge whether the data volume summation that this key value is stored surpasses predetermined threshold values M, if be no more than predetermined threshold values M, then storing process finishes; If the data volume summation that this key value is stored surpasses predetermined threshold values M, then this key value is divided, it is C new key value that this key value is split into predetermined number.
That mentions in the said process divides key value, its detailed process is as shown in Figure 2: produce at random C new key value, and one by one each the new key value that produces is inquired about, the data of guaranteeing to store in each new key value are for empty, if the data that certain new key value is stored are not for empty, then regenerate until the new key value number that produces arrives C, with C new key value the producing next stage key value as key value to be divided.Then, all data that key value to be divided is stored store in each next stage key value dispersedly, store the address of C next stage key value into key value to be divided.Wherein, C is the number of predetermined key value division.
Embodiment 1
In the present embodiment, adopt the DHT system of the memory load equalization methods of DHT network of the present invention, wherein, suppose that M is that 4, C is 3, in this DHT system, to the key value K storage data V that stores 10 data
New, because stored 10 data on the key value, surpassed predetermined threshold values 4, so this key value carried out once division, suppose that three sub-key values of key value K are P
1, P
2And P
3, sub-key value P
1On store data V
1, V
2, V
3And V
4, sub-key value P
2On store data V
5, V
6And V
7, and sub-key value P
3On store data V
8, V
9And V
10, to key value K storage data V
NewProcess be: at first, the content of the upper storage of inquiry key value K, Query Result is P
1, P
2And P
3The address, then, select at random a P
1The address, find key value P
1, to key value P
1Storage data V
NewEqually, in method of the present invention, to the flow process of key value storage data, inquiry key value P
1The content of upper storage, Query Result are data V
1, V
2, V
3And V
4So, with data V to be stored
NewStore key value P into
1On, key value P then
1The data of storing are V
1, V
2, V
3, V
4And V
New, data volume has surpassed predetermined threshold values 4, therefore need to be to key value P
1Divide, the process of division is: produce at random 3 new key value P
4, P
5And P
6, and guarantee P by inquiry
4, P
5And P
6Middle institute memory contents is empty, does not store data, deletion P
1In data be V
1, V
2, V
3, V
4And V
New, will divide the key value P that produces
4, P
5And P
6The address store key value P into
1In.Then with key value P
1In former data V
1, V
2, V
3, V
rAnd V
NewStore randomly successively key value P into
1The key value P that division produces
4, P
5And P
6In, for example can be key value P
4Middle storage data V
3,, key value P
5Middle storage data V
2And V
New, and key value P
6Middle storage data V
1And V
4.Wherein, to key value P
4Middle storage data V
3, to key value P
5The middle data V that stores respectively
2And V
NewAnd to key value P
6The middle data V that stores respectively
1And V
4Process need by above-mentioned whole flow process, in the present embodiment, because key value P equally
4, P
5And P
6Middle data volume of storing does not all have to surpass 4, so key value P
4, P
5And P
6Do not need to continue division.So far, storing process finishes.
Embodiment 2
In the present embodiment, adopt the DHT system of the memory load equalization methods of DHT network of the present invention, wherein, suppose that M is that 4, C is 3, in this DHT system, to storing data V
1Key value K storage data V
NewProcess be: at first inquire about the content of the upper storage of key value K, inquiring memory contents is data, then with data V to be stored
NewStore key value K into, because V in the present embodiment
NewWith V
1The data volume sum less than 4, so so far, storing process finishes.
More than introduced in the DHT system of the memory load equalization methods that adopts DHT network of the present invention, detailed process to key value storage data, the below is still take embodiment 2 as example, introduces the process that the data of in the DHT system of the memory load equalization methods of using DHT network of the present invention key value being stored are inquired about.
Storing on the key value K and comprising V
1To V
10And V
New11 data, key value K has three sub-key value P
1, P
2And P
3, wherein, sub-key value P
1Three sub-key value P are arranged again
4, P
5And P
6, key value P
2On store data V
5, V
6And V
7, and key value P
3On store data V
8, V
9And V
10, key value P
4On store data V
3, key value P
5On store data V
2And V
New, and key value P
6On store data V
1And V
4Inquire about the total data of storing on the key value K, at first, the content that key value K stores is inquired about, the content of storing is key value P
1, P
2And P
3The address, then, to key value P
1, P
2And P
3Proceed respectively inquiry, to key value P
1Inquire about, institute's memory contents is key value P
4, P
5And P
6The address, and to key value P
2And P
3The result who inquires about, memory contents are respectively data V
5, V
6, V
7With data V
8, V
9, V
10, with data V
5, V
6, V
7With data V
8, V
9, V
10Output to Query Result, then, continue key value P
1Sub-key value P
4, P
5And P
6Inquire about, obtain data V
3With data V
2, V
NewAnd data V
1, V
4, so far, 11 data that can obtain storing on the key value K.
Predetermined key value division number C in the above execution mode and predetermined amount of data threshold values M are not limited only to adopt among the embodiment, can set up on their own as required.
The memory load equalization methods of the DHT network of the present invention by above introduction, divide by the key value that the corresponding storage data volume of the node in DHT network summation is surpassed reservation threshold, its data of storing are distributed to a plurality of sub-key values, thereby avoid storing the generation of focus, save network node resource, effectively prevent the generation of the network bottleneck that causes owing to the access to the excessive key value of data memory space, realize Network Load Balance.
More than describe preferred embodiment of the present invention in detail.Should be appreciated that those of ordinary skill in the art need not creative work and just can design according to the present invention make many modifications and variations.Therefore, all those skilled in the art all should be in the determined protection range by claims under this invention's idea on the basis of existing technology by the available technical scheme of logical analysis, reasoning, or a limited experiment.
Claims (3)
1. the memory load equalization methods of a DHT network is characterized in that, if the institute of the node in the DHT network is right
The data volume summation of all data that the key value of answering is stored surpasses a certain predetermined threshold values M, then will
The data that described key value is stored are distributed to a plurality of sons pass that predetermined number associated with it is C
Key assignments, the address of only preserving described a plurality of sub-key values in described key value is until each height
The data volume summation of all data that key value is stored is no more than described predetermined threshold values M, concrete bag
Draw together following steps:
Step 1), the key value of data to be stored is inquired about, if Query Result is to store in the described key value
Data volume for empty or store data, then carry out step 2, if Query Result is in the described key value
Store the address of C key value, then from a described C key value, determine at random a key value,
And with the key value of definite key value as data to be stored, enter step 1;
Step 2), data to be stored is stored in the described key value, judge and store in the described key value
Whether the data volume summation of all data surpasses described reservation threshold M, if surpass described reservation threshold M,
Then enter step 3, if be no more than described reservation threshold M, then finish storage;
Step 3), described key value is split into the key value of C next stage, C is predetermined number.
2. the memory load equalization methods of DHT network as claimed in claim 1, wherein, described step 3
The key value that key value is split into C next stage of stating is specially:
Step 3.1), produce at random C new key value, and each new key value to producing one by one
Inquire about, the data of guaranteeing to store in each new key value are for empty, if certain new key
The data that value is stored then regenerate until the new key value number that produces arrives not for empty
C, wherein C is predetermined number, with C new key value producing as key value to be divided
The next stage key value;
All data of step 3.2), key value to be divided being stored store into dispersedly C next
In the level key value, store the address of C next stage key value into key value to be divided.
3. the memory load equalization methods of DHT network as claimed in claim 2, wherein, described predetermined number
C is 3, and the threshold values of predetermined data volume summation is 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210434973.7A CN102970349B (en) | 2012-11-02 | 2012-11-02 | A kind of memory load equalization methods of DHT network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210434973.7A CN102970349B (en) | 2012-11-02 | 2012-11-02 | A kind of memory load equalization methods of DHT network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102970349A true CN102970349A (en) | 2013-03-13 |
| CN102970349B CN102970349B (en) | 2015-11-25 |
Family
ID=47800228
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210434973.7A Expired - Fee Related CN102970349B (en) | 2012-11-02 | 2012-11-02 | A kind of memory load equalization methods of DHT network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102970349B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103246484A (en) * | 2013-05-03 | 2013-08-14 | 华为技术有限公司 | Data storing method, data storing device and data storing system |
| CN105045806A (en) * | 2015-06-04 | 2015-11-11 | 中国科学院信息工程研究所 | Dynamic splitting and maintenance method of quantile query oriented summary data |
| CN107426055A (en) * | 2017-07-31 | 2017-12-01 | 努比亚技术有限公司 | Abnormal flow monitoring method, device and computer-readable recording medium |
| CN108965387A (en) * | 2018-06-09 | 2018-12-07 | 西安电子科技大学 | A kind of equalization methods and system improving P2P data storage survivability |
| CN110427538A (en) * | 2019-07-30 | 2019-11-08 | 北京奇艺世纪科技有限公司 | A kind of data query method, storage method, device and electronic equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1937557A (en) * | 2006-09-05 | 2007-03-28 | 华为技术有限公司 | Structured reciprocal network system and its load query, transfer and resource seeking method |
| CN101414962A (en) * | 2007-10-19 | 2009-04-22 | 华为技术有限公司 | Method and node for implementing load balance |
| CN101645039A (en) * | 2009-06-02 | 2010-02-10 | 中国科学院声学研究所 | Method for storing and reading data based on Peterson graph |
| CN102238202A (en) * | 2010-04-23 | 2011-11-09 | 华为技术有限公司 | Method and device for storing and searching index information |
| US20120203745A1 (en) * | 2011-02-08 | 2012-08-09 | Wavemarket Inc. | System and method for range search over distributive storage systems |
-
2012
- 2012-11-02 CN CN201210434973.7A patent/CN102970349B/en not_active Expired - Fee Related
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1937557A (en) * | 2006-09-05 | 2007-03-28 | 华为技术有限公司 | Structured reciprocal network system and its load query, transfer and resource seeking method |
| CN101414962A (en) * | 2007-10-19 | 2009-04-22 | 华为技术有限公司 | Method and node for implementing load balance |
| CN101645039A (en) * | 2009-06-02 | 2010-02-10 | 中国科学院声学研究所 | Method for storing and reading data based on Peterson graph |
| CN102238202A (en) * | 2010-04-23 | 2011-11-09 | 华为技术有限公司 | Method and device for storing and searching index information |
| US20120203745A1 (en) * | 2011-02-08 | 2012-08-09 | Wavemarket Inc. | System and method for range search over distributive storage systems |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103246484A (en) * | 2013-05-03 | 2013-08-14 | 华为技术有限公司 | Data storing method, data storing device and data storing system |
| CN105045806A (en) * | 2015-06-04 | 2015-11-11 | 中国科学院信息工程研究所 | Dynamic splitting and maintenance method of quantile query oriented summary data |
| CN105045806B (en) * | 2015-06-04 | 2019-04-09 | 中国科学院信息工程研究所 | A kind of summary data Dynamic Division and maintaining method towards quantile inquiry |
| CN107426055A (en) * | 2017-07-31 | 2017-12-01 | 努比亚技术有限公司 | Abnormal flow monitoring method, device and computer-readable recording medium |
| CN108965387A (en) * | 2018-06-09 | 2018-12-07 | 西安电子科技大学 | A kind of equalization methods and system improving P2P data storage survivability |
| CN108965387B (en) * | 2018-06-09 | 2021-04-06 | 西安电子科技大学 | Balancing method and system for improving survivability of P2P data storage |
| CN110427538A (en) * | 2019-07-30 | 2019-11-08 | 北京奇艺世纪科技有限公司 | A kind of data query method, storage method, device and electronic equipment |
| CN110427538B (en) * | 2019-07-30 | 2023-01-20 | 北京奇艺世纪科技有限公司 | Data query method, data storage method, data query device, data storage device and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102970349B (en) | 2015-11-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109343963B (en) | Application access method and device for container cluster and related equipment | |
| CN109684092B (en) | Resource allocation method and device | |
| CA2964461C (en) | Composite partition functions | |
| CN106407207B (en) | Real-time newly-added data updating method and device | |
| JP6542909B2 (en) | File operation method and apparatus | |
| US20180027061A1 (en) | Method and apparatus for elastically scaling virtual machine cluster | |
| US10356150B1 (en) | Automated repartitioning of streaming data | |
| CN109032803B (en) | Data processing method and device and client | |
| CN105991660B (en) | System for resource sharing among multiple cloud storage systems | |
| CN105956035A (en) | File storage method and device | |
| CN110321225B (en) | Load balancing method, metadata server and computer readable storage medium | |
| CN102970349A (en) | Distributed hash table (DHT) network storage load balancing method | |
| CN102970379A (en) | Method for realizing load balance among multiple servers | |
| CN106326239A (en) | Distributed file system and file meta-information management method thereof | |
| US20210132829A1 (en) | Method, device and computer program product of balance of storage space for file system | |
| CN108399175B (en) | Data storage and query method and device | |
| US20230267015A1 (en) | Resource scheduling method and apparatus, electronic device and computer readable storage medium | |
| CN110569302A (en) | method and device for physical isolation of distributed cluster based on lucene | |
| CN105814864B (en) | A kind of input and output I/O request processing method and file server | |
| CN109145053B (en) | Data processing method and device, client and server | |
| CN111008071A (en) | Task scheduling system, method and server | |
| CN110309229A (en) | The data processing method and distributed system of distributed system | |
| CN112422611B (en) | Virtual bucket storage processing method and system based on distributed object storage | |
| CN113873052B (en) | Domain name resolution method, device and equipment of Kubernetes cluster | |
| CN106899515B (en) | Method and device for selecting monitoring service and service monitoring system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20151125 Termination date: 20181102 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |