CN109246250A

CN109246250A - The method for adjusting dummy node quantity according to the change of number of servers

Info

Publication number: CN109246250A
Application number: CN201811327654.XA
Authority: CN
Inventors: 程桂平
Original assignee: Individual
Current assignee: Individual
Priority date: 2018-11-08
Filing date: 2018-11-08
Publication date: 2019-01-18

Abstract

The method for adjusting dummy node quantity according to the change of number of servers, comprising: step 1: it obtains in big data network, the quantity n of the server of initial setting up；Step 2: judging whether n is less than lowest service device quantity, if it is less, executing step 3, otherwise, execute step 4；Preferably, lowest service device quantity therein is equal to any one integer in 5-9.

Description

The method for adjusting dummy node quantity according to the change of number of servers

Technical field

The invention belongs to the load balancing fields of big data.

Background technique

Consistency hash algorithm (Consistent Hashing Algorithm) is a kind of distributed algorithm, is usually used in bearing It carries balanced.Memcached client also selects this algorithm, solves key-value being evenly distributed to numerous Memcached Problem on server.It can replace traditional modulo operation, and additions and deletions Memcached can not be coped with by solving modulo operation The problem of Server, (additions and deletions server will lead to the same key, and in get operation, distribution is really stored less than data Server, hit rate can sharply decline).

But in the prior art, when the cryptographic Hash of the hash algorithm of initial setting up consistency Hash, uses regular length and breathe out The method of uncommon value.

Such method has certain value at the initial stage that big data develops, because of the method for fixed setting, convenient for subsequent Processing, all processing modes all can be carried out corresponding setting as long as keeping a close watch on the fixed value.But there is also many for fixed setting Limitation.If maximum problem is that the length of cryptographic Hash is too long, overhead will increase, and if too short, it will lead to It can not will be on server equiblibrium mass distribution all hash space ring.

Therefore, it is badly in need of a kind of new method, the method that cryptographic Hash length can be adjusted according to number of servers dynamic.

Summary of the invention

The method for adjusting dummy node quantity according to the change of number of servers, comprising:

Step 1: obtaining in big data network, the quantity n of the server of initial setting up；

Step 2: judging whether n is less than lowest service device quantity, if it is less, executing step 3, otherwise, execute step 4；It is excellent Choosing, lowest service device quantity therein is equal to any one integer in 5-9；

Step 3: it needs that dummy node is arranged for server, m dummy node is arranged in each server, and m is greater than 1 integer, Wherein,, N is the pre-set integer of system；Preferably, N therein, any one that can be in 50-75 are whole Number；

Step 4: if n < 75, setting 16 for cryptographic Hash length K, otherwise K=16+;

Step 5: the value space of hash function be 0-() hash space ring on, server or server are set Dummy node；

Step 6: monitoring whether to have increased server newly, step 6 is repeated if not monitored newly-increased server, otherwise Execute step 7；

Step 7: judging that on hash space ring be server or dummy node, if it is server then directly in hash space Newly-increased server addition is got on ring, if it is dummy node, then any one dummy node of random erasure, then at this The newly-increased server is arranged on position；And repeat step 6.

Step 2: judging whether n is less than lowest service device quantity, if it is less, executing step 3, otherwise, execute step 4；It is excellent Choosing, lowest service device quantity therein is equal to 10；

Step 4: if n < 75, setting 16 for cryptographic Hash length K, otherwise K=16+；

It is an advantage of the invention that cryptographic Hash length is arranged, according to the quantity of server or dummy node so as to dynamic Adjust cryptographic Hash length, it is too long or too short to avoid cryptographic Hash length, and can service carry out during, according to server Increase adjust dummy node so that total node is in the reasonable range.

Detailed description of the invention

Fig. 1 is schematic diagram 1 according to an embodiment of the present invention；

Fig. 2 is schematic diagram 2 according to an embodiment of the present invention；

Fig. 3 is schematic diagram 3 according to an embodiment of the present invention；

Fig. 4 is schematic diagram 4 according to an embodiment of the present invention.

Specific embodiment

Consistency Hash principle

In simple terms, entire hash-value space is organized into a virtual annulus by consistency Hash, such as assumes certain hash function H Value space be 0-() (i.e. cryptographic Hash is one K without symbol shaping).It is illustrated below with K=32.

Entire hash space ring is as shown in Figure 1, entire space is organized in the direction of the clock.0 and () in zero point Direction is overlapped.

Each server is subjected to a Hash using H in next step, specifically can choose the ip or host masterpiece of server Hash is carried out for keyword, every machine so just can determine that its position on hash space ring, it is assumed here that will above Three servers use as shown in Figure 2 in the position of annular space after ip Address-Hash.

Next respective server is accessed using following algorithm location data: data key is counted using identical function H Cryptographic Hash h is calculated, logical to determine position of this data on ring according to h, " walking " clockwise along ring from this position, First encounters Server be exactly server that it should be navigated to.

Such as we have tetra- data objects of A, B, C, D, position such as Fig. 3 institute after Hash calculation, on annular space Show: according to consistency hash algorithm, data A can be decided to be on Server 1, and D is decided to be on Server 3, and B, C distinguish It is decided to be on Server 2.

Dummy node principle

In order to solve the problems, such as that this data skew, consistency hash algorithm introduce dummy node mechanism, i.e., to each service Node calculates multiple Hash, this service node, referred to as dummy node are placed in each calculated result position.Specific practice can It is realized with increasing number behind server ip or host name.Such as situation above, we are determined as every server Three dummy nodes are calculated, " Memcached Server 1#1 ", " Memcached Server 1# can be then calculated separately 2”、“Memcached Server 1#3”、“Memcached Server 2#1”、“Memcached Server 2#2”、 The cryptographic Hash of " Memcached Server 2#3 " then forms six dummy nodes, as shown in Figure 4.

Embodiment 1

The setting of cryptographic Hash length will follow following several principles:

Principle 1: excessively long cryptographic Hash can bring two harm, and one is that the calculation amount of Hash calculation can become larger, and increase and be It unites expense, the other is being easy to lead to follow-up storage data so that server is unevenly distributed weighing apparatus on entire hash space ring It is unbalanced.Therefore, cryptographic Hash cannot be too long.

Principle 2: if cryptographic Hash is too short, since in present big data system, the quantity of server is continuous Increase, it is not enough that too short cryptographic Hash will lead to hash space ring.When not enough, need to recalculate whole network data Kazakhstan It is uncommon, and storage is rearranged, this is unacceptable.

Principle 3: if number of servers is very little, can have data skew, need that dummy node is arranged, but if clothes Device quantity of being engaged in is enough, then there is no data skew problems.

In 4: one big data systems of principle, the quantity of the server of initial setting up is generally possible to embody this big data The scale of construction of project, even if having increasing for server during subsequent running, but generally or identical in initial number Number of levels on.

Principle 5: when number of servers increases, the sum of newly-increased server and initial dummy node is controlled, otherwise Quantity is too many, and the probability that will lead to cryptographic Hash conflict rises, and needs constantly to modify server name to avoid cryptographic Hash conflict, this Sample will lead to the decline of system effectiveness.

Step 4: if n < 75, setting 16 for cryptographic Hash length K, otherwise K=16+;

Embodiment 2

Claims

1. the method for adjusting dummy node quantity according to the change of number of servers, comprising:

Step 4: if n < 75, setting 16 for cryptographic Hash length K, otherwise K=16+;

2. the method for adjusting dummy node quantity according to the change of number of servers, comprising:

3. a kind of computer program, for executing any one method in method 1-2.

4. the system for adjusting dummy node quantity according to the change of number of servers, comprising: central processing unit, memory, institute State includes computer program, the computer program, for executing any one method in method 1-2 on memory.