CN104951561B - HBase hot spots removing method and system - Google Patents
HBase hot spots removing method and system Download PDFInfo
- Publication number
- CN104951561B CN104951561B CN201510398124.4A CN201510398124A CN104951561B CN 104951561 B CN104951561 B CN 104951561B CN 201510398124 A CN201510398124 A CN 201510398124A CN 104951561 B CN104951561 B CN 104951561B
- Authority
- CN
- China
- Prior art keywords
- region
- hot spots
- regionserver
- cluster
- canceled
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/217—Database tuning
Abstract
The invention discloses a kind of HBase hot spots removing method and systems, and wherein the HBase hot spot removing methods include：Cluster is detected, the Region hot spots in cluster are added to hot spot machine list；Judge whether the quantity for the Region hot spots that hot spot machine list includes is 0, if so, terminating flow, if it is not, then selecting a Region hot spot to be canceled；The RegionServer that Region hot spots to be canceled split into two Region or selection one does not have Region hot spots from the cluster is exchanged with the Region hot spots to be canceled；The Region hot spots to be canceled are removed from the hot spot machine list.The present invention can detect and eliminate automatically Region hot spots and RegionServer hot spots, shorten the request time of data, ensure that the normal operation of whole system, have the advantages that quickly and efficiently.
Description
Technical field
The present invention relates to internet big data field more particularly to a kind of a kind of HBase (distributed, opening towards row
Source database) hot spot removing method and system.
Background technology
Include multiple Region in HBase table, certain data are stored in each Region.In cluster
RegionServer may be assigned to one or more Region.It, can due to the difference of the data request amount on each Region
One or more Region hot spots can be will appear, so-called Region hot spots refer to that request amount is more than one first request amount threshold value
Region.The appearance of these Region hot spots can make the request time of data elongated, largely effect on the stability of entire cluster
And availability, systemic breakdown is possibly even caused when serious.
Currently, HBase has found there is no Region hot spots and eliminates tool automatically, included split (Java can only be utilized
A kind of middle sentence) method splits specified Region, but this method, which needs technical staff manually to set, to tear open
The Region divided, cumbersome, efficiency is not also high.
Invention content
The technical problem to be solved by the present invention is to how overcome to be difficult to eliminate lacking for the hot spot in HBase in the prior art
It falls into, a kind of HBase hot spots removing method and system that can eliminate hot spot automatically is provided.
The present invention is to solve above-mentioned technical problem by following technical proposals：
The present invention provides a kind of HBase hot spots removing method, its main feature is that, including：
S1, set the initial value by exchange times of every Region in the HBase table of cluster as 0；
S2, detection cluster, the Region hot spots in cluster are added to a hot spot machine list, Region hot spots are request
Measure the Region more than one first request amount threshold value；
S3, judge whether the quantity for the Region hot spots that the hot spot machine list includes is 0, if so, terminate flow, if
It is no, then execute S4；
S4, select a Region hot spot as Region hot spots to be canceled from the hot spot machine list；
S5, judge the Region hot spots to be canceled by exchange times whether more than one exchange threshold value executed if being more than
S6, if not exceeded, then executing S7；
S6, the Region hot spots to be canceled are split into two Region, then execute S8；
S7, from the cluster selection one do not have the RegionServer of Region hot spots, and by the Region to be canceled
Hot spot is exchanged with the Region of the selected RegionServer without Region hot spots, and updates the Region to be canceled
Hot spot by exchange times；
S8, the Region hot spots to be canceled are removed from the hot spot machine list, then execute S3。
Wherein, step S2Statistical method can be used for statistical analysis to the HBase table of cluster, obtain data point
Cloth and variance determine the first request amount threshold value according to the average load of cluster, and then determine the Region hot spots in cluster.It is logical
Cross step S7, Region hot spots have been switched on other RegionServer.It just will load from a machine, turn as a result,
Have been moved to another machine., and have some Region hot spots because requesting itself amount is excessive, even across multiple exchange, no
Non- Region hot spots can be become can just be converted into non-Region hot spots at this point, can only be split to this kind of Region hot spots.
The HBase hot spots removing method can not only detect Region hot spots automatically, moreover it is possible to eliminate hot spot automatically, shorten the request of data
Time has the advantages that fast and efficiently.
Preferably, S1Further include：
S11, set cluster detected number initial value as 0；
S12, judge whether the detected number of cluster reaches a detection threshold value, if reaching, terminate flow, if not up to,
Then execute S2；
S2It further include the update detected number；
S3It replaces with：S3', judge whether the quantity for the Region hot spots that the hot spot machine list includes is 0, if so, holding
Row S12, if it is not, then executing S4。
Wherein, if having had reached detected number, do not allow to operate cluster again, when limiting the operation of this tool
Between, other times are returned into HBase itself, some mechanism of HBase itself is allowed to be handled, it need not again constantly
Carry out the detection and elimination of Region hot spots.The detected time number limiting end time point of flow, prevents HBase hot spots
Removing method is constantly running, to bring burden to whole system.
Preferably, S7Further include：
S71, choose from the RegionServer of request amount minimum in the cluster Region of request amount minimum, and should
Region hot spots to be canceled are exchanged with the Region being selected.
Preferably, S7Further include：Judge whether wrapped in the RegionServer without Region hot spots in the cluster
The Region with the same table of Region hot spots to be canceled is included, if so, executing S72, if it is not, then executing S71；
S72, the Region hot spots to be canceled are exchanged with the Region of the same table.
Preferably, S72Including：It is minimum that visit capacity is chosen from the Region of the same table of Region hot spots to be canceled
Region, the Region hot spots to be canceled Region minimum with the visit capacity of selection is exchanged.
Preferably, S2Further include that the Region hot spots in the hot spot machine list sort from high to low according to visit capacity；S4
Further include selecting the highest Region hot spots of visit capacity as Region hot spots to be canceled from the hot spot machine list.
Preferably, step S1Later, further include：
T1, obtain cluster HBase table RegionServer hot spots, RegionServer hot spots be request amount be more than one
The RegionServer of second request amount threshold value；
T2, calculate the RegionServer hot spots the highest N number of Region of visit capacity variance, N is positive integer；
T3, judge the variance whether be less than a variance threshold values, if so, execute T4, if it is not, then executing T5；
T4, select from the cluster request amount to be less than the RegionServer of the second request amount threshold value, and should
The highest M of visit capacity of RegionServer hot spots1A Region is less than the second request amount threshold with selected request amount
The Region of the RegionServer of value is exchanged, M1For the positive integer less than or equal to N, then terminate flow；
T5, by the highest M of visit capacity2A Region is added as Region hot spots to hot spot machine list, then executes
S3, M2For the positive integer less than or equal to N.
Wherein, step T1Statistical method can be used for statistical analysis to the HBase table of cluster, obtain data point
Cloth and variance determine the second request amount threshold value according to the average load of cluster, and then determine the RegionServer in cluster
Hot spot.In view of in HBase, although there is some RegionServer not include Region hot spots, it includes
The load summation of Region is excessive, will also result in RegionServer overheats, becomes RegionServer hot spots.Above-mentioned steps
In, if variance is less than the variance threshold values, illustrates that the request amount of the Region on RegionServer hot spots is average, pass through
The highest Region of request amount is exchanged on other RegionServer, changes the distribution of Region on cluster, can make
It obtains RegionServer hot spots and is converted into non-RegionServer hot spots；If variance is not less than the variance threshold values, illustrate
The request amount of Region on RegionServer hot spots is very unequal, has the request amount of individual Region may be larger, that
It can be handled using this kind of Region as Region hot spots.The HBase hot spots removing method can not only be automatic
Detect RegionServer hot spots, moreover it is possible to eliminate hot spot automatically, and make the Hot spots detection of entire HBase more comprehensively.
The present invention also provides a kind of HBase hot spots to eliminate system, its main feature is that, including：
One setup unit, for set the initial value by exchange times of every Region in the HBase table of cluster as
0；
The Region hot spots in cluster are added to a hot spot machine list for detecting cluster for one detection unit,
Region hot spots are the Region that request amount is more than one first request amount threshold value；
One first judging unit, for judging whether the quantity of Region hot spots that the hot spot machine list includes is 0, if
It is then to terminate to call, if it is not, then calling a selecting unit；
The selection unit, for selecting a Region hot spot as Region to be canceled heat from the hot spot machine list
Point；
One second judgment unit, for judge the Region hot spots to be canceled by exchange times whether more than one exchange threshold
Value, if being more than, calls a split cells, if not exceeded, then calling a crosspoint；
Then the split cells calls one to remove list for the Region hot spots to be canceled to be split into two Region
Member；
The crosspoint, for RegionServer of the selection one without Region hot spots from the cluster, and should
Region hot spots to be canceled are exchanged with the Region of the selected RegionServer without Region hot spots, and updating should
Region hot spots to be canceled by exchange times；
The removal unit, for removing the Region hot spots to be canceled from the hot spot machine list, then calling should
First judging unit.
Preferably, the setup unit further includes：
One initial value setting module, for set cluster detected number initial value as 0；
One third judgment module, if reaching, is tied for judging whether the detected number of cluster reaches a detection threshold value
Beam calls, if not up to, calling the detection unit；
The detection unit is additionally operable to update the detected number；
First judging unit replaces with：For judge Region hot spots that the hot spot machine list includes quantity whether
It is 0, if so, the third judgment module is called, if it is not, then calling the selection unit.
Preferably, the crosspoint further includes：
One first Switching Module, it is minimum for choosing request amount from the RegionServer of request amount minimum in the cluster
Region, and the Region hot spots to be canceled are exchanged with the Region being selected.
Preferably, the crosspoint further includes：
One the 4th judgment module, for judge in the RegionServer without Region hot spots in the cluster whether
Include the Region with the same table of Region hot spots to be canceled, if so, one second Switching Module is called, if it is not, then adjusting
With first Switching Module；
Second Switching Module, for exchanging the Region hot spots to be canceled with the Region of the same table.
Preferably, second Switching Module, for being selected from the Region of the same table of Region hot spots to be canceled
The Region that visit capacity is minimum is taken, the Region hot spots to be canceled Region minimum with the visit capacity of selection is exchanged.
Preferably, the detection unit be additionally operable to by the Region hot spots in the hot spot machine list according to visit capacity by height to
Low sequence；The selection unit is additionally operable to select the highest Region hot spots of visit capacity as to be canceled from the hot spot machine list
Region hot spots.
Preferably, further including：
One acquiring unit, the RegionServer hot spots of the HBase table for obtaining cluster, RegionServer hot spots are
Request amount is more than the RegionServer of one second request amount threshold value；
One computing unit, the variance of the highest N number of Region of visit capacity for calculating the RegionServer hot spots, N
For positive integer；
One the 5th judging unit, for judging whether the variance is less than a variance threshold values, if so, a third is called to exchange
Unit, if it is not, then calling an adding device；
The third crosspoint, for selecting a request amount to be less than the second request amount threshold value from the cluster
RegionServer, and by the highest M of the visit capacity of the RegionServer hot spots1A Region and selected request amount are not
Region more than the RegionServer of the second request amount threshold value is exchanged, M1For the positive integer less than or equal to N；
The adding device is used for the highest M of visit capacity2A Region is added as Region hot spots to the hot spot machine
Then first judging unit, M are called in list2For the positive integer less than or equal to N.
On the basis of common knowledge of the art, above-mentioned each optimum condition can be combined arbitrarily to get each preferable reality of the present invention
Example.
The positive effect of the present invention is that：The present invention can detect and eliminate automatically Region hot spots and
RegionServer hot spots shorten the request time of data, ensure that the normal operation of whole system, have quickly and efficiently excellent
Point.
Description of the drawings
Fig. 1 is the flow chart of the HBase hot spot removing methods of embodiment 1.
Fig. 2 is that the HBase hot spots of embodiment 1 eliminate the system schematic of system.
Fig. 3 is the flow chart of the HBase hot spot removing methods of embodiment 2.
Fig. 4 is that the HBase hot spots of embodiment 2 eliminate the system schematic of system.
Fig. 5 is the flow chart of the HBase hot spot removing methods of embodiment 3.
Fig. 6 is that the HBase hot spots of embodiment 3 eliminate the system schematic of system.
Specific implementation mode
It is further illustrated the present invention below by the mode of embodiment, but does not therefore limit the present invention to the reality
It applies among a range.
Embodiment 1
A kind of HBase hot spots removing method, as shown in Figure 1, including：
Step 101 sets the initial value by exchange times of every Region in the HBase table of cluster as 0.
Step 102, detection cluster, it is for statistical analysis to the HBase table of cluster using statistical method, obtain data
Distribution and variance, determine one first request amount threshold value according to the average load of cluster, then first find the region more than threshold value,
Sorted out further according to the regionServer where these region, and is added into hot spot machine list.Also, herein
The region numbers to be operated can be limited, for example, in the case where detecting 20 hot spot region, can handle first
Preceding 10 hot spot region therein；
Step 103 judges whether the quantity for the Region hot spots that the hot spot machine list includes is 0, if so, terminating stream
Journey, if it is not, thening follow the steps 104.It is, judge in the hot spot machine list whether to include Region hot spots, if not including,
Then explanation does not have Region hot spots, cluster to stablize, can terminate flow, if including, illustrate that there are Region hot spots, cluster is not
Stablize, needs to eliminate Region hot spots.
Step 104 selects a Region hot spot as Region hot spots to be canceled from the hot spot machine list.Its
In, the Region hot spots as Region hot spots to be canceled can be an optional Region in the hot spot machine list
Hot spot can also be the highest Region hot spot of the amount of may I ask in the hot spot machine list.
Step 105, judge the Region hot spots to be canceled by exchange times whether more than one exchange threshold value, if being more than,
106 are thened follow the steps, if not exceeded, thening follow the steps 107.The Region hot spots to be canceled are split into two by step 106
Then Region executes step 108.
Step 107, selection one does not have the RegionServer of Region hot spots from the cluster, and this is to be canceled
Region hot spots are exchanged with the Region of selected RegionServer, and update being exchanged for the Region hot spots to be canceled
Elimination Region hot spots (are added 1) by number by exchange times.Wherein, it is exchanged with the Region hot spots to be canceled
Region can be any one Region on the RegionServer without Region hot spots, or pass through step
1071 Region selected：
Step 1071, the Region that request amount minimum is chosen from the RegionServer of request amount minimum in the cluster,
And the Region hot spots to be canceled are exchanged with the Region being selected；
Or the Region to be selected by following steps：
Step 1073 judges whether in the RegionServer without Region hot spots in the cluster include being waited for this
The Region for eliminating the same table of Region hot spots, if so, 1072 are thened follow the steps, if it is not, thening follow the steps 1071；
Step 1072 exchanges the Region hot spots to be canceled with the Region of the same table.Wherein, in order to reach
Optimal hot spot eradicating efficacy prevents occurring other Region hot spots again in cluster next time detects, in step 1072 from
With visit capacity minimum Region is preferentially chosen in the Region of the same table of Region hot spots to be canceled, this is to be canceled
The Region hot spots Region minimum with the visit capacity of selection is exchanged.
Such as, the exchange threshold value preset in step 105 is 3, has been exchanged 3 times but still has been Region hot spots
Region can be split as two Region by step 106, and the request amount of single Region is reduced with this, so that cluster
In no longer there are Region hot spots；It is less than 3 times and can be exchanged by step 107 for the Region of Region hot spots by exchange times
To other RegionServer.
Step 108 removes the Region hot spots to be canceled from the hot spot machine list, then executes step 103.
The HBase hot spots of the present embodiment eliminate system, as shown in Fig. 2, including：
One setup unit 201, the initial value by exchange times for setting every Region in the HBase table of cluster
It is 0.
One detection unit 202 carries out statistical using statistical method for detecting cluster to the HBase table of cluster
Analysis obtains data distribution and variance, determines one first request amount threshold value according to the average load of cluster, request amount in cluster is surpassed
The Region for crossing one first request amount threshold value is added as Region hot spots to a hot spot machine list.
One first judging unit 203, for judge Region hot spots that the hot spot machine list includes quantity whether be
0, if so, terminating to call, if it is not, then calling a selecting unit 204.
The selection unit 204, for selecting a Region hot spot as to be canceled from the hot spot machine list
Region hot spots.Wherein, the Region hot spots as Region hot spots to be canceled can arbitrarily be selected in the hot spot machine list
The Region hot spot selected can also be the highest Region hot spot of the amount of may I ask in the hot spot machine list.
One second judgment unit 205, for judge the Region hot spots to be canceled by exchange times whether more than one hand over
Threshold value is changed, if being more than, calls a split cells 206, if not exceeded, then calling a crosspoint 207.
Then the split cells 206 calls one to remove for the Region hot spots to be canceled to be split into two Region
Unit 208.
The crosspoint 207, for RegionServer of the selection one without Region hot spots from the cluster, and will
The Region hot spots to be canceled are exchanged with the Region of selected RegionServer, and update the Region hot spots to be canceled
By exchange times (elimination Region hot spots are added 1 by exchange times).Wherein, it is handed over the Region hot spots to be canceled
The Region changed can be any one Region on the RegionServer without Region hot spots, or pass through
The Region that one first Switching Module is selected：
First Switching Module, it is minimum for choosing request amount from the RegionServer of request amount minimum in the cluster
Region, and the Region hot spots to be canceled are exchanged with the Region being selected；
Or to pass through the Region selected with lower module：
One the 4th judgment module, for judge in the RegionServer without Region hot spots in the cluster whether
Include the Region with the same table of Region hot spots to be canceled, if so, one second Switching Module is called, if it is not, then adjusting
With first Switching Module；
Second Switching Module, for exchanging the Region hot spots to be canceled with the Region of the same table.Its
In, for the hot spot eradicating efficacy being optimal, prevent occurring other Region hot spots again in cluster next time detects, it should
Second Switching Module, which can also be further used for choosing from the Region of the same table of Region hot spots to be canceled, to be accessed
Minimum Region is measured, the Region hot spots to be canceled Region minimum with the visit capacity of selection is exchanged.
Then the removal unit 208 is called for removing the Region hot spots to be canceled from the hot spot machine list
First judging unit 203.
Embodiment 2
The HBase hot spots removing method of the present embodiment and the HBase hot spot removing methods of embodiment 1 are essentially identical, different
Place is that the HBase hot spot removing methods of this implementation are further comprising the steps of：
As shown in figure 3, step 101 is replaced with following steps：
Step 1011 sets the initial value by exchange times of every Region in the HBase table of cluster as 0, and
The initial value of the detected number of cluster is set as 0.
Step 1012 judges whether the detected number of cluster reaches a detection threshold value, if reaching, terminates flow, if
Not up to, 102 ' are thened follow the steps.The detection threshold value can freely be set according to actual needs by technical staff, if technical staff
Cluster of detection is merely desired to, then the detection threshold value can be set as 1, it, can should if technical staff wants repeated detection cluster
Detection threshold value is set as the value of bigger.
Step 102 ', detection cluster and update this and be detected number, using statistical method to the HBase table of cluster into
Row statistical analysis obtains data distribution and variance, one first request amount threshold value is determined according to the average load of cluster, will be in cluster
Request amount is more than that the Region of the first request amount threshold value is added as Region hot spots to a hot spot machine list, by the hot spot
Region hot spots in machine list sort from high to low according to visit capacity.
Step 103 ', judge whether the quantity for the Region hot spots that the hot spot machine list includes is 0, if so, execute
Step 1012, if it is not, thening follow the steps 104 '.
Step 104 ', select the highest Region hot spots of visit capacity as Region to be canceled from the hot spot machine list
Then hot spot executes step 105.
The HBase hot spot removing methods that the HBase hot spots of the present embodiment eliminate system and embodiment 1 are essentially identical, different
Place is that the HBase hot spots of this implementation eliminate system and further include：
As shown in figure 4, the setup unit 201 further includes：
One initial value setting module 2011, for set cluster detected number initial value as 0.
One third judgment module 2012, for judging whether the detected number of cluster reaches a detection threshold value, if reaching,
Then terminate to call, if not up to, calling the detection unit 202.
The detection unit 202 is additionally operable to update the detected number, and the Region hot spots in the hot spot machine list are pressed
It sorts from high to low according to visit capacity.
First judging unit 203 replaces with：Quantity for judging Region hot spots that the hot spot machine list includes
Whether it is 0, if so, the third judgment module 2012 is called, if it is not, then calling the selection unit 204.
The selection unit 204, for selecting the highest Region hot spots of visit capacity as waiting for from the hot spot machine list
Region hot spots are eliminated, the second judgment unit 205 is then called.
Embodiment 3
The HBase hot spots removing method of the present embodiment and the HBase hot spot removing methods of embodiment 1 are essentially identical, no
It is with place, as shown in figure 5, the HBase hot spot removing methods of this implementation further include executing following step after step 101
Suddenly：
Step 001, acquisition data distribution and side for statistical analysis to the HBase table of cluster using statistical method
Difference determines the second request amount threshold value according to the average load of cluster, obtains the RegionServer heat of the HBase table of cluster
Point, RegionServer hot spots are the RegionServer that request amount is more than the second request amount threshold value.
The variance of the highest N number of Region of step 002, the visit capacity for calculating the RegionServer hot spots, N are just whole
Number.The numerical value of N can freely be set by technical staff according to actual conditions.
Step 003 judges whether the variance is less than a variance threshold values, if so, illustrating the request amount phase of N number of Region
To average, execute step 004 has if it is not, then illustrating that the request amount of the Region on RegionServer hot spots is very unequal
The request amount of individual Region may be larger, executes step 005.
Step 004 selects a request amount to be less than the RegionServer of the second request amount threshold value from the cluster, and
By the highest M of the visit capacity of the RegionServer hot spots1The Region of a Region and selected RegionServer is handed over
It changes, M1For the positive integer less than or equal to N, then terminate flow.Wherein, with the M1The selection for the Region that a Region is exchanged
It can be with the selection mode of the Region exchanged with the Region hot spots to be canceled in the step 107 in reference implementation example 1.
Step 005, by the highest M of visit capacity2A Region is added to the hot spot machine list, so as Region hot spots
Step 103, M are executed afterwards2For the positive integer less than or equal to N.
The HBase hot spots of the present embodiment eliminate system and the HBase hot spots elimination system of embodiment 1 is essentially identical, no
It is with place, as shown in fig. 6, it further includes being called after the setup unit 201 that the HBase hot spots of this implementation, which eliminate system,
With lower unit：
One acquiring unit 301, for for statistical analysis to the HBase table of cluster using statistical method, acquisition number
According to distribution and variance, which is determined according to the average load of cluster, obtains the HBase table of cluster
RegionServer hot spots, RegionServer hot spots are the RegionServer that request amount is more than the second request amount threshold value.
One computing unit 302, the side of the highest N number of Region of visit capacity for calculating the RegionServer hot spots
Difference, N are positive integer.
One the 5th judging unit 303, for judging whether the variance is less than a variance threshold values, if so, calling a third
Crosspoint 304, if it is not, then calling an adding device 305.
The third crosspoint 304, for selecting a request amount to be less than the second request amount threshold value from the cluster
RegionServer, and by the highest M of the visit capacity of the RegionServer hot spots1A Region with it is selected
The Region of RegionServer is exchanged, M1For the positive integer less than or equal to N, then terminate to call.
The adding device 305 is used for the highest M of visit capacity2A Region is added as Region hot spots to the hot spot
Then first judging unit 203, M is called in machine list2For the positive integer less than or equal to N.
Although specific embodiments of the present invention have been described above, it will be appreciated by those of skill in the art that these
It is merely illustrative of, protection scope of the present invention is defined by the appended claims.Those skilled in the art is not carrying on the back
Under the premise of from the principle and substance of the present invention, many changes and modifications may be made, but these are changed
Protection scope of the present invention is each fallen with modification.
Claims (12)
1. a kind of HBase hot spots removing method, which is characterized in that including：
S1, set the initial value by exchange times of every Region in the HBase table of cluster as 0；
S2, detection cluster, the Region hot spots in cluster are added to a hot spot machine list, Region hot spots are super for request amount
Cross the Region of one first request amount threshold value；
S3, judge whether the quantity for the Region hot spots that the hot spot machine list includes is 0, if so, terminate flow, if it is not, then
Execute S4；
S4, select a Region hot spot as Region hot spots to be canceled from the hot spot machine list；
S5, judge the Region hot spots to be canceled by exchange times whether more than one exchange threshold value execute S if being more than6If
It is less than, then executes S7；
S6, the Region hot spots to be canceled are split into two Region, then execute S8；
S7, from the cluster selection one do not have Region hot spots RegionServer, and by the Region hot spots to be canceled with
The Region of the selected RegionServer without Region hot spots is exchanged, and updates the Region hot spots to be canceled
By exchange times；
S8, the Region hot spots to be canceled are removed from the hot spot machine list, then execute S3；
Step S1Later, further include：
T1, obtain cluster HBase table RegionServer hot spots, RegionServer hot spots be request amount be more than one second
The RegionServer of request amount threshold value；
T2, calculate the RegionServer hot spots the highest N number of Region of visit capacity variance, N is positive integer；
T3, judge the variance whether be less than a variance threshold values, if so, execute T4, if it is not, then executing T5；
T4, select from the cluster request amount to be less than the RegionServer of the second request amount threshold value, and should
The highest M of visit capacity of RegionServer hot spots1A Region is less than the second request amount threshold with selected request amount
The Region of the RegionServer of value is exchanged, M1For the positive integer less than or equal to N, then terminate flow；
T5, by the highest M of visit capacity2A Region is added as Region hot spots to hot spot machine list, then executes S3, M2For
Positive integer less than or equal to N.
2. HBase hot spots removing method as described in claim 1, which is characterized in that S1Further include：
S11, set cluster detected number initial value as 0；
S12, judge whether the detected number of cluster reaches a detection threshold value, if reaching, terminate flow, if not up to, hold
Row S2；
S2It further include the update detected number；
S3It replaces with：S3', judge whether the quantity for the Region hot spots that the hot spot machine list includes is 0, if so, execute
S12, if it is not, then executing S4。
3. HBase hot spots removing method as described in claim 1, which is characterized in that S7Further include：
S71, choose from the RegionServer of request amount minimum in the cluster Region of request amount minimum, and this is waited disappearing
Except Region hot spots are exchanged with the Region being selected.
4. HBase hot spots removing method as claimed in claim 3, which is characterized in that S7Further include：Judge in the cluster not
Whether include Region with the same table of Region hot spots to be canceled in RegionServer with Region hot spots,
If so, executing S72, if it is not, then executing S71；
S72, the Region hot spots to be canceled are exchanged with the Region of the same table.
5. HBase hot spots removing method as claimed in claim 4, which is characterized in that S72Including：From with the Region to be canceled
The minimum Region of visit capacity is chosen in the Region of the same table of hot spot, by the visit of the Region hot spots and selection to be canceled
The minimum Region of the amount of asking is exchanged.
6. HBase hot spots removing method as described in claim 1, which is characterized in that S2Further including will be in the hot spot machine list
Region hot spots sort from high to low according to visit capacity；S4Further include selecting visit capacity highest from the hot spot machine list
Region hot spots are as Region hot spots to be canceled.
7. a kind of HBase hot spots eliminate system, which is characterized in that including：
One setup unit, for setting the initial value by exchange times of every Region in the HBase table of cluster as 0；
The Region hot spots in cluster are added to a hot spot machine list, Region heat for detecting cluster for one detection unit
Point is the Region that request amount is more than one first request amount threshold value；
One first judging unit, for judging whether the quantity of Region hot spots that the hot spot machine list includes is 0, if so,
Then terminate to call, if it is not, then calling a selecting unit；
The selection unit, for selecting a Region hot spot as Region hot spots to be canceled from the hot spot machine list；
One second judgment unit, for judge the Region hot spots to be canceled by exchange times whether more than one exchange threshold value,
If being more than, a split cells is called, if not exceeded, then calling a crosspoint；
Then the split cells calls one to remove unit for the Region hot spots to be canceled to be split into two Region；
The crosspoint for RegionServer of the selection one without Region hot spots from the cluster, and this is waited disappearing
Except Region hot spots are exchanged with the Region of the selected RegionServer without Region hot spots, and updates this and wait disappearing
Except Region hot spots are by exchange times；
The removal unit, for the Region hot spots to be canceled to be removed from the hot spot machine list, then call this first
Judging unit；
The HBase hot spots eliminate system：
One acquiring unit, the RegionServer hot spots of the HBase table for obtaining cluster, RegionServer hot spots are request
Measure the RegionServer more than one second request amount threshold value；
One computing unit, the variance of the highest N number of Region of visit capacity for calculating the RegionServer hot spots, N are just
Integer；
One the 5th judging unit, for judging whether the variance is less than a variance threshold values, if so, it is single to call a third to exchange
Member, if it is not, then calling an adding device；
The third crosspoint, for selecting a request amount to be less than the second request amount threshold value from the cluster
RegionServer, and by the highest M of the visit capacity of the RegionServer hot spots1A Region and selected request amount are not
Region more than the RegionServer of the second request amount threshold value is exchanged, M1For the positive integer less than or equal to N；
The adding device is used for the highest M of visit capacity2A Region is added as Region hot spots to the hot spot machine list,
Then first judging unit, M are called2For the positive integer less than or equal to N.
8. HBase hot spots as claimed in claim 7 eliminate system, which is characterized in that the setup unit further includes：
One initial value setting module, for set cluster detected number initial value as 0；
One third judgment module, if reaching, terminates to adjust for judging whether the detected number of cluster reaches a detection threshold value
With if not up to, calling the detection unit；
The detection unit is additionally operable to update the detected number；
First judging unit replaces with：For judging whether the quantity of Region hot spots that the hot spot machine list includes is 0,
If so, the third judgment module is called, if it is not, then calling the selection unit.
9. HBase hot spots as claimed in claim 7 eliminate system, which is characterized in that the crosspoint further includes：
One first Switching Module, for choosing request amount minimum from the RegionServer of request amount minimum in the cluster
Region, and the Region hot spots to be canceled are exchanged with the Region being selected.
10. HBase hot spots as claimed in claim 9 eliminate system, which is characterized in that the crosspoint further includes：
One the 4th judgment module, for judge in the RegionServer without Region hot spots in the cluster whether include
It is somebody's turn to do if so, calling one second Switching Module if it is not, then calling with the Region of the same table of Region hot spots to be canceled
First Switching Module；
Second Switching Module, for exchanging the Region hot spots to be canceled with the Region of the same table.
11. HBase hot spots as claimed in claim 10 eliminate system, which is characterized in that second Switching Module, for from
The minimum Region of visit capacity is chosen in the Region of the same table of Region hot spots to be canceled, by the Region to be canceled
The hot spot Region minimum with the visit capacity of selection is exchanged.
12. HBase hot spots as claimed in claim 7 eliminate system, which is characterized in that the detection unit is additionally operable to the hot spot
Region hot spots in machine list sort from high to low according to visit capacity；The selection unit is additionally operable to from the hot spot machine list
It is middle to select the highest Region hot spots of visit capacity as Region hot spots to be canceled.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510398124.4A CN104951561B (en) | 2015-07-08 | 2015-07-08 | HBase hot spots removing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510398124.4A CN104951561B (en) | 2015-07-08 | 2015-07-08 | HBase hot spots removing method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104951561A CN104951561A (en) | 2015-09-30 |
| CN104951561B true CN104951561B (en) | 2018-09-07 |
Family
ID=54166219
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510398124.4A Active CN104951561B (en) | 2015-07-08 | 2015-07-08 | HBase hot spots removing method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104951561B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110019528A (en) * | 2017-12-26 | 2019-07-16 | 中国移动通信集团湖北有限公司 | Database manipulation load-balancing method, device, equipment and medium |
| CN110099082B (en) * | 2018-01-30 | 2022-03-11 | 浙江宇视科技有限公司 | Method and device for reducing Hbase hot spots and readable storage medium |
| CN112988703B (en) * | 2019-12-18 | 2022-09-16 | 中国移动通信集团四川有限公司 | Read-write request balancing method and device |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104158902A (en) * | 2014-08-27 | 2014-11-19 | 浙江宇视科技有限公司 | Method and device of distributing Hbase data blocks based on number of requests |
| CN104243621A (en) * | 2014-10-17 | 2014-12-24 | 浪潮电子信息产业股份有限公司 | Improved HBASE load balancing strategy |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9128949B2 (en) * | 2012-01-18 | 2015-09-08 | Cloudera, Inc. | Memory allocation buffer for reduction of heap fragmentation |
-
2015
- 2015-07-08 CN CN201510398124.4A patent/CN104951561B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104158902A (en) * | 2014-08-27 | 2014-11-19 | 浙江宇视科技有限公司 | Method and device of distributing Hbase data blocks based on number of requests |
| CN104243621A (en) * | 2014-10-17 | 2014-12-24 | 浪潮电子信息产业股份有限公司 | Improved HBASE load balancing strategy |
Non-Patent Citations (2)
| Title |
|---|
| An Optimized Load Balance Based on Data Popularity on HBASE;Linjuan Xia.etc;《2nd International Conference on Information Technology and Electronic Commerce》;20141221;第234-238页 * |
| 基于HBase 的交通流数据实时存储系统;陆婷等;《计算机应用》;20150110;第103-135页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104951561A (en) | 2015-09-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104951561B (en) | HBase hot spots removing method and system | |
| CN110545326B (en) | Cluster load scheduling method and device, electronic equipment and storage medium | |
| CN105530138B (en) | Data monitoring method and device | |
| CN109491788B (en) | Method and device for realizing load balance of virtualization platform | |
| JP2001027986A5 (en) | ||
| CN107800756A (en) | A kind of load-balancing method and load equalizer | |
| CN111045808B (en) | Distributed network task scheduling method and device | |
| CN101963923A (en) | Batch processing multiple method | |
| US20170153819A1 (en) | Method of storing data and data storage managing server | |
| CN104980515B (en) | Message distribution treating method and apparatus in a kind of cloud storage system | |
| Van der Boor et al. | Scalable load balancing in networked systems: Universality properties and stochastic coupling methods | |
| CN106059940B (en) | A kind of flow control methods and device | |
| CN104158902B (en) | A kind of Hbase data blocks distribution method and device based on number of request | |
| CN104219163B (en) | The load-balancing method that a kind of node based on dynamic copies method and dummy node method dynamically moves forward | |
| CN108073408A (en) | The method for updating system and device of self-aided terminal | |
| CN108241639B (en) | A kind of data duplicate removal method | |
| CN104899072B (en) | Fine granularity resource scheduling system based on virtual platform and method | |
| JP7417950B2 (en) | Passage time calculation device, passage time calculation method, and program | |
| CN105373451B (en) | The method and device that a kind of virtual machine is placed | |
| CN109474758A (en) | A kind of processing method and processing device | |
| CN103391225A (en) | Parallel automated testing system for test case in futures and security industry | |
| CN109561083B (en) | Method, device, equipment and storage medium for bypass processing | |
| CN105187498B (en) | The Region distribution methods and system of HBase table | |
| CN106657284A (en) | Data stream processing method and device | |
| CN103869231B (en) | A kind of probe card automatic replacing options of wafer acceptability test |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C41 | Transfer of patent application or patent right or utility model | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20160205 Address after: 200335 Shanghai city Changning District Admiralty Road No. 968 Building No. 16 10 floor Applicant after: SHANGHAI XIECHENG BUSINESS CO., LTD. Address before: 200335 Shanghai City, Changning District Fuquan Road No. 99, Ctrip network technology building Applicant before: Ctrip computer technology (Shanghai) Co., Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |