CN104243621A - Improved HBASE load balancing strategy - Google Patents
Improved HBASE load balancing strategy Download PDFInfo
- Publication number
- CN104243621A CN104243621A CN201410551035.4A CN201410551035A CN104243621A CN 104243621 A CN104243621 A CN 104243621A CN 201410551035 A CN201410551035 A CN 201410551035A CN 104243621 A CN104243621 A CN 104243621A
- Authority
- CN
- China
- Prior art keywords
- region
- request
- server
- minimum
- load
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000005012 migration Effects 0.000 claims abstract description 29
- 238000013508 migration Methods 0.000 claims abstract description 29
- 238000000034 method Methods 0.000 abstract description 7
- 230000000694 effects Effects 0.000 abstract description 3
- 230000006872 improvement Effects 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000000712 assembly Effects 0.000 description 2
- 238000000429 assembly Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
Landscapes
- Computer And Data Communications (AREA)
Abstract
The invention provides an improved HBASE load balancing strategy, which is characterized in that whether Region migration is needed in each Region server is calculated according to the number of regions in each Region server, and the number of regions needing migration in the Region server is determined; then, determining the specific Region transferred in the Region Server according to the most recent service request number of the Region in each Region Server; the invention effectively takes the two factors of the number of regions in the Region Server and the number of service requests of the regions in the cluster as the basis of HBase load balancing to achieve the purpose of load balancing; the method has the effect of ensuring the balance of the number of regions in each Region Server.
Description
Technical field
The HBASE load balancing of the present invention's specifically a kind of improvement.
Background technology
Along with the continuous appearance of social application platform and ecommerce class internet, applications, indicate that people have stepped into the more senior life epoch.When this mass data, high Concurrency Access, database can provide stable, reliable performance to be vital.But traditional database can not meet the problem of this large data, under this technical background, distributed database has arisen at the historic moment.HBase is the realization of a NoSQL database on Hadoop, be also simultaneously one distributed, towards the PostgreSQL database of row.It provide powerful extensibility and data manipulation performance.In HBase, Master is responsible for distributing on Region to each different Region Server, and Region Server is responsible for storing actual data.If but service request quantity in the load imbalance opened of Region Server or each Region Server is unbalanced, these all will affect the service performance of HBase.Therefore, HBase load balancing is very important.
The HBase load-balancing algorithm of current acquiescence is the Region quantity in simple balanced Region Server.First all Region Server are sorted with the quantity containing Region in this algorithm, if load is identical, sort according to ServerName.Then judge whether to need to carry out load balancing according to default algorithm, if desired load balancing then calculates minimum load and maximum load, the remaining Region exceeding maximum load value is recorded in regionsToMove in each Region Server.Again, the Region Server being less than minimum load is taken out from regionsToMove queue be all added in minimum load.Finally check that all minimum load Region Server that is less than are when being raised to minimum load, regionsToMove queue also has when residue region and then reaches maximum by initial for the Region Server of traversal minimum load.If the region number obtaining needs when also having RS load to be less than minimum value from maximum load Region Server puts into regionsToMove queue, carrying out region distribution, making the Region Server being less than minimum value reach minimum load.But the situation that the method can cause overheated Data distribution8 uneven, therefore, using the index of the nearest service request number of each Regoin as equilibrated dsc data, in overall balance Region Server, Region quantity and these two factors of equilibrated dsc data propose a kind of HBase load balancing of improvement.This strategy can make the number of requests relative equilibrium of Region quantity on every platform Region Server and service, and the situation of improved dsc data access, improves the access performance of service request, reach the object of load balancing simultaneously.
Summary of the invention
The object of the invention is to overcome the deficiencies in the prior art, a kind of HBASE load balancing of improvement is provided, for the foundation of these two factors of the equilibrated dsc data of nearest service request number as load balancing balancing Region quantity and each Region in Region Server, thus realize the object of load balancing, make the number of requests relative equilibrium of Region quantity on every platform Region Server and service simultaneously.
Technical scheme of the present invention realizes in the following manner, first calculate the need of Region migration in each Region Server according to the quantity of Region in each Region Server in its structure, and determine the Region number needing migration in Region Server; Then, the concrete Region moved in Region Server is determined according to the Region nearest service request number in each Region Server; Comprehensively above-mentioned two aspects can to reach in each Region Server the balanced simultaneously also balanced situation of overheated data access of Region number;
Determine that in the module of the Region quantity needing migration in Region Server be calculate minimax load value according to the quantity of Region, and determine according to minimax load value the Region number needing migration in Region Server; First all Region Server are sorted with the quantity containing Region, if load is identical, sort according to ServerName; Then judge whether to need to carry out load balancing according to default algorithm; Again, if desired load balancing then calculates minimum load and maximum load, and wherein minimum load equals Region sum and the ratio of Region Server sum; Maximum load is that if 0, then maximum load equals minimum value, otherwise maximum load equals maximum according to Region sum and Region Server sum remainder; Last is the upper limit according to maximum load value, is worth the Region number in Region Server to be recorded in regionsToMove by being greater than maximum load;
The service request number first nearest according to each Region for nearest the served number of request of Region equilibrated dsc data access modules calculates minimum number of request and largest request number, then selects according to how many decisions of service request number in Region Server the Region needing migration; Minimum number of request is the total number of request according to each Region and the ratio of Region Server sum; Largest request number is total number of request and the Region Server sum remainder of Region, if 0, then largest request number equals to get maximum between minimum number of request and maximum single Region number of request, otherwise largest request number equals the minimum petition adds and get maximum between minimum single Region number of request and maximum single Region number of request; I.e. total number of request %RegionServer sum==0 of largest request number=max{Region? minimum number of request: minimum number of request+minimum single Region number of request, maximum single Region number of request }; Then the Region of needs migration is selected according to the Region number needing in Region Server to move, the now selection of Region determines according to largest request number, get off the value record exceeding largest request number M, then moved away by the Region closest to M in Region Server; Be placed on need Region quantity and need in the Region Server of service request number.
Advantage of the present invention is:
The HBASE load balancing of a modification of the present invention compared to the prior art, effectively according to these two factors of service request quantity of Region quantity and Region in Region Server in cluster as HBase load balancing according to the object reaching load balancing; First the effect that the method is brought is the equilibrium of the Region number that ensure that in each Region Server, increases degree of parallelism when data access; Next is the situation improving overheated data access on this basis, improves the access performance of service request.
Accompanying drawing explanation
Fig. 1 is HBase load balancing flow chart.
Region quantity in each Region Server in Fig. 2 cluster.
Fig. 3 is for the quantity of Region in migration Region Server.
The nearest number of requests of Region in Fig. 4 Region Server.
Execution mode
Be described in detail below below in conjunction with the HBASE load balancing of accompanying drawing to a modification of the present invention.
As shown in Figure 1, the HBASE load balancing of a modification of the present invention carries out equilibrium for the Region quantity on Region Server each in distributed type assemblies and accesses according to the equilibrated dsc data of nearest the served number of request of Region, these two factors are effectively combined the foundation as load balancing, it is advantageous that the number of requests relative equilibrium making Region quantity on every platform Region Server and service, the situation of improved dsc data access, improve the access performance of service request, reach the object of load balancing simultaneously.
Region quantity on balanced each Region Server is the Region quantity obtaining needing migration according to the load-balancing algorithm of acquiescence HBase, makes the Region quantity relative equilibrium on each Region Server.First all Region Server are sorted with the quantity containing Region in this algorithm, if load is identical, sort according to ServerName.Then judge whether to need to carry out load balancing according to default algorithm.Again, if desired load balancing then calculates minimum load and maximum load, finally the Region quantity needed in the Region Server of migration is recorded in regionsToMove.
Go out in each Region Server to have how many number of requests to need migration according to the mean value of service request quantity and max min policy calculation for nearest the served number of request of Region equilibrated dsc data access modules.The number of requests of moving as required selects each Region Sever in regionsToMove to need the specific Region of migration.The selection of specific Region is according to the number of request needing in Region Server to move and needs the number of moving Region to carry out comprehensively, needs the service request number that moves closest in the number of request sum that the while of finding the Region quantity of migration of satisfying the demand, these Region serve recently and this Region Server.The specific Region meeting these two factors is moved to the equilibrium of needs Region number and lack in the Region Server of service request number.
Region quantity on balanced each Region Server refers to the ability meeting Region Server load balancing.It is the migration quantity obtaining Region Server according to the load-balancing algorithm of acquiescence, and this module is only responsible for the Region number of determining to need in Region Server to move.Which Region of concrete migration is then that namely equilibrated dsc data module decides according to newly-increased module.
Data problems of excessive heat is mainly considered for the equilibrated dsc data access modules of nearest the served number of request of Region.Can consider service request situation in Region Server according to this module, the situation of improved dsc data access, improves the access performance of service request.
Embodiment
First disposing distributed type assemblies environment, is that centos6.3 installs hadoop assembly according to official document in operating system.Then hdfs, mapreduce, hbase service is opened.There are 6 Datanode nodes in the cluster, each node have 1 Region Server.HBase load balancing (the RS:Region Server as shown in Figure 1 improved, R:Region), when HBase load balancing will be carried out, first calculate minimax load value and minimax R number according to the quantity of Region, and determine according to minimax R numerical value the Region number needing migration in Region Server.If the R quantity in cluster in RS as shown in Figure 2: RS1 is 22, RS2 be 20, RS3 be 18, RS4 be 20, RS5 be 10, RS6 is 4.By the R quantity sequence in RS, be RS6 4, RS5 10, RS3 18, RS2 20, RS4 20, RS1 22.Minimum R number=R sum/RS sum=15; Maximum R number=R sum %RS sum==0? minimum R number: minimum R number+1, therefore maximum R number=16.According to putting in regionToMove by the R quantity be greater than in the RS of maximum R number shown in Fig. 1, at this moment each RS situation is: RS6 4, RS5 10, RS3 16, RS216, RS4 16, RS1 16; In wish migration RS in regionToMove, the quantity of R as shown in Figure 3: RS3 2, RS2 4, RS4 4, RS1 6.
The equilibrated dsc data access modules of nearest the served number of request of Region determines the R of concrete migration, this rule of determining to move R is added in each RS for nearest the served number of request of migration R quantity, if recently M (exceeding the part of largest request number) time, then select this group R to need the Region moved in RS for this reason.In the case, if the nearest number of requests of the R in cluster in RS as shown in Figure 4: RS1 is 330, RS2 be 330, RS3 be 270, RS4 be 300, RS5 be 150, RS6 is 60.Minimum number of request=R service request sum/RS sum=235; Total number of request %RegionServer sum==0 of largest request number==max{Region? minimum number of request: minimum number of request+minimum single Region number of request, maximum single Region number of request }=235, wherein minimum single Region service request quantity is 0, and maximum single Region service request quantity is 100.Therefore, need the number of request sum of the R in the RS of computation migration (the R number according to the RS of each needs migration), migration is needed as there being two R in RS3, determine that the method for two concrete R is by the R in traversal RS3, every two nearest number of requests of R are sued for peace, choose closest to M, at the total number of request-largest request number=270-235=35 of the M=RS3 of RS3.Then decide two R are put into MoveUpdate, if when needing the nearest number of request of R of moving in RS to be less than minimum number of request, just the random R choosing corresponding number puts into MoveUpdate.The RS that last basis is less than minimum R number takes out in the RS being added to minimum R number from MoveUpdate queue, this process ensures that nearest service request number now meets minimum service number of request, if this is less than service request number in the RS of minimum R number and exceedes minimum service number of request, then can choose the R satisfied condition that service request number is minimum.
Accordingly, the effect that the method is brought first is the equilibrium of the Region number that ensure that in each Region Server, improve the situation of overheated data access on this basis, improve the access performance of service request, reach the number of requests relative equilibrium of Region quantity on every platform Region Server and service simultaneously.
Its processing and fabricating of HBASE load balancing of a modification of the present invention is very simple and convenient, can process to specifications shown in accompanying drawing.
Except the technical characteristic described in specification, be the known technology of those skilled in the art.
Claims (1)
1. the HBASE load balancing improved, it is characterized in that first calculating the need of Region migration in each Region Server according to the quantity of Region in each Region Server, and determine the Region number needing migration in Region Server; Then, the concrete Region moved in Region Server is determined according to the Region nearest service request number in each Region Server; Comprehensively above-mentioned two aspects can to reach in each Region Server the balanced simultaneously also balanced situation of overheated data access of Region number;
Determine that in the module of the Region quantity needing migration in Region Server be calculate minimax load value according to the quantity of Region, and determine according to minimax load value the Region number needing migration in Region Server; First all Region Server are sorted with the quantity containing Region, if load is identical, sort according to ServerName; Then judge whether to need to carry out load balancing according to default algorithm; Again, if desired load balancing then calculates minimum load and maximum load, and wherein minimum load equals Region sum and the ratio of Region Server sum; Maximum load is that if 0, then maximum load equals minimum value, otherwise maximum load equals maximum according to Region sum and Region Server sum remainder; Last is the upper limit according to maximum load value, is worth the Region number in Region Server to be recorded in regionsToMove by being greater than maximum load;
The service request number first nearest according to each Region for nearest the served number of request of Region equilibrated dsc data access modules calculates minimum number of request and largest request number, then selects according to how many decisions of service request number in Region Server the Region needing migration; Minimum number of request is the total number of request according to each Region and the ratio of Region Server sum; Largest request number is total number of request and the Region Server sum remainder of Region, if 0, then largest request number equals to get maximum between minimum number of request and maximum single Region number of request, otherwise largest request number equals the minimum petition adds and get maximum between minimum single Region number of request and maximum single Region number of request; I.e. total number of request %RegionServer sum==0 of largest request number=max{Region? minimum number of request: minimum number of request+minimum single Region number of request, maximum single Region number of request }; Then the Region of needs migration is selected according to the Region number needing in Region Server to move, the now selection of Region determines according to largest request number, get off the value record exceeding largest request number M, then moved away by the Region closest to M in Region Server; Be placed on need Region quantity and need in the Region Server of service request number.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410551035.4A CN104243621A (en) | 2014-10-17 | 2014-10-17 | Improved HBASE load balancing strategy |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410551035.4A CN104243621A (en) | 2014-10-17 | 2014-10-17 | Improved HBASE load balancing strategy |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104243621A true CN104243621A (en) | 2014-12-24 |
Family
ID=52230949
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410551035.4A Pending CN104243621A (en) | 2014-10-17 | 2014-10-17 | Improved HBASE load balancing strategy |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104243621A (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104951561A (en) * | 2015-07-08 | 2015-09-30 | 携程计算机技术（上海）有限公司 | HBase hotspot eliminating method and system |
| CN105187498A (en) * | 2015-08-10 | 2015-12-23 | 携程计算机技术（上海）有限公司 | Region allocation method and system for HBase table |
| CN105630945A (en) * | 2015-12-23 | 2016-06-01 | 浪潮集团有限公司 | HBase region data overheating-based balancing method |
| CN107491448A (en) * | 2016-06-12 | 2017-12-19 | 中国移动通信集团四川有限公司 | A kind of HBase resource adjusting methods and device |
| CN107783720A (en) * | 2016-08-24 | 2018-03-09 | 深圳市深信服电子科技有限公司 | A kind of data balancing method and storage device |
| CN107894942A (en) * | 2017-12-04 | 2018-04-10 | 北京小度信息科技有限公司 | The monitoring method and device of tables of data visit capacity |
| CN107895030A (en) * | 2017-11-21 | 2018-04-10 | 上海帝联网络科技有限公司 | HBase Region merging methods and device, computer-readable recording medium |
| CN110019528A (en) * | 2017-12-26 | 2019-07-16 | 中国移动通信集团湖北有限公司 | Database manipulation load-balancing method, device, equipment and medium |
| CN110515726A (en) * | 2019-08-14 | 2019-11-29 | 苏州浪潮智能科技有限公司 | A kind of database loads equalization methods and device |
| CN110888919A (en) * | 2019-12-04 | 2020-03-17 | 阳光电源股份有限公司 | HBase-based big data statistical analysis method and device |
| CN112395318A (en) * | 2020-11-24 | 2021-02-23 | 福州大学 | Distributed storage middleware based on HBase + Redis |
| CN112988703A (en) * | 2019-12-18 | 2021-06-18 | 中国移动通信集团四川有限公司 | Read-write request balancing method and device |
| CN116069594A (en) * | 2023-03-07 | 2023-05-05 | 武汉工程大学 | Load balancing prediction method, device and system and storage medium |
-
2014
- 2014-10-17 CN CN201410551035.4A patent/CN104243621A/en active Pending
Cited By (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104951561A (en) * | 2015-07-08 | 2015-09-30 | 携程计算机技术（上海）有限公司 | HBase hotspot eliminating method and system |
| CN104951561B (en) * | 2015-07-08 | 2018-09-07 | 上海携程商务有限公司 | HBase hot spots removing method and system |
| CN105187498B (en) * | 2015-08-10 | 2018-05-08 | 携程计算机技术（上海）有限公司 | The Region distribution methods and system of HBase table |
| CN105187498A (en) * | 2015-08-10 | 2015-12-23 | 携程计算机技术（上海）有限公司 | Region allocation method and system for HBase table |
| CN105630945A (en) * | 2015-12-23 | 2016-06-01 | 浪潮集团有限公司 | HBase region data overheating-based balancing method |
| CN107491448A (en) * | 2016-06-12 | 2017-12-19 | 中国移动通信集团四川有限公司 | A kind of HBase resource adjusting methods and device |
| CN107783720A (en) * | 2016-08-24 | 2018-03-09 | 深圳市深信服电子科技有限公司 | A kind of data balancing method and storage device |
| CN107895030A (en) * | 2017-11-21 | 2018-04-10 | 上海帝联网络科技有限公司 | HBase Region merging methods and device, computer-readable recording medium |
| CN107895030B (en) * | 2017-11-21 | 2020-08-14 | 上海帝联网络科技有限公司 | Region merging method and device of HBase and computer readable storage medium |
| CN107894942A (en) * | 2017-12-04 | 2018-04-10 | 北京小度信息科技有限公司 | The monitoring method and device of tables of data visit capacity |
| CN107894942B (en) * | 2017-12-04 | 2020-06-02 | 北京星选科技有限公司 | Method and device for monitoring data table access amount |
| CN110019528A (en) * | 2017-12-26 | 2019-07-16 | 中国移动通信集团湖北有限公司 | Database manipulation load-balancing method, device, equipment and medium |
| CN110515726A (en) * | 2019-08-14 | 2019-11-29 | 苏州浪潮智能科技有限公司 | A kind of database loads equalization methods and device |
| CN110888919A (en) * | 2019-12-04 | 2020-03-17 | 阳光电源股份有限公司 | HBase-based big data statistical analysis method and device |
| CN112988703A (en) * | 2019-12-18 | 2021-06-18 | 中国移动通信集团四川有限公司 | Read-write request balancing method and device |
| CN112988703B (en) * | 2019-12-18 | 2022-09-16 | 中国移动通信集团四川有限公司 | Read-write request balancing method and device |
| CN112395318A (en) * | 2020-11-24 | 2021-02-23 | 福州大学 | Distributed storage middleware based on HBase + Redis |
| CN116069594A (en) * | 2023-03-07 | 2023-05-05 | 武汉工程大学 | Load balancing prediction method, device and system and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104243621A (en) | Improved HBASE load balancing strategy | |
| CN106233276B (en) | The coordination admission control of network-accessible block storage device | |
| CN104850450B (en) | A kind of load-balancing method and system towards mixed cloud application | |
| US9647943B2 (en) | Method for providing tiered load balancing for a hosted voice-over internet protocol (VoIP) private branch exchange (PBX) | |
| Vanitha et al. | Effective resource utilization in cloud environment through a dynamic well-organized load balancing algorithm for virtual machines | |
| CN102611735A (en) | Load balancing method and system of application services | |
| US11336718B2 (en) | Usage-based server load balancing | |
| CN103473365B (en) | A kind of file memory method based on HDFS, device and distributed file system | |
| Alfarrarjeh et al. | Scalable spatial crowdsourcing: A study of distributed algorithms | |
| Razzaghzadeh et al. | Probabilistic modeling to achieve load balancing in expert clouds | |
| Nazir et al. | Cuckoo optimization algorithm based job scheduling using cloud and fog computing in smart grid | |
| JP2014513852A5 (en) | ||
| KR20150019359A (en) | A method for processing data in distributed system | |
| CN103441918A (en) | Self-organizing cluster server system and self-organizing method thereof | |
| CN104283963B (en) | A kind of CDN load-balancing methods of Distributed Cooperative formula | |
| US9755924B2 (en) | Network resource requirements of traffic through a multiple stage switch network | |
| US9519441B1 (en) | Automated storage provisioning and management using a centralized database | |
| CN106101276A (en) | A kind of cluster load balancing method and device | |
| CN105426255B (en) | ReduceTask data locality dispatching method in Hadoop big data platform based on network I/O cost evaluation | |
| CN104917839A (en) | Load balancing method used in cloud computing environment | |
| CN104158902B (en) | A kind of Hbase data blocks distribution method and device based on number of request | |
| KR20190132009A (en) | System for managing transaction based on blockchain and operationg method thereof | |
| Srivastava et al. | A dominance of the channel capacity in load balancing of software defined network | |
| CN105959368B (en) | A kind of method of social activity cloud hot point resource prediction and deployment | |
| JP2017037445A (en) | Server management device and server management method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20141224 |