CN110445725B - Method and storage medium for shunting newly-added load node - Google Patents
Method and storage medium for shunting newly-added load node Download PDFInfo
- Publication number
- CN110445725B CN110445725B CN201910609417.0A CN201910609417A CN110445725B CN 110445725 B CN110445725 B CN 110445725B CN 201910609417 A CN201910609417 A CN 201910609417A CN 110445725 B CN110445725 B CN 110445725B
- Authority
- CN
- China
- Prior art keywords
- node
- newly added
- utilization rate
- proportion
- disk
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/12—Avoiding congestion; Recovering from congestion
- H04L47/125—Avoiding congestion; Recovering from congestion by balancing the load, e.g. traffic engineering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/5022—Workload threshold
Abstract
The invention provides a method and a storage medium for shunting newly added load nodes, wherein the method comprises the following steps: each node attaches the disk utilization rate of the server to a heartbeat packet periodically sent to the central node; slowly writing data into the newly added node at a preset speed until the utilization rate of a disk corresponding to the newly added node reaches a first preset value; and after the utilization rate of the disks corresponding to the newly added node reaches a first preset value, calculating the flow distributed to the newly added node according to a formula 1/total proportion fraction x maximum proportion. The invention can not only overcome the problem that the new node is broken down due to overlarge short-term load pressure; and the relative load balance of each node of the system can be achieved as soon as possible after the new node is added, so that the availability of the cluster is obviously improved.
Description
Technical Field
The invention relates to the field of distributed nodes, in particular to a method and a storage medium for shunting newly added load nodes.
Background
In many projects, because of the large amount of data, many systems or software are deployed in a distributed manner. The software that needs to be deployed in a distributed manner is most often deployed in a cluster manner, a plurality of service nodes are implemented in the cluster, each service node provides the same function, a uniform traffic inlet is provided outside the cluster, and traffic is shunted in a certain policy manner, for example, a certain request arrives at a node a this time, and arrives at b node according to a certain rule next time.
In this case, if the service node is not enough, a new service node is needed to provide more traffic throughput. Generally, each cluster has a function of a central node, when a new service node joins, the cluster registers with the central node to provide its own related information, and when a subsequent request comes, the central node can send a request to the new node through a routing policy.
At this time, two strategies are generally adopted in the old mode: (1) polling, each node in turn sends a request. (2) And the weight value of each node is set according to the configuration file, and the request is sent according to the weight. The polling mode can cause the requests to be evenly distributed to each node, and the flow born by the newly added node is always lower than that of the old node, so that the requests of each node are unevenly distributed; the weight mode can balance the new node in a short time, but the weight mode needs to be modified frequently subsequently, which is not intelligent, and meanwhile, if the weight of the newly added node is set to be too large, the newly added node can be caused to crash in a short time due to too large short-term load, which results in that the service is not available.
Therefore, it is necessary to provide a new offloading method, which can relatively balance the load of each service node as much as possible for the situation after a new node is added, thereby avoiding the problem of excessive short-term load pressure.
Disclosure of Invention
The technical problem to be solved by the invention is as follows: the method and the storage medium for shunting the newly added load nodes are provided, so that the load of each service node can be relatively balanced, and the availability of the cluster is improved.
In order to solve the technical problems, the technical scheme adopted by the invention is as follows:
a method for shunting newly-added load nodes comprises the following steps:
each node attaches the disk utilization rate of a server in a heartbeat packet periodically sent to a central node;
slowly writing data into the newly added node at a preset speed until the utilization rate of the disk corresponding to the newly added node reaches a first preset value;
and after the utilization rate of the disk corresponding to the newly added node reaches a first preset value, calculating the flow distributed to the newly added node according to a formula 1/the maximum proportion of the total proportion.
The invention provides another technical scheme as follows:
a computer readable storage medium, having a computer readable program stored thereon, which when executed by a processor, is capable of implementing the steps included in a method of offloading a new-joining load node as described above.
The invention has the beneficial effects that: the central node can master the disk utilization rate of each node, and when a new node is added, the request is processed in a slow data writing mode, so that the collapse caused by overlarge load pressure due to large data volume written in a short period is effectively prevented; when the disk utilization rate of the new node reaches a first preset value, recalculating the flow distributed to the new node according to a formula 1/total proportion fraction-maximum proportion, namely, performing flow distribution in a maximum proportion mode so as to slowly improve the disk utilization rate of the new node and improve the flow throughput capacity of the system; meanwhile, the load of each node can be balanced relatively, and the availability of the cluster is improved on the whole.
Drawings
Fig. 1 is a schematic flowchart of a method for offloading a newly added load node according to an embodiment of the present invention;
fig. 2 is a schematic flowchart of a method for offloading a newly added load node according to a first embodiment of the present invention.
Detailed Description
In order to explain technical contents, achieved objects, and effects of the present invention in detail, the following description is made with reference to the accompanying drawings in combination with the embodiments.
The most key concept of the invention is as follows: when a new node is just added, the request is processed in a slow data writing mode, so that the phenomenon that the load pressure is overlarge in a short period and is broken down is prevented; and when the disk utilization rate of the new node reaches a first preset value, carrying out flow distribution by adopting a maximum proportion mode, and slowly improving the disk utilization rate of the new node.
Referring to fig. 1, the present invention provides a method for offloading a newly added load node, including:
each node attaches the disk utilization rate of the server to a heartbeat packet periodically sent to the central node;
slowly writing data into the newly added node at a preset speed until the utilization rate of the disk corresponding to the newly added node reaches a first preset value;
and after the utilization rate of the disks corresponding to the newly added node reaches a first preset value, calculating the flow distributed to the newly added node according to a formula 1/total proportion fraction x maximum proportion.
Further, the slow writing of data to the newly added node at the preset speed specifically includes:
and allocating the flow with the corresponding proportion to the newly added node according to the first preset value, and slowly writing data into the newly added node at a preset speed.
As can be seen from the above description, taking the first preset value as a traffic distribution standard when a new node is just added has rationality, can ensure that the distributed traffic is moderate and not too large, and supports slow write without affecting the request processing efficiency.
Further, the calculating the flow rate distributed to the newly added node according to the formula 1/total proportion fraction × maximum proportion further includes:
and the rest flow is evenly distributed to other nodes in a polling mode.
It can be known from the above description that the existing flow distribution mode is still adopted for the original other nodes in the system without changing, so that the implementation difficulty can be reduced, and the implementation simplicity can be improved.
Further, when the central node judges that the difference between the disk utilization rates of the servers of the nodes is within a second preset value, the step of evenly distributing the residual flow to other nodes in a polling mode is executed.
It can be known from the above description that when the disk usage rates of the nodes in the system are all equivalent, the existing traffic distribution mode is directly adopted for distribution, and the relative load balance of the nodes in the system can be realized.
Further, the method also comprises the following steps:
after the newly added node is added into the distributed system, acquiring resource information of a server of the newly added node, wherein the resource information comprises a CPU utilization rate, a memory utilization rate and a disk utilization rate;
and when the newly joining node registers to the central node, the resource information is simultaneously sent.
It can be known from the above description that the new node needs to report the resource information of its own server when registering, so that the central node can perform more accurate traffic allocation accordingly.
Further, the first preset value is 3-10%.
As can be seen from the above description, the number is reasonable in arrangement and has high feasibility.
The invention provides another technical scheme as follows:
a computer readable storage medium, having a computer readable program stored thereon, which when executed by a processor, is capable of implementing the steps included in the above-mentioned various methods for offloading a new-joining load node.
As can be understood from the above description, those skilled in the art can understand that all or part of the processes in the above technical solutions can be implemented by instructing related hardware through a computer program, where the program can be stored in a computer-readable storage medium, and when executed, the program can include the processes of the above methods. Meanwhile, after the processes are executed, the beneficial effects of the corresponding method can be realized.
The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.
Example one
Referring to fig. 2, in this embodiment, for a situation of a newly added node in a distributed system, a method for offloading the newly added load node is provided, so that load of each node in the system is relatively balanced, and availability of a cluster is improved. The embodiment is suitable for the software stored in the mongo, cassandra or other databases, and when the cluster deployment is carried out, a plurality of distributed nodes are deployed.
The embodiment specifically comprises the following steps:
s1: after the new node is added into the cluster system, the new node registers to a central node of the cluster and provides relevant information of the new node, wherein the information comprises the name of a server, an intranet IP (Internet protocol), a port and other configuration information;
particularly, in this embodiment, the information further includes resource information, where the resource information includes a CPU utilization rate, a memory utilization rate, and a disk utilization rate of a system of the new node itself. Correspondingly, each node in the cluster system has the capability of acquiring the resource information related to the server.
S2: each node in the cluster system sends a heartbeat packet to the central node at regular time through a communication channel established after being associated with the central node so as to confirm whether each node is alive, namely whether the service can be normally provided.
Particularly, in this embodiment, each node attaches resource information of its own server to each heartbeat packet, and optionally, only needs to attach a disk usage rate.
S3: the central node collects heartbeat packets sent by each node at regular time and monitors the disk utilization rate of each node;
s4: and performing the requested route distribution according to a preset route strategy.
Specifically, when one node is free in the cluster system, all request information is sent to the node;
when the number of the nodes is larger than one, the nodes are allocated according to the proportion of the utilization rate of the disk, and the following detailed description is given:
aiming at the node newly added into the cluster system, at the moment, the utilization rate of a magnetic disk of the new node is generally 0 and is lower than a first preset value, so that a slow data writing mode is adopted, and data are slowly written into the newly added node at a preset speed; the first preset value is optionally 3-10%, and preferably 5%. By adopting the mode of slowly writing data, the phenomenon that the service is unavailable due to the fact that the node is overloaded and crashed in a short period because of writing of an overlarge data volume in a short period can be prevented.
Preferably, before the disk usage of the new node reaches a first preset value, allocating a corresponding proportion of traffic to the new joining node according to the first preset value, for example, splitting the request according to a configured proportion of 5%. For example, there are 2 nodes in the current system: a and b, wherein the corresponding average disk utilization rates are both 50%; a node c is newly added; at this time, the usage rate of the node c is 0; the system determines that the utilization rate of the node c is lower than the preset 5%, and sends some requests to the node c within a period of time in a slow data writing mode.
And when the disk utilization rate corresponding to the new node reaches a preset value, calculating the flow to be distributed to the newly added node according to the formula 1/the maximum proportion of the total proportion fraction. I.e. the largest proportion is used for re-splitting. For example, the disk usage ratio of the nodes a: b: c is 60:50:5, and at this time, the resource occupancy rate of the new node c exceeds 5%, the ratio 60:50:5 is reduced to 11:10:1, and then the traffic of the new node c is allocated according to the traffic ratio of 1/22 × 11, that is, the data with the maximum ratio (11) is used as the traffic ratio, so that the traffic is directed to the node c, and thus the resource of the node c is slowly increased, and the resource occupancy ratio of the existing nodes a and b is slowly approached, and finally a balance is achieved.
And optionally, averagely writing the rest traffic into other nodes in a polling mode. Of course, other ways of allocation may be used.
Preferably, when the central node determines that the difference between the disk utilization rates of the servers of the nodes is within a second preset value, all the traffic of the system is averagely distributed to the nodes in a polling mode. Wherein the second preset value is preferably the same as the first preset value, i.e. 5%.
By combining the shunting strategy and the polling strategy aiming at the new node, the flow is distributed by using an equal sharing mode after the resource occupancy rate of the newly added node is equivalent to that of other nodes, so that the load of each node in the system is relatively balanced.
The embodiment realizes the relative balance of the load of each node in the system, and after a new node is added, the problem of large short-term load pressure is avoided, the load balance of the system is not damaged, and the load balance can be achieved again, so that the availability of the cluster is obviously improved.
Example two
Corresponding to the first embodiment, the present embodiment provides a computer-readable storage medium, where a computer-readable program is stored, and when the program is executed by a processor, the steps included in the method for offloading a new join load node in the first embodiment can be implemented, and specific step contents are not repeated here, and refer to the description in the first embodiment for details.
In summary, the method and the storage medium for shunting the newly added load node provided by the invention can overcome the problem that the new node is broken down due to overlarge short-term load pressure; moreover, the load of each node of the system can be relatively balanced as soon as possible after a new node is added, so that the availability of the cluster is remarkably improved; furthermore, the system also has the advantages of high practicability, low implementation difficulty and the like.
The above description is only an embodiment of the present invention, and not intended to limit the scope of the present invention, and all equivalent changes made by using the contents of the present specification and the drawings, or applied directly or indirectly to the related technical fields, are included in the scope of the present invention.
Claims (5)
1. A method for offloading a newly added load node is characterized by comprising the following steps:
each node attaches the disk utilization rate of a server in a heartbeat packet periodically sent to a central node;
the central node writes data into the newly added node slowly at a preset speed until the utilization rate of a disk corresponding to the newly added node reaches a first preset value;
calculating the flow distributed to the newly added node according to the proportion between the disk utilization rates of all nodes and the maximum proportion of the formula 1/total proportion after the disk utilization rate corresponding to the newly added node reaches a first preset value;
the total proportion number is the sum of numerical values corresponding to the utilization rates of the node disks in the proportion among the utilization rates of the node disks, and the maximum proportion is the maximum value of the numerical values corresponding to the utilization rates of the node disks in the proportion among the utilization rates of the node disks;
and when the central node judges that the difference between the disk utilization rates of the servers of the nodes is within a second preset value, the residual flow is evenly distributed to other nodes in a polling mode.
2. The method for offloading a new load node, according to claim 1, wherein the slow writing of data to the new load node at a preset speed includes:
and distributing the flow with the corresponding proportion to the newly added node according to the first preset value, and slowly writing data into the newly added node at a preset speed.
3. The method for offloading a new-joining load node of claim 1, further comprising:
after the newly added node is added into the distributed system, acquiring resource information of a server per se, wherein the resource information comprises CPU (Central processing Unit) utilization rate, memory utilization rate and disk utilization rate;
and when the newly joining node registers to the central node, the resource information is simultaneously sent.
4. The method according to claim 1, wherein the first predetermined value is 3-10%.
5. A computer readable storage medium having a computer readable program stored thereon, wherein the program is capable of implementing the steps of a method for offloading a new-joining load node of any of claims 1-4 when executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910609417.0A CN110445725B (en) | 2019-07-08 | 2019-07-08 | Method and storage medium for shunting newly-added load node |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910609417.0A CN110445725B (en) | 2019-07-08 | 2019-07-08 | Method and storage medium for shunting newly-added load node |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110445725A CN110445725A (en) | 2019-11-12 |
| CN110445725B true CN110445725B (en) | 2022-07-26 |
Family
ID=68429588
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910609417.0A Active CN110445725B (en) | 2019-07-08 | 2019-07-08 | Method and storage medium for shunting newly-added load node |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110445725B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016165242A1 (en) * | 2015-04-14 | 2016-10-20 | 中兴通讯股份有限公司 | Method of adjusting number of nodes in system and device utilizing same |
| CN106416173A (en) * | 2014-05-30 | 2017-02-15 | 华为技术有限公司 | Resource allocation method and device |
| CN107800744A (en) * | 2016-09-06 | 2018-03-13 | 北京金山云网络技术有限公司 | A kind of service request retransmission method, apparatus and system |
| CN109167699A (en) * | 2018-08-20 | 2019-01-08 | 众安信息技术服务有限公司 | The method and apparatus for handling the state of the node of block chain |
| CN109803024A (en) * | 2019-01-28 | 2019-05-24 | 北京中科晶上科技股份有限公司 | A method of for cluster node network |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7894463B2 (en) * | 2006-11-21 | 2011-02-22 | Oracle America, Inc. | Topological approach for improving network and system utilization in a non-uniform network |
| US10693951B2 (en) * | 2017-06-01 | 2020-06-23 | Salesforce.Com, Inc. | Decentralized, resource aware load distribution in a distributed system |
-
2019
- 2019-07-08 CN CN201910609417.0A patent/CN110445725B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106416173A (en) * | 2014-05-30 | 2017-02-15 | 华为技术有限公司 | Resource allocation method and device |
| WO2016165242A1 (en) * | 2015-04-14 | 2016-10-20 | 中兴通讯股份有限公司 | Method of adjusting number of nodes in system and device utilizing same |
| CN107800744A (en) * | 2016-09-06 | 2018-03-13 | 北京金山云网络技术有限公司 | A kind of service request retransmission method, apparatus and system |
| CN109167699A (en) * | 2018-08-20 | 2019-01-08 | 众安信息技术服务有限公司 | The method and apparatus for handling the state of the node of block chain |
| CN109803024A (en) * | 2019-01-28 | 2019-05-24 | 北京中科晶上科技股份有限公司 | A method of for cluster node network |
Non-Patent Citations (2)
| Title |
|---|
| 一种基于负载均衡的Kubernetes调度改进算法;谭莉,陶宏才;《成都信息工程大学学报》;20190320;第3卷(第34期);第228-231页 * |
| 一种改进的HDFS副本放置策略;陈伟;《长春师范大学学报》;20180420;第37卷(第04期);第15-20页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110445725A (en) | 2019-11-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11546644B2 (en) | Bandwidth control method and apparatus, and device | |
| CN109618002B (en) | Micro-service gateway optimization method, device and storage medium | |
| CN102369688B (en) | Method for adjusting resources dynamically and scheduling device | |
| CN107026907A (en) | A kind of load-balancing method, load equalizer and SiteServer LBS | |
| CN109933431B (en) | Intelligent client load balancing method and system | |
| CN112671928B (en) | Equipment centralized management architecture, load balancing method, electronic equipment and storage medium | |
| CN110764963B (en) | Service exception handling method, device and equipment | |
| CN108881512B (en) | CTDB virtual IP balance distribution method, device, equipment and medium | |
| CN107105013B (en) | File processing method, server, terminal and system | |
| CN103036979A (en) | Server loading balancing method and loading balancer | |
| CN101447939A (en) | Functional distribution method and load balancer | |
| US8356098B2 (en) | Dynamic management of workloads in clusters | |
| CN110244901B (en) | Task allocation method and device and distributed storage system | |
| CN107395458B (en) | System monitoring method and device | |
| CN107979539B (en) | Message processing method and device | |
| CN113810304A (en) | Load balancing method, device, equipment and computer storage medium | |
| US20210048956A1 (en) | Storage system, and method and apparatus for allocating storage resources | |
| WO2020038450A1 (en) | Bandwidth adjustment method and apparatus, communication device, and computer readable storage medium | |
| CN110661708A (en) | Network optimization method, system and network equipment | |
| JP2016051446A (en) | Calculator system, calculator, and load dispersing method and program | |
| CN109981768A (en) | I/o multipath planning method and equipment in distributed network storage system | |
| CN114364031B (en) | Service providing method, device and storage medium | |
| CN110445725B (en) | Method and storage medium for shunting newly-added load node | |
| CN111930710A (en) | Method for distributing big data content | |
| CN112416594A (en) | Micro-service distribution method, electronic equipment and computer storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |