CN104852857B - Distributed data transport method and system based on load balancing - Google Patents
Distributed data transport method and system based on load balancing Download PDFInfo
- Publication number
- CN104852857B CN104852857B CN201410153071.5A CN201410153071A CN104852857B CN 104852857 B CN104852857 B CN 104852857B CN 201410153071 A CN201410153071 A CN 201410153071A CN 104852857 B CN104852857 B CN 104852857B
- Authority
- CN
- China
- Prior art keywords
- node
- transmission
- load balancing
- transmission node
- state
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Abstract
An embodiment of the present invention provides a kind of distributed data transport method and system based on load balancing.This method includes mainly：Each transmission node in distributed data transport system is divided into many levels, include multiple transmission nodes in each level, one load balancing management node is set in distributed data transport system, load balancing management node is controlled according to the node status information of each transmission node received according to data transfer throughput of the load balancing of setting between each transmission node.The embodiment of the present invention, which is realized, to entire data transmission procedure effectively detect and manage, it can be controlled according to data transfer throughput of the load balancing of setting between each transmission node according to the node status information of each transmission node, in real time adjustment distributed data transport system the network architecture, to improve distributed data transport system data transmission efficiency.
Description
Technical field
The present invention relates to distributed data transport technical field more particularly to a kind of distributed datas based on load balancing
Transmission method and system.
Background technology
Information technology is fast-developing in recent years, and web database technology increases very fast.Cloud computing simultaneously, technology of Internet of things
Development also produces a large amount of network data.There is a big chunk data to need long range, length in these network datas
The network transmission of time.Such as the cloud computing platform for data mining, it needs the gathered data from each collection point and transmits
Into cloud platform.Urban traffic control system based on Internet of Things then needs in real time from each key node gathered data in city,
And in real time data stored, merged, being handled etc. with operations.
It is current over long distances, there are mainly two types of prolonged network data transmissions, a kind of is exactly the server applied in network
Middle open interface, for receiving data, each outer net node is by the interface to server transmission data；It is another then be to make
With the method for distributed data transport, by the way that a large amount of, multi-level intermediate caching nodes are arranged, to collected data into
Row is converged and is finally pooled in server.
Recently as the development of internet, there is distributed data transport systems in many service applications, such as
Internet of Things, traffic information management, network monitoring etc..However these distributed data transport systems have some disadvantages.
In this kind of distributed data transport system, the operations such as data needs are transmitted between different nodes, shunt, converging.Along with
The increase of node, the probability for situations such as node failure, node load is excessive occur will greatly increase, and these situations will all be led
Cause loss of data.These distributed data transport systems only could find to ask after wherein some node breaks down failure
Topic, and the data for being at this moment buffered in the node then can not probably be given for change.The net of these distributed data transport systems simultaneously
Network transmission rate is also prodigious restraining factors, when multiple upper one successively point next put transmitting data in parallel successively to some simultaneously
When, which possibly can not handle reception data, so as to cause loss of data.
Invention content
The embodiment provides a kind of distributed data transport method and system based on load balancing, to improve
The efficiency of the data transmission of distributed data transport system.
The present invention provides following schemes：
A kind of distributed data transport method based on load balancing, by each transmission in distributed data transport system
Node is divided into many levels, includes multiple transmission nodes in each level, and one is arranged in distributed data transport system
Load balancing management node, the method specifically include：
The load balancing management node, into row data communication, receives each transmission node and is transmitted across with each transmission node
The node status information come；
The load balancing management node is according to the node status information of each transmission node received, according to setting
Data transfer throughput of the load balancing between each transmission node controls.
The load balancing management node, into row data communication, receives each transmission node and is transmitted across with each transmission node
The node status information come, including：
Heartbeat module is set in each transmission node in distributed data transport system, and each transmission node obtains in real time
Take the node status information of transmission node itself；
Each transmission node at set time intervals by the heartbeat module using remote procedure call protocol to
The load balancing management node sends the heartbeat message for carrying the node status information, and the load balancing management node is logical
It crosses designated port and receives the heartbeat message that each transmission node sends over, obtain the node state carried in the heartbeat message
Information.
The node status information includes meshed network flow rate, node maximum bandwidth, cpu load, memory usage amount.
The load balancing management node is according to the node status information of each transmission node received, according to setting
Data transfer throughput of the load balancing between each transmission node controls, including：
The load balancing management node is not received by the heartbeat that some transmission node sends within a certain period of time
Information, the load balancing management node confirm that the state of some transmission node is failure；
The load balancing management node judges whether to be located at same level, same position with some described transmission node
Unit is set, and state is effective other transmission nodes, if it is, being located at described in load balancing management node selection
Same level, same position unit, and state is the spare of effective other transmission nodes some transmission node as described in
The data traffic for being sent to some transmission node is directed to the standby transport node by transmission node；Otherwise, under execution
State processing step；
The load balancing management node judges whether to be located at same level with some described transmission node, state is
Effective other transmission nodes, if it is, the load balancing management node select it is described positioned at same level, state to have
Standby transport node of the other transmission nodes of effect as some transmission node will be sent to some transmission node
Data traffic is directed to the standby transport node.
The load balancing management node is according to the node status information of each transmission node received, according to setting
Data transfer throughput of the load balancing between each transmission node controls, including：
After the load balancing management node receives the heartbeat message that some transmission node sends over, the heartbeat is obtained
The node status information carried in information judges that the CPU usage amounts of some transmission node are more than according to the node status information
80%, memory usage amount be more than 80%, data cached size account for entire spatial cache size be more than 90% and/or network bandwidth use
Amount is more than 90%, then the load balancing management node confirms that the state of some transmission node is overload；
The load balancing management node judges whether to be located at same level, same position with some described transmission node
Unit is set, and state is effective other transmission nodes, if it is, being located at described in load balancing management node selection
Same level, same position unit, and state is the spare of effective other transmission nodes some transmission node as described in
The all or part of data traffic for being sent to some transmission node is directed to the standby transport section by transmission node
Point；Otherwise, following processing steps are executed；
The load balancing management node judges whether to be located at same level with some described transmission node, state is
Effective other transmission nodes, if it is, the load balancing management node select it is described positioned at same level, state to have
Standby transport node of the other transmission nodes of effect as some transmission node will be sent to some transmission node
All or part of data traffic is directed to the standby transport node.
The load balancing management node is according to the node status information of each transmission node received, according to setting
Data transfer throughput of the load balancing between each transmission node controls, including：
After the load balancing management node receives the heartbeat message that some transmission node subsequently sends over, obtaining should
The node status information carried in heartbeat message judges the state of some transmission node as just according to the node status information
Often, then the load balancing management node will be sent to the data traffic of some transmission node from the standby transport node
Re-direct some described transmission node.
A kind of distributed data transport system based on load balancing, including：Multiple transmission nodes and a load balancing
Management node, each transmission node are divided into many levels, include multiple transmission nodes in each level；
The load balancing management node, for, into row data communication, receiving each transmission section with each transmission node
The node status information that point sends over, according to the node status information of each transmission node received, according to the negative of setting
Data transfer throughput of the balance policy between each transmission node is carried to control.
The transmission node obtains the node status information of transmission node itself in real time for heartbeat module to be arranged；It presses
It is sent out to the load balancing management node using remote procedure call protocol by the heartbeat module according to the time interval of setting
Send the heartbeat message for carrying the node status information；
The load balancing management node, for receiving the heartbeat that each transmission node sends over by designated port
Message obtains the node status information carried in the heartbeat message.
The load balancing management node is sent specifically for being not received by some transmission node within a certain period of time
The heartbeat message to come over confirms that the state of some transmission node is failure；
It judges whether to be located at same level, same position unit with some described transmission node, and state is to have
Other transmission nodes of effect, if it is, selection is described to be located at same level, same position unit, and state is effective
Standby transport node of other transmission nodes as some transmission node will be sent to the data of some transmission node
Flow is directed to the standby transport node；Otherwise, following processing steps are executed；
Judge whether to be located at same level with some described transmission node, state is effective other transmission nodes,
If it is, being located at same level described in selecting, state is effective other transmission nodes some transmission node as described in
The data traffic for being sent to some transmission node is directed to the standby transport node by standby transport node.
The load balancing management node, the heartbeat message sended over specifically for receiving some transmission node
Afterwards, the node status information carried in the heartbeat message is obtained, the CPU of some transmission node is judged according to the node status information
Usage amount is more than that 80%, memory usage amount is more than that account for entire spatial cache size be more than 90% and/or net to 80%, data cached size
Network bandwidth usage amount is more than 90%, then the load balancing management node confirms that the state of some transmission node is overload；
It judges whether to be located at same level, same position unit with some described transmission node, and state is to have
Other transmission nodes of effect, if it is, selection is described to be located at same level, same position unit, and state is effective
Standby transport node of other transmission nodes as some transmission node will be sent to the whole of some transmission node
Or partial data flow is directed to the standby transport node；Otherwise, following processing steps are executed；
Judge whether to be located at same level with some described transmission node, state is effective other transmission nodes,
If it is, being located at same level described in selecting, state is effective other transmission nodes some transmission node as described in
The all or part of data traffic for being sent to some transmission node is directed to the standby transport by standby transport node
Node.
The load balancing management node is believed specifically for receiving the heartbeat that some transmission node subsequently sends over
After breath, the node status information carried in the heartbeat message is obtained, judges that some described transmission saves according to the node status information
The state of point is normal, then re-directs the data traffic for being sent to some transmission node from the standby transport node
Some described transmission node.
The embodiment of the present invention passes through in distributed number it can be seen from the technical solution that embodiments of the invention described above provide
According to load balancing management node is arranged in Transmission system, the load balancing management node with each transmission node by being led to
Letter, obtains the node status information of each transmission node, is effectively detected to entire data transmission procedure to realize
And management, can according to the node status information of each transmission node according to the load balancing of setting to each transmission node
Between data transfer throughput controlled, the network architecture of distributed data transport system is adjusted in real time, to improve point
The efficiency of the data transmission of cloth data transmission system.
Description of the drawings
In order to illustrate the technical solution of the embodiments of the present invention more clearly, required use in being described below to embodiment
Attached drawing be briefly described, it should be apparent that, drawings in the following description are only some embodiments of the invention, for this
For the those of ordinary skill of field, without having to pay creative labor, other are can also be obtained according to these attached drawings
Attached drawing.
Fig. 1 is a kind of net for distributed data transport system based on load-balancing mechanism that the embodiment of the present invention one provides
Network topological structure schematic diagram；
Fig. 2 is that a kind of load balancing management node that the embodiment of the present invention one provides is negative to transmission node progress data traffic
Carry the principle schematic of equilibrium treatment；
Fig. 3 is that the embodiment of the present invention three provides a kind of specific reality of the distributed data transport system based on load balancing
Existing structure chart.
Specific implementation mode
For ease of the understanding to the embodiment of the present invention, done further by taking several specific embodiments as an example below in conjunction with attached drawing
Explanation, and each embodiment does not constitute the restriction to the embodiment of the present invention.
Embodiment one
For above-mentioned analysis, the embodiment of the present invention proposes a kind of distributed data transport side with load-balancing mechanism
Method is monitored the transmission process of data by introducing a load balancing management node, and according to message transmission rate
Data transfer process is adjusted in real time with factors such as node states, the transmission safety of data is ensured, improves data transmission
Fault-tolerance.
A kind of network topology structure for distributed data transport system based on load-balancing mechanism that the embodiment provides
Schematic diagram is as shown in Figure 1.In distributed data transport system shown in Fig. 1, each transmission node is divided into many levels, every
A level includes multiple transmission nodes.A load balancing management node is also set up in distributed data transport system, it should
Load balancing management node is communicated with each transmission node, according to result of communication to entire distributed data transport system
Data transmission flow is measured in real time, and accomplishes to adjust the network architecture in real time, achievees the purpose that ensure data transmission security.
In embodiments of the present invention, heartbeat module is set in each transmission node during distributed transmission, it is described
Load balancing management node, into row data communication, receives the node state that each transmission node sends over each transmission node
Information.Then, the load balancing management node is according to the node status information of each transmission node received, according to setting
Data transfer throughput of the load balancing between each transmission node control.
The place that above-mentioned load balancing management node is monitored each transmission node in distributed data transport system
Reason process is as follows：
1, monitoring of the load balancing management node to each node during distributed transmission.
Heartbeat module is set in each transmission node in distributed data transport system, and each transmission node obtains in real time
Take the node status information of transmission node itself.Each transmission node is adopted by the heartbeat module at set time intervals
Use RPC（Remote Procedure Call Protocol, remote procedure call protocol）To the load balancing management node
Send the heartbeat message for carrying the node status information, the node status information include meshed network flow rate, node most
Big bandwidth, cpu load, memory usage amount.Open designated port is used to receive each transmission in load balancing management node simultaneously
The heartbeat message that node is sent.In order to improve node communication efficiency, each transmission node directly uses RPC to carry out with management node
Communication.
In embodiments of the present invention, the state of transmission node can be divided into failure, overload or effective.When the load is equal
When weighing apparatus management node is not received by the heartbeat message that some transmission node sends within a certain period of time, the load balancing
Management node confirms that the state of some transmission node is failure.Then, above-mentioned load balancing management node judges whether to deposit
It is being located at same level, same position unit with some described transmission node, and state is effective other transmission nodes, such as
Fruit is that then the load balancing management node selection is described is located at same level, same position unit, and state is effective
Standby transport node of other transmission nodes as some transmission node will be sent to the data of some transmission node
Flow is directed to the standby transport node；Otherwise, following processing steps are executed；
The load balancing management node judges whether to be located at same level with some described transmission node, state is
Effective other transmission nodes, if it is, the load balancing management node select it is described positioned at same level, state to have
Standby transport node of the other transmission nodes of effect as some transmission node will be sent to some transmission node
Data traffic is directed to the standby transport node.
When the quantity of above-mentioned other transmission nodes is multiple, then load balancing management node randomly chooses a number of other biographies
A transmission node in defeated node is as standby node.In above-mentioned processing procedure, the preferential transmission node selected and fail
The standby transport node of co-located unit, above-mentioned same position unit can be same computer room etc..
2, the real-time adjustment of the distributed data transport network architecture.
A kind of load balancing management node that the embodiment provides carries out data traffic loads equilibrium treatment to transmission node
Principle schematic it is as shown in Figure 2.The load balancing management node receives the heartbeat letter that each transmission node sends over
After breath, the node status information carried in the heartbeat message is obtained, the node status information of each transmission node is stored in number
According in library.
When the load balancing management node is believed according to the node state of some transmission node stored in above-mentioned database
Breath judges that the CPU usage amounts of some transmission node be more than 80%, memory usage amount are more than that 80%, data cached size accounts for entire caching
Space size is more than 90% and/or network bandwidth usage amount is more than 90%, then the load balancing management node confirm it is described some
The state of transmission node is overload；
The load balancing management node judges whether to be located at same level, same position with some described transmission node
Unit is set, and state is effective other transmission nodes, if it is, being located at described in load balancing management node selection
Same level, same position unit, and state is the spare of effective other transmission nodes some transmission node as described in
The all or part of data traffic for being sent to some transmission node is directed to the standby transport section by transmission node
Point, to mitigate the data traffic pressure of some above-mentioned transmission node；Otherwise, following processing steps are executed；
The load balancing management node judges whether to be located at same level with some described transmission node, state is
Effective other transmission nodes, if it is, the load balancing management node select it is described positioned at same level, state to have
Standby transport node of the other transmission nodes of effect as some transmission node will be sent to some transmission node
All or part of data traffic is directed to the standby transport node, to mitigate the data traffic of some above-mentioned transmission node
Pressure.
Later, after the load balancing management node receives the heartbeat message that some transmission node subsequently sends over,
The node status information carried in the heartbeat message is obtained, the shape of some transmission node is judged according to the node status information
State is normal, then the load balancing management node will be sent to the data traffic of some transmission node from the spare biography
Defeated node re-directs some described transmission node.
When the quantity of above-mentioned other transmission nodes is multiple, then load balancing management node randomly chooses a number of other biographies
A transmission node in defeated node is as standby node.In above-mentioned processing procedure, the preferential transmission node selected and fail
The standby transport node of co-located unit, above-mentioned same position unit can be same computer room etc..
Embodiment two
A kind of load balancing management node that the embodiment provides carries out transmission node the mistake of node failure issue handling
Journey is as follows, including following processing step：
1, each transmission node timing in a manner of RPC communication disappears to the heartbeat of management node transmission carrying node status information
Breath, above-mentioned node status information includes meshed network flow rate, node maximum bandwidth, cpu load, memory usage amount, node
Data cached size accounts for entire spatial cache size；
2, load balancing management node obtains the node state carried in the heartbeat message that each transmission node sends over
Information, in the database by node status information storage；
3, when transmission node A does not send heartbeat message, then load balancing to load balancing management node whithin a period of time
It is failure node that management node, which adjudicates transmission node A,；
4, load balancing management node inquiry whether there is with transmission node A in same physical location for example in same computer room
Other transmission nodes；
If 5, in the presence of with transmission node A same physical location transmission node B, load balancing management node notify institute
There is the node for transmitting data to transmission node A to transfer data to transmission node B；If stochastic searching and transmission node there is no if
Other transmission node Cs of the A in same level；
6, load balancing management node notifies all nodes for transmitting data to transmission node A to transfer data to transmission section
Point C.To complete the processing procedure of transmission node A Problem of Failure.
A kind of load balancing management node that the embodiment provides carries out transmission node the mistake of node overload issue handling
Journey is as follows, including following processing step：
1, each transmission node timing in a manner of RPC communication disappears to the heartbeat of management node transmission carrying node status information
Breath, above-mentioned node status information includes meshed network flow rate, node maximum bandwidth, cpu load, memory usage amount, node
Data cached size accounts for entire spatial cache size；
2, load balancing management node obtains the node state carried in the heartbeat message that each transmission node sends over
Information, in the database by node status information storage；
3, load balancing management node is according to the node status information of each transmission node stored in database, use with
Lower four conditions detect each transmission node：a）The CPU usage amounts of the node are more than 80%；b）The memory usage amount of the node
More than 80%；c）It is more than 90% that nodal cache size of data, which accounts for entire spatial cache size, i.e. mass data has not been transmitted next
Node；d）The network bandwidth usage amount of the node is more than 90%, i.e., toward the data traffic of the node-node transmission close to the node maximum belt
It is wide.
4, when transmission node A meets any one of the above condition, the shunting of load balancing management node is transferred to the transmission
The data traffic of node A, to mitigate the data traffic pressure of transmission node A；
5, load balancing management node inquiry whether there is with transmission node A in same physical location for example in same computer room
Other nodes；
If 6, in the presence of, in the transmission node B of same physical location, load balancing management node selects at random with transmission node A
Take the transmission node C toward the transmission node transmission data, by transmission node C be transferred to the traffic steering of transmission node A to biography
On the transmission node B of the defeated same levels of node A；It is saved in other transmission of same level if the stochastic searching there is no and if node A
Point D；
7, load balancing management node randomly selects the transmission node C toward the transmission node transmission data, by transmission node C
Be transferred to the traffic steering of transmission node A to on the transmission node D of the same levels of transmission node A.To complete transmission node
The processing procedure of A overload problems.
Embodiment three
The distributed data transport system based on load balancing that this embodiment offers a kind of, specific implementation structure is as schemed
Shown in 3, it can specifically include：Multiple transmission nodes and a load balancing management node, each transmission node are divided into more
A level includes multiple transmission nodes in each level；
The load balancing management node, for, into row data communication, receiving each transmission section with each transmission node
The node status information that point sends over；According to the node status information of each transmission node received, according to the negative of setting
Data transfer throughput of the balance policy between each transmission node is carried to control.
Specifically, the transmission node obtains the node state of transmission node itself in real time for heartbeat module to be arranged
Information；Use remote procedure call protocol to the load balancing management by the heartbeat module at set time intervals
Node sends the heartbeat message for carrying the node status information；
The load balancing management node, for receiving the heartbeat that each transmission node sends over by designated port
Message obtains the node status information carried in the heartbeat message.
Specifically, the load balancing management node, specifically for being not received by some transmission within a certain period of time
The heartbeat message that node sends over confirms that the state of some transmission node is failure；
It judges whether to be located at same level, same position unit with some described transmission node, and state is to have
Other transmission nodes of effect, if it is, selection is described to be located at same level, same position unit, and state is effective
Standby transport node of other transmission nodes as some transmission node will be sent to the data of some transmission node
Flow is directed to the standby transport node；Otherwise, following processing steps are executed；
Judge whether to be located at same level with some described transmission node, state is effective other transmission nodes,
If it is, being located at same level described in selecting, state is effective other transmission nodes some transmission node as described in
The data traffic for being sent to some transmission node is directed to the standby transport node by standby transport node.
Specifically, the load balancing management node, the heart sended over specifically for receiving some transmission node
After hop-information, the node status information carried in the heartbeat message is obtained, judges that some transmission saves according to the node status information
The CPU usage amounts of point are more than that 80%, memory usage amount is more than that account for entire spatial cache size be more than 90% to 80%, data cached size
And/or network bandwidth usage amount is more than 90%, then the load balancing management node confirms that the state of some transmission node is
Overload；
It judges whether to be located at same level, same position unit with some described transmission node, and state is to have
Other transmission nodes of effect, if it is, selection is described to be located at same level, same position unit, and state is effective
Standby transport node of other transmission nodes as some transmission node will be sent to the whole of some transmission node
Or partial data flow is directed to the standby transport node；Otherwise, following processing steps are executed；
Judge whether to be located at same level with some described transmission node, state is effective other transmission nodes,
If it is, being located at same level described in selecting, state is effective other transmission nodes some transmission node as described in
The all or part of data traffic for being sent to some transmission node is directed to the standby transport by standby transport node
Node.
Specifically, the load balancing management node, subsequently sends over specifically for receiving some transmission node
Heartbeat message after, obtain the node status information carried in the heartbeat message, according to the node status information judge it is described certain
The state of a transmission node is normal, then will be sent to the data traffic of some transmission node from the standby transport node
Re-direct some described transmission node.
With the system of the embodiment of the present invention carry out the detailed process of the distributed data transport based on load balancing with it is aforementioned
Embodiment of the method is similar, and details are not described herein again.
In conclusion the embodiment of the present invention by distributed data transport system be arranged load balancing management node,
The load balancing management node obtains the node status information of each transmission node by being communicated with each transmission node,
Entire data transmission procedure effectively detect and manage to realize, it can be according to the node shape of each transmission node
State information is controlled according to data transfer throughput of the load balancing of setting between each transmission node, is adjusted in real time
The network architecture of distributed data transport system, to improve distributed data transport system data transmission efficiency.
Load balancing management node in the embodiment of the present invention can in time be sent out according to the node status information of transmission node
Existing failed transmission node and overload transmission node, and the total data flow of failed transmission node is shunted in time, ensure
Flowing to the data traffic of failed transmission node will not lose.In time to overload transmission node all or part of data traffic into
Row shunting, mitigates the data traffic pressure of overload transmission node, ensures that overload transmission node can handle the quantity stream received
Amount.
One of ordinary skill in the art will appreciate that：Attached drawing is the schematic diagram of one embodiment, module in attached drawing or
Flow is not necessarily implemented necessary to the present invention.
As seen through the above description of the embodiments, those skilled in the art can be understood that the present invention can
It is realized by the mode of software plus required general hardware platform.Based on this understanding, technical scheme of the present invention essence
On in other words the part that contributes to existing technology can be expressed in the form of software products, the computer software product
It can be stored in a storage medium, such as ROM/RAM, magnetic disc, CD, including some instructions are used so that a computer equipment
（Can be personal computer, server or the network equipment etc.）Execute the certain of each embodiment or embodiment of the invention
Method described in part.
Each embodiment in this specification is described in a progressive manner, identical similar portion between each embodiment
Point just to refer each other, and each embodiment focuses on the differences from other embodiments.Especially for system or
For system embodiment, since it is substantially similar to the method embodiment, so describing fairly simple, related place is referring to method
The part of embodiment illustrates.System and system embodiment described above is only schematical, wherein the conduct
The unit that separating component illustrates may or may not be physically separated, the component shown as unit can be or
Person may not be physical unit, you can be located at a place, or may be distributed over multiple network units.It can root
According to actual need that some or all of module therein is selected to achieve the purpose of the solution of this embodiment.Ordinary skill
Personnel are without creative efforts, you can to understand and implement.
The foregoing is only a preferred embodiment of the present invention, but scope of protection of the present invention is not limited thereto,
Any one skilled in the art in the technical scope disclosed by the present invention, the change or replacement that can be readily occurred in,
It should be covered by the protection scope of the present invention.Therefore, protection scope of the present invention should be with scope of the claims
Subject to.
Claims (11)
1. a kind of distributed data transport method based on load balancing, which is characterized in that will be in distributed data transport system
Each transmission node be divided into many levels, include multiple transmission nodes in each level, in distributed data transport system
One load balancing management node of middle setting, the method specifically include：
The load balancing management node, into row data communication, receives what each transmission node sended over each transmission node
Node status information；
The load balancing management node is according to the node status information of each transmission node received, according to the load of setting
Data transfer throughput of the balance policy between each transmission node controls, including：
The load balancing management node is not received by the heartbeat message that some transmission node sends within a certain period of time,
The load balancing management node confirms that the state of some transmission node is failure；
The load balancing management node judges whether to be located at same level, same position list with some described transmission node
Member, and state is effective other transmission nodes.
2. the distributed data transport method according to claim 1 based on load balancing, which is characterized in that the load
Balanced management node, into row data communication, receives the node state letter that each transmission node sends over each transmission node
Breath, including：
Heartbeat module is set in each transmission node in distributed data transport system, and each transmission node obtains biography in real time
The defeated node node status information of itself；
Each transmission node at set time intervals by the heartbeat module using remote procedure call protocol to described
Load balancing management node sends the heartbeat message for carrying the node status information, and the load balancing management node passes through finger
Fixed end mouthful receives the heartbeat message that each transmission node sends over, and obtains the node state letter carried in the heartbeat message
Breath.
3. the distributed data transport method according to claim 2 based on load balancing, which is characterized in that the node
Status information includes meshed network flow rate, node maximum bandwidth, cpu load, memory usage amount.
4. the distributed data transport method according to claim 1 or 2 or 3 based on load balancing, which is characterized in that institute
Node status information of the load balancing management node according to each transmission node received is stated, according to the load balancing plan of setting
Data transfer throughput slightly between each transmission node controls, and further includes：
The load balancing management node judges whether to be located at same level, same position list with some described transmission node
Member, and state is effective other transmission nodes, if it is, load balancing management node selection is described positioned at same
Level, same position unit, and state is standby transport of the effective other transmission nodes as some transmission node
The data traffic for being sent to some transmission node is directed to the standby transport node by node；Otherwise, following places are executed
Manage step；
The load balancing management node judges whether to be located at same level with some described transmission node, state is effective
Other transmission nodes, if it is, the load balancing management node select it is described be located at same level, state is effective
Standby transport node of other transmission nodes as some transmission node will be sent to the data of some transmission node
Flow is directed to the standby transport node.
5. the distributed data transport method according to claim 1 or 2 or 3 based on load balancing, which is characterized in that institute
Node status information of the load balancing management node according to each transmission node received is stated, according to the load balancing plan of setting
Data transfer throughput slightly between each transmission node controls, including：
After the load balancing management node receives the heartbeat message that some transmission node sends over, the heartbeat message is obtained
The node status information of middle carrying, according to the node status information judge the CPU usage amounts of some transmission node be more than 80%, it is interior
It is more than that account for entire spatial cache size be more than that 90% and/or network bandwidth usage amount are super to 80%, data cached size to deposit usage amount
90% is crossed, then the load balancing management node confirms that the state of some transmission node is overload；
The load balancing management node judges whether to be located at same level, same position list with some described transmission node
Member, and state is effective other transmission nodes, if it is, load balancing management node selection is described positioned at same
Level, same position unit, and state is standby transport of the effective other transmission nodes as some transmission node
The all or part of data traffic for being sent to some transmission node is directed to the standby transport node by node；It is no
Then, following processing steps are executed；
The load balancing management node judges whether to be located at same level with some described transmission node, state is effective
Other transmission nodes, if it is, the load balancing management node select it is described be located at same level, state is effective
Standby transport node of other transmission nodes as some transmission node will be sent to the whole of some transmission node
Or partial data flow is directed to the standby transport node.
6. the distributed data transport method according to claim 5 based on load balancing, which is characterized in that the load
Balanced management node is according to the node status information of each transmission node received, according to the load balancing of setting to each
Data transfer throughput between a transmission node is controlled, including：
After the load balancing management node receives the heartbeat message that some transmission node subsequently sends over, the heartbeat is obtained
The node status information carried in information, according to the node status information judge the state of some transmission node be it is normal,
Then the load balancing management node will be sent to the data traffic of some transmission node from the standby transport node weight
It is new to be oriented to some described transmission node.
7. a kind of distributed data transport system based on load balancing, which is characterized in that including：Multiple transmission nodes and one
Load balancing management node, the multiple transmission node are divided into many levels, include multiple transmission nodes in each level；
The load balancing management node, for, into row data communication, receiving each transmission node hair with each transmission node
The node status information brought, it is equal according to the load of setting according to the node status information of each transmission node received
Data transfer throughput of the weighing apparatus strategy between each transmission node controls,
The load balancing management node is sended over specifically for being not received by some transmission node within a certain period of time
Heartbeat message, confirm that the state of some transmission node is failure；
It judges whether to be located at same level, same position unit with some described transmission node, and state is effective
Other transmission nodes.
8. the distributed data transport system according to claim 7 based on load balancing, it is characterised in that：
The transmission node obtains the node status information of transmission node itself in real time for heartbeat module to be arranged；According to setting
Fixed time interval is taken using remote procedure call protocol to load balancing management node transmission by the heartbeat module
Heartbeat message with the node status information；
The load balancing management node disappears for receiving the heartbeat that each transmission node sends over by designated port
Breath, obtains the node status information carried in the heartbeat message.
9. the distributed data transport system according to claim 7 or 8 based on load balancing, it is characterised in that：
The load balancing management node judges whether to be located at same level, same position with some described transmission node
Unit is set, and state is effective other transmission nodes, including：
If it is, selection is described to be located at same level, same position unit, and state is that effective other transmission nodes are made
For the standby transport node of some transmission node, the data traffic for being sent to some transmission node is directed to described
Standby transport node；Otherwise, following processing steps are executed；
It judges whether to be located at same level, state with some described transmission node for effective other transmission nodes, if
Be then select it is described positioned at same level, state be the spare of effective other transmission nodes some transmission node as described in
The data traffic for being sent to some transmission node is directed to the standby transport node by transmission node.
10. the distributed data transport system according to claim 9 based on load balancing, it is characterised in that：
The load balancing management node obtains after receiving the heartbeat message that some transmission node sends over
The node status information carried in the heartbeat message is taken, judges that the CPU of some transmission node is used according to the node status information
Amount is more than that account for entire spatial cache size be more than 90% and/or network to 80%, data cached size more than 80%, memory usage amount
Bandwidth usage amount is more than 90%, then the load balancing management node confirms that the state of some transmission node is overload；
It judges whether to be located at same level, same position unit with some described transmission node, and state is effective
Other transmission nodes, if it is, selection is described to be located at same level, same position unit, and state is effective other
Standby transport node of the transmission node as some transmission node, will be sent to some transmission node whole or
Partial data flow is directed to the standby transport node；Otherwise, following processing steps are executed；
It judges whether to be located at same level, state with some described transmission node for effective other transmission nodes, if
Be then select it is described positioned at same level, state be the spare of effective other transmission nodes some transmission node as described in
The all or part of data traffic for being sent to some transmission node is directed to the standby transport section by transmission node
Point.
11. the distributed data transport system according to claim 10 based on load balancing, it is characterised in that：
The load balancing management node, the heartbeat message subsequently sended over specifically for receiving some transmission node
Afterwards, the node status information carried in the heartbeat message is obtained, some described transmission node is judged according to the node status information
State be normal, then the data traffic that will be sent to some transmission node re-directs institute from the standby transport node
State some transmission node.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410153071.5A CN104852857B (en) | 2014-02-14 | 2014-04-16 | Distributed data transport method and system based on load balancing |
Applications Claiming Priority (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410051152 | 2014-02-14 | ||
| CN2014100511524 | 2014-02-14 | ||
| CN201410153071.5A CN104852857B (en) | 2014-02-14 | 2014-04-16 | Distributed data transport method and system based on load balancing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104852857A CN104852857A (en) | 2015-08-19 |
| CN104852857B true CN104852857B (en) | 2018-07-31 |
Family
ID=53852221
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410153071.5A Active CN104852857B (en) | 2014-02-14 | 2014-04-16 | Distributed data transport method and system based on load balancing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104852857B (en) |
Families Citing this family (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105183387A (en) * | 2015-09-14 | 2015-12-23 | 联想(北京)有限公司 | Control method and controller and storage equipment |
| CN105376317A (en) * | 2015-11-19 | 2016-03-02 | 网宿科技股份有限公司 | Load balancing control method and load balancing control device |
| CN105517056B (en) * | 2015-12-07 | 2019-04-02 | 重庆邮电大学 | Interlayer based on network formats switching in multi-standard mobile radio communication cascades load-balancing method |
| US20170171085A1 (en) * | 2015-12-15 | 2017-06-15 | Huawei Technologies Co., Ltd. | Traffic Engineering System and Method for a Communications Network |
| CN105516328A (en) * | 2015-12-18 | 2016-04-20 | 浪潮(北京)电子信息产业有限公司 | Dynamic load balancing method and system, and devices used for distributed storage system |
| CN105721577B (en) * | 2016-02-15 | 2018-12-14 | 安徽大学 | A kind of software-oriented defines the server load balancing method of network |
| CN107231661B (en) * | 2016-03-24 | 2019-09-27 | 中国移动通信集团山东有限公司 | A kind of method and micro-base station of flow equalization |
| CN107404441B (en) * | 2016-05-19 | 2020-02-14 | 华为技术有限公司 | Method and equipment for data stream splitting in slicing network |
| CN107483332B (en) * | 2016-06-07 | 2022-08-02 | 中兴通讯股份有限公司 | Service transmission management method, device and system for subnet connection protection link |
| CN106453122B (en) * | 2016-09-23 | 2019-06-04 | 北京奇虎科技有限公司 | A kind of choosing method and device of Stream Data Transmission node |
| CN106648979A (en) * | 2016-12-09 | 2017-05-10 | 曙光信息产业(北京)有限公司 | Backup method and system for distributed real-time file system |
| CN108965381B (en) * | 2018-05-31 | 2023-03-21 | 康键信息技术（深圳）有限公司 | Nginx-based load balancing implementation method and device, computer equipment and medium |
| CN108600401A (en) * | 2018-08-17 | 2018-09-28 | 青岛海尔能源动力有限公司 | a kind of data transmission method and system |
| CN109272752B (en) * | 2018-10-11 | 2021-03-02 | 南威软件股份有限公司 | Transmission method and transmission system of intersection vehicle picture acquisition system |
| CN110730110A (en) * | 2019-10-18 | 2020-01-24 | 深圳市网心科技有限公司 | Node exception handling method, electronic device, system and medium |
| CN112416888B (en) * | 2020-10-16 | 2024-03-12 | 上海哔哩哔哩科技有限公司 | Dynamic load balancing method and system for distributed file system |
| CN113038065B (en) * | 2021-05-27 | 2021-10-29 | 江苏怀业信息技术股份有限公司 | Data transmission method and data transmission device of video conference system |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN100411401C (en) * | 2002-12-31 | 2008-08-13 | 联想(北京)有限公司 | Method for network equipment self adaption load equalization |
| CN1251110C (en) * | 2002-12-31 | 2006-04-12 | 联想(北京)有限公司 | Method for node load information transfer and node survival detection in machine group |
| US20050015511A1 (en) * | 2003-07-02 | 2005-01-20 | Nec Laboratories America, Inc. | Accelerated large data distribution in overlay networks |
| CN102123087B (en) * | 2011-02-18 | 2014-01-08 | 天津博宇铭基信息科技有限公司 | Method for quickly calibrating multi-level forwarding load balance and multi-level forwarding network system |
| CN103188161B (en) * | 2011-12-30 | 2017-11-21 | 中国移动通信集团公司 | Distributed data loads dispatching method and system |
-
2014
- 2014-04-16 CN CN201410153071.5A patent/CN104852857B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN104852857A (en) | 2015-08-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104852857B (en) | Distributed data transport method and system based on load balancing | |
| CN104243337B (en) | A kind of method and device across cluster load balance | |
| CN105940639B (en) | Data are transmitted between first network node and the second network node | |
| CN103746911B (en) | A kind of SDN structure and its communication means | |
| CN107613030A (en) | A kind of method and system of processing business request | |
| US8533731B2 (en) | Apparatus and method for distrubuting complex events based on correlations therebetween | |
| CN105357246B (en) | Caching method based on information centre's network and system | |
| CN107005431A (en) | The system and method for changing the data plane configuration based on service | |
| CN106411733A (en) | SDN dynamic load balanced scheduling method based on real-time load of link | |
| CN106911584A (en) | It is a kind of based on leaf-flow load sharing method of ridge topological structure, apparatus and system | |
| TW201036353A (en) | Dynamic load balancing of Fibre Channel traffic | |
| CN103259809A (en) | Load balancer, load balancing method and stratified data center system | |
| CN103929782B (en) | A kind of resources balance multi-path route method suitable for industrial wireless sensor network | |
| CN106953926A (en) | A kind of method for routing and device | |
| CN109728981A (en) | A kind of cloud platform fault monitoring method and device | |
| CN108632869A (en) | A kind of transfer control method and device | |
| CN104506337B (en) | Mapping method of virtual network and device based on regional faults prediction | |
| CN109617710A (en) | The big data transmission bandwidth dispatching method for thering is deadline to constrain between data center | |
| CN105515977A (en) | Method, device and system for obtaining transmission path in network | |
| CN105721256B (en) | A kind of Audit data De-weight method of distributed deployment audit platform | |
| CN106385363A (en) | SDN data plane data-flow backup method and device | |
| CN107197039B (en) | A kind of PAAS platform service packet distribution method and system based on CDN | |
| CN114513467B (en) | Network traffic load balancing method and device of data center | |
| CN107018018A (en) | A kind of server delta online upgrading method and system based on SDN | |
| CN105406986B (en) | A kind of content distribution control method and a kind of master controller |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |