CN111277510A - Link load balancing method and device, controller and gateway equipment - Google Patents
Link load balancing method and device, controller and gateway equipment Download PDFInfo
- Publication number
- CN111277510A CN111277510A CN202010073494.1A CN202010073494A CN111277510A CN 111277510 A CN111277510 A CN 111277510A CN 202010073494 A CN202010073494 A CN 202010073494A CN 111277510 A CN111277510 A CN 111277510A
- Authority
- CN
- China
- Prior art keywords
- port
- link
- gateway device
- interconnection
- state information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/12—Avoiding congestion; Recovering from congestion
- H04L47/125—Avoiding congestion; Recovering from congestion by balancing the load, e.g. traffic engineering
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0896—Bandwidth or capacity management, i.e. automatically increasing or decreasing capacities
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a link load balancing method, a device, a controller and gateway equipment, wherein the method is executed by the controller; the method comprises the following steps: sending a reading instruction to each gateway device to enable each gateway device to send link state information to the controller; updating a link state information table according to the received link state information; calculating the weighted bandwidth of each WAN port according to the link state information table and the link weighted bandwidth evaluation method, and correspondingly storing the weighted bandwidth into the link state information table; acquiring the role and the weighted bandwidth of each interconnection port according to a link state information table, the role of the interconnection port and a link weighted bandwidth evaluation method, and correspondingly storing the roles and the weighted bandwidth into the link state information table; and generating link control information according to the link state information table, and sending a write instruction carrying the link control information to each gateway device, so that each gateway device performs load balancing processing according to the write instruction. The embodiment of the invention can effectively realize the link load balance of multi-gateway multi-WAN-port networking.
Description
Technical Field
The present invention relates to the field of communications technologies, and in particular, to a link load balancing method and apparatus, a controller, and a gateway device.
Background
With the wide application and explosion of Internet technology, many schools and enterprises choose to access multiple Internet service providers in order to avoid single point failure and performance bottleneck caused by one Internet link, in a small local area network, a single gateway with multiple WAN ports can meet the requirements of multiple links for accessing to external networks such as operators, etc., backup and load balance can be realized among the links in a single gateway system, and in enterprises, campuses or other local area networks with larger scale, a single gateway can become a performance bottleneck, therefore, networking with multiple gateways is a common networking mode, and the egress traffic can be reasonably routed and scheduled according to factors such as different regions, different department architectures, different charging prices, etc., each gateway bears a part of network egress traffic, and simultaneously, multiple WAN ports of each egress gateway access to the external networks, therefore, a network topology structure of N multiplied by M (the number of gateways multiplied by the number of WAN ports) exit links is constructed, and the communication quality and the stable reliability of the whole uplink and downlink network are effectively guaranteed.
In a network topology structure with multiple gateways and multiple WAN ports, backup and load balancing between links are a very critical ring, and currently adopted methods generally include: (1) a central load balancer is arranged between the gateway and the intranet, balanced routing of outlet flow before reaching the gateway is realized by configuring a policy route and other methods in the central load balancer, and when the single-point gateway fails, a corresponding preposed routing policy table can be adjusted; (2) and policy routing is configured on the gateway system to realize balanced routing among a plurality of WAN ports, and when a certain WAN port link is disconnected, balanced routing can be performed among the links of the rest available WAN ports.
The existing link backup and sharing method of multiple gateways and multiple WAN ports has the following defects: for the method (1), all outlet flows need to be scheduled by the central load balancer first, and then data is forwarded to different gateways, the networking mode limits the application in some scenes, the whole network is affected by the performance bottleneck of the central load balancer, and an external ISP link is uncontrollable, when the communication quality of a certain WAN port link is reduced or even disconnected, the central load balancer cannot quickly sense and adjust the scheduling strategy, and meanwhile, the method needs to additionally deploy the central load balancer, so that the hardware cost is increased; in the method (2), although each gateway can independently perform load balancing scheduling on the WAN port link, when there is an imbalance between WAN port links of different gateways, traffic redistribution cannot be performed efficiently.
Disclosure of Invention
The technical problem to be solved by the embodiments of the present invention is to provide a link load balancing method, device, controller and gateway device, which can effectively implement link load balancing of a multi-gateway multi-WAN-port networking system, and is not limited by a central load balancer, thereby saving hardware cost.
In order to solve the foregoing technical problem, a first aspect of an embodiment of the present invention provides a link load balancing method, where the method is applicable to a multi-gateway multi-WAN-port networking system, where the system includes a controller and at least one gateway device that is in connection communication with the controller through a network; each gateway device comprises at least one LAN port, at least one WAN port and at most two interconnection ports; each gateway device is connected and networked with the interconnection ports of other gateway devices through the interconnection port of the gateway device; the method is performed by the controller; the method comprises the following steps:
sending a reading instruction to each gateway device, so that each gateway device sends link state information to the controller according to the received reading instruction;
updating a preset link state information table according to the received link state information sent by each gateway device to obtain a first link state information table;
calculating the weighted bandwidth of each WAN port of each gateway device according to the first link state information table and a preset link weighted bandwidth evaluation method, and correspondingly storing the weighted bandwidth into the first link state information table to obtain a second link state information table;
acquiring the role and the weighted bandwidth of each interconnection port of each gateway device according to the second link state information table, a preset interconnection port role and a preset link weighted bandwidth evaluation method, and correspondingly storing the roles and the weighted bandwidth into the second link state information table to acquire a third link state information table; the interconnection port role comprises a blocking interconnection port, a WAN interconnection port and a LAN interconnection port;
generating corresponding link control information according to the third link state information table, and sending a write instruction carrying the link control information to each gateway device, so that each gateway device performs load balancing processing according to the received write instruction; the link control information includes a weighted bandwidth of each WAN port, a role of each interconnect port, and a weighted bandwidth of each interconnect port.
Further, the calculating the weighted bandwidth of each WAN port of each gateway device according to the first link state information table and a preset link weighted bandwidth evaluation method specifically includes:
for the ith WAN port of the kth gateway device, acquiring the state information of the ith WAN port of the kth gateway device according to the first link state information table; wherein the status information of the ith WAN port at least comprises a downlink bandwidth average value DBkiAverage value UB of uplink bandwidthkiA weight factor AkiDownlink flow statistics DTkiAnd upstream traffic statistics UTki;
According to the formulaCalculating weighted bandwidth WB of ith WAN port of kth gateway deviceki(ii) a Wherein k is more than or equal to 1 and less than or equal to M, M represents the number of gateway equipment in the system, and M is more than or equal to 1; i is more than or equal to 1 and less than or equal to N, N represents the number of WAN ports of the kth gateway equipment, and N is more than or equal to 1.
Further, the obtaining of the role and the weighted bandwidth of each interconnect of each gateway device according to the second link state information table, the preset interconnect role, and the preset link weighted bandwidth evaluation method specifically includes:
acquiring state information of all the interconnection ports according to the second link state information table; the state information of the interconnection port at least comprises a home terminal MAC address and an opposite terminal MAC address of the interconnection port;
acquiring interconnection port connection topological graphs among all gateway devices according to the state information of all interconnection ports and a preset interconnection topology construction method;
and determining the role of each interconnection port of each gateway device according to the topological graph, the preset interconnection port role and the preset link weighted bandwidth evaluation method, and calculating the weighted bandwidth of each interconnection port of each gateway device.
Further, the method further comprises:
when receiving an event message sent by any gateway equipment, sending a reading instruction to each gateway equipment; the event message indicates that the gateway equipment generates a preset event currently and needs to adjust link parameters; the preset event at least comprises a WAN port disconnection event or a bandwidth change event.
In order to solve the foregoing technical problem, a second aspect of the embodiments of the present invention provides a link load balancing method, where the method is applicable to a multi-gateway multi-WAN-port networking system, where the system includes a controller and at least one gateway device that is in connection communication with the controller through a network; each gateway device comprises at least one LAN port, at least one WAN port and at most two interconnection ports; each gateway device is connected and networked with the interconnection ports of other gateway devices through the interconnection port of the gateway device; the method is performed by any gateway device; the method comprises the following steps:
when a read instruction sent by the controller is received, link state information is obtained and sent to the controller, so that the controller generates corresponding link control information according to the received link state information sent by each gateway device, a preset interconnection port role and a preset link weighted bandwidth evaluation method, and sends a write instruction carrying the link control information to each gateway device; the link control information comprises the weighted bandwidth of each WAN port of each gateway device, the role of each interconnection port and the weighted bandwidth of each interconnection port; the interconnection port role comprises a blocking interconnection port, a WAN interconnection port and a LAN interconnection port;
and when a write command sent by the controller is received, carrying out load balancing processing according to the link control information carried in the received write command.
Further, the acquiring the link state information specifically includes:
acquiring state information of each WAN port; the state information of the WAN port at least comprises a downlink bandwidth average value, an uplink bandwidth average value and a preset weight factor of the WAN port;
acquiring state information of each interconnection port; the state information of the interconnection port at least comprises a home terminal MAC address and an opposite terminal MAC address of the interconnection port.
Further, when receiving the write command sent by the controller, the load balancing processing is performed according to the link control information carried in the received write command, which specifically includes:
when a write instruction sent by the controller is received, setting corresponding load balancing control parameters and interconnection port control parameters according to link control information carried in the received write instruction;
adding a WAN interconnection port to a preset WAN port schedulable list and a preset WAN port ongoing scheduling list, adjusting a weight parameter of a policy route according to the load balancing control parameter, and performing policy route selection according to the WAN port ongoing scheduling list and the weight parameter of the policy route;
and adjusting the role of the interconnection port according to the interconnection port control parameters, and starting a dialing function of the WAN interconnection port and an address allocation service of the LAN interconnection port.
Further, the method further comprises:
when receiving a system message, sending an event message to the controller, so that the controller sends a reading instruction to each gateway device according to the received event message; the system message indicates that the gateway equipment generates a preset event currently and needs to adjust link parameters; the preset event at least comprises a WAN port disconnection event or a bandwidth change event.
In order to solve the foregoing technical problem, a third aspect of the embodiments of the present invention provides a link load balancing apparatus, where the apparatus is suitable for a multi-gateway multi-WAN port networking system, where the system includes a controller, and at least one gateway device that is in connection communication with the controller through a network; each gateway device comprises at least one LAN port, at least one WAN port and at most two interconnection ports; each gateway device is connected and networked with the interconnection ports of other gateway devices through the interconnection port of the gateway device; the device is arranged in the controller; the device comprises a core control module, a link evaluation module and an interconnection evaluation module;
the core control module is used for sending a reading instruction to each gateway device, so that each gateway device sends link state information to the controller according to the received reading instruction;
the core control module is further configured to update a preset link state information table according to the received link state information sent by each gateway device, so as to obtain a first link state information table;
the link evaluation module is used for calculating the weighted bandwidth of each WAN port of each gateway device according to the first link state information table and a preset link weighted bandwidth evaluation method, and correspondingly storing the weighted bandwidth into the first link state information table to obtain a second link state information table;
the interconnection evaluation module is used for acquiring the role and the weighted bandwidth of each interconnection port of each gateway device according to the second link state information table, a preset interconnection port role and a preset link weighted bandwidth evaluation method, and correspondingly storing the roles and the weighted bandwidth into the second link state information table to acquire a third link state information table; the interconnection port role comprises a blocking interconnection port, a WAN interconnection port and a LAN interconnection port;
the core control module is further configured to generate corresponding link control information according to the third link state information table, and send a write instruction carrying the link control information to each gateway device, so that each gateway device performs load balancing processing according to the received write instruction; the link control information includes a weighted bandwidth of each WAN port, a role of each interconnect port, and a weighted bandwidth of each interconnect port.
In order to solve the foregoing technical problem, a fourth aspect of the embodiments of the present invention provides a link load balancing apparatus, where the apparatus is suitable for a multi-gateway multi-WAN-port networking system, and the system includes a controller and at least one gateway device that is in connection communication with the controller through a network; each gateway device comprises at least one LAN port, at least one WAN port and at most two interconnection ports; each gateway device is connected and networked with the interconnection ports of other gateway devices through the interconnection port of the gateway device; the device is arranged in any gateway equipment; the device comprises:
the information acquisition module is used for acquiring link state information and sending the link state information to the controller when receiving a read instruction sent by the controller, so that the controller generates corresponding link control information according to the received link state information sent by each gateway device, a preset interconnection port role and a preset link weighted bandwidth evaluation method, and sends a write instruction carrying the link control information to each gateway device; the link control information comprises the weighted bandwidth of each WAN port of each gateway device, the role of each interconnection port and the weighted bandwidth of each interconnection port; the interconnection port role comprises a blocking interconnection port, a WAN interconnection port and a LAN interconnection port;
and the load balancing module is used for carrying out load balancing processing according to the link control information carried in the received write instruction when the write instruction sent by the controller is received.
An embodiment of the present invention further provides a controller, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, where the processor, when executing the computer program, implements the link load balancing method according to any one of the first aspect.
An embodiment of the present invention further provides a gateway device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, where the processor, when executing the computer program, implements the link load balancing method according to any one of the second aspects.
Compared with the prior art, the embodiment of the invention provides a link load balancing method, a link load balancing device, a controller and gateway equipment, which are suitable for a multi-gateway multi-WAN-port networking system, wherein the system comprises the controller and at least one gateway equipment which is in connection communication with the controller through a network, each gateway equipment comprises at least one LAN port, at least one WAN port and at most two interconnection ports, and each gateway equipment is connected with the interconnection ports of other gateway equipment through the interconnection port of the gateway equipment and is networked; the controller sends a reading instruction to each gateway device, so that each gateway device sends link state information to the controller according to the received reading instruction, the controller updates a preset link state information table according to the received link state information sent by each gateway device to obtain a first link state information table, the controller calculates the weighted bandwidth of each WAN port of each gateway device according to the first link state information table and a preset link weighted bandwidth evaluation method, and correspondingly stores the weighted bandwidth into the first link state information table to obtain a second link state information table, the controller obtains the role and the weighted bandwidth of each interconnection port of each gateway device according to the second link state information table, the preset interconnection port role and the preset link weighted bandwidth evaluation method, and correspondingly stores the role and the weighted bandwidth into the second link state information table to obtain a third link state information table, the controller generates corresponding link control information according to the third link state information table, and sends a write instruction carrying the link control information to each gateway device, so that each gateway device performs load balancing processing according to the received write instruction, link load balancing of the multi-gateway multi-WAN-port networking system can be effectively achieved, limitation of a central load balancer is not needed, and hardware cost is saved.
Drawings
FIG. 1 is a schematic structural diagram of a preferred embodiment of a multi-gateway multi-WAN port networking system provided by the present invention;
fig. 2 is a flowchart of a preferred embodiment of a link load balancing method according to the first aspect of the present invention;
fig. 3A to fig. 3C are schematic diagrams of application topologies of an interconnection topology construction method provided by the present invention;
fig. 4 is a flowchart of a preferred embodiment of a link load balancing method according to a second aspect of the present invention;
fig. 5 is a block diagram of a preferred embodiment of a link load balancing apparatus according to a third aspect of the present invention;
fig. 6 is a block diagram of a preferred embodiment of a link load balancing apparatus according to a fourth aspect of the present invention;
FIG. 7 is a block diagram of a preferred embodiment of a controller provided by the present invention;
fig. 8 is a block diagram of a preferred embodiment of a gateway device provided by the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without any inventive step, are within the scope of the present invention.
Referring to fig. 1, it is a schematic structural diagram of a preferred embodiment of a multi-gateway multi-WAN-port networking system provided by the present invention, the system includes a controller (e.g. an SDN controller in an SDN network) and three gateway devices, namely a gateway device 1, a gateway device 2 and a gateway device 3, which are respectively in connection communication with the controller through a network, where the controller has a stable network connection with each gateway device and is capable of transceiving control messages and information messages, where the gateway device 1 includes two LAN ports, two WAN ports and an interconnection port, the gateway device 2 includes two LAN ports, two WAN ports and two interconnection ports, the gateway device 3 includes three LAN ports, two WAN ports and an interconnection port, each gateway device is connected with interconnection ports of other gateway devices through its own interconnection port for networking, and the two interconnection ports connected with each other form an interconnection port group, for example, in fig. 1, the group of interconnection ports between the gateway device 1 and the gateway device 2, and the group of interconnection ports between the gateway device 2 and the gateway device 3 belong to a dedicated network connection.
The interconnection port can be switched among three roles, wherein the specific roles are a blocking interconnection port, a WAN interconnection port and a LAN interconnection port, the initial role of the interconnection port is the blocking interconnection port, the blocking interconnection port does not forward messages, the WAN interconnection port serves as the WAN port of the gateway device, the LAN interconnection port serves as the LAN port of the gateway device, and the interconnection port group is composed of the LAN interconnection port and the WAN interconnection port or a pair of blocking interconnection ports.
Each gateway device has a plurality of LAN ports, a plurality of WAN ports, and at most two interconnection ports, the number of network interfaces of different gateway devices may be the same or different, the LAN ports, the WAN ports, and the interconnection ports may be physical interfaces or virtual interfaces based on a VLAN, the LAN ports are used to connect devices such as switches, routers, or hosts in the local area network, the WAN ports are used to connect external networks such as an ISP (Internet Service Provider), the interconnection ports are dedicated to connect interconnection ports of another gateway device for networking, and a topology diagram formed by all gateway devices connected based on the interconnection ports is one or more serial structures or a ring structure, for example, the serial structure in fig. 1.
It should be noted that the schematic structural diagram shown in fig. 1 is merely an example of the multi-gateway multi-WAN port networking system, and does not constitute a specific limitation to the present system, and may include more or less gateway devices than those shown in the drawings, or other similar network devices.
A first aspect of the embodiment of the present invention provides a link load balancing method, and as shown in fig. 2, is a flowchart of a preferred embodiment of the link load balancing method provided in the first aspect of the present invention, where the method is applicable to a multi-gateway multi-WAN port networking system, and the system includes a controller and at least one gateway device that is in connection communication with the controller through a network; each gateway device comprises at least one LAN port, at least one WAN port and at most two interconnection ports; each gateway device is connected and networked with the interconnection ports of other gateway devices through the interconnection port of the gateway device; the method is performed by the controller; the method includes steps S11 to S15:
step S11, sending a reading instruction to each gateway device, so that each gateway device sends link state information to the controller according to the received reading instruction;
step S12, updating a preset link state information table according to the received link state information sent by each gateway device to obtain a first link state information table;
step S13, calculating a weighted bandwidth of each WAN port of each gateway device according to the first link state information table and a preset link weighted bandwidth evaluation method, and storing the weighted bandwidth into the first link state information table correspondingly to obtain a second link state information table;
step S14, obtaining a role and a weighted bandwidth of each interconnect port of each gateway device according to the second link state information table, a preset interconnect port role, and a preset link weighted bandwidth evaluation method, and storing the roles and the weighted bandwidths into the second link state information table correspondingly to obtain a third link state information table; the interconnection port role comprises a blocking interconnection port, a WAN interconnection port and a LAN interconnection port;
step S15, generating corresponding link control information according to the third link state information table, and sending a write instruction carrying the link control information to each gateway device, so that each gateway device performs load balancing processing according to the received write instruction; the link control information includes a weighted bandwidth of each WAN port, a role of each interconnect port, and a weighted bandwidth of each interconnect port.
Specifically, the embodiment of the present invention is applicable to a system (for example, the system shown in fig. 1) in which a plurality of gateway devices and a plurality of WAN ports perform networking, and in the system, a controller sends a read instruction to each gateway device in the system to notify each gateway device of reporting information.
After receiving the read instruction sent by the controller, each gateway device executes the read instruction and reports the current link state information of the gateway device to the controller.
The controller updates a link state information table LIST preset in the controller according to the received link state information reported by all the gateway devices (the link state information table LIST is initialized { }), correspondingly stores the received link state information reported by each gateway device into the link state information table LIST, and correspondingly obtains a first link state information table LIST 1; the controller calculates and obtains the weighted bandwidth of each WAN port of each gateway device by using a preset link weighted bandwidth evaluation method according to the link state information in the obtained first link state information LIST LIST1, correspondingly stores the weighted bandwidth of each WAN port of each gateway device into the first link state information LIST LIST1, and correspondingly obtains a second link state information LIST LIST 2; the controller obtains the role of each interconnection port and the weighted bandwidth of each interconnection port of each gateway device by using a preset interconnection port role (interconnection port roles comprise a blocked interconnection port, a WAN interconnection port and a LAN interconnection port) and a preset link weighted bandwidth evaluation method according to the link state information in the second link state information table LIST2, correspondingly stores the role of each interconnection port of each gateway device and the weighted bandwidth of each interconnection port into the second link state information table LIST2, and correspondingly obtains a third link state information table LIST 3; the controller generates corresponding link control information (the link control information at least includes the weighted bandwidth of each WAN port of each gateway device, the role of each interconnect port, and the weighted bandwidth of each interconnect port) according to the third link state information table LIST3, and sends a write instruction carrying the link control information to each gateway device.
Each gateway device executes a write instruction after receiving the write instruction sent by the controller, and performs load balancing processing according to link control information carried in the write instruction, the gateway device sets corresponding load balancing control parameters and interconnection port control parameters according to the link control information carried in the received write instruction, the gateway device adds a WAN interconnection port of the gateway device to a preset WAN port schedulable list and a preset WAN port ongoing scheduling list, and adjusts weight parameters of a policy route according to the set load balancing control parameters, so as to perform policy route selection according to the latest WAN port ongoing scheduling list and the adjusted weight parameters of the policy route; the gateway equipment adjusts the role of the interconnection port according to the interconnection port control parameters, and starts the dialing function of the WAN interconnection port and the address allocation service of the LAN interconnection port.
It should be noted that, when the role of the interconnection port is configured to block the interconnection port, the routing forwarding of the data packet with other network interfaces in the gateway device is prohibited, and when the role of the interconnection port is configured to be the WAN interconnection port, the WAN interconnection port is added to the load balancing module of the gateway device for scheduling, and the dialing module is notified to start a dialing routine for the WAN interconnection port; when the role of the interconnection port is configured to be the LAN interconnection port, informing an address allocation module of the gateway equipment to start address allocation service for the LAN interconnection port; the gateway device starts a broadcast function for each internet port, can periodically send an ECHO broadcast packet to an opposite-end internet port, and immediately sends a REPLY broadcast packet to the opposite-end internet port when receiving the ECHO broadcast packet sent by the opposite-end internet port, and the internet port can acquire an MAC address of the opposite-end internet port according to the received ECHO broadcast packet or the REPLY broadcast packet.
The link load balancing method provided by the embodiment of the invention is characterized in that a controller is responsible for collecting link state information reported by each gateway device so as to calculate the weighted bandwidth of a link of each WAN port of each gateway device, decide the role of each interconnection port and calculate the weighted bandwidth of each WAN interconnection port, and control and configuration messages composed of calculation and decision results are sent to each gateway device; the gateway equipment executes a read-write instruction issued by the controller, configures the weight parameter of a local load balancing module, changes the role of an interconnection port, and starts the address allocation service of the LAN interconnection port and the dialing function of the WAN interconnection port; the whole system has the advantage of decision making of a controller center, the state of each WAN port link in the system can be analyzed, dynamic distribution and sharing of outlet flow in all WAN port links are realized by adjusting the roles of the interconnection ports, flow distribution of the whole system is more balanced, and therefore link load balancing of a multi-gateway multi-WAN port networking system can be effectively realized.
In another preferred embodiment, the calculating a weighted bandwidth of each WAN port of each gateway device according to the first link state information table and a preset link weighted bandwidth evaluation method specifically includes:
for the ith WAN port of the kth gateway device, acquiring the state information of the ith WAN port of the kth gateway device according to the first link state information table; wherein the status information of the ith WAN port at least comprises a downlink bandwidth average value DBkiAverage value UB of uplink bandwidthkiA weight factor AkiDownlink flow statistics DTkiAnd upstream traffic statistics UTki;
According to the formulaCalculating weighted bandwidth WB of ith WAN port of kth gateway deviceki(ii) a Wherein k is more than or equal to 1 and less than or equal to M, M represents the number of gateway equipment in the system, and M is more than or equal to 1; i is more than or equal to 1 and less than or equal to N, N represents the number of WAN ports of the kth gateway equipment, and N is more than or equal to 1.
Specifically, the controller calculates and obtains the weighted bandwidth of each WAN port of each gateway device by using a preset link weighted bandwidth evaluation method according to the link state information in the obtained first link state information table LIST1, where the calculation method of the weighted bandwidth of each WAN port is the same, and here, the ith WAN port of the kth gateway device is taken as an example to explain: with reference to the above embodiment, the controller obtains the status information of the ith WAN port of the kth gateway device according to the obtained link status information in the first link status information table LIST1, where the status information at least includes the downlink bandwidth average DB of the WAN portkiAverage value UB of uplink bandwidthkiA weight factor AkiDownlink flow statistics DTkiAnd upstream traffic statistics UTkiRoot of Chinese characterAccording to the formulaCalculating to obtain weighted bandwidth WB of ith WAN port of kth gateway deviceki。
It should be noted that, a link status information table LIST is preset in the controller, and the link status information table LIST is initialized { }; after receiving the link state information reported by each gateway device, the controller correspondingly inserts the link state information reported by each gateway device into the link state information table LIST, and correspondingly obtains a first link state information table LIST1, which takes the system shown in fig. 1 as an example and includes three gateway devices, where the first link state information table LIST1 is { D ═ D1,D2,D3In which D is1~D3Respectively representing the link state information of gateway equipment 1 to gateway equipment 3 in the system; the controller, after calculating the weighted bandwidth Q of each WAN port, sets { Q) of the weighted bandwidths of all WAN ports of all gateway devices1,Q2,···,Q6Correspondingly updating into the first link state information table LIST1, and accordingly obtaining the second link state information table LIST2 { { D { }1,Q1},{D1,Q2},···,{D3,Q6} in which Q is equal to1~Q6Respectively representing the weighted bandwidth of the WAN ports 1-6 after all the WAN ports in the system are numbered from left to right as 1-6, namely corresponding to the WB obtained by calculation according to the formula11、WB12、WB21、WB22、WB31And WB32(ii) a After obtaining the role of each interconnect and the weighted bandwidth of each interconnect, the controller correspondingly updates the set of the roles and the weighted bandwidths of all the interconnects to the second link state information table LIST2, and correspondingly obtains a third link state information table LIST 3.
For example, table 1 shows link state information of a corresponding kth gateway device in the first link state information table LIST1, and table 2 shows link state information of a corresponding kth gateway device in the second link state information table LIST2, assuming that the kth gateway device has N WAN ports and two interconnect ports, where the number of the interconnect ports of the gateway device may be 0, 1, or 2, and when there is no interconnect port in the kth gateway device, the corresponding position of the interconnect port state information in tables 1 and 2 is empty.
TABLE 1
TABLE 2
In another preferred embodiment, the obtaining, according to the second link state information table, the preset role of the interconnect port, and the preset link weighted bandwidth evaluation method, the role and the weighted bandwidth of each interconnect port of each gateway device specifically includes:
acquiring state information of all the interconnection ports according to the second link state information table; the state information of the interconnection port at least comprises a home terminal MAC address and an opposite terminal MAC address of the interconnection port;
acquiring interconnection port connection topological graphs among all gateway devices according to the state information of all interconnection ports and a preset interconnection topology construction method;
and determining the role of each interconnection port of each gateway device according to the topological graph, the preset interconnection port role and the preset link weighted bandwidth evaluation method, and calculating the weighted bandwidth of each interconnection port of each gateway device.
Specifically, with reference to the foregoing embodiment, the controller obtains, according to the link state information in the second link state information table LIST2, state information of all the interconnect ports, where the information at least includes a home MAC address and an opposite MAC address of each interconnect port, obtains, according to the state information of all the interconnect ports and a preset interconnect topology construction method, an interconnect port connection topology diagram among all the gateway devices, determines, according to the constructed topology diagram, a role of each interconnect port of each gateway device by using a preset interconnect port role and a preset link weighted bandwidth evaluation method, and calculates and obtains a weighted bandwidth of each interconnect port of each gateway device.
It should be added that the interconnection topology construction method specifically includes the following steps:
TABLE 3
Gateway device | 1 st interconnection port | 2 nd interconnection port |
1 st gateway device | W11=0 | W12=0 |
2 nd gateway device | W21=0 | W22=0 |
… | … | … |
Mth gateway device | WM1=0 | WM2=0 |
(1) The topology graph may be represented by a neighbor table with initial values as shown in table 3, WkjThe number of the gateway equipment connected to the jth interconnection port of the kth gateway equipment is represented, the initial state is all 0, the number of the interconnection ports of the opposite end which are not connected with any gateway equipment is represented, the number of the interconnection ports of each gateway equipment is at most 2, namely j is more than or equal to 0 and less than or equal to 2, the number of the interconnection ports of the gateway equipment can be 0, 1 or 2, and when the number of the interconnection ports of a certain gateway equipment is 0, the corresponding position in the table 3 is empty;
(2) starting to traverse all the interconnection ports of all the gateway devices, searching for the neighbor of each interconnection port, assuming that x and y respectively represent the gateway device number and the interconnection port number of the neighbor which needs to be searched currently, and x/y can sequentially take a group of numbers in 1/1, 1/2, 2/1, 2/2, …, M/1 and M/2, and the number is not selected in the x/y initial state;
(3) selecting a certain group of numbers 1/1, 1/2, 2/1, 2/2, …, M/1 and M/2 for x/y, and processing according to the conditions: if x/y is unnumbered and selectable, jumping to the step (6); if W isxyIf not, skipping to the step (3) to take the next group of numbers of x/y; if the situation is other, jumping to the step (4);
(4) starting to traverse all interconnection ports of all gateway equipment, searching for the numbers of the gateway equipment and the interconnection ports adjacent to x/y, and assuming that g and h are found to respectively represent the numbers of the gateway equipment and the interconnection ports adjacent to x/y; similarly, g/h can take a group of numbers in 1/1, 1/2, 2/1, 2/2, …, M/1 and M/2 in sequence (sequentially traversing all the interconnection ports of all the gateway devices), and no number is selected in the initial state;
(5) g/h is selected from a certain group of numbers of 1/1, 1/2, 2/1, 2/2, …, M/1 and M/2, and is treated according to the conditions: if g/h has no number and is selectable, skipping to the step (3) to take the next group of numbers of x/y; if D isxMAC ofx#yIs equal to DgMAC inghThen set Wxy=g,WghX, skipping to the step (3) to take the next group of numbers of x/y;
(6) and (6) ending.
Fig. 3A to fig. 3C are schematic diagrams of application topologies of an interconnection topology construction method provided by the present invention, where fig. 3A is a topology one, a system includes only one gateway device 1, and the gateway device 1 includes an interconnection port a and an interconnection port b; fig. 3B shows a topology two, the system includes a gateway device 1, a gateway device 2, and a gateway device 3, the gateway device 1 has an interconnection port a and an interconnection port B, the gateway device 2 has an interconnection port c and an interconnection port d, the gateway device 3 has an interconnection port e and an interconnection port f, the interconnection port B is connected to the interconnection port c, the interconnection port d is connected to the interconnection port e, and the topology two is a series structure; in fig. 3C, the system includes a gateway device 1, a gateway device 2, and a gateway device 3, where the gateway device 1 has an interconnection port a and an interconnection port b, the gateway device 2 has an interconnection port C and an interconnection port d, the gateway device 3 has an interconnection port e and an interconnection port f, the interconnection port a is connected to the interconnection port f, the interconnection port b is connected to the interconnection port C, the interconnection port d is connected to the interconnection port e, and the formed topology is a ring structure.
According to the interconnection topology construction method, the following steps can be obtained: the neighbor table is shown in table 4 for topology one in fig. 3A, in table 5 for topology two in fig. 3B, and in table 6 for topology three in fig. 3C.
TABLE 4
Gateway device | 1 st interconnection port | 2 nd interconnection port |
Gateway device 1 | W1a=0 | W1b=0 |
TABLE 5
Gateway device | 1 st interconnection port | 2 nd interconnection port |
Gateway device 1 | W1a=0 | W1b=2 |
Gateway device 2 | W2c=1 | W2d=3 |
Gateway device 3 | W3e=2 | W3f=0 |
TABLE 6
Gateway device | 1 st interconnection port | 2 nd interconnection port |
Gateway device 1 | W1a=3 | W1b=2 |
Gateway device 2 | W2c=1 | W2d=3 |
Gateway device 3 | W3e=2 | W3f=1 |
It should be added that the method for evaluating role and weighted bandwidth of an interconnection port specifically includes the following steps:
(1) let the gateway device number set ORIGN be {1, 2, …, M }, and the role of the interconnection port and the weighted bandwidth of the kth gateway device be Pk={PRk1=0,PBk1=0,PRk2=0,PBk20} where, PRk1The role of the 1 st interconnection port of the kth gateway device is represented, the value 1/2/3 correspondingly represents the blocked interconnection port/WAN interconnection port/LAN interconnection port, the value 0 represents the unexvaluated role, and the PB isk1Indicating the weighted bandwidth of the 1 st interconnect of the kth gateway device, PR, for the same reasonk2/PBk2Respectively representing the role/weighted bandwidth of the 2 nd interconnection port of the kth gateway device;
(2) if the set ORIGN is empty, directly jumping to the step (7) to finish evaluation, otherwise, selecting a number k from the set ORIGN, performing recursive search according to a topological graph recorded by a neighbor table, finding a set UNION with all numbers which can be reached by the number k, deleting all numbers existing in the set UNION from the set ORIGN, and assuming that the number of the numbers in the set UNION is z;
(3) calculating the weighted bandwidth sum of the k gateway device in the set UNION as Ck=WBk1+WBk2+…+WBkN(ii) a The average of the weighted bandwidth sums of all gateway devices in the set UNION is calculated asR1=R2=…=RM=(C1+C2+…+CM)/z;
(4) And (3) traversing and selecting a number k from the set UNION as a starting point, and processing according to the following cases: if the set UNION has only one unique number k, then P is set directlyk={PRk1=1,PBk1=0,PRk2=1,PBk20 and jumping to the step (2); if a number k can be found, which has only one interconnection port t with neighbor and the other interconnection port r without neighbor, PR is set firstkr=1,PBkrJumping to the step (5) again when the value is equal to 0; if all numbers contain two neighbors, randomly selecting a k, firstly assuming that the 1 st interconnection port is t-1, the 2 nd interconnection port is r-2, and assuming that o is the neighbor W of kkrAssuming that f is the number of the interconnection port adjacent to the r interconnection port of k in o, PR is setkr=1,PBkr=0,PRof=1,PBofIf not, jumping to the step (5);
(5) let us assume a neighbor W of kktAnd assuming that u is the number of the interconnection port adjacent to the t interconnection port of k in v and e is the number of the other interconnection port in v, then processing according to the following cases: if (C)k+PBkr) Is equal to RkSetting PRkt=1,PBkt=0,PRvu=1,PBvuIf not, jumping to the step (6); if (C)k+PBkr) Less than RkSetting PRkt=2,PBkt=Rk-(Ck+PBkr),PRvu=3,PBvu=(Ck+PBkr)-RkAnd jumping to the step (6); if (C)k+PBkr) Greater than RkSetting PRkt=3,PBkt=Rk-(Ck+PBkr),PRvu=2,PBvu=(Ck+PBkr)-RkAnd jumping to the step (6);
(6) if PRveIf not, jumping to the step (2); if W isveEqual to 0, PR is setve=1,PBve0, andskipping to the step (2); if the situation is the other situation, firstly setting k as v, t as e and u as r, and then jumping to the step (5);
(7) and (6) ending.
The interconnection port role and weighted bandwidth set of the k-th gateway device finally evaluated by the controller are { P1, P2, …, PkH, weighted bandwidth set of N WAN ports with the kth gateway device { Q1, Q2, …, QNSummarizing to obtain the link control information sent to the kth gateway device, as shown in table 7.
TABLE 7
Taking the topology three shown in fig. 3C as an example, the evaluation is performed according to the above-mentioned role of the interconnect and the weighted bandwidth evaluation method:
(1) performing weighted bandwidth calculation according to the link state information of the WAN port, assuming that a number after the WAN port identification colon in the topology three represents a weighted bandwidth value obtained by calculation, for example, the weighted bandwidth of the WAN1 of the gateway device 1 is 5, and the other same reason;
(2) the weighted bandwidth sum, C, of the gateway device 1, the gateway device 2 and the gateway device 3 is calculated separately1=WB11+WB12=5+10=15,C2=WB21+WB22=3+3=6、C3=WB31+WB32The sum of the weighted bandwidths of the three gateway devices is calculated as R, 13+ 8-211=R2=R3=R=(15+6+21)/3=14;
(3) All gateway devices are provided with a left neighbor and a right neighbor, the gateway device 1 is selected, the interconnection port b is set as a blocking interconnection port, the weighted bandwidth value is 0, the interconnection port c of the neighbor gateway device 2 of the interconnection port b is also set as a blocking interconnection port, and the weighted bandwidth value is 0; since the sum of the weighted bandwidths of the gateway device 1 is 15 and is greater than R ═ 14, the interconnection port a is set as the LAN interconnection port, and the weighted bandwidth value is-1; setting an interconnection port f of the neighbor gateway device 3 of the interconnection port a as a WAN interconnection port, and setting a weighted bandwidth value as 1;
(4) selecting a gateway device 3, wherein the role of the interconnection port f is selected, and setting the role of the interconnection port e as a LAN interconnection port and the weighted bandwidth value as-8, because the sum 21 of the weighted bandwidths of the gateway device 3 and the weighted bandwidth 1 borrowed from the interconnection port f are equal to 22 and greater than R ═ 14; setting an interconnection port d of the neighbor gateway device 2 of the interconnection port e as a WAN interconnection port, and setting a weighted bandwidth value as 8;
(5) because the roles of the interconnection port c and the interconnection port d of the gateway device 2 are set, the interconnection port role and the weighted bandwidth evaluation are completed; the link control information of the three gateway devices is finally obtained as shown in table 8.
TABLE 8
In yet another preferred embodiment, the method further comprises:
when receiving an event message sent by any gateway equipment, sending a reading instruction to each gateway equipment; the event message indicates that the gateway equipment generates a preset event currently and needs to adjust link parameters; the preset event at least comprises a WAN port disconnection event or a bandwidth change event.
Specifically, with reference to the foregoing embodiment, the controller may actively and periodically send a read instruction to each gateway device, or send a read instruction to each gateway device when receiving an event message sent by any gateway device in the system, where subsequent processing flows are the same as those in the foregoing embodiment and are not described here again.
It should be noted that the event message indicates that a preset event occurs at the gateway device currently and link parameters need to be adjusted, where the preset event at least includes a WAN port disconnection event or a bandwidth change event, for example, when the gateway device determines that an increase or a decrease of a currently detected bandwidth average value with respect to a bandwidth average value obtained by last detection reaches 30%, the gateway device submits the bandwidth change event to a system message list, and when receiving the system message, the gateway device sends the event message to the controller.
A second aspect of the present invention further provides a link load balancing method, which is a flowchart shown in fig. 4 and is a preferred embodiment of the link load balancing method provided in the second aspect of the present invention, where the method is applied to a multi-gateway multi-WAN port networking system, and the system includes a controller and at least one gateway device that is in connection communication with the controller through a network; each gateway device comprises at least one LAN port, at least one WAN port and at most two interconnection ports; each gateway device is connected and networked with the interconnection ports of other gateway devices through the interconnection port of the gateway device; the method is performed by any gateway device; the method comprises steps S21 to S22:
step S21, when a read instruction sent by the controller is received, link state information is obtained and sent to the controller, so that the controller generates corresponding link control information according to the received link state information sent by each gateway device, a preset interconnection port role and a preset link weighted bandwidth evaluation method, and sends a write instruction carrying the link control information to each gateway device; the link control information comprises the weighted bandwidth of each WAN port of each gateway device, the role of each interconnection port and the weighted bandwidth of each interconnection port; the interconnection port role comprises a blocking interconnection port, a WAN interconnection port and a LAN interconnection port;
and step S22, when receiving the write command sent by the controller, performing load balancing processing according to the link control information carried in the received write command.
Specifically, the embodiment of the present invention is applicable to a system (for example, the system shown in fig. 1) in which a plurality of gateway devices and a plurality of WAN ports perform networking, and in the system, after receiving a read instruction sent by a controller, the gateway device executes the read instruction, obtains current link state information of the gateway device, and reports the obtained link state information to the controller.
The controller updates a link state information table LIST preset in the controller according to the received link state information reported by all the gateway devices (the link state information table LIST is initialized { }), correspondingly stores the received link state information reported by each gateway device into the link state information table LIST, and correspondingly obtains a first link state information table LIST 1; the controller calculates and obtains the weighted bandwidth of each WAN port of each gateway device by using a preset link weighted bandwidth evaluation method according to the link state information in the obtained first link state information LIST LIST1, correspondingly stores the weighted bandwidth of each WAN port of each gateway device into the first link state information LIST LIST1, and correspondingly obtains a second link state information LIST LIST 2; the controller obtains the role of each interconnection port and the weighted bandwidth of each interconnection port of each gateway device by using a preset interconnection port role (interconnection port roles comprise a blocked interconnection port, a WAN interconnection port and a LAN interconnection port) and a preset link weighted bandwidth evaluation method according to the link state information in the second link state information table LIST2, correspondingly stores the role of each interconnection port of each gateway device and the weighted bandwidth of each interconnection port into the second link state information table LIST2, and correspondingly obtains a third link state information table LIST 3; the controller generates corresponding link control information (the link control information at least includes the weighted bandwidth of each WAN port of each gateway device, the role of each interconnect port, and the weighted bandwidth of each interconnect port) according to the third link state information table LIST3, and sends a write instruction carrying the link control information to each gateway device.
The gateway equipment executes a write instruction after receiving the write instruction sent by the controller, sets corresponding load balancing control parameters and interconnection port control parameters according to link control information carried in the write instruction, so as to configure weight parameters of a local load balancing module and change the role of an interconnection port, thereby carrying out load balancing processing.
The link load balancing method provided by the embodiment of the invention is characterized in that a controller is responsible for collecting link state information reported by each gateway device so as to calculate the weighted bandwidth of a link of each WAN port of each gateway device, decide the role of each interconnection port and calculate the weighted bandwidth of each WAN interconnection port, and control and configuration messages composed of calculation and decision results are sent to each gateway device; the gateway equipment executes a read-write instruction issued by the controller, configures the weight parameter of a local load balancing module, changes the role of an interconnection port, and starts the address allocation service of the LAN interconnection port and the dialing function of the WAN interconnection port; the whole system has the advantage of decision making of a controller center, the state of each WAN port link in the system can be analyzed, dynamic distribution and sharing of outlet flow in all WAN port links are realized by adjusting the roles of the interconnection ports, flow distribution of the whole system is more balanced, and therefore link load balancing of a multi-gateway multi-WAN port networking system can be effectively realized.
In another preferred embodiment, the acquiring the link status information specifically includes:
acquiring state information of each WAN port; the state information of the WAN port at least comprises a downlink bandwidth average value, an uplink bandwidth average value and a preset weight factor of the WAN port;
acquiring state information of each interconnection port; the state information of the interconnection port at least comprises a home terminal MAC address and an opposite terminal MAC address of the interconnection port.
Specifically, with reference to the above embodiment, the gateway device collects status information of each WAN port, where the status information of the WAN port at least includes a downlink bandwidth average value, an uplink bandwidth average value, and a preset weight factor (the weight factor may be input through a user interface); and collecting state information of each interconnection port, where the state information of the interconnection port at least includes a home terminal MAC address and an opposite terminal MAC address of the interconnection port, and correspondingly obtaining current link state information of the gateway device, for example, link state information of a kth gateway device shown in table 1.
In another preferred embodiment, when receiving a write command sent by the controller, the load balancing processing according to the link control information carried in the received write command specifically includes:
when a write instruction sent by the controller is received, setting corresponding load balancing control parameters and interconnection port control parameters according to link control information carried in the received write instruction;
adding a WAN interconnection port to a preset WAN port schedulable list and a preset WAN port ongoing scheduling list, adjusting a weight parameter of a policy route according to the load balancing control parameter, and performing policy route selection according to the WAN port ongoing scheduling list and the weight parameter of the policy route;
and adjusting the role of the interconnection port according to the interconnection port control parameters, and starting a dialing function of the WAN interconnection port and an address allocation service of the LAN interconnection port.
Specifically, with reference to the foregoing embodiment, after receiving a write instruction sent by a controller, the gateway device sets a corresponding load balancing control parameter and an interconnection port control parameter according to link control information carried in the write instruction, adds a WAN interconnection port of the gateway device to a preset WAN port schedulable list and a preset WAN port ongoing scheduling list, and adjusts a weight parameter of a policy route according to the set load balancing control parameter, so as to perform policy route selection according to a latest WAN port ongoing scheduling list and the adjusted weight parameter of the policy route; the gateway equipment adjusts the role of the interconnection port according to the interconnection port control parameters, and starts the dialing function of the WAN interconnection port and the address allocation service of the LAN interconnection port.
It should be noted that, when the role of the interconnection port is configured to block the interconnection port, the routing forwarding of the data packet with other network interfaces in the gateway device is prohibited, and when the role of the interconnection port is configured to be the WAN interconnection port, the WAN interconnection port is added to the load balancing module of the gateway device for scheduling, and the dialing module is notified to start a dialing routine for the WAN interconnection port; when the role of the interconnection port is configured to be the LAN interconnection port, informing an address allocation module of the gateway equipment to start address allocation service for the LAN interconnection port; the gateway device starts a broadcast function for each interconnection port, can periodically send a broadcast packet to the opposite-end interconnection port, immediately sends the broadcast packet to the opposite-end interconnection port when receiving the broadcast packet sent by the opposite-end interconnection port, and acquires the MAC address of the opposite-end interconnection port according to the received broadcast packet.
In yet another preferred embodiment, the method further comprises:
when receiving a system message, sending an event message to the controller, so that the controller sends a reading instruction to each gateway device according to the received event message; the system message indicates that the gateway equipment generates a preset event currently and needs to adjust link parameters; the preset event at least comprises a WAN port disconnection event or a bandwidth change event.
Specifically, with reference to the foregoing embodiment, when the gateway device receives a system message, the gateway device sends an event message to the controller, so that the controller sends a read instruction to each gateway device according to the received event message, and the subsequent processing flow is the same as that in the foregoing embodiment, and is not described here again.
It should be noted that the system message indicates that a preset event occurs in the gateway device currently and the link parameter needs to be adjusted, where the preset event at least includes a WAN disconnection event or a bandwidth change event.
For example, when the gateway device determines that the rising or falling of the currently detected bandwidth average value relative to the bandwidth average value obtained by the last detection reaches 30%, it determines that a bandwidth change event occurs, and submits the bandwidth change event to the system message list, and when the gateway device determines that a certain WAN port is dialing and disconnected, it submits the WAN port disconnection event to the system message list, and removes the WAN port from the WAN port scheduling list; when the disconnected WAN port is connected with the dial-up again successfully, the disconnected WAN port is added into the scheduling list of the WAN port again.
A third aspect of the embodiments of the present invention further provides a link load balancing apparatus, which is capable of implementing all the processes of the link load balancing method described in any embodiment of the first aspect, and the functions and implemented technical effects of each module and unit in the apparatus are respectively the same as those of the link load balancing method described in the embodiment of the first aspect, and are not described herein again.
Referring to fig. 5, it is a block diagram of a preferred embodiment of a link load balancing apparatus provided in a third aspect of the present invention, where the apparatus is suitable for a multi-gateway multi-WAN port networking system, and the system includes a controller, and at least one gateway device in connection communication with the controller through a network; each gateway device comprises at least one LAN port, at least one WAN port and at most two interconnection ports; each gateway device is connected and networked with the interconnection ports of other gateway devices through the interconnection port of the gateway device; the device is arranged in the controller; the device comprises a core control module 11, a link evaluation module 12 and an interconnection evaluation module 13;
the core control module 11 is configured to send a read instruction to each gateway device, so that each gateway device sends link state information to the controller according to the received read instruction;
the core control module 11 is further configured to update a preset link state information table according to the received link state information sent by each gateway device, so as to obtain a first link state information table;
the link evaluation module 12 is configured to calculate a weighted bandwidth of each WAN port of each gateway device according to the first link state information table and a preset link weighted bandwidth evaluation method, and correspondingly store the weighted bandwidth into the first link state information table to obtain a second link state information table;
the interconnection evaluation module 13 is configured to obtain a role and a weighted bandwidth of each interconnection port of each gateway device according to the second link state information table, a preset interconnection port role, and a preset link weighted bandwidth evaluation method, and store the role and the weighted bandwidth of each interconnection port of each gateway device into the second link state information table correspondingly to obtain a third link state information table; the interconnection port role comprises a blocking interconnection port, a WAN interconnection port and a LAN interconnection port;
the core control module 11 is further configured to generate corresponding link control information according to the third link state information table, and send a write instruction carrying the link control information to each gateway device, so that each gateway device performs load balancing processing according to the received write instruction; the link control information includes a weighted bandwidth of each WAN port, a role of each interconnect port, and a weighted bandwidth of each interconnect port.
Preferably, the link evaluation module 12 specifically includes:
a WAN port state information acquiring unit, configured to acquire, for an ith WAN port of a kth gateway device, state information of an ith WAN port of the kth gateway device according to the first link state information table; wherein the status information of the ith WAN port at least comprises a downlink bandwidth average value DBkiAverage value UB of uplink bandwidthkiA weight factor AkiDownlink flow statistics DTkiAnd upstream traffic statistics UTki;
A link evaluation unit for evaluating a link according to a formulaCalculating weighted bandwidth WB of ith WAN port of kth gateway deviceki(ii) a Wherein k is more than or equal to 1 and less than or equal to M, M represents the number of gateway equipment in the system, and M is more than or equal to 1; i is more than or equal to 1 and less than or equal to N, N represents the number of WAN ports of the kth gateway equipment, and N is more than or equal to 1.
Preferably, the interconnection evaluation module 13 specifically includes:
an interconnection port state information obtaining unit, configured to obtain state information of all interconnection ports according to the second link state information table; the state information of the interconnection port at least comprises a home terminal MAC address and an opposite terminal MAC address of the interconnection port;
a topological graph obtaining unit, configured to obtain interconnection port connection topological graphs among all gateway devices according to state information of all interconnection ports and a preset interconnection topology construction method;
and the interconnection evaluation unit is used for deciding the role of each interconnection port of each gateway device according to the topological graph, the preset interconnection port role and the preset link weighted bandwidth evaluation method, and calculating the weighted bandwidth of each interconnection port of each gateway device.
Preferably, the core control module 11 is further configured to:
when receiving an event message sent by any gateway equipment, sending a reading instruction to each gateway equipment; the event message indicates that the gateway equipment generates a preset event currently and needs to adjust link parameters; the preset event at least comprises a WAN port disconnection event or a bandwidth change event.
A fourth aspect of the embodiments of the present invention further provides a link load balancing apparatus, which is capable of implementing all the processes of the link load balancing method described in any embodiment of the second aspect, and the functions and implemented technical effects of each module and unit in the apparatus are respectively the same as those of the link load balancing method described in the embodiment of the second aspect, and are not described herein again.
Referring to fig. 6, it is a block diagram of a preferred embodiment of a link load balancing apparatus according to a fourth aspect of the present invention, where the apparatus is suitable for a multi-gateway multi-WAN port networking system, and the system includes a controller, and at least one gateway device connected and communicating with the controller through a network; each gateway device comprises at least one LAN port, at least one WAN port and at most two interconnection ports; each gateway device is connected and networked with the interconnection ports of other gateway devices through the interconnection port of the gateway device; the device is arranged in any gateway equipment; the device comprises:
the information acquisition module 21 is configured to, when receiving a read instruction sent by the controller, acquire link state information and send the link state information to the controller, so that the controller generates corresponding link control information according to the received link state information sent by each gateway device, a preset interconnect angle, and a preset link weighted bandwidth evaluation method, and sends a write instruction carrying the link control information to each gateway device; the link control information comprises the weighted bandwidth of each WAN port of each gateway device, the role of each interconnection port and the weighted bandwidth of each interconnection port; the interconnection port role comprises a blocking interconnection port, a WAN interconnection port and a LAN interconnection port;
and the load balancing module 22 is configured to, when receiving the write instruction sent by the controller, perform load balancing processing according to the link control information carried in the received write instruction.
Preferably, the information acquisition module 21 specifically includes:
the bandwidth detection unit is used for acquiring the state information of each WAN port; the state information of the WAN port at least comprises a downlink bandwidth average value, an uplink bandwidth average value and a preset weight factor of the WAN port;
the first interconnection port control unit is used for acquiring the state information of each interconnection port; the state information of the interconnection port at least comprises a home terminal MAC address and an opposite terminal MAC address of the interconnection port.
Preferably, the load balancing module 22 specifically includes:
the load balancing configuration unit is used for setting corresponding load balancing control parameters and interconnection port control parameters according to link control information carried in the received write instruction when the write instruction sent by the controller is received;
the load balancing control unit is used for adding the WAN interconnection port to a preset WAN port schedulable list and a preset WAN port ongoing scheduling list, adjusting the weight parameter of the policy route according to the load balancing control parameter, and performing policy route selection according to the WAN port ongoing scheduling list and the weight parameter of the policy route;
and the second interconnection port control unit is used for adjusting the role of the interconnection port according to the interconnection port control parameters and starting a dialing function of the WAN interconnection port and address allocation service of the LAN interconnection port.
Preferably, the apparatus further comprises a core interaction module, the core interaction module being configured to:
when receiving a system message, sending an event message to the controller, so that the controller sends a reading instruction to each gateway device according to the received event message; the system message indicates that the gateway equipment generates a preset event currently and needs to adjust link parameters; the preset event at least comprises a WAN port disconnection event or a bandwidth change event.
An embodiment of the present invention further provides a controller, which is shown in fig. 7 and is a block diagram of a preferred embodiment of the controller provided in the present invention, where the controller includes a processor 101, a memory 102, and a computer program stored in the memory 102 and configured to be executed by the processor 101, and the processor 101, when executing the computer program, implements the link load balancing method according to any embodiment of the first aspect.
Preferably, the computer program can be divided into one or more modules/units (e.g. computer program 1, computer program 2,) which are stored in the memory 102 and executed by the processor 101 to accomplish the present invention. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution of the computer program in the controller.
The Processor 101 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, a discrete hardware component, etc., the general purpose Processor may be a microprocessor, or the Processor 101 may be any conventional Processor, the Processor 101 is a control center of the controller, and various interfaces and lines are used to connect various parts of the controller.
The memory 102 mainly includes a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function, and the like, and the data storage area may store related data and the like. In addition, the memory 102 may be a high speed random access memory, a non-volatile memory such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash Card (Flash Card), and the like, or the memory 102 may be other volatile solid state memory devices.
It should be noted that the above controller may include, but is not limited to, a processor and a memory, and those skilled in the art will understand that the structural block diagram of fig. 7 is only an example of the above controller, and does not constitute a limitation to the controller, and may include more or less components than those shown, or combine some components, or different components.
An embodiment of the present invention further provides a gateway device, as shown in fig. 8, which is a block diagram of a preferred embodiment of the gateway device provided in the present invention, the gateway device includes a processor 201, a memory 202, and a computer program stored in the memory 202 and configured to be executed by the processor 201, and when executing the computer program, the processor 201 implements the link load balancing method according to any embodiment of the second aspect.
Preferably, the computer program can be divided into one or more modules/units (e.g. computer program 1, computer program 2,) which are stored in the memory 202 and executed by the processor 201 to accomplish the present invention. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution of the computer program in the gateway device.
The Processor 201 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, a discrete hardware component, etc., the general purpose Processor may be a microprocessor, or the Processor 201 may be any conventional Processor, the Processor 201 is a control center of the gateway device, and various interfaces and lines are used to connect various parts of the gateway device.
The memory 202 mainly includes a program storage area that may store an operating system, an application program required for at least one function, and the like, and a data storage area that may store related data and the like. In addition, the memory 202 may be a high speed random access memory, may also be a non-volatile memory, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash Card (Flash Card), and the like, or the memory 202 may also be other volatile solid state memory devices.
It should be noted that the above gateway device may include, but is not limited to, a processor and a memory, and those skilled in the art will understand that the structural block diagram of fig. 8 is only an example of the above gateway device and does not constitute a limitation of the gateway device, and may include more or less components than those shown in the drawings, or may combine some components, or different components.
To sum up, the link load balancing method, apparatus, controller and gateway device provided in the embodiments of the present invention have the following beneficial effects:
(1) in a system scene without a central load balancer, the link load balancing among a plurality of exit gateways can be realized, the performance bottleneck influence of the central load balancer is avoided, the method is also suitable for medium and small network scale scenes, the hardware cost can be saved, and the network topology design is simplified;
(2) the dispatching distribution of the outlet flow is globally and integrally planned by the controller, so that compared with the link load dispatching of a single gateway system, the global load balancing is more favorably realized, and near real-time dispatching adjustability can be realized by dynamically monitoring the link state;
(3) the method can be used in cooperation with a central load balancer in the prior art, so that the defect that the central load balancer cannot sense the WAN port link change and the bandwidth change is overcome.
The above description is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, several modifications and variations can be made without departing from the technical principle of the present invention, and these modifications and variations should also be regarded as the protection scope of the present invention.
Claims (12)
1. A link load balancing method is characterized in that the method is suitable for a multi-gateway multi-WAN-port networking system, and the system comprises a controller and at least one gateway device which is in connection communication with the controller through a network; each gateway device comprises at least one LAN port, at least one WAN port and at most two interconnection ports; each gateway device is connected and networked with the interconnection ports of other gateway devices through the interconnection port of the gateway device; the method is performed by the controller; the method comprises the following steps:
sending a reading instruction to each gateway device, so that each gateway device sends link state information to the controller according to the received reading instruction;
updating a preset link state information table according to the received link state information sent by each gateway device to obtain a first link state information table;
calculating the weighted bandwidth of each WAN port of each gateway device according to the first link state information table and a preset link weighted bandwidth evaluation method, and correspondingly storing the weighted bandwidth into the first link state information table to obtain a second link state information table;
acquiring the role and the weighted bandwidth of each interconnection port of each gateway device according to the second link state information table, a preset interconnection port role and a preset link weighted bandwidth evaluation method, and correspondingly storing the roles and the weighted bandwidth into the second link state information table to acquire a third link state information table; the interconnection port role comprises a blocking interconnection port, a WAN interconnection port and a LAN interconnection port;
generating corresponding link control information according to the third link state information table, and sending a write instruction carrying the link control information to each gateway device, so that each gateway device performs load balancing processing according to the received write instruction; the link control information includes a weighted bandwidth of each WAN port, a role of each interconnect port, and a weighted bandwidth of each interconnect port.
2. The link load balancing method according to claim 1, wherein the calculating the weighted bandwidth of each WAN port of each gateway device according to the first link state information table and a preset link weighted bandwidth evaluation method specifically includes:
for the ith WAN port of the kth gateway device, acquiring the state information of the ith WAN port of the kth gateway device according to the first link state information table; wherein the status information of the ith WAN port at least comprises a downlink bandwidth average value DBkiAverage value UB of uplink bandwidthkiA weight factor AkiDownlink flow statistics DTkiAnd upstream traffic statistics UTki;
According to the formulaCalculating weighted bandwidth WB of ith WAN port of kth gateway deviceki(ii) a Wherein k is more than or equal to 1 and less than or equal to M, M represents the number of gateway equipment in the system, and M is more than or equal to 1; i is more than or equal to 1 and less than or equal to N, N represents the number of WAN ports of the kth gateway equipment, and N is more than or equal to 1.
3. The link load balancing method according to claim 1, wherein the obtaining of the role and the weighted bandwidth of each interconnect port of each gateway device according to the second link state information table, the preset interconnect port role, and the preset link weighted bandwidth evaluation method specifically includes:
acquiring state information of all the interconnection ports according to the second link state information table; the state information of the interconnection port at least comprises a home terminal MAC address and an opposite terminal MAC address of the interconnection port;
acquiring interconnection port connection topological graphs among all gateway devices according to the state information of all interconnection ports and a preset interconnection topology construction method;
and determining the role of each interconnection port of each gateway device according to the topological graph, the preset interconnection port role and the preset link weighted bandwidth evaluation method, and calculating the weighted bandwidth of each interconnection port of each gateway device.
4. The link load balancing method according to claim 1, wherein the method further comprises:
when receiving an event message sent by any gateway equipment, sending a reading instruction to each gateway equipment; the event message indicates that the gateway equipment generates a preset event currently and needs to adjust link parameters; the preset event at least comprises a WAN port disconnection event or a bandwidth change event.
5. A link load balancing method is characterized in that the method is suitable for a multi-gateway multi-WAN-port networking system, and the system comprises a controller and at least one gateway device which is in connection communication with the controller through a network; each gateway device comprises at least one LAN port, at least one WAN port and at most two interconnection ports; each gateway device is connected and networked with the interconnection ports of other gateway devices through the interconnection port of the gateway device; the method is performed by any gateway device; the method comprises the following steps:
when a read instruction sent by the controller is received, link state information is obtained and sent to the controller, so that the controller generates corresponding link control information according to the received link state information sent by each gateway device, a preset interconnection port role and a preset link weighted bandwidth evaluation method, and sends a write instruction carrying the link control information to each gateway device; the link control information comprises the weighted bandwidth of each WAN port of each gateway device, the role of each interconnection port and the weighted bandwidth of each interconnection port; the interconnection port role comprises a blocking interconnection port, a WAN interconnection port and a LAN interconnection port;
and when a write command sent by the controller is received, carrying out load balancing processing according to the link control information carried in the received write command.
6. The link load balancing method according to claim 5, wherein the obtaining of the link state information specifically includes:
acquiring state information of each WAN port; the state information of the WAN port at least comprises a downlink bandwidth average value, an uplink bandwidth average value and a preset weight factor of the WAN port;
acquiring state information of each interconnection port; the state information of the interconnection port at least comprises a home terminal MAC address and an opposite terminal MAC address of the interconnection port.
7. The link load balancing method according to claim 5, wherein when receiving the write command sent by the controller, performing load balancing processing according to link control information carried in the received write command specifically includes:
when a write instruction sent by the controller is received, setting corresponding load balancing control parameters and interconnection port control parameters according to link control information carried in the received write instruction;
adding a WAN interconnection port to a preset WAN port schedulable list and a preset WAN port ongoing scheduling list, adjusting a weight parameter of a policy route according to the load balancing control parameter, and performing policy route selection according to the WAN port ongoing scheduling list and the weight parameter of the policy route;
and adjusting the role of the interconnection port according to the interconnection port control parameters, and starting a dialing function of the WAN interconnection port and an address allocation service of the LAN interconnection port.
8. The link load balancing method according to claim 5, wherein the method further comprises:
when receiving a system message, sending an event message to the controller, so that the controller sends a reading instruction to each gateway device according to the received event message; the system message indicates that the gateway equipment generates a preset event currently and needs to adjust link parameters; the preset event at least comprises a WAN port disconnection event or a bandwidth change event.
9. A link load balancing device is characterized in that the device is suitable for a multi-gateway multi-WAN-port networking system, and the system comprises a controller and at least one gateway device which is in connection communication with the controller through a network; each gateway device comprises at least one LAN port, at least one WAN port and at most two interconnection ports; each gateway device is connected and networked with the interconnection ports of other gateway devices through the interconnection port of the gateway device; the device is arranged in the controller; the device comprises a core control module, a link evaluation module and an interconnection evaluation module;
the core control module is used for sending a reading instruction to each gateway device, so that each gateway device sends link state information to the controller according to the received reading instruction;
the core control module is further configured to update a preset link state information table according to the received link state information sent by each gateway device, so as to obtain a first link state information table;
the link evaluation module is used for calculating the weighted bandwidth of each WAN port of each gateway device according to the first link state information table and a preset link weighted bandwidth evaluation method, and correspondingly storing the weighted bandwidth into the first link state information table to obtain a second link state information table;
the interconnection evaluation module is used for acquiring the role and the weighted bandwidth of each interconnection port of each gateway device according to the second link state information table, a preset interconnection port role and a preset link weighted bandwidth evaluation method, and correspondingly storing the roles and the weighted bandwidth into the second link state information table to acquire a third link state information table; the interconnection port role comprises a blocking interconnection port, a WAN interconnection port and a LAN interconnection port;
the core control module is further configured to generate corresponding link control information according to the third link state information table, and send a write instruction carrying the link control information to each gateway device, so that each gateway device performs load balancing processing according to the received write instruction; the link control information includes a weighted bandwidth of each WAN port, a role of each interconnect port, and a weighted bandwidth of each interconnect port.
10. A link load balancing device is characterized in that the device is suitable for a multi-gateway multi-WAN-port networking system, and the system comprises a controller and at least one gateway device which is in connection communication with the controller through a network; each gateway device comprises at least one LAN port, at least one WAN port and at most two interconnection ports; each gateway device is connected and networked with the interconnection ports of other gateway devices through the interconnection port of the gateway device; the device is arranged in any gateway equipment; the device comprises:
the information acquisition module is used for acquiring link state information and sending the link state information to the controller when receiving a read instruction sent by the controller, so that the controller generates corresponding link control information according to the received link state information sent by each gateway device, a preset interconnection port role and a preset link weighted bandwidth evaluation method, and sends a write instruction carrying the link control information to each gateway device; the link control information comprises the weighted bandwidth of each WAN port of each gateway device, the role of each interconnection port and the weighted bandwidth of each interconnection port; the interconnection port role comprises a blocking interconnection port, a WAN interconnection port and a LAN interconnection port;
and the load balancing module is used for carrying out load balancing processing according to the link control information carried in the received write instruction when the write instruction sent by the controller is received.
11. A controller comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, the processor when executing the computer program implementing the link load balancing method of any one of claims 1 to 4.
12. A gateway device comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, the processor implementing the link load balancing method according to any one of claims 5 to 8 when executing the computer program.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010073494.1A CN111277510A (en) | 2020-01-22 | 2020-01-22 | Link load balancing method and device, controller and gateway equipment |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010073494.1A CN111277510A (en) | 2020-01-22 | 2020-01-22 | Link load balancing method and device, controller and gateway equipment |
Publications (1)
Publication Number | Publication Date |
---|---|
CN111277510A true CN111277510A (en) | 2020-06-12 |
Family
ID=71001246
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202010073494.1A Pending CN111277510A (en) | 2020-01-22 | 2020-01-22 | Link load balancing method and device, controller and gateway equipment |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN111277510A (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114513464A (en) * | 2021-12-31 | 2022-05-17 | 深圳市联洲国际技术有限公司 | Flow load balancing scheduling method, device, equipment and storage medium |
CN114553964A (en) * | 2020-11-20 | 2022-05-27 | 中移动信息技术有限公司 | Control method, device and equipment of simulcast system and simulcast system |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106357433A (en) * | 2016-08-30 | 2017-01-25 | 上海寰创网络科技有限公司 | Implementation method of solid-interface and multi-WAN-port gateway |
US9762495B1 (en) * | 2016-09-13 | 2017-09-12 | International Business Machines Corporation | Weighted distribution across paths of degraded quality |
CN107579923A (en) * | 2017-09-18 | 2018-01-12 | 迈普通信技术股份有限公司 | The balancing link load method and SDN controllers of a kind of SDN |
-
2020
- 2020-01-22 CN CN202010073494.1A patent/CN111277510A/en active Pending
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106357433A (en) * | 2016-08-30 | 2017-01-25 | 上海寰创网络科技有限公司 | Implementation method of solid-interface and multi-WAN-port gateway |
US9762495B1 (en) * | 2016-09-13 | 2017-09-12 | International Business Machines Corporation | Weighted distribution across paths of degraded quality |
CN107579923A (en) * | 2017-09-18 | 2018-01-12 | 迈普通信技术股份有限公司 | The balancing link load method and SDN controllers of a kind of SDN |
Non-Patent Citations (1)
Title |
---|
李闯等: "基于模糊算法的VoWLAN切换策略研究", 《信息通信》 * |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114553964A (en) * | 2020-11-20 | 2022-05-27 | 中移动信息技术有限公司 | Control method, device and equipment of simulcast system and simulcast system |
CN114513464A (en) * | 2021-12-31 | 2022-05-17 | 深圳市联洲国际技术有限公司 | Flow load balancing scheduling method, device, equipment and storage medium |
CN114513464B (en) * | 2021-12-31 | 2024-03-29 | 深圳市联洲国际技术有限公司 | Traffic load balancing scheduling method, device, equipment and storage medium |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN109194577B (en) | Traffic engineering method and device of segmented routing network based on partial deployment | |
CA2882535C (en) | Control device discovery in networks having separate control and forwarding devices | |
EP1246415B1 (en) | Method and apparatus for communications traffic engineering | |
US9210071B2 (en) | Automated traffic engineering for fat tree networks | |
EP2817928B1 (en) | Controller placement for fast failover in the split architecture | |
EP2552065A1 (en) | Controller placement for fast failover in the split architecture | |
US20130028070A1 (en) | Method and apparatus for resilient routing of control traffic in a split-architecture system | |
US11336577B2 (en) | Method and apparatus for implementing load sharing | |
US9800499B2 (en) | Ethernet switch and method for routing Ethernet data packets | |
CN105471747B (en) | A kind of intelligent router route selecting method and device | |
CN102143042A (en) | Virtual cluster router system and flow sharing method thereof, controller and sub routers | |
CN106713177A (en) | Multi-controller wmSDN networking method | |
CN111277510A (en) | Link load balancing method and device, controller and gateway equipment | |
CN104243334A (en) | Method and equipment for preventing link congestion | |
CN102035741B (en) | The retransmission method of unicast message and equipment in a kind of ring topology network | |
US9825850B2 (en) | Network controlling method and network controller | |
US20150036508A1 (en) | Method and Apparatus For Gateway Selection In Multilevel SPB Network | |
CN110213161B (en) | Routing scheduling method and related equipment | |
CN110912823A (en) | SDN and IPv6 reliable routing mechanism oriented to heterogeneous wired and wireless converged network | |
Nithin et al. | Efficient load balancing for multicast traffic in data center networks using SDN | |
US20160065455A1 (en) | Flow cache based mechanism of packet redirection in multiple border routers for application awareness | |
CN111385182B (en) | Traffic transmission method, device and system | |
US11876680B2 (en) | Method and apparatus for determining link for forwarding service flow | |
CN115378875A (en) | Reliable flow scheduling method suitable for different network environments | |
CN112714061B (en) | Routing method and device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
AD01 | Patent right deemed abandoned |
Effective date of abandoning: 20230228 |
|
AD01 | Patent right deemed abandoned |