CN112751762A - Automatic routing platform for multi-operator network link load outbound - Google Patents

Automatic routing platform for multi-operator network link load outbound Download PDF

Info

Publication number
CN112751762A
CN112751762A CN202011641784.8A CN202011641784A CN112751762A CN 112751762 A CN112751762 A CN 112751762A CN 202011641784 A CN202011641784 A CN 202011641784A CN 112751762 A CN112751762 A CN 112751762A
Authority
CN
China
Prior art keywords
routing
link
data
platform
route
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
CN202011641784.8A
Other languages
Chinese (zh)
Inventor
刘文平
高宏松
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Jingmen Huiyijia Information Technology Co ltd
Original Assignee
Jingmen Huiyijia Information Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Jingmen Huiyijia Information Technology Co ltd filed Critical Jingmen Huiyijia Information Technology Co ltd
Priority to CN202011641784.8A priority Critical patent/CN112751762A/en
Publication of CN112751762A publication Critical patent/CN112751762A/en
Withdrawn legal-status Critical Current

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L45/00Routing or path finding of packets in data switching networks
    • H04L45/12Shortest path evaluation
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L45/00Routing or path finding of packets in data switching networks
    • H04L45/12Shortest path evaluation
    • H04L45/121Shortest path evaluation by minimising delays
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L45/00Routing or path finding of packets in data switching networks
    • H04L45/12Shortest path evaluation
    • H04L45/125Shortest path evaluation based on throughput or bandwidth
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L45/00Routing or path finding of packets in data switching networks
    • H04L45/74Address processing for routing
    • H04L45/745Address table lookup; Address filtering
    • H04L45/7453Address table lookup; Address filtering using hashing
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00Traffic control in data switching networks
    • H04L47/10Flow control; Congestion control
    • H04L47/12Avoiding congestion; Recovering from congestion
    • H04L47/125Avoiding congestion; Recovering from congestion by balancing the load, e.g. traffic engineering

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

The multi-operator network link load outbound automatic routing platform provided by the invention is oriented to the outbound situations of multiple operator links encountered by small and medium-sized enterprises and personal families, solves the outbound routing problem, improves the accuracy in outbound routing, reduces the link load outbound cost, and meets the requirement that when a user access request arrives, a proper link can be selected for the user from multiple internet links by relying on an accurate IP database and an efficient retrieval algorithm, and the outbound routing link realizes load balancing; the method comprises the steps of accessing external network resources when the user goes out of a station, keeping links connected by adopting a session maintenance technology under the condition of special application so as to avoid influencing the application effect, ensuring user experience in the process of selecting a route and ensuring that the user does not feel link interruption, meeting the requirements of unit or individual network resource access quality and efficiency by adopting a network link health detection algorithm, and selecting a proper link when accessing the resources so as to ensure the quality and stability of link access.

Description

Automatic routing platform for multi-operator network link load outbound
Technical Field
The invention relates to a multi-operator network routing platform, in particular to a multi-operator network link load outbound automatic routing platform, and belongs to the technical field of multi-operator network automatic routing.
Background
Internet service providers generally include mobile networks, communication networks, telecommunication networks, educational networks, etc. in China, because of the needs of practical applications, no matter individuals or units have multiple network links to access, but the resources operated and maintained by each network operator are different, which brings a problem to the user access, that is, if a unit rents more than two network links at the same time, when personnel in the unit access the internet resources by surfing the internet, the problem of delay of cross-link access may be encountered, for example, the resources of the telecommunication network are accessed but accessed through the communication network links, the time is delayed in the routing process, the efficiency is reduced, and the internet speed and the connection effectiveness are not guaranteed. Secondly, a plurality of links deployed by a unit are difficult to be fully and effectively utilized in a balanced manner, which may cause the situation that a telecommunication link is very busy and a communication link is idle in bandwidth, thereby causing the waste of network resources.
Inbound traffic refers to traffic generated when an external user accesses internal resources through the internet, and is required as a service provider to be able to select an appropriate link from a plurality of internet links for the user when a user access request arrives; meanwhile, when a certain link is interrupted, the access of a user can be switched to other normal links, the stable outbound flow of the access is ensured to be the flow generated when an internal user accesses external resources through the Internet, and a proper link needs to be selected when the access to the resources to meet the requirements of the access quality and efficiency of network resources of units or individuals, so that the access quality and stability of the link are ensured.
The prior art has a plurality of technologies for solving network link outbound, but the product efficiency is not high overall, the use experience of the product is not good, and there are always the following problems, and the problems in the prior art mainly include:
first, configuration management is inconvenient: currently, many units rent a plurality of network links, no efficient and practical method exists for selecting the network links, and routing selection is performed according to a router, the method is very mechanical, the problem of link can be caused in many cases, the experience of a user is unstable due to the fact that the routing is not very flexible, the configuration is very complex, special routing configuration professional personnel is needed for configuration, the operation management is complex, and the maintenance cost is high;
second, link utilization is low: on the basis of a traditional network routing algorithm, if a link cannot be intelligently selected, bandwidth waste is often caused, the utilization rate is low, and the use experience of a user is seriously influenced, multiple network links are shared, so that the user experience is improved, the problem cannot be well solved in the prior art, and an intelligent network routing algorithm platform is urgently required to be developed, so that the network bandwidth utilization rate is improved;
third, the scheme is complex and load unbalanced: in many current network environments, multiple internet access links become necessary choices for ensuring the access speed and stability of the internet, however, in the prior art, an algorithm facing multiple links and configured based on routing and routing allocates a different IP address network segment for each internet link intranet through a static invariance principle to realize the selection of the links, such a solution depends on a complex design, although the problem of single-point faults of some access links can be solved, real balanced load is not realized, and the management method is complex;
fourth, the degree of intelligence is very low: because the configured route does not know the current traffic load and active session of the selected link, the mechanism for balancing the load is very inaccurate, only the route to be used is mechanically routed to different router outlets in turn, which is not scientific and violates the connection principle, in addition, when the intranet user accesses the internet, some links are easier to pass than other links, although the routing protocol relates to partial proximity and accessibility, but it is impossible to combine various factors such as the HOP number of routers, the delay to a target network and the load condition of the links, and the intelligence degree of outbound routing is very low.
According to the invention, by developing a multi-operator network link load outbound automatic routing platform based on the modified linux kernel, the problem that medium and small enterprises and individuals surf the internet and face the bottleneck of network link selection is solved, the efficiency of network link selection is improved, the link load outbound cost is reduced, and a high-efficiency network link selection scheme is brought to the medium and small enterprises and the individuals.
The user range oriented by the multi-operator network link load outbound automatic routing platform is the outbound situation of a plurality of operator links encountered by small and medium-sized enterprises and personal families, is a basis for solving the outbound routing problem, and the core technology and key innovation point for realizing the accuracy and the high efficiency of routing are the core technology and the key innovation point of the invention.
The accuracy is improved in outbound routing, the problem that the link load is balanced in reality needs to be solved by means of an accurate IP database and an efficient retrieval algorithm, and the problem that the link load is not uniform in the process of outbound routing is solved by the high efficiency of outbound routing.
The outbound accesses the external network resources, and a session maintenance technology is needed under special application conditions, so that the link is kept connected, and the application effect is not influenced.
In the routing process, a network link health detection algorithm is needed to ensure the user experience and ensure that the user does not feel the link interruption.
The automatic routing platform for multi-operator network link load outbound is developed, and basic requirements of users, attribute information such as network setting, network card configuration and the like, routing rule setting and acl setting are guaranteed, so that the platform can approach the network, and a log platform is added.
Disclosure of Invention
Aiming at the defects of the prior art, the invention provides a light and efficient solution for multi-operator network link selection, the system architecture of a platform is not required to be changed, the system architecture is mounted under the platform in a hook function mode, the network link is provided with an outbound automatic routing result in a bypass mode by intercepting and analyzing data flow, the access efficiency is not influenced, a kernel source code is changed through the hook function, a kernel module is developed, a route or a circuit is selected according to a result inquired and returned in an IP database, the route is selected in the kernel, the problem of long inquiry time under the condition of a large number of iptables rules is solved, a routing method is modified through modifying the kernel module, an IP section database is led into the kernel, an algorithm is added in the IP database search, the searching speed is higher than that of the iptables rules, the database search function is programmed into the kernel, the outbound circuit is ensured to be smooth through updating and maintenance of the database, the quality and stability of the link access are guaranteed.
In order to achieve the technical effects, the technical scheme adopted by the invention is as follows:
the automatic routing platform for the multi-operator network link load outbound carries out linux kernel transformation under the linux platform, accesses link data accessed in the outbound according to intelligent classification of resource types without changing the system architecture of the platform, is mounted under the platform in the form of hook functions, intercepts data flow and analyzes the data flow, and gives an automatic routing result for the network link load outbound in the form of a bypass, and comprises the following steps: setting a network link, selecting an outbound link, and selecting a routing algorithm of various links, developing a platform kernel module, interacting a user layer and a kernel layer, and keeping a session;
the outbound data routing method comprises the following steps after a data packet enters a platform: firstly, entering a data packet processing module of a platform, marking the data packet by detaching the data packet, then selecting a route, matching the customized route, and if not, continuing to enter an automatic route selection algorithm; secondly, entering a static routing algorithm and proceeding downwards all the time according to the sequence of the priority; thirdly, if the automatic routing is unsuccessful, selecting default routing;
the conversation maintaining technology ensures that data accessing the same target place walks the same link, the conversation maintaining technology performs cache automatic registration at the position matched with the link, and the specific working process of the improved algorithm is as follows:
firstly, according to a source IP, a target IP, a protocol type and a source IP address of a route in a current routing node in a visiting data stream, collecting and storing parameters in a data structure, and determining whether to carry out session maintenance according to a feedback result;
secondly, if the session is required to be maintained, entering a cache table, judging whether parameters are in the table according to a source IP and a target IP of the visiting data, if no data record exists in the table, indicating that the visiting is the first time, if the route is unobstructed and selectable, adding the data record into the table, otherwise, continuously matching the next route;
thirdly, if the session is not required to be maintained, routing is carried out according to a polling or weighted polling algorithm selected by the user;
the cache table HASH chain table is directly read by the kernel file, and the format of the data record is as follows: the first number is the source IP of the initial visiting data stream, the target IP of the second visiting data stream, the third number is the source IP of the route, http represents the protocol, the last number is the time, and the platform determines whether the data is deleted according to whether the current time and the recorded time exceed a certain difference value; modifying the routing module, adding the cache table to store data records, and recompiling the kernel module;
the realization of the cache function is realized in a linux inner core, temporary data is stored by adding a HASH chain table data structure, data records are quickly searched through a HASH algorithm, the life cycle of the data records is mastered by a timer in real time, and the position of the linux inner core is reformed: under the route.c file under/BUILD/kernel-2.6.32-358. el6/linux-2.6.32-358.el6.x86_64/net/ipv4, the modification module is: adding list matching in a fib _ semantic _ match function, adding a hash list initialization in a fib _ init () function, adding a session _ list _ add () in the fib _ hash _ match function, adding a function of the linked list information in the function, storing the incoming data information which is not in the linked list and needs to be kept, and judging whether corresponding information exists by the session _ list _ search (ip) function according to whether the target ip information is in the cache list or not; session _ list _ del is a Timer which is used for regularly clearing the overtime data record in the linked list, Timer is a Timer executed according to the CPU frequency, timing is absolute, and after the Timer is started, the corresponding action can be executed in the specified time;
the working process of the kernel related module is as follows:
the first process is that data stream is obtained in a network layer, a data packet is disassembled, and a source IP, a target IP and an access type parameter in the data stream are obtained;
if the access parameter group is in the cache table, directly returning the corresponding link information, namely, not re-matching the route, or not walking different links, thereby realizing the session maintenance function;
if the access parameter group is not in the cache table and the route is not matched, continuing to search for a proper route, and finally returning the route if the access parameter group is not in the cache table but has a passed route, adding data into the cache table, otherwise, exiting the cycle;
and if no appropriate routing table exists, selecting a default route, clearing the expired data by the Timer in time, and deleting the useless data records within a certain time by the Timer according to the set critical value.
The multi-operator network link load outbound automatic routing platform is developed under a linux operating platform, the whole platform is based on a BS framework and is operated through a management platform, the key technology lies in communication between a user layer and a kernel layer, the user layer sends an instruction to the kernel layer and sends data to the kernel, the kernel interacts according to the instruction after receiving the data, and communication between the kernel and a user is realized through a fast socket technology;
the link balance load outbound platform is deployed at the edge of a unit network and rents a plurality of links, and the whole structure comprises:
and (3) link customization: selecting a link according to special needs of a user, wherein the link has the highest priority, the judging stage of the link is positioned at a PREROUTING point, the data entering a platform is analyzed, and an outbound link is selected according to the limited range of iptables;
an automated routing module: the automatic routing algorithm structure comprises a database routing algorithm, a polling routing algorithm, a weighted polling routing algorithm, a minimum bandwidth routing algorithm and a fastest response routing algorithm;
a static routing module: the static routing mode is close to the static route, but different from the static route, the static routing can set an IP section and set the IP section on the target IP section;
a default routing module: the routing is not default routing and is used as a branch selection for automatic routing and non-communication, so that the smooth link is ensured under the condition of maximum limit;
session maintenance: the same route is guaranteed to be taken among multiple sessions, a cache is needed to record the taken link at all times, and multiple routing is guaranteed to select the same link.
The multi-operator network link load outbound automatic routing platform is characterized in that the link outbound platform intelligently selects and accesses a data outbound link, link setting, a routing scheme and session maintenance are all necessary modules, health detection and DNS proxy technologies are all hidden modules, and a specific mechanism module of the platform is as follows:
link definition, namely links provided by each operator, the links are accessed to personal or unit equipment through line media, the connection between the SSA and each link is through a network interface, the defined links find out an outlet connected with the link through a route, and the link design is divided into an extranet link definition and an intranet link definition;
a routing algorithm, wherein routing is to select a link from the network, judge the line where the accessed server resource is located, need a comprehensive IP section database for judgment, and then perform routing;
customizing routing, and forcing data to go through a certain line according to the needs of a user;
static routing, wherein a user sets a certain IP section to go through one network operator and a certain IP section to go through another network operator, and performs link selection according to a target IP section;
default routing, namely, directing the link which fails to be selected in the database to the default link by setting the default route;
the conversation is kept, and under the condition of starting the conversation keeping, the conversation can be kept on the same link under the condition of dynamically modifying the polling link;
the hidden module is used for detecting the connection condition of the link through a third-party tool so as to judge the health condition, judging the situations of automatic routing, static routing and default routing in real time and analyzing the connection condition of the link;
the multi-operator network link load outbound automatic routing platform further performs network connectivity diagnosis through a network tool, performs server switching in time and ensures service continuity;
the SSA detects the health condition of the link at regular time when the station is out of the station, and the specific method is that a loop program detects the link at regular time, analyzes the response result and feeds back the result in real time according to the analysis result;
the specific technical details comprise: adopting a Sleep function in the platform for timeout and time interval, adding a # include < windows.h >, and then directly calling the Sleep function in the main function, wherein parameters comprise Sleep time and unit millisecond; the alarm function can be called, and an alarm module in the SSA platform sends a mail to an administrator; tool: performing ping operation on the link gateway, entering a sleep state if the returned result is yes, and calling an alarm module if the returned result is not yes, wherein other tools, namely Nmap and Tracepath, have the same test logic with the ping command; designing an interface: basic settings are selected, and the required input on the interface comprises a detection method, a time interval and timeout time.
The multi-operator network link load outbound automatic routing platform comprises the following steps:
1) firstly, adding an SSA _ LINK _ LOAD.ko module in a kernel layer, then adding all LINK IP database information into the kernel by calling a user layer interface client, setting LINK conditions in a database by judging LINKs in an application layer, listing all communication situations of the current LINK, carrying out route selection if the LINK is completely communicated, transmitting an obstructed LINK and an unobstructed LINK into the kernel if the LINK is not completely communicated, calling a diagnosis client program, introducing data leading to the obstructed LINK into an adjacent unobstructed LINK, and putting LINKs of different operators into different tables;
2) after the selection of the polling algorithm, if the database routing algorithm is adopted before, the unloading database routing algorithm module: ko, deleting the routing rule in the routing table corresponding to the operator in the LINK; if a weighted polling algorithm is adopted, a table named table25 is found to delete the routing rule therein, and all the unobstructed links are added into a specific routing table for polling.
The multi-operator network link load outbound automatic routing platform further comprises the following steps of: and (3) storing basic information of the link, putting the added link information into a database, and carrying out interaction by displaying, editing and deleting:
1) displaying the basic information of the port aggregation, displaying the port aggregation information acquired from the database in the foreground, and writing a file by adopting Python language as follows:
/usr/local/lib/python2.7/site-packages/SSA/System/Link_Load_bancer/inter_agg_set.py;
2) adding port aggregation basic information, adding the port aggregation information into a database according to parameters input by a user and the selected type, and calling a port aggregation command to generate a port aggregation object;
3) editing the port aggregation basic information, judging according to the input parameters of the user, and then modifying the data of the database and the port aggregation information under the platform;
4) removing the basic port aggregation information, and deleting the corresponding data records in the database and the port aggregation information under the platform according to the parameters input by the user;
5) and acquiring an available interface function, listing available network cards for selection by reading network card information under ifconfig, and further judging when selecting for use, wherein the network card which is not marked can only be in one vlan, and the network card which is marked can be in a plurality of vlans.
The multi-operator network link load outbound automatic routing platform further has the following logic structure of an ACL rule:
firstly, extracting ACL rule parameters in a database, displaying the ACL rule parameters on a foreground page, and judging whether a data table exists or not and whether reading is abnormal or not;
secondly, adding ACL rules, adding parameters transmitted from a foreground, judging the address formats of a source IP and a target IP, the range of port numbers and the opening of IP fragments, and if the added data is correct, firstly adding the data under the platform and then adding the data into a database to ensure that the data and the rules under the platform are synchronous;
thirdly, editing an ACL rule, firstly judging parameters of the editing rule, then judging the correctness of the rule, adding all rules in the nat chain again when adding into the platform to ensure the sequence, and adding and using a shell script,/etc/nginx/Sys _ iptables/iptables _ m.sh again;
deleting the ACL rules, namely deleting a plurality of rules, firstly deleting the rules under the platform, deleting the rules in the database under the condition of successful deletion, deleting the rules under the platform, simultaneously considering the iptables rules of other functions, and modifying the iptables rules of the platform by using the same adding interface script;
and fifthly, calling the iptables rules to generate a unified interface, and writing all the iptables rules into/etc/nginx/sys _ iptables/iptables _ m.sh.
The multi-operator network link load outbound automatic routing platform comprises the following steps: routing part codes in Linux source codes forward data packets of each layer of a network, the routing part codes are positioned under a D \ Linux-2.6.11.12\ net \ ipv4 path, routing in a kernel is an important part, an IP _ rt _ init () function is called in an entry function IP _ init (), initialization operation of a routing function table is completed, routing codes are initialized, the platform is called by an IP _ init interface of an initialized IP sub-platform when being started, initialization of strategy routing is realized by a fib _ rules _ init function and initialized to a netdev _ chain notification chain registration processing hook, and the registration processing hook is fib _ rules _ event.
The multi-operator network link load outbound automatic routing platform further comprises the following steps of: the user sets the selected link according to the accessed resource parameters, customizes the link by combining the routing rule through the forwarding control or marking control of the iptables, or sets the static route according to the network segment, the iptables is limited, otherwise the priority is reduced to the third, and the link balancing load solves the routing problem by configuring the firewall and the routing rule;
the intranet client is connected to the Internet through a Linux host, and when the Linux host is connected with the Internet, two lines are arranged to perform static routing on the intranet;
before routing a data packet, marking a mark for the data packet according to a protocol and a target port of the data packet, then appointing a corresponding rule, performing strategy routing according to the mark of the data packet, marking a specific data packet by using a mangle table, and marking a PREROUTING chain before routing;
the command adds a rule in a PREROUTING chain of a mangle table, a data packet from an eth0 interface is marked, matching rules of the command are respectively a TCP protocol, a target port number is 80 and a UDP protocol, a target port number is 53, the marked values are respectively 1 and 2, after the data packet passes through the PREROUTING chain, the data packet enters a routing module, and policy routing is carried out on the data packet to execute the following two commands, and a corresponding rule is added:
1)ip rule add from all fwmark 1 table 10;
2)ip rule add from all fwmark 2 table 20;
the two commands indicate that all packets with the flag of 1 are routed by using the routing table10, while all packets with the flag of 2 are routed by using the routing table20, the routing tables 10 and 20 respectively use gateways on ChinaNet and Cernet lines as default gateways, and the specific setting commands are as follows:
1)ip route add default via 10.10.1.1 dev eth1 table 10;
2)ip route add default via 10.10.2.1 dev eth2 table 20;
the two commands specify 10.10.1.1 and 10.10.2.1 on routing tables 10 and 20, respectively, as default gateways, which are located on the ChinaNet and Cernet lines, respectively, with packets using routing table10 going out through the ChinaNet line and packets using routing table20 going out through the Cernet line.
The multi-operator network link load outbound automatic routing platform is characterized in that the global session maintenance is that a cache mechanism table is added in a kernel routing module to register parameters of a source IP, a target IP, time and a protocol type, data needing session maintenance next time comes and then is firstly judged by a progress table, if the data does not exist in the cache table, the data is added into the table, if the data exists, the line is moved, the time in the table is modified, a regular detection table of a timer is set according to the registration time of the table, if the difference between the data time of the table and the current time is greater than a certain value, the fact that the time is not newly connected is indicated, namely the data is deleted if the time is overtime.
Compared with the prior art, the invention has the following contributions and innovation points:
firstly, the multi-operator network link load outbound automatic routing platform is oriented to a user range of the outbound situation of a plurality of operator links encountered by small and medium-sized enterprises and personal families, solves the outbound routing problem, improves the accuracy in outbound routing, relies on an accurate IP database and an efficient retrieval algorithm to solve the problem, and realizes load balance of the outbound routing links; outbound access to external network resources, and a session maintenance technology is adopted under the special application condition, so that links are kept connected, and the application effect is not influenced; the user experience is ensured in the routing process, the user can not feel the link interruption, and the core technology and key innovation points of the invention are that the accuracy and the high efficiency of routing are realized by adopting a network link health detection algorithm;
secondly, the invention solves the problem that medium and small enterprises and individuals surf the internet and face the bottleneck of network link selection by developing a multi-operator network link load outbound automatic routing platform based on the modified linux kernel, improves the efficiency of network link selection, reduces the cost of link load outbound, and brings an efficient network link selection scheme for the medium and small enterprises and individuals. Serving as a service provider, and selecting a proper link from a plurality of internet links for a user when a user access request arrives; meanwhile, when a certain link is interrupted, the access of a user can be switched to other normal links, the stability of the access is ensured, the requirements of the access quality and efficiency of network resources of units or individuals are met, and the proper link is selected when the resources are accessed, so that the quality and the stability of the access of the link are ensured;
thirdly, configuration management is convenient: currently, many units rent a plurality of network links, aiming at the selection of the network links, the invention provides a high-efficiency and practical method, mechanical route selection is not performed according to a router, the problem of link is avoided, the route is intelligent and flexible, the user experience is stable, the configuration is relatively simple, special route configuration professional configuration is not needed, the operation management is simple, and the maintenance cost is low;
fourthly, the link utilization rate is high: the multi-operator network link load outbound automatic routing platform can intelligently select links, bandwidth waste is avoided, the utilization rate is high, the user experience is good, and multiple network links are shared to improve the user experience;
fifth, the scheme is clear and load balancing: the invention is oriented to the algorithm of multiple links, allocates a different IP address network segment for the intranet of each internet link through an intelligent algorithm to realize the selection of the link, has clear and definite solution scheme, not only can solve the single-point failure problem of some access links, but also can realize the real balanced load;
sixth, the intelligent degree is higher: the invention provides a light and high-efficiency solution for multi-operator network link selection, without changing the system architecture of a platform, the system architecture is mounted under the platform in a hook function mode, the network link is provided with an outbound automatic routing result in a bypass mode by intercepting and analyzing data flow, the access efficiency is not influenced, a kernel source code is changed through the hook function, a kernel module is developed, a route or a circuit is selected according to a result inquired and returned in an IP database, the route is selected in the kernel, the problem of long inquiry time under the condition of a large number of iptables rules is solved, the routing method is modified by modifying the kernel module, an IP section database is led into the kernel, an algorithm is added in IP database search, the searching speed is higher than the iptables rule search speed, the database search function is programmed into the kernel, the outbound route is ensured to be smooth through the updating and maintenance of the database, the quality and stability of the link access are guaranteed, which is also the key technology of the invention.
Drawings
Fig. 1 is a schematic diagram of the overall structure of the automated routing platform of the present invention.
Fig. 2 is a flow chart of the link health detection of the present invention.
Fig. 3 is a flow diagram of a multi-operator network link load outbound automated routing platform routing process.
Fig. 4 is a data flow diagram of a multi-operator network link load outbound automated routing platform module.
FIG. 5 is a schematic diagram of the logical structure of the ACL rules of the present invention.
Fig. 6 is a flow chart of the session holding method of the present invention.
FIG. 7 is a schematic flow chart of a simulation experiment of the present invention.
Detailed Description
The technical solution of the multi-operator network link load outbound automatic routing platform provided by the present invention is further described below with reference to the accompanying drawings, so that those skilled in the art can better understand the present invention and can implement the same.
The invention provides an automatic routing platform for multi-operator network link load outbound aiming at link balanced load outbound, linux kernel transformation is carried out under a linux platform, and link data accessed in outbound are intelligently classified and accessed according to resource types.
The solutions of the prior art are endless, but the implementation principle, efficiency and cost are all unreasonable, and in order to solve the problem of multi-operator network routing, the invention provides a multi-operator network link load outbound automatic routing platform, which solves the problems that: setting network links, selecting outbound links, routing algorithms of various links, development of platform kernel modules, interaction between a user layer and a kernel layer, and a session holding method.
The invention provides a light and efficient solution for selecting the network links of multiple operators, the system architecture of a platform is not required to be changed, the network links are mounted under the platform in a hook function mode, the data flow is intercepted and analyzed, the network link load outbound automatic routing result is given in a bypass mode, the access efficiency is not influenced, and the invention is also a bright point.
In the link balancing load outbound platform, link setting is the most basic content, an outbound routing algorithm is the core algorithm of the whole outbound platform, the invention comprises an automatic routing, a customized routing, a static routing and a default routing algorithm of the link outbound, wherein the automatic routing covers most scenes, and the invention further comprises a polling routing algorithm, a weighted polling routing algorithm, a minimum bandwidth proportion routing algorithm and a fastest response routing algorithm, which are the core technology of the invention, and a session maintenance algorithm.
The routing algorithm is a method for selecting which network link to walk according to the network resource accessed by the user and the current link condition, the routing method of the invention comprises the steps of customized routing, automatic routing, static routing and default routing, according to the data packet principle, the four routing methods include the highest priority of customized routing, the lowest priority of automatic routing, the third priority of static routing and the lowest priority of default routing. Because the platform is generally placed on the edge of a network, in order to separate the intranet address from the extranet address, an access control technology is added to make SNAT or DNAT, and the conversion between the intranet address and the extranet address is realized. The database routing algorithm is based on the target IP of the resource accessed by the user and then by judging the operator to which the IP belongs in the database.
The link is the most basic selected object for outbound, and is divided into an extranet link and an internal link and an external link, specifically, a routing rule set by a platform, and data coming to the link balancing load device can be forwarded to an extranet operator server through the routing rule.
Design of first, link outbound route selection platform
Automated routing platform architecture
The invention is developed under a linux operating platform, the whole platform is based on a BS framework and is operated through a management platform, the key technology lies in the communication between a user layer and a kernel layer, the user layer sends an instruction to the kernel layer and sends data to the kernel, and the kernel carries out interaction according to the instruction after receiving the data.
1. Routing platform architecture
The link balancing load outbound platform is deployed at the edge of a unit network, rents a plurality of links, and is designed as the whole structure shown in fig. 1.
And (3) link customization: and selecting a link according to the special needs of a user, wherein the link has the highest priority, the judging stage of the link is positioned at a PREROUTING point, the data entering the platform is analyzed, and the outbound link is selected according to the limited range of iptables.
An automated routing module: the automatic routing algorithm structure comprises a database routing algorithm, a polling routing algorithm, a weighted polling routing algorithm, a minimum bandwidth routing algorithm and a fastest response routing algorithm.
A static routing module: the static routing mode is close to the static routing, but the static routing mode is different from the static routing mode in that an IP section can be set and set on the target IP section.
A default routing module: the routing is not default routing, and is used as a branch selection for automatic routing and non-communication, so that the smooth link is ensured under the condition of maximum limit.
Session maintenance: special applications require that links cannot be broken and service interruptions can cause harm and even loss. The multi-time interaction is that the server without interruption returns data to the client after receiving access, some services cannot be completed by one-time interaction, multiple interactions are needed, at the moment, the balance cannot be obtained in time, and the same link needs to be ensured to be used in each interaction, namely, a session maintaining technology. In order to ensure that the same route is taken among multiple sessions, the session keeping technology needs a buffer to record the taken link at any time, and the same link is ensured to be selected by multiple routes.
2. Routing platform mechanism module
The link outbound platform intelligently selects the outbound links for accessing data, and the most important is to select a route and ensure efficiency and accuracy, and in addition, conversation is kept and the same link is continuously dropped. The link setting, the routing scheme and the session maintenance are all necessary modules, the health detection and DNS proxy technology is a hidden module, and the specific mechanism module of the platform is as follows:
(1) link definition, namely links provided by each operator, mainly includes in china: mobile, telecommunications, connectivity, educational, and others, with links to individual or unit devices via a line medium, with the connection of the SSA to each link being via a network interface. Defining a link is to find an exit connecting the link through a route, and the link design is divided into an extranet link definition and an intranet link definition.
(2) The routing algorithm is to select a link from the network, to judge the line where the accessed server resource is located, to judge by a comprehensive IP section database, and then to route.
(3) And customizing routing, namely forcing data to go along a certain line according to the needs of a user, for example, accessing a video resource from telecommunication, and customizing the link trend by setting a forced connection smooth link.
(4) And (4) statically selecting a route, wherein a user sets a certain IP section to be communicated with a telecommunication line and a certain IP section to be communicated with a telecommunication line, and selects a link according to a target IP section.
(5) And (4) default routing, namely, directing the link which fails to be selected in the database to the default link by setting the default route.
(6) Session maintenance, in the case of open session maintenance, the session can be maintained on the same link in the case of polling link dynamic modification.
(7) And the hidden module and the health detection algorithm are used for detecting the connection condition of the link through a third-party tool so as to judge the health condition, judging the situations of automatic routing, static routing and default routing in real time and analyzing the connection condition of the link.
Network connectivity diagnosis is carried out through a network tool, fault detection is carried out on a server and a server port in multiple modes through the application delivery equipment, server switching is carried out in time, and service continuity is guaranteed.
The SSA detects the health condition of the link at regular time when the station is out of the station, and the specific method is that a loop program detects the link at regular time, analyzes the response result and feeds back the response result in real time according to the analysis result, and the design scheme is shown in figure 2.
The specific technical details comprise: adopting a Sleep function in the platform for timeout and time interval, adding a # include < windows.h >, and then directly calling the Sleep function in the main function, wherein parameters comprise Sleep time and unit millisecond; the alarm function can be called, and an alarm module in the SSA platform sends a mail to an administrator; tool: performing ping operation on the link gateway, entering a sleep state if the returned result is yes, and calling an alarm module if the returned result is not yes, wherein other tools, namely Nmap and Tracepath, have the same test logic with the ping command; designing an interface: basic settings are selected, and the required input on the interface comprises a detection method, a time interval and timeout time.
(II) outbound data routing method
The network data comes to the platform for routing and then goes out of the platform, which needs a basic model of data routing, the platform routing flow is shown in fig. 3, and the steps of the data packet after entering the platform are:
firstly, entering a data packet processing module of the platform, marking the data packet by splitting the data packet and then selecting a route. Matching is carried out on the customized routing, and if the customized routing is not matched, the automatic routing algorithm is continuously entered; secondly, entering a static routing algorithm, and moving downwards according to the priority order; third, if the automated routing is unsuccessful, a default route is selected.
The conversation keeping technology ensures that data accessing the same target place walks the same link, and the conversation keeping technology performs cache automatic registration at the position matched with the link.
The specific working process of the improved algorithm is as follows:
firstly, according to a source IP, a target IP, a protocol type and a source IP address of a route in a current routing node in a visiting data stream, collecting and storing parameters in a data structure, and determining whether to carry out session maintenance according to a feedback result;
secondly, if the session is required to be maintained, entering a cache table, judging whether parameters are in the table according to a source IP and a target IP of the visiting data, if no data record exists in the table, indicating that the visiting is the first time, if the route is unobstructed and selectable, adding the data record into the table, otherwise, continuously matching the next route;
third, if session maintenance is not required, routing is performed according to a polling or weighted polling algorithm selected by the user.
The cache table HASH chain table is directly read by the kernel file, and the format of the data record is as follows: the first number is the source IP of the initial visiting data stream, the second visiting data stream is the target IP, the third number is the source IP of the route, http represents the protocol, the last number is the time, and the platform determines whether the data is deleted according to whether the current time and the recorded time exceed a certain difference value.
And modifying the routing module, adding the routing module into the cache table to store the data records, and recompiling the kernel module.
The implementation of the cache function of the invention is that temporary data is stored in a linux kernel by adding a HASH linked list data structure, data records are quickly searched by a HASH algorithm, a timer controls the life cycle of the data records in real time, and the linux kernel is modified by the following steps: under the route.c file under/BUILD/kernel-2.6.32-358. el6/linux-2.6.32-358.el6.x86_64/net/ipv4, the modification module is: adding list matching in a fib _ semantic _ match function, adding a hash list initialization in a fib _ init () function, adding a session _ list _ add () in the fib _ hash _ match function, adding a function of the linked list information in the function, storing the incoming data information which is not in the linked list and needs to be kept, and judging whether corresponding information exists by the session _ list _ search (ip) function according to whether the target ip information is in the cache list or not; session _ list _ del is a Timer which is used for clearing the overtime data record in the linked list at regular time so as to prevent the linked list from being too large, Timer is a Timer executed according to the CPU frequency, timing is absolute, and corresponding action can be executed in the specified time after the Timer is started.
The data flow in the module is shown in fig. 4, and the working process of the kernel-related module is as follows:
the first process is that data stream is obtained in a network layer, a data packet is disassembled, and a source IP, a target IP and an access type parameter in the data stream are obtained;
if the access parameter group is in the cache table, directly returning the corresponding link information, namely, not re-matching the route, or not walking different links, thereby realizing the session maintenance function;
if the access parameter group is not in the cache table and the route is not matched, continuing to search for a proper route, and finally returning the route if the access parameter group is not in the cache table but has a passed route, adding data into the cache table, otherwise, exiting the cycle;
and if no appropriate routing table exists, selecting a default route, clearing the expired data by the Timer in time, and deleting the useless data records within a certain time by the Timer according to the set critical value.
(III) detailed design of each module
1. Program set description
And placing the set link in a database, waiting for the routing algorithm to be called, and flexibly changing the routing algorithm, such as routing from the database to a polling algorithm, wherein rules need to be changed, and database link parameters are not changed.
The link selection is a pivot of the whole automatic routing, and a user sets a specific rule by sending an instruction to a kernel layer after selecting an algorithm, wherein the method comprises the following steps:
(1) database routing method
1) Firstly adding an SSA _ LINK _ LOAD.ko module in a kernel layer, then adding all LINK IP database information into the kernel by calling a user layer interface client, setting LINK conditions in a database by judging LINKs in an application layer, listing all communication situations of the current LINK, carrying out route selection if the LINK is completely communicated, transmitting an obstructed LINK and an unobstructed LINK into the kernel if the LINK is not completely communicated, calling a diagnosis client program, introducing data leading to the obstructed LINK into an adjacent unobstructed LINK, and putting LINKs of different operators into different tables.
2) After the selection of the polling algorithm, if the database routing algorithm is adopted before, the unloading database routing algorithm module: ko, deleting the routing rule in the routing table corresponding to the operator in the LINK; if a weighted polling algorithm is adopted, a table named table25 is found to delete the routing rule therein, and all the unobstructed links are added into a specific routing table for polling.
(2) Default routing method
The default link can be selected more, multiple links exist in a routing table in a polling or weighted polling mode, a policy route represented as 14 is set, a selected link rule is added into the routing table with the corresponding name of 14, the links are taken out from a data table set by the link, a default link user selects a default link object, when the operator to which the link belongs or the accessed resource does not belong to any link operator resource cannot be found out in a database routing algorithm, one link is selected from the default route, and the priority of the default link is the lowest.
(3) Network port aggregation
And storing the basic information of the link, putting the added link information into a database, and carrying out interaction by displaying, editing and deleting.
1) Displaying the basic information of the port aggregation, displaying the port aggregation information acquired from the database in the foreground, and writing a file by adopting Python language as follows:
/usr/local/lib/python2.7/site-packages/SSA/System/Link_Load_bancer/inter_agg_set.py。
2) adding port aggregation basic information, adding the port aggregation information into a database according to parameters input by a user and the selected type, and calling a port aggregation command to generate a port aggregation object.
3) Editing the port aggregation basic information, judging according to the input parameters of the user, and then modifying the data of the database and the port aggregation information under the platform.
4) And removing the basic port aggregation information, and deleting the corresponding data records in the database and the port aggregation information under the platform according to the parameters input by the user.
5) And acquiring an available interface function, listing available network cards for selection by reading network card information under ifconfig, and further judging when selecting for use, wherein the network card which is not marked can only be in one vlan, and the network card which is marked can be in a plurality of vlans.
(2) ACL rule setting
The logic structure of the ACL rule is shown in fig. 5, and mainly includes the following points:
firstly, extracting ACL rule parameters in a database, displaying the ACL rule parameters on a foreground page, and judging whether a data table exists or not and whether reading is abnormal or not;
secondly, adding ACL rules, adding parameters transmitted from a foreground, judging the address formats of a source IP and a target IP, the range of port numbers and the opening of IP fragments, and if the added data is correct, firstly adding the data under the platform and then adding the data into a database to ensure that the data and the rules under the platform are synchronous;
thirdly, editing an ACL rule, firstly judging parameters of the editing rule, then judging the correctness of the rule, adding all rules in the nat chain again when adding into the platform to ensure the sequence, and adding and using a shell script,/etc/nginx/Sys _ iptables/iptables _ m.sh again;
deleting the ACL rules, namely deleting a plurality of rules, firstly deleting the rules under the platform, deleting the rules in the database under the condition of successful deletion, deleting the rules under the platform, simultaneously considering the iptables rules of other functions, and modifying the iptables rules of the platform by using the same adding interface script;
and fifthly, calling the iptables rules to generate a unified interface, and writing all the iptables rules into/etc/nginx/sys _ iptables/iptables _ m.sh.
Second, realization of link outbound routing platform
(one) Link setup
The link setting in the outbound platform is divided into extranet link setting and intranet link setting, the extranet link setting sets different routing tables for different operators, and sets a mobile link mark as 1, a communication link mark as 3, a telecommunication link mark as 5 and an education link mark as 7.
(II) Link customization
The highest priority is policy routing, which directly controls the network layer data according to the Iptables, for example, to organize the rules for accessing the resources of a certain target IP: the IPtables-AINPUT-ptcp-ieth 0-d207.46.18.94-dport 80-JDROP limits the access target IP to 80 ports of 207.46.18.94, controls the access outbound flow through the IPtables at the network layer, tears down the data packet for detection before routing, and the priority of strategy routing is higher than that of a routing algorithm.
Example (b): the method comprises the steps of simulating an external network link, walking a telecommunication line when accessing 80 ports of a website, walking a communication line when accessing 50 ports, and in order to simulate the network access environment more vividly, setting a link balancing loader, a client, a router and 2 web servers, realizing link balancing load by configuring an iptables firewall rule and a policy routing rule, wherein the iptables is used for marking the access from the ports, for example, marking '1' on a data packet accessing 80 ports, marking '2' on a data packet accessing 81, a routing policy directs the data packet to different routing tables according to different marks of the data packet, and the line type of an internet service provider is selected according to a gateway in the routing tables. Simulating an outbound in an intranet, wherein four machines are matched with and simulate a simple model, namely a client, a link balancing loader, a route and a website server, which is a model for simulating the whole link load outbound in the intranet to carry out route configuration, and the setting environment and configuration steps are as follows:
the method comprises the following steps: and (3) client IP: 10.7.12.166, balanced loaders eth0:10.7.13.7, eth2:192.168.5.6, eth3:192.168.11.33, route: eth2:192.168.5.8, eth3:192.168.11.44, eth1:192.168.55.26 website server eth1: 192.168.55.88;
step two: in order to access the website by crossing the ip section, a router is added with a default route, route add default gw 192.168.5.6;
step three: adding a route, route add default gw 192.168.55.26 on the website server;
step four: the SSA and the router both start the forwarding mode of the network card;
step five: at a client access 192.168.55.88:80 port, adopting tcpdump packet capture to check the data condition of eth2 in the link load balancer, if no data packet exists, indicating that the link is wrong and needing detection;
step six: and at the client access 192.168.55.88:53 port, adopting tcpdump packet capture to check the packet condition of the network card eth3 in the router, and if no packet exists, indicating that the link is wrong and needing detection.
Step seven: and replacing the client and the website server, and performing multiple tests.
In the simulation of an intranet, a connected network structure is constructed to achieve the connected state in a real link, devices in different network segments in the intranet need to be configured with default routes for mutual communication, a packet capturing tool tcpdump displays captured data packets to prove that specific lines are visited, and the rule is as follows:
routing rules are as follows: adding a table101 and a table102 under a file/etc/iproute 2/rt _ tables, configuring iptables rules and routing rules on link balancing load, adopting virtual machine simulation by a router, wherein Eth2, Eth3 and Eth0 are different network segments, Eth0 of a Web server and Eth0 of the router are the same network segment, Eth2 of the balancing load and Eth2 of the router belong to the same IP segment, and Eth3 of the balancing load and Eth3 of the router belong to the same IP segment. When the 80 port is accessed, the network card has data packets, the request is judged to come from the client side by analyzing the content of the data packets, and the path from eth2 for balancing load to eth2 of the router is shown to be taken at the moment.
(III) network routing method
The invention discloses a network routing method, which is simple in the prior art, has the inherent defects that access can only be realized in a basic polling mode, other algorithms cannot be realized, and the defects of large load, slow path searching, low efficiency and more errors can be caused under the condition of more rules.
Routing part codes in Linux source codes forward data packets of each layer of a network, the routing part codes are positioned under a D \ Linux-2.6.11.12\ net \ ipv4 path, routing in a kernel is an important part, an IP _ rt _ init () function is called in an entry function IP _ init (), initialization operation of a routing function table is completed, routing codes are initialized, the platform is called by an IP _ init interface of an initialized IP sub-platform when being started, initialization of strategy routing is realized by a fib _ rules _ init function and initialized to a netdev _ chain notification chain registration processing hook, and the registration processing hook is fib _ rules _ event.
(IV) implementation of automatic routing
1. Database routing method
Truthful database: first, all IP addresses are stored according to a shaping, and assuming that one IP address is a.b.c.d, the calculation formula is: ip ═ a × 256+ B) × 256+ C, which is calculated only to segments: a.b.c.0 to a.b.c.255.
The method comprises the following steps: the user writes the program operating database number using the SQL language as follows: SELECT FROM ' ip _ groups _ for _ city ' WHERE ' ip _ start ═ 4891709ORDERByip _ startDESCIMIT 1; the results are shown below: ip _ start _ count _ code | region _ code | city | zipcode | site | longitude48891709| US | CA | Mount V iew |94043|37.4192| -122.057.
The second method comprises the following steps: using the database via online mode, the web site as shown below is used: http:// blogama. org/ip _ query. phpip. 74.125.45.100& output. XML thus, XML data is obtained.
The IP database is complete, but the accuracy rate of the IP database is only about 60%, so that the IP database is not in line with the requirements, and in addition, the operator cannot be directly judged according to the IP by the genuine version database.
And the data routing algorithm is used for putting all IP sections into a database, then storing the IP sections into a binary tree, quickly searching the tree according to the target IP extracted from the data packet, judging the IP section to which the target IP belongs, and then further judging the line to which the target IP belongs. The method of matching in a binary tree is an advantage and a feature of the present invention, and results can be obtained by comparing several times without affecting efficiency.
Adding a module in a linux kernel, hooking the module at a PREROUTING position, intercepting a data stream and unpacking an analysis target IP, finding a corresponding link according to a matching result, realizing communication between the module and an application layer through a fastsocket technology, adding data to the kernel layer by a user layer, and realizing data interaction from the user layer to the kernel layer.
And the binary tree data structure is used for storing all IP sections, disassembling the data packet to obtain a target IP, obtaining the target IP of the data packet in the developed module, and obtaining the serial number of the corresponding link section in the created binary tree. And the database routing algorithm is characterized in that the core routing module judges the line to which the accessed resource belongs in the database according to the access target IP by loading the IP database, returns the query result and selects a route according to the query result.
2. Polling method
The outbound algorithm and inbound algorithm are similar in principle, except that the inbound detection is the load case of the server and the outbound detection is the load case of the link. The polling algorithm is used for selecting different links in turn when data is selected, which is also the reason for needing the support of the session maintenance technology, and the polling and weighted polling algorithms are realized by configuring the parameter weight of the routing module.
Polling: the polling algorithm assigns each new connection request to the next server or gateway in sequence, eventually splitting all requests equally among all servers. The round robin algorithm is such that data access will be routed equally when selecting links, but it works better if the load balancing devices are not perfectly equal in processing speed, connection speed and memory.
Weighted polling: the number of the connections accepted by each machine is distributed according to the weight proportion, the method is an improvement on a common polling algorithm, the polling algorithm and the weighted polling algorithm are similar, and a conclusion can be read through experiments.
To verify the availability of the polling algorithm and the weighted polling algorithm, the outbound platform additionally creates two routing tables: and T1 and T2 are added into/etc/iproute 2/rt _ tables, then a default route is set, and then a routing rule is set, so that all response data are ensured to return from the original route of the corresponding network card. And (3) simulating experiments of a polling algorithm and a weighted polling algorithm, balancing the routing of two Internet service providers, and appointing one Internet service provider to have higher priority than the other Internet service provider by adjusting a weight parameter.
If the two weights are equal, the algorithm is a polling algorithm, if the weights are defined by the user, the algorithm belongs to a weighted polling algorithm, and if the algorithm is a plurality of internet service providers, the setting method is the same.
(V) static routing
The user sets the selected link according to the accessed resource parameters, customizes the link by combining the forwarding control or marking control of the iptables and the routing rule, or sets the static route according to the network segment, and the iptables is limited, otherwise the priority is reduced to the third. The link balancing load solves the routing problem by configuring a firewall and a routing rule.
The intranet client is connected to the Internet through a Linux host, two lines are arranged when the Linux host is connected with the Internet, static route selection is carried out on the intranet, data accessing 192.168.1.x sections are all sent out through a telecommunication line, and all accesses accessing 192.168.2.x sections are sent out through a communication line.
Before routing the data packet, a mark is made for the data packet according to the protocol and the target port of the data packet, then a corresponding rule is assigned, strategy routing is carried out according to the mark of the data packet, a flag is made for a specific data packet to use a mangle table, and a PREROUTING chain is used for making a flag before routing.
The command adds a rule in a PREROUTING chain of a mangle table, a data packet from an eth0 interface is marked, matching rules of the command are respectively a TCP protocol, a target port number is 80 and a UDP protocol, a target port number is 53, the marked values are respectively 1 and 2, after the data packet passes through the PREROUTING chain, the data packet enters a routing module, and policy routing is carried out on the data packet to execute the following two commands, and a corresponding rule is added:
(1)ip rule add from all fwmark 1table 10;
(2)ip rule add from all fwmark 2table 20。
the two commands indicate that all packets with the flag of 1 are routed by using the routing table10, while all packets with the flag of 2 are routed by using the routing table20, the routing tables 10 and 20 respectively use gateways on ChinaNet and Cernet lines as default gateways, and the specific setting commands are as follows:
(1)ip route add default via 10.10.1.1dev eth1 table 10;
(2)ip route add default via 10.10.2.1dev eth2 table 20。
the two commands specify 10.10.1.1 and 10.10.2.1 on routing tables 10 and 20, respectively, as default gateways, which are located on the ChinaNet and Cernet lines, respectively, with packets using routing table10 going out through the ChinaNet line and packets using routing table20 going out through the Cernet line.
(VI) Session Hold
The global session is maintained by adding a cache mechanism table in the kernel routing module, the principle is as shown in fig. 6, registering source IP, target IP, time and protocol type parameters, advancing table judgment after data needing session maintenance comes next time, if the data does not exist in the cache table, adding the data into the table, if the data exists, walking the line, and modifying the time in the table, setting a regular detection table of a timer according to the registration time of the table, if the difference between the data time of the table and the current time is greater than a certain value, it is indicated that the time is not newly connected, that is, the data is deleted if the time is overtime.
The method comprises the following steps:
step one, according to a source IP, a target IP, a protocol type and a source IP address of a route in a current routing node in a visiting data stream, learning a neural network, and determining whether to carry out session maintenance or not according to a feedback result;
step two, if the session is required to be maintained, entering a cache table, judging whether parameters are in the table according to a source IP and a target IP of the visiting data, if no data record exists in the table, indicating that the visiting is the first time, if the route is smooth and optional, adding the data record into the table, otherwise, continuously matching the next route;
and thirdly, if the session is not required to be kept, routing is carried out according to a polling or weighted polling algorithm selected by the user.
The cache table is directly read by the kernel file, and the format of the data record is as follows: the first number is the source IP of the initial visiting data flow, the second is the target IP of the visiting data flow, the third number is the source IP of the route, the last number is the time, and the platform determines whether the data is deleted or not according to whether the current time and the recorded time exceed a certain difference value or not.
After the kernel module is modified, the kernel is required to be recompiled, the kernel is recompiled and loaded through the modified kernel code, the kernel is compiled for many times in this period, if the source code is not modified properly, the kernel is compiled with errors, and the kernel compiling step is as follows:
step 1: $ make mrprep;
step 2: various functions of the kernel module are configured, and interface configuration modes are various, including:
the configuration mode of the make config// text needs the user to write the configuration file by himself,
the configuration mode of the make menuconfig// text menu can be used only under a terminal with characters;
(3) # make dep// for one to two minutes;
(4) # make clean// clear some unnecessary files;
(5) # make bzImage// compilation image File
If the kernel is compiled, bzImage is put in/usr/src/linux-2.6.20/arch/i 386/boot/directory, otherwise the platform cannot be started;
(6) # make modules, the key element selected as M in make menuconfig, all compiled into modules and placed in/usr/src/linux-2.4.25;
(7) make initrd. img, # make modules _ install, mkinitrd;
(8)#make install
and (3) environment configuration: four virtual machines are set to form a test environment, the technical progress of the modified routing kernel is verified by simulating an intranet to access an extranet, and an experimental simulation flow is shown in fig. 7.
SSA addition rules:
IP rule add from all fwmark3 table10// policy routing
Ip route add default via 192.168.44.12 dev eth2 table10// link
Ip rule add from all fwmark 5 table 20
Ip route add default via 192.168.55.99 dev eth 1table 20
An experimental process, a network environment is configured, a default route pointing to an SSA is added on a router, after the whole virtual network is communicated, a client accesses a web server, if a 50 port of a website is accessed, routing telecommunication is agreed, if an 80 port of the website is accessed, a communicated line is agreed, a routing table10 is simulation of telecommunication, a table20 is simulation of communication, routing telecommunication is agreed when 192.168.33.25:50 is accessed, an eth1 of the router is subjected to tcp packet grabbing, an eth0 of the router is empty, and when an 192.168.33.25:80 port is accessed, an eth0 is subjected to tcp packet grabbing through tcpdump. The experiment is developed on the basis of customizing two route options, different routes are taken by simulating and accessing different ports, and the route selection method is changed by compiling the route source code. If the route is selected according to the return value by quickly searching the return line of the database, experiments just prove that the modification idea is feasible and effective.
(VII) platform communication
The platform communication adopts a netlink technology, a user is directly connected with a kernel, a user layer interacts with a kernel layer, data of an application layer is transmitted to the kernel layer, the kernel is used by other modules through the transmitted data, the setting of the user is placed in a database or a file at the application layer, and when a machine is restarted, the platform defaults and loads the setting of the user, so that the setting is prevented from being lost. And developing a user interface and an interface of the kernel layer, transmitting data from the user layer to the kernel layer, and changing the logic of the kernel after the kernel responds.
By designing a simulated network environment, different ports are accessed to replace accessing different ips, different routes are added to the platform, and different routes are taken by data by accessing different ports. The invention changes the kernel source code through the hook function, develops the kernel module, selects the route or the line according to the result of inquiring and returning in the IP database, the thought is feasible, the method for selecting the route in the kernel is different from the traditional method for marking and selecting the route rule by iptables, the problem of long inquiring time under the condition of a large number of iptables rules is avoided, the finding of the iptables rules is sequential finding, and the time complexity is maximum. In addition, the efficiency of searching the line in the kernel is required to be higher than the matching efficiency of the iptables rule, the iptables rule is sequentially matched and has lower speed, the IP section database is imported into the kernel by modifying the kernel module and modifying the routing method, the algorithm is added in the IP database search, the searching speed is obviously higher than that of the iptables rule, the smooth of the outbound line is ensured by compiling the database searching function into the kernel and updating and maintaining the database, and the key technology of the invention is also disclosed.

Claims (10)

1. The multi-operator network link load outbound automatic routing platform is characterized in that a linux kernel is reformed under the linux platform, link data accessed in an outbound mode are accessed according to intelligent classification of resource types, the system architecture of the platform is not changed, the link data are mounted under the platform in a hook function mode, and the network link load outbound automatic routing result is given in a bypass mode through intercepting data flow and analyzing, and the method comprises the following steps: setting a network link, selecting an outbound link, and selecting a routing algorithm of various links, developing a platform kernel module, interacting a user layer and a kernel layer, and keeping a session;
the outbound data routing method comprises the following steps after a data packet enters a platform: firstly, entering a data packet processing module of a platform, marking the data packet by detaching the data packet, then selecting a route, matching the customized route, and if not, continuing to enter an automatic route selection algorithm; secondly, entering a static routing algorithm and proceeding downwards all the time according to the sequence of the priority; thirdly, if the automatic routing is unsuccessful, selecting default routing;
the conversation maintaining technology ensures that data accessing the same target place walks the same link, the conversation maintaining technology performs cache automatic registration at the position matched with the link, and the specific working process of the improved algorithm is as follows:
firstly, according to a source IP, a target IP, a protocol type and a source IP address of a route in a current routing node in a visiting data stream, collecting and storing parameters in a data structure, and determining whether to carry out session maintenance according to a feedback result;
secondly, if the session is required to be maintained, entering a cache table, judging whether parameters are in the table according to a source IP and a target IP of the visiting data, if no data record exists in the table, indicating that the visiting is the first time, if the route is unobstructed and selectable, adding the data record into the table, otherwise, continuously matching the next route;
thirdly, if the session is not required to be maintained, routing is carried out according to a polling or weighted polling algorithm selected by the user;
the cache table HASH chain table is directly read by the kernel file, and the format of the data record is as follows: the first number is the source IP of the initial visiting data stream, the target IP of the second visiting data stream, the third number is the source IP of the route, http represents the protocol, the last number is the time, and the platform determines whether the data is deleted according to whether the current time and the recorded time exceed a certain difference value; modifying the routing module, adding the cache table to store data records, and recompiling the kernel module;
the realization of the cache function is realized in a linux inner core, temporary data is stored by adding a HASH chain table data structure, data records are quickly searched through a HASH algorithm, the life cycle of the data records is mastered by a timer in real time, and the position of the linux inner core is reformed: under the route.c file under/BUILD/kernel-2.6.32-358. el6/linux-2.6.32-358.el6.x86_64/net/ipv4, the modification module is: adding list matching in a fib _ semantic _ match function, adding a hash list initialization in a fib _ init () function, adding a session _ list _ add () in the fib _ hash _ match function, adding a function of the linked list information in the function, storing the incoming data information which is not in the linked list and needs to be kept, and judging whether corresponding information exists by the session _ list _ search (ip) function according to whether the target ip information is in the cache list or not; session _ list _ del is a Timer which is used for regularly clearing the overtime data record in the linked list, Timer is a Timer executed according to the CPU frequency, timing is absolute, and after the Timer is started, the corresponding action can be executed in the specified time;
the working process of the kernel related module is as follows:
the first process is that data stream is obtained in a network layer, a data packet is disassembled, and a source IP, a target IP and an access type parameter in the data stream are obtained;
if the access parameter group is in the cache table, directly returning the corresponding link information, namely, not re-matching the route, or not walking different links, thereby realizing the session maintenance function;
if the access parameter group is not in the cache table and the route is not matched, continuing to search for a proper route, and finally returning the route if the access parameter group is not in the cache table but has a passed route, adding data into the cache table, otherwise, exiting the cycle;
and if no appropriate routing table exists, selecting a default route, clearing the expired data by the Timer in time, and deleting the useless data records within a certain time by the Timer according to the set critical value.
2. The multi-operator network link load outbound automatic routing platform of claim 1, wherein the automatic routing platform is developed under a linux operating platform, the whole platform is based on a BS architecture and is operated through a management platform, the key technology lies in communication between a user layer and a kernel layer, the user layer sends an instruction to the kernel layer and sends data to the kernel, the kernel performs interaction according to the instruction after receiving the data, and communication between the kernel and a user is realized through a fast socket technology;
the link balance load outbound platform is deployed at the edge of a unit network and rents a plurality of links, and the whole structure comprises:
and (3) link customization: selecting a link according to special needs of a user, wherein the link has the highest priority, the judging stage of the link is positioned at a PREROUTING point, the data entering a platform is analyzed, and an outbound link is selected according to the limited range of iptables;
an automated routing module: the automatic routing algorithm structure comprises a database routing algorithm, a polling routing algorithm, a weighted polling routing algorithm, a minimum bandwidth routing algorithm and a fastest response routing algorithm;
a static routing module: the static routing mode is close to the static route, but different from the static route, the static routing can set an IP section and set the IP section on the target IP section;
a default routing module: the routing is not default routing and is used as a branch selection for automatic routing and non-communication, so that the smooth link is ensured under the condition of maximum limit;
session maintenance: the same route is guaranteed to be taken among multiple sessions, a cache is needed to record the taken link at all times, and multiple routing is guaranteed to select the same link.
3. The multi-operator network link load outbound automatic routing platform of claim 1, wherein the link outbound platform intelligently selects access to data outbound links, link setup, routing schemes, session maintenance are all necessary modules, health detection and DNS proxy technologies are all hidden modules, and the specific mechanism module of the platform is:
link definition, namely links provided by each operator, the links are accessed to personal or unit equipment through line media, the connection between the SSA and each link is through a network interface, the defined links find out an outlet connected with the link through a route, and the link design is divided into an extranet link definition and an intranet link definition;
a routing algorithm, wherein routing is to select a link from the network, judge the line where the accessed server resource is located, need a comprehensive IP section database for judgment, and then perform routing;
customizing routing, and forcing data to go through a certain line according to the needs of a user;
static routing, wherein a user sets a certain IP section to go through one network operator and a certain IP section to go through another network operator, and performs link selection according to a target IP section;
default routing, namely, directing the link which fails to be selected in the database to the default link by setting the default route;
the conversation is kept, and under the condition of starting the conversation keeping, the conversation can be kept on the same link under the condition of dynamically modifying the polling link;
and the hidden module and the health detection algorithm are used for detecting the connection condition of the link through a third-party tool so as to judge the health condition, judging the situations of automatic routing, static routing and default routing in real time and analyzing the connection condition of the link.
4. The multi-operator network link load outbound automatic routing platform of claim 3, wherein network connectivity diagnosis is performed through a network tool, server switching is performed in time, and service continuity is guaranteed;
the SSA detects the health condition of the link at regular time when the station is out of the station, and the specific method is that a loop program detects the link at regular time, analyzes the response result and feeds back the result in real time according to the analysis result;
the specific technical details comprise: adopting a Sleep function in the platform for timeout and time interval, adding a # include < windows.h >, and then directly calling the Sleep function in the main function, wherein parameters comprise Sleep time and unit millisecond; the alarm function can be called, and an alarm module in the SSA platform sends a mail to an administrator; tool: performing ping operation on the link gateway, entering a sleep state if the returned result is yes, and calling an alarm module if the returned result is not yes, wherein other tools, namely Nmap and Tracepath, have the same test logic with the ping command; designing an interface: basic settings are selected, and the required input on the interface comprises a detection method, a time interval and timeout time.
5. The multi-operator network link load outbound automated routing platform of claim 1, wherein the database routing method:
1) firstly, adding an SSA _ LINK _ LOAD.ko module in a kernel layer, then adding all LINK IP database information into the kernel by calling a user layer interface client, setting LINK conditions in a database by judging LINKs in an application layer, listing all communication situations of the current LINK, carrying out route selection if the LINK is completely communicated, transmitting an obstructed LINK and an unobstructed LINK into the kernel if the LINK is not completely communicated, calling a diagnosis client program, introducing data leading to the obstructed LINK into an adjacent unobstructed LINK, and putting LINKs of different operators into different tables;
2) after the selection of the polling algorithm, if the database routing algorithm is adopted before, the unloading database routing algorithm module: ko, deleting the routing rule in the routing table corresponding to the operator in the LINK; if a weighted polling algorithm is adopted, a table named table25 is found to delete the routing rule therein, and all the unobstructed links are added into a specific routing table for polling.
6. The multi-operator network link load outbound automated routing platform of claim 1, wherein network port aggregation: and (3) storing basic information of the link, putting the added link information into a database, and carrying out interaction by displaying, editing and deleting:
1) displaying the basic information of the port aggregation, displaying the port aggregation information acquired from the database in the foreground, and writing a file by adopting Python language as follows:
/usr/local/lib/python2.7/site-packages/SSA/System/Link_Load_bancer/inter_agg_set.py;
2) adding port aggregation basic information, adding the port aggregation information into a database according to parameters input by a user and the selected type, and calling a port aggregation command to generate a port aggregation object;
3) editing the port aggregation basic information, judging according to the input parameters of the user, and then modifying the data of the database and the port aggregation information under the platform;
4) removing the basic port aggregation information, and deleting the corresponding data records in the database and the port aggregation information under the platform according to the parameters input by the user;
5) and acquiring an available interface function, listing available network cards for selection by reading network card information under ifconfig, and further judging when selecting for use, wherein the network card which is not marked can only be in one vlan, and the network card which is marked can be in a plurality of vlans.
7. The multi-operator network link load outbound automated routing platform of claim 1, wherein the logical structure of the ACL rules is mainly:
firstly, extracting ACL rule parameters in a database, displaying the ACL rule parameters on a foreground page, and judging whether a data table exists or not and whether reading is abnormal or not;
secondly, adding ACL rules, adding parameters transmitted from a foreground, judging the address formats of a source IP and a target IP, the range of port numbers and the opening of IP fragments, and if the added data is correct, firstly adding the data under the platform and then adding the data into a database to ensure that the data and the rules under the platform are synchronous;
thirdly, editing an ACL rule, firstly judging parameters of the editing rule, then judging the correctness of the rule, adding all rules in the nat chain again when adding into the platform to ensure the sequence, and adding and using a shell script,/etc/nginx/Sys _ iptables/iptables _ m.sh again;
deleting the ACL rules, namely deleting a plurality of rules, firstly deleting the rules under the platform, deleting the rules in the database under the condition of successful deletion, deleting the rules under the platform, simultaneously considering the iptables rules of other functions, and modifying the iptables rules of the platform by using the same adding interface script;
and fifthly, calling the iptables rules to generate a unified interface, and writing all the iptables rules into/etc/nginx/sys _ iptables/iptables _ m.sh.
8. The multi-operator network link load outbound automated routing platform of claim 1, wherein the network routing method: routing part codes in Linux source codes forward data packets of each layer of a network, the routing part codes are positioned under a D \ Linux-2.6.11.12\ net \ ipv4 path, routing in a kernel is an important part, an IP _ rt _ init () function is called in an entry function IP _ init (), initialization operation of a routing function table is completed, routing codes are initialized, the platform is called by an IP _ init interface of an initialized IP sub-platform when being started, initialization of strategy routing is realized by a fib _ rules _ init function and initialized to a netdev _ chain notification chain registration processing hook, and the registration processing hook is fib _ rules _ event.
9. The multi-operator network link load outbound automated routing platform of claim 1, wherein static routing: the user sets the selected link according to the accessed resource parameters, customizes the link by combining the routing rule through the forwarding control or marking control of the iptables, or sets the static route according to the network segment, the iptables is limited, otherwise the priority is reduced to the third, and the link balancing load solves the routing problem by configuring the firewall and the routing rule;
the intranet client is connected to the Internet through a Linux host, and when the Linux host is connected with the Internet, two lines are arranged to perform static routing on the intranet;
before routing a data packet, marking a mark for the data packet according to a protocol and a target port of the data packet, then appointing a corresponding rule, performing strategy routing according to the mark of the data packet, marking a specific data packet by using a mangle table, and marking a PREROUTING chain before routing;
the command adds a rule in a PREROUTING chain of a mangle table, a data packet from an eth0 interface is marked, matching rules of the command are respectively a TCP protocol, a target port number is 80 and a UDP protocol, a target port number is 53, the marked values are respectively 1 and 2, after the data packet passes through the PREROUTING chain, the data packet enters a routing module, and policy routing is carried out on the data packet to execute the following two commands, and a corresponding rule is added:
1)ip rule add from all fwmark 1table 10;
2)ip rule add from all fwmark 2table 20;
the two commands indicate that all packets with the flag of 1 are routed by using the routing table10, while all packets with the flag of 2 are routed by using the routing table20, the routing tables 10 and 20 respectively use gateways on ChinaNet and Cernet lines as default gateways, and the specific setting commands are as follows:
1)ip route add default via 10.10.1.1dev eth1 table 10;
2)ip route add default via 10.10.2.1dev eth2 table 20;
the two commands specify 10.10.1.1 and 10.10.2.1 on routing tables 10 and 20, respectively, as default gateways, which are located on the ChinaNet and Cernet lines, respectively, with packets using routing table10 going out through the ChinaNet line and packets using routing table20 going out through the Cernet line.
10. The multi-operator network link load outbound automated routing platform of claim 1, wherein the global session maintenance is performed by the kernel routing module by adding a cache mechanism table to register source IP, target IP, time, protocol type parameters, proceeding table judgment after data requiring session maintenance comes next time, if no such data exists in the cache table, adding the data to the table, if the data exists, proceeding the line, and modifying the time in the table, according to the registration time of the table, setting a detection table with a timer period, if the difference between the data time of the table and the current time is greater than a certain value, it indicates that there is no new connection in the period, that is, if the time is out, the data is deleted.
CN202011641784.8A 2020-12-31 2020-12-31 Automatic routing platform for multi-operator network link load outbound Withdrawn CN112751762A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011641784.8A CN112751762A (en) 2020-12-31 2020-12-31 Automatic routing platform for multi-operator network link load outbound

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011641784.8A CN112751762A (en) 2020-12-31 2020-12-31 Automatic routing platform for multi-operator network link load outbound

Publications (1)

Publication Number Publication Date
CN112751762A true CN112751762A (en) 2021-05-04

Family

ID=75649424

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011641784.8A Withdrawn CN112751762A (en) 2020-12-31 2020-12-31 Automatic routing platform for multi-operator network link load outbound

Country Status (1)

Country Link
CN (1) CN112751762A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114598625A (en) * 2022-03-30 2022-06-07 缀初网络技术(上海)有限公司 Method and storage medium for adaptively adjusting lines in unstable multi-line scene

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030174648A1 (en) * 2001-10-17 2003-09-18 Mea Wang Content delivery network by-pass system
US20120163180A1 (en) * 2010-12-28 2012-06-28 Deepak Goel Systems and Methods for Policy Based Routing for Multiple Hops
CN103036788A (en) * 2012-12-14 2013-04-10 上海斐讯数据通信技术有限公司 Implementation method of multi-interface gateway equipment data transmitting
CN104270325A (en) * 2014-09-05 2015-01-07 烽火通信科技股份有限公司 System and method of implementing limitation of public network access user number based on Linux for CPE (Customer Premise Equipment)
CN104994029A (en) * 2015-06-23 2015-10-21 浪潮电子信息产业股份有限公司 Link load balance outbound session persistence method based on intelligent learning algorithm

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030174648A1 (en) * 2001-10-17 2003-09-18 Mea Wang Content delivery network by-pass system
US20120163180A1 (en) * 2010-12-28 2012-06-28 Deepak Goel Systems and Methods for Policy Based Routing for Multiple Hops
CN103036788A (en) * 2012-12-14 2013-04-10 上海斐讯数据通信技术有限公司 Implementation method of multi-interface gateway equipment data transmitting
CN104270325A (en) * 2014-09-05 2015-01-07 烽火通信科技股份有限公司 System and method of implementing limitation of public network access user number based on Linux for CPE (Customer Premise Equipment)
CN104994029A (en) * 2015-06-23 2015-10-21 浪潮电子信息产业股份有限公司 Link load balance outbound session persistence method based on intelligent learning algorithm

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114598625A (en) * 2022-03-30 2022-06-07 缀初网络技术(上海)有限公司 Method and storage medium for adaptively adjusting lines in unstable multi-line scene
CN114598625B (en) * 2022-03-30 2024-06-14 派欧云计算(上海)有限公司 Method for adaptively adjusting lines in unstable multi-line scene and storage medium

Similar Documents

Publication Publication Date Title
AbdelSalam et al. Implementation of virtual network function chaining through segment routing in a linux-based NFV infrastructure
US11570090B2 (en) Flow tracing operation in container cluster
CN103812704B (en) A kind of public network IP dynamic management approach of Virtual machine
US8458319B2 (en) System and method for tracking network resources
CN109952746A (en) Physics and virtual network function are integrated in business chain network environment
US7769873B1 (en) Dynamically inserting filters into forwarding paths of a network device
CN106911778A (en) A kind of flow bootstrap technique and system
US20220052923A1 (en) Data processing method and device, storage medium and electronic device
US10693753B2 (en) Network device snapshots
CN108600260A (en) A kind of industry Internet of Things security configuration check method
US10574680B2 (en) Malware detection in distributed computer systems
CN112202940A (en) Pod service mode for external exposure of kubernets
CN112751762A (en) Automatic routing platform for multi-operator network link load outbound
US10560284B2 (en) System and methods for mapping a network service path
Kumar et al. Scaling telecom core network functions in public cloud infrastructure
US7747849B2 (en) Secure communications equipment for processing data packets according to the send mechanism
CN113630301A (en) Data transmission method, device and equipment based on intelligent decision and storage medium
CN110191067A (en) Private line network access control method, device, equipment and readable storage medium storing program for executing
CN106161051B (en) Custom end intelligent route selection
CN114726785A (en) Directional routing debugging mechanism based on request label expression
JP2018026734A (en) Fault part specification program, device, and method
CN114221808B (en) Security policy deployment method and device, computer equipment and readable storage medium
Mente MASTERARBEIT/MASTER’S THESIS
CN111107178B (en) Method and equipment for designating local address used by message
CN116319260B (en) Network fault diagnosis method, device, equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
WW01 Invention patent application withdrawn after publication

Application publication date: 20210504

WW01 Invention patent application withdrawn after publication