CN113890789A - UDP tunnel traffic shunting method and traffic forwarding method suitable for data center - Google Patents
UDP tunnel traffic shunting method and traffic forwarding method suitable for data center Download PDFInfo
- Publication number
- CN113890789A CN113890789A CN202111153796.0A CN202111153796A CN113890789A CN 113890789 A CN113890789 A CN 113890789A CN 202111153796 A CN202111153796 A CN 202111153796A CN 113890789 A CN113890789 A CN 113890789A
- Authority
- CN
- China
- Prior art keywords
- udp tunnel
- packet
- traffic
- udp
- virtual router
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L12/00—Data switching networks
- H04L12/28—Data switching networks characterised by path configuration, e.g. LAN [Local Area Networks] or WAN [Wide Area Networks]
- H04L12/46—Interconnection of networks
- H04L12/4633—Interconnection of networks using encapsulation techniques, e.g. tunneling
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L12/00—Data switching networks
- H04L12/28—Data switching networks characterised by path configuration, e.g. LAN [Local Area Networks] or WAN [Wide Area Networks]
- H04L12/46—Interconnection of networks
- H04L12/4641—Virtual LANs, VLANs, e.g. virtual private networks [VPN]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L45/00—Routing or path finding of packets in data switching networks
- H04L45/58—Association of routers
- H04L45/586—Association of routers of virtual routers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/16—Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
- H04L69/164—Adaptation or special uses of UDP protocol
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/22—Parsing or analysis of headers
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Environmental & Geological Engineering (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The application provides a UDP tunnel traffic shunting method suitable for a data center, wherein the data center comprises a plurality of virtual routers, and the method comprises the following steps: setting an SRO module at a node of at least one virtual router, and setting a UDP tunnel port for the SRO module; sending the UDP tunnel flow from the node of the first virtual router to the node of the second virtual router by using an MPLSoverUDP encapsulation mode, and setting monitoring at a UDP tunnel port; judging whether the UDP tunnel flow packet is a local data packet or not; responding to the UDP tunnel flow packet as a local data packet, executing software packet merging operation, and sending the UDP tunnel flow packet after executing the GRO operation to the second virtual router after decapsulating. According to the embodiment of the application, the analysis and the shunt of the cross-node north-south flow can be realized, the GRO processing is skipped, the process of segmenting after aggregation on a path is reduced, the network forwarding efficiency is improved, and the CPU resource is saved.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a UDP tunnel traffic offload method suitable for a data center and a traffic forwarding method suitable for a data center.
Background
With the development of web sites and electronic commerce, networks have become one of the life necessities that people cannot leave at present. Since the first time, cloud computing was proposed in a search engine meeting of 8 months 2006, the cloud computing became a third revolution of the internet, and belongs to one type of distributed computing, a huge data computing processing program is decomposed into countless small programs through network cloud, and then the small programs are processed and analyzed through a system consisting of a plurality of servers to obtain results and are returned to users. In recent years, cloud computing is becoming a strategic focus of development of information technology industries, and information technology enterprises around the world are transformed into cloud computing in disputes. In recent years, with the development of network, virtualization and storage technologies, we have advanced into the cloud computing era, and various large cloud computing manufacturers or large IT enterprises establish their own data centers to support and develop their own services.
If high-speed forwarding and shunting of flow data cannot be realized in a data center, immeasurable loss is brought to users. The existing gro (generic receiving offloading) technology is implemented as lro (large receiving offloading) software, and a plurality of received VXLAN/TCP data packets are aggregated into one large data packet on a network card driver layer, and then uploaded to a protocol stack for processing. Therefore, the overhead of protocol stack processing can be reduced, and the capability and efficiency of a system for receiving TCP data are improved.
However, existing GRO technologies only support UDP tunnels for the mainstream. For example, if a vxlan is used to encapsulate a data packet, then a receive flow is supported; GRO will not be possible using the mpls over udp tunnel to encapsulate the packet. And the traffic of the cross-node access router out of the external network does not need GRO originally, but cannot be intelligently distinguished, so that the traffic still needs to be subjected to packet subpackaging again after GRO and before packet sending, and CPU resources are additionally occupied.
As shown in fig. 1, which is a structural schematic diagram of an existing virtual router, a cross-node uses an Mpls over Udp tunnel, and when vm1 accesses vm2 across nodes, a network card driver plane cannot support GRO, so that a data packet is sent to a kernel protocol stack with a length of a data segment smaller than mtu and then is forwarded through decapsulation (decap). Taking mtu length 1500 as an example, the maximum message length is 1514 bytes, if the GRO of the mpls oudp data packet can be supported, the data packet can be waited and made into a large packet with 64K length and then sent to the protocol stack, and then 42 times of decapsulation processing can be executed, thereby achieving the purposes of saving cpu resources and improving the forwarding performance in turn.
As shown in fig. 2, another structure of the existing virtual router, a vxlan tunnel is used across nodes, and an external network router is at the Controller02 node. When vm1 accesses the extranet client, the data packet will be forwarded from the Con roller01 node to the extranet router of Controller02 across the node through the vxlan tunnel and then sent to the extranet client. In the scenario, when a vxlan tunnel data packet reaches a Controller02 node, a 64K big packet is formed after GRO processing and then sent to a protocol stack, and after decapsulation and routing inspection, the data packet is found to be required to be forwarded to an external network card, and is required to be segmented according to the external network card mtu and then sent to an external network client. However, if the data packets in such a scenario can be intelligently determined not to be subjected to gro processing, CPU resources consumed by data packets and segments can be saved.
Based on this, there is a need for an improvement on the way of offloading UDP tunnel traffic in the existing data center.
Disclosure of Invention
The purpose of the application is to provide a flow distribution method suitable for UDP tunnel flow of a data center and a flow forwarding method suitable for the data center, improve network forwarding efficiency and save CPU resources.
The purpose of the application is realized by adopting the following technical scheme:
in a first aspect, the present application provides a method for offloading UDP tunnel traffic applicable to a data center, where the data center includes a plurality of virtual routers, and the method includes: setting an intelligent receiving and distributing (SRO) module at a node of at least one virtual router, and setting a UDP tunnel port for the SRO module; setting monitoring at the UDP tunnel port to receive UDP tunnel traffic packets between nodes of different virtual routers; and judging whether the UDP tunnel flow packet is a local data packet or not, and executing a software packet merging (GRO) operation in response to the UDP tunnel flow packet being the local data packet. The data packet is intelligently identified to be the local packet, the GRO function of the proto over UDP tunnel flow of the cross-node east-west flow is realized, the GRO function is aggregated into a big packet and then sent to a protocol stack, the decapsulation times of a subsequent path are greatly reduced, the use cost of a CPU is reduced, and the forwarding performance is improved. When the data packet is identified to be an external network data packet, the analysis and shunting of the cross-node north-south flow are realized, the GRO process is skipped for direct forwarding, the process of aggregation and segmentation on the path is reduced, and the use expense of a CPU is reduced.
In some optional embodiments, the SRO module is further configured to set an initial offset or initialize a register GRO handling function.
In some optional embodiments, the setting of the monitoring at the UDP tunnel port includes:
and at the node of at least one virtual router, judging whether the destination port of the UDP tunnel flow is the UDP tunnel port set by the SRO module.
In some optional embodiments, the nodes of the virtual router include a node of a first virtual router and a node of a second virtual router, and the UDP tunnel traffic packet is received across the nodes by: and sending the UDP tunnel traffic from the node of the first virtual router to the node of the second virtual router by using an MPLSoverUDP encapsulation mode.
In some optional embodiments, determining whether the UDP tunnel traffic packet is a local data packet includes: extracting the MPLS header according to the initial offset and comparing the MPLS header with the data packet cached on the GRO list; responding to the UDP tunnel traffic packet without finding the same header, temporarily stripping the header of the UDP tunnel traffic packet and taking out an inner IP of the UDP tunnel traffic packet; and querying a local routing table by using the inner layer IP to judge whether the UDP tunnel flow packet is a local data packet.
In some optional embodiments, before determining whether the UDP tunnel traffic packet is a local data packet, the method further includes: and carrying out validity check on the UDP tunnel flow packet according to the initial offset, and stopping the GRO operation and/or stopping the operation of judging whether the UDP tunnel flow packet is a local data packet or not in response to the fact that the UDP tunnel flow packet is not valid.
In some optional embodiments, the method further comprises: and responding to the UDP tunnel flow packet as an external network data packet, and stopping the GRO operation.
In a second aspect, the present application provides a traffic forwarding method applicable to a data center, where the data center includes a first virtual router and a second virtual router, and the method includes: performing flow encapsulation on the first virtual router; setting an SRO module at a node of a first virtual router and/or a node of a second virtual router, and setting a UDP tunnel port for the SRO module; sending the UDP tunnel flow from the node of the first virtual router to the node of the second virtual router by using an MPLSoverUDP encapsulation mode, and setting monitoring at the port of the UDP tunnel; judging whether the UDP tunnel flow packet is a local data packet or not; responding to the UDP tunnel traffic packet as a local data packet, executing GRO operation, decapsulating the UDP tunnel traffic packet after the GRO operation, and sending the UDP tunnel traffic packet to the second virtual router; or responding to the UDP tunnel traffic packet as an external network data packet, decapsulating the UDP tunnel traffic packet and sending the decapsulated UDP tunnel traffic packet to the second virtual router. The technical scheme has the advantages that the SRO module can configure GRO parameters of UDP encapsulation in each node in a user-defined manner; the GRO function of protoover UDP tunnel flow of cross-node east-west flow can be realized, the GRO function is aggregated into a large packet and then sent to a protocol stack, the decapsulation times of a subsequent path are greatly reduced, the use cost of a cpu is reduced, and the forwarding performance is improved; the method can realize the analysis and shunting of the cross-node north-south flow, skip the GRO process for direct forwarding, reduce the process of aggregation and segmentation on the path and reduce the use cost of a CPU.
In some optional embodiments, the SRO module is further configured to set an initial offset or initialize a register GRO handling function.
In some optional embodiments, setting listening at the UDP tunnel port includes:
and at a node of a second virtual router, judging whether a destination port of the UDP tunnel flow is a UDP tunnel port set by the SRO module.
In some optional embodiments, determining whether the UDP tunnel traffic packet is a local data packet includes: extracting the MPLS header according to the initial offset and comparing the MPLS header with the data packet cached on the GRO list; responding to the UDP tunnel traffic packet without finding the same header, temporarily stripping the header of the UDP tunnel traffic packet and taking out an inner IP of the UDP tunnel traffic packet; and querying a local routing table by using the inner layer IP to judge whether the UDP tunnel flow packet is a local data packet.
In some optional embodiments, before determining whether the UDP tunnel traffic packet is a local data packet, the method further includes: and carrying out validity check on the UDP tunnel flow packet according to the initial offset, and stopping the GRO operation and/or stopping the operation of judging whether the UDP tunnel flow packet is a local data packet or not in response to the fact that the UDP tunnel flow packet is not valid.
In a third aspect, the present application provides an electronic device, which includes a memory, a processor, and a hardware module for performing tasks, where the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.
In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program which, when executed by a processor, implements the steps of any of the methods described above.
Drawings
The present application is further described below with reference to the drawings and examples.
FIG. 1 is a schematic diagram of a prior art virtual router structure;
FIG. 2 is a schematic diagram of another prior art virtual router;
fig. 3 is a schematic flowchart of a UDP tunnel traffic flow splitting method applied to a data center according to an embodiment of the present application;
fig. 4 is a schematic diagram illustrating east-west traffic forwarding according to an embodiment of the present application;
FIG. 5 is a schematic diagram illustrating the forwarding of north-south traffic according to an embodiment of the present application;
fig. 6 is a flowchart illustrating configuration initialization of an SRO module and forwarding of a data stream to the SRO module according to an embodiment of the present application;
fig. 7 is a schematic diagram illustrating a processing flow of a data flow in an SRO module according to an embodiment of the present application;
fig. 8 is a schematic flow chart of a traffic forwarding method applied to a data center according to an embodiment of the present application;
fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present application;
fig. 10 is a schematic structural diagram of a program product for implementing a type selection method according to an embodiment of the present application.
Detailed Description
The present application is further described with reference to the accompanying drawings and the detailed description, and it should be noted that, in the present application, the embodiments or technical features described below may be arbitrarily combined to form a new embodiment without conflict.
Referring to fig. 3, the embodiment of the present application provides a UDP tunnel traffic offload method applicable to a data center, which is an offload (offload) of UDP tunnel traffic that can be configured by a user, and the method is applicable to specific scenarios such as check and transmission of electronic mailboxes, online storage, business cooperation of instant messaging, virtual office, and can implement high-speed forwarding and offload of traffic data in the data center. The control method comprises steps S101-S103, and in this embodiment, the data center may comprise a plurality of virtual routers.
Step S101, an intelligent receive offload (SRO) module is set at a node of at least one virtual router, and a UDP tunnel port is set for the SRO module. Illustratively, a Virtual Router (VRouter) in a data center may be installed using mplSoverUDP tunnel encapsulation deployment, an SRO module is installed and deployed in each VRouter node in the data center, and an own UDP tunnel port is set. In this embodiment, the specific execution command is: 6635 and 4, and restarting the VRouter agent of the node.
Step S102, setting monitoring at UDP tunnel port to receive UDP tunnel flow package between nodes of different virtual router. And after loading the SRO module, establishing a monitoring UDP socket at a UDP tunnel port. The nodes of the virtual router may include a node of a first virtual router and a node of a second virtual router. The UDP tunnel port (tunnel socket) establishing monitoring is set when the SRO module is loaded, and only the first virtual router node flow plus encapsulation (vxlan/mpls) is sent to the node of the second virtual router, and the node of the second virtual router judges whether to register the monitored socket for the SRO according to the encapsulation protocol port, namely the destination port of the data packet sent to the node of the second virtual router by the first virtual router node plus encapsulation. In this embodiment, a listening UDP socket 6635 is created by the SRO module based on port 6635, and the mounting GRO receive/complete processing function is set, the data header length offset is offset 4, and the setting of the initial offset is completed. Optionally, the SRO module will further initialize a registration GRO handling function according to the aforementioned configuration. Through the above operation, when the SRO module is deployed, the UDP tunnel port and the offset length used by the configuration data center are specified, and the SRO module completes the initialization operation.
Step S103, determining whether the UDP tunnel traffic packet is a local data packet, and executing a software packet merging (GRO) operation in response to the UDP tunnel traffic packet being the local data packet.
For example, when the node data network cards of a plurality of virtual routers included in the data center receive packets, the SR O device monitors and captures UDP data packets configured when the destination port is deployed.
Optionally, the determining whether the UDP tunnel traffic packet is a local data packet includes: extracting the MPLS header according to the initial offset and comparing the MPLS header with the data packet cached on the GRO list; responding to the situation that the UDP tunnel flow packet with the same header r is not found, temporarily stripping the header of the UDP tunnel flow packet and taking out the inner layer IP of the UDP tunnel flow packet; and inquiring a local routing table by using the inner layer IP to judge whether the UDP tunnel flow packet is a local data packet.
Optionally, in order to ensure the detection efficiency, before determining whether the UDP tunnel traffic packet is a local data packet, the following steps are further performed: and carrying out validity check on the UDP tunnel flow packet according to the initial offset, and stopping GRO operation and/or stopping the operation of judging whether the UDP tunnel flow packet is a local data packet or not in response to the fact that the UDP tunnel flow packet is not valid.
In one embodiment, the UDP tunnel traffic packet is determined to be a local packet by:
firstly, the SRO module sends the data packet/UDP tunnel traffic packet to a GRO receive/complete processing function (GRO function) for processing, acquires an encapsulation header according to offset, and performs validity check on the data packet. Carrying out subsequent processing on the legal data packet; and if the data packet is illegal, stopping subsequent operation.
Secondly, extracting the mpls header according to the initial offset and comparing the mpls header with the data packet cached in the GRO list, setting the same value as 1 if the same value is same with the same value of the same sample flow, setting flush value as 0 after stripping the mpls header according to the offset, and sending the flush value as 0 to a corresponding kernel GRO function for processing according to the next layer of protocol type; if the mpls header is taken out according to the initial offset and compared with the data packet cached in the GRO list, but the same header data packet is not found, the data packet header is temporarily stripped to take out the inner layer IP and inquire a local routing table, and whether the detection is a local data packet is judged.
In an embodiment, when the UDP tunnel traffic packet/data packet is detected as a local data packet, the UDP tunnel traffic packet/data packet is an eastern-western traffic, and after the data packet is added to the GRO list, the mpls header is stripped according to the offset, and then flush is set to 0, and the data packet is sent to a corresponding kernel GRO function for processing according to the next layer of protocol type. In one embodiment, when the UDP tunnel traffic packet is detected as an external network packet, the packet is a north-south traffic, and flush is set to 1, and no GRO processing is performed.
According to the UDP tunnel flow shunting method applicable to the data center, the SRO module is arranged at the node of the virtual router, and configured UDP tunnel flow shunting can be customized; monitoring is set according to a UDP tunnel port by utilizing an SR O module so as to receive UDP tunnel flow packets among nodes of different virtual routers; and judging whether the UDP tunnel flow packet is a local data packet or not, and executing a software packet merging (GRO) operation in response to the UDP tunnel flow packet being the local data packet. The method can realize the GRO function of the user-defined UDP tunnel encapsulation data packet, realize the analysis and shunt of the cross-node north-south flow and skip the GRO processing, reduce the process of segmenting after aggregation on the path, improve the network forwarding efficiency and save the CPU resource.
Fig. 4 is a schematic diagram illustrating east-west traffic forwarding according to an embodiment of the present application. Wherein, node 1(Controller1) corresponds to virtual router 1(VRouter1), which includes virtual machine VM 1; node 2 (controller 2) corresponds to virtual router 2(VRouter2), which includes virtual machine VM 2. After the flow is sent from the VM1, the flow is sent from the node controller1 to the node controller2 through Virtual Routing Forwarding (VRF) and one or more encapsulation operations, using mpls over udp in an encapsulation mode; after receiving the UDP tunnel data packet, the controller2 node network card driver sends the flow to a UDP GRO receive interface of the kernel and traverses and searches a local UDP socket list, namely, monitoring is set at a UDP tunnel port, and UDP socket matching is executed; in this embodiment, the mplSoverUDP tunnel is encapsulated by the UDP destination port 6635, and the SRO registration creation listens to the 6635 port, so that the data packet with the destination port 6635 is captured to the SRO module for processing. In this embodiment, the node 2 also performs Custom GRO func processing, acquires an encapsulation header according to offset, and performs validity check on a data packet; then, the data packet is confirmed to be a local data packet (local detection), and after the data packet is determined to be legal and the local data packet is determined, the GRO operation and the subsequent parsing/decapsulating operation are executed.
Fig. 5 is a schematic diagram illustrating the forwarding of north-south traffic according to an embodiment of the present application. Wherein, the node 3(Controller3) corresponds to the virtual router 3(VRouter3), which includes a virtual machine VM 3; node 4 (controller 4) corresponds to virtual router 4(VRouter4), which includes virtual machine VM 4. After the flow is sent from the VM3, the flow is sent from the node controller3 to the node controller4 through Virtual Routing Forwarding (VRF) and one or more encapsulation operations, using mpls over udp in an encapsulation mode; after receiving the UDP tunnel data packet, the controller4 node network card driver sends the traffic to the UD P GRO receive interface of the kernel and traverses the UDP socket list to search for the local UDP socket list, i.e., performs UDP socket matching, in this embodiment, the MPLSoverUDP tunnel is encapsulated by the UDP destination port 6635, and the SRO registration creates and monitors the 6635 port, so that the data packet with the destination port 6635 is captured to the SR O module for processing. In this embodiment, if the packet is detected as an extranet packet, the packet is a north-south traffic, and specifically, the flush is set to 1, and the GRO processing is not performed.
In one embodiment, the SRO module configuration initialization and data flow forwarding to SRO module flow diagram is shown in fig. 6. Installing and deploying an SRO module at each VRouter node of a data center, setting a self-defined UDP tunnel port, and specifically executing a command: and restarting the vrouter agent of the node where the modprobe sro port is 6635offset is 4. After loading the SRO module, the SRO module creates a listening UDP socket 6635 based on port 6635, and sets a mount GRO receive/complete processing function, with the data header length offset being offset 4.
When the cross-node traffic of each node is processed, as described above, after receiving the UDP tunnel data packet from the node of the first virtual router and the node of the second virtual router using the mpls over UDP encapsulation mode traffic, the node network card driver (kernel) of the second virtual router sends the traffic to the UDP GRO receive interface of the kernel and searches through the local UDP socket list, that is, the destination port is used to query the local socket option. In this embodiment, the destination port hits socket A. Further, the flow enters a custom GRO interface for processing. Then, when the validity check does not pass, setting a flush 1 identifier, not carrying out subsequent GRO treatment, and directly sending the GRO to a kernel; and when the validity check passes, setting flush as 0, sending the flush to a corresponding kernel GRO function for processing according to the next layer of protocol type, and continuing to check and merge layer by layer.
In one embodiment, the data flow is processed in the SRO module as shown in fig. 7.
Firstly, for executing custom GRO func, flow of legalization inspection is completed;
next, the tunnel header A is obtained according to the offset length. In this embodiment, the mpls header A is fetched based on the offset offs et 4 and checked for correctness. If the header A is checked to be correct, continuing to execute subsequent operations; if the check header A is incorrect/wrong, the flush is set to 1, and no subsequent operation is performed.
Then, the current packet traversal is used to find whether the GROlist has the same flow. In this embodiment, the header a is compared with the data packet cached in the gro list, if the same stream is set to 1, after the mpls header is stripped according to the offset, flush is set to 0, and the data packet is sent to the corresponding kernel gro function for processing according to the next layer of protocol type; if the same header data packet is not found, the IP address of the inner layer header is taken, the data packet is decapsulated, the data packet header is temporarily stripped to take out the inner layer IP and inquire a local routing table (whether the destination is a local IP is checked), and whether the detection is a local data packet is judged.
In one embodiment, when the local packet is detected, the local packet is an eastern and western flow, and the specific data flow is as shown in fig. 4, the packet is added to a GRO list, the mpls header is stripped according to offset, then the plus h is set to 0, and the packet is sent to a corresponding kernel GRO function for processing according to a next layer protocol type. In an embodiment, when the packet is detected as an external network packet, the packet is a north-south traffic, and the specific data flow is as shown in fig. 5, where flush is set to 1, and the GRO processing is not performed. In the embodiment of the application, after the SRO module is used, the data packet is encapsulated in the UDP tunnel, the performance of a single stream can be increased from 5Gbits/sec to 9Gbits/sec by taking a 10G network card as an example, and the cpu utilization rate is reduced by 25% under the same flow rate of the outbound network data packet.
It should be noted that the above description of the flow of the offloading method for UDP tunnel traffic is provided for illustrative purposes only and is not intended to limit the scope of the present application. Many variations and modifications may be made to the teachings of the present application by those of ordinary skill in the art in light of the present disclosure. However, such changes and modifications do not depart from the scope of the present application. In some embodiments, the flow of the offloading method of UDP tunnel traffic may be accomplished with one or more additional operations not described and/or without one or more of the operations discussed.
In an embodiment, the present application provides a traffic forwarding method applied to a data center, where the data center includes a first virtual router and a second virtual router, and a User Datagram Protocol (UDP) encapsulated GRO parameter can be configured in a customized manner in each virtual router node, and the method shown in fig. 8 includes:
step S201, performing flow encapsulation on a first virtual router;
step S202, an SRO module is arranged at a node of the first virtual router and/or the second virtual router, and a UDP tunnel port is set for the SRO module; optionally, the SRO module is further configured to set an initial offset or initialize a registration GRO handling function;
step S203, sending the UDP tunnel flow from the node of the first virtual router to the node of the second virtual router by using an MPLSoverUDP encapsulation mode, and setting monitoring at the UDP tunnel port; optionally, the UDP tunnel port (tunnel socket) creation monitoring is set when the SRO module is loaded, and only the first virtual router node traffic plus encapsulation (vxlan/mpls) is sent to the node of the second virtual router, and the node of the second virtual router determines, according to an encapsulation protocol port, that is, a destination port of a packet sent to the node of the second virtual router by adding encapsulation to the first virtual router node, whether the packet is a socket determination processing for SRO registration monitoring;
step S204, judging whether the UDP tunnel flow packet is a local data packet;
step S205, in response to the UDP tunnel traffic packet being a local data packet, executing a GRO operation, decapsulating the UDP tunnel traffic packet after executing the GRO operation, and sending the decapsulated UDP tunnel traffic packet to the second virtual router;
step S206, responding to the UDP tunnel traffic packet as an extranet data packet, decapsulating the UDP tunnel traffic packet and sending the decapsulated UDP tunnel traffic packet to the second virtual router.
Optionally, the determining whether the UDP tunnel traffic packet is a local data packet includes:
extracting the MPLS header according to the initial offset and comparing the MPLS header with the data packet cached on the GRO list; responding to the UDP tunnel traffic packet without finding the same header, temporarily stripping the header of the UDP tunnel traffic packet and taking out an inner layer IP of the UDP tunnel traffic packet; and inquiring a local routing table by using the inner layer IP to judge whether the UDP tunnel flow packet is a local data packet.
Optionally, before determining whether the UDP tunnel traffic packet is a local data packet, the method further includes: and carrying out validity check on the UDP tunnel flow packet according to the initial offset, and stopping GRO operation and/or stopping the operation of judging whether the UDP tunnel flow packet is a local data packet or not in response to the fact that the UDP tunnel flow packet is not valid.
In the embodiment of the application, the data packet is intelligently identified to be the local packet, the GRO function of proto over UDP tunnel flow of cross-node east-west flow is realized, the GRO function is aggregated into a big packet and then sent to a protocol stack, the decapsulation times of a subsequent path are greatly reduced, the use cost of a CPU is reduced, and the forwarding performance is improved. When the data packet is identified to be an external network data packet, the analysis and shunting of the cross-node north-south flow are realized, the GRO process is skipped for direct forwarding, the process of aggregation and segmentation on the path is reduced, and the use expense of a CPU is reduced.
Referring to fig. 9, an embodiment of the present application further provides an electronic device 200, where the electronic device 200 includes at least one memory 210, at least one processor 220, and a bus 230 connecting different platform systems.
The memory 210 may include readable media in the form of volatile memory, such as Random Access Memory (RAM)211 and/or cache memory 212, and may further include Read Only Memory (ROM) 213.
The memory 210 further stores a computer program, and the computer program can be executed by the processor 220, so that the processor 220 executes a flow splitting method suitable for UDP tunnel traffic of the data center and/or a traffic forwarding method suitable for the data center.
The UDP tunnel traffic flow distribution method applicable to the data center comprises the following steps: setting an SRO module at a node of at least one virtual router, and setting a UDP tunnel port for the SRO module; setting monitoring at a UDP tunnel port to receive UDP tunnel traffic packets between nodes of different virtual routers; judging whether the UDP tunnel flow packet is a local data packet or not, and executing GRO operation in response to the UDP tunnel flow packet being the local data packet; and responding to the UDP tunnel flow packet as an external network data packet, and stopping GRO operation.
The traffic forwarding method applicable to the data center comprises the following steps: performing flow encapsulation on the first virtual router; setting an SRO module at a node of the first virtual router and/or the second virtual router, and setting a UDP tunnel port for the SRO module; sending the UDP tunnel flow from the node of the first virtual router to the node of the second virtual router by using an MPLSoverUDP encapsulation mode, and setting monitoring at the port of the UDP tunnel; judging whether the UDP tunnel flow packet is a local data packet or not; responding to the UDP tunnel traffic packet as a local data packet, executing GRO operation, decapsulating the UDP tunnel traffic packet after executing GRO operation, and sending the decapsulated UDP tunnel traffic packet to a second virtual router; or responding to the UDP tunnel traffic packet as an external network data packet, decapsulating the UDP tunnel traffic packet and sending the decapsulated UDP tunnel traffic packet to the second virtual router.
The specific implementation manner is consistent with the implementation manner and the achieved technical effect described in the embodiment of the method, and some contents are not described again. Memory 210 may also include a program/utility 214 having a set (at least one) of program modules 215, such program modules 215 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.
Accordingly, processor 220 may execute the computer programs described above, as well as may execute programs/utilities 214.
The electronic device 200 may also communicate with one or more external devices 240, such as a keyboard, pointing device, Bluetooth device, etc., and may also communicate with one or more devices capable of interacting with the electronic device 200, and/or with any devices (e.g., routers, modems, etc.) that enable the electronic device 200 to communicate with one or more other computing devices. Such communication may occur via an input/output (I/O) interface 250. Also, the electronic device 200 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via the network adapter 260. The network adapter 260 may communicate with other modules of the electronic device 200 via the bus 230. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the electronic device 200, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, to name a few.
Embodiments of the present application further provide a computer-readable storage medium for storing a computer program, where the computer program is executed to implement:
setting an SRO module at a node of at least one virtual router, and setting a UDP tunnel port for the SRO module; setting monitoring at a UDP tunnel port to receive UDP tunnel traffic packets between nodes of different virtual routers; judging whether the UDP tunnel flow packet is a local data packet or not, and executing GRO operation in response to the UDP tunnel flow packet being the local data packet; and responding to the UDP tunnel flow packet as an external network data packet, and stopping GRO operation.
Or, performing traffic encapsulation on the first virtual router; setting an SRO module at a node of the first virtual router and/or the second virtual router, and setting a UDP tunnel port for the SRO module; sending the UDP tunnel flow from the node of the first virtual router to the node of the second virtual router by using an MPLSoverUDP encapsulation mode, and judging whether a destination port of the UDP tunnel flow is a UDP tunnel port set by the SRO module or not at the node of the second virtual router; judging whether the UDP tunnel flow packet is a local data packet or not; responding to the UDP tunnel traffic packet as a local data packet, executing GRO operation, decapsulating the UDP tunnel traffic packet after executing GRO operation, and sending the decapsulated UDP tunnel traffic packet to a second virtual router; or responding to the UDP tunnel traffic packet as an external network data packet, decapsulating the UDP tunnel traffic packet and sending the decapsulated UDP tunnel traffic packet to the second virtual router.
The specific implementation manner is consistent with the implementation manner and the achieved technical effect described in the embodiment of the method, and some contents are not described again. Fig. 10 shows a program product 300 provided by the present embodiment for implementing the method, which may employ a portable compact disc read only memory (CD-ROM) and include program codes, and may be run on a terminal device, such as a personal computer. However, the program product 300 of the present invention is not so limited, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. Program product 300 may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
A computer readable storage medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable storage medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing. Program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
The foregoing description and drawings are only for purposes of illustrating the preferred embodiments of the present application and are not intended to limit the present application, which is, therefore, to the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the present application.
Claims (14)
1. A method for offloading UDP tunnel traffic applicable to a data center, the data center including a plurality of virtual routers, the method comprising:
setting an intelligent receiving and distributing (SRO) module at a node of at least one virtual router, and setting a UDP tunnel port for the SRO module;
setting monitoring at the UDP tunnel port to receive UDP tunnel traffic packets between nodes of different virtual routers;
and judging whether the UDP tunnel flow packet is a local data packet or not, and executing a software packet merging (GRO) operation in response to the UDP tunnel flow packet being the local data packet.
2. The method for offloading UDP tunnel traffic applicable to a data center of claim 1, wherein the SRO module is further configured to set an initial offset or initialize a registration GRO handling function.
3. The method for offloading UDP tunnel traffic applicable to the data center according to claim 1 or 2, wherein the setting listening on the UDP tunnel port includes:
and at the node of at least one virtual router, judging whether the destination port of the UDP tunnel flow is the UDP tunnel port set by the SRO module.
4. The method for offloading UDP tunnel traffic applicable to a data center of claim 1, wherein the nodes of the virtual router comprise a node of a first virtual router and a node of a second virtual router, and wherein the UDP tunnel traffic packet is received across the nodes by:
and sending the UDP tunnel traffic from the node of the first virtual router to the node of the second virtual router by using an MPLSoverUDP encapsulation mode.
5. The method for offloading UDP tunnel traffic applicable to the data center of claim 4, wherein determining whether the UDP tunnel traffic packet is a local data packet comprises:
extracting the MPLS header according to the initial offset and comparing the MPLS header with the data packet cached on the GRO list;
responding to the UDP tunnel traffic packet without finding the same header, temporarily stripping the header of the UDP tunnel traffic packet and taking out an inner IP of the UDP tunnel traffic packet;
and querying a local routing table by using the inner layer IP to judge whether the UDP tunnel flow packet is a local data packet.
6. The method for offloading UDP tunnel traffic applicable to the data center of claim 5, further comprising, before determining whether the UDP tunnel traffic packet is a local data packet:
and carrying out validity check on the UDP tunnel flow packet according to the initial offset, and stopping the GRO operation and/or stopping the operation of judging whether the UDP tunnel flow packet is a local data packet or not in response to the fact that the UDP tunnel flow packet is not valid.
7. The method for offloading UDP tunnel traffic applicable to a data center according to any one of claims 1, 2 or 4, wherein the method further comprises:
and responding to the UDP tunnel flow packet as an external network data packet, and stopping the GRO operation.
8. A traffic forwarding method applicable to a data center, where the data center includes a first virtual router and a second virtual router, the method comprising:
performing flow encapsulation on the first virtual router;
setting an intelligent receiving and distributing (SRO) module at a node of a first virtual router and/or a node of a second virtual router, and setting a UDP tunnel port for the SRO module;
sending the UDP tunnel flow from the node of the first virtual router to the node of the second virtual router by using an MPLSoverUDP encapsulation mode, and setting monitoring at the port of the UDP tunnel;
judging whether the UDP tunnel flow packet is a local data packet or not;
responding to the UDP tunnel traffic packet as a local data packet, executing a software packet merging (GRO) operation, decapsulating the UDP tunnel traffic packet after the GRO operation, and sending the UDP tunnel traffic packet to the second virtual router; alternatively, the first and second electrodes may be,
and responding to the UDP tunnel traffic packet as an external network data packet, and sending the UDP tunnel traffic packet to the second virtual router after decapsulating the UDP tunnel traffic packet.
9. The traffic forwarding method applicable to data centers as claimed in claim 8, wherein the SRO module is further configured to set an initial offset or initialize a registration GRO handling function.
10. The traffic forwarding method applicable to the data center according to claim 8 or 9, wherein the setting of the listening on the UDP tunnel port comprises:
and at a node of a second virtual router, judging whether a destination port of the UDP tunnel flow is a UDP tunnel port set by the SRO module.
11. The traffic forwarding method applicable to the data center according to claim 8, wherein determining whether the UDP tunnel traffic packet is a local data packet comprises:
extracting the MPLS header according to the initial offset and comparing the MPLS header with the data packet cached on the GRO list;
responding to the UDP tunnel traffic packet without finding the same header, temporarily stripping the header of the UDP tunnel traffic packet and taking out an inner IP of the UDP tunnel traffic packet;
and querying a local routing table by using the inner layer IP to judge whether the UDP tunnel flow packet is a local data packet.
12. The traffic forwarding method applicable to the data center according to claim 8 or 11, wherein before determining whether the UDP tunnel traffic packet is a local data packet, the method further comprises:
and carrying out validity check on the UDP tunnel flow packet according to the initial offset, and stopping the GRO operation and/or stopping the operation of judging whether the UDP tunnel flow packet is a local data packet or not in response to the fact that the UDP tunnel flow packet is not valid.
13. An electronic device comprising a memory, a processor and hardware modules for performing tasks, the memory storing a computer program, the processor implementing the steps of the method according to any of claims 1-12 when executing the computer program.
14. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 12.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111153796.0A CN113890789B (en) | 2021-09-29 | 2021-09-29 | UDP tunnel traffic shunting method and traffic forwarding method suitable for data center |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111153796.0A CN113890789B (en) | 2021-09-29 | 2021-09-29 | UDP tunnel traffic shunting method and traffic forwarding method suitable for data center |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113890789A true CN113890789A (en) | 2022-01-04 |
| CN113890789B CN113890789B (en) | 2023-03-21 |
Family
ID=79008375
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111153796.0A Active CN113890789B (en) | 2021-09-29 | 2021-09-29 | UDP tunnel traffic shunting method and traffic forwarding method suitable for data center |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113890789B (en) |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2002191026A (en) * | 1999-02-05 | 2002-07-05 | Toshiba Corp | Stream information processing system |
| CN101217464A (en) * | 2007-12-28 | 2008-07-09 | 北京大学 | UDP data package transmission method |
| CN105162674A (en) * | 2015-10-16 | 2015-12-16 | 上海斐讯数据通信技术有限公司 | Method and network card for physical machine to access to virtual network |
| CN107078936A (en) * | 2014-08-19 | 2017-08-18 | 思杰系统有限公司 | For the system and method for the fine granularity control for providing the MSS values connected to transport layer |
| US20180123943A1 (en) * | 2016-11-03 | 2018-05-03 | Futurewei Technologies, Inc. | Global Resource Orchestration System for Network Function Virtualization |
| CN108337183A (en) * | 2017-12-19 | 2018-07-27 | 南京大学 | A kind of method of data center network current load equilibrium |
| EP3370158A1 (en) * | 2017-03-01 | 2018-09-05 | Juniper Networks, Inc. | Network interface card switching for virtual networks |
| WO2018227519A1 (en) * | 2017-06-16 | 2018-12-20 | 唐全德 | System, method and apparatus for implementing network interconnection |
| CN109246189A (en) * | 2018-08-03 | 2019-01-18 | 广州广哈通信股份有限公司 | Network data distributing method and device, storage medium, server-side |
| CN109510740A (en) * | 2019-01-02 | 2019-03-22 | 郑州云海信息技术有限公司 | A kind of method, apparatus, terminal and the storage medium of automatic monitoring network interface card MTU value |
| CN109639488A (en) * | 2018-12-18 | 2019-04-16 | 无锡华云数据技术服务有限公司 | A kind of more outer nets shunt accelerated method and system |
| US20200021533A1 (en) * | 2018-07-13 | 2020-01-16 | Samsung Electronics Co., Ltd. | Apparatus and method for processing data packet of electronic device |
| CN112532468A (en) * | 2019-09-19 | 2021-03-19 | 华为技术有限公司 | Network measurement system, method, device and storage medium |
| WO2021128089A1 (en) * | 2019-12-25 | 2021-07-01 | 华为技术有限公司 | Forwarding device, network card, and message forwarding method |
-
2021
- 2021-09-29 CN CN202111153796.0A patent/CN113890789B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2002191026A (en) * | 1999-02-05 | 2002-07-05 | Toshiba Corp | Stream information processing system |
| CN101217464A (en) * | 2007-12-28 | 2008-07-09 | 北京大学 | UDP data package transmission method |
| CN107078936A (en) * | 2014-08-19 | 2017-08-18 | 思杰系统有限公司 | For the system and method for the fine granularity control for providing the MSS values connected to transport layer |
| CN105162674A (en) * | 2015-10-16 | 2015-12-16 | 上海斐讯数据通信技术有限公司 | Method and network card for physical machine to access to virtual network |
| US20180123943A1 (en) * | 2016-11-03 | 2018-05-03 | Futurewei Technologies, Inc. | Global Resource Orchestration System for Network Function Virtualization |
| EP3370158A1 (en) * | 2017-03-01 | 2018-09-05 | Juniper Networks, Inc. | Network interface card switching for virtual networks |
| WO2018227519A1 (en) * | 2017-06-16 | 2018-12-20 | 唐全德 | System, method and apparatus for implementing network interconnection |
| CN108337183A (en) * | 2017-12-19 | 2018-07-27 | 南京大学 | A kind of method of data center network current load equilibrium |
| US20200021533A1 (en) * | 2018-07-13 | 2020-01-16 | Samsung Electronics Co., Ltd. | Apparatus and method for processing data packet of electronic device |
| CN109246189A (en) * | 2018-08-03 | 2019-01-18 | 广州广哈通信股份有限公司 | Network data distributing method and device, storage medium, server-side |
| CN109639488A (en) * | 2018-12-18 | 2019-04-16 | 无锡华云数据技术服务有限公司 | A kind of more outer nets shunt accelerated method and system |
| CN109510740A (en) * | 2019-01-02 | 2019-03-22 | 郑州云海信息技术有限公司 | A kind of method, apparatus, terminal and the storage medium of automatic monitoring network interface card MTU value |
| CN112532468A (en) * | 2019-09-19 | 2021-03-19 | 华为技术有限公司 | Network measurement system, method, device and storage medium |
| WO2021128089A1 (en) * | 2019-12-25 | 2021-07-01 | 华为技术有限公司 | Forwarding device, network card, and message forwarding method |
Non-Patent Citations (4)
| Title |
|---|
| GUANGLONGXISHUI: "GRO的实现 （linux网络子系统学习 第七节）", 《HTTPS://BLOG.51CTO.COM/YAOYANG/1303010》 * |
| PAOLO ABENI: "udp: implement GRO support", 《LWN.NET》 * |
| YIRAN ZHANG: "VMS: Load Balancing Based on the Virtual Switch Layer in Datacenter Networks" * |
| 胡进军: "网络流量识别技术的研究与实现" * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113890789B (en) | 2023-03-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9602307B2 (en) | Tagging virtual overlay packets in a virtual networking system | |
| CN109547580B (en) | Method and device for processing data message | |
| US9385912B1 (en) | Framework for stateless packet tunneling | |
| TWI626537B (en) | Methods and systems for analyzing record and usage in post package repair | |
| CN112968844B (en) | Method and device for sending fragment message | |
| CN115134245A (en) | Techniques for network packet processing between cloud and telecommunications networks | |
| US9910687B2 (en) | Data flow affinity for heterogenous virtual machines | |
| CN111131037A (en) | Data transmission method, device, medium and electronic equipment based on virtual gateway | |
| US9619272B1 (en) | Virtual machine networking | |
| CN115589383B (en) | EBPF-based virtual machine data transmission method, eBPF-based virtual machine data transmission device, eBPF-based virtual machine data transmission apparatus, eBPF-based virtual machine data transmission device, eBPF-based virtual machine data transmission storage medium, and eBPF-based virtual machine data transmission program product | |
| KR20120062174A (en) | Apparatus and method for dynamic processing a variety of characteristics packet | |
| US10608889B2 (en) | High-level interface to analytics engine | |
| CN113890789B (en) | UDP tunnel traffic shunting method and traffic forwarding method suitable for data center | |
| CN116346649A (en) | Virtual service packet grabbing method and device of load balancing equipment | |
| US10298606B2 (en) | Apparatus, system, and method for accelerating security inspections using inline pattern matching | |
| CN115277558B (en) | Message sending method and device, computer storage medium and electronic equipment | |
| CN111371723B (en) | Method and device for realizing PPTP VPN network isolation under DPDK framework | |
| CN116132435B (en) | Double-stack cross-node communication method and system of container cloud platform | |
| CN115208780B (en) | Method and device for determining maximum transmission unit, storage medium and electronic equipment | |
| CN108519912B (en) | Data cleaning method and device, computer readable storage medium and electronic equipment | |
| US20240205050A1 (en) | Computing device with a virtual network layer installed with user plane function (upf) plugin and method for processing upf packet | |
| CN115208780A (en) | Method and device for determining maximum transmission unit, storage medium and electronic equipment | |
| CN117793038A (en) | Message processing method, device, electronic equipment and computer readable storage medium | |
| CN113965511A (en) | Tag data transmission method based on RDMA (remote direct memory Access), and related device and system | |
| CN115514771A (en) | Data transmission method and device, electronic equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |