CN117178523A - Multi-uplink path quality aware IPSEC - Google Patents

Abstract

Some embodiments provide a method of collecting metrics for one or more paths of a first tunnel implementing a first Security Association (SA) and for one or more paths of a second tunnel implementing a second SA. The method selects a path based on the collected metrics of the paths of the first tunnel and the second tunnel. When the selected path belongs to the first tunnel, the method encrypts data to be transmitted as an encrypted payload of the first SA and transmits the encrypted payload in the first tunnel. When the selected path belongs to the second tunnel, the method encrypts data to be transmitted as an encrypted payload of the second SA and transmits the encrypted payload in the second tunnel.

Description

Multi-uplink path quality aware IPSEC

Background

Internet protocol security (IPsec) is a set of protocols used together to set up encrypted connections between devices so that private data can be sent securely over a public network. IPsec is often used to set up Virtual Private Networks (VPN) by encrypting IP packets and authenticating the source of the packets. IPsec VPNs are widely used by enterprises to interconnect geographically diverse branch locations thereof via a Wide Area Network (WAN) or the internet, particularly in the software defined WAN (SD-WAN) era. Cloud providers also use IPsec to encrypt IP traffic across data center interconnected WANs to meet security and compliance requirements, especially in financial and government cloud environments.

Internet Key Exchange (IKE) is a protocol for establishing a secure and authenticated communication channel between two parties. IKE is typically authenticated using public key infrastructure certificates and uses a key exchange protocol to set up a shared session key. IKE is part of IPsec and is responsible for negotiating Security Associations (SAs), which is a collection of keys and algorithms agreed upon by both parties for use by both parties attempting to establish VPN connections/tunnels.

Modern data center networks or WAN networks include redundant paths between endpoints. It is important for modern cloud workloads to fully exploit multiple links or paths for better performance, better reliability, faster adaptation to route disruptions or misconfigurations, etc.

Equal cost multi-path routing (ECMP) is a routing strategy in which packet forwarding to a single destination can occur on multiple best paths with the same routing priority. ECMP is a per-hop decision made independently on each router. It can significantly increase bandwidth by load balancing traffic over multiple paths.

Disclosure of Invention

Some embodiments of the present disclosure provide a method of transmitting IPsec data using multiple paths in multiple different Security Associations (SAs). The gateway collects metrics for one or more paths of the first tunnel and one or more paths of the second tunnel. The gateway receives data to be transmitted from a first network endpoint to a second network endpoint. When the selected path belongs to the first tunnel, the gateway encrypts the received data as an encrypted payload of the first SA and encapsulates the encrypted payload by appending (i) a first source address identifying the first tunnel and (ii) a first source port identifying the selected path. The gateway then transmits the encapsulated encrypted payload in a first tunnel. When the selected path belongs to the second tunnel, the gateway encrypts the received data as an encrypted payload of the second SA and encapsulates the encrypted payload by appending (i) a second source address identifying the second tunnel and (ii) a second source port identifying the selected path. The gateway transmits the encapsulated encrypted payload in a second tunnel.

In some embodiments, a gateway of a data center negotiates a first Virtual Private Network (VPN) tunnel implementing a first SA and a second VPN tunnel implementing a second SA. The first and second SAs and tunnels are established as part of a VPN session, where the gateway is a VPN client and the remote gateway is a VPN server. One tunnel may include a path through the internet while the other tunnel does not include a path through the internet or includes only a direct connection within or between data centers.

In some embodiments, the gateway sends a probe message and receives a response to the probe message. Metrics collected for one or more paths of the first and second tunnels are determined based on the received responses to the probe messages. In some embodiments, the metrics of the path include at least one of connectivity, latency, packet loss rate, jitter of the path.

In some embodiments, the first network endpoint is hosted by a first data center and the second network endpoint is hosted by a second data center. In some embodiments, data is received at a single routing layer interface (or VTI) for encryption and transmission in a first tunnel using a first SA and in a second tunnel using a second SA. In some embodiments, data is received from an application at a bound interface at an application layer, and the bound interface logically combines a first routing layer interface for encrypting and encapsulating the received data for transmission in a first tunnel using a first SA and a second routing layer interface for encrypting and encapsulating the received data for transmission in a second tunnel using a second SA. The gateway selects a path based on the collected metrics of the paths of the first and second tunnels.

The foregoing summary is intended to serve as a brief description of some embodiments of the invention. It is not meant to be an introduction or overview of all inventive subject matter disclosed in this document. The embodiments described in the summary of the invention and in the other embodiments will be further described in the following detailed description and with reference to the drawings of the detailed description. Accordingly, a full appreciation of the inventive content, the detailed description, the drawings, and the claims can be gained by taking the entire specification, claims, through the examples described herein. Furthermore, the claimed subject matter is not limited to the illustrative details in the summary, detailed description, and accompanying drawings.

Drawings

The novel features of the invention are set forth in the appended claims. However, for purposes of explanation, several embodiments of the invention are set forth in the following figures.

Figure 1 conceptually illustrates a network in which there are multiple paths between network endpoints.

Figure 2 conceptually illustrates the transmission of IPsec data from one endpoint to another endpoint over multiple paths.

Figure 3 conceptually illustrates a VPN session established to securely transport or migrate data from a first data center to a second data center.

Fig. 4 conceptually illustrates that a gateway gathers path quality information in order to perform path selection for transmitting IPsec data.

Figure 5 conceptually illustrates load balancing across multiple active paths for security association SA 1.

Figure 6 conceptually illustrates a VPN client using multiple paths in multiple uplinks or tunnels to send IPsec data across a network to a VPN server.

Figure 7 conceptually illustrates a single VTI associated with different SAs for IPsec encryption.

Fig. 8 conceptually illustrates a plurality of VTIs associated with different SAs for encryption logically combined into a bound VTI.

Fig. 9A-B illustrate a gateway using aggregated path information to select a best path from a plurality of different VPN tunnels.

Fig. 10 conceptually illustrates a process of transmitting IPsec data using multiple paths in multiple different SAs.

Fig. 11 illustrates a block diagram of a system that probes multiple paths to find the best path and updates the IP address of the SA to use the best path.

Fig. 12 illustrates a VPN session in which the SA may be configured to use different paths by changing the source and destination addresses.

Fig. 13A-E conceptually illustrate a gateway that uses the MOBIKE protocol to change the source and destination IP addresses of the SA in order to select the best path.

Fig. 14 conceptually illustrates a process of transmitting IPsec data using multipath by changing the IP address of an SA.

Fig. 15 illustrates a block diagram of a system that probes multiple paths to find the best path and updates the IP address of the SA to use the best path.

Figure 16 conceptually illustrates a gateway that uses multiple active uplink interfaces to send IPsec data to its VPN peers.

Figure 17 conceptually illustrates a path pool including paths of several different uplink interfaces.

Figure 18 conceptually illustrates removing a path from a path pool when an uplink interface fails.

Figure 19 conceptually illustrates identifying network paths included in a path pool based on bandwidth.

Fig. 20 illustrates data flow within a gateway for load balancing across multiple paths in multiple uplinks.

Fig. 21 conceptually illustrates a process of performing load balancing when IPsec packets are transmitted across multiple active uplinks.

Figure 22 conceptually illustrates an RSS scheme for assigning IPsec processing to processing cores.

Figure 23 conceptually illustrates different streams of the same SA being processed by different processing cores.

Figure 24 conceptually illustrates the flow of different SAs handled by different processing cores.

Figure 25 conceptually illustrates the flow of different SAs with the same port identifier handled by different processing cores.

Fig. 26 illustrates the generation of IPsec packets in which identifiers such as ports, IP addresses, and SPI are set for load balancing among a plurality of CPUs or processing cores.

Figure 27 conceptually illustrates a process of distributing IPsec workloads among multiple processor cores using flow identifiers.

Figure 28 conceptually illustrates a gateway that selects a particular path for each packet based on the QoS required by the packet.

Fig. 29 shows load balancing between active paths of the same QoS class.

Fig. 30 illustrates assigning packets with different QoS requirements to gateways of paths with different SAs.

Fig. 31 illustrates data flows within a gateway for performing QoS provisioning in a multipath IPsec environment.

Fig. 32 conceptually illustrates a process for performing QoS provisioning in a multipath IPsec environment.

FIG. 33 illustrates a computing device that functions as a host machine running virtualization software.

FIG. 34 conceptually illustrates a computer system with which some embodiments of the invention are implemented.

Detailed Description

In the following detailed description of the present invention, numerous details, examples, and embodiments of the present invention are set forth and described. It will be apparent, however, to one skilled in the art that the invention is not limited to the illustrated embodiments, and that the invention may be practiced without some of the specific details and examples discussed.

When a particular packet flow is delivered to the same destination across a network having multiple paths, the underlying physical network infrastructure (or underlying) typically relies on ECMP to select a path for the flow. This involves hashing of flow related data in the packet header, such as source and destination IP, source and destination ports, and five tuples of the protocol. However, when deploying an IPsec VPN on a network, ECMP is limited to hashing two tuples (external IP pairs) to select a path, as the inner packets are encrypted for IPsec ESP tunnel traffic. When the binary hash is constant (e.g., always the IP address of the corresponding TEP in the IPsec header), only one path can be selected per endpoint side. Thus, the following occurs: the best route is through a particular path, but the route selects another path.

Some embodiments provide a path aware IPsec gateway that selects a path at run-time for sending packets through a particular IPsec tunnel (or security association) based on path quality information collected from different paths of a probing network. In some embodiments, the collected information includes connectivity, latency, packet loss rate, jitter, and/or other metrics indicative of dynamic quality of the different paths. The selected path is indicated, for example, by a corresponding port identifier in the UDP header of the encapsulated packet. Thus, the path aware IPsec gateway detects path quality dynamics and selects the best path for the IPsec session at run time. The control of the selection and switching paths is driven by the IPsec VPN and is not route dependent.

Figure 1 conceptually illustrates a network 100 in which there are multiple paths between network endpoints such that IPsec can use multiple paths to transport data from source and destination endpoints in a VPN session. Network 100 interconnects network endpoints 102, 104, and 106, and network endpoints 102, 104, and 106 may refer to physical machines or virtual machines capable of initiating and/or receiving data packet traffic through network 100. Network 100 is implemented by the underlying physical infrastructure of wired and/or wireless communication media, routers, switches, and the like. Network 100 may include the internet and any direct connections between some of network endpoints 102, 104, and 106. Direct connection may refer to an interconnection between network endpoints within the same data center and/or the same physical device, or other proprietary network connections that interconnect endpoints 102 and 104 behind a gateway or firewall.

As shown, data traffic from network endpoint 102 may reach network endpoint 104 through any of a plurality of network paths 110, 112, 114, 116, and 118. Paths 110, 112, and 114 are paths of direct connections between network endpoints 102 and 104 without going through the internet, and network paths 116 and 118 are network paths through the internet.

Figure 2 conceptually illustrates the transmission of IPsec data from one endpoint to another endpoint over multiple paths. In this example, IPsec data is based on Security Association (SA) 200 (labeled "SA 1"), which is defined for the addresses of endpoints 102 and 104. As shown, IPsec tunnel 202 has been established for transmitting data 210 from endpoint 102 to endpoint 104. Multiple paths may be used to deliver packet traffic for SA 200, including paths 110, 112, 114, and 116.

A security association is a shared security attribute established between two network entities (e.g., between network endpoints 102 and 104, or between two gateways of two different data centers) to support secure communications. The SA may correspond to a unidirectional or simplex connection. The SA may include attributes such as encryption algorithms and modes, traffic encryption keys, parameters of the network data to be communicated over the connection. The SA is a form of contract between two network entities detailing how information is exchanged and protected between each other, including indicating how to encrypt/decrypt data. Each SA may include a commonly agreed key, one or more security protocols, and a Security Parameter Index (SPI) value identifying the SA, among other data.

The data 210 is the payload of an internal packet 220 having an internal IP address 222 and internal port information 224. The inner packet 220 is encrypted according to the SA 200 as IPsec authenticated data in an Encapsulating Security Payload (ESP) 230. Since the internal IP address 222 is encrypted with the internal packet 220 and cannot be used to route the packet, a new IP field 244 is appended to the ESP 230 to specify the external source and destination IP addresses. The external source and destination IP addresses are unencrypted and may be used to route packets. In this example, security association 200 ("SA 1") uses an external source IP address 10.10.10.1 and an external destination IP address 20.20.20.2 to route packets. In some embodiments, the source and/or destination IP addresses are used with a Security Parameter Index (SPI) of the packet to identify the SA. (SPI is the unique identifier of SA.)

The authenticated data 230 may be further encapsulated by a UDP header 242 into a User Datagram Protocol (UDP) -encapsulated packet 240. In some embodiments, if Network Address Translation (NAT) is enabled in the path used by the SA 200 and if NAT traversal (NAT-T) is used to deliver IPsec authenticated data 230, UDP encapsulation is performed. The UDP header 242 may specify a set of external source and destination ports (or UDP ports). In some embodiments, NAT-T is not enabled and packet 220 does not include UDP header 242.

In some embodiments, the gateway of the first data center may establish a VPN session to securely transport data to the second data center across multiple paths, either through a direct connection or through the internet. Fig. 3 conceptually illustrates a VPN session 300 established to securely transport or migrate data from a first data center 310 to a second data center 320. The data center 310 has a gateway 312 for managing data center traffic with external networks, including any direct connection to the second data center 320 or the internet. VPN session 300 may use multiple IPsec tunnels and establish multiple SAs, and gateway 312 manages VPN session 300 and multiple IPsec tunnels. Gateway 312 may use the VPN session to transport IPsec data on behalf of the network endpoint of the first data center. In some embodiments, gateway 312 may also use multiple addresses of local endpoints to establish multiple SA or IPsec tunnels. For VPN session 300, gateway 312 of the first data center is a VPN client and gateway 322 of the second data center is a VPN server. The path connecting the two data centers may support one or more active uplinks from the VPN client to the VPN server.

The gateway 312 uses the path quality information to identify the best performing or most appropriate path rather than relying on simple ECMP to perform path selection based on fixed external IP addresses. In some embodiments, gateway 312 obtains path quality information by sending probe messages to different paths and receiving responses to the probe messages to collect metrics.

Fig. 4 conceptually illustrates that a gateway gathers path quality information in order to perform path selection for transmitting IPsec data. Gateway 312 of data center 310 periodically sends out probe messages to various paths that may reach data center 320. These probe messages may be used to obtain dynamic or real-time measurements or metrics regarding connectivity, latency, packet loss rate, jitter, etc. Is provided. For example, the gateway may send a probe message to ping the path to measure the delay of the path or to determine the liveness of the path. Gateway 312 tabulates the performance metrics for the different paths and periodically updates the metrics. The gateway may also maintain a pool of paths with performance metrics above a certain threshold that may be used to send IPsec data.

In this example, gateway 312 sends the probe message to the path identified by the source IP address and destination IP address pair 10.10.10.1 and 20.20.20.2 defining security association "SA 1". Gateway 312 then uses the metrics obtained for those paths to identify the best execution path for the given security association. Gateway 312 may indicate the selected path to the routing layer. In some embodiments, different paths are associated with different source and/or destination ports, and gateway 312 indicates the selected path (e.g., 242) in the UDP header by setting the source and/or destination ports to a value corresponding to the selected path. Probing paths to obtain path performance metrics is also described in commonly owned U.S. patent application Ser. No.17/016,596, entitled "PATH SELECTION FOR DATA PACKETS ENCRYPTED BASED ON AN IPSEC PROTOCOL," filed 9/10/2020. U.S. patent application Ser. No.17/016,596 is incorporated herein by reference in its entirety.

In some embodiments, gateway 312 reserves multiple active paths for a given security association and performs load balancing by distributing outgoing IPsec packets for the given security association among the multiple active paths. Multiple active paths may concurrently transmit packets for security association. In some embodiments, the gateway identifies any paths that may be used to send packets to VPN peers as active paths for load balancing. In some embodiments, the gateway identifies paths with performance metrics above a particular threshold as active paths or best performing paths for load balancing. In the example of fig. 4, paths with performance metrics higher than 80 are identified by gateway 312 as active paths, so path 1 (metric 100), path 2 (metric 83), path 6 (metric 90), and path 9 (metric 89) are used for load balancing multiple active paths for security association SA1, while other paths are not used for load balancing.

Figure 5 conceptually illustrates load balancing across multiple active paths for security association SA 1. As shown, gateway 312 dispatches packets for delivery to gateway 322, which is a VPN peer. The load balancer module 500 of the gateway allocates assigned packets among the paths identified as active paths or best performing paths for sending IPsec packets for SA1, and these active paths (path 1, path 2, path 6, and path 9 in the example) may be concurrently active when delivering packets to the gateway 322.

The load balancer 500 may select a path among a plurality of active paths based on a hash value derived from a particular field of the internal payload (e.g., port number, source IP address, destination IP address, protocol identifier, etc.). The hash value may be calculated based on the 5-tuple included in the inner L3/L4 header. The 5-tuple may include a source IP address, a destination IP address, a source port identifier, a destination port identifier, and a protocol identifier. In some of these embodiments, the gateway may direct the load balancer to select a particular path by setting a particular field of the packet to a value corresponding to the particular path.

It should be noted that while certain embodiments are described with respect to communication between gateways, these techniques may similarly be applicable to communication between any suitable computing machines (e.g., virtual computing instances, physical computing devices, etc.).

In some embodiments, when multiple tunnels in different uplinks (e.g., one uplink over a direct connection and one uplink over the internet) have the same reachability (i.e., both can be used to reach the VPN server from the VPN client), the information generated by the path probing is used to select the best path among the different tunnels in the different uplinks. Different uplinks may be used to transmit data for different security associations.

Figure 6 conceptually illustrates a VPN client using multiple paths in multiple uplinks or tunnels to send IPsec data across network 100 to a VPN server. As shown, gateway 312 has established VPN session 600 as a VPN client with gateway 322 as a VPN server. The VPN session uses two security associations SA1 and SA2 to send IPsec data across the network 100. The security association SA1 has multiple paths with a source IP of 10.10.10.1 and a destination IP of 20.20.20.2. The security association SA2 has several paths with a source IP of 10.10.22.2 and a destination IP of 20.20.20.2. SA1 is used to encrypt and authenticate IPsec data for VPN tunnel (or uplink) 630, and SA2 is used to encrypt and authenticate IPsec data for VPN tunnel (or uplink) 640. In particular, any flow transmitted from an endpoint in a first data center to an endpoint in a second data center may be encrypted at the first data center using SA1 and sent over VPN tunnel 630 or encrypted using SA2 and sent over VPN tunnel 640.

In some embodiments, gateway 312 is configured with a Virtual Tunnel Interface (VTI) to handle data traffic going into and out of VPN tunnels. The VTI is a logical routing layer interface configured at the end of a VPN tunnel to support a route-based VPN and append an IPsec profile to the end of the tunnel. Outbound traffic from the VTI is encrypted and sent to the VPN peer, and the SA associated with the tunnel decrypts inbound traffic to the VTI.

In some embodiments, a single VTI is configured at the source gateway for bundles of multiple different SAs. The destination gateway is similarly configured with a single corresponding VTI for bundles of different SAs. Each SA has a different SPI value associated with it, and tuples of header values of packets tunneled across different VPNs may be hashed to different CPUs at the destination gateway for processing.

Because there is a single VTI interface, routes are installed for the single VTI interface, avoiding ECMP-based load distribution asymmetric routes due to multiple interfaces for multiple SAs. All packets routed through the VTI distribute the load among the SA bundles set for the VTI. The load distribution of packets on the SA may be done using a simple hash of the 5 tuples in the packet or using an algorithm agreed between the peer and the gateway.

Figure 7 conceptually illustrates one single VTI associated with different SAs for IPsec encryption. As shown, gateway 312 implements VTI 710 at its application layer. VTI 710 receives data traffic to be encrypted by SA1 and data traffic to be encrypted by SA 2. IPsec traffic for SA1 uses a first VPN tunnel 630 and IPsec traffic for SA2 uses a second VPN tunnel 640.

In some embodiments, multiple VTIs may be configured at the source gateway, with each VTI being associated with a different SA for encryption. The destination gateway is similarly configured with a plurality of corresponding VTIs, each associated with the same corresponding different SA for decryption. In this way, the source gateway and the destination gateway implement multiple VPN tunnels, each tunnel corresponding to a different VTI and each tunnel being associated with a different SA. Each SA has a different SPI value associated with it, and tuples of header values of packets tunneled across different VPNs may be hashed to different CPUs at the destination gateway for processing.

In some embodiments, from an application layer (e.g., L7 of OSI) perspective, the gateway for VPN traffic implements a single collaboration (metering) interface or device (or bundled VTI) for VPN session 600. However, from the perspective of the routing layer (L3 of OSI), the gateway implements multiple VTIs corresponding to multiple VPN tunnels or SAs. A single cooperating interface or bound VTI logically combines different VTI tunnels into one IPsec VPN tunnel. VPN traffic may be forwarded to the remote gateway and upper layer protocol traffic may proceed uninterrupted as long as at least one of the VPN tunnels is available for the cooperating interfaces. In some embodiments, all information about the different paths and VTIs is transparent to the administrator. In some embodiments, different VTIs are visible to an administrator of the data center, allowing different firewall or MTU configurations to be applied to different tunnels, providing greater flexibility for the administrator. Commonly owned U.S. patent application Ser. No.16/514,647, entitled "USING VTI TEAMING TO ACHIEVE LOAD BALANCE AND REDUNDANCY," filed on 7/17/2019, further describes the collaboration of multiple VTIs into one bundled VTI. U.S. patent application Ser. No.16/514,647 was disclosed as U.S. patent publication No.2021/0021523, incorporated herein by reference in its entirety, at 21 of 2021.

Fig. 8 conceptually illustrates a plurality of VTIs associated with different SAs for encryption logically combined into a bound VTI. As shown, gateway 312 implements a bundled VTI 810 at its application layer. The bound VTI 810 has two L3 slave VTIs: a first slave VTI 830 using SA1 to VPN tunnel 630 and a second slave VTI 840 using SA2 to VPN tunnel 640. Similarly, gateway 322 implements a bundled VTI 815 having a first slave VTI 835 for receiving IPsec data from VPN tunnel 630 and a second slave VTI 845 for receiving IPsec data from VPN tunnel 640.

In some embodiments, gateway 312 aggregates path information for paths used by both VPN tunnels 620 and 630 (as well as any other paths used by VPN session 600). In particular, the gateway sends probe messages to paths of different VPN tunnels and different SAs to obtain dynamic quality of those different paths. For each packet to be delivered using VPN session 600, gateway 312 selects the best path from the paths of the different VPN tunnels based on the aggregated path information.

Fig. 9A-B illustrate a gateway using aggregated path information to select a best path from a plurality of different VPN tunnels. Gateway 312 receives application data 900 at virtual interface 910 (e.g., single VTI 710 or bound VTI 810) to be sent to gateway 312 through VPN tunnel 630 or VPN tunnel 640. In this example, gateway 312 has sent probe messages to the paths of VPN tunnel 630 and VPN tunnel 640 and obtained a set of path quality metrics. Gateway 312 maintains a pool of paths from both VPN tunnels by identifying paths that have performance metrics above a particular threshold. The path performance metrics may be updated in real-time so that gateway 312 selects the best path based on the dynamic, real-time information.

Fig. 9A illustrates gateway 312 selecting path 910 corresponding to source port 5001, which is the path of VPN tunnel 630 for SA 1. Path 910 has the best performance metric among all paths used by SA1 and SA 2. Since this path is one of paths used by the VPN tunnel 630, the gateway 312 encrypts the received application data 900 according to SA1 and encapsulates the encrypted data with a UDP header. The UDP header indicates the port number (source port 5001) corresponding to the selected path 910. The routing layer in turn performs ECMP and hashes the UDP header to use path 910. In some embodiments, path 910 is one of several active paths that may be used to transport packets for SA1 or VPN tunnel 630.

Fig. 9B illustrates gateway 312 selecting path 920 corresponding to source port 6003, which is the path of VPN tunnel 640 for SA 2. Path 920 has the best performance metric among all paths used by SA1 and SA 2. Since path 920 is one of the paths used by VPN tunnel 640, gateway 312 encrypts received application data 900 according to SA2 and encapsulates the encrypted data with a UDP header. The UDP header indicates the port number (source port 6003) corresponding to the selected path 920. The routing layer in turn performs ECMP and hashes the UDP header to use path 920. In some embodiments, path 920 is one of several active paths that may be used to transport packets for SA2 or VPN tunnel 640.

For some embodiments, fig. 10 conceptually illustrates a process 1000 for transmitting IPsec data using multiple paths in multiple different SAs. In some embodiments, one or more processing units (e.g., processors) of a computing device implementing gateway 312 perform process 1000 by executing instructions stored in a computer-readable medium.

In some embodiments, process 1000 begins when a gateway negotiates (at 1010) a first (VPN) tunnel implementing a first SA and a second (VPN) tunnel implementing a second SA. The first and second SAs and tunnels are established as part of a VPN session, where the gateway is a VPN client and the remote gateway is a VPN server. One tunnel may include a path through the internet while the other tunnel does not include a path through the internet, or simply includes a direct connection within or between data centers.

The gateway collects (at 1015) metrics of one or more paths of the first tunnel and one or more paths of the second tunnel. In some embodiments, the gateway sends a probe message and receives a response to the probe message. Metrics collected for one or more paths of the first and second tunnels are determined based on the received responses to the probe messages. In some embodiments, the metrics of the path include at least one of connectivity, latency, packet loss rate, jitter of the path.

The gateway receives (at 1020) data to be transmitted from the first network endpoint to the second network endpoint. In some embodiments, the first network endpoint is hosted by a first data center and the second network endpoint is hosted by a second data center. The gateway is an edge device (apple) of the first data center. The VPN server is a gateway or edge device of the second data center. In some embodiments, data is received at a single routing layer interface (or VTI) for encryption and transmission in a first tunnel using a first SA and in a second tunnel using a second SA. In some embodiments, data is received from an application at a bound interface at an application layer, and the bound interface logically combines a first routing layer interface for encrypting and encapsulating the received data for transmission in a first tunnel using a first SA and a second routing layer interface for encrypting and encapsulating the received data for transmission in a second tunnel using a second SA. The gateway selects (at 1025) a path based on the collected metrics of the paths of the first and second tunnels. In some embodiments, the metrics of the collected paths are used to identify a pool of best performing paths, and the gateway selects a path from the pool of best performing paths for load balancing.

The gateway determines (at 1030) whether the selected path belongs to the first tunnel or the second tunnel (or another tunnel established for the VPN session). If the selected path belongs to the first tunnel, the process proceeds to 1040. If the selected path belongs to the second tunnel, then the process proceeds to 1060.

At 1040, the gateway encrypts the received data as an encrypted payload of the first SA. The gateway encapsulates (at 1045) the encrypted payload by appending (i) a first source address identifying the first tunnel and (ii) a first source port identifying the selected path. In some embodiments, the encapsulating includes storing a UDP header for the first source port. The gateway transmits (at 1050) the encapsulated encrypted payload in a first tunnel. The process may return to 1015 so that the gateway continues to collect path performance metrics and select a path for delivering subsequent IPsec data.

At 1060 (when the selected path belongs to the second tunnel), the gateway encrypts the received data into an encrypted payload of the second SA. The gateway encapsulates (at 1065) the encrypted payload by appending (i) a second source address identifying the second tunnel and (ii) a second source port identifying the selected path. In some embodiments, the encapsulating includes storing a UDP header for the second source port. The gateway transmits (at 1070) the encapsulated encrypted payload in a second tunnel. The process may return to 1015 so that the gateway continues to collect path performance metrics and select a path for delivering subsequent IPsec data.

Fig. 11 illustrates a block diagram of a system 1100, the system 1100 probing multiple paths to find the best path and updating the IP address of the SA to use the best path. In some embodiments, system 1100 is implemented in a gateway or edge device of a data center, such as gateway 312. The system 1100 may be implemented by a bare metal computing device or a host machine running virtualization software that operates a gateway in one or more virtual machines. In some embodiments, system 1100 represents a VPN control plane.

As shown, system 1100 implements IKE control stack 1110, probe manager 1120, path analyzer 1130, traffic analyzer 1140, and IPsec tunnel data path 1150. In some embodiments, modules 1110-1140 are sub-modules of a VPN control plane, while module 1150 represents a VPN data plane. In some embodiments, modules 1110-1150 are modules of software instructions that are executed by one or more processing units (e.g., processors) of a computing device. In some embodiments, modules 1110-1150 are modules of hardware circuitry implemented by one or more Integrated Circuits (ICs) of an electronic device. Although modules 1110, 1120, 1130, 1140, and 1150 are shown as separate modules, some of these modules may be combined into a single module.

IKE control stack 1110 controls the operation of IPsec including establishing and maintaining VPN sessions and SAs. The IKE control stack provides the necessary authentication key data to IPsec tunnel data path 1150 for authentication and encryption of the payload. The IKE control stack 1110 also identifies paths determined to be available to reach the VPN server and maps those paths to UDP port identifiers. A list of available paths or identifiers of UDP port identifiers are provided to the probe manager 1120 to probe those paths.

The probe manager 1120 periodically probes all available paths to compute metrics for the different paths. In some embodiments, the probe manager 1120 is configured with the number of probe packets per path. The path metrics are provided to a path analyzer 1130. When the probe manager 1120 generates a packet to probe a path and calculates and updates a path metric according to the probe result, the path analyzer 1130 identifies the best path among all paths based on the path metric.

The path analyzer 1130 drives the selection of the best path from among the different paths across the different SAs. The path analyzer 1130 may also consider link throughput, runtime, traffic load, liveness, route optimization, RTT, load balancing, and path MTU when determining a new path. The path analyzer 1130 also uses input from the traffic analyzer 1140 to influence the path change decisions based on traffic characteristics. Once the best path selection is made, IKE control stack 1110 provides the corresponding SA information and UDP information to IPsec tunnel data path 1150. In some embodiments, the path analyzer 1130 may trigger a path switch based on traffic characteristics (provided by the traffic analyzer 1140) or QoS requirements.

IPsec tunnel data path 1150 performs the operations of the various VPN tunnels and provides traffic statistics of the tunnels to traffic analyzer 1140. In some embodiments, IPsec tunnel data path 1150 may include various VPN data plane modules. IPsec tunnel data path 1150 also performs encryption and authentication of payloads based on SA information provided by IKE control stack 1110. The IPsec tunnel data path also encapsulates the encrypted payload in a UDP header that includes a UDP port number to identify the best path selected.

IPsec tunnel data path 1150 receives application data at routing interface VTI 910 when an application uses a gateway to send certain application data in VPN session 600. The application data is wrapped as an internal packet 1165. The encryption module 1170 encrypts the inner packet into an IPsec encrypted packet 1175 according to encryption parameters of the SA information (specified by the IKE control stack 1110 to select either SA1 or SA 2). The encryption module 1170 also appends other IPsec related fields (e.g., ESP header, ESP trailer, ESP authentication, new IP, etc.) based on the SA information. Encapsulation module 1180 encapsulates IPsec encrypted packet 1175 into UDP encapsulated packet 1185 having a UDP encapsulation header, which may include a UDP port number indicating the selected path. Data plane routing module 1190 then sends UDP encapsulated packet 1185.

In some embodiments, a Security Association (SA) may be configured to use different paths by changing the source address and the destination address. When the gateway establishes an SA with a VPN server for a VPN session to send IPsec data from a first site to a second site (e.g., from data center 310 to data center 320), the specific source and destination addresses are used by the SA to route IPsec packets (SPI is used to identify the SA). In addition, the gateway associates each path that can be used to reach the VPN server with a different source address and destination address pair. In some embodiments, when the information generated by the path probing is used to select the best performing path, the gateway may indicate the selected path by informing the VPN server SA that the source and destination address pair has changed to the address pair associated with the selected path.

In some embodiments, the VPN client and VPN server are each configured with a list of a plurality of local endpoint addresses. These local endpoints may be routed through a single uplink or multiple uplinks. The VPN client exchanges its list of local endpoint addresses with the list of local endpoint addresses of the VPN server and pairs the address of the VPN client and the address of the VPN server are used as source and destination addresses to identify possible paths for the SA to be probed. For example, if a first site that is a VPN client has n IP addresses and a second site B that is a VPN server has m IP addresses, the total number of paths to be probed is n×m. As a further example, if a first station has n links and each link has m IP addresses and a second station has p links and each link has q IP addresses, then a total of (n x m) (p x q) paths will be probed and made available for selection as the best path. Thus, the gateway maintains a dynamic pool of local endpoints or loopback ips to have ECMP entropy on the IPSec network path for reaching VPN peers. Each path in the pool is also periodically monitored for its quality (e.g., latency, drop count).

Fig. 12 illustrates a VPN session in which the SA may be configured to use different paths by changing the source and destination addresses. As shown, for a VPN session, gateway 312 has established SA 1200 as a VPN client and gateway 322 acts as a VPN server. SA 1200 is currently configured with a source IP 10.10.10.1 and a destination IP 20.20.20.2, hashed to a value corresponding to the path labeled "X1-Y1" from gateway 312 to gateway 322.

In addition to path "X1-Y1," there are other paths that may be used by VPN client 312 to send IPsec traffic to VPN server 322 but are not currently used by SA 1200. These different paths correspond to different pairs of local endpoint addresses used by gateway 312 and gateway 322. In this example, gateway 312 is configured with local addresses 10.10.10.1 (labeled "X1"), 10.10.11.1 (labeled "X2"), 10.10.12.1 (labeled "X3"), and 10.10.13.1 (labeled "X4"), while gateway 322 is configured with local addresses 20.20.20.2 (labeled "Y1"), 20.20.21.2 (labeled "Y2"), 20.20.22.2 (labeled "Y3"), and 20.20.23.2 (labeled "Y4"). Each pairing of the local address of gateway 312 (as the source address) and the local address of gateway 322 (as the destination address) hashes to a value corresponding to a different path (labeled "X1-Y1", "X1-Y2", "X4-Y4", etc.). In some embodiments, some of the endpoint addresses may be loopback IP addresses that are introduced to enhance ECMP entropy.

Gateway 312 also issues probe messages to obtain path performance metrics for the different paths. In some embodiments, the gateway uses liveness detection to check the reachability of the available network addresses. These same messages are used to obtain path performance metrics for different paths. In some embodiments, the performance metrics of the paths may include at least one of Round Trip Time (RTT), link throughput/bandwidth, traffic load, load balancing, path Maximum Transmission Unit (MTU), path optimization, packet loss per path, and the like. For different pairs of source and destination addresses in the path matrix 1210, metrics for different paths are aggregated and tabulated, with each entry corresponding to a path. The path matrix 1210 may also be referred to as a probe matrix, as entries of the matrix 1210 are populated and updated by metrics determined by probing different paths. The matrix may be maintained by gateway 312 or in data center 310 as a VPN site.

The gateway may select the best path based on the contents of the path matrix 1210 and then modify the source and destination addresses of the SA 1200 to correspond to the selected best path. In some embodiments, gateway 312 communicates with VPN server 322 using IKEv2 mobility and multi-homing protocol (MOBIKE) to change the address of the SA without disrupting the operation of the SA so that the SA does not need to be re-established due to the change in address. Before changing the IP address of the SA using MOBIKE, both sides of the SA exchange their respective lists of local endpoint addresses using MOBIKE. After exchanging the list of local endpoint addresses using MOBIKE, the peers/sides of the SA know the available paths based on the IP addresses exchanged using MOBIKE.

In some embodiments, the probe message sent to collect path performance metrics is a MOBIKE reachability/liveness probe. This allows the probe mechanism to interoperate with any IPSec peer supporting MOBIKE. In MOBIKE, these probe messages are used for liveness checking of the path. In some embodiments, the probe messages are used at regular intervals. In some embodiments, bi-directional latency information based on these liveness probes and drop counts per path are maintained by the gateway.

Gateway 312 may perform path or address selection based on policies that apply weights to different paths according to predefined settings. The weights applied to particular paths may also be based on certain traffic characteristics or quality of service (QoS) requirements of the VPN session. For example, real-time traffic may require a higher level of bandwidth and paths with more bandwidth/throughput and faster RTT may be selected by address/path selection policies.

Fig. 13A-E conceptually illustrate a gateway 312 that uses the MOBIKE protocol to change the source and destination IP addresses of the SA in order to select the best path. Fig. 13A shows VPN client gateway 312 sending its list 1310 of local addresses to VPN server gateway 322 and VPN server gateway 322 sending its list 1315 of local addresses to VPN client gateway 312. Based on the exchange of the list of local addresses, VPN client 312 generates a matrix 1320 (4 x4 in this example) whose entries correspond to paths defined by the pairing of addresses from list 1310 and list 1315. Fig. 13B shows gateway 312 sending probes to different paths and populating corresponding entries in matrix 1320. Fig. 13C shows gateway 312 using path (X4-Y2) with the best performance metric (45) according to matrix 1320. The figure also shows the contents of IPsec packet 1310 using path X4-Y2. Specifically, the new IP field 1315 of the packet specifies that the source IP address is endpoint X4 (10.10.13.1) and the destination IP address is endpoint Y2 (20.20.21.2).

As the gateway 312 continues to probe for paths and update the matrix 1320, the gateway monitors the matrix 1320. Fig. 13D shows gateway 1320 monitoring matrix 1320 to detect that another path (X2-Y3) now has the best performance metric (59). While still using path X4-Y2, gateway 312 communicates with VPN server gateway 322 to change to use the new best performing path X2-Y3. Specifically, the VPN client gateway 312 changes the source IP and the destination IP of the SA 1200 from (X4, Y2) to (X2, Y3) using the MOBIKE protocol. Fig. 13E shows VPN client gateway 312 using paths X2-Y3 to send IPsec data to VPN server gateway 322 using SA 1200. Even if the IP address of the SA 1200 has changed, the SA is not interrupted and does not need to be re-established. The figure also shows the contents of IPsec packet 1320 using paths X2-Y3. Specifically, the new IP field 1325 of the packet specifies that the source IP address is endpoint X2 (10.10.11.1) and the destination IP address is endpoint Y3 (20.20.22.2).

For some embodiments, fig. 14 conceptually illustrates a process 1400 of transmitting IPsec data using multiple paths by changing the IP address of an SA. In particular, the gateway of the first site performs a process 1400 for sending IPsec data to the second site. In some embodiments, one or more processing units (e.g., processors) of a computing device implementing gateway 312 perform process 1400 by executing instructions stored in a computer-readable medium.

Process 1400 begins when the gateway establishes (at 1410) a Security Association (SA) for transmitting encrypted payloads from a first site to a second site in a VPN session. Thus, the gateway of the first site is a VPN client of the VPN session and the gateway of the second site is a VPN server of the VPN session. In some embodiments, only one path at a time in one uplink is active for a VPN session, and only one active path of the multiple paths has the best path metric.

The gateway (of the first site) exchanges (at 1420) the first list of endpoint addresses of the first site for a second list of endpoint addresses of the second site for VPN sessions with the gateway of the second site. The gateway in turn maintains a pool of multiple local endpoint addresses from both ends of the VPN session so as to have underlying ECMP entropy. The gateway identifies (at 1430) a plurality of paths between the first site and the second site for the VPN session. Each path is defined by a pair of an endpoint address in the first site and an endpoint address in the second site.

The gateway obtains (at 1440) metrics for the plurality of identified paths by, for example, sending a probe message. The metric of the path may be determined based on at least one of connectivity, latency, packet loss rate, jitter of the path. Metrics of the paths may also include at least one of Round Trip Time (RTT), link throughput/bandwidth, traffic load, load balancing, path Maximum Transmission Unit (MTU), path optimization, packet loss per path, and the like. In some embodiments, the gateway sends a probe message and receives a response to the probe message. The obtained metrics for the identified paths are determined based on the received responses to the probe messages. In some embodiments, the obtained metrics are stored in a path matrix (e.g., path matrix 1210) specified based on the first and second lists of endpoint addresses.

The gateway selects (at 1450) a path from the plurality of paths based on the obtained metrics. The selected path is defined by a first endpoint address in the first site and a second endpoint address in the second site and is the best execution path among the plurality of paths. The first endpoint address is identified in a first list of endpoint addresses and the second endpoint address is identified in a second list of endpoint addresses. The gateway then determines (at 1455) whether the selected path is the path currently used by the SA. If so, then the process proceeds to 1475. If the selected path is not the path currently used by the SA, then the process proceeds to 1460.

The gateway sends (at 1460) a message from the first station to the second station to update the SA to switch from using the original path to using the selected path. The message indicates the first and second endpoint addresses. In some embodiments, messages sent to the second station using the MOB IKE protocol to update the SA and to update the SA to use the selected path do not interrupt or re-establish the SA. The gateway encrypts the payload according to the updated SA (at 1470). The process then proceeds to 1480.

At 1475, the gateway encrypts the payload according to the SA without updating the address indicating the selected path. The gateway transmits (at 1480) a packet including the encrypted payload. ECMP routing is to be performed based on the first and second endpoint addresses defining the selected path. The external (tunnel header) address of the packet is updated based on the first and second endpoint addresses, while the address and other traffic selectors used to route the packet within the VPN tunnel remain unchanged. The process may return to 1440 to continue probing paths and obtaining path metrics.

Fig. 15 illustrates a block diagram of a system 1500 that probes multiple paths to find the best path and updates the IP address of the SA to use the best path. In some embodiments, system 1500 is implemented in a gateway or edge device of a data center, such as gateway 312. In some embodiments, system 1500 represents a VPN control plane. The system 1500 may be implemented by a bare metal computing device or host machine running virtualization software operating a gateway in one or more virtual machines.

As shown, system 1500 implements IKE control stack 1510, probe manager 1520, path analyzer 1530, traffic analyzer 1540, and IPsec tunnel manager 1550. In some embodiments, modules 1510-1540 are sub-modules of a VPN control plane, while module 1550 represents a VPN data plane. In some embodiments, modules 1510-1550 are modules of software instructions executed by one or more processing units (e.g., processors) of a computing device. In some embodiments, modules 1510-1550 are modules of hardware circuitry implemented by one or more Integrated Circuits (ICs) of an electronic device. Although modules 1510, 1520, 1530, 1540, and 1550 are shown as separate modules, some of these modules may be combined into a single module.

IKE control stack 1510 controls the operation of IPsec, including the establishment and maintenance of VPN sessions and SAs. IKE control stack 1510 also includes a MOBIKE extension that drives communications with VPN servers in the MOBIKE protocol. The IKE control stack provides the necessary authentication key data to IPsec tunnel manager 1550 to authenticate and encrypt the payload. IKE control stack 1510 also identifies a list of available local endpoint addresses and uses its MOBIKE extension 1515 to communicate those addresses to the VPN server. IKE control stack 1510 receives a list of exchanged endpoint addresses from the VPN server. The list of endpoint addresses is provided to probe manager 1520 to probe those paths. The MOBIKE extension 1515 is also used to communicate with a VPN server to change the IP address of the SA when the path analyzer 1530 selects a new path.

The probe manager 1520 is initialized based on endpoint address information exchanged between the VPN client and the VPN server using MOBIKE. The probe manager 1520 periodically probes all available paths and populates a path matrix (e.g., path matrix 1210). Probe manager 1520 is configured with the number of probe packets per path and probe timeout in order to re-trigger path matrix computation. Then, the probe manager 1520 generates a specified number of probe packets for each path. When the probe manager 1520 generates a packet to probe a path and populates a path matrix according to the probe, the path analyzer 1530 identifies the best path among all paths by using the path matrix. The path analyzer 1530 then triggers a MOBIKE message to update the SA.

The path analyzer 1530 drives selecting an endpoint from among a plurality of local endpoints configured for the IPsec session. The path analyzer 1530 may also consider link throughput, runtime, traffic load, liveness, route optimization, RTT, load balancing, and path MTU when determining a new path. The path analyzer 1530 also uses input from the traffic analyzer 1540 to influence the path change decisions based on the traffic characteristics. Once the best path is selected, the IKE MOBIKE extension 1515 in IKE control stack 1510 is used to switch the VPN session (or SA) to a different endpoint address corresponding to the selected best path so that IPsec tunnel data path 1550 can begin using the newly selected best path. In some embodiments, the path analyzer 1530 may trigger a path switch based on traffic characteristics (provided by the traffic analyzer 1540) or QoS requirements.

IPsec tunnel datapath 1550 performs the operations of the various VPN tunnels, including encryption and authentication of the payloads based on the SAs maintained and updated by IKE control stack 1510. For some embodiments, IPsec tunnel data path 1550 represents a VPN data plane. The IPsec tunnel also provides traffic statistics about the VPN tunnel to traffic analyzer 1540. In some embodiments, IPsec tunnel data path 1550 may trigger path switching if multiple local endpoints are configured with different uplinks (such as a direct connection and an internet) and have the same reachability.

When an application uses a gateway to send certain application data in a VPN session, IPsec tunnel data path 1550 receives the application data at routing interface VTI 1560. The application data is packaged as an internal packet 1565. Encryption module 1570 encrypts the inner packet into IPsec encrypted packet 1575 according to encryption parameters of SA 1200 (specified in the SA information provided by IKE control stack 1510). The encryption module 1570 also appends other IPsec related fields (e.g., ESP header, ESP trailer, ESP authentication, new IP, etc.) based on the SA information. Encapsulation module 1580 encapsulates IPsec encrypted packet 1575 with an external IP corresponding to the selected endpoint address. The data plane routing module 1590 then sends the encapsulated packet 1585.

In some embodiments, the gateway directs IPsec VPN traffic over multiple paths made available by multiple active uplink interfaces and performs load balancing over the multiple paths. The gateway also provides failover or redundancy between the multiple uplink interfaces so that if one of the uplinks fails, traffic will drop back to the other uplink without further synchronization or session renegotiation overhead.

Figure 16 conceptually illustrates a gateway that uses multiple active uplink interfaces to send IPsec data to its VPN peers. As shown, gateway 312 has established a VPN session 1600 with gateway 322 for transmitting IPsec data from data center 310 to data center 320. The gateway 312 has a first interface 1612 to a first uplink 1610 that allows the gateway to access a path through a direct connection between two data centers. Gateway 322 has a second interface 1622 to a second uplink 1620, which allows the gateway to access paths through the internet.

Both interfaces 1612 and 1622 are used to transmit IPsec packets encrypted according to security association SA 1. The same SA information is used for multiple network paths following different uplink interfaces. For IPsec traffic, gateway 312 will load balance VPN traffic on all available or active network paths while maintaining the same SA. Thus, an application using VPN session 1600 may use only one single virtual interface (VTI) for the SA while load balancing across multiple paths in multiple uplinks in the physical network floor. Thus, IKE control packets can still use a single interface to send the packet. However, in the data plane, ESP packets may be sent over multiple interfaces. In the example of fig. 16, there is one direct connection uplink and one internet uplink. In some embodiments, the network topology may provide multiple direct connection uplinks and/or multiple internet uplinks. In some embodiments, the network topology may provide a single uplink, and the gateway may maintain multiple paths through the single uplink and provide load balancing across the paths.

By maintaining a single VPN session across multiple uplinks, there will be no asymmetric routing problem because there is only one single VTI routing interface for the VPN session. Furthermore, a single VTI for multiple uplink VPN sessions allows a stateful firewall to operate without further changes. In some embodiments where the software stack of the gateway includes a routing layer and an IPsec layer, the routing layer of the gateway sees only one SA, and therefore load balancing does not select from multiple SAs. Load balancing across multiple uplink paths is managed by the IPsec layer, which tracks all network paths on a single VPN tunnel. Load balancing single VPN tunnel traffic over multiple different uplinks or external IPs can further improve RSS throughput and performance. Thus, multiple CPU cores may be selected to handle different traffic flows. It also makes more efficient use of available network bandwidth by propagating IPsec traffic over multiple paths and helps overcome traffic control in some cloud networks. Maintaining IPsec settings over a single link can be quite simple. But as the number of redundant or additional links increases, the number of SAs that must be negotiated and maintained also increases. Maintaining multiple simultaneous IPsec connections to ensure reliable and secure communications can present significant network overhead and management challenges. By maintaining a single VPN session across multiple links, only a single IKE SA, a single IPsec SA, and a single VTI need be maintained, thereby reducing signaling and configuration overhead, enabling optimal network control.

In some embodiments, gateway 312 implements path aware IPsec by detecting path quality dynamics and selecting the best execution path for a VPN session at run-time. Gateway 312 is configured to send traffic using all of the best paths available. The selected best path is identified as a pool of available best paths for the data plane. Selecting the paths included in the path pool may include paths for both the first uplink interface 1612 and the second uplink interface 1622. Gateway 312 may dynamically add and/or remove paths to and/or from the path pool based on real-time path performance metrics collected from path probing. Gateway 312 in turn performs load balancing by selecting paths from the path pool to transport IPsec packets. In some embodiments, control of the selection and switching paths is driven by an IPsec VPN and is not route dependent.

Figure 17 conceptually illustrates a path pool including paths of several different uplink interfaces. As shown, a gateway (e.g., gateway 312) has at least three uplink interfaces A, B and C. Each interface allows the gateway to access several paths, in particular uplink interface a may access paths A1-A7, uplink interface B may access paths B1-B7, and uplink C may access paths C1-C7.

The gateway obtains path quality dynamics for the paths of the three uplink interfaces, e.g., by probing the paths to obtain performance metrics for the paths. In the illustrated example, the performance metric for path A1 is 74, the performance metric for path A2 is 101, the performance metric for path B1 is 93, and so on. Based on these performance metrics, the gateway identifies several best execution paths as part of path pool 1710. In the example of fig. 17, paths A1, A2, A5, B1, B3, C2, C3, and C4 are identified as best execution paths and are included in the path pool 1710. In an embodiment, paths with performance metrics above a certain threshold (70 in the example) are identified and included in the path pool 1710. Thus, path pool 1710 provides paths for multiple different uplink interfaces, and the gateway can use path pool 1710 to load balance the transmission of IPsec packets on the different uplink interfaces.

In some embodiments, when one uplink interface is down, the gateway removes all paths using the failed interface from the data plane by removing the path of the interface from the path pool 1710. In other words, the path of the failed interface will not be used for transmission. Figure 18 conceptually illustrates removing a path from a path pool when an uplink interface fails. In the example of fig. 18, the gateway detects that interface B has failed. The gateway in turn removes all paths using interface B, specifically paths B1 and B3, from path pool 1710 and the path that did not use interface B will be used to transmit the IPsec packet. Thus, if any of the uplink interfaces were down, the VPN session would continue using the next available interface.

In some embodiments, path selection for load balancing is weighted based on the bandwidths of the different interfaces. For example, a direct-connect interface may have more network paths in the path pool than an interface for the internet because the direct connection has a higher bandwidth than the internet. Figure 19 conceptually illustrates identifying network paths included in a path pool based on bandwidth. The gateway may dynamically measure the bandwidth of the different interfaces or rely on predefined bandwidth parameters. In the example of fig. 19, the interface for uplink a has a bandwidth metric of 1000, the interface for uplink B has a bandwidth metric of 100, and the interface for uplink C has a bandwidth metric of 500. Based on the bandwidth metrics of interfaces A, B and C, the gateway identifies seven network paths from interface a to include in path pool 1710, only one network path from interface B to include in path pool 1710, and three network paths from interface C. In other words, the number of paths included in the path pool 1710 is weighted or determined based on the bandwidths of the different interfaces.

Fig. 20 illustrates data flow within a gateway for load balancing across multiple paths in multiple uplinks. The figure shows a portion of a computing device implementing gateway 312. The computing device may be a bare metal device or a host machine running virtualization software, where the gateway is implemented by a virtual machine. The gateway includes several sub-modules in the VPN control plane including a path/link performance monitor 2010, a best path identifier 2020. The gateway also includes several sub-modules in the VPN data plane including a routing interface (VTI) 2005, a load balancer/path selector 2030, an encryption module 2040, an encapsulation module 2050, and a routing module 2060.

As shown, performance monitor 2010 obtains performance metrics 2015 for each path and uplink interface by, for example, sending probe messages to those paths. Performance monitor 2010 may continue to monitor and provide the latest performance metrics for the path and uplink interface. The best path identifier 2020 uses performance metrics 2015 to identify paths to include in the path pool 1710. The best path identifier 2020 may prefer interfaces (e.g., direct connections) by including network paths that use more of the preferred interfaces, or may disfavor interfaces (e.g., to the internet) by including network paths that use less of the undesirable interfaces. When an interface fails, the path identifier 2020 may remove all paths belonging to the failed interface from the path pool 1710 such that the path pool 1710 only includes good execution paths for active uplinks.

The path selector 2030 in turn selects a path from the path pool 1710 to send the IPsec packet to the VPN peer. The path selector 2030 performs path selection based on a hash of a particular field of the outgoing packet in order to achieve load balancing between active paths. In some embodiments, the fields of the outgoing packet that are hashed for path selection may include the internal IP address (e.g., 222) and/or internal port information (e.g., 224) of the internal packet 220 prior to encryption.

In some embodiments, loopback IP may be used to support more network paths, thereby increasing entropy in load balancing. A gateway performing a VPN session may listen to multiple loopback IPs instead of directly listening to the uplink. More entropy/network paths with multiple UDP ports per uplink path are also contemplated.

When an application uses a gateway to send certain application data in VPN session 1600, the application data is received at routing interface VTI 2005 for security association SA 1. VTI 2005 is a single VTI for VPN session 1600. The application data is wrapped as an inner packet 2035 (e.g., inner packet 220) of an IPsec packet having internal IP and port information. The encryption module 2040 encrypts the inner packet into an IPsec encrypted packet 2045 according to encryption parameters of the security association SA1 and appends other IPsec related fields (e.g., ESP header, ESP trailer, ESP authentication, new IP, etc.) based on the SA 1.

In some embodiments, when NAT-T is enabled, the encapsulation module 2050 encapsulates the IPsec encrypted packet 2045 into a UDP encapsulated packet 2055 having a UDP encapsulation header (e.g., UDP header 242), which may include UDP port information. In some embodiments, when NAT-T is not enabled, IPsec encrypted internal packets will not be encapsulated by UDP and will not include UDP port information.

The data plane routing module 2060 then sends the IPsec encrypted packet 2045 (or the UDP encapsulated packet 2055) using the path selected by the load balancer 2030. The load balancer 2030 indicates information about the selected path to the data plane routing module 2060, including uplink interface information 2032 and an IP address 2034 for the selected path. The uplink interface information 2032 may include parameters, next hop IP address, etc. for accessing a particular type of physical medium of the uplink or selected path. When the selected path is the first uplink, the data plane routing module 2060 transmits IPsec packets using the interface of the first uplink. And when the selected path is a second uplink, the data plane routing module 2060 uses the interface of the second uplink.

For some embodiments, fig. 21 conceptually illustrates a process 2100 of performing load balancing when IPsec packets are sent across multiple active uplinks. In some embodiments, the gateway of the first site performs process 2100 when sending IPsec data to the second site. In some embodiments, one or more processing units (e.g., processors) of a computing device implementing gateway 312 perform process 2100 by executing instructions stored in a computer-readable medium.

The gateway establishes (at 2110) a Virtual Private Network (VPN) session with the VPN peer using a plurality of active uplinks having a first uplink interface for accessing the first set of paths and a second uplink interface for accessing the second set of paths. In some embodiments, a single VPN session with a single IKE SA and IPSec SA is used across multiple active uplink paths. In some embodiments, each path in the first set of paths is a direct direction of travel and each path in the second set of paths is through the internet.

The gateway collects (at 2120) performance metrics for paths in the first and second path sets. The gateway identifies (at 2130) paths from the first and second sets of paths to include in the path pool based on the collected performance metrics. In some embodiments, paths in the path pool are identified based on bandwidths of the first and second uplink interfaces such that the path pool has more paths belonging to the higher bandwidth uplink interface than paths belonging to the lower bandwidth uplink interface. For example, the path pool may include more paths through the direct connection than paths through the internet, because the uplink interface to the direct connection has a higher bandwidth than the uplink interface to the internet. The process may return to 2120 to continue to collect performance metrics for the path and update the path pool. In some embodiments, when an uplink interface fails, the gateway excludes (at 2135) the path of the failed uplink interface from the path pool.

The gateway receives (at 2140) data to be transmitted to the VPN peer in IPsec packets. In some embodiments, the VPN session uses a single Virtual Tunnel Interface (VTI) for the SA to receive data for the first and second uplink interfaces. The gateway selects (at 2150) a path from the path pool by using a hash value derived from the received data. In some embodiments, the hash value is further derived from the source port, the destination port, and the protocol identifier of the internal payload. In some embodiments, the hash value may also be derived from the source IP, destination IP, source port, destination port, and protocol identifier of the internal payload. In some embodiments, NAT-T is not enabled and the IPsec packet is not encapsulated by UDP.

The gateway encrypts the received data according to the SA (at 2160). The gateway transmits (at 2170) the encrypted data using the uplink interface corresponding to the selected path. For example, when the selected path is accessible through the first uplink interface, the gateway uses the first uplink interface to transmit the encrypted data as IPsec packets; when the selected path is accessible through the second uplink interface, the gateway uses the second uplink interface to transmit the encrypted data as IPsec packets.

Receive Side Scaling (RSS) refers to the distribution of network workload across multiple CPUs or processing cores. When RSS is enabled, data processing for a particular TCP connection is shared across multiple processors or processor cores. The hash function is used to calculate a hash value of a predetermined area or field within the received network data. For ESP packets, the RSS scheme for IPsec processing may hash fields such as source IP, destination IP, and SPI to determine which CPU to use for encryption or decryption, as these fields of the ESP packet are not encrypted.

As mentioned, in some embodiments, ESP packets are encapsulated with a UDP header, and a UDP port identifier in the UDP encapsulation is used to indicate path selection when multiple paths may be used to send IPsec data. In some embodiments, different traffic flows of an ESP tunnel are assigned different UDP port identifiers, and the hash function used to select the CPU or processing core considers the UDP port identifiers to achieve better load balancing. In other words, when the UDP port changes to indicate a different network traffic flow and/or a different path, a different CPU or processing core may be selected. In some embodiments, tuples of port number, source IP, destination IP, and SPI are used as flow identifiers, and a hash of the tuples of flow identifiers is used to select a CPU or processing core for IPsec processing.

Figure 22 conceptually illustrates an RSS scheme for assigning IPsec processing to processing cores. The figures illustrate portions of a computing device 2200 that implements an RSS scheme. Computing device 2200 may be a physical computing device, a bare metal device, or a host machine running virtualized software. Computing device 2200 may also implement a gateway or edge device for a data center. The computing device has at least four CPUs or processing cores 2201-2204 that can independently perform computations.

As shown, computing device 2200 receiving IPsec packets is using RSS to distribute authentication and decryption workload among multiple CPUs or processing cores. As shown, computing device 2200 receives IPsec packet 2214 for the VPN tunnel from network 100 at RX interface 2212. IPsec packet 2214 has an encrypted payload 2216 and unencrypted header fields 2218 such as UDP port identifiers, source and destination IP addresses, and SPI. A hash function 2220 is applied to some of the unencrypted header fields and the result of the hash is used to select one of the processing cores 2201-2204 (2202 in this example). The selected processing core decrypts payload 2216 into decrypted payload 2224 according to the SA. The decrypted data is provided to data path 2222 for further processing based on the stream identifier mapped from unencrypted header field 2218. Data path 2222 may be other processing elements of computing device 2200, or processing elements of another computing device that may be reached by network 100. The stream mapping function 2226 maps the tuples of UDP port identifiers, source and destination IP addresses, and SPI in the unencrypted header field 2218 into the stream identifier 2228 for the data path 2222 so that the decrypted payload 2224 can be properly aggregated with the data of the same stream.

In some embodiments, different traffic flows of a single SA are assigned different UDP port identifiers, so different flows may be handled by different cores. These different flows may have the same source and destination IP addresses and SPI. Figure 23 conceptually illustrates different streams of the same SA being processed by different processing cores. This figure conceptually illustrates an SA 2300 (SA a) that has been established for a VPN session by a computing device 2200 that is a VPN server or VPN client. The VPN session has at least four flows 2311-2314 encrypted according to the SA 2300. The packets of these four flows have the same source and destination IP addresses (10.10.10.1 and 20.20.20.2) and the same SPI. However, the four streams 2311-2314 have different UDP port identifiers that are hashed to different processing cores. Packets of these flows are encrypted or decrypted at these cores according to SA 2300.

In some embodiments, the computing device may encrypt or decrypt streams of IPsec packets belonging to different SAs. Figure 24 conceptually illustrates the flow of different SAs handled by different processing cores. In this example, the computing device 2200, which is a VPN server or VPN client, has established an SA 2300 (SA a) and a second SA 2400 (SA B) for the VPN session. The VPN session has at least two streams 2311-2312 encrypted in the SA 2300 and at least two streams 2413-2414 encrypted in the SA 2400. The flows 2311 and 2312 have port identifiers 8010 and 8020 hashed to the processing cores 2201 and 2202, respectively, and the flows 2313 and 2314 have port identifiers 8030 and 8040 hashed to the processing cores 2203 and 2204, respectively.

Flows of different SAs may have the same port number (e.g., because the path selection selects the same path). In some embodiments, flows of different SAs are assigned different SPIs (because the SPIs uniquely identify the SAs), and thus flows of different SAs may be hashed to different processing cores based on the different SPIs, even though they have the same port number. Figure 25 conceptually illustrates the flow of different SAs with the same port identifier handled by different processing cores.

In this example, at least two streams 2311-2313 are encrypted in the SA 2300, and at least two streams 2411 and 2413 are encrypted in the SA 2400. The flows 2311 and 2313 have port identifiers (8010 and 8030) that are the same as the port identifiers (8010 and 8030) of the flows 2411 and 2413. However, since the flows of the SA 2300 have different SPIs (spi=a versus spi=b) than the flows of the SA 2400, the flows of different SAs may be assigned to different processing cores for encryption or decryption, although having the same port number (e.g., because the same path is selected by the path selection)) and the same IP address.

Fig. 26 illustrates the generation of IPsec packets in which identifiers such as ports, IP addresses, and SPI are set for load balancing among a plurality of CPUs or processing cores. The figure illustrates the flow of data between the functional modules of computing device 2600. Computing device 2600 may be a bare metal device or a host machine running virtualization software. The computing device may also implement a gateway or edge device of a data center, such as gateway 312. The computing device has multiple CPUs or processing cores 2650 that can independently perform computations.

In computing device 2600, path monitoring module 2602 generates path metrics 2604 by probing different paths (as described above with reference to fig. 4). CPU monitor module 2606 monitors CPU 2600 to generate CPU metrics 2608, which may include current and predicted performance of CPU 2600. Core selection module 2610 uses generated CPU metrics 2608 to select one of the processing cores in CPU 2600. The path selection module 2612 uses the generated path metric 2604 to select a path and indicates the selected path as the UDP port number 2620.CPU 2600 receives payload 2614 from Receive (RX) interface 2616 and uses the selected processing core to perform authentication and encryption to generate encrypted payload 2618. The UDP encapsulation module 2626 encapsulates the encrypted payload 2618 to create an encapsulated packet 2622, the encapsulated packet 2622 including a UDP header including a UDP port number 2620. The network scheduling module 2624 provides additional flow identifiers 2628 (e.g., IP address and SPI) to the UDP encapsulation module 2626 for inclusion in the packet 2622. The transport interface 2630 then transports the encapsulated packets 2622.

In some embodiments, path monitoring module 2602, CPU monitoring module 2606, core selection module 2610, path selection module 2612, RX interface 2616, network scheduling module 2624, UDP encapsulation module 2626, and TX interface 2630 are modules of software instructions executed by one or more processing units (e.g., processors) of a computing device. In some embodiments, modules 2602, 2606, 2610, 2612, 2616, 2624, 2626, and 2630 are modules of hardware circuitry implemented by one or more Integrated Circuits (ICs) of an electronic device. Although the modules 2602, 2606, 2610, 2612, 2616, 2624, 2626, and 2630 are shown as separate modules, some of these modules may be combined into a single module.

For some embodiments, fig. 27 conceptually illustrates a process 2700 of distributing IPsec workloads among a plurality of processor cores using flow identifiers. In some embodiments, the gateway of the first site performs process 2700 when IPsec data is received from the second site. In some embodiments, one or more processing units (e.g., processors) of a computing device implementing gateway 312 perform process 2700 by executing instructions stored in a computer readable medium.

Process 2700 begins when the gateway receives (at 2710) an encapsulated packet for a VPN session. The encapsulated packet includes (i) a set of flow identifiers for the network traffic flow including a UDP port number, and (ii) a payload encrypted according to a security association. The packet is encapsulated by a UDP header including a UDP port number. In some embodiments, the UDP port number is determined based on a random number. In some embodiments, the UDP port number is a NAT translated port when NAT-T is detected between VPN peers.

In some embodiments, the UDP port number corresponds to a path selected to transmit the packet from the VPN client to the VPN server, the path being selected from a plurality of paths based on a performance metric of the path calculated from dynamic monitoring (e.g., by probing) of the path. In some embodiments, the UDP port number is adjusted according to congestion status information associated with different paths.

The gateway hashes (at 2720) the set of flow identifiers for the network traffic flows to obtain a hash value. The gateway selects (at 2730) a processor core from the plurality of processor cores based on the hash value. The gateway decrypts the payload according to the Security Association (SA) using (at 2740) the selected processor core.

Different streams of the same SA may be processed by different processing cores. In particular, a first set of stream identifiers for a first stream including a first UDP port number may be hashed to select a first processor core for decrypting a first packet, and a second set of stream identifiers for a second stream including a second UDP port number may be hashed to select a second, different processor core for decrypting a second packet. Flows of different SAs may also be handled by different processing cores, even though the flows have the same IP address and UDP port. In particular, a first set of stream identifiers for a first stream including a first SPI may be hashed to select a first processor core for decrypting a first packet, and a second set of stream identifiers for a second stream including a second, different SPI may be hashed to select a second, different processor core for decrypting a second packet.

Since the data of the IPsec packet is encrypted, it is difficult to enforce a specific QoS in the intermediate router. The external IPsec headers (e.g., tunnel source IP and destination IP) provide limited visibility into the network path. However, in modern cloud data centers, connectivity based on multiple network paths is often available to reach VPN peers, and different available paths may have different QoS characteristics for sending encrypted data packets. The QoS of an application depends on the network path that the application uses to send IPsec data to its peers. Using encrypted ESP payloads, VPN traffic will always select one of the paths based on the external ESP tunnel address, even if multiple network paths (ECMP routes) are available, and will eventually end up with QoS specific to that particular network path for all encrypted payloads.

Some embodiments of the present disclosure provide a mechanism for leveraging different QoS characteristics of different paths in a multipath VPN environment. Specifically, IPsec or VPN gateway classifies packets and paths based on their bandwidth requirements and their network characteristics (e.g., jitter, delay, packet loss, etc.). The VPN gateway may learn the network characteristics of multiple network paths by, for example, probing the paths to collect a set of performance metrics for each path. When applying or pre-configuring QoS, the IPsec gateway takes advantage of the network characteristics of multiple paths and selects a particular path for each packet based on the QoS required by the packet.

Figure 28 conceptually illustrates a gateway that selects a particular path for each packet based on the QoS required by the packet. The gateway classifies the data to be transmitted according to its QoS requirements. The gateway also classifies each path based on its network characteristics, in particular in terms of the QoS levels that the path can support.

As shown, gateway 312 of data center 310 is in a VPN session to send IPsec data to gateway 322 of data center 320. Gateway 312 may use several paths to reach gateway 322 for the VPN session, including paths 2801-2806 (labeled "path 1" through "path 6"). Gateway 312 uses these paths to send packets encrypted according to security association 2800 (SA 1).

The gateway gathers performance metrics and other states about the different paths by, for example, periodically sending probe messages through the paths and obtaining responses to the probe messages. The performance metrics of the path may include connectivity, latency, packet loss rate, and jitter of the path. In some embodiments, different paths are identified or defined by their source and/or destination IP addresses. In some embodiments, different paths may be identified by different port numbers (e.g., UDP port numbers). Based on the performance metrics collected from the probe paths, the gateway classifies each path according to the QoS level that the path can support. For example, paths with long latency, high packet loss rate, and low connectivity may be classified as supporting only network traffic with low QoS requirements, while paths with small latency and low packet loss rate may be classified as supporting network traffic with high QoS requirements. Gateway 312 uses the network characteristics or performance metrics of the different paths to generate path classification table 2815, where each path is assigned a QoS class. According to table 2815, "path 1", "path 3", and "path 7" (paths 2801, 2803, 2807) are classified as QoS class a, "path 2" and "path 5" (paths 2802 and 2805) are classified as QoS class B, "path 4" (path 2804) is assigned QoS class C, "path 6" (path 2806) is assigned QoS class D, and so on. In some embodiments, the gateway may assign two or more paths to the same QoS class or category. In some embodiments, the gateway assigns each path a unique QoS class according to the particular network characteristics of the path.

Gateway 312 also classifies the packets based on their QoS requirements. The data of the application may have a specific set of quality of service requirements, such as guaranteed delay or guaranteed bandwidth. Such requirements may be expressed as Differentiated Services Code Points (DSCPs) for applications or for data packets generated by applications. The data packets generated by the application have Differentiated Service Code Point (DSCP) values that are typically observed by intermediate routers between VPN peers. DSCP is a means of classifying and managing network traffic and providing QoS in a layer 3IP network. It uses the 6-bit Differentiated Services (DS) field in the IP header for packet classification. In some embodiments, the gateway may determine the QoS requirements of the packet based on the type or priority level of the application generating the packet. The gateway may also determine the QoS requirements of the packet based on (running) account information of the user generating the payload. In some embodiments, the QoS class of the packet is determined based on at least one of the DSCP field, the application type, and the internal port. The gateway in turn selects a path that can meet the QoS requirements of the packet, e.g., a QoS class with an assignment that matches the QoS class of the packet. In this example, packet 2825 is classified as QoS class C based on the QoS requirements of the packet. Gateway 312 accordingly selects path 2804 ("path 4") which is assigned QoS class C according to path classification table 2815.

As mentioned, gateway 312 uses multiple active paths to send IPsec packets and load balancing is performed across the multiple active paths. In some embodiments, gateway 312 performs load balancing on active paths of the same QoS class. Fig. 29 shows load balancing between active paths of the same QoS class. As shown, for QoS class a packets, gateway 312 performs load balancing between the active paths of QoS class a (paths 2801, 2803, and 2807). For QoS class B packets, the gateway performs load balancing between the active paths of QoS class B (paths 2802 and 2805). For QoS class C packets, the gateway uses a unique QoS class C path (path 2804). For QoS class D packets, the gateway uses a unique active QoS class D path (path 2806). In some embodiments, the gateway may perform dynamic path addition based on the required QoS. For example, if the gateway determines that paths 2801, 2803, and 2807 are not performing sufficiently to maintain QoS class a, the gateway may dynamically add one or more paths to QoS class a load balancing so that the VPN session may satisfy QoS class a requirements.

In the example of fig. 28, different paths for reaching the VPN server are used by the same SA 2800 (SA 1). In some embodiments, a gateway that is a VPN client may establish multiple IPsec SAs with a VPN server. Each SA is used to handle a particular QoS class. In some embodiments, each SA or QoS class is linked with a particular network path such that there is a one-to-one mapping between SA, qoS class and network path to give a particular QoS.

Fig. 30 illustrates assigning packets with different QoS requirements to gateways of paths with different SAs. As shown, gateway 312 may use at least 7 paths to reach gateway 322, labeled "path 1" through "path 7". Gateway 312 has established a different security association for each path (labeled "SA1" through "SA 7") based on the endpoints of those paths. Gateway 312 also assigns QoS classes to each of those paths based on their network characteristics, such that "path 1" is assigned QoS class a, "path 2" is assigned QoS class B, "path 3" is assigned QoS class C, and so on. The mapping between SA, path and QoS class is stored in the path classification table 3015. Based on the path classification table 3015, the first packet with QoS requirements classified as QoS class E will be sent over "path 5" and encrypted according to "SA 5"; a second packet with QoS requirements classified as QoS class B will be sent over "path 2" and encrypted according to "SA 2".

Fig. 31 illustrates data flows within a gateway for performing QoS provisioning in a multipath IPsec environment. The figure shows a portion of a computing device implementing gateway 312. The computing device may be a bare metal device or a host machine running virtualization software, where the gateway is implemented by a virtual machine.

As shown, gateway 312 receives application data 3100 at a Receive (RX) interface 3102. RX interface 3102 may refer to a network interface of a gateway that receives application data from other network endpoints, or to a software interface that receives data from a processing or data path element within the same computing device hosting the gateway. The RX interface 3102 provides application data 3100 as a packet payload 3106 to a cryptographic engine 3108. The cryptographic engine 3108 in turn encrypts the payload 3106 according to the security association to create an encrypted payload 3110.

Application data 3100 is associated with a set of QoS requirements 3104. The QoS requirements may include DSCP values, identifiers of applications or application types generating application data 3100, internal port numbers, account information, and/or any information that may be used to determine QoS requirements for application data. Packet classifier 3112 uses QoS requirements 3104 to assign packet classification 3114, e.g., by mapping different QoS requirements to different QoS classes using a lookup table.

The probe manager 3116 collects path performance metrics for different paths that may be used to transmit packets. Path performance metrics for a path may include packet loss rate, connectivity, latency, and other measures indicative of the level of service that the path may be able to support. Probe manager 3116 may periodically send probe messages to different paths to obtain updated path performance metrics thereof. The path classifier 3120 uses the collected path performance metrics 3118 to classify paths such that each path that may be used to reach a VPN peer is assigned a QoS class. In some embodiments, a lookup table is used to map different path performance metrics to different QoS classes.

The path classifier outputs a path classification table 3122 (e.g., path classification table 2815 of fig. 28) listing the assigned QoS classes for the different paths. In some embodiments, as the probe manager 3116 continuously probes paths to obtain new path performance metrics, the path classifier 3120 also continuously updates the path classification table 3122 so that the QoS class assigned to the path is up-to-date.

Packet classification 3114 and path classification table 3122 are provided to path selector 3124 to select a path for transmission of a packet containing application data 3100. Specifically, the path selector 3124 selects a path from the path classification table 3122 by identifying a path having an assigned QoS class that matches the QoS class of the packet indicated in the packet classification 3114. The path selector 3124 may indicate the selected path through the selected path identifier 3126. In some embodiments, the path selector 3124 performs load balancing for each QoS class by distributing packets of that QoS class among multiple active paths of the QoS class.

The gateway 312 in turn sends the encrypted payload 3110 using the selected path. In some embodiments, gateway 312 encapsulates the encrypted payload 3110 (at packet encapsulation module 3128) with a UDP header, which indicates the selected path via a UDP port number. Encapsulation results in an encapsulated packet 3130 that is transmitted to the network at a Transmission (TX) interface 3132. In some embodiments, if the selected path is identified by an IP address pair, the gateway does not perform UDP encapsulation unless a true NAT is detected.

In some embodiments, RX interface 3102, crypto engine 3108, packet classifier 3112, probe manager 3116, path classifier 3120, path selector 3124, packet encapsulation module 3128, and TX interface 3132 are modules of software instructions that are executed by one or more processing units (e.g., processors) of a computing device. In some embodiments, modules 3102, 3108, 3112, 3116, 3120, 3124, 3128, and 3132 are modules of hardware circuitry implemented by one or more Integrated Circuits (ICs) of an electronic device. Although the modules 3102, 3108, 3112, 3116, 3120, 3124, 3128 and 3132 are shown as separate modules, some of these modules may be combined into a single module. In some embodiments, the probe manager 3116, the path classifier 3120, the path performance metrics 3118, and the path classification table 3122 are components of a VPN control plane, while the RX interface 3102, the cryptographic engine 3108, the packet classifier 3112, the path selector 3124, the packet encapsulation module 3128, and the TX interface 3132 are components of a VPN data plane.

For some embodiments, fig. 32 conceptually illustrates a process 3200 for performing QoS provisioning in a multipath IPsec environment. In some embodiments, the gateway of the first site performs process 3200 when transmitting IPsec data to the second site. In some embodiments, one or more processing units (e.g., processors) of a computing device implementing gateway 312 perform process 3200 by executing instructions stored in a computer-readable medium.

The gateway gathers (at 3210) performance metrics or network characteristics of the multiple paths that the gateway, which is the VPN client, may use to reach the VPN server. In some embodiments, the gateway sends probe messages to the plurality of paths and receives responses to the probe messages, and the gateway collects performance metrics for the plurality of paths based on the received responses to the probe messages. The performance metrics of the path may be one or more of delay, packet loss rate, link capacity, and current bandwidth. The gateway assigns (at 3220) a QoS class to each of the plurality of paths based on the collected performance metrics. In some embodiments, the process continues to perform operations 3210 and 3220 to continuously update the path QoS assignment based on dynamic network characteristics.

The gateway receives (at 3230) data to be transmitted as a payload in the packet. The gateway identifies (at 3240) the QoS class of the packet. In some embodiments, the QoS class of the packet is determined based on a Differentiated Services Code Point (DSCP) of the packet. DSCP may be supplied by an application generating data to be transmitted. The QoS class of a packet may also be determined based on the application type and internal port value.

The gateway selects (at 3250) a path from the plurality of paths based on the QoS class of the identified packet and the QoS class assigned to each path of the plurality of paths. In some embodiments, the gateway selects a path with an assigned QoS class that matches the QoS class of the packet, for example, by using path classification table 3122.

The gateway encrypts (at 3255) the payload of the packet according to a security association established between the gateway as VPN client and the VPN server. In some embodiments, different QoS classes may use different SAs, or different paths may have different SAs. For example, a first packet having a first QoS class is encrypted according to a first security association of a VPN session and a second packet having a second QoS class is encrypted according to a second security association of the VPN session.

The gateway transmits (at 3260) the packet with the encrypted payload using the selected path. In some embodiments, the packet is encapsulated in a UDP header including a port number or identifier, and the port number is set to correspond to the selected path. In some embodiments, the IP address of the packet (e.g., the external source IP address) is set to correspond to the selected path.

In some embodiments, the gateway or edge device may be implemented by a host machine running virtualization software that acts as a virtual network forwarding engine. Such virtual network forwarding engines are also known as Managed Forwarding Elements (MFEs) or hypervisors. Virtualization software allows a computing device to host a set of Virtual Machines (VMs) and perform packet forwarding operations (including L2 switching and L3 routing operations). These computing devices are therefore also referred to as host machines. The packet forwarding operations of the virtualization software are managed and controlled by a set of central controllers, and thus in some embodiments, the virtualization software is also referred to as a Managed Software Forwarding Element (MSFE). In some embodiments, when the virtualization software of the host machine operates a local instance of a logical forwarding element as a physical forwarding element, the MSFE performs its packet forwarding operations on one or more logical forwarding elements. Some of these physical forwarding elements are Managed Physical Routing Elements (MPREs) for performing L3 routing operations for Logical Routing Elements (LREs), and some of these physical forwarding elements are Managed Physical Switching Elements (MPSE) for performing L2 switching operations for Logical Switching Elements (LSEs). FIG. 33 illustrates a computing device 3300 that serves as a host machine running virtualization software for some embodiments of the present invention.

As shown, computing device 3300 may access a physical network 3390 through a Physical NIC (PNIC) 3395. Host 3300 also runs virtualization software 3305 and hosts VMs 3311-3314. Virtualization software 3305 serves as an interface between the hosted VM and physical NIC 3395 (as well as other physical resources such as processors and memory). Each of the VMs includes a Virtual NIC (VNIC) for accessing a network through virtualization software 3305. Each VNIC in a VM is responsible for exchanging packets between the VM and the virtualization software 3305. In some embodiments, the VNIC is a software abstraction of a physical NIC implemented by a virtual NIC emulator.

Virtualization software 3305 manages the operation of VMs 3311-3314 and includes several components for managing the VM's access to the physical network (in some embodiments, through the logical network to which the VM is connected). As shown, the virtualization software includes several components including a MPSE 3320, a set of MPREs 3330, a controller agent 3340, a network data storage device 3345, a VTEP 3350, and a set of uplink pipes 3370.

VTEP (VXLAN tunnel endpoint) 3350 allows host machine 3300 to function as a tunnel endpoint for logical network traffic (e.g., VXLAN traffic). VXLAN is an overlay network encapsulation protocol. The overlay network created by the VXLAN encapsulation is sometimes referred to as a VXLAN network, or simply VXLAN. When a VM on host 3300 sends a packet (e.g., an ethernet frame) to another VM in the same VXLAN network but on a different host, the VTEP will encapsulate the data packet with the VNI of the VXLAN network and the network address of the VTEP before sending the packet to the physical network. Packets are tunneled through the physical network (i.e., encapsulation makes the underlying packets transparent to the intermediate network element) to the destination host. The VTEP decapsulates the packet at the destination host and forwards only the original internal data packet to the destination VM. In some embodiments, the VTEP module serves only as a controller interface for VXLAN encapsulation, while encapsulation and decapsulation of VXLAN packets is done at the uplink module 3370.

The controller agent 3340 receives control plane messages from a controller or cluster of controllers. In some embodiments, these control plane messages include configuration data for configuring various components of the virtualization software (such as MPSE3320 and MPRE 3330) and/or the virtual machine. In the example shown in fig. 33, the controller agent 3340 receives control plane messages from the controller cluster 3360 from the physical network 3390 and then provides the received configuration data to the MPREs 3330 through the control channel without passing through the MPSE 3320. However, in some embodiments, the controller agent 3340 receives control plane messages from a direct data pipe (not shown) that is independent of the physical network 3390. In some other embodiments, the controller agent receives the control plane message from the MPSE3320 and forwards the configuration data to the router 3330 through the MPSE 3320.

In some embodiments, network data storage device 3345 stores some of the data used and generated by the logical forwarding elements of host machine 3300, such as the logical forwarding elements of MPSE3320 and MPRE 3330. In some embodiments, such stored data includes forwarding and routing tables, connection mapping, and packet traffic statistics. In some embodiments, these stored data may be accessed by the controller agent 3340 and delivered to another computing device that is operating the troubleshooting system (e.g., 150).

The MPSE 3320 delivers network data to the physical NIC 3395 that interfaces with the physical network 3390 and receives network data from the physical NIC 3395. The MPSE also includes a plurality of virtual ports (vPorts) that communicatively interconnect the physical NICs with the VMs 3311-3314, MPRE 3330, and controller agent 3340. In some embodiments, each virtual port is associated with a unique L2MAC address. The MPSE performs L2 link layer packet forwarding between any two network elements connected to its virtual ports. The MPSE also performs L2 link layer packet forwarding between any network element connected to any of its virtual ports and an reachable L2 network element on the physical network 3390 (e.g., another VM running on another host). In some embodiments, the MPSE is a local instance of a Logical Switch Element (LSE) that operates across different host machines and may perform L2 packet switching between VMs on the same host machine or on different host machines. In some embodiments, the MPSE performs the switching functions of several LSEs according to the configuration of those logical switches.

MPRE 3330 performs L3 routing on data packets received from virtual ports on MPSE 3320. In some embodiments, this routing operation requires resolving the L3 IP address into a next-hop L2MAC address and a next-hop VNI (i.e., the VNI of the L2 segment of the next hop). Each routed data packet is then sent back to the MPSE 3320 to be forwarded to its destination according to the parsed L2MAC address. The destination can be another VM connected to a virtual port on the MPSE 3320 or an reachable L2 network element on the physical network 3390 (e.g., another VM running on another host, a physical non-virtualized machine, etc.).

As mentioned, in some embodiments, MPREs are local instances of Logical Routing Elements (LREs) that operate across different host machines, and L3 packet forwarding may be performed between VMs on the same host machine or on different host machines. In some embodiments, the host machine may have multiple MPREs connected to a single MPSE, where each MPRE in the host machine implements a different LRE. In some embodiments, MPRE and MPSE are also referred to as "physical" routing/switching elements, even though they are implemented in software, in order to distinguish them from "logical" routing/switching elements. In some embodiments, the MPRE is referred to as a "software router" and the MPSE is referred to as a "software switch". In some embodiments, LREs and LSEs are collectively referred to as Logical Forwarding Elements (LFEs), while MPREs and MPSEs are collectively referred to as Managed Physical Forwarding Elements (MPFEs). Some Logical Resources (LR) referred to throughout this document are LREs or LSEs that have corresponding local MPREs or local MPSE running in each host machine.

In some embodiments, MPRE 2630 includes one or more Logical Interfaces (LIFs), each of which serves as an interface to a particular segment of the network (L2 segment or VXLAN). In some embodiments, each LIF is addressable by its own IP address and serves as a default gateway or ARP proxy for network nodes (e.g., VMs) of its particular segment of the network. In some embodiments, all MPREs in different host machines are addressable by the same "virtual" MAC address (or vMAC), and each MPRE is also assigned a "physical" MAC address (or pMAC) to indicate in which host machine the MPRE is operating from the bottom.

Uplink module 3370 relays data between MPSE 3320 and physical NIC 3395. The uplink module 3370 includes an egress chain and an ingress chain each performing a plurality of operations. Some of these operations are pre-processing and/or post-processing operations for MPRE 3330.

As shown in fig. 33, virtualization software 3305 has multiple MPREs for multiple different LREs. In a multi-tenant environment, a host machine may operate virtual machines from multiple different users or tenants (i.e., connected to different logical networks). In some embodiments, each user or tenant has a corresponding MPRE instance of its LRE in the host for handling its L3 routes. In some embodiments, while different MPREs belong to different tenants, they share the same vPort on MPSE 3320 and thus share the same L2 MAC address (vMAC or pMAC). In some other embodiments, each different MPRE belonging to a different tenant has its own port to the MPSE.

The MPSE 3320 and MPRE 3330 make it possible for data packets to be forwarded between the VMs 3311-3314 without being sent over the external physical network 3390 (as long as the VMs are connected to the same logical network, since the VMs of different tenants will be isolated from each other). Specifically, the MPSE performs the functions of the local logical switch by using VNIs of various L2 segments of various logical networks (i.e., their corresponding L2 logical switches). Also, MPRE performs the function of logical routers by using VNIs of those various L2 segments. Since each L2 segment/L2 switch has its own unique VNI, the host machine 3300 (and its virtualization software 3305) is able to direct packets of different logical networks to their correct destinations and effectively separate traffic of different logical networks from each other.

Many of the features and applications described above are implemented as software processes, which are specified as a set of instructions recorded on a computer-readable storage medium (also referred to as a computer-readable medium). When executed by one or more processing units (e.g., one or more processors, processor cores, or other processing units), cause the processing unit(s) to perform the actions indicated in the instructions. Examples of computer readable media include, but are not limited to, CD-ROM, flash memory drives, RAM chips, hard drives, EPROMs, and the like. Computer readable media does not include carrier waves and electronic signals transmitted over a wireless or wired connection.

In this specification, the term "software" refers to applications stored in magnetic storage that include firmware residing in read-only memory or that can be read into memory for processing by a processor. Furthermore, in some embodiments, multiple software inventions may be implemented as sub-portions of a larger program, while maintaining the apparent software invention. In some embodiments, multiple software inventions may also be implemented as separate programs. Finally, any combination of separate programs that together implement the software invention described herein is within the scope of the present invention. In some embodiments, when a software program is installed to operate on one or more electronic systems, the software program defines one or more particular machine implementations that perform the operations of the software program.

FIG. 34 conceptually illustrates a computer system 3400 with which some embodiments of the invention are implemented. The computer system 3400 may be used to implement any of the hosts, controllers, and managers described above. It can therefore be used to perform any of the above-described processes. This computer system includes various types of non-transitory machine-readable media and interfaces for various other types of machine-readable media. The computer system 3400 includes a bus 3405, processing unit(s) 3410, a system memory 3425, a read only memory 3430, a persistent storage device 3435, an input device 3440, and an output device 3445.

Bus 3405 collectively represents all system, peripheral, and chipset buses that communicatively connect many internal devices of computer system 3400. For example, a bus 3405 communicatively connects the processing unit(s) 3410 with the read-only memory 3430, the system memory 3425, and the persistent storage 3435.

The processing unit(s) 3310 retrieve instructions to be executed and data to be processed from these various memory units in order to perform the processes of the present invention. The processing unit(s) may be a single processor or a multi-core processor in different embodiments.

Read Only Memory (ROM) 3430 stores static data and instructions required by processing unit(s) 3410 and other modules of the computer system. On the other hand, persistent storage 3435 is a read-write memory device. This device is a non-volatile memory unit that stores instructions and data even when computer system 3400 is turned off. Some embodiments of the invention use a mass storage device (such as a magnetic or optical disk and its corresponding disk drive) as persistent storage 3435.

Other embodiments use removable storage devices (such as floppy disks, flash drives, etc.) as persistent storage devices. Like persistent storage 3435, system memory 3425 is a read-write memory device. However, unlike the storage device 3435, the system memory is a volatile read-write memory, such as a random access memory. The system memory stores some of the instructions and data that the processor needs at runtime. In some embodiments, the processes of the present invention are stored in system memory 3425, persistent storage 3435 and/or read-only memory 3430. The processing unit(s) 3410 retrieve the instructions to be executed and the data to be processed from these various memory units in order to perform the processes of some embodiments.

The bus 3405 is also connected to input and output devices 3440 and 3445. Input devices enable users to communicate information to computer systems and select commands to electronic systems. Input devices 3440 include an alphanumeric keyboard and pointing device (also referred to as a "cursor control device"). The output device 3445 displays images generated by the computer system. The output devices include printers and display devices, such as Cathode Ray Tubes (CRTs) or Liquid Crystal Displays (LCDs). Some embodiments include devices that function as both input and output devices, such as touch screens.

Finally, as shown in fig. 34, the bus 3405 also couples the electronic system 3400 to the network 3465 through a network adapter (not shown). In this manner, the computer may be part of a network of computers, such as a local area network ("LAN"), a wide area network ("WAN"), or an intranet, or a network of networks, such as the Internet. Any or all of the components of computer system 3400 may be used in conjunction with the present invention.

Some embodiments include electronic components such as microprocessors, storage devices and memories that store computer program instructions in a machine-readable or computer-readable medium (alternatively referred to as a computer-readable storage medium, a machine-readable medium, or a machine-readable storage medium). Some examples of such computer-readable media include RAM, ROM, compact disk read-only (CD-ROM), compact disk recordable (CD-R), compact disk rewriteable (CD-RW), digital versatile disk read-only (e.g., DVD-ROM, dual layer DVD-ROM), various recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD cards, mini SD cards, micro SD cards, etc.), magnetic and/or solid state hard disk drives, read-only and recordable Blu-ray (CD-RW) drives A disk, a super-density optical disk, any other optical or magnetic medium, and a floppy disk. The computer readable medium may store a computer program executable by at least one processing unit and including a set of instructions for performing various operations. Examples of a computer program or computer code include machine code, such as produced by a compiler, and files containing higher level code that are executed by the computer, electronic components, or microprocessor using an interpreter.

Although the discussion above refers primarily to microprocessors or multi-core processors executing software, some embodiments are performed by one or more integrated circuits, such as Application Specific Integrated Circuits (ASICs) or Field Programmable Gate Arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored on the circuit itself.

As used in this specification, the terms "computer," "server," "processor," and "memory" all refer to electronic or other technical devices. These terms do not include a person or group of people. For the purposes of this specification, the term display or displaying means displaying on an electronic device. The terms "computer-readable medium," "plurality of computer-readable media," and "machine-readable medium" as used in this specification are entirely limited to tangible, physical objects that store information in a form readable by a computer. These terms do not include any wireless signals, wired download signals, and any other transitory signals.

Although the invention has been described with reference to numerous specific details, one of ordinary skill in the art will recognize that the invention can be embodied in other specific forms without departing from the spirit of the invention. The several embodiments described above include various data in the overlay encapsulation header. One of ordinary skill will recognize that other embodiments may not use encapsulation headers to relay all such data.

Moreover, several figures conceptually illustrate the processes of some embodiments of the invention. In other embodiments, the specific operations of these processes may not be performed in the exact order shown and described in these figures. Certain operations may not be performed in a series of sequential operations, and different certain operations may be performed in different embodiments. Furthermore, a process may be implemented using several sub-processes, or as part of a larger macro process. It will be understood, therefore, by one of ordinary skill in the art that the present invention is not limited by the foregoing illustrative details, but is defined by the following claims.

Claims (22)

1. A method, comprising:

collecting metrics for one or more paths of a first tunnel for implementing a first Security Association (SA) and one or more paths of a second tunnel for implementing a second SA;

Selecting a path based on the collected metrics of the paths of the first tunnel and the second tunnel;

encrypting data to be transmitted as an encrypted payload of the first SA and transmitting the encrypted payload in the first tunnel when the selected path belongs to the first tunnel; and

when the selected path belongs to the second tunnel, data to be transmitted as an encrypted payload of the second SA is encrypted and the encrypted payload is transmitted in the second tunnel.

2. The method of claim 1, wherein the data to be transmitted is received at a single routing layer interface for encryption and transmission in a first tunnel using a first SA and a second tunnel using a second SA.

3. The method of claim 1, wherein the data to be transmitted is received at a bound interface at an application layer, wherein the bound interface comprises

A first routing layer interface for encrypting received data for transmission in a first tunnel using a first SA, an

And a second routing layer interface for encrypting the received data for transmission in a second tunnel using a second SA.

4. The method of claim 1, further comprising sending a probe message and receiving a response thereto, wherein the collected metrics for the one or more paths of the first tunnel and the second tunnel are determined based on the received response to the probe message.

5. The method of claim 4, wherein the metrics of the path include at least one of connectivity, latency, packet loss rate, and jitter of the path.

6. The method of claim 1, wherein the encapsulated encrypted payload comprises a User Datagram Protocol (UDP) header storing the first source port.

7. The method of claim 1, wherein:

the data to be transmitted is from a first network endpoint to a second network endpoint;

the first network endpoint is hosted by a first data center and the second network endpoint is hosted by a second data center; and

the gateway is an edge device of the first data center.

8. The method of claim 1, wherein the first tunnel includes a path through the internet and the second tunnel does not include a path through the internet.

9. The method of claim 1, wherein:

encapsulating the encrypted payload by appending (i) a first source address identifying the first tunnel and (ii) a first source port identifying the selected path when the selected path belongs to the first tunnel; and

when the selected path belongs to the second tunnel, the encrypted payload is encapsulated by appending (i) a second source address identifying the second tunnel and (ii) a second source port identifying the selected path.

10. The method of claim 1, wherein the collected metrics are used to identify a pool of best execution paths, and a path is selected from the pool of best execution paths for load balancing.

11. A method, comprising:

establishing a plurality of active uplinks for a Virtual Private Network (VPN) session with a VPN peer using a first uplink interface for accessing a first set of paths and a second uplink interface for accessing a second set of paths;

selecting a path from a pool of paths by using a hash value derived from data to be transmitted to a peer in a VPN session, wherein the paths in the pool are identified from a first set of paths and a second set of paths based on a performance metric;

transmitting data as IPsec packets on the first uplink interface when the selected path is accessible by the first uplink interface; and

when the selected path is accessible by the second uplink interface, data is transmitted as IPsec packets on the second uplink interface, wherein the data is encrypted according to a Security Association (SA).

12. The method of claim 11, wherein each path in the first set of paths is through a direct direction and each path in the second set of paths is through the internet.

13. The method of claim 12, wherein the pool of paths includes more paths through the direct connection than paths through the internet.

14. The method of claim 11, wherein paths in the path pool are identified based on bandwidths of the first and second uplink interfaces such that the path pool has more paths belonging to a higher bandwidth uplink interface than paths belonging to a lower bandwidth uplink interface.

15. The method of claim 11, further comprising determining whether an uplink interface has failed and excluding a path of the failed uplink interface from a path pool.

16. The method of claim 11, wherein the hash value is derived from an internal IP address of the IPsec packet.

17. The method of claim 16, wherein the hash value is further derived from a source port, a destination port, and a protocol identifier of the internal payload.

18. The method of claim 11, wherein the VPN session uses a single Virtual Tunnel Interface (VTI) for the SA to receive data for the first uplink interface and the second uplink interface.

19. A machine readable medium storing a program which, when implemented by at least one processing unit, implements the method of any of claims 1-18.

20. An electronic device, comprising:

a collection of processing units; and

a machine readable medium storing a program which when implemented by at least one of the processing units implements the method according to any one of claims 1-18.

21. A system comprising means for implementing the method of any one of claims 1-18.

22. A computer program product comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1-18.