WO2018109536A1

WO2018109536A1 - Method and apparatus for monitoring virtual extensible local area network (vxlan) tunnel with border gateway protocol (bgp)-ethernet virtual private network (evpn) infrastructure

Info

Publication number: WO2018109536A1
Application number: PCT/IB2016/057748
Authority: WO
Inventors: Vinayak Joshi
Original assignee: Telefonaktiebolaget Lm Ericsson (Publ)
Priority date: 2016-12-17
Filing date: 2016-12-17
Publication date: 2018-06-21

Abstract

A method and apparatus for monitoring a status of a virtual extensible local area network (VXLAN) tunnel between a first network device and a second network device are described. A first packet is received from the first network device through the VXLAN tunnel. The first packet includes a network identifier which identifies a VXLAN segment and an inner destination address that identifies the first network device within the VXLAN segment. In response to receiving the first packet, the second network device transmits, based upon the inner destination address and the network identifier, a second packet towards the first network device through the VXLAN tunnel, where the transmitting the second packet causes the first network device to determine whether the VXLAN tunnel is active.

Description

METHOD AND APPARATUS FOR MONITORING VIRTUAL EXTENSIBLE LOCAL AREA NETWORK (VXLAN) TUNNEL WITH BORDER GATEWAY PROTOCOL (BGP)- ETHERNET VIRTUAL PRIVATE NETWORK (EVPN) INFRASTRUCTURE

TECHNICAL FIELD

[0001] Embodiments of the invention relate to the field of computer networks; and more specifically, to monitoring Virtual Extensible Local Area Network (VXLAN) tunnel with border gateway protocol (BGP)-Ethernet virtual private network (EVPN) infrastructure.

BACKGROUND

[0002] A virtual network is a logical abstraction of a physical network that provides network services (e.g., L2 and/or L3 services). A virtual network can be implemented as an overlay network (sometimes referred to as a network virtualization overlay) that provides network services (e.g., layer 2 (L2, data link layer) and/or layer 3 (L3, network layer) services) over an underlay network (e.g., an L3 network, such as an Internet Protocol (IP) network that uses tunnels (e.g., generic routing encapsulation (GRE), layer 2 tunneling protocol (L2TP), IPSec, Virtual Extensible Local Area Network (VXLAN)) to create the overlay network).

[0003] A network virtualization edge (NVE) sits at the edge of the underlay network and participates in implementing the network virtualization; the network-facing side of the NVE uses the underlay network to tunnel frames to and from other NVEs; the outward-facing side of the NVE sends and receives data to and from systems outside the network. A virtual network instance is a specific instance of a virtual network on a NVE (e.g., a network element (NE) / virtual network element (VNE) on an network device (ND), a part of a NE/VNE on a ND where that NE/VNE is divided into multiple VNEs through emulation); one or more virtual network instances can be instantiated on an NVE (e.g., as different VNEs on an ND).

[0004] VXLAN (Virtual extensible Local Area Network) is a Layer 2 overlay scheme on a Layer 3 network. Each overlay network is referred to as a VXLAN segment. VXLAN was introduced to address the multiple requirements of the Layer 2 and Layer 3 data center network infrastructure in the presence of virtual computing elements (e.g., virtual machines (VMs), containers, etc.) in a multi-tenant environment. VXLAN runs over existing networking infrastructure and provides a means to "stretch" a Layer 2 network. In VXLAN, only network elements (virtual or physical) that are within the same VXLAN segment can communicate with each other through the VXLAN tunnel. Each VXLAN segment is identified through a 24-bit segment ID, termed the "VXLAN Network Identifier (VNI)." This allows up to 16M VXLAN segments to coexist within the same administrative domain. [0005] Typically, connectivity of the overlay tunnels is monitored using keep-alive packets exchanged between two endpoints of the tunnels. Keep-alive requests are originated from a first tunnel endpoint, and a response is expected back from the second endpoint of the tunnel.

[0006] In some approaches, Bidirectional Forwarding Detection (BFD) can be used to monitor VXLAN tunnels. A first tunnel endpoint sends periodic BFD requests and expects a response from the second tunnel endpoint. However, this mechanism is very limited in scenarios where the network device (e.g., a data center gateway) including one of the tunnel endpoint terminates large scales of tunnels, due to BFD scaling issues.

SUMMARY

[0007] One general aspect includes a method of monitoring a status of a virtual extensible local area network (VXLAN) tunnel between a first network device and a second network device. The method includes: receiving, from the first network device through the VXLAN tunnel, a first packet, where the first packet includes a network identifier which identifies a VXLAN segment and an inner destination address that identifies the first network device within the VXLAN segment; and in response to receiving the first packet, transmitting, based upon the inner destination address and the network identifier, a second packet towards the first network device through the VXLAN tunnel, where the transmitting the second packet causes the first network device to determine whether the VXLAN tunnel is active.

[0008] One general aspect includes a network device for monitoring virtual extensible local area network (VXLAN) tunnel status between a first network device and a second network device. The network device includes: a non-transitory computer readable medium to store instructions; and a processor coupled with the non-transitory computer readable medium to process the stored instructions to: receive, from the first network device through the VXLAN tunnel, a first packet, where the first packet includes a network identifier which identifies a VXLAN segment and an inner destination address that identifies the first network device within the VXLAN segment; and in response to receipt of the first packet, transmit, based upon the inner destination address and the network identifier, a second packet towards the first network device through the VXLAN tunnel, where to transmit the second packet causes the first network device to determine whether the VXLAN tunnel is active.

[0009] One general aspect includes a non-transitory computer readable storage medium that provide instructions, which when executed by a processor of a second network device that is coupled with a first network device through a virtual extensible local area network (VXLAN) tunnel, cause said processor to perform operations including: receiving, from a first network device through a virtual extensible local area network (VXLAN) tunnel, a first packet, where the first packet includes a network identifier which identifies a VXLAN segment and an inner destination address that identifies the first network device within the VXLAN segment; and in response to receiving the first packet, transmitting, based upon the inner destination address and the network identifier, a second packet towards the first network device through the VXLAN tunnel, where the transmitting the second packet causes the first network device to determine whether the VXLAN tunnel is active.

BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The invention may best be understood by referring to the following description and accompanying drawings that are used to illustrate embodiments of the invention. In the drawings:

[0011] Figure 1A illustrates a block diagram of an exemplary network operative to perform monitoring of VXLAN tunnels in accordance with some embodiments.

[0012] Figure IB illustrates a block diagram of an exemplary network according to a centralized routing approach, operative to perform monitoring of VXLAN tunnels in accordance with some embodiments.

[0013] Figure 2A illustrates an exemplary advertisement message indicating a route between a first and a second device within a VXLAN segment, in accordance with some embodiments.

[0014] Figure 2B illustrates an exemplary packet for transmitting a keep-alive request from a first network device to a second network devices through a VXLAN tunnel, in accordance with some embodiments.

[0015] Figure 2C illustrates an exemplary packet for transmitting a keep-alive response from the second network device to the first network devices through a VXLAN tunnel, in accordance with some embodiments.

[0016] Figure 3A illustrates exemplary operations for forwarding a keep-alive response packet in response to receiving a keep-alive request from a first network device in accordance with some embodiments.

[0017] Figure 3B illustrates exemplary operations for forwarding a keep-alive request packet in accordance with some embodiments.

[0018] Figure 4A illustrates connectivity between network devices (NDs) within an exemplary network, as well as three exemplary implementations of the NDs, according to some embodiments of the invention.

[0019] Figure 4B illustrates an exemplary way to implement a special-purpose network device according to some embodiments of the invention. [0020] Figure 4C illustrates a network with a single network element (NE) on each of the NDs, and within this straight forward approach contrasts a traditional distributed approach (commonly used by traditional routers) with a centralized approach for maintaining reachability and forwarding information (also called network control), according to some embodiments of the invention.

[0021] Figure 5 illustrates a general purpose control plane device with centralized control plane (CCP) software 550), according to some embodiments of the invention.

DETAILED DESCRIPTION

[0022] The following description describes methods and apparatus for monitoring Virtual Extensible Local Area Network (VXLAN) tunnel with border gateway protocol (BGP)-Ethernet virtual private network (EVPN) infrastructure. In the following description, numerous specific details such as logic implementations, opcodes, means to specify operands, resource partitioning/sharing/duplication implementations, types and interrelationships of system components, and logic partitioning/integration choices are set forth in order to provide a more thorough understanding of the present invention. It will be appreciated, however, by one skilled in the art that the invention may be practiced without such specific details. In other instances, control structures, gate level circuits and full software instruction sequences have not been shown in detail in order not to obscure the invention. Those of ordinary skill in the art, with the included descriptions, will be able to implement appropriate functionality without undue experimentation.

[0023] References in the specification to "one embodiment," "an embodiment," "an example embodiment," etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to affect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.

[0024] Bracketed text and blocks with dashed borders (e.g., large dashes, small dashes, dot- dash, and dots) may be used herein to illustrate optional operations that add additional features to embodiments of the invention. However, such notation should not be taken to mean that these are the only options or optional operations, and/or that blocks with solid borders are not optional in certain embodiments of the invention. [0025] In the following description and claims, the terms "coupled" and "connected," along with their derivatives, may be used. It should be understood that these terms are not intended as synonyms for each other. "Coupled" is used to indicate that two or more elements, which may or may not be in direct physical or electrical contact with each other, co-operate or interact with each other. "Connected" is used to indicate the establishment of communication between two or more elements that are coupled with each other.

[0026] An electronic device stores and transmits (internally and/or with other electronic devices over a network) code (which is composed of software instructions and which is sometimes referred to as computer program code or a computer program) and/or data using machine -readable media (also called computer-readable media), such as machine -readable storage media (e.g., magnetic disks, optical disks, read only memory (ROM), flash memory devices, phase change memory) and machine -readable transmission media (also called a carrier) (e.g., electrical, optical, radio, acoustical or other form of propagated signals - such as carrier waves, infrared signals). Thus, an electronic device (e.g., a computer) includes hardware and software, such as a set of one or more processors coupled to one or more machine -readable storage media to store code for execution on the set of processors and/or to store data. For instance, an electronic device may include non-volatile memory containing the code since the non-volatile memory can persist code/data even when the electronic device is turned off (when power is removed), and while the electronic device is turned on that part of the code that is to be executed by the processor(s) of that electronic device is typically copied from the slower nonvolatile memory into volatile memory (e.g., dynamic random access memory (DRAM), static random access memory (SRAM)) of that electronic device. Typical electronic devices also include a set or one or more physical network interface(s) to establish network connections (to transmit and/or receive code and/or data using propagating signals) with other electronic devices. One or more parts of an embodiment of the invention may be implemented using different combinations of software, firmware, and/or hardware.

[0027] A network device (ND) is an electronic device that communicatively interconnects other electronic devices on the network (e.g., other network devices, end-user devices). Some network devices are "multiple services network devices" that provide support for multiple networking functions (e.g., routing, bridging, switching, Layer 2 aggregation, session border control, Quality of Service, and/or subscriber management), and/or provide support for multiple application services (e.g., data, voice, and video).

[0028] Overlay based connectivity is increasingly used in data centers. For example, overlay tunnels are established between the DC-GW and virtual switches of the respective compute nodes. The DC-GW is a network device that connects the compute nodes of the data center with external networks. Generic Routing Encapsulation (GRE) and VXLAN are examples of overlay technologies that can be used for overlay tunnel encapsulations in data centers. Overlay tunnels are established between a Data Center Gateway (DC-GW) and the compute nodes that make up the data center (e.g., telco clusters, datacenters, server farms).

[0029] VXLAN has emerged as a network virtualization overlay standard. It addresses data plane needs using overlay networks within virtualized data centers that accommodate multiple tenants. VXLAN encapsulates an Ethernet Media Access Control (MAC) frame using User Datagram Protocol (UDP). To meet the massive tenant endpoint scaling requirements, its 24-bit VPN identifier provides more than 16 million VXLAN segment IDs. Since VXLAN is routable with IP, it allows the use of existing underlay IP network resiliency and load balancing mechanisms such as Equal Cost Multi-Path and IP Fast Reroute.

[0030] A routing and reachability protocol is used to announce and receive prefixes of the network devices independently of the underlay network. For example, Border Gateway Protocol (BGP) Layer 3 Virtual Private Network (L3VPN) or BGP-Ethernet Virtual Private Network (EVPN) can be used to announce routes in the overlay network and enable tenant separation in addition to providing external connectivity to compute host of a data center. BGP can be hosted on a Software Defined Networking (SDN) controller, or alternatively can be run inside a Virtual Router route processor (RP). BGP-EVPN is used for extending L2 domains in a cloud outside the data center (across data centers, into enterprise premises etc.) and BGP-L3VPN is used for extending the L3 domains inside the cloud outside the data center. Traditionally, Multiprotocol Label Switching (MPLS) over GRE tunneling is used for L3VPN and VXLAN tunneling is used for EVPN.

[0031] Typically, connectivity of the overlay tunnels is monitored using keep-alive packets exchanged between two endpoints of the tunnels. Keep-alive requests are originated from a first tunnel endpoint, and a response is expected back from the second endpoint of the tunnel. In some approaches, Bidirectional Forwarding Detection (BFD) can be used to monitor VXLAN tunnels. A first tunnel endpoint sends periodic BFD requests and expects a response from the second tunnel endpoint. However, this mechanism is very limited in scenarios where the network device (e.g., a data center gateway) including one of the tunnel endpoint terminates large scales of tunnels, due to BFD scaling issues. When GRE is used as the tunneling protocol, a mechanism referred to as GRE-Keep Alive (GRE-KA) can be used. However this mechanism covers only GRE tunnels and is not applicable to VXLAN tunnels.

[0032] Embodiments of the present invention disclose methods and apparatuses for monitoring VXLAN tunnel with BGP-EVPN infrastructure. A dedicated EVPN instance is created (EVI_0). The EVPN instance is associated with a unique VXLAN network identifier (VNI_0). VNI_0 is unique in the entire network such that there is no VNI overlap and hence no overlap of L2 domains. As will be discussed in further details below, the unique VNI_0 is used for forwarding keep-alive packets of a VXLAN tunnel between two network devices. Thus there is a one to one mapping between EVI_0 and VNI_0.

[0033] Once the VNI and EVI are created, a network controller advertises a unique route within the VNI. The route identifies two end points of a VXLAN tunnel and is advertised to a second one of the endpoints. Following the advertisement and installation of the routes at the second endpoint, the first endpoint starts transmitting keep-alive request on the VXLAN tunnel connecting the first and the second endpoints. The second endpoint receives, from the first network device through the VXLAN tunnel, a first packet, where the first packet includes a network identifier which identifies the VXLAN segment and an inner destination address that identifies the first network device within the VXLAN segment; and in response to receiving the first packet, the second endpoint transmits, based upon the inner destination address and the network identifier, a second packet towards the first network device through the VXLAN tunnel. The second packet causes the first network device to determine whether the VXLAN tunnel is active. In some embodiments, if the first network device receives the second packet within a predetermined period of time from the transmission of the first packet, it determines that the VXLAN tunnel is still active. Alternatively if the first network device does not receive the second packet within the predetermined period of time, it determines that the VXLAN tunnel is down.

[0034] The embodiments of the present invention provide a scalable mechanism for VXLAN tunnel monitoring in a BGP-EVPN network with VXLAN overlay. The embodiments do not require a specific configuration at the endpoints of the tunnel. As opposed to standard approaches which need substantial storage and processing resources for state tracking of BFD sessions, the embodiments of the present invention present a stateless mechanism that scales well for enabling monitoring of millions of VXLAN tunnels at a given network device. The embodiments are scalable on the network device receiving the requests (e.g., a gateway device of a cloud data center), even when it has very high fan-out because as the embodiments involve only a look up of the Forwarding Information Base (FIB) at the network device. Further the embodiments, are applicable in both centralized routing approaches as well as distributed routing approaches as will be described in further details below.

[0035] Figure 1A illustrates a block diagram of an exemplary network operative to perform monitoring of VXLAN tunnels in accordance with some embodiments. The network 100 includes a first network device (ND) 111 A and a second network device 112. ND 111 A is a network device that is coupled with ND 112 through a VXLAN tunnel 106 A implemented over an underlay network 102. The underlay network can be an Internet Protocol (IP) Layer 3 protocol. Each one of the NDs 111 A-N, and 112 can be implemented as described in further details below with reference to Figures 4A-C. In some embodiments, ND 111A may be one of multiple network devices (e.g., hosts) of a data center 110. For example, the data center 110 may include 111 A- 11 IN. While some embodiments will be described with the example of a data center, in other embodiments, the mechanism of the present invention apply to any networks where BGP-EVPN with VXLAN overlay is deployed.

[0036] The network 100 includes another network device ND 112 that is coupled with one or more network devices (111 A- 111 N) through overlay VXLAN tunnels. ND 112 is a VXLAN tunnel termination node (or as it may be referred to as a VXLAN Tunnel End Point (VTEP)). The ND 112 maps VLANs to VXLANs and handles the VXLAN encapsulation and decapsulation so that the non-virtualized resources do not need to support the VXLAN protocol. In some embodiments, ND 112 can be a gateway network device coupling each network device (e.g., host) of data center 110 with external networks (e.g., other data centers, enterprise networks, VLANs, Internet, etc.) through an IP/MPLS network 104. Sitting at the edge of the data center network, the gateway device functions as a data center gateway— providing the interface to the IP/MPLS WAN for interworking Layer 2 and Layer 3 VPN services to remote centers and branch locations. These services provide seamless connectivity between multiple data centers on the same or different IP subnets using the same or different Layer 2 or Layer 3 encapsulation mechanisms. It also enables full integration of data center and VPN services for seamless connectivity between data center and branch locations. In some embodiments, ND 112 can be a Top of the Rack (ToR)/access switch or a switch higher up in the topology of data center 110 (e.g., it can be a core or Wide Area Network (WAN) edge device). For example, ND 112 can be a provider edge (PE) router that terminates VXLAN tunnels on a hybrid cloud environment.

[0037] A dedicated EVPN instance is created (EVI_0). The EVPN instance is associated with a unique VXLAN network identifier (VNI_0). VNI_0 is unique in the entire network such that there is no VNI overlap and hence no overlap of L2 domains. As will be discussed in further details below, the unique VNI_0 is used for forwarding keep-alive packets of a VXLAN tunnel between two network devices. Thus there is a one to one mapping between EVI_0 and VNI_0. Thus at a configuration phase of the present invention, the network devices ND 112, ND 111 A are configured with the Layer 2 domain identified with VNI_0. This L2 domain is not added to any Layer 3 (L3) domain forwarding tables such that it in total isolation with respect to other Layer 2 and Layer 3 domains configured at the network devices. A MAC and IP address are assigned to the ND 111 A within the domain EVI_0 (e.g., MACOl and IP01 (115A)). In some embodiments, the MAC and IP addresses can be randomly assigned to the network device as the domain is isolated from other L2 and L3 domains therefore there would not be any MAC/IP address overlap. That is, the [IP, MAC] pair has to be unique only within the isolated L2 domain. The MAC and IP addresses within the EVI_0 domain are defined separately from the MAC and IP address of the device within the underlay physical network. As illustrated in Figure

1 A, each of the NDs 11 lA-11 IN is assigned a couple [MACOX, IPOX] 115A-115N within the VNI_0 and a couple of addresses [MAC_X, IP_X] 114A-114N for the underlay physical network.

[0038] At operation (131a), ND 112 receives an advertisement message indicating a route between ND 111A and ND 112 within VNI_0. In some embodiments, the advertisement message is a BGP-EVPN MAC/IP Advertisement route message (i.e., BGP-EVPN Route Type -

2 (RT-2)) as illustrated with reference to Figure 2A. Figure 2A illustrates an exemplary advertisement message indicating a route between a first and a second device within a VXLAN segment, in accordance with some embodiments. The advertisement message 240 is used to announce a unique BGP-EVPN [IP-MAC] route for each ND 111 A-N to ND 112 in the EVI_0 domain. The MAC and IP address of each one of the NDs 111 A-N is advertised within a message 240. The message 240 includes a route distinguisher 241 (this is configured to be unique at the creation of the EVPN instance), an Ethernet Segment identifier 242 (this field may be configured to zero as the routes are not advertised beyond the ND 112), an Ethernet Tag ID 243 (which includes VNI_0), a field 244 that indicates the length of the MAC address, a MAC address field 245 (which includes the MAC address of ND 111 A within the VNI_0, i.e., MACOl). The message 240 further includes a field 246 that indicates the length of the IP address, an IP address field 247 (which includes the IP address of ND 111 A within the VNI_0, i.e.JPOl). The message 240 also includes a field 248 for indicating the type of route described within the message, and the next hop (NH) for the route, which is ND 111 A.

[0039] When the ND 112 is coupled with N network devices ND 111A-N, a route is advertised for each one of the devices with corresponding [IP;, MAG] 115i per device. The couple [IP;, MAC;] is unique for each ND within the domain VNI_0. This route is used exclusively for transmission of keep alive packets (requests/responses). Thus, the message 240 advertises a fake BGP route. It advertises a non-existent pair of MAC and IP addresses (i.e., the pair [IP;, MAC;], e.g., [MACOl, IP01] for ND 111A) for every ND 111A-N towards the ND 112 (in other words a route that is not used to transmit data between the two NDs). Once the message 240 is received at ND 112, it updates (at operation 131b) a forwarding table associated with the network identifier (VNI_0) that identifies a VXLAN segment to include an entry for the address of ND 111A within the VNI_0 (i.e., [MACOl, IP01]). When ND 112 is coupled with N NDs 111 A-N, it receives N route advertisement messages (240) and updates its forwarding table accordingly. Thus, ND 112 would have N number of EVPN L2 routes in its Forwarding Information Base (FIB), each pointing to a VXLAN towards the corresponding ND.

[0040] In some embodiments, the advertisement message 240 is received from a network controller. The network controller can be implemented according to a centralized routing approach or alternatively according to a distributed routing approach. In one embodiment, the network controller is implemented as described with Control Plane device 504 of Figure 5. In other embodiments, the advertisement message is received from a route processor (e.g., virtual route processor vRP) operative to implement a BGP speaker and to advertise BGP-EVPN MAC/IP routes to ND 112, the route processor being part of the control plane of the network devices (e.g., ND 111A-N, ND 112).

[0041] Figure IB illustrates a block diagram of an exemplary network according to a centralized routing approach, operative to installing routes for monitoring of VXLAN tunnels in accordance with some embodiments. The BGP speaker (e.g., network controller 117) transmits advertisement messages for installing route 1 to route N at the network device 112 (operation 130). Each of the routes identifies the destination of the route (NH = ND111 A to NH=ND 11 IN), the prefix to use for looking up the forwarding table ( at least one of [IP;, MAC;] , e.g., [IP01, MACOl],... [IP0N, MACON]). The network controller can further be used to configure the NDs 111A-111N.

[0042] The routes for [IP;, MAC;] are installed in addition to regular VXLAN routes that are used for forwarding traffic. For example, ND 111 may be assigned another pair (116A) of addresses [IP 11 , MAC11] defined within a second VXLAN segment (e.g., VNI_1) between ND 111 A and ND 112 and that is used for transmission of regular data between these two NDs.

[0043] Once the route is installed at ND 112, ND 111 A starts sending (operation 132) periodic VXLAN Keep- Alive Requests on its VXLAN tunnel towards ND 112. At operation 133, ND 112 receives a first packet (103 A) which was transmitted by ND 111 A. In response to receiving the packet 103A, ND 112 retrieves (at operation 134), from the forwarding table 113 and based upon an inner destination address, a next hop that is the first network device ND 111 A. In some embodiments, the first packet is a keep alive request and is described in further details with reference to Figure 2B. Figure 2B illustrates an exemplary packet for transmitting a keep-alive request from a first network device to a second network devices through a VXLAN tunnel, in accordance with some embodiments. The packet 103 A is transmitted from the ND 111 A to ND 112 and represents a keep-alive request for the VXLAN tunnel between the two network devices. The packet 103 A includes an outer IP header 202 including the Dst IP address

(Destination IP address) as the IP address of ND 112 (IP_2), a Src IP address (source IP address) as the IP address of the originating ND 111A (IP_A), and IP protocol that is UDP. The packet further includes a UDP header 204, which indicates that VXLAN is the overlay network, and a VXLAN header 206 indicating that the VXLAN segment for this packet is VNI_0. The packet then includes an inner Layer 2 Header 208, with a destination MAC address being the address of the ND 111 A within the VXLAN domain VNI_0 (MACOl); and an inner IP header 210, which includes a destination IP address (DIP) being the address of the ND 111 A within the VXLAN domain VNI_0 (IP01). The packet 103A further includes a payload 212. In some embodiments, the payload 212 is a dummy payload (that is it does not include any data or relevant information) and is therefore ignored by ND 112. In other embodiments, the payload 212 may include data that can be used at the ND 111 A in the mechanism of tunnel status verification. For example, the payload may include a sequence number or a time stamp that identifies the keep alive request 103A. For example, the payload 212 may include a sequence number generated by ND 111A that identifies the packet 103 A. In other embodiments, the payload 212 may include a time stamp that is generated at ND 111 A at the time of transmission of packet 103 A.

[0044] At operation 134, in response to receiving the packet 103 A, ND 112 retrieves, from the forwarding table 113 and based upon the inner destination address, a next hop that is the first network device. For example, and with reference to Figure 2B, upon receipt of the packet 103 A, ND 112 decapsulates the outer IP header 202 and determines that the packet is to be forwarded based on the VXLAN port. The VXLAN packet 216 is then forwarded at the ND 112, based on the forwarding table associated with VNI_0, identified in VXLAN header 206. An entry associated with at least one of the inner addresses (DMAC = MACOl of inner Layer 2 header 208 or DIP address = IP01 of inner IP header 210) of the EVPN forwarding table of EVI_0 associated with VNI_0 is looked-up. As per the route installed for these two addresses, at operation 131b, the next hop for this entry is ND 111A. Therefore, at operation 135, ND 112 transmits based upon the identified next hop a second packet 105 A towards the first network device through the VXLAN tunnel.

[0045] The second packet 105 A is a response to the keep alive request 103 A received from the ND 111 A. Figure 2C illustrates an exemplary packet for transmitting a keep-alive response from the second network device to the first network devices through a VXLAN tunnel, in accordance with some embodiments. The packet 105 A is transmitted from the ND 112 to ND 111 A in response to the receipt of the packet 103 A and as a result of the forwarding of the packet 103 A at ND 112. Packet 105 A represents a keep-alive response for the VXLAN tunnel between the two network devices. The packet 105A includes an outer IP header 222 including the Dst IP address (Destination IP address) as the IP address of ND 111A (IP_A), a Src IP address (source IP address) as the IP address of the originating ND 112 (IP_2), and IP protocol that is UDP. The packet further includes a UDP header 224, which indicates that VXLAN is the overlay network, and a VXLAN header 226 indicating that the VXLAN segment for this packet is VNI_0. The packet then includes an inner Layer 2 Header 228, with a destination MAC address being the address of the ND 111 A within the VXLAN domain VNI_0 (MACOl); and an inner IP header 230, which includes a destination IP address (DIP) being the address of the ND 111 A within the VXLAN domain VNI_0 (IP01).

[0046] The packet 105 A further includes a payload 232. In some embodiments, the payload 232 is a dummy payload (that is it does not include any data or relevant information) and is therefore ignored by ND 111 A. In other embodiments, the payload 232 may include data that can be used at the ND 111 A in the mechanism of tunnel status verification. For example, the payload may include a sequence number or a time stamp that identifies the keep alive request 103 A and response 105 A. For example, the payload 232 may include the sequence number that was transmitted within the payload of the packet 103A. In other embodiments, the payload 232 may include a time stamp that was generated at ND 111 A at the time of transmission of packet 103 A. In some embodiments, in addition or alternatively to the time stamp from ND 111 A, the payload may include a time stamp added by ND 112 at the time of transmission of the packet 105 A towards ND 111A.

[0047] At operation 136, ND 111 A determines whether the VXLAN tunnel is active based upon the second packet 105A. In some embodiments, ND 111 A is operative to use the received packet 105 A to determine whether the VXLAN tunnel between ND 112 and ND 111 A is active or inactive (i.e., down). When the packet 105A arrives at ND 111A, the VNI_0 and inner MAC [MACOl] indicate to ND 111 A that packet 105 A is a response to a keep alive packet previously transmitted to ND 112. The determination that the VXLAN tunnel is still active can be performed via various mechanisms. In one exemplary embodiment, the ND 111 A can determine that the VXLAN tunnel is active upon receipt of the packet 105 A. In another embodiment, the ND 111A can determine that the VXLAN tunnel is active upon receipt of the packet 105A within a predefined period of time from the time the packet 103 A was sent. In other embodiments, ND 111 A can determine that the tunnel is active or down based on a number of received packets vs. a number of packets sent. For example, ND 111 A may be configured with a threshold value of a ratio of received packets vs. sent packets; if this threshold value is not attained within a given time period, the ND 111 A determines that the VXLAN tunnel is down. Alternatively if the ratio of actual packets received vs. packet sent is above the threshold, the ND 111 A determines that the VXLAN tunnel is active. In some embodiments, this

determination can be performed based on the data stored in the payload 232. For example, ND 111 A may store sequence numbers of each keep-alive request sent to ND 112 and determines upon receipt of the response (e.g., 105A) that the response includes the sequence number.

[0048] The embodiments of the present invention provide a scalable mechanism for VXLAN tunnel monitoring in a BGP-EVPN network. The embodiments enable monitoring of a VXLAN tunnel status. The tunnel's status can serve to provide High Availability networking (e.g., in the presence of multiple data center gateways). The tunnel's status can additionally or alternatively be used to generate an alarm such that the network underlay provider can better manage the underlay connectivity and remedy any network failures. Further, the embodiments do not require a specific configuration at the endpoints of the tunnel. As opposed to standard approaches which need substantial storage and processing resources for state tracking of BFD sessions, the embodiments of the present invention present a stateless mechanism that scales well for enabling monitoring of millions of VXLAN tunnels at a given network device. The embodiments are scalable on the network device receiving the requests (e.g., a gateway device of a cloud data center), even when it has very high fan-out as the embodiments involve only a look up of the Forwarding Information Base (FIB) at the network device. Further the embodiments, are applicable in both centralized routing approaches as well as distributed routing.

[0049] The operations in the flow diagrams will be described with reference to the exemplary embodiments of the other figures. However, it should be understood that the operations of the flow diagrams can be performed by embodiments of the invention other than those discussed with reference to the other figures, and the embodiments of the invention discussed with reference to these other figures can perform operations different than those discussed with reference to the flow diagrams.

[0050] Figure 3A illustrates exemplary operations for forwarding a keep-alive response packet in response to receiving a keep-alive request from a first network device in accordance with some embodiments. At operation 300, the network device (e.g., ND 112) receives an announcement message indicating a route from the first network device towards the second network device through the VXLAN tunnel and within the VXLAN segment. In some embodiments, the advertisement message is a BGP-EVPN MAC/IP Advertisement route message as illustrated with reference to Figure 2A. For example, the advertisement message 240 is used to announce a unique BGP-EVPN [IP-MAC] route for each ND 111 A-N to ND 112 in the EVI_0 domain. The MAC and IP [MACi, IPi] address of each one of the NDs 111 A-N is advertised within a message 240. In some embodiments, the advertisement message 240 is received from a network controller. The network controller can be implemented according to a centralized routing approach or alternatively according to a distributed routing approach. In one embodiment, the network controller is implemented as described with Control Plane device 504 of Figure 5. In other embodiments, the advertisement message is received from a route processor (e.g., virtual route processor vRP) operative to implement a BGP speaker and to advertise BGP- EVPN MAC/IP routes to ND 112, the route processor being part of the control plane of the network devices (e.g., ND 111A-N, ND 112).

[0051] Flow then moves to operation 302, at which ND 112 updates a forwarding table associated with the network identifier to include an entry for the inner destination address. For example, ND 112 updates a forwarding table associated with the network identifier (VNI_0) that identifies the VXLAN segment to include an entry for the address of ND 111 A within the VNI_0 (i.e., [MACOl, IP01]). When ND 112 is coupled with N NDs 111 A-N, it receives N route advertisement messages (240) and updates its forwarding table accordingly. Thus, ND 112 would have N number of EVPN L2 routes in its Forwarding Information Base (FIB), each pointing to a VXLAN towards the corresponding ND.

[0052] Flow then moves to operation 304 at which, the network device (ND 112) receives, from the first network device (ND 111 A) through the VXLAN tunnel, a first packet (103 A). The first packet (103 A) includes a network identifier which identifies a VXLAN segment (VNI_0) and an inner destination address [MACOl, IP01] that identifies the first network device within the VXLAN segment.

[0053] Flow then moves to operation 306, at which the network device decapsulates the packet to obtain the inner packet (e.g., inner packet 214). Flow then moves to operation 308, at which the network device (e.g., ND 112) forwards the inner header based upon the inner destination address (e.g., MACOl or IP01). Flow moves to operation 310, at which a next hop is identified in the forwarding table based upon the inner destination address (of the inner IP header, or inner Layer 2 header), where the next hop is identified to be the first network device from which the first packet was received, consequently resulting in a packet being transmitted in response to the first packet received.

[0054] At operation 312, in response to receiving the first packet, the network device transmits, based upon the inner destination address and the network identifier, a second packet towards the first network device through the VXLAN tunnel. The transmission of the second packet causes the first network device to determine whether the VXLAN tunnel is active.

[0055] Figure 3B illustrates exemplary operations for forwarding a keep-alive request packet in accordance with some embodiments. At operation 322, a network device (one of NDs 111 A- N) transmits a first packet (103 A) that includes a network identifier (VNI_0) which identifies a VXLAN segment and inner destination address that identifies the first network device within the VXLAN segment (e.g., [MACOl, IP01] of ND 111A within VXLAN domain VNI_0). The first packet is representative of keep-alive request sent from the first network device in order to obtain a status of the VXLAN tunnel. Flow then moves to operation 324, at which in some embodiments, the network device receives through the VXLAN tunnel, a second packet including the network identifier. The second packet (e.g., 105A) is received in response to the transmission of the first packet and as a response to the keep alive request. In some

embodiments, if the VXLAN tunnel is down (e.g., if the physical underlay network is experiencing faults) the second packet is not received at the first network device.

[0056] Flow then moves to operation 326, at which the ND determines whether the VXLAN tunnel is active. Thus upon determination that a valid response (e.g., 105 A) is received at the network device in response to the keep-alive request (packet 103 A) sent, the ND determines that the VXLAN tunnel is active (operation 328). Alternatively, upon determination that an invalid response (e.g., 105A) is received at the network device in response to the keep-alive request (packet 103 A) sent, or that the response has not been received the ND determines that the VXLAN tunnel is down (operation 330).

[0057] The determination that the VXLAN tunnel is still active can be performed via various mechanisms. In one exemplary embodiment, the ND 111 A can determine that the VXLAN tunnel is active upon receipt of the packet 105 A. In another embodiment, the ND 111 A can determine that the VXLAN tunnel is active upon receipt (operation 327) of the packet 105 A within a predefined period of time from the time the packet 103 A was sent. In other

embodiments, ND 111A can determine (operation 329) that the tunnel is active or down based on a number of received packets vs. a number of packets sent. For example, ND 111 A may be configured with a threshold value of a ratio of received packets vs. sent packets, if this threshold value is not attained within a given time period, the ND 111 A determines that the VXLAN tunnel is down. Alternatively if the ratio of actual packets received vs. packet sent is above the threshold, the ND 111 A determines that the VXLAN tunnel is active. In some embodiments, this determination can be performed based on the data stored in the payload 232. For example, ND 111 A may store sequence numbers of each keep-alive request sent to ND 112 and determines upon receipt of the response (e.g., 105A) that the response includes the sequence number.

[0058] Architecture:

[0059] Figure 4A illustrates connectivity between network devices (NDs) within an exemplary network, as well as three exemplary implementations of the NDs, according to some embodiments of the invention. Figure 4A shows NDs 400A-H, and their connectivity by way of lines between 400A-400B, 400B-400C, 400C-400D, 400D-400E, 400E-400F, 400F-400G, and 400A-400G, as well as between 400H and each of 400A, 400C, 400D, and 400G. These NDs are physical devices, and the connectivity between these NDs can be wireless or wired (often referred to as a link). An additional line extending from NDs 400A, 400E, and 400F illustrates that these NDs act as ingress and egress points for the network (and thus, these NDs are sometimes referred to as edge NDs; while the other NDs may be called core NDs).

[0060] Two of the exemplary ND implementations in Figure 4 A are: 1) a special-purpose network device 402 that uses custom application-specific integrated-circuits (ASICs) and a special-purpose operating system (OS); and 2) a general purpose network device 404 that uses common off-the-shelf (COTS) processors and a standard OS.

[0061] The special-purpose network device 402 includes networking hardware 410 comprising compute resource(s) 412 (which typically include a set of one or more processors), forwarding resource(s) 414 (which typically include one or more ASICs and/or network processors), and physical network interfaces (NIs) 416 (sometimes called physical ports), as well as non- transitory machine readable storage media 418 having stored therein networking software 420. A physical NI is hardware in a ND through which a network connection (e.g., wirelessly through a wireless network interface controller (WNIC) or through plugging in a cable to a physical port connected to a network interface controller (NIC)) is made, such as those shown by the connectivity between NDs 400A-H. During operation, the networking software 420 may be executed by the networking hardware 410 to instantiate a set of one or more networking software instance(s) 422. Each of the networking software instance(s) 422, and that part of the networking hardware 410 that executes that network software instance (be it hardware dedicated to that networking software instance and/or time slices of hardware temporally shared by that networking software instance with others of the networking software instance(s) 422), form a separate virtual network element 430A-R. Each of the virtual network element(s) (VNEs) 430A-R includes a control communication and configuration module 432A-R (sometimes referred to as a local control module or control communication module) and forwarding table(s) 434A-R, such that a given virtual network element (e.g., 430A) includes the control communication and configuration module (e.g., 432A), a set of one or more forwarding table(s) (e.g., 434A), and that portion of the networking hardware 410 that executes the virtual network element (e.g., 430A). The networking software includes VXLAN monitor 423, which when instantiated as on or more instances 433A-R cause the network device 402 to perform the operations described with reference to Figures 1A-3B.

[0062] The special-purpose network device 402 is often physically and/or logically considered to include: 1) a ND control plane 424 (sometimes referred to as a control plane) comprising the compute resource(s) 412 that execute the control communication and configuration module(s) 432A-R; and 2) a ND forwarding plane 426 (sometimes referred to as a forwarding plane, a data plane, or a media plane) comprising the forwarding resource(s) 414 that utilize the forwarding table(s) 434A-R and the physical NIs 416. By way of example, where the ND is a router (or is implementing routing functionality), the ND control plane 424 (the compute resource(s) 412 executing the control communication and configuration module(s) 432A-R) is typically responsible for participating in controlling how data (e.g., packets) is to be routed (e.g., the next hop for the data and the outgoing physical NI for that data) and storing that routing information in the forwarding table(s) 434A-R, and the ND forwarding plane 426 is responsible for receiving that data on the physical NIs 416 and forwarding that data out the appropriate ones of the physical NIs 416 based on the forwarding table(s) 434A-R.

[0063] Figure 4B illustrates an exemplary way to implement the special-purpose network device 402 according to some embodiments of the invention. Figure 4B shows a special- purpose network device including cards 438 (typically hot pluggable). While in some embodiments the cards 438 are of two types (one or more that operate as the ND forwarding plane 426 (sometimes called line cards), and one or more that operate to implement the ND control plane 424 (sometimes called control cards)), alternative embodiments may combine functionality onto a single card and/or include additional card types (e.g., one additional type of card is called a service card, resource card, or multi-application card). A service card can provide specialized processing (e.g., Layer 4 to Layer 7 services (e.g., firewall, Internet Protocol Security (IPsec), Secure Sockets Layer (SSL) / Transport Layer Security (TLS), Intrusion Detection System (IDS), peer-to-peer (P2P), Voice over IP (VoIP) Session Border Controller, Mobile Wireless Gateways (Gateway General Packet Radio Service (GPRS) Support Node (GGSN), Evolved Packet Core (EPC) Gateway)). By way of example, a service card may be used to terminate IPsec tunnels and execute the attendant authentication and encryption algorithms. These cards are coupled together through one or more interconnect mechanisms illustrated as backplane 436 (e.g., a first full mesh coupling the line cards and a second full mesh coupling all of the cards).

[0064] Returning to Figure 4A, the general purpose network device 404 includes hardware 440 comprising a set of one or more processor(s) 442 (which are often COTS processors) and network interface controller(s) 444 (NICs; also known as network interface cards) (which include physical NIs 446), as well as non-transitory machine readable storage media 448 having stored therein software 450. During operation, the processor(s) 442 execute the software 450 to instantiate one or more sets of one or more applications 464A-R. While one embodiment does not implement virtualization, alternative embodiments may use different forms of virtualization. For example, in one such alternative embodiment the virtualization layer 454 represents the kernel of an operating system (or a shim executing on a base operating system) that allows for the creation of multiple instances 462A-R called software containers that may each be used to execute one (or more) of the sets of applications 464A-R; where the multiple software containers (also called virtualization engines, virtual private servers, or jails) are user spaces (typically a virtual memory space) that are separate from each other and separate from the kernel space in which the operating system is run; and where the set of applications running in a given user space, unless explicitly allowed, cannot access the memory of the other processes. In another such alternative embodiment the virtualization layer 454 represents a hypervisor (sometimes referred to as a virtual machine monitor (VMM)) or a hypervisor executing on top of a host operating system, and each of the sets of applications 464A-R is run on top of a guest operating system within an instance 462A-R called a virtual machine (which may in some cases be considered a tightly isolated form of software container) that is run on top of the hypervisor - the guest operating system and application may not know they are running on a virtual machine as opposed to running on a "bare metal" host electronic device, or through para-virtualization the operating system and/or application may be aware of the presence of virtualization for optimization purposes. In yet other alternative embodiments, one, some or all of the applications are implemented as unikernel(s), which can be generated by compiling directly with an application only a limited set of libraries (e.g., from a library operating system (LibOS) including drivers/libraries of OS services) that provide the particular OS services needed by the application. As a unikernel can be implemented to run directly on hardware 440, directly on a hypervisor (in which case the unikernel is sometimes described as running within a LibOS virtual machine), or in a software container, embodiments can be implemented fully with unikernels running directly on a hypervisor represented by virtualization layer 454, unikernels running within software containers represented by instances 462A-R, or as a combination of unikernels and the above-described techniques (e.g. , unikernels and virtual machines both run directly on a hypervisor, unikernels and sets of applications that are run in different software containers). In some embodiments, the software 450 includes VXLAN monitor 453, which when instantiated as on or more instances cause the network device 404 to perform the operations described with reference to Figures 1A-3B.

[0065] The instantiation of the one or more sets of one or more applications 464A-R, as well as virtualization if implemented, are collectively referred to as software instance(s) 452. Each set of applications 464A-R, corresponding virtualization construct (e.g., instance 462A-R) if implemented, and that part of the hardware 440 that executes them (be it hardware dedicated to that execution and/or time slices of hardware temporally shared), forms a separate virtual network element(s) 460A-R.

[0066] The virtual network element(s) 460A-R perform similar functionality to the virtual network element(s) 430A-R - e.g., similar to the control communication and configuration module(s) 432A and forwarding table(s) 434A (this virtualization of the hardware 440 is sometimes referred to as network function virtualization (NFV)). Thus, NFV may be used to consolidate many network equipment types onto industry standard high volume server hardware, physical switches, and physical storage, which could be located in Data centers, NDs, and customer premise equipment (CPE). While embodiments of the invention are illustrated with each instance 462A-R corresponding to one VNE 460A-R, alternative embodiments may implement this correspondence at a finer level granularity (e.g., line card virtual machines virtualize line cards, control card virtual machine virtualize control cards, etc.); it should be understood that the techniques described herein with reference to a correspondence of instances 462A-R to VNEs also apply to embodiments where such a finer level of granularity and/or unikernels are used.

[0067] In certain embodiments, the virtualization layer 454 includes a virtual switch that provides similar forwarding services as a physical Ethernet switch. Specifically, this virtual switch forwards traffic between instances 462A-R and the NIC(s) 444, as well as optionally between the instances 462A-R; in addition, this virtual switch may enforce network isolation between the VNEs 460A-R that by policy are not permitted to communicate with each other (e.g., by honoring virtual local area networks (VLANs) or virtual extensible local area networks (VXLANs)).

[0068] The third exemplary ND implementation in Figure 4A is a hybrid network device 406, which includes both custom ASICs/special-purpose OS and COTS processors/standard OS in a single ND or a single card within an ND. In certain embodiments of such a hybrid network device, a platform VM (i.e., a VM that that implements the functionality of the special-purpose network device 402) could provide for para-virtualization to the networking hardware present in the hybrid network device 406.

[0069] Regardless of the above exemplary implementations of an ND, when a single one of multiple VNEs implemented by an ND is being considered (e.g., only one of the VNEs is part of a given virtual network) or where only a single VNE is currently being implemented by an ND, the shortened term network element (NE) is sometimes used to refer to that VNE. Also in all of the above exemplary implementations, each of the VNEs (e.g., VNE(s) 430A-R, VNEs 460A-R, and those in the hybrid network device 406) receives data on the physical NIs (e.g., 416, 446) and forwards that data out the appropriate ones of the physical NIs (e.g., 416, 446). For example, a VNE implementing IP router functionality forwards IP packets on the basis of some of the IP header information in the IP packet; where IP header information includes source IP address, destination IP address, source port, destination port (where "source port" and

"destination port" refer herein to protocol ports, as opposed to physical ports of a ND), transport protocol (e.g., user datagram protocol (UDP), Transmission Control Protocol (TCP), and differentiated services code point (DSCP) values.

[0070] The NDs of Figure 4A, for example, may form part of the Internet or a private network; and other electronic devices (not shown; such as end user devices including workstations, laptops, netbooks, tablets, palm tops, mobile phones, smartphones, phablets, multimedia phones, Voice Over Internet Protocol (VOIP) phones, terminals, portable media players, GPS units, wearable devices, gaming systems, set-top boxes, Internet enabled household appliances) may be coupled to the network (directly or through other networks such as access networks) to communicate over the network (e.g., the Internet or virtual private networks (VPNs) overlaid on (e.g., tunneled through) the Internet) with each other (directly or through servers) and/or access content and/or services. Such content and/or services are typically provided by one or more servers (not shown) belonging to a service/content provider or one or more end user devices (not shown) participating in a peer-to-peer (P2P) service, and may include, for example, public webpages (e.g., free content, store fronts, search services), private webpages (e.g.,

username/password accessed webpages providing email services), and/or corporate networks over VPNs. For instance, end user devices may be coupled (e.g., through customer premise equipment coupled to an access network (wired or wirelessly)) to edge NDs, which are coupled (e.g., through one or more core NDs) to other edge NDs, which are coupled to electronic devices acting as servers. However, through compute and storage virtualization, one or more of the electronic devices operating as the NDs in Figure 4A may also host one or more such servers (e.g., in the case of the general purpose network device 404, one or more of the software instances 462A-R may operate as servers; the same would be true for the hybrid network device 406; in the case of the special-purpose network device 402, one or more such servers could also be run on a virtualization layer executed by the compute resource(s) 412); in which case the servers are said to be co-located with the VNEs of that ND.

[0071] A virtual network is a logical abstraction of a physical network (such as that in Figure 4A) that provides network services (e.g., L2 and/or L3 services). A virtual network can be implemented as an overlay network (sometimes referred to as a network virtualization overlay) that provides network services (e.g., layer 2 (L2, data link layer) and/or layer 3 (L3, network layer) services) over an underlay network (e.g., an L3 network, such as an Internet Protocol (IP) network that uses tunnels (e.g., generic routing encapsulation (GRE), layer 2 tunneling protocol (L2TP), IPSec) to create the overlay network).

[0072] A network virtualization edge (NVE) sits at the edge of the underlay network and participates in implementing the network virtualization; the network-facing side of the NVE uses the underlay network to tunnel frames to and from other NVEs; the outward-facing side of the NVE sends and receives data to and from systems outside the network. A virtual network instance (VNI) is a specific instance of a virtual network on a NVE (e.g., a NE/VNE on an ND, a part of a NE/VNE on a ND where that NE/VNE is divided into multiple VNEs through emulation); one or more VNIs can be instantiated on an NVE (e.g., as different VNEs on an ND). A virtual access point (VAP) is a logical connection point on the NVE for connecting external systems to a virtual network; a VAP can be physical or virtual ports identified through logical interface identifiers (e.g., a VLAN ID).

[0073] Examples of network services include: 1) an Ethernet LAN emulation service (an Ethernet-based multipoint service similar to an Internet Engineering Task Force (IETF) Multiprotocol Label Switching (MPLS) or Ethernet VPN (EVPN) service) in which external systems are interconnected across the network by a LAN environment over the underlay network (e.g., an NVE provides separate L2 VNIs (virtual switching instances) for different such virtual networks, and L3 (e.g., IP/MPLS) tunneling encapsulation across the underlay network); and 2) a virtualized IP forwarding service (similar to IETF IP VPN (e.g., Border Gateway Protocol (BGP)/MPLS IP VPN) from a service definition perspective) in which external systems are interconnected across the network by an L3 environment over the underlay network (e.g., an NVE provides separate L3 VNIs (forwarding and routing instances) for different such virtual networks, and L3 (e.g., IP/MPLS) tunneling encapsulation across the underlay network)). Network services may also include quality of service capabilities (e.g., traffic classification marking, traffic conditioning and scheduling), security capabilities (e.g., filters to protect customer premises from network - originated attacks, to avoid malformed route announcements), and management capabilities (e.g., full detection and processing).

[0074] Fig. 4C illustrates a network with a single network element on each of the NDs of Figure 4A, and within this straight forward approach contrasts a traditional distributed approach (commonly used by traditional routers) with a centralized approach for maintaining reachability and forwarding information (also called network control), according to some embodiments of the invention. Specifically, Figure 4C illustrates network elements (NEs) 470A-H with the same connectivity as the NDs 400A-H of Figure 4A.

[0075] Figure 4C illustrates that the distributed approach 472 distributes responsibility for generating the reachability and forwarding information across the NEs 470A-H; in other words, the process of neighbor discovery and topology discovery is distributed.

[0076] For example, where the special-purpose network device 402 is used, the control communication and configuration module(s) 432A-R of the ND control plane 424 typically include a reachability and forwarding information module to implement one or more routing protocols (e.g., an exterior gateway protocol such as Border Gateway Protocol (BGP), Interior Gateway Protocol(s) (IGP) (e.g., Open Shortest Path First (OSPF), Intermediate System to Intermediate System (IS-IS), Routing Information Protocol (RIP), Label Distribution Protocol (LDP), Resource Reservation Protocol (RSVP) (including RSVP-Traffic Engineering (TE): Extensions to RSVP for LSP Tunnels and Generalized Multi-Protocol Label Switching

(GMPLS) Signaling RSVP-TE)) that communicate with other NEs to exchange routes, and then selects those routes based on one or more routing metrics. Thus, the NEs 470A-H (e.g., the compute resource(s) 412 executing the control communication and configuration module(s) 432A-R) perform their responsibility for participating in controlling how data (e.g., packets) is to be routed (e.g., the next hop for the data and the outgoing physical NI for that data) by distributively determining the reachability within the network and calculating their respective forwarding information. Routes and adjacencies are stored in one or more routing structures (e.g., Routing Information Base (RIB), Label Information Base (LIB), one or more adjacency structures) on the ND control plane 424. The ND control plane 424 programs the ND forwarding plane 426 with information (e.g., adjacency and route information) based on the routing structure(s). For example, the ND control plane 424 programs the adjacency and route information into one or more forwarding table(s) 434A-R (e.g., Forwarding Information Base (FIB), Label Forwarding Information Base (LFIB), and one or more adjacency structures) on the ND forwarding plane 426. For layer 2 forwarding, the ND can store one or more bridging tables that are used to forward data based on the layer 2 information in that data. While the above example uses the special-purpose network device 402, the same distributed approach 472 can be implemented on the general purpose network device 404 and the hybrid network device 406.

[0077] Figure 4C illustrates that a centralized approach 474 (also known as software defined networking (SDN)) that decouples the system that makes decisions about where traffic is sent from the underlying systems that forwards traffic to the selected destination. The illustrated centralized approach 474 has the responsibility for the generation of reachability and forwarding information in a centralized control plane 476 (sometimes referred to as a SDN control module, controller, network controller, OpenFlow controller, SDN controller, control plane node, network virtualization authority, or management control entity), and thus the process of neighbor discovery and topology discovery is centralized. The centralized control plane 476 has a south bound interface 482 with a data plane 480 (sometime referred to the infrastructure layer, network forwarding plane, or forwarding plane (which should not be confused with a ND forwarding plane)) that includes the NEs 470A-H (sometimes referred to as switches, forwarding elements, data plane elements, or nodes). The centralized control plane 476 includes a network controller 478, which includes a centralized reachability and forwarding information module 479 that determines the reachability within the network and distributes the forwarding information to the NEs 470A-H of the data plane 480 over the south bound interface 482 (which may use the OpenFlow protocol). Thus, the network intelligence is centralized in the centralized control plane 476 executing on electronic devices that are typically separate from the NDs.

[0078] For example, where the special-purpose network device 402 is used in the data plane 480, each of the control communication and configuration module(s) 432A-R of the ND control plane 424 typically include a control agent that provides the VNE side of the south bound interface 482. In this case, the ND control plane 424 (the compute resource(s) 412 executing the control communication and configuration module(s) 432A-R) performs its responsibility for participating in controlling how data (e.g., packets) is to be routed (e.g., the next hop for the data and the outgoing physical NI for that data) through the control agent communicating with the centralized control plane 476 to receive the forwarding information (and in some cases, the reachability information) from the centralized reachability and forwarding information module 479 (it should be understood that in some embodiments of the invention, the control communication and configuration module(s) 432A-R, in addition to communicating with the centralized control plane 476, may also play some role in determining reachability and/or calculating forwarding information - albeit less so than in the case of a distributed approach; such embodiments are generally considered to fall under the centralized approach 474, but may also be considered a hybrid approach).

[0079] While the above example uses the special-purpose network device 402, the same centralized approach 474 can be implemented with the general purpose network device 404 (e.g., each of the VNE 460A-R performs its responsibility for controlling how data (e.g., packets) is to be routed (e.g., the next hop for the data and the outgoing physical NI for that data) by communicating with the centralized control plane 476 to receive the forwarding information (and in some cases, the reachability information) from the centralized reachability and forwarding information module 479; it should be understood that in some embodiments of the invention, the VNEs 460A-R, in addition to communicating with the centralized control plane 476, may also play some role in determining reachability and/or calculating forwarding information - albeit less so than in the case of a distributed approach) and the hybrid network device 406. In fact, the use of SDN techniques can enhance the NFV techniques typically used in the general purpose network device 404 or hybrid network device 406 implementations as NFV is able to support SDN by providing an infrastructure upon which the SDN software can be run, and NFV and SDN both aim to make use of commodity server hardware and physical switches. In some embodiments, the centralized reachability and forwarding information module 479 includes the VXLAN monitoring controller 481, which when instantiated perform the operations described with reference to the network controller in Figures 1A-3B. [0080] Figure 4C also shows that the centralized control plane 476 has a north bound interface 484 to an application layer 486, in which resides application(s) 488. The centralized control plane 476 has the ability to form virtual networks 492 (sometimes referred to as a logical forwarding plane, network services, or overlay networks (with the NEs 470A-H of the data plane 480 being the underlay network)) for the application(s) 488. Thus, the centralized control plane 476 maintains a global view of all NDs and configured NEs/VNEs, and it maps the virtual networks to the underlying NDs efficiently (including maintaining these mappings as the physical network changes either through hardware (ND, link, or ND component) failure, addition, or removal).

[0081] While Figure 4C shows the distributed approach 472 separate from the centralized approach 474, the effort of network control may be distributed differently or the two combined in certain embodiments of the invention. For example: 1) embodiments may generally use the centralized approach (SDN) 474, but have certain functions delegated to the NEs (e.g., the distributed approach may be used to implement one or more of fault monitoring, performance monitoring, protection switching, and primitives for neighbor and/or topology discovery); or 2) embodiments of the invention may perform neighbor discovery and topology discovery via both the centralized control plane and the distributed protocols, and the results compared to raise exceptions where they do not agree. Such embodiments are generally considered to fall under the centralized approach 474, but may also be considered a hybrid approach.

[0082] While Figure 4C illustrates the simple case where each of the NDs 400A-H implements a single NE 470A-H, it should be understood that the network control approaches described with reference to Figure 4C also work for networks where one or more of the NDs 400A-H implement multiple VNEs (e.g., VNEs 430A-R, VNEs 460A-R, those in the hybrid network device 406). Alternatively or in addition, the network controller 478 may also emulate the implementation of multiple VNEs in a single ND. Specifically, instead of (or in addition to) implementing multiple VNEs in a single ND, the network controller 478 may present the implementation of a VNE/NE in a single ND as multiple VNEs in the virtual networks 492 (all in the same one of the virtual network(s) 492, each in different ones of the virtual network(s) 492, or some combination). For example, the network controller 478 may cause an ND to implement a single VNE (a NE) in the underlay network, and then logically divide up the resources of that NE within the centralized control plane 476 to present different VNEs in the virtual network(s) 492 (where these different VNEs in the overlay networks are sharing the resources of the single VNE/NE implementation on the ND in the underlay network).

[0083] While some embodiments of the invention implement the centralized control plane 476 as a single entity (e.g., a single instance of software running on a single electronic device), alternative embodiments may spread the functionality across multiple entities for redundancy and/or scalability purposes (e.g., multiple instances of software running on different electronic devices).

[0084] Similar to the network device implementations, the electronic device(s) running the centralized control plane 476, and thus the network controller 478 including the centralized reachability and forwarding information module 479, may be implemented a variety of ways (e.g., a special purpose device, a general-purpose (e.g., COTS) device, or hybrid device). These electronic device(s) would similarly include compute resource(s), a set or one or more physical NICs, and a non-transitory machine-readable storage medium having stored thereon the centralized control plane software. For instance, Figure 5 illustrates, a general purpose control plane device 504 including hardware 540 comprising a set of one or more processor(s) 542 (which are often COTS processors) and network interface controller(s) 544 (NICs; also known as network interface cards) (which include physical NIs 546), as well as non-transitory machine readable storage media 548 having stored therein centralized control plane (CCP) software 550.

[0085] In embodiments that use compute virtualization, the processor(s) 542 typically execute software to instantiate a virtualization layer 554 (e.g., in one embodiment the virtualization layer 554 represents the kernel of an operating system (or a shim executing on a base operating system) that allows for the creation of multiple instances 562A-R called software containers (representing separate user spaces and also called virtualization engines, virtual private servers, or jails) that may each be used to execute a set of one or more applications; in another embodiment the virtualization layer 554 represents a hypervisor (sometimes referred to as a virtual machine monitor (VMM)) or a hypervisor executing on top of a host operating system, and an application is run on top of a guest operating system within an instance 562A-R called a virtual machine (which in some cases may be considered a tightly isolated form of software container) that is run by the hypervisor ; in another embodiment, an application is implemented as a unikernel, which can be generated by compiling directly with an application only a limited set of libraries (e.g., from a library operating system (LibOS) including dri vers/libraries of OS services) that provide the particular OS services needed by the application, and the unikernel can run directly on hardware 540, directly on a hypervisor represented by virtualization layer 554 (in which case the unikernel is sometimes described as running within a LibOS virtual machine), or in a software container represented by one of instances 562A-R). Again, in embodiments where compute virtualization is used, during operation an instance of the CCP software 550 (illustrated as CCP instance 576A) is executed (e.g., within the instance 562A) on the virtualization layer 554. In embodiments where compute virtualization is not used, the CCP instance 576A is executed, as a unikernel or on top of a host operating system, on the "bare metal" general purpose control plane device 504. The instantiation of the CCP instance 576A, as well as the virtualization layer 554 and instances 562A-R if implemented, are collectively referred to as software instance(s) 552.

[0086] In some embodiments, the CCP instance 576A includes a network controller instance 578. The network controller instance 578 includes a centralized reachability and forwarding information module instance 579 (which is a middleware layer providing the context of the network controller 478 to the operating system and communicating with the various NEs), and an CCP application layer 580 (sometimes referred to as an application layer) over the middleware layer (providing the intelligence required for various network operations such as protocols, network situational awareness, and user - interfaces). At a more abstract level, this CCP application layer 580 within the centralized control plane 476 works with virtual network view(s) (logical view(s) of the network) and the middleware layer provides the conversion from the virtual networks to the physical view. In some embodiments, the CCP software includes the VXLAN monitoring controller 551 , which when instantiated as VXLAN monitoring controller instance 581 performs the operations described with reference to the network controller in Figures 1A-3B.

[0087] The centralized control plane 476 transmits relevant messages to the data plane 480 based on CCP application layer 580 calculations and middleware layer mapping for each flow. A flow may be defined as a set of packets whose headers match a given pattern of bits; in this sense, traditional IP forwarding is also flow-based forwarding where the flows are defined by the destination IP address for example; however, in other implementations, the given pattern of bits used for a flow definition may include more fields (e.g., 10 or more) in the packet headers. Different NDs/NEs/VNEs of the data plane 480 may receive different messages, and thus different forwarding information. The data plane 480 processes these messages and programs the appropriate flow information and corresponding actions in the forwarding tables (sometime referred to as flow tables) of the appropriate NE/VNEs, and then the NEs/VNEs map incoming packets to flows represented in the forwarding tables and forward packets based on the matches in the forwarding tables.

[0088] Standards such as OpenFlow define the protocols used for the messages, as well as a model for processing the packets. The model for processing packets includes header parsing, packet classification, and making forwarding decisions. Header parsing describes how to interpret a packet based upon a well-known set of protocols. Some protocol fields are used to build a match structure (or key) that will be used in packet classification (e.g., a first key field could be a source media access control (MAC) address, and a second key field could be a destination MAC address). [0089] Packet classification involves executing a lookup in memory to classify the packet by determining which entry (also referred to as a forwarding table entry or flow entry) in the forwarding tables best matches the packet based upon the match structure, or key, of the forwarding table entries. It is possible that many flows represented in the forwarding table entries can correspond/match to a packet; in this case the system is typically configured to determine one forwarding table entry from the many according to a defined scheme (e.g., selecting a first forwarding table entry that is matched). Forwarding table entries include both a specific set of match criteria (a set of values or wildcards, or an indication of what portions of a packet should be compared to a particular value/values/wildcards, as defined by the matching capabilities - for specific fields in the packet header, or for some other packet content), and a set of one or more actions for the data plane to take on receiving a matching packet. For example, an action may be to push a header onto the packet, for the packet using a particular port, flood the packet, or simply drop the packet. Thus, a forwarding table entry for IPv4/IPv6 packets with a particular transmission control protocol (TCP) destination port could contain an action specifying that these packets should be dropped.

[0090] Making forwarding decisions and performing actions occurs, based upon the forwarding table entry identified during packet classification, by executing the set of actions identified in the matched forwarding table entry on the packet.

[0091] However, when an unknown packet (for example, a "missed packet" or a "match-miss" as used in OpenFlow parlance) arrives at the data plane 480, the packet (or a subset of the packet header and content) is typically forwarded to the centralized control plane 476. The centralized control plane 476 will then program forwarding table entries into the data plane 480 to accommodate packets belonging to the flow of the unknown packet. Once a specific forwarding table entry has been programmed into the data plane 480 by the centralized control plane 476, the next packet with matching credentials will match that forwarding table entry and take the set of actions associated with that matched entry.

[0092] A network interface (NI) may be physical or virtual; and in the context of IP, an interface address is an IP address assigned to a NI, be it a physical NI or virtual NI. A virtual NI may be associated with a physical NI, with another virtual interface, or stand on its own (e.g., a loopback interface, a point-to-point protocol interface). A NI (physical or virtual) may be numbered (a NI with an IP address) or unnumbered (a NI without an IP address). A loopback interface (and its loopback address) is a specific type of virtual NI (and IP address) of a NE/VNE (physical or virtual) often used for management purposes; where such an IP address is referred to as the nodal loopback address. The IP address(es) assigned to the NI(s) of a ND are referred to as IP addresses of that ND; at a more granular level, the IP address(es) assigned to NI(s) assigned to a NE/VNE implemented on a ND can be referred to as IP addresses of that NE/VNE.

[0093] Each VNE (e.g., a virtual router, a virtual bridge (which may act as a virtual switch instance in a Virtual Private LAN Service (VPLS) is typically independently administrable. For example, in the case of multiple virtual routers, each of the virtual routers may share system resources but is separate from the other virtual routers regarding its management domain, AAA (authentication, authorization, and accounting) name space, IP address, and routing database(s). Multiple VNEs may be employed in an edge ND to provide direct network access and/or different classes of services for subscribers of service and/or content providers.

[0094] Within certain NDs, "interfaces" that are independent of physical NIs may be configured as part of the VNEs to provide higher-layer protocol and service information (e.g., Layer 3 addressing). The subscriber records in the AAA server identify, in addition to the other subscriber configuration requirements, to which context (e.g., which of the VNEs/NEs) the corresponding subscribers should be bound within the ND. As used herein, a binding forms an association between a physical entity (e.g., physical NI, channel) or a logical entity (e.g., circuit such as a subscriber circuit or logical circuit (a set of one or more subscriber circuits)) and a context's interface over which network protocols (e.g., routing protocols, bridging protocols) are configured for that context. Subscriber data flows on the physical entity when some higher- layer protocol interface is configured and associated with that physical entity.

[0095] Some NDs provide support for implementing VPNs (Virtual Private Networks) (e.g., Layer 2 VPNs and/or Layer 3 VPNs). For example, the ND where a provider's network and a customer's network are coupled are respectively referred to as PEs (Provider Edge) and CEs (Customer Edge). In a Layer 2 VPN, forwarding typically is performed on the CE(s) on either end of the VPN and traffic is sent across the network (e.g., through one or more PEs coupled by other NDs). Layer 2 circuits are configured between the CEs and PEs (e.g., an Ethernet port, an ATM permanent virtual circuit (PVC), a Frame Relay PVC). In a Layer 3 VPN, routing typically is performed by the PEs. By way of example, an edge ND that supports multiple VNEs may be deployed as a PE; and a VNE may be configured with a VPN protocol, and thus that VNE is referred as a VPN VNE.

[0096] Some NDs provide support for VPLS (Virtual Private LAN Service). For example, in a VPLS network, end user devices access content/services provided through the VPLS network by coupling to CEs, which are coupled through PEs coupled by other NDs. VPLS networks can be used for implementing triple play network applications (e.g., data applications (e.g., highspeed Internet access), video applications (e.g., television service such as IPTV (Internet Protocol Television), VoD (Video-on-Demand) service), and voice applications (e.g., VoIP (Voice over Internet Protocol) service)), VPN services, etc. VPLS is a type of layer 2 VPN that can be used for multi-point connectivity. VPLS networks also allow end use devices that are coupled with CEs at separate geographical locations to communicate with each other across a Wide Area Network (WAN) as if they were directly attached to each other in a Local Area Network (LAN) (referred to as an emulated LAN).

[0097] In VPLS networks, each CE typically attaches, possibly through an access network (wired and/or wireless), to a bridge module of a PE via an attachment circuit (e.g., a virtual link or connection between the CE and the PE). The bridge module of the PE attaches to an emulated LAN through an emulated LAN interface. Each bridge module acts as a "Virtual Switch Instance" (VSI) by maintaining a forwarding table that maps MAC addresses to pseudowires and attachment circuits. PEs forward frames (received from CEs) to destinations (e.g., other CEs, other PEs) based on the MAC destination address field included in those frames.

[0098] An Ethernet Virtual Private Network (EVPN) is a type of VPN technology which introduces routing Media Access Control (MAC) addresses using Multiprotocol Border Gateway Protocol (MP-BGP) over Multiprotocol Label Switching (MPLS). As with other types of VPNs, an EVPN is comprised of customer edge (CE) devices (host, router, or switch) connected to provider edge (PE) devices that form the edge of an MPLS infrastructure. A CE may be a host, a router, or a switch. The PEs provide virtual Layer 2 bridged connectivity between the CEs. There may be multiple EVPN instances in the provider' s network. The PEs may be connected by an MPLS Label Switched Path (LSP) infrastructure, which provides the benefits of MPLS technology, such as fast reroute, resiliency, etc. The PEs may also be connected by an IP infrastructure, in which case IP/GRE (Generic Routing Encapsulation) tunneling or other IP tunneling can be used between the PEs. The CEs can connect to multiple active points of attachment (i.e., to multiple PEs).

[0099] In EVPN, PEs advertise the MAC addresses learned from the CEs that are connected to them, along with an MPLS label to other PEs in the control plane using BGP. Control-plane route learning through MP-BGP, offers greater control over a MAC route learning process, and enables the introduction of restriction on which device learns which information as well as the ability to apply policies. It further enables load balancing of traffic to and from CEs that are multi-homed to multiple PEs and improves convergence times in the event of certain network failures.

[00100] For example, while the flow diagrams in the figures show a particular order of operations performed by certain embodiments of the invention, it should be understood that such order is exemplary (e.g., alternative embodiments may perform the operations in a different order, combine certain operations, overlap certain operations, etc.).

[00101] While the invention has been described in terms of several embodiments, those skilled in the art will recognize that the invention is not limited to the embodiments described, can be practiced with modification and alteration within the spirit and scope of the appended claims. The description is thus to be regarded as illustrative instead of limiting.

Claims

What is claimed is:

1. A method of monitoring a status of a Virtual Extensible Local Area Network (VXLAN) tunnel between a first network device and a second network device, the method comprising: receiving (304), from the first network device (ND111A) through the VXLAN tunnel

(106A), a first packet (103A), wherein the first packet (103 A) includes a network identifier which identifies a VXLAN segment and an inner destination address (116 A) that identifies the first network device (111 A) within the VXLAN segment; and

in response to receiving the first packet (103 A), transmitting (312), based upon the inner destination address (116A) and the network identifier, a second packet (105 A) towards the first network device (111 A) through the VXLAN tunnel (106A), wherein the transmitting the second packet (105 A) causes the first network device (111 A) to determine whether the VXLAN tunnel is active.

2. The method of claim 1, wherein the network identifier is included in a VXLAN header (206) of the first packet (103 A) and the inner destination address (116 A) is included in an inner header (208, 210) encapsulated within the VXLAN header.

3. The method of claim 2, wherein the method further comprises:

prior to the receiving and the transmitting, receiving (300) an advertisement message

(240) indicating a route from the first network device towards the second network device through the VXLAN tunnel and within the VXLAN segment, and updating (302) a forwarding table (113) associated with the network identifier to include an entry for the inner destination address; and

responsive to receiving the first packet, decapsulating (306) the first packet (103 A) to obtain an inner packet (214) including the inner header; and

forwarding (308) the inner header (214) based upon the inner destination address (116A).

4. The method of claim 3, further comprising identifying (310) a next hop in the forwarding table (113) based upon the inner destination address, wherein the next hop is the first network device (111A), and transmitting (312) the second packet (105 A) is performed based upon the identified next hop.

5. The method of claim 4, wherein the second packet (105 A) includes an outer Internet Protocol (IP) header (202) encapsulating a VXLAN packet (216) and an IP address of the first network device (111 A) as a destination address of the outer IP header (202).

6. The method of claim 3, wherein the inner header (208, 210) is at least one of a Layer 3 header and a Layer 2 header.

7. The method of claim 1, wherein the inner destination address (116A) is at least one of an Internet Protocol (IP) address and a Media Access Control (MAC) address.

8. The method of claim 1, wherein the first packet (103 A) includes a dummy payload (212).

9. A network device for monitoring Virtual Extensible Local Area Network (VXLAN) tunnel status between a first network device and a second network device, the network device comprising:

a non-transitory computer readable medium to store instructions; and

a processor coupled with the non-transitory computer readable medium to process the stored instructions to:

receive (304), from the first network device (ND111A) through the VXLAN tunnel (106A), a first packet (103A), wherein the first packet (103A) includes a network identifier which identifies a VXLAN segment and an inner destination address (116 A) that identifies the first network device (111 A) within the VXLAN segment; and

in response to receipt of the first packet, transmit (312), based upon the inner destination address (116A) and the network identifier, a second packet (105 A) towards the first network device (111 A) through the VXLAN tunnel (106A), wherein to transmit the second packet (105A) causes the first network device (111 A) to determine whether the VXLAN tunnel is active.

10. The network device of claim 9, wherein the network identifier is included in a VXLAN header (206) of the first packet (103 A) and the inner destination address (116 A) is included in an inner header (208, 210) encapsulated within the VXLAN header.

11. The network device of claim 10, wherein the processor is further to:

prior to the receipt and transmission, receive (300) an advertisement message (240) indicating a route from the first network device towards the second network device through the VXLAN tunnel and within the VXLAN segment, and update

(302) a forwarding table (113) associated with the network identifier to include an entry for the inner destination address; and

responsive to the receipt of the first packet, decapsulate (306) the first packet (103 A) to obtain an inner packet (214) including the inner header; and

forward (308) the inner header (214) based upon the inner destination address (116A).

12. The network device of claim 11, wherein the processor is further to identify (310) a next hop in the forwarding table (113) based upon the inner destination address, wherein the next hop is the first network device (111A), and to transmit (312) the second packet (105 A) is performed based upon the identified next hop.

13. The network device of claim 12, wherein the second packet (105 A) includes an outer Internet Protocol (IP) header (202) encapsulating a VXLAN packet (216) and an IP address of the first network device (111 A) as a destination address of the outer IP header (202).

14. The network device of claim 11, wherein the inner header (208, 210) is at least one of a Layer 3 header and a Layer 2 header.

15. The network device of claim 9, wherein the inner destination address (116A) is at least one of an Internet Protocol (IP) address and a Media Access Control (MAC) address.

16. The network device of claim 9, wherein the first packet (103A) includes a dummy payload (212).

17. A non-transitory computer readable storage medium that provide instructions, which when executed by a processor of a second network device that is coupled with a first network device through a Virtual Extensible Local Area Network (VXLAN) tunnel, cause said processor to perform operations comprising:

receiving (304), from the first network device (ND111A) through the VXLAN tunnel

18. The non-transitory computer readable storage medium of claim 17, wherein the network identifier is included in a VXLAN header (206) of the first packet (103 A) and the inner destination address (116A) is included in an inner header (208, 210) encapsulated within the VXLAN header.

19. The non-transitory computer readable storage medium of claim 18, wherein the operations further comprise:

20. The non-transitory computer readable storage medium of claim 19, wherein the operations further comprise identifying (310) a next hop in the forwarding table (113) based upon the inner destination address, wherein the next hop is the first network device (111A), and transmitting (312) the second packet (105 A) is performed based upon the identified next hop.

21. The non-transitory computer readable storage medium of claim 20, wherein the second packet (105 A) includes an outer Internet Protocol (IP) header (202) encapsulating a VXLAN packet (216) and an IP address of the first network device (111 A) as a destination address of the outer IP header (202).

22. The non-transitory computer readable storage medium of claim 19, wherein the inner header (208, 210) is at least one of a Layer 3 header and a Layer 2 header.

23. The non-transitory computer readable storage medium of claim 17, wherein the inner destination address (116A) is at least one of an Internet Protocol (IP) address and a Media Access Control (MAC) address.

24. The non-transitory computer readable storage medium of claim 17, wherein the first packet (103 A) includes a dummy payload (212).