WO2021224931A1

WO2021224931A1 - System and a method to efficiently exchange echo and stats messages between sdn controller and the open vswitches

Info

Publication number: WO2021224931A1
Application number: PCT/IN2020/050402
Authority: WO
Inventors: Siva Kumar PERUMALLA; Ganapathy Raman MADANAGOPAL
Original assignee: Telefonaktiebolaget Lm Ericsson (Publ)
Priority date: 2020-05-04
Filing date: 2020-05-04
Publication date: 2021-11-11

Abstract

A method of aggregating control messages in a software defined networking (SDN) network is provided. The method includes receiving a control message with control information from at least one data plane node in a region of the SDN network, generating an aggregated control message with the received control information, and sending the aggregated control message to an SDN controller for the region.

Description

SYSTEM AND A METHOD TO EFFICIENTLY EXCHANGE ECHO AND STATS MESSAGES BETWEEN SDN CONTROLLER AND THE OPEN VSWITCHES

TECHNICAL FIELD

[0001] Embodiments of the invention relate to the field of software defined networks (SDNs); and more specifically, to a method and system to improve message exchanges between an SDN and switches in the SDN.

BACKGROUND ART

[0002] Computer networks are composed of groups of networking devices that communicate data and control information. The handling of data is referred to as the data plane and the handling of control information is referred to as the control plane. The handling of the control plane and the data plane is generally implemented completely in each networking devices.

[0003] Software-defined networking (SDN) technology is an approach to network device design and network management that removes a significant aspect of the control plane from most network devices to enable more cost effective and efficient network configuration. SDN technology also improves network performance and monitoring. SDN technology is intended to change the static architecture of traditional networks that are decentralized and complex to provide networks that are more flexible and easier to troubleshoot. SDN technology centralizes network intelligence in a smaller subset of network devices by disassociating the forwarding process of network traffic(i.e., the dataplane) from the routing process (i.e., the control plane). In an SDN network, the control plane is implemented by one or more controllers that manage the SDN network. The ‘intelligence’ of the SDN network is thus concentrated in the SDN controllers.

[0004] SDN networks can enable updates to network device operation without requiring hardware upgrades, because the control plane functions at the controller can be improved separate from the simpler data plane functions. In some SDN networks a flow control protocol (e.g., the OpenFlow protocol) is used between the SDN network dataplane devices and the SDN controllers. The flow control protocol enables dynamic programming of flow control policies in a production SDN based network.

[0005] When using SDN, the network devices in the data plane, data path nodes (DPN) (e.g., Open vSwitches (OvS)) are simple forwarding engines that are programmed by the SDN controller using a standard flowcontrol protocol rules (e.g., using OpenFlow). However, the volume of messages between the SDN controllers and the dataplane nodes can adversely impact the operation of the network.

SUMMARY

[0006] In one embodiment, a method of aggregating control messages in a software defined networking (SDN) network is provided. The method includes receiving a control message with control information from at least one data plane node in a region of the SDN network, generating an aggregated control message with the received control information, and sending the aggregated control message to an SDN controller for the region.

[0007] In another embodiment, a network device executes the method of aggregating control messages in the SDN network. The network device includes a non-transitory computer-readable storage medium having stored therein an aggregator, and a processor coupled to the non-transitory computer-readable storage medium, the processor to execute the aggregator, the aggregator to receive a control message with control information from at least one data plane node in a region of the SDN network, generate an aggregated control message with the received control information, and send the aggregated control message to an SDN controller for the region.

[0008] In a further embodiment, a computing device executes a method of aggregating control messages in the SDN network. The network device executes a plurality of virtual machines. The plurality of virtual machines implements network function virtualization (NFV). The computing device includes a non-transitory computer-readable storage medium having stored therein an aggregator, and a processor coupled to the non-transitory computer- readable storage medium, the processor to execute one of the plurality of virtual machines, the one of the plurality of virtual machines to execute the aggregator, the aggregator to receive a control message with control information from at least one data plane node in a region of the SDN network, generate an aggregated control message with the received control information, and send the aggregated control message to an SDN controller for the region.

[0009] In one embodiment, a control plane device executes a method of aggregating control messages in the SDN network. The control plane device includes a non-transitory computer-readable storage medium having stored therein an aggregator manager, and a processor coupled to the non-transitory computer-readable storage medium, the processor to execute the aggregator manager, the aggregator manager to receive an aggregated control message with control information from at least one data plane node in a region of the SDN network, and to update control information maintained per data plane node based on the received control information in the aggregated control message.

BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The invention may best be understood by referring to the following description and accompanying drawings that are used to illustrate embodiments of the invention. In the drawings:

[0011] Figure 1 is a diagram of an example SDN architecture showing exchange of control messages between an SDN controller and a set of OvSes.

[0012] Figure 2 is a diagram of one embodiment where the DPN of the SDN network are organized into regions.

[0013] Figure 3 is a diagram of one example embodiment for managing high overhead control messages in an SDN network.

[0014] Figure 4 is a diagram of one example embodiment of an aggregated control message format.

[0015] Figure 5A is a flowchart of one embodiment of a process of an aggregator at a master node.

[0016] Figure 5B is a flowchart of one embodiment of a process of an aggregator manager at an SDN controller.

[0017] Figure 5C is a flowchart of one embodiment of a process of a DPN to support aggregation.

[0018] Figure 6 is a diagram of one example embodiment of a region configured to exchange control messages using spanning tree protocol (STP)/ rapid spanning tree protocol (RSTP).

[0019] Figure 7A illustrates connectivity between network devices (NDs) within an exemplary network, as well as three exemplary implementations of the NDs, according to some embodiments of the invention.

[0020] Figure 7B illustrates an exemplary way to implement a special-purpose network device according to some embodiments of the invention.

[0021] Figure 7C illustrates various exemplary ways in which virtual network elements (VNEs) may be coupled according to some embodiments of the invention.

[0022] Figure 7D illustrates a network with a single network element (NE) on each of the NDs, and within this straight forward approach contrasts a traditional distributed approach (commonly used by traditional routers) with a centralized approach for maintaining reachability and forwarding information (also called network control), according to some embodiments of the invention.

[0023] Figure 7E illustrates the simple case of where each of the NDs implements a single NE, but a centralized control plane has abstracted multiple of the NEs in different NDs into (to represent) a single NE in one of the virtual network(s), according to some embodiments of the invention.

[0024] Figure 7F illustrates a case where multiple VNEs are implemented on different NDs and are coupled to each other, and where a centralized control plane has abstracted these multiple VNEs such that they appear as a single VNE within one of the virtual networks, according to some embodiments of the invention.

[0025] Figure 8 illustrates a general purpose control plane device with centralized control plane (CCP) software 850), according to some embodiments of the invention.

DETAILED DESCRIPTION

[0026] The following description describes methods and apparatus for improving the operation of a software defined networking (SDN) network. In particular, the embodiments provide a system and processes for improving control plane operations to reduce load on SDN controllers. The embodiments reduce load caused by the exchange of flow control protocol messages such as the Echo and Stats messages and enable the SDN controller to assist data plane nodes (DPNs) in running critical applications by reducing the computing and networking resources required for handling flow control protocol messages.

[0027] In the following description, numerous specific details such as logic implementations, opcodes, means to specify operands, resource partitioning/sharing/duplication implementations, types and interrelationships of system components, and logic partitioning/integration choices are set forth in order to provide a more thorough understanding of the present invention. It will be appreciated, however, by one skilled in the art that the invention may be practiced without such specific details. In other instances, control structures, gate level circuits and full software instruction sequences have not been shown in detail in order not to obscure the invention. Those of ordinary skill in the art, with the included descriptions, will be able to implement appropriate functionality without undue experimentation.

[0028] References in the specification to “one embodiment,” “an embodiment,” “an example embodiment,” etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to affect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described. [0029] Bracketed text and blocks with dashed borders (e.g., large dashes, small dashes, dot-dash, and dots) may be used herein to illustrate optional operations that add additional features to embodiments of the invention. However, such notation should not be taken to mean that these are the only options or optional operations, and/or that blocks with solid borders are not optional in certain embodiments of the invention.

[0030] In the following description and claims, the terms “coupled” and “connected,” along with their derivatives, may be used. It should be understood that these terms are not intended as synonyms for each other. “Coupled” is used to indicate that two or more elements, which may or may not be in direct physical or electrical contact with each other, co-operate or interact with each other. “Connected” is used to indicate the establishment of communication between two or more elements that are coupled with each other.

[0031] An electronic device stores and transmits (internally and/or with other electronic devices over a network) code (which is composed of software instructions and which is sometimes referred to as computer program code or a computer program) and/or data using machine-readable media (also called computer-readable media), such as machine -readable storage media (e.g., magnetic disks, optical disks, solid state drives, read only memory (ROM), flash memory devices, phase change memory) and machine-readable transmission media (also called a carrier) (e.g., electrical, optical, radio, acoustical or other form of propagated signals - such as carrier waves, infrared signals). Thus, an electronic device (e.g., a computer) includes hardware and software, such as a set of one or more processors (e.g., wherein a processor is a microprocessor, controller, microcontroller, central processing unit, digital signal processor, application specific integrated circuit, field programmable gate array, other electronic circuitry, a combination of one or more of the preceding) coupled to one or more machine-readable storage media to store code for execution on the set of processors and/or to store data. For instance, an electronic device may include non-volatile memory containing the code since the non-volatile memory can persist code/data even when the electronic device is turned off (when power is removed), and while the electronic device is turned on that part of the code that is to be executed by the processor(s) of that electronic device is typically copied from the slower non-volatile memory into volatile memory (e.g., dynamic random access memory (DRAM), static random access memory (SRAM)) of that electronic device. Typical electronic devices also include a set or one or more physical network interface(s) (NI(s)) to establish network connections (to transmit and/or receive code and/or data using propagating signals) with other electronic devices. For example, the set of physical NIs (or the set of physical NI(s) in combination with the set of processors executing code) may perform any formatting, coding, or translating to allow the electronic device to send and receive data whether over a wired and/or a wireless connection. In some embodiments, a physical NI may comprise radio circuitry capable of receiving data from other electronic devices over a wireless connection and/or sending data out to other devices via a wireless connection. This radio circuitry may include transmitter(s), receiver(s), and/or transceiver(s) suitable for radiofrequency communication. The radio circuitry may convert digital data into a radio signal having the appropriate parameters (e.g., frequency, timing, channel, bandwidth, etc.). The radio signal may then be transmitted via antennas to the appropriate recipient(s). In some embodiments, the set of physical NI(s) may comprise network interface controller(s) (NICs), also known as a network interface card, network adapter, or local area network (LAN) adapter. The NIC(s) may facilitate in connecting the electronic device to other electronic devices allowing them to communicate via wire through plugging in a cable to a physical port connected to a NIC. One or more parts of an embodiment of the invention may be implemented using different combinations of software, firmware, and/or hardware.

[0032] A network device (ND) is an electronic device that communicatively interconnects other electronic devices on the network (e.g., other network devices, end-user devices). Some network devices are “multiple services network devices” that provide support for multiple networking functions (e.g., routing, bridging, switching, Layer 2 aggregation, session border control, Quality of Service, and/or subscriber management), and/or provide support for multiple application services (e.g., data, voice, and video).

[0033] Overview

[0034] An SDN controller interacts with DPNs using a variety of flow control protocol messages. Examples of these flow control protocol messages include Echo and Status messages. Echo (keep-alive) messages play a vital role in detecting DPN (also referred to as a ‘vSwitch’) failures. To verify the liveness of the connection between the DPN and SDN controller, a DPN sends Echo request messages (e.g., an OpenFlow OFPT_ECHO_REQUEST) or similar keep-alive messages periodically and the SDN controller responds back with Echo reply (e.g., OpenFlow OFPT_ECHO_REPLY) messages.

[0035] Inventory and statistics messages (e.g., OpenFlow OFPT_STATS_REQUEST and OFPT_STATS_REPLY messages) from each DPN are sent to the SDN controller periodically to report various metrics related to the operation of the respective DPN. Statistics can be gathered using various counters in the DPN from a statistics table (e.g., OpenFlow OvS-vSwitchDB) that contain the number of active entries and processed packets. These statistics help the SDN controller and related management applications (e.g., a virtual infrastructure manager (VIM)) to monitor the SDN network efficiently.

[0036] Control flow protocol message frequency can vary depending on the message and network configuration. For example, the OpenFlow specification doesn’t specify a standard time on how frequently Echo messages or Statistics packets must be exchanged between an OpenvSwitch (OvS) and the SDN controller. But generally, Echo messages are sent out frequently (default 1 second) so that the disconnect between the SDN controller and OvS can be detected early and appropriate recovery actions can be taken by the OvS and the SDN controller. A timeout and retransmission timer are also kept small to detect the disconnect immediately. Similarly, in OpenFlow the OvS statistics has to be updated to the SDN controller regularly, so that the applications interacting within SDN controller over the north-bound interface can react to the network changes happening based on the updated stats pushed to the SDN controller. In OpenFlow based SDN networks, the SDN controller has to regularly poll every OvS for the required statistics. In turn OvSes send replies with the required statistics. Instead of polling for the statistics, some OpenFlow protocols support push-based mechanism where the OvSes automatically sends the requested statistics to the SDN controller for the pre-defined interval regularly.

[0037] Regardless of the flow control protocol or SDN network configuration, the overhead of handling the flow control messaging increases substantially as the number of DPNs (e.g., OvSes) managed by the SDN controller increases. For example, each OvS will send Echo requests at the pre-defined interval repeatedly. The SDN controller has to receive and process all the Echo request messages and has to send the responses back to all the OvS switches. These Echo (keep-alive) messages takes precedence over other messages and the SDN controller must process these Echo messages with high-priority. Similarly, the SDN controller has to manage a large number of OvSes and in turn multiple larger statistics packets are pushed to the SDN controller. [0038] In SDN networks with a large number of DPNs sending regular control messages (e.g., Echo messages) at a high rate can overwhelm the SDN controller. The SDN controller has to spend significant compute resources (e.g., central processing unit (CPU) cycles) in processing and replying to the burst of control messages (e.g., Echo and Stats messages). Apart from compute resource utilization, DPNs in the upstream path of the control messages and the general link bandwidth utilization in the SDN network are impacted by a high volume of control messages since the DPNs have to forward the control messages to/from the controller to the respective originating DPN. These issues make the existing approach of control message handling not suitable when an SDN network is large, or where the cluster size is large (i.e., where the number of DPNs to SDN controllers is large).

[0039] When the SDN controller is unable to process a large rate or burst of control messages (e.g., Echo or Stats messages), this can have severe consequences on the DPNs. For example, if the SDN controller fails to respond to an incoming Echo request from a DPN consecutively three-times, the corresponding DPN terminates the existing connection with the SDN controller and flushes all the flows installed in the DPN and then re-initiates a new connection with the SDN controller which in turn will result in the SDN controller pushing the rules for all of the potentially thousands of existing flows to be re-installed in the DPN. Similarly, not receiving the latest stats message at the SDN controller can make the underlying SDN network prone to mishaps and mismanagement.

[0040] There are several possible approaches to handling the issues related to control message handling. One possible approach would be to reduce the frequency at which the control messages (e.g., Echo messages) are generated. This might reduce the load on the SDN controller processing control messages. However, the effectiveness of having the information from the control messages (e.g., keep-alive and stats messages) to detect and quickly react to outages of a connection between the DPN and the SDN controller or similar issues will be lowered. Another approach can be to have a dedicated secondary SDN controller and a proxy-agent which diverts some control messages (e.g., the Echo and Stats messages) to the secondary controller, thus trying to reduce the compute resource utilization on the primary SDN controller. However, this approach still has the chance of a proxy-agent getting bottlenecked or the secondary SDN controller overwhelmed. For example, the number of Echo messages exchanged with the secondary SDN controller remains the same. Also, this method requires provisioning of additional resources such as the second SDN controller and proxy-agents. Further, the load on the upstream DPNs is not necessarily reduced. [0041] The embodiments address and overcome these issues to provide a device, a system and a method that not only reduces the load on the SDN controller created by the control messages such as the Echo and Stats messages, but also helps the SDN controller to efficiently focus on DPNs running critical applications by reducing the overall control message (e.g., Echo messages) propagation across the network.

[0042] The embodiments introduce an aggregator, for example a Custom Messages Handler Daemon (CMHd) application ,that is deployed on required compute nodes such as DPNs (e.g., OvS) to help with sending aggregated control messages (e.g., Echo and Stats messages). For the required DPNs (e.g., OvSes), the SDN controller can disable or can set a large value for the frequency of control message (e.g., Echo and Stats messages generation). The DPNs (e.g., OvSes) can be clustered into various regions (e.g., identified by networks/subnet groups). Each region can have a Root/Master node, referred to herein as a ‘master node’ that is elected or assigned. The master node can also be referred to as a root- node where the spanning tree protocol (STP) is utilized (i.e., consistent with STP terminology) or can be referred to as a root/master where a user datagram protocol is utilized. For sake of clarity, the term ‘master node’ is primarily utilized herein with some examples referring to root/master or root nodes. All the nodes in the particular region send out at least certain types of control messages (e.g., those that announces aliveness), by sending periodic messages to the master node. The aggregator, e.g., CMHd, periodically polls the local DPNs and retrieves the DPN health status and similar information. The aggregator, e.g., CMHd, sends out the aliveness-message or similar information only if the health of the DPN is stable.

[0043] The embodiments further provide that an aggregator, e.g., CMHd daemon, running only on a master node can establish a new session with the SDN controller. The master node collects information relevant to managed control message (e.g., from aliveness-messages) from other nodes in the region. The master-node creates one aggregated control message (e.g., OpenFlow Echo message) with a list of all DPN identifiers (IDs) (e.g., OvS IDs) appended (e.g., into an arbitrary-length data field of the Echo message header) and sends the aggregated information to the SDN controller. On reception of the aggregated information (e.g., aliveness-message), the SDN Controller parses the list of DPN IDs (e.g., OvS IDs) in the message. For each DPN ID (e.g., OvS ID) present in the aggregated information (e.g., each aliveness-message), the SDN controller records the information for the corresponding DPN. For example, a DPN can be marked as ‘alive’ in its record when the collected aggregate information is aliveness-messages. The SDN controller can also appropriately respond to received aggregate information with an aggregated response. For example, the SDN controller can send Echo Reply only to the master-node which generated the aggregated aliveness-message. Similarly, for the example of the stats messages, the CMHd present in non-master nodes, will periodically send stats to the master-node. The CMHd running in master node, collects and appends all the stats received and publishes them as an aggregated message to the SDN controller.

[0044] The embodiments have many advantages over the existing art. The advantages include performance and architectural advantages. The performance advantages include reduced control messages, e.g., a lesser number of Echo messages gets exchanged between OvSes and the SDN Controller where OpenFlow is utilized. Performance advantages also include an efficient and scalable approach when there are large number of DPNs under the SDN controller management. The embodiments provide reduced overhead in compute resource utilization at the SDN controller and on upstream DPNs. There is also less overall link bandwidth utilization. For example, at an SDN controller there are fewer CPU cycles spent in processing the Echo and Stats packets. At the upstream DPNs fewer Stats and Echo packets are forwarded between the DPN (i.e., OvSes) thus increasing the available bandwidth for other packets. For larger clusters, the SDN controller can closely monitor the DPN with high-priority network virtualization function (NVF) applications using the existing architecture (e.g., OvS-to-Controller), while the non-critical NVF applications on DPNs can be grouped to make use of the embodiments, thus helping the SDN controller to efficiently maintain focus on critical applications.

[0045] The architectural advantages of the embodiments require no modification to existing flow control protocols (e.g., the embodiments are compatible with the OpenFlow protocol and similar flow control protocols). The embodiments do not require modification to the DPN code (e.g., Open vSwitch opensource code). Similarly, the embodiments can be utilized in SDN networks that encompasses existing Non-SDN and OvS Client only vendor switches.

[0046] Figure 1 is a diagram of an example SDN architecture showing exchange of control messages between an SDN controller and a set of OvSes. In a basic SDN network configuration, an SDN controller 101 communicates directly with a set of DPNs 103 (e.g., OvS-1 to OvS-8). An SDN controller 101 can manage any number of DPNs 103, however, without the embodiments provided herein, the SDN controller 101 can be overwhelmed by the number of control messages to process from the DPNs 103 creating a limit on the number of DPNs 103 that can be reliably managed. In the example, a single SDN controller 101 manages the set of DPNs 103. In other embodiments, the SDN controller 101 can work in coordination with other SDN controllers to manage the DPNs 103. The DPNs 103 are shown with a direct communication link with the SDN controller 101, however, the DPN 103 can communicate along paths with intermediate devices that may include any type of networking devices including other DPN 103.

[0047] Figure 2 is a diagram of one embodiment where the DPN of the SDN network are organized into regions. In some embodiments, the DPNs 103 are organized into regions 201A-C. A region can have any number of DPNs 103. An SDN controller 101 can manage any number and variety of regions. The regions 201A-C can be correlated with geographic areas, network topology, or similar characteristics. Regions 201A-C can have any number of DPNs 103 and can have differing characteristics. In the illustrated example, there are three regions managed by the SDN controller 101 including a high priority region 201C where control messages of DPNs 103 from this region can be given processing priority over those of the other regions 201 A and 20 IB.

[0048] The embodiments utilize the following components to provide a device, a system, and a method to efficiently exchange high overhead control messages (e.g., Echo messages) in an SDN enabled environment. An SDN controller that is a centralized entity that monitors and orchestrates the flow of packets through network devices functioning as DPN (e.g., Open vSwitches) by installing flow rules in these DPN. The DPN (e.g., an Open vS witch (OvS) is an entity that forwards network traffic packets based on the flow rules embedded in the DPN by the SDN controller. The DPN communicates with the controller via a flow control protocol (e.g., the OpenFlow protocol). In some embodiments, the DPN entity is part of compute nodes (e.g., servers) where NVF applications are hosted (e.g., complete OvS). In other embodiments, the DPN is a simple client (e.g., OvS) present in a physical network device (i.e., vSwitches). In both the case, high overhead control messages (e.g., Echo messages) are generated by the DPN and exchanged with SDN controller for management information reporting (e.g., reporting liveness, bandwidth and latency information).

[0049] In some embodiments, a cloud networking agent is involved, which is an entity that is responsible to provide networking support for NFV applications to communicate through overlay networks. An example of a cloud networking agent is Openstack Neutron. Th cloud networking agent also communicates with the SDN controller to install suitable flows on DPNs (e.g., OvS switches) so that packets flow to/from the destined NFV applications. [0050] The embodiments introduce an aggregator, e.g., a Custom Messages Handler Daemon (CMHd). The aggregator, e.g., CMHd, is a software entity that resides on all compute nodes (e.g., DPNs) responsible for sending aggregated control messages (e.g., aggregated Echo and Stats messages).

[0051] Figure 3 is a diagram of one example embodiment for managing high overhead control messages in an SDN network. The embodiments are described with reference to the diagram of Figure 3 in which an SDN controller 101 is managing regions 201A-C having DPNs 103 A-Z. The example is described with relation to the handling of Echo and Stats messages for an OpenFlow based SDN implementation. However, one skilled in the art would understand that the embodiments are not limited to an OpenFlow based implementation and the principles, function, and structures described with relation to the handling of Echo and Stats messages are also applicable to high overhead control messages of any flow control protocol.

[0052] In the example embodiments, the SDN network is setup and configured. A Custom Messages Handler Daemon (CMHd) is pre-deployed on all the compute nodes (DPNs 103 A-Z). The CMHd can be deployed by an SDN operator such as a cloud computing operator. On boot-up each DPN (e.g., an OvS or compute node executing an OvS instance) is uniquely identified by a Datapathld or similar identifier. The identifier is unique within the SDN network to enable the unambiguous identification of each DPN. Each DPN, e.g., each OvS instance, tries to exchange initial messages (e.g., OFPT_HELLO and OFPT_FEATURES_REQUEST messages) with the SDN controller 101 and establishes transport channel between the DPN 103 and the SDN controller 101. In one example embodiment, as part of the OvS database (OvSDB) configuration (i.e., DPN switch table configuration), the field named “system type” can be set to “CMHd enabled compute nodes ” on all compute nodes that have the CMHd daemon deployed and enabled. Based on the system type field or similar information, an SDN Controller 101 can classify nodes as CMHd enabled (e.g., OvSes having capability to run CMHd on the compute node) or other (e.g., OvS client only) DPNs not having capability to run CMHd. [0053] Based on SDN network configuration requirements, the SDN Controller 101 identifies a set of CMHd enabled DPNs (e.g., vS witches) and clusters them into various regions. A cloud networking agent 301 (e.g., Openstack Neutron agent) can instruct the SDN Controller regarding the OvS nodes handling high-priority applications, so that the SDN Controller 101 can efficiently and accurately form regions. [0054] For each region 201A-C, the SDN Controller 101 identifies and assigns a DPN 103 as a master node. A master nodes 303 will take the responsibility of collecting and sending aggregated flow control (e.g., Echo messages) to the SDN controller 101 for other DPN nodes 103 in the respective region. A DPN selected as a master node requires the capability to run CMHd, so that it can collect and aggregate control message (e.g., Echo messages). If a region does not include nodes that can run CMHd, e.g., non-SDN switches or vS witches and OvS client only physical switches that can’t be chosen as a candidate for master-node, then the region will not have a master node. Nodes in such a region (e.g., region 201C) will communicate directly with the SDN controller without aggregation.

[0055] Any logic or process can be used to cluster vSwitches to form a region and identify a master node for a particular region. For example, regions can be formed out of clustering non-critical vSwitches or clustering all the vSwitches which are in a secondary high availability path. The number of regions and appropriate cluster size for each region can be decided by the SDN or cloud operator based on any set of policies or requirements. Also, to identify the master-node for a region, strategies such as selecting a DPN with more compute resources (e.g., CPU, Memory, or similar resources) or node priority to the SDN controller 101 or similar criteria can be utilized. In some embodiments, once the regions 201A-C are created, the SDN controller 101 requests the cloud networking agent 301 such as Openstack Neutron to create a separate subnet for exchanging specific types of control messages (e.g., Echo/aliveness messages). No data traffic flows through this subnet, just the aggregated control messages.

[0056] An aggregated message contains the DPN IDs (e.g., of the OVS-Switches) in that region from which the relevant control messages (e.g., Echo aliveness-message) is received. The aggregated messages are exchanged between the master node 301 and the SDN controller 101. In some embodiments, a designated IP:Port of the master node is provided to all the other nodes in the region for exchanging control messages to be aggregated (e.g., Echo aliveness messages). For example, this information can be communicated to the CMHd using OvSDB or Out-of-band mechanisms.

[0057] The aggregator, e.g., CMHd, running on the master- node 300 responsible for a particular region establishes a session with SDN Controller 101. To enable the SDN Controller 101 to differentiate aggregated control messages (e.g., aggregated OFPT_ECHO messages) from the aggregator, e.g., CMHd, from control messages from DPNs, the aggregated control messages (e.g., ECHO messages) from the aggregator, e.g., CMHd, are appended with details such as master-node ID, corresponding DPN (e.g., OvS) Id, Region- id, and similar information. In the example case of the aggregated echo messages, this additional information can be included as part of the “element payload” field of the OFPT_ECHO_REQUEST messages. An aggregator, e.g., a CMHd daemon, is not equivalent to a full-fledged DPN (e.g., OvS with all vSwitch) capabilities. Rather, the aggregator is an application that can establish a transmission control protocol (TCP) connection with the SDN Controller 101 and create messages in flow control (e.g., OpenFlow) format and exchange with the Controller 101.

[0058] Figure 4 is a diagram of one example embodiment of an aggregated control message format. In the illustrated examples a format of an aggregated echo message and an aggregated stats message are shown. In the example aggregated messages a 32 bit header with version, type, and length information is followed by a xid, an identifier of the sending DPN. The payload of each message format can have an arbitrary length. The aggregated echo message payload provides a list of DPN IDs (e.g., OvS IDs) that have sent aliveness messages to the master node. Similarly, the payload of the stats message includes a list of different Stats information from DPNs that report to the master where each stats information component includes a DPN ID of the reporting node (e.g., OvS ID), along with a table ID, pad, and config fields that provide statistics and configuration information about the reporting DPN.

[0059] The operations in the flow diagrams will be described with reference to the exemplary embodiments of the other figures. However, it should be understood that the operations of the flow diagrams can be performed by embodiments of the invention other than those discussed with reference to the other figures, and the embodiments of the invention discussed with reference to these other figures can perform operations different than those discussed with reference to the flow diagrams.

[0060] Figure 5A is a flowchart of one embodiment of a process of an aggregator at a master node. In one embodiment, the aggregator (e.g., a CMHd) establishes a connection with the SDN controller (Block 501). The connection can be any type of connection (e.g., transmission control protocol (TCP), user datagram protocol (UPD), Internet Protocol (IP), other communication protocol) or session to enable communication with the SDN controller. The aggregator can receive configuration information from the SDN controller once the communication session is established with the SDN controller. Similarly, the aggregator or other component of the DPN executing the aggregator can provide information about the DPN including whether the DPN is capable of serving as a master for a region. The configuration information that is received from the SDN controller can include configuration information indicating that the aggregator has been selected as a master for a region of the SDN (Block 503).

[0061] In some embodiments, the aggregator can poll or similarly request that each DPN in the region send specific types of control messages or a set of specific types of control messages (e.g., Echo and Stats messages) to the aggregator (Block 505). The aggregator can periodically poll the DPN in the region or can configure or similarly establish that the DPN are to send the relevant control messages to the aggregator at regular intervals. The aggregator can collect the relevant types of control messages from each of the DPN in the region (Block 507). The control information of the received control messages can be collected. The aggregator can generate an aggregated control message at regular intervals that contain the collected control information (Block 509). The aggregated control message can be generated at any interval, or responsive to receiving all of the expected control messages. In some embodiments, the SDN controller can configure the intervals at which the aggregated control messages are to be sent. Once the aggregated control message is generated and populated and the pre-determined interval has expired, then the control message can be sent to the SDN controller (Block 511).

[0062] Figure 5B is a flowchart of one embodiment of a process of an aggregator manager at an SDN controller. In one embodiment, the aggregator manager uses knowledge of a topology of the SDN network to define a set of regions and to assign each of the DPNs in the SDN controller to one of the regions (Block 551). Any number of regions based on any criteria or characteristics for the DPNs can be utilized to define the regions. With the regions defined, the aggregator manager can determine a master node for each region (Block 553). The master node can be any DPN that is capable of executing an aggregator (e.g., a CMHd).

[0063] The SDN controller establishes a connection with each DPN in the SDN network (Block 555) including the DPN selected to be the master for the region. The connection can be any type of connection TCP, UPD, IP, or other communication protocol or session to enable communication between the SDN controller and each DPN. The aggregator manager can send configuration information to each of the DPN in the region once the communication session is established between the SDN controller and the DPN including the DPN selected as the master node. Similarly, the aggregator manager can receive information about each DPN including whether the DPN is capable of serving as a master for a region. [0064] In some embodiments, the aggregator manager can poll or similarly request that each master node in each corresponding region send specific types of control messages or a set of specific types of control messages (e.g., Echo and Stats messages) to the aggregator manager. The aggregator manager can periodically poll the master nodes in each region or can configure or similarly establish that the master nodes are to send the relevant control messages to the aggregator manager at regular intervals. The aggregator manager can collect the relevant types of control messages from each of the managers from each of the regions handled by the SDN controller (Block 559). The control information of the received control messages can be collected. The aggregator manager can check the received an aggregated control message at the expected regular intervals from each master node (i.e., from each region) (Block 559). The aggregated control messages can be generated at any interval, in response to polling from the aggregator manager, or responsive to receiving all of the expected control messages at the master. In some embodiments, the SDN controller can configure the intervals at which the aggregated control messages are to be sent.

[0065] If an aggregated control message is not received from a particular master node, then the SDN controller can query the master node to determine if the communication session is operational, if the master has failed, or similar problems exist (Block 567). When the nature of the problem is determined, then the SDN controller can attempt to remedy the issue. If the communication session failed, then it can be reestablished. If the master node has failed, then a new master node for the region can be selected.

[0066] Similarly, if an aggregated control message is received for a region, then the aggregator manager can check whether all of the DPNs for the region are included in the aggregated control message information (Block 563). If a DPN is missing, then the aggregator manager can query the missing DPN directly to determine the status of the DPN (Block 569). The SDN controller can attempt to remedy any issue with the DPN, including restarting the DPN, reconfiguring the DPN, or similarly addressing the detected issue. Once the aggregated control message is received then the received control information received in the aggregated control message can be passed to applications of the SDN controller that utilize this information or the information can be stored of updating of the information can be done in tracking data structures (Block 565).

[0067] Figure 5C is a flowchart of one embodiment of a process of a DPN to support the message aggregation. The process of the DPN related to supporting aggregation is shown by way of example and other operations of the DPN are omitted for sake of clarity. The DPN can initiate the process by establishing a connection with the SDN controller (Block 571). The connection can be initiated upon DPN start, reset, or in similar conditions. After the connection is established, the SDN controller can send configuration information to the DPN (Block 573). The configuration information can include information about a region that the DPN is assigned to as well as a master node for that region.

[0068] Depending on the configuration, the DPN can be polled by the master node for control messages or can send control messages at defined intervals. The DPN detects a polling request from the SDN controller or expiration an interval (Block 575). In response to receiving the polling request or the expired interval, the control messages of the DPN are sent to the master node (Block 577). The received configuration information can specify the set of control messages (e.g., Echo and/or Stats messages) that are to be aggregated at the master node.

[0069] Examples of the aggregator and aggregator operations are described with relation to handling Echo and Stats messages by CMHd. In particular for handling Echo and Stats messages in one example embodiment, for each of the participating Open vSwitches, the SDN controller can disable or could set a large value for the frequency of Echo and Stats messages generation. In a first example embodiment, a transport layer protocol like user datagram protocol (UDP) is utilized for exchanging aliveness messages between CMHds. All the CMHd in each DPN open an UDP socket and listen on pre-communicated (e.g., the SDN controller provides this information during setup) IP address and UDP port. The SDN controller assigns the pre-communicated IP address to the master node for the region or a virtual IP (VIP) address can be used. Each CMHd periodically polls the local OvS and retrieves the OvS health status. Each CMHd sends out the aliveness-message only if the health of the OvS is stable.

[0070] All CMHds generate the custom UDP aliveness-message at regular intervals and send it to the master node (i.e., via the configured IP: UDP). The master node receives and process the custom UDP aliveness-message. The master node then sends the aggregated OpenFlow Echo packets to the SDN controller. The SDN controller can directly query the respective OvS for the liveness, in the case that a particular OvS ID is not present in ‘N’ consecutive aggregated aliveness-messages. A similar process it utilized for handling stats packets.

[0071] In the case of a master node failure, things would be back to place just by reassigning the VIP to another node (since are aliveness-message are always sent to VIP:UDP_PORT). On receiving the custom UDP aliveness-message from other nodes, the node will automatically start to act as the new master node. The process of exchanging UDP messages between CMHd’s does not require additional OpenFlow rules to be installed in each OVS, rather the same number of rules are utilized with the substitution of the destination being the master node rather than the SDN controller.

[0072] Another embodiment for handling the example of Echo and Stats messages is described with reference to Figure 6. Figure 6 is a diagram of one example embodiment of a region configured to exchange control messages using spanning tree protocol (STP)/ rapid spanning tree protocol (RSTP). In addition to the UDP based mechanism for handling control message aggregation, the sharing and aggregating of aliveness-messages can be realized in multiple ways. One such example is through STP/RSTP. In this example, a cloud networking agent or similar component can provision a separate network/subnet per region 603. The cloud networking agent schedules a virtual machine or container (VM/container) containing an STP daemon (STPd) and a CMHd on the required servers.

[0073] The SDN controller 601 provisions the required OpenFlow rules for providing connectivity between the VMs in the region 603. STPd running inside the VMs exchange bridge protocol data units (BPDUs) between other VMs in the region 603. The elected STP root node acts as the root node (i.e., master node) for that region 603. The customer message handler daemon (CMHd) periodically generates a dummy Topology Change Notification (TCN). The TCN acts as the aliveness-messages for the respective OvS. The generated TCNs are forwarded to the root node of the region. A CMHd running on the root node aggregates the media access control (MAC) address from the received TCN BPDUs and creates an aggregated Echo message. The Aggregated Echo message is sent to the SDN controller 601 via PUNT_TO_CONTROLLER message with CUSTOM_ETH_TYPE. [0074] The SDN Controller 601 receives the P AC KET_IN_MES SAGE with CUSTOM_ETH_TYPE. The SDN controller 601 decodes the received aggregated Echo message as a message from a CMHd. The SDN controller 101 maintains mapping of MAC address of the VM to the OVS Id for tracking the aliveness of the OvS. In the case where a root/master node failure occurs, the STP convergence will re-elect the root node. After the convergence, the new root node acts as the master node.

[0075] In another example embodiment, OpenFlow stats messages can be aggregated. The SDN Controller set a Stats message timer to a large value on all participating OvSes in the SDN network or region. The same master node can collect and send aggregate stats to the SDN controller as is used for aggregating other types of control messages. An OvS generally collects different types of statistics such as aggregate flow statistics, flow table statistics, port statistics, group and meter statistics, queue statistics, and similar information and stores the information in a database (e.g., an OvS-vSwitchDB).

[0076] A master node collects all the statistics received from different OvSes. Along with its own statistics, without any manipulation/modification to the received statistics, the master node appends all the received stats in an arbitrary-length data field and sends the aggregated message to the SDN controller at regular intervals. In some embodiments, the master node appends all the details received from the other DPNs without summarizing or manipulation of the received values because only the SDN controller can make sense of the various statistics since it has an overall picture of the SDN network. For example, port statistics to virtual private networking (VPN) mapping and similar statistics. In some embodiments, the SDN controller can dynamically add or remove DPN (e.g., OvSes) of a particular region. If the SDN controller wants an OvS to directly report Echo messages, then it could disable vPort and enable the Echo and Stats messages to be sent directly.

[0077] In some embodiments, the SDN controller can detect issues and make repairs to DPN operations in the SDN network. The SDN controller can implement failure detection and recovery for DPNs. If a DPN fails (e.g., an OvS fails at a DPN), the SDN can determine whether the DPN has failed or VMs including an OvS have failed. The SDN controller can detect failure where the corresponding DPN ID or DataPathld for the OvS is not listed on the consecutive ‘N’ aggregated Echo messages. The SDN controller can directly probe the corresponding OvS or DPN ‘M’ times, e.g., with the

OFPT_Echo_Request message. In the case of no response from the DPN or OvS, a recovery mechanism such as node reboot, migrating VMs to other nodes or similar recovery mechanism can be implemented. If the controller receives a response from the DPN or OvS, then the OvS the TCP connection with the SDN controller is up and running. [0078] In one embodiment, an SDN controller can determine whether a session disconnection has occurred. In a case where there is a disconnect in the TCP session between the OvS or DPN and the SDN controller, the SDN controller can detect and remedy this problem. The SDN controller can detect that a DPN ID or DataPathld of the OvS is listed in an aggregated control message (e.g. , Echo messages), because the aggregator (e.g., a CMHd) is up and running, but the SDN controller is unable to communicate with the corresponding DPN or OvS directly. The SDN controller can implement a recovery mechanism such as trying to reboot the DPN or OvS. [0079] In one embodiment, the SDN controller can detect and remedy a case where an aggregator fails. In this case an aggregator (e.g., CMHd daemon) running on a DPN goes down. Detection can operate in the same manner as DPN failure detection by probing the corresponding DPN or OvS with a control message (e.g., an Echo request) and if the SDN controller gets a response, this indicates the aggregator failure. The SDN controller, in the case of an aggregator (e.g., CHMd failure, can implement a scheduled command or task (cron job) running on a DPN node to help to detect the failure of the aggregator (e.g., CMHd), which can also help to restart the aggregator (e.g., CMHd). The aggregator (e.g., CMHd) doesn’t maintain any state. So, crash/rebooting the aggregator (e.g., CMHd daemons) doesn’t affect the working of the system. In a worst-case scenario, if an aggregator (e.g., CMHd) never comes up, then the SDN controller can remove the DPN or OvS from the region and reset the Echo message timer to default value so that the OvS can directly exchange Echo message with the SDN controller. If aggregator (e.g., CMHd) on the master node goes down, then the disconnect in the TCP session between aggregator (e.g., CMHd) and SDN controller indicates a new leader to be elected. The SDN controller can re-assign the VIP to another node in that region, which will start acting as master- node.

[0080] In further example embodiments, instead of having an aggregator (e.g., a CMHd) as a separate daemon running on computer node, the aggregator (e.g., CMHd) can also be implemented inside an OvS or similar virtual switch component in a DPN as a vendor extension so that existing TCP session between the OvS and the SDN controller can used for communication between the CMHd and SDN controller. In some embodiments, a CHMd can have a session with secondary controller or even directly with a cloud controller and push aggregated stats to those component.

[0081] Thus the embodiments provide an advantage by exchanging fewer number of high overhead control messages (e.g., Echo and Stats messages), thereby ensuring time and resources of an SDN controller are spent to process a larger number of control packets. Thus, an SDN controller might have adequate time and resources to process critical and useful messages (e.g., OFPT_PACKET_IN and OFPT_FLOW_REMOVED and similar control messages). For Echo messages, the embodiments aggregate and summarize all the Echo messages into a single Echo messages where the information about the active OvS Ids are appended as list and sent to controller. For Stats messages, the embodiments collect the statistics information from the other OvSes append the original (unmodified) stats information compliant to OpenFlow standard, appended into a single message and send to the SDN controller.

[0082] Figure 7A illustrates connectivity between network devices (NDs) within an exemplary network, as well as three exemplary implementations of the NDs, according to some embodiments of the invention. Figure 7A shows NDs 700A-H, and their connectivity by way of lines between 700A-700B, 700B-700C, 700C-700D, 700D-700E, 700E-700F, 700F-700G, and 700A-700G, as well as between 700H and each of 700A, 700C, 700D, and 700G. These NDs are physical devices, and the connectivity between these NDs can be wireless or wired (often referred to as a link). An additional line extending from NDs 700A, 700E, and 700F illustrates that these NDs act as ingress and egress points for the network (and thus, these NDs are sometimes referred to as edge NDs; while the other NDs may be called core NDs).

[0083] Two of the exemplary ND implementations in Figure 7A are: 1) a special-purpose network device 702 that uses custom application-specific integrated-circuits (ASICs) and a special-purpose operating system (OS); and 2) a general purpose network device 704 that uses common off-the-shelf (COTS) processors and a standard OS.

[0084] The special-purpose network device 702 includes networking hardware 710 comprising a set of one or more processor(s) 712, forwarding resource(s) 714 (which typically include one or more ASICs and/or network processors), and physical network interfaces (NIs) 716 (through which network connections are made, such as those shown by the connectivity between NDs 700A-H), as well as non-transitory machine readable storage media 718 having stored therein networking software 720. During operation, the networking software 720 may be executed by the networking hardware 710 to instantiate a set of one or more networking software instance(s) 722. Each of the networking software instance(s) 722, and that part of the networking hardware 710 that executes that network software instance (be it hardware dedicated to that networking software instance and/or time slices of hardware temporally shared by that networking software instance with others of the networking software instance(s) 722), form a separate virtual network element 730A-R. Each of the virtual network element(s) (VNEs) 730A-R includes a control communication and configuration module 732A-R (sometimes referred to as a local control module or control communication module) and forwarding table(s) 734A-R, such that a given virtual network element (e.g., 730A) includes the control communication and configuration module (e.g., 732A), a set of one or more forwarding table(s) (e.g., 734A), and that portion of the networking hardware 710 that executes the virtual network element (e.g., 730A). [0085] The special-purpose network device 702 is often physically and/or logically considered to include: 1) a ND control plane 724 (sometimes referred to as a control plane) comprising the processor(s) 712 that execute the control communication and configuration module(s) 732A-R; and 2) a ND forwarding plane 726 (sometimes referred to as a forwarding plane, a data plane, or a media plane) comprising the forwarding resource(s) 714 that utilize the forwarding table(s) 734A-R and the physical NIs 716. By way of example, where the ND is a router (or is implementing routing functionality), the ND control plane 724 (the processor(s) 712 executing the control communication and configuration module(s) 732A-R) is typically responsible for participating in controlling how data (e.g., packets) is to be routed (e.g., the next hop for the data and the outgoing physical NI for that data) and storing that routing information in the forwarding table(s) 734A-R, and the ND forwarding plane 726 is responsible for receiving that data on the physical NIs 716 and forwarding that data out the appropriate ones of the physical NIs 716 based on the forwarding table(s) 734A- R.

[0086] Figure 7B illustrates an exemplary way to implement the special-purpose network device 702 according to some embodiments of the invention. Figure 7B shows a special- purpose network device including cards 738 (typically hot pluggable). While in some embodiments the cards 738 are of two types (one or more that operate as the ND forwarding plane 726 (sometimes called line cards), and one or more that operate to implement the ND control plane 724 (sometimes called control cards)), alternative embodiments may combine functionality onto a single card and/or include additional card types (e.g., one additional type of card is called a service card, resource card, or multi-application card). A service card can provide specialized processing (e.g., Layer 4 to Layer 7 services (e.g., firewall, Internet Protocol Security (IPsec), Secure Sockets Layer (SSL) / Transport Layer Security (TLS), Intrusion Detection System (IDS), peer-to-peer (P2P), Voice over IP (VoIP) Session Border Controller, Mobile Wireless Gateways (Gateway General Packet Radio Service (GPRS) Support Node (GGSN), Evolved Packet Core (EPC) Gateway)). By way of example, a service card may be used to terminate IPsec tunnels and execute the attendant authentication and encryption algorithms. These cards are coupled together through one or more interconnect mechanisms illustrated as backplane 736 (e.g., a first full mesh coupling the line cards and a second full mesh coupling all of the cards).

[0087] Returning to Figure 7 A, the general purpose network device 704 includes hardware 740 comprising a set of one or more processor(s) 742 (which are often COTS processors) and physical NIs 746, as well as non-transitory machine readable storage media 748 having stored therein software 750. During operation, the processor(s) 742 execute the software 750 to instantiate one or more sets of one or more applications 764A-R. While one embodiment does not implement virtualization, alternative embodiments may use different forms of virtualization. For example, in one such alternative embodiment the virtualization layer 754 represents the kernel of an operating system (or a shim executing on a base operating system) that allows for the creation of multiple instances 762A-R called software containers that may each be used to execute one (or more) of the sets of applications 764 A-R; where the multiple software containers (also called virtualization engines, virtual private servers, or jails) are user spaces (typically a virtual memory space) that are separate from each other and separate from the kernel space in which the operating system is run; and where the set of applications running in a given user space, unless explicitly allowed, cannot access the memory of the other processes. In another such alternative embodiment the virtualization layer 754 represents a hypervisor (sometimes referred to as a virtual machine monitor (VMM)) or a hypervisor executing on top of a host operating system, and each of the sets of applications 764 A-R is run on top of a guest operating system within an instance 762 A-R called a virtual machine (which may in some cases be considered a tightly isolated form of software container) that is run on top of the hypervisor - the guest operating system and application may not know they are running on a virtual machine as opposed to running on a “bare metal” host electronic device, or through para-virtualization the operating system and/or application may be aware of the presence of virtualization for optimization purposes. In yet other alternative embodiments, one, some or all of the applications are implemented as unikernel(s), which can be generated by compiling directly with an application only a limited set of libraries (e.g., from a library operating system (LibOS) including drivers/libraries of OS services) that provide the particular OS services needed by the application. As a unikernel can be implemented to run directly on hardware 740, directly on a hypervisor (in which case the unikemel is sometimes described as running within a LibOS virtual machine), or in a software container, embodiments can be implemented fully with unikernels running directly on a hypervisor represented by virtualization layer 754, unikernels running within software containers represented by instances 762 A-R, or as a combination of unikemels and the above-described techniques (e.g., unikemels and virtual machines both run directly on a hypervisor, unikemels and sets of applications that are run in different software containers).

[0088] The instantiation of the one or more sets of one or more applications 764 A-R, as well as virtualization if implemented, are collectively referred to as software instance(s) 752. Each set of applications 764A-R, corresponding virtualization construct (e.g., instance 762A-R) if implemented, and that part of the hardware 740 that executes them (be it hardware dedicated to that execution and/or time slices of hardware temporally shared), forms a separate virtual network element(s) 760A-R.

[0089] The virtual network element(s) 760A-R perform similar functionality to the virtual network element(s) 730A-R - e.g., similar to the control communication and configuration module(s) 732A and forwarding table(s) 734A (this virtualization of the hardware 740 is sometimes referred to as network function virtualization (NFV)). Thus, NFV may be used to consolidate many network equipment types onto industry standard high volume server hardware, physical switches, and physical storage, which could be located in Data centers, NDs, and customer premise equipment (CPE). While embodiments of the invention are illustrated with each instance 762A-R corresponding to one VNE 760A-R, alternative embodiments may implement this correspondence at a finer level granularity (e.g., line card virtual machines virtualize line cards, control card virtual machine virtualize control cards, etc.); it should be understood that the techniques described herein with reference to a correspondence of instances 762A-R to VNEs also apply to embodiments where such a finer level of granularity and/or unikernels are used.

[0090] In certain embodiments, the virtualization layer 754 includes a virtual switch that provides similar forwarding services as a physical Ethernet switch. Specifically, this virtual switch forwards traffic between instances 762A-R and the physical NI(s) 746, as well as optionally between the instances 762A-R; in addition, this virtual switch may enforce network isolation between the VNEs 760A-R that by policy are not permitted to communicate with each other (e.g., by honoring virtual local area networks (VLANs)). [0091] The third exemplary ND implementation in Figure 7 A is a hybrid network device 706, which includes both custom ASICs/special-purpose OS and COTS processors/standard OS in a single ND or a single card within an ND. In certain embodiments of such a hybrid network device, a platform VM (i.e., a VM that that implements the functionality of the special-purpose network device 702) could provide for para-virtualization to the networking hardware present in the hybrid network device 706. [0092] Regardless of the above exemplary implementations of an ND, when a single one of multiple VNEs implemented by an ND is being considered (e.g., only one of the VNEs is part of a given virtual network) or where only a single VNE is currently being implemented by an ND, the shortened term network element (NE) is sometimes used to refer to that VNE. Also in all of the above exemplary implementations, each of the VNEs (e.g., VNE(s) 730A- R, VNEs 760A-R, and those in the hybrid network device 706) receives data on the physical NIs (e.g., 716, 746) and forwards that data out the appropriate ones of the physical NIs (e.g., 716, 746). For example, a VNE implementing IP router functionality forwards IP packets on the basis of some of the IP header information in the IP packet; where IP header information includes source IP address, destination IP address, source port, destination port (where “source port” and “destination port” refer herein to protocol ports, as opposed to physical ports of a ND), transport protocol (e.g., user datagram protocol (UDP), Transmission Control Protocol (TCP), and differentiated services code point (DSCP) values.

[0093] Figure 7C illustrates various exemplary ways in which VNEs may be coupled according to some embodiments of the invention. Figure 7C shows VNEs 770A.1-770A.P (and optionally VNEs 770A.Q-770A.R) implemented in ND 700A and VNE 770H.1 in ND 700H. In Figure 7C, VNEs 770A.1-P are separate from each other in the sense that they can receive packets from outside ND 700A and forward packets outside of ND 700A; VNE 770A.1 is coupled with VNE 770H.1, and thus they communicate packets between their respective NDs; VNE 770A.2-770A.3 may optionally forward packets between themselves without forwarding them outside of the ND 700A; and VNE 770A.P may optionally be the first in a chain of VNEs that includes VNE 770A.Q followed by VNE 770A.R (this is sometimes referred to as dynamic service chaining, where each of the VNEs in the series of VNEs provides a different service - e.g., one or more layer 4-7 network services). While Figure 7C illustrates various exemplary relationships between the VNEs, alternative embodiments may support other relationships (e.g., more/fewer VNEs, more/fewer dynamic service chains, multiple different dynamic service chains with some common VNEs and some different VNEs).

[0094] The NDs of Figure 7A, for example, may form part of the Internet or a private network; and other electronic devices (not shown; such as end user devices including workstations, laptops, netbooks, tablets, palm tops, mobile phones, smartphones, phablets, multimedia phones, Voice Over Internet Protocol (VOIP) phones, terminals, portable media players, GPS units, wearable devices, gaming systems, set-top boxes, Internet enabled household appliances) may be coupled to the network (directly or through other networks such as access networks) to communicate over the network (e.g., the Internet or virtual private networks (VPNs) overlaid on (e.g., tunneled through) the Internet) with each other (directly or through servers) and/or access content and/or services. Such content and/or services are typically provided by one or more servers (not shown) belonging to a service/content provider or one or more end user devices (not shown) participating in a peer- to-peer (P2P) service, and may include, for example, public webpages (e.g., free content, store fronts, search services), private webpages (e.g., username/password accessed webpages providing email services), and/or corporate networks over VPNs. For instance, end user devices may be coupled (e.g., through customer premise equipment coupled to an access network (wired or wirelessly)) to edge NDs, which are coupled (e.g., through one or more core NDs) to other edge NDs, which are coupled to electronic devices acting as servers. However, through compute and storage virtualization, one or more of the electronic devices operating as the NDs in Figure 7A may also host one or more such servers (e.g., in the case of the general purpose network device 704, one or more of the software instances 762A-R may operate as servers; the same would be true for the hybrid network device 706; in the case of the special-purpose network device 702, one or more such servers could also be run on a virtualization layer executed by the processor(s) 712); in which case the servers are said to be co-located with the VNEs of that ND.

[0095] A virtual network is a logical abstraction of a physical network (such as that in Figure 7A) that provides network services (e.g., L2 and/or L3 services). A virtual network can be implemented as an overlay network (sometimes referred to as a network virtualization overlay) that provides network services (e.g., layer 2 (L2, data link layer) and/or layer 3 (L3, network layer) services) over an underlay network (e.g., an L3 network, such as an Internet Protocol (IP) network that uses tunnels (e.g., generic routing encapsulation (GRE), layer 2 tunneling protocol (L2TP), IPSec) to create the overlay network).

[0096] A network virtualization edge (NVE) sits at the edge of the underlay network and participates in implementing the network virtualization; the network-facing side of the NVE uses the underlay network to tunnel frames to and from other NVEs; the outward-facing side of the NVE sends and receives data to and from systems outside the network. A virtual network instance (VNI) is a specific instance of a virtual network on a NVE (e.g., a NE/VNE on an ND, a part of a NE/VNE on a ND where that NE/VNE is divided into multiple VNEs through emulation); one or more VNIs can be instantiated on an NVE (e.g., as different VNEs on an ND). A virtual access point (VAP) is a logical connection point on the NVE for connecting external systems to a virtual network; a VAP can be physical or virtual ports identified through logical interface identifiers (e.g., a VLAN ID).

[0097] Examples of network services include: 1) an Ethernet LAN emulation service (an Ethernet-based multipoint service similar to an Internet Engineering Task Force (IETF) Multiprotocol Label Switching (MPLS) or Ethernet VPN (EVPN) service) in which external systems are interconnected across the network by a LAN environment over the underlay network (e.g., an NVE provides separate L2 VNIs (virtual switching instances) for different such virtual networks, and L3 (e.g., IP/MPLS) tunneling encapsulation across the underlay network); and 2) a virtualized IP forwarding service (similar to IETF IP VPN (e.g., Border Gateway Protocol (BGP)/MPLS IP VPN) from a service definition perspective) in which external systems are interconnected across the network by an L3 environment over the underlay network (e.g., an NVE provides separate L3 VNIs (forwarding and routing instances) for different such virtual networks, and L3 (e.g., IP/MPLS) tunneling encapsulation across the underlay network)). Network services may also include quality of service capabilities (e.g., traffic classification marking, traffic conditioning and scheduling), security capabilities (e.g., filters to protect customer premises from network - originated attacks, to avoid malformed route announcements), and management capabilities (e.g., full detection and processing).

[0098] Fig. 7D illustrates a network with a single network element on each of the NDs of Figure 7 A, and within this straight forward approach contrasts a traditional distributed approach (commonly used by traditional routers) with a centralized approach for maintaining reachability and forwarding information (also called network control), according to some embodiments of the invention. Specifically, Figure 7D illustrates network elements (NEs) 770A-H with the same connectivity as the NDs 700A-H of Figure 7A.

[0099] Figure 7D illustrates that the distributed approach 772 distributes responsibility for generating the reachability and forwarding information across the NEs 770A-H; in other words, the process of neighbor discovery and topology discovery is distributed.

[00100] For example, where the special-purpose network device 702 is used, the control communication and configuration module(s) 732A-R of the ND control plane 724 typically include a reachability and forwarding information module to implement one or more routing protocols (e.g., an exterior gateway protocol such as Border Gateway Protocol (BGP), Interior Gateway Protocol(s) (IGP) (e.g., Open Shortest Path First (OSPF), Intermediate System to Intermediate System (IS-IS), Routing Information Protocol (RIP), Fabel Distribution Protocol (FDP), Resource Reservation Protocol (RSVP) (including RSVP- Traffic Engineering (TE): Extensions to RSVP for ESP Tunnels and Generalized Multi- Protocol Label Switching (GMPLS) Signaling RSVP-TE)) that communicate with other NEs to exchange routes, and then selects those routes based on one or more routing metrics. Thus, the NEs 770A-H (e.g., the processor(s) 712 executing the control communication and configuration module(s) 732A-R) perform their responsibility for participating in controlling how data (e.g., packets) is to be routed (e.g., the next hop for the data and the outgoing physical NI for that data) by distributively determining the reachability within the network and calculating their respective forwarding information. Routes and adjacencies are stored in one or more routing structures (e.g., Routing Information Base (RIB), Label Information Base (LIB), one or more adjacency structures) on the ND control plane 724. The ND control plane 724 programs the ND forwarding plane 726 with information (e.g., adjacency and route information) based on the routing structure(s). For example, the ND control plane 724 programs the adjacency and route information into one or more forwarding table(s) 734A-R (e.g., Forwarding Information Base (FIB), Label Forwarding Information Base (LFIB), and one or more adjacency structures) on the ND forwarding plane 726. For layer 2 forwarding, the ND can store one or more bridging tables that are used to forward data based on the layer 2 information in that data. While the above example uses the special-purpose network device 702, the same distributed approach 772 can be implemented on the general purpose network device 704 and the hybrid network device 706.

[00101] Figure 7D illustrates that a centralized approach 774 (also known as software defined networking (SDN)) that decouples the system that makes decisions about where traffic is sent from the underlying systems that forwards traffic to the selected destination. The illustrated centralized approach 774 has the responsibility for the generation of reachability and forwarding information in a centralized control plane 776 (sometimes referred to as a SDN control module, controller, network controller, OpenFlow controller, SDN controller, control plane node, network virtualization authority, or management control entity), and thus the process of neighbor discovery and topology discovery is centralized. The centralized control plane 776 has a south bound interface 782 with a data plane 780 (sometime referred to the infrastructure layer, network forwarding plane, or forwarding plane (which should not be confused with a ND forwarding plane)) that includes the NEs 770A-H (sometimes referred to as switches, forwarding elements, data plane elements, or nodes). The centralized control plane 776 includes a network controller 778, which includes a centralized reachability and forwarding information module 779 that determines the reachability within the network and distributes the forwarding information to the NEs 770A- H of the data plane 780 over the south bound interface 782 (which may use the OpenFlow protocol). Thus, the network intelligence is centralized in the centralized control plane 776 executing on electronic devices that are typically separate from the NDs.

[00102] For example, where the special-purpose network device 702 is used in the data plane 780, each of the control communication and configuration module(s) 732A-R of the ND control plane 724 typically include a control agent that provides the VNE side of the south bound interface 782. In this case, the ND control plane 724 (the processor(s) 712 executing the control communication and configuration module(s) 732A-R) performs its responsibility for participating in controlling how data (e.g., packets) is to be routed (e.g., the next hop for the data and the outgoing physical NI for that data) through the control agent communicating with the centralized control plane 776 to receive the forwarding information (and in some cases, the reachability information) from the centralized reachability and forwarding information module 779 (it should be understood that in some embodiments of the invention, the control communication and configuration module(s) 732A-R, in addition to communicating with the centralized control plane 776, may also play some role in determining reachability and/or calculating forwarding information - albeit less so than in the case of a distributed approach; such embodiments are generally considered to fall under the centralized approach 774, but may also be considered a hybrid approach). [00103] While the above example uses the special-purpose network device 702, the same centralized approach 774 can be implemented with the general purpose network device 704 (e.g., each of the VNE 760A-R performs its responsibility for controlling how data (e.g., packets) is to be routed (e.g., the next hop for the data and the outgoing physical NI for that data) by communicating with the centralized control plane 776 to receive the forwarding information (and in some cases, the reachability information) from the centralized reachability and forwarding information module 779; it should be understood that in some embodiments of the invention, the VNEs 760A-R, in addition to communicating with the centralized control plane 776, may also play some role in determining reachability and/or calculating forwarding information - albeit less so than in the case of a distributed approach) and the hybrid network device 706. In fact, the use of SDN techniques can enhance the NFV techniques typically used in the general purpose network device 704 or hybrid network device 706 implementations as NFV is able to support SDN by providing an infrastructure upon which the SDN software can be run, and NFV and SDN both aim to make use of commodity server hardware and physical switches.

[00104] Figure 7D also shows that the centralized control plane 776 has a north bound interface 784 to an application layer 786, in which resides application(s) 788. The centralized control plane 776 has the ability to form virtual networks 792 (sometimes referred to as a logical forwarding plane, network services, or overlay networks (with the NEs 770A-H of the data plane 780 being the underlay network)) for the application(s) 788. Thus, the centralized control plane 776 maintains a global view of all NDs and configured NEs/VNEs, and it maps the virtual networks to the underlying NDs efficiently (including maintaining these mappings as the physical network changes either through hardware (ND, link, or ND component) failure, addition, or removal).

[00105] While Figure 7D shows the distributed approach 772 separate from the centralized approach 774, the effort of network control may be distributed differently or the two combined in certain embodiments of the invention. For example: 1) embodiments may generally use the centralized approach (SDN) 774, but have certain functions delegated to the NEs (e.g., the distributed approach may be used to implement one or more of fault monitoring, performance monitoring, protection switching, and primitives for neighbor and/or topology discovery); or 2) embodiments of the invention may perform neighbor discovery and topology discovery via both the centralized control plane and the distributed protocols, and the results compared to raise exceptions where they do not agree. Such embodiments are generally considered to fall under the centralized approach 774, but may also be considered a hybrid approach.

[00106] While Figure 7D illustrates the simple case where each of the NDs 700A-H implements a single NE 770A-H, it should be understood that the network control approaches described with reference to Figure 7D also work for networks where one or more of the NDs 700A-H implement multiple VNEs (e.g., VNEs 730A-R, VNEs 760A-R, those in the hybrid network device 706). Alternatively or in addition, the network controller 778 may also emulate the implementation of multiple VNEs in a single ND. Specifically, instead of (or in addition to) implementing multiple VNEs in a single ND, the network controller 778 may present the implementation of a VNE/NE in a single ND as multiple VNEs in the virtual networks 792 (all in the same one of the virtual network(s) 792, each in different ones of the virtual network(s) 792, or some combination). For example, the network controller 778 may cause an ND to implement a single VNE (a NE) in the underlay network, and then logically divide up the resources of that NE within the centralized control plane 776 to present different VNEs in the virtual network(s) 792 (where these different VNEs in the overlay networks are sharing the resources of the single VNE/NE implementation on the ND in the underlay network).

[00107] On the other hand, Figures 7E and 7F respectively illustrate exemplary abstractions of NEs and VNEs that the network controller 778 may present as part of different ones of the virtual networks 792. Figure 7E illustrates the simple case of where each of the NDs 700A-H implements a single NE 770A-H (see Figure 7D), but the centralized control plane 776 has abstracted multiple of the NEs in different NDs (the NEs 770A-C and G-H) into (to represent) a single NE 7701 in one of the virtual network(s) 792 of Figure 7D, according to some embodiments of the invention. Figure 7E shows that in this virtual network, the NE 7701 is coupled to NE 770D and 770F, which are both still coupled to NE 770E.

[00108] Figure 7F illustrates a case where multiple VNEs (VNE 770A.1 and VNE 770H.1) are implemented on different NDs (ND 700 A and ND 700H) and are coupled to each other, and where the centralized control plane 776 has abstracted these multiple VNEs such that they appear as a single VNE 770T within one of the virtual networks 792 of Figure 7D, according to some embodiments of the invention. Thus, the abstraction of a NE or VNE can span multiple NDs.

[00109] While some embodiments of the invention implement the centralized control plane 776 as a single entity (e.g., a single instance of software running on a single electronic device), alternative embodiments may spread the functionality across multiple entities for redundancy and/or scalability purposes (e.g., multiple instances of software running on different electronic devices).

[00110] Similar to the network device implementations, the electronic device(s) running the centralized control plane 776, and thus the network controller 778 including the centralized reachability and forwarding information module 779, may be implemented a variety of ways (e.g., a special purpose device, a general-purpose (e.g., COTS) device, or hybrid device). These electronic device(s) would similarly include processor(s), a set or one or more physical NIs, and a non-transitory machine -readable storage medium having stored thereon the centralized control plane software. For instance, Figure 8 illustrates, a general purpose control plane device 804 including hardware 840 comprising a set of one or more processor(s) 842 (which are often COTS processors) and physical NIs 846, as well as non- transitory machine readable storage media 848 having stored therein centralized control plane (CCP) software 850.

[00111] In embodiments that use compute virtualization, the processor(s) 842 typically execute software to instantiate a virtualization layer 854 (e.g., in one embodiment the virtualization layer 854 represents the kernel of an operating system (or a shim executing on a base operating system) that allows for the creation of multiple instances 862A-R called software containers (representing separate user spaces and also called virtualization engines, virtual private servers, or jails) that may each be used to execute a set of one or more applications; in another embodiment the virtualization layer 854 represents a hypervisor (sometimes referred to as a virtual machine monitor (VMM)) or a hypervisor executing on top of a host operating system, and an application is run on top of a guest operating system within an instance 862A-R called a virtual machine (which in some cases may be considered a tightly isolated form of software container) that is run by the hypervisor ; in another embodiment, an application is implemented as a unikemel, which can be generated by compiling directly with an application only a limited set of libraries (e.g., from a library operating system (LibOS) including drivers/libraries of OS services) that provide the particular OS services needed by the application, and the unikernel can run directly on hardware 840, directly on a hypervisor represented by virtualization layer 854 (in which case the unikemel is sometimes described as running within a LibOS virtual machine), or in a software container represented by one of instances 862 A-R). Again, in embodiments where compute virtualization is used, during operation an instance of the CCP software 850 (illustrated as CCP instance 876A) is executed (e.g., within the instance 862A) on the virtualization layer 854. In embodiments where compute virtualization is not used, the CCP instance 876A is executed, as a unikemel or on top of a host operating system, on the “bare metal” general purpose control plane device 804. The instantiation of the CCP instance 876A, as well as the virtualization layer 854 and instances 862A-R if implemented, are collectively referred to as software instance(s) 852.

[00112] In some embodiments, the CCP instance 876A includes a network controller instance 878. The network controller instance 878 includes a centralized reachability and forwarding information module instance 879 (which is a middleware layer providing the context of the network controller 778 to the operating system and communicating with the various NEs), and an CCP application layer 880 (sometimes referred to as an application layer) over the middleware layer (providing the intelligence required for various network operations such as protocols, network situational awareness, and user - interfaces). At a more abstract level, this CCP application layer 880 within the centralized control plane 776 works with virtual network view(s) (logical view(s) of the network) and the middleware layer provides the conversion from the virtual networks to the physical view.

[00113] The centralized control plane 776 transmits relevant messages to the data plane 780 based on CCP application layer 880 calculations and middleware layer mapping for each flow. A flow may be defined as a set of packets whose headers match a given pattern of bits; in this sense, traditional IP forwarding is also flow-based forwarding where the flows are defined by the destination IP address for example; however, in other implementations, the given pattern of bits used for a flow definition may include more fields (e.g., 10 or more) in the packet headers. Different NDs/NEs/VNEs of the data plane 780 may receive different messages, and thus different forwarding information. The data plane 780 processes these messages and programs the appropriate flow information and corresponding actions in the forwarding tables (sometime referred to as flow tables) of the appropriate NE/VNEs, and then the NEs/VNEs map incoming packets to flows represented in the forwarding tables and forward packets based on the matches in the forwarding tables. [00114] Standards such as OpenFlow define the protocols used for the messages, as well as a model for processing the packets. The model for processing packets includes header parsing, packet classification, and making forwarding decisions. Header parsing describes how to interpret a packet based upon a well-known set of protocols. Some protocol fields are used to build a match structure (or key) that will be used in packet classification (e.g., a first key field could be a source media access control (MAC) address, and a second key field could be a destination MAC address).

[00115] Packet classification involves executing a lookup in memory to classify the packet by determining which entry (also referred to as a forwarding table entry or flow entry) in the forwarding tables best matches the packet based upon the match structure, or key, of the forwarding table entries. It is possible that many flows represented in the forwarding table entries can correspond/match to a packet; in this case the system is typically configured to determine one forwarding table entry from the many according to a defined scheme (e.g., selecting a first forwarding table entry that is matched). Forwarding table entries include both a specific set of match criteria (a set of values or wildcards, or an indication of what portions of a packet should be compared to a particular value/values/wildcards, as defined by the matching capabilities - for specific fields in the packet header, or for some other packet content), and a set of one or more actions for the data plane to take on receiving a matching packet. For example, an action may be to push a header onto the packet, for the packet using a particular port, flood the packet, or simply drop the packet. Thus, a forwarding table entry for IPv4/IPv6 packets with a particular transmission control protocol (TCP) destination port could contain an action specifying that these packets should be dropped.

[00116] Making forwarding decisions and performing actions occurs, based upon the forwarding table entry identified during packet classification, by executing the set of actions identified in the matched forwarding table entry on the packet.

[00117] However, when an unknown packet (for example, a “missed packet” or a “match- miss” as used in OpenFlow parlance) arrives at the data plane 780, the packet (or a subset of the packet header and content) is typically forwarded to the centralized control plane 776. The centralized control plane 776 will then program forwarding table entries into the data plane 780 to accommodate packets belonging to the flow of the unknown packet. Once a specific forwarding table entry has been programmed into the data plane 780 by the centralized control plane 776, the next packet with matching credentials will match that forwarding table entry and take the set of actions associated with that matched entry.

[00118] While the invention has been described in terms of several embodiments, those skilled in the art will recognize that the invention is not limited to the embodiments described, can be practiced with modification and alteration within the spirit and scope of the appended claims. The description is thus to be regarded as illustrative instead of limiting.

Claims

CLAIMS:

1. A method of aggregating control messages in a software defined networking (SDN) network, the method comprising: receiving a control message with control information from at least one data plane node in a region of the SDN network; generating an aggregated control message with the received control information; and sending the aggregated control message to an SDN controller for the region.

2. The method of claim 1, wherein the control message is an Echo message or a Stats message.

3. The method of claim 1, wherein the control message is a user datagram protocol (UDP) or a spanning tree protocol (STP) message.

4. The method of claim 1, further comprising: receiving configuration from the SDN controller indicating that a receiving data plane node is to function as a master node for the region.

5. The method of claim 1, further comprising: polling at least one data plane node in the region to obtain control information.

6. A network device to execute a method of aggregating control messages in a software defined networking (SDN) network, the network device comprising: a non-transitory computer-readable storage medium having stored therein an aggregator; and a processor coupled to the non-transitory computer-readable storage medium, the processor to execute the aggregator, the aggregator to receive a control message with control information from at least one data plane node in a region of the SDN network, generate an aggregated control message with the received control information, and send the aggregated control message to an SDN controller for the region.

7. The network device of claim 6, wherein the control message is an Echo message or a Stats message.

8. The network device of claim 6, wherein the control message is a user datagram protocol (UDP) or a spanning tree protocol (STP) message.

9. The network device of claim 6, wherein the aggregator is further configured to receive configuration from the SDN controller indicating that a receiving data plane node is to function as a master node for the region.

10. The network device of claim 6, wherein the aggregator is further configured to polling at least one data plane node in the region to obtain control information.

11. A computing device to execute a method of aggregating control messages in a software defined networking (SDN) network, the network device to execute a plurality of virtual machines, the plurality of virtual machines implementing network function virtualization (NFV), the computing device comprising: a non-transitory computer-readable storage medium having stored therein an aggregator; and a processor coupled to the non-transitory computer-readable storage medium, the processor to execute one of the plurality of virtual machines, the one of the plurality of virtual machines to execute the aggregator, the aggregator to receive a control message with control information from at least one data plane node in a region of the SDN network, generate an aggregated control message with the received control information, and send the aggregated control message to an SDN controller for the region.

12. The computing device of claim 11, wherein the control message is an Echo message or a Stats message.

13. The computing device of claim 11, wherein the control message is a user datagram protocol (UDP) or a spanning tree protocol (STP) message.

14. The computing device of claim 1, wherein the aggregator is further configured to receive configuration from the SDN controller indicating that a receiving data plane node is to function as a master node for the region.

15. The computing device of claim 11, wherein the aggregator is further configured to polling at least one data plane node in the region to obtain control information.

16. A control plane device to execute a method of aggregating control messages in a software defined networking (SDN) network, the control plane device comprising: a non-transitory computer-readable storage medium having stored therein an aggregator manager; and a processor coupled to the non-transitory computer-readable storage medium, the processor to execute the aggregator manager, the aggregator manager to receive an aggregated control message with control information from at least one data plane node in a region of the SDN network, and to update control information maintained per data plane node based on the received control information in the aggregated control message.

17. The control plane device of claim 16, wherein the aggregated control message includes aggregated data from a plurality of Echo messages or a Stats messages.

18. The control plane device of claim 16, wherein the aggregated control message is a user datagram protocol (UDP) or a spanning tree protocol (STP) message.

19. The control plane device of claim 16, wherein the aggregator manager is further configured to determine a failure of a master node in the region and reestablish a connection with the master node or select a replacement master node.

20. The control plane device of claim 16, wherein the aggregator manager is further configured to query at least one data plane node in the region to determine a status of the data plane node in response to an absence of control information for the data plane node in the aggregated flow control.