CN117099082A - User interface for cloud native software defined network architecture - Google Patents

Abstract

In general, techniques are described for creating virtual network routers via a User Interface (UI) presented by a Software Defined Network (SDN) architecture. A network controller including memory and processing circuitry may perform these techniques. The memory may store a UI and the processing circuitry may present the UI and execute the control node. The UI may graphically represent a topology of a network including the first virtual network and the second virtual network. The UI may dynamically generate a graphical element representing a virtual network router through which the first virtual network and the second virtual network are interconnected. The virtual network router may represent a logical abstraction of one or more policies that cause one or more of ingress and egress of routing information between the first virtual network and the second virtual network. The control node configures the first virtual network and the second virtual network according to one or more policies.

Description

User interface for cloud native software defined network architecture

RELATED APPLICATIONS

The present application claims the benefit of U.S. provisional application No. 63/364,283 filed on 5/6 of 2022 and indian provisional application No. 202141044924 filed on 4 of 2021, each of which is incorporated herein by reference in its entirety.

Technical Field

The present disclosure relates to virtualized computing infrastructure, and more particularly, to cloud native networking.

Background

In a typical cloud data center environment, there is a large set of interconnected servers that provide computing and/or storage capacity to run various applications. For example, a data center may include facilities that host applications and services for subscribers (i.e., customers of the data center). The data center may, for example, host all infrastructure equipment such as networking and storage systems, redundant power supplies, and environmental controls. In a typical data center, clusters of storage systems and application servers are interconnected via a high-speed switching fabric provided by one or more layers of physical network switches and routers. More complex data centers provide an infrastructure throughout the world with subscriber support equipment located in various physical hosting facilities.

Virtualized data centers are becoming the core foundation of modern Information Technology (IT) infrastructure. In particular, modern data centers have widely utilized virtualized environments in which virtual hosts (also referred to herein as virtual execution elements, such as virtual machines or containers) are deployed and executed on the underlying computing platform of a physical computing device.

Virtualization within a data center or any environment that includes one or more servers may provide several advantages. One advantage is that virtualization may provide a significant improvement in efficiency. With the advent of multi-core microprocessor architectures with a large number of cores per physical CPU, the underlying physical computing devices (i.e., servers) have become more and more powerful and virtualization has become easier and more efficient. A second advantage is that virtualization provides significant control over the computing infrastructure. As physical computing resources become alternative resources, such as in a cloud-based computing environment, provisioning and management of computing infrastructure becomes easier. Thus, enterprise IT personnel often prefer virtualized computing clusters in a data center because of their management advantages in addition to the efficiency and increased Return On Investment (ROI) provided by virtualization.

Containerization is a virtualization scheme based on operating system level virtualization. The containers are lightweight and portable actuators for applications that are isolated from each other and from the host. Because the container is not tightly coupled to the host hardware computing environment, the application may be bound to the container image and executed as a single lightweight package on any host or virtual host that supports the underlying container architecture. In this way, the container solves the problem of how to make software work in different computing environments. The container provides promises for consistent operation from one computing environment to another virtual or physical environment.

With the inherently lightweight nature of containers, a single host can typically support more container instances than a traditional Virtual Machine (VM). Short-term containers can be created and moved more efficiently than VMs, typically short-term (as compared to most VMs), and they can also be managed as logically related groups of elements (e.g., sometimes referred to as "pod" for some orchestration platforms, such as Kubernetes). These container characteristics affect the requirements for a container networking solution: the network should be agile and scalable. The VM, container, and bare metal server may need to coexist in the same computing environment with communication enabled between diverse deployments of applications. The container network should also be agnostic to work with multiple types of orchestration platforms for deploying containerized applications.

The computing infrastructure that manages the infrastructure of deployment and application execution may involve two main roles: (1) Orchestration-deployment, scaling, and operation for automating applications across host clusters, and providing computing infrastructure, which may include container-centric computing infrastructure; and (2) network management-creating a virtual network in a network infrastructure to enable packet communication in applications running on a virtual execution environment such as a container or VM, as well as in applications running on a traditional (e.g., physical) environment. The software defined network facilitates network management.

Disclosure of Invention

In general, techniques are described for providing a user interface (e.g., a graphical user interface) for interfacing with a cloud native Software Defined Network (SDN) architecture. In some examples, the SDN architecture may include data plane elements implemented in computing nodes and network devices such as routers or switches, and the SDN architecture may also include network controllers for creating and managing virtual networks. The SDN architecture configuration and control plane is designed with laterally-expanding cloud native software that supports a container-based micro-service architecture for in-service upgrades. Configuration nodes for configuring planes may be implemented to expose custom resources. These custom resources for SDN architecture configuration may include configuration elements that are traditionally exposed by network controllers, but configuration elements may be incorporated with Kubernetes local/built-in resources to support a unified intent model, exposed by an aggregated API layer, implemented by Kubernetes controllers and custom resource controllers working to coordinate the actual state and intent state of the SDN architecture.

Custom resources for SDN architecture configuration include virtual network routers that represent logical abstractions for interconnecting and implementing communications between virtual networks implemented by the SDN architecture. Instances of virtual network router customization resources facilitate the exchange of routing information (referred to as asymmetric or symmetric ingress and egress) using ingress and egress routing targets configured within the routing instances of the respective virtual networks. Thus, in practice, virtual network routers enable routing leakage between virtual network resources defined for SDN architecture. Because virtual network router resources hide complex detailed information of virtual routing and forwarding instance (VRF or "routing instance") configuration in VRFs used to implement various virtual network resources, the technique provides an intuitive intent-based model for defining interconnectivity between virtual network resources and corresponding virtual networks configured in the SDN architecture and improves usability even for those who may be unfamiliar with border gateway protocols and VRF configurations.

The addition of VNRs for abstracting policies that control the ingress and/or egress of routing information may simplify the stitching together of network elements (such as virtual networks) to facilitate interconnection between such network elements. Rather than resorting to complex interfaces to define VNRs involving complex statements that may require complex network construction knowledge (such as routing targets, labels identifying network elements, etc.), aspects of the techniques described in this disclosure may provide a user interface configured to graphically present a topology of a network that may include various network elements (such as virtual networks). A user may interface with a user interface, which may refer to a Graphical User Interface (GUI), to visually configure a VNR to interconnect two or more network elements.

The user interface may receive user input (which may be referred to as input via a GUI input), such as a graphical user interface input for dragging and dropping or otherwise defining a VNR. The user interface may communicate these inputs to the network controller as a request to configure the VNR. The user may input through interaction with the GUI to graphically define the VNR, select a virtual network and other network elements for which the VNR establishes interconnectivity (e.g., grid connectivity and/or center spoke connectivity). The network controller may process these inputs and update the GUI to present prompts and other GUIs by which to prompt the user for additional inputs that may be needed to define the VNR. The network controller may reduce these inputs defining the VNR into various policies that control the ingress and/or egress of routing information between network elements, and configure policies in the appropriate network elements to facilitate the exchange of routing information that enables interconnectivity between network elements.

These techniques may provide additional one or more technical advantages. For example, cloud-native (SDN) architecture may address limitations in traditional SDN architecture that relate to complexity of lifecycle management, mandatory high-resource analysis components, scale limitations in configuration management, and lack of command-line interface (CLI) based interfaces. For example, a network controller for an SDN architecture is a cloud-native, lightweight distributed application with a simplified installation footprint. This also facilitates easier and modular upgrades of configuration nodes of the configuration and control plane and of the various component microservices of the control node.

The techniques may also enable optional cloud native monitoring (telemetry) and user interface, high performance data planes for containers using virtual routers connected to DPDK-enabled pod based data plane forwarding tools (DPDK-based), and in some cases cloud native configuration management with a configuration framework for existing orchestration platforms such as Kubernetes or Openstack. As a cloud native architecture, the network controller is scalable and resilient to address and support multiple clusters. In some cases, the network controller may also support scalability and performance requirements of Key Performance Indicators (KPIs).

In addition, user interfaces configured in accordance with aspects of the technology described in the present disclosure may enable inexperienced users (in the context of SDN architecture) to define VNRs without extensive knowledge of network architecture (such as those listed above). In this way, a user may more efficiently interact with the user interface to define a VNR to meet enterprise and/or customer goals. The user interface may guide the user to reduce and possibly eliminate errors in configuring the VNR, while also enabling the user to view the topology of the network and get a better understanding of the network itself. By being easy to use and eliminating errors, the user interface may reduce the need for a user to define complex configuration data while also avoiding such errors that may cause inefficient operation of the network controller (in terms of computing resources such as processing cycles, memory bus bandwidth, etc., and associated power). In this way, the user interface may improve the operation of the network controller itself.

In one example, aspects of the present technology relate to a network controller for a Software Defined Network (SDN) architecture system, the network controller comprising: a memory configured to store a user interface; processing circuitry configured to present the user interface and execute the control node, wherein the user interface is configured to: graphically representing a topology of a network supported by the Software Defined Network (SDN) architecture system, the network including a first virtual network and a second virtual network; and dynamically generating a graphical element representing a virtual network router through which the first virtual network and the second virtual network are interconnected, the virtual network router representing a logical abstraction of one or more policies that cause one or more of ingress and egress of routing information between the first virtual network and the second virtual network, wherein the control node configures the first virtual network and the second virtual network in accordance with the one or more policies to enable one or more of ingress and egress of routing information between the first virtual network and the second virtual network via the virtual network router.

In another example, aspects of the present technology relate to a method comprising: presenting, by a network controller for a Software Defined Network (SDN) architecture system, a user interface, wherein the user interface is configured to: graphically representing a topology of a network supported by the Software Defined Network (SDN) architecture system, the network including a first virtual network and a second virtual network; and dynamically generating a graphical element representing a virtual network router through which the first virtual network and the second virtual network are interconnected, the virtual network router representing a logical abstraction of one or more policies that cause one or more of ingress and egress of routing information between the first virtual network and the second virtual network; and executing, by the network controller, a control node that configures the first virtual network and the second virtual network according to one or more policies to enable one or more of ingress and egress of routing information between the first virtual network and the second virtual network via the virtual network router.

In another example, aspects of the present technology relate to a non-transitory computer-readable medium including instructions for causing processing circuitry of a network controller of a Software Defined Network (SDN) architecture system to: presenting a user interface, wherein the user interface is configured to: graphically representing a topology of a network supported by the Software Defined Network (SDN) architecture system, the network including a first virtual network and a second virtual network; and dynamically generating a graphical element representing a virtual network router through which the first virtual network and the second virtual network are interconnected, the virtual network router representing a logical abstraction of one or more policies that cause one or more of ingress and egress of routing information between the first virtual network and the second virtual network; and executing a control node that configures the first virtual network and the second virtual network according to the one or more policies to enable one or more of ingress and egress of routing information between the first virtual network and the second virtual network via the virtual network router.

The details of one or more examples of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims.

Drawings

FIG. 1 is a block diagram illustrating an exemplary computing infrastructure in which examples of the techniques described herein may be implemented.

Fig. 2 is a block diagram illustrating an example of a cloud native SDN architecture for cloud native networking in accordance with the techniques of this disclosure.

Fig. 3 is a block diagram illustrating another view of components of an SDN architecture in greater detail, in accordance with the techniques of this disclosure.

Fig. 4 is a block diagram illustrating exemplary components of an SDN architecture in accordance with the techniques of this disclosure.

Fig. 5 is a block diagram of an exemplary computing device in accordance with the techniques described in this disclosure.

Fig. 6 is a block diagram of an exemplary computing device operating as a computing node of one or more clusters of an SDN architecture system in accordance with the techniques of this disclosure.

Fig. 7A is a block diagram illustrating a control/routing plane for an underlying network and overlay network configuration using an SDN architecture in accordance with the techniques of this disclosure.

Fig. 7B is a block diagram illustrating a configured virtual network using a configured tunnel in an underlying network to connect pods in accordance with the techniques of this disclosure.

Fig. 8 is a block diagram illustrating an example of a custom controller for custom resources of an SDN architecture configuration in accordance with the techniques of this disclosure.

FIG. 9 is a block diagram illustrating an exemplary flow of creation, observation, and reconciliation between custom resource types that have dependencies on different custom resource types.

Fig. 10A-10M are diagrams illustrating examples of graphical user interfaces providing graphical representations of the topology of a network in which virtual network routers may be graphically defined to allow connectivity between different network elements.

Fig. 11-17 are diagrams illustrating various examples in which virtual network routers may be configured to implement interconnectivity between virtual networks in accordance with aspects of the technology described in the present disclosure.

Fig. 18A-18D are flowcharts illustrating operations performed when one or more virtual network routers are established within an SDN architecture in accordance with aspects of the technology described in the present disclosure.

Fig. 19 is another flow chart illustrating operation of the computer architecture shown in the example of fig. 1 in performing aspects of the technology described herein.

Like reference numerals refer to like elements throughout the specification and drawings.

Detailed Description

FIG. 1 is a block diagram illustrating an exemplary computing infrastructure 8 in which examples of the techniques described herein may be implemented. Current implementations of Software Defined Networking (SDN) architecture for virtual networks present challenges to cloud native adoption due to, for example, the complexity of lifecycle management, mandatory high-resource analysis components, scale restrictions in configuration modules, and the lack of Command Line Interface (CLI) -like interfaces. The computing infrastructure 8 includes a cloud-native SDN architecture system described herein that addresses these challenges and modernizes the Telco cloud-native era (Telco closed-active era). Example use cases of cloud-native SDN architecture include 5G mobile networks, cloud and enterprise cloud-native use cases. The SDN architecture may include data plane elements implemented in computing nodes (e.g., servers 12) and network devices such as routers or switches, and may also include SDN controllers (e.g., network controllers 24) for creating and managing virtual networks. The SDN architecture configuration and control plane is designed with laterally-expanding cloud native software that supports a container-based micro-service architecture for in-service upgrades.

As a result, the SDN architecture component is a micro-service and, in contrast to existing network controllers, the SDN architecture assumes a basic container orchestration platform to manage the lifecycle of the SDN architecture component. Building SDN framework components by utilizing a container arranging platform; the SDN architecture uses a cloud-native monitoring tool that may be integrated with customer-provided cloud-native options; the SDN architecture provides a declarative way of resources using APIs for aggregation of SDN architecture objects (i.e., custom resources). SDN architecture upgrades may follow cloud native patterns and SDN architecture may utilize Kubernetes constructs such as Multus, authentication and authorization, cluster API, kubeFederation, kubeVirt, and Kata containers. The SDN architecture may support data plane forwarding tool (DPDK) pod and may be extended to support Kubernetes with virtual network policies and global security policies.

For service providers and enterprises, SDN architecture automates network resource provisioning and orchestration to dynamically create highly scalable virtual networks, and links Virtualized Network Functions (VNFs) and Physical Network Functions (PNFs) to form differentiated service chains on demand. The SDN architecture may be integrated with orchestration platforms such as Kubernetes, openShift, mesos, openStack, VMware vspheres (e.g., orchestrator 23), and with service provider operation support systems/business support systems (OSS/BSS).

In general, one or more data centers 10 provide an operating environment for applications and services of a client station 11 (illustrated as "client 11") having one or more client networks coupled to the data center through a service provider network 7. Each of the data centers 10 may, for example, host infrastructure equipment such as networking and storage systems, redundant power supplies, and environmental control. The service provider network 7 is coupled to a public network 15, which may represent one or more networks managed by other providers, and thus may form part of a large-scale public network infrastructure, such as the internet. Public network 15 may represent, for example, a Local Area Network (LAN), wide Area Network (WAN), the internet, a Virtual LAN (VLAN), an enterprise LAN, a layer 3 Virtual Private Network (VPN), an Internet Protocol (IP) intranet operated by a service provider running service provider network 7, an enterprise IP network, or some combination thereof.

Although the client station 11 and public network 15 are primarily illustrated and described as edge networks of the service provider network 7, in some examples one or more of the client station 11 and public network 15 may be a tenant network within any of the data centers 10. For example, the data center 10 may host multiple tenants (customers), each associated with one or more Virtual Private Networks (VPNs), each of which may implement one of the customer sites 11.

The service provider network 7 provides packet-based connectivity to attached client stations 11, data centers 10 and public networks 15. The service provider network 7 may represent a network owned and operated by a service provider to interconnect multiple networks. The service provider network 7 may implement multiprotocol label switching (MPLS) forwarding and may be referred to as an MPLS network or MPLS backbone in such instances. In some examples, service provider network 7 represents a plurality of interconnected autonomous systems, such as the internet, that provide services from one or more service providers.

In some examples, each of the data centers 10 may represent one of a number of network data centers that are geographically distributed, which may be connected to each other via a service provider network 7, a dedicated network link, a dark fiber, or other connection. As shown in the example of fig. 1, the data center 10 may include facilities for providing network services to customers. The clients of the service provider may be collective entities such as businesses and governments or individuals. For example, a network data center may host web services for several enterprises and end users. Other exemplary services may include data storage, virtual private networks, traffic engineering, file services, data mining, scientific or super computing, and so forth. Although illustrated as a separate edge network of the service provider network 7, elements of the data center 10, such as one or more Physical Network Functions (PNFs) or Virtualized Network Functions (VNFs), may be included within the service provider network 7 core.

In this example, the data center 10 includes storage and/or computing servers (or "nodes") interconnected via a switch fabric 14 provided by one or more layers of physical network switches and routers, with the servers 12A-12X (herein "servers 12") being depicted as coupled to roof-top switches 16A-16N. The server 12 is a computing device and may also be referred to herein as a "computing node," host, "or" host device. Although only server 12A coupled to TOR switch 16A is shown in detail in fig. 1, data center 10 may include many additional servers coupled to other TOR switches 16 of data center 10.

The switching fabric 14 in the illustrated example includes interconnected top of rack (TOR) (or other "leaf") switches 16A-16N (collectively, "TOR switches 16") that are coupled to a distribution layer of chassis (or "backbone" or "core") switches 18A-18M (collectively, "chassis switches 18"). Although not shown, the data center 10 may also include, for example, one or more non-edge switches, routers, centers, gateways, security devices such as firewalls, intrusion detection and/or prevention devices, servers, computer terminals, laptop computers, printers, databases, wireless mobile devices such as cellular telephones or personal digital assistants, wireless access points, bridges, cable modems, application accelerators, or other network devices. Data center 10 may also include one or more Physical Network Functions (PNFs) such as physical firewalls, load balancers, routers, route reflectors, broadband Network Gateways (BNGs), mobile core network elements, and other PNFs.

In this example, TOR switch 16 and chassis switch 18 provide server 12 with redundant (multi-homed) connections to IP fabric 20 and service provider network 7. The chassis switches 18 aggregate traffic flows and provide connections between TOR switches 16. TOR switches 16 may be network devices that provide layer 2 (MAC) and/or layer 3 (e.g., IP) routing and/or switching functions. TOR switch 16 and chassis switch 18 may each include one or more processors and memory and may execute one or more software processes. Chassis switch 18 is coupled to IP fabric 20, which may perform layer 3 routing to route network traffic between data center 10 and client stations 11 through service provider network 7. The switching architecture of the data center 10 is merely an example. For example, other switching fabrics may have more or fewer switching layers. IP fabric 20 may include one or more gateway routers.

The term "packet flow", "traffic flow" or simply "flow" refers to a group of packets originating from a particular source device or endpoint and sent to a particular destination device or endpoint. A single packet stream may be identified by a 5-tuple: < source network address, destination network address, source port, destination port, protocol >. The 5-tuple typically identifies the packet stream to which the received packet corresponds. An n-tuple refers to any n items extracted from a 5-tuple. For example, a 2-tuple of a data packet may refer to a < source network address, destination network address > or a combination of < source network address, source port > of the data packet.

The servers 12 may each represent a computing server or a storage server. For example, each server 12 may represent a computing device configured to operate in accordance with the techniques described herein, such as an x86 processor-based server. Server 12 may provide Network Function Virtualization Infrastructure (NFVI) for NFVI architecture.

Any of the servers 12 may be configured with virtual execution elements, such as pod or virtual machines, by virtualizing the resources of the servers to provide some measure of isolation between one or more processes (applications) executing on the servers. "hypervisor-based" or "hardware level" or "platform" virtualization refers to creating virtual machines, each virtual machine including a guest operating system for performing one or more processes. Typically, virtual machines provide a virtualized/guest operating system for executing applications in an isolated virtual environment. Because the virtual machines are virtualized from the physical hardware of the host server, the executing applications are isolated from the hardware of the other virtual machines and hosts. Each virtual machine may be configured with one or more virtual network interfaces for communicating over a corresponding virtual network.

Virtual networks are logical constructs implemented on top of physical networks. Virtual networks may be used to replace VLAN-based quarantine and provide multi-tenancy in virtualized data centers, such as data center 10. Each tenant or application may have one or more virtual networks. Each virtual network may be isolated from all other virtual networks unless explicitly allowed by the security policy.

The virtual network may be connected to and extended over a physical multiprotocol label switching (MPLS) layer 3 virtual private network (L3 VPN) and an Ethernet Virtual Private Network (EVPN) network using a data center 10 gateway router (not shown in fig. 1). Virtual networks may also be used to implement Network Function Virtualization (NFV) and service chaining.

The virtual network may be implemented using various mechanisms. For example, each virtual network may be implemented as a Virtual Local Area Network (VLAN), a Virtual Private Network (VPN), or the like. The virtual network may also be implemented using two networks, a physical underlay network consisting of the IP fabric 20 and the switch fabric 14, and a virtual overlay network. The role of the physical underlay network is to provide an "IP fabric" that provides unicast IP connectivity from any physical device (server, storage device, router, or switch) to any other physical device. The underlying network may provide uniform low-latency, non-blocking, high-bandwidth connectivity from any point in the network to any other point in the network.

As described further below with respect to virtual router 21 (shown and also referred to herein as "vruter 21"), virtual routers running in server 12 create a virtual overlay network over the physical underlay network using a grid of dynamic "tunnels" between them. These overlay tunnels may be MPLS over GRE/UDP tunnels, or VXLAN tunnels, or NVGRE tunnels, for example. The underlying physical routers and switches may not store any per-tenant state of the virtual machine or other virtual execution element, such as any Media Access Control (MAC) address, IP address, or policy. The forwarding tables of the underlying physical routers and switches may contain, for example, only the IP prefix or MAC address of the physical server 12. (gateway routers or switches connecting virtual networks to physical networks are exceptions and may contain tenant MAC or IP addresses

The virtual router 21 of the server 12 typically contains per-tenant states. For example, they may contain separate forwarding tables (routing instances) for each virtual network. The forwarding table contains the IP prefix (in the case of layer 3 coverage) or MAC address (in the case of layer 2 coverage) of the virtual machine or other virtual execution element (e.g., pod of the container). No single virtual router 21 needs to contain all IP prefixes or all MAC addresses of all virtual machines in the entire data center. A given virtual router 21 need only contain those route instances that exist locally on the server 12 (i.e., that have at least one virtual execution element that exists on the server 12).

"container-based" or "operating system" virtualization refers to the virtualization of an operating system to run multiple isolation systems on a single machine (virtual or physical). Such quarantine systems represent containers such as those provided by open source DOCKER container applications or by CoreOS Rkt ("rock"). As with virtual machines, each container is virtualized and can remain isolated from hosts and other containers. However, unlike virtual machines, each container may omit a separate operating system, but instead provide application suites and application-specific libraries. Typically, the containers are executed by the host as isolated user space instances, and the operating system and common library may be shared with other containers executing on the host. Thus, the container may require less processing power, storage, and network resources than a virtual machine ("VM"). The group of one or more containers may be configured to share one or more virtual network interfaces for communication over a corresponding virtual network.

In some examples, containers are managed by their host kernel to allow limitation and prioritization of resources (CPUs, memory, block I/O, networks, etc.) without the need to launch any virtual machines, and in some cases use namespace isolation functions including process trees, networking, user identifiers, and installed file systems that allow for completely isolating the operating environment view of an application (e.g., a given container). In some examples, the container may be deployed according to a Linux container (LXC), which is an operating system level virtualization method for running multiple isolated Linux systems (containers) on a control host using a single Linux kernel.

Server 12 hosts virtual network endpoints of one or more virtual networks operating on a physical network represented herein by IP fabric 20 and switch fabric 14. Although described primarily with respect to a data center-based switching network, other physical networks, such as the service provider network 7, may carry one or more virtual networks.

Each server 12 may host one or more virtual execution elements, each virtual execution element having at least one virtual network endpoint for one or more virtual networks configured in a physical network. A virtual network endpoint of a virtual network may represent one or more virtual execution elements that share a virtual network interface of the virtual network. For example, a virtual network endpoint may be a virtual machine, a set of one or more containers (e.g., pod), or another virtual execution element, such as a layer 3 endpoint for a virtual network. The term "virtual execution element" encompasses virtual machines, containers, and other virtualized computing resources that provide an at least partially independent execution environment for an application. The term "virtual actuator" may also encompass the pod of one or more containers. The virtual execution element may represent an application workload. As shown in fig. 1, server 12A hosts a virtual network endpoint in the form of pod 22 having one or more containers. However, given the hardware resource limitations of the server 12, the server 12 may execute as many virtual execution elements as practical. Each virtual network endpoint may use one or more virtual network interfaces to perform packet I/O or otherwise process packets. For example, the virtual network endpoint may use one virtual hardware component (e.g., SR-IOV virtual function) enabled by NIC 13A to perform packet I/O and receive/transmit packets over one or more communication links with TOR switch 16A. Other examples of virtual network interfaces are described below.

The servers 12 each include at least one Network Interface Card (NIC) 13 that each includes at least one interface to exchange data packets with the TOR switch 16 over a communications link. For example, the server 12A includes the NIC 13A. Any of the NICs 13 may provide one or more virtual hardware components 21 for virtualizing input/output (I/O). The virtual hardware component for I/O may be a virtualization of a physical NIC ("physical function"). For example, in single root I/O virtualization (SR-IOV), which is described in the peripheral interface special interest group SR-IOV specification, the PCIe physical functions of a network interface card (or "network adapter") are virtualized to present one or more virtual network interfaces as "virtual functions" for use by the corresponding endpoints executing on server 12. In this way, the virtual network endpoints may share the same PCIe physical hardware resources, and the virtual functions are examples of virtual hardware components 21. As another example, one or more servers 12 may implement virtualization (Virtio), such as a paravirtualized framework available to the Linux operating system that provides emulated NIC functionality as a type of virtual hardware component to provide virtual network interfaces to virtual network endpoints. As another example, one or more servers 12 may implement an open vSwitch to perform distributed virtual multi-layer switching between one or more virtual NICs (vnics) of a hosted virtual machine, where such vnics may also represent one type of virtual hardware component that provides a virtual network interface to a virtual network endpoint.

In some examples, the virtual hardware component is a virtual I/O (e.g., NIC) component. In some examples, the virtual hardware component is an SR-IOV virtual function. In some examples, any of servers 12 may implement a Linux bridge that emulates a hardware bridge and forwards data packets between virtual network interfaces of servers or between virtual network interfaces of servers and physical network interfaces of servers. For a Docker implementation of containers hosted by a server, linux bridge, or other operating system bridge executing on the server, exchanging data packets between containers may be referred to as a "Docker bridge. The term "virtual router" as used herein may include a Contrail or Tungsten structured virtual router, an Open VSwitch (OVS), an OVS bridge, a Linux bridge, a Docker bridge, or other devices and/or software located on a host device and performing switching, bridging, or routing of data packets between virtual network endpoints of one or more virtual networks, where the virtual network endpoints are hosted by one or more servers 12.

Any of the NICs 13 may include an internal device switch to exchange data between virtual hardware components associated with the NIC. For example, for an NIC supporting SR-IOV, the internal device switch may be a Virtual Ethernet Bridge (VEB) to exchange between SR-IOV virtual functions and, correspondingly, between endpoints configured to use SR-IOV virtual functions, where each endpoint may include a guest operating system. The internal device switch may alternatively be referred to as a NIC switch, or for SR-IOV implementation, as a SR-IOV NIC switch. The virtual hardware component associated with NIC 13A may be associated with a layer 2 destination address that may be assigned by NIC 13A or a software process responsible for configuring NIC 13A. A physical hardware component (or "physical function" implemented by the SR-IOV) is also associated with the layer 2 destination address.

Each of the one or more servers 12 may include a virtual router 21 that executes one or more routing instances for corresponding virtual networks within the data center 10 to provide virtual network interfaces and route data packets between virtual network endpoints. Each routing instance may be associated with a network forwarding table. Each routing instance may represent a virtual route forwarding instance (VRF) for an internet protocol-virtual private network (IP-VPN). The data packets received by the virtual router 21 of the server 12A, for example, from the underlying physical network fabric (i.e., IP fabric 20 and switching fabric 14) of the data center 10, may include an outer header to allow the physical network fabric to tunnel the payload or "inner data packet" to the physical network address of the network interface card 13A of the server 12A executing the virtual router. The outer header may include not only the physical network address of the network interface card 13A of the server, but also a virtual network identifier, such as a VxLAN label or a multiprotocol label switching (MPLS) label, that identifies one of the virtual networks and the corresponding routing instance performed by the virtual router 21. The inner data packet includes an inner header having a destination network address conforming to a virtual network addressing space of the virtual network identified by the virtual network identifier.

The virtual router 21 terminates the virtual network overlay tunnel and determines the virtual network of the received packet based on the tunnel encapsulation header of the packet and forwards the packet to the appropriate destination virtual network endpoint of the packet. For server 12A, for example, for each data packet outbound from a virtual network endpoint (e.g., pod 22) hosted by server 12A, virtual router 21 appends a tunnel encapsulation header of the virtual network that indicates the data packet to generate an encapsulated or "tunnel" data packet, and virtual router 21 outputs the encapsulated data packet to a physical destination computing device, such as another one of servers 12, via an overlay tunnel of the virtual network. As used herein, virtual router 21 may perform operations of a tunneling endpoint to encapsulate an inner packet originating from a virtual network endpoint source to generate a tunnel packet, and decapsulate the tunnel packet to obtain an inner packet for routing to other virtual network endpoints.

In some examples, virtual router 21 may be kernel-based and execute as part of the kernel of the operating system of server 12A.

In some examples, the virtual router 21 may be a virtual router that enables a data plane forwarding tool (Data Plane Development Kit, DPDK). In such an example, the virtual router 21 uses DPDK as a data plane. In this mode, the virtual router 21 operates as a user space application linked to a DPDK library (not shown). This is a performance version of the virtual router and is typically used by telco corporation, where VNFs are typically DPDK-based applications. The performance of the virtual router 21 as a DPDK virtual router can achieve ten times higher throughput than a virtual router operating as a kernel-based virtual router. The physical interface is used by the Polling Mode Driver (PMD) of DPDK instead of the interrupt-based driver of Linux kernel.

user-I/O (UIO) core modules such as vfio or uio_pci_geneic may be used to expose registers of the physical network interface into the user space so that they are accessible by the DPDKPMD. When NIC 13A is bound to the UIO driver, it moves from Linux kernel space to user space and is therefore no longer managed or visible by the Linux OS. Thus, it is the DPDK application (i.e., virtual router 21A in this example) that fully manages NIC 13. This includes packet polling, packet processing, and packet forwarding. The user data packet processing step may be performed by the virtual router 21DPDK data plane with limited or no participation by the core (where the core is not shown in fig. 1). The nature of this "polling mode" makes virtual router 21DPDK data plane packet processing/forwarding much more efficient than interrupt mode, especially when packet rates are high. There is limited or no interruption and context switch during packet I/O. Additional details of an example of DPDK vruter can be found below: "DAY ONE: CONTRAIL DPDK vROUTER, "2021, kiran KN et al, juniper Networks Inc., which is incorporated herein by reference in its entirety.

The computing infrastructure 8 implements an automation platform for automating the deployment, scaling, and operation of virtual execution elements across servers 12 to provide a virtualized infrastructure for executing application workloads and services. In some examples, the platform may be a container orchestration system that provides a container-centric infrastructure for automating the deployment, scaling, and manipulation of containers to provide a container-centric infrastructure. "orchestration" in the context of virtualized computing infrastructure generally refers to provisioning, scheduling, and managing virtual execution elements and/or applications and services executing on such virtual execution elements to host servers available to an orchestration platform. Container orchestration may facilitate container reconciliation and involve deploying, managing, scaling, and configuring containers to host servers, e.g., by a container orchestration platform. Illustrative examples of orchestration platforms include Kubernetes (container orchestration system), docker Swarm, meso/Marathon, openShift, openStack, VMware, and Amazon ECS.

The elements of the automation platform of the computing infrastructure 8 include at least a server 12, an orchestrator 23, and a network controller 24. The container may be deployed to the virtualized environment using a cluster-based framework in which a cluster master node of the cluster manages the deployment of the container to and operation of one or more cluster work nodes of the cluster. The terms "master node" and "working node" as used herein include different orchestration platform terms for similar devices that distinguish between the primary management elements of the cluster and the primary container hosting devices of the cluster. For example, the Kubernetes platform uses the terms "cluster hosts" and "working nodes", while the Docker Swarm platform refers to a cluster manager and cluster nodes.

The orchestrator 23 and the network controller 24 may execute on separate computing devices, on the same computing device. Each of orchestrator 23 and network controller 24 may be a distributed application executing on one or more computing devices. Orchestrator 23 and network controller 24 may implement respective master nodes for one or more clusters, each cluster having one or more working nodes (also referred to as "compute nodes") implemented by respective servers 12.

In general, network controller 24 controls the network configuration of the data center 10 architecture, for example, to establish one or more virtual networks for packet communications between virtual network endpoints. Network controller 24 provides a logically and in some cases physically centralized controller to facilitate the operation of one or more virtual networks within data center 10. In some examples, network controller 24 may operate in response to configuration inputs received from orchestrator 23 and/or an administrator/operator. Additional information regarding exemplary operation of the network controller 24 in connection with other devices or other software-defined network operations of the data center 10 is found in international application number PCT/US2013/044378 entitled "PHYSICAL PATH DETERMINATION FOR VIRTUAL NETWORK PACKET FLOWS" filed on day 5 of 6 in 2013 and U.S. patent application number 14/226,509 entitled "TUNNELED PACKET AGGREGATION FOR VIRTUAL NETWORKS" filed on day 26 of 3 in 2014, each of which is incorporated herein by reference as if fully set forth herein.

In general, orchestrator 23 controls the deployment, scaling, and operation of containers across the cluster of servers 12, as well as the provision of computing infrastructure, which may include container-centric computing infrastructure. The orchestrator 23 and in some cases the network controller 24 may implement respective cluster hosts for one or more Kubernetes clusters. As an example, kubernetes is a container management platform that provides portability across public and private clouds, each of which may provide a virtualization infrastructure to the container management platform. Exemplary components of the Kubernetes orchestration system are described below with reference to fig. 3.

In one example, pod 22 is an example of a Kubernetes pod and virtual network endpoint. A pod is a group of one or more logically related containers (not shown in fig. 1), shared storage of containers, and options for how to run containers. When instantiated for execution, a pod may alternatively be referred to as a "pod copy". Each container of pod 22 is one example of a virtual execution unit. The containers of the pod are always co-located on a single server, co-scheduled, and run in a shared context. The shared context of pod may be a set of Linux namespaces, cgroups, and other aspects of isolation.

In the context of pod, a separate application may apply additional child isolation. Typically, the containers within the pod have public IP addresses and port space and are able to detect each other via a local host. Because they have shared context, the containers within the pod also communicate with each other using inter-process communication (IPC). Examples of IPC include SystemV semaphores or POSIX shared memory. In general, containers that are members of different pod have different IP addresses, and cannot communicate through IPC in the absence of a configuration for implementing this feature. Containers that are members of different pod typically instead communicate with each other via the pod IP address.

The server 12A includes a container platform 19 for running containerized applications, such as the containerized application of pod 22. Container platform 19 receives requests from orchestrator 23 to obtain and host containers in server 12A. The container platform 19 obtains and executes containers.

The Container Network Interface (CNI) 17 configures a virtual network interface for the virtual network endpoint. Orchestrator 23 and container platform 19 use CNI 17 to manage networking of pod including pod 22. For example, the CNI 17 creates a virtual network interface to connect a pod to the virtual router 21, and enables the container of such a pod to communicate with other virtual network endpoints through the virtual network via the virtual network interface. The CNI 17 may, for example, insert a virtual network interface for a virtual network into a network namespace for a container in the pod 22, and configure (or request configuration) the virtual network interface for the virtual network in the virtual router 21, such that the virtual router 21 is configured to send data packets received from the virtual network to the container of the pod 22 via the virtual network interface, and send data packets received from the container of the pod 22 on the virtual network via the virtual network interface. The CNI 17A may assign a network address (e.g., a virtual IP address for a virtual network) and may establish a route for the virtual network interface.

In Kubernetes, by default, all pods can communicate with all other pods without using Network Address Translation (NAT). In some cases, orchestrator 23 and network controller 24 create a service virtual network and a pod virtual network that are shared by all namespaces, from which service and pod network addresses are assigned, respectively. In some cases, all of the pod's in all namespaces caused in the Kubernetes cluster may be able to communicate with each other, and the network addresses of all of the pod's may be allocated from the pod subnet specified by the orchestrator 23. When a user creates an isolated namespace for a pod, orchestrator 23 and network controller 24 can create a new pod virtual network and a new shared services virtual network for the new isolated namespace. The pod in the isolated namespace caused in the Kubernetes cluster extracts the network address from the new pod virtual network, and the corresponding service for such pod extracts the network address from the new service virtual network.

CNI 17 may represent a library, plug-in, module, runtime, or other executable code for server 12A. The CNI 17A may at least partially conform to the Container Network Interface (CNI) specification or rkt networking proposal. CNI 17 may represent Contrail, openContrail, multus, calico, cRPD or other CNIs. CNI 17 may alternatively be referred to as a network plug-in or CNI instance. For example, separate CNIs can be invoked by the Multus CNI to establish different virtual network interfaces for pod 22.

CNI 17 may be invoked by orchestrator 23. For the purposes of the CNI specification, the container may be considered synonymous with the Linux network namespace. What elements correspond to depends on the particular container runtime implementation: for example, in an implementation of an application container specification such as rkt, each pod runs in a unique network namespace. However, in Docker, a network namespace typically exists for each individual Docker container. For the purposes of the CNI specification, a network refers to a set of entities that are uniquely addressable and that can communicate with each other. This may be a separate container, a machine/server (real or virtual) or some other network device (e.g., a router). Containers may be conceptually added to or removed from one or more networks. The CNI specification specifies a number of considerations for conforming to a plug-in ("CNI plug-in").

pod 22 includes one or more containers. In some examples, pod 22 includes a containerized DPDK workload designed to use DPDK to accelerate packet processing, such as by exchanging data with other components using a DPDK library. In some examples, virtual router 21 may execute as a containerized DPDK workload.

The pod 22 is configured with a virtual network interface 26 for sending and receiving data packets with the virtual router 21. The virtual network interface 26 may be the default interface of the pod 22. pod 22 may implement virtual network interface 26 as an ethernet interface (e.g., named "eth 0"), while virtual router 21 may implement virtual network interface 26 as a tap interface, a virtio-user interface, or other type of interface.

The pod 22 and virtual router 21 exchange data packets using the virtual network interface 26. The virtual network interface 26 may be a DPDK interface. The pod 22 and virtual router 21 can establish the virtual network interface 26 using vhost. pod 22 may operate according to an aggregation model. pod 22 may use a virtual device, such as a virtio device with a vhost user adapter, for user space container inter-process communication of virtual network interface 26.

CNI 17 may configure virtual network interface 26 for pod 22 in conjunction with one or more of the other components shown in fig. 1. Any container of pod 22 can utilize (i.e., share) virtual network interface 26 of pod 22.

Virtual network interface 26 may represent a virtual ethernet ("veth") pair, where each end of the pair is a separate device (e.g., a Linux/Unix device), one end of the pair is assigned to pod 22, and one end of the pair is assigned to virtual router 21. The veth pair or the end of the veth pair is sometimes referred to as a "port". The virtual network interface may represent a macvlan network with Media Access Control (MAC) addresses assigned to pod 22 and virtual router 21 for communication between the pods 22 and the containers of virtual router 21. The virtual network interface may alternatively be referred to as, for example, a Virtual Machine Interface (VMI), pod interface, container network interface, tap interface, veth interface, or simply a network interface (in a particular context).

In the exemplary server 12A of fig. 1, pod 22 is a virtual network endpoint in one or more virtual networks. Orchestrator 23 may store or otherwise manage configuration data for application deployment that specifies a virtual network and that pod 22 (or one or more containers therein) is a virtual network endpoint of the virtual network. Orchestrator 23 may receive configuration data from, for example, a user, operator/administrator, or other computing system.

As part of the process of creating pod 22, orchestrator 23 requests network controller 24 to create a corresponding virtual network interface for one or more virtual networks (indicated in the configuration data). The pod 22 may have a different virtual network interface for each virtual network to which the pod 22 belongs. For example, virtual network interface 26 may be a virtual network interface for a particular virtual network. Additional virtual network interfaces (not shown) may be configured for other virtual networks.

The network controller 24 processes the request to generate interface configuration data for the virtual network interface of the pod 22. The interface configuration data may include a container or pod unique identifier and a list or other data structure specifying network configuration data for configuring the virtual network interfaces for each virtual network interface. The network configuration data of the virtual network interface may include a network name, an assigned virtual network address, a MAC address, and/or a domain name server value. The following is an example of interface configuration data in JavaScript Object Notation (JSON) format.

The network controller 24 transmits the interface configuration data to the server 12A, more specifically, in some cases, to the virtual router 21. To configure the virtual network interface of pod 22, orchestrator 23 may invoke CNI 17. The CNI 17 acquires interface configuration data from the virtual router 21 and processes it. The CNI 17A creates each virtual network interface specified in the interface configuration data. For example, the CNI 17 can attach one end of a veth pair implementing the management interface 26 to the virtual router 21, and can attach the other end of the same veth pair to a pod 22 that can implement it using a virtio-user.

The following are exemplary interface configuration data for pod 22 of virtual network interface 26.

[{

//virtual network interface 26

"id":"fe4bab62-a716-11e8-abd5-0cc47a698428",

"instance-id":"fe3edca5-a716-11e8-822c-0cc47a698428",

"ip-address":"10.47.255.250",

"plen":12,

"vn-id":"56dda39c-5e99-4a28-855e-6ce378982888",

"vm-project-id":"00000000-0000-0000-0000-000000000000",

"mac-address":"02:fe:4b:ab:62:a7",

"system-name":"tapeth0fe3edca",

"rx-vlan-id":65535,

"tx-vlan-id":65535,

"vhostuser-mode":0,

“v6-ip-address”:“::“,

“v6-plen”:,

“v6-dns-server”:“::”,

“v6-gateway”:“::”,

"dns-server":"10.47.255.253",

"gateway":"10.47.255.254",

"author":"/usr/bin/contrail-vrouter-agent",

"time":"426404:56:19.863169"

}]

Conventional CNI plug-ins are invoked by the container platform/runtime, receive Add commands from the container platform to Add the container to a single virtual network, and such plug-ins may then be invoked to receive Del (ete) commands from the container/runtime and remove the container from the virtual network. The term "call" may refer to an instantiation of a software component or module in memory as executable code for execution by a processing circuit.

In accordance with the techniques of this disclosure, network controller 24 is a cloud-native distributed network controller for a Software Defined Network (SDN) that is implemented using one or more configuration nodes 30 and one or more control nodes 32. Each configuration node 30 itself may be implemented using one or more cloud native component microservices. Each control node 32 itself may be implemented using one or more cloud native component microservices.

In some examples, configuration node 30 may be implemented by extending a local orchestration platform to support custom resources of the orchestration platform for the software-defined network, and more specifically, to provide a northbound interface to the orchestration platform to support the intended driven/declarative creation and management of the virtual network by, for example, configuring virtual network interfaces for virtual execution elements, configuring the underlying network of connection servers 12, configuring overlay routing functions including overlay tunnels for the virtual network and overlay trees for multicast layers 2 and 3.

As part of the SDN architecture shown in fig. 1, network controller 24 may be multi-tenant aware and support multi-tenants for orchestration platforms. For example, network controller 24 may support Kubernetes role-based access control (RBAC) architecture, local Identity Access Management (IAM), and external IAM integration. The network controller 24 may also support Kubernetes defined networking fabric and high cascading networking features such as virtual networking, BGPaaS, networking policies, service chaining, and other telco features. The network controller 24 may support network isolation using a virtual network architecture and support layer 3 networking.

To interconnect multiple virtual networks, network controller 24 may use (and be configured in the underlying and/or virtual router 21) ingress and egress policies defined using Virtual Network Router (VNR) resources. The virtual network router resources may be used to define connectivity between virtual networks by configuring the ingress and egress of routing information between respective routing instances for implementing the virtual networks in the SDN architecture. A single network controller 24 may support multiple Kubernetes clusters, and the VNR thus allows for connecting multiple virtual networks in namespaces, virtual networks in different namespaces, kubernetes clusters, and cross-Kubernetes clusters. The VNR may also be extended to support virtual network connectivity between multiple instances of the network controller 24. VNR may also be alternatively referred to herein as a Virtual Network Policy (VNP) or virtual network topology.

As shown in the example of fig. 1, network controller 24 may maintain configuration data (e.g., config. 30) representing Virtual Networks (VNs) 50A-50N ("VNs 50") that represent policies and other configuration data for establishing VNs 50 within data center 10 through a physical underlying network and/or virtual router, such as virtual router 21 ("vruter 21"). Network controller 24 may also maintain configuration data (e.g., config.30) representing Virtual Network Routers (VNRs) 52A-52N ("VNRs 52") that may be implemented, at least in part, using policies and other configuration data in order to establish interconnectivity between VNs 50.

A user, such as an administrator, may interact with UI 60 of network controller 24 to define VN 50 and VNR 52. In some cases, UI 60 represents a Graphical User Interface (GUI) that facilitates entry of configuration data defining VNs 50 and VNRs 52. In other examples, UI 60 may represent a Command Line Interface (CLI) or other type of interface. Assuming UI 60 represents a graphical user interface, the administrator may define VN 50 by arranging graphical elements representing different pods, such as pod 22, to associate the pods with VN 50, with any of VN 50 enabling communication between one or more pods assigned to the VN.

In this regard, the administrator may understand Kubernetes or other orchestration platform, but may not fully understand the underlying infrastructure supporting VN 50. Some controller architectures, such as Contrail, may configure VN 50 based on a network protocol that is similar, if not substantially similar, to the routing protocol in a traditional physical network. For example, contrail may utilize concepts from Border Gateway Protocol (BGP), which is a routing protocol used to communicate routing information within, and sometimes between, so-called Autonomous Systems (ASs).

Different versions of BGP exist, such AS Internal BGP (iBGP) for transporting routing information within the ases and external BGP (eBGP) for transporting routing information between ases. ASE may relate to the concept of items within Contrail, which is also similar to the namespaces in Kubernetes. In each instance of an AS, project, and namespace, an AS like project and namespace may represent a set of one or more networks (e.g., one or more of VNs 50) that may share routing information and thereby facilitate interconnectivity between networks (or, in this instance, VNs 50).

In its simplest form, VNR 52 represents a logical abstraction of a router that is set in the context of Kubernetes, where VNR 52 may be defined as a custom resource that facilitates interconnectivity between VNs 50. Given that the Kubernetes administrator may not fully understand the complex dissemination of routing information in accordance with a complex routing protocol, such as BGP, various aspects of cloud-originated networking techniques may facilitate abstraction of the underlying routing protocol (or a complementary process of a Contrail or other controller architecture) as the VNR 52.

That is, rather than taking a means of defining how routing occurs between two or more VNRs 50, an administrator may define one or more VNRs 52 to interconnect VNs 50 without having to manually develop and deploy extensive policies and/or routing instance configurations to enable exchange of routing information between such VNs 50. Instead, an administrator (who may have little understanding of the routing protocol) may define custom resources (e.g., one or more of VNRs 52) using familiar Kubernetes syntax/semantics (or even merely by dragging a graphical element and specifying an interconnection between that graphical element representing, for example, VNR 52A and graphical elements representing, for example, VNs 50A and 50N).

In this regard, an administrator may readily interconnect VNs 50 to VNR 50 using the logical abstraction shown in the example of fig. 1, whereupon network controller 24 may convert VNR 50 to the underlying route target to automatically (meaning with little or no human intervention) cause routing information of VNs 50A and 50N to be exchanged and enable communication (meaning exchange of packets or other data) between VNs 50A and 50N.

Assuming that an administrator may employ familiar Kubernetes syntax/semantics to configure VNR 50 instead of configuring complex configuration data that conforms to routing protocol syntax/semantics, network controller 24 may facilitate a better user experience while also facilitating more efficient operation of data center 8 itself. That is, having an administrator enter such configuration data that is unfamiliar to the administrator may cause the underlying resources of data center 8 (in terms of processing cycles, memory, bus bandwidth, etc. and associated power) to be wasted, while also delaying the proper implementation of the network topology (which may prevent successful routing of packets and other data between VNs 50). This delay can not only frustrate the administrator, but can also frustrate the clients associated with the VN 50, which may require rapid operation of the VN 50 to achieve various business goals. By enabling an administrator to easily facilitate communication between VNs 50 using a logical abstraction shown as VNR 50, data center 8 itself may experience more efficient operations (including processor cycles, memory, bus bandwidth, and associated power in terms of the computing resources described above) while providing a better user experience for the administrator and clients.

Network controller 24, representing an SDN architecture system of data center 10, includes processing circuitry to implement configuration nodes and control nodes (as described in more detail with respect to the example of fig. 3). The network controller 24 may be configured to interconnect a first virtual network (e.g., VN 50A) and a second virtual network (e.g., VN 50N) operating within an SDN architecture system represented by the data center 10. The network controller 24 may be configured to define a logical abstraction of one or more policies in order to perform such interconnection via one or more VNRs 52, e.g., VNR 52A.

Policies may include ingress and egress policies regarding routing information maintained by the virtual network (which may refer to VNs 50A and 50N in this example). That is, kubernetes may be extended by custom resources on behalf of VNR 52A to translate VNR 52A into one or more ingress and egress policies that are deployed with respect to VNs 50A and 50N to configure interworking via routing information distribution between VNs 50A and 50N. Once configured, VN 50A may bring out routing information (e.g., a route representing VN 50A) to VN 50N and bring in routing information (e.g., a route representing VN 50N) to VN 50A. Likewise, VN 50N may bring out routing information (e.g., a route representing VN 50N) to VN 50A and bring in routing information (e.g., a route representing VN 50A) to VN 50N.

The abstraction may hide the underlying routing configuration to enable such routing leakage, such as defining routing information to the routing targets for implementing the ingress and egress of the routing instances of VNs 50A and 50N. Instead, network controller 24 may translate VNR 52A into a common routing target and configure communication of routing information via the common routing target for implementing routing instances of VNs 50A and 50N (in this example).

To achieve mesh connectivity, network controller 24 may configure the ingress and egress of routing instances of VNs 50A, VN 50N and VNR 52A with routing targets associated with VNs 50A, VN N and VNR 52A. To achieve center spoke connectivity, network controller 24 may configure the extraction of routing instances associated with VNs 50A and 50N to extract routing information to routing instances associated with VNR 52A (acting as a center), and configure the routing instances of VNR 52A to extract routing information to routing instances associated with VNs 50A and 50N. In this center spoke connectivity, VN 50A and VN 50N may not communicate directly with each other. More information about VNR can be found in U.S. application No. 17/809,659, entitled "VIRTUAL NETWORK ROUTERS FOR CLOUD NATIVE SOFTWARE-DEFINED NETWORK ARCHITECTURES," filed on 7 month 29 of 2022, the entire contents of which are incorporated herein by reference.

While VNR may reduce complexity in automatically converting VNR abstractions into underlying routing configurations, it is still difficult to visualize how a VNR will adapt a network topology that may include one or more VNs (such as VN 50A and VN 50N) to facilitate connectivity between VN 50A and VN 50N. Such difficulties may cause user errors in configuring VNRs such as VNR 52A, which may cause efficient operation of network controller 24 and the underlying network managed by network controller 24.

According to various aspects of the technology described in this disclosure, the network controller 24 may present a UI 60 (e.g., a graphical user interface, which may be represented as GUI 60), visualize the topology of the network through the UI 60, and interactively define VNRs, such as VNR 52A, through various interactions with the GUI 60. Adding VNR 52 to abstract policies for controlling the ingress and/or egress of routing information may simplify stitching network elements (such as VNs 50A and 50N) together to facilitate interconnection between such network elements. Aspects of the techniques described in this disclosure may provide a GUI 60 that is configured to graphically present a network topology that may include various network elements (such as VNs 50A and 50N) rather than defining VNR 52 involving complex declarations that may be required to network construction (such as routing targets, labels identifying network elements, etc.) by means of complex interfaces. The user may interface with the GUI 60 to visually configure the VNR 52A to interconnect two or more network elements.

The GUI 60 may receive user input (which may be referred to as input entered via the GUI 60), such as a graphical user input that drags and drops or otherwise defines the VNR 52A. The GUI 60 may communicate these inputs to the network controller 24 as a request to configure the VNR 52A. The user may input through interaction with GUI 60 to graphically define VNR 52A, select VNs 50A and 50N, and other network elements for which VNR 52A establishes interconnectivity (e.g., grid connectivity and/or center spoke connectivity). The network controller 24 may process these inputs and update the GUI 60 to present prompts and other GUIs 60 by which the user is prompted to define additional inputs required by the VNR 52A. The network controller 24 may reduce these inputs defining VNR 52A into various policies that control the ingress and/or egress of routing information between VNs 50A and 50N, and configure the policies in the appropriate network elements to facilitate the exchange of routing information that achieves interconnectivity between VNs 50A and 50N.

In operation, the network controller 24 may present the GUI 60 locally (via a local display). In some examples, GUI 60 may represent a web-based GUI that network controller 24 may remotely host for display via a client device (such as a remote computer or other terminal). Regardless of how the GUI 60 is presented, the GUI 60 may be configured to graphically represent the topology of the networks supported by the Software Defined Network (SDN) architecture system 8. For illustration purposes, it is assumed that the network includes VNs 50A and 50N. However, the network may include any number of VNs 50 or other network elements.

The network controller 24 may maintain configuration data (not shown in the example of fig. 1) defining the topology of the network, which the network controller 24 may transform into a graphical representation of the topology of the network. The graphical representation of the network topology may facilitate viewing of a portion or all of the network, wherein the GUI 60 may provide filtering and other operations to enable a user to view the graphical representation of the network topology at various levels of granularity, as described in more detail below.

GUI 60 may also be configured to dynamically generate graphical elements representing VNR 52A by which VN 50A and VN 50N are interconnected. GUI 60 may receive input from a user identifying VNRs 52A and VNs 50A and 50N, such as input indicating that a graphical element (e.g., a graphical icon) representing the VNR has been dragged over a network topology that may be proximate, and/or on one or more of VNs 50A and 50N. The GUI 60 may provide these inputs back to the network controller 24, which network controller 24 may update the GUI 60 to include additional cues for information about VNR 52A and VNs 50A and 50N in response to these inputs.

GUI 60 (via network controller 24) may iterate in this manner until VNR 52A has been successfully defined in a manner that achieves connectivity between VNs 50A and 50N. The network controller 24 may execute the configuration node 30 to verify the VNR 52A before invoking the control node 32 to configure the VNs 50A and 50N. Once successfully authenticated, control node 32 configures VNs 50A and 50N according to one or more policies to enable one or more of ingress and egress of routing information between VN 50A and VN 50N via VNR 52A.

As such, the GUI 60 configured in accordance with aspects of the technology described in this disclosure may enable inexperienced users (in the context of SDN architecture) to define VNR 52 without having extensive knowledge of network architecture (such as those listed above). In this way, the user may more effectively interact with the GUI 60 to define the VNR 52 to meet business and/or customer goals. GUI 60 may guide the user in reducing and possibly eliminating errors in configuring VNR 52 while also enabling the user to view the topology of the network and gain a better understanding of the network itself. By being easy to use and eliminating errors, the GUI 60 may reduce the need for a user to define complex configuration data while also avoiding such errors that may cause inefficient operation of the network controller (in terms of computing resources, such as processing cycles, memory bus bandwidth, etc., and associated power). In this way, the GUI 60 may improve the operation of the network controller itself.

In addition, network controller 24 may use network policies to implement multi-layer security. Kubernetes default behavior is that the pod communicate with each other. In order to apply network security policies, the SDN architecture implemented by the network controller 24 and the virtual router 21 may operate with CNI 17 as the CNI of Kubernetes. For layer 3, the isolation occurs at the network level, and the virtual network operates at L3. The virtual networks are connected by policies. Kubernetes local network policy provides security at layer 4. The SDN architecture may support Kubernetes network policies. Kubernetes network policies operate at Kubernetes namespace boundaries. SDN architecture may add custom resources for enhancing network policies. The SDN architecture may support application-based security. (in some cases, these security policies may be based on meta-tags in order to apply granular security policies in an extensible manner). For layer 4+, SDN architecture may support integration with container security devices and/or issio in some examples and may provide encryption support.

As part of the SDN architecture shown in fig. 1, the network controller 24 may support multi-cluster deployment, which is important for telco cloud and high-end enterprise use cases. For example, an SDN architecture may support multiple Kubernetes clusters. The cluster API may be used to support lifecycle management of Kubernetes clusters. KubefedV2 may be used to join configuration nodes 32 across Kubernetes clusters. Cluster API and KubefedV2 are optional components for supporting a single instance of the network controller 24 supporting multiple Kubernetes clusters.

SDN architecture may use web user interface and telemetry components to provide insight into infrastructure, clusters, and applications. Telemetry nodes may be cloud-native and include micro-services to support insight.

As a result of the above features and other features that will be described elsewhere herein, the computing infrastructure 8 implements an SDN architecture as cloud native and may present one or more of the following technical advantages. For example, the network controller 24 is a cloud-native, lightweight distributed application with a simplified installation footprint. This also facilitates easier and modular upgrades of the various component microservices of configuration node 30 and control node 32 (as well as any other components of other examples of network controllers described in this disclosure). The techniques may also enable optional cloud native monitoring (telemetry) and user interfaces, high performance data planes using containers connected to DPDK-enabled pod-based virtual routers, and in some cases cloud native configuration management with a configuration framework for existing orchestration platforms (such as Kubernetes or Openstack). As a cloud native architecture, the network controller 24 is a scalable and resilient architecture for addressing and supporting multiple clusters. In some cases, the network controller 24 may also support scalability and performance requirements for Key Performance Indicators (KPIs).

SDN architectures having features and technical advantages such as those described herein may be used to implement cloud-native telco clouds to support, for example, 5G mobile networking (and subsequent generations) and edge computing, as well as enterprise Kubernetes platforms including, for example, high-performance cloud-native application hosting. telco cloud applications are rapidly evolving towards containerized cloud-native methods. 5G fixed and mobile networks are driving the need to deploy workloads with significantly separated micro services, especially in the 5G Next-Gen RAN (5 GNR). The 5G NextGen Core (5 GNC) might be deployed as a set of micro-service based applications corresponding to each of the different components described by 3 GPP. When considered as a group of micro-service delivery applications, 5 GNCs can be a highly complex combination of pod with complex networking, security, and policy requirements. For such use cases, the cloud native SDN architecture described herein may be utilized, with well-defined constructs for networking, security, and policy. The network controller 24 may provide a relevant API capable of creating these complex structures.

Also, the User Plane Function (UPF) within 5GNC will be an ultra-high performance application. It can be delivered as a set of highly distributed high performance pods. The SDN architecture described herein is capable of providing very high throughput data planes (in terms of bits per section (bps) and packets per second (pps)). Integration with DPDK virtual router with recent performance enhancements, eBPF and with SmartNIC will help achieve the required throughput. The DPDK-based virtual router is described in more detail in U.S. application No. 17/649,632, filed on 1/2/2022, entitled "CONTAINERIZED ROUTER WITH VIRTUAL NETWORKING," which is incorporated herein by reference in its entirety.

High performance processing may also be relevant in gilans, as workloads migrate from more traditional virtualized workloads to containerized microservices. In the data plane of UPF and GiLAN services, such as GiLAN firewall, intrusion detection and prevention, virtualized IP multimedia subsystem (vIMS) voice/video, etc., the throughput will be high and continuous in terms of bps and pps. For control plane functions of 5GNC functions, such as access and mobility management functions (AMF), session Management Functions (SMF), etc., and for some GiLAN services (e.g., IMS), while absolute traffic may be modest in terms of bps, the advantage of small packets means that pps will remain high. In some examples, the SDN controller and data plane provide millions of packets per second per virtual router 21, as implemented on server 12. In a 5G Radio Access Network (RAN), in order to be remote from the proprietary vertical integrated RAN stack provided by a legacy radio provider, an open RAN decouples RAN hardware and software from among a number of components including non-RT Radio Intelligent Controllers (RIC), near real-time RIC, centralized Unit (CU) control planes and user planes (CU-CP and CU-UP), distributed Units (DU) and Radio Units (RU). The software components are deployed on the commodity server architecture, supplementing the programmable accelerator if necessary. The SDN architecture described herein may support O-RAN specifications.

Edge computation may be primarily directed to two different use cases. The first would be as support for a containerized telco infrastructure (e.g., 5G RAN, UPF, security functions) and the second would be for containerized service workload from telco and from third parties such as vendors or enterprise customers. In both cases, edge computation is in fact a special case of GiLAN, where traffic is interrupted for special handling at highly distributed locations. In many cases, these locations will have limited resources (power, cooling, space).

The SDN architecture described herein may be well suited to support the requirements of very lightweight coverage areas, may support computing and storage resources in stations remote from associated control functions, and may be location aware in a manner that deploys workload and storage. Some stations may have as few as one or two compute nodes that deliver a very specific set of services to a highly localized user or other set of services. There may be a hierarchy of stations where a central station is densely connected with many paths, a regional station is multiply connected with two to four uplink paths, and a remote edge station may have a connection to only one or two upstream stations.

This requires great flexibility in the way that SDN architecture can be deployed and the way (and location) in which tunnelled traffic in the overlay is terminated and bound into the core transport network (SRv, MPLS, etc.). Likewise, in a station hosting a telco cloud infrastructure workload, the SDN architecture described herein may support the dedicated hardware (GPU, smartNIC, etc.) required for high performance workloads. There may also be a workload that requires an SR-IOV. Thus, the SDN architecture may also support creating a VTEP at ToR and linking it back to the overlay as VXLAN.

It is expected that there will be a mix of fully distributed Kubernetes micro clusters, where each station runs its own master device, and SDN architecture can support a remote computing like scenario.

For use cases involving enterprise Kubernetes platforms, high performance cloud native applications power financial service platforms, online gaming services, and hosted application service providers. Cloud platforms delivering these applications must provide high performance, resilience to failures, and high security and visibility. Applications hosted on these platforms tend to be developed in-house. The application developer and platform owner cooperate with the infrastructure team to deploy and operate instances of the organized application. These applications often require high throughput (> 20Gbps per server) and low latency. Some applications may also use multicasting for signaling or payload traffic. Additional hardware and network infrastructure may be utilized to ensure availability. Applications and micro-services will utilize namespaces within the cluster for partitioning. Isolation between namespaces is critical in a high security environment. Although the default rejection policy is a standard situation in a zero-trust application deployment environment, additional network segments using virtual routing and forwarding instances (VRFs) add additional layers of security and allow overlapping network scopes to be used. Overlapping network scopes are key requirements of managed application hosting environments that tend to be standardized across a set of reachable endpoints for all managed clients.

Complex microservice-based applications tend to utilize complex network filtering. The SDN architecture described herein may deliver high performance firewalls for scaling. Such filtering may exhibit consistent forwarding performance with less delay degradation regardless of ruleset length or order. Some customers may also have some of the same management pressure as telcos about application separation, not only at the network layer, but also in the kernel. Finance and other aspects require data plane encryption, particularly when running on public clouds. In some examples, the SDN architecture described herein may include features for meeting these requirements.

In some examples, when the SDN architecture is automated through application of dev/test/stage/prod continuous integration/continuous development (CI/CD) pipelines, the SDN architecture may provide a GitOps friendly UX for strict change management control, auditing, and reliability of changes made in production several times per day, even hundreds of times per day.

Fig. 2 is a block diagram illustrating an example of a cloud native SDN architecture for cloud native networking in accordance with the techniques of this disclosure. SDN architecture 200 is shown in a manner that abstracts the underlying connectivity between the various components. In this example, the network controller 24 of the SDN architecture 200 includes configuration nodes 230A-230N ("configuration nodes" or "config nodes", collectively, "configuration nodes 230") and control nodes 232A-232K (collectively, "control nodes 232"). Configuration node 230 and control node 232 may represent exemplary embodiments of configuration node 30 and control node 32, respectively, of fig. 1. Configuration node 230 and control node 232, while illustrated as separate from server 12, may be implemented as one or more workloads on server 12.

Configuration node 230 provides a northbound, representational state transfer (REST) interface to support the intent-driven configuration of SDN architecture 200. Example platforms and applications that may be used to push intent to configuration node 230 include virtual machine orchestrator 240 (e.g., openstack), container orchestrator 242 (e.g., kubernetes), user interface 242, or other one or more applications 246. In some examples, SDN architecture 200 has Kubernetes as its underlying platform.

SDN architecture 200 is divided into a configuration plane, a control plane, and a data plane, and optionally a telemetry (or analysis) plane. The configuration plane is implemented with a horizontal scalable configuration node 230, the control plane is implemented with a horizontal scalable control node 232, and the data plane is implemented with a compute node.

At a high level, configuration nodes 230 use configuration store 224 to manage the state of configuration resources of SDN architecture 200. In general, a configuration resource (or more simply "resource") is a named object schema that includes data and/or methods describing a custom resource, and defines an Application Programming Interface (API) to create and manipulate data through an API server. One is naming of object patterns. The configuration resources may include Kubernetes native resources such as Pod, ingress, configuration map, service, role, namespace, node, network policy, or LoadBalancer.

The configuration resources may also include custom resources for extending the Kubernetes platform by defining Application Program Interfaces (APIs) that may not be available in the default installation of the Kubernetes platform. In examples of SDN architecture 200, the customized resources may describe physical infrastructure, virtual infrastructure (e.g., VN 50 and/or VNR 52), configuration, and/or other resources of SDN architecture 200. As part of configuring and operating SDN architecture 200, various custom resources (e.g., VNR 52 within vruter 21) may be instantiated. The instantiated resources (whether native or customized) may be referred to as objects or instances of resources, which are persistent entities in the SDN architecture 200, representing the intent (desired state) and the condition (actual state) of the SDN architecture 200.

Configuration node 230 provides an aggregated API for performing operations (i.e., creating, reading, updating, and deleting) on configuration resources of SDN architecture 200 in configuration store 224. Load balancer 226 represents one or more load balancer objects that load balance configuration requests among configuration nodes 230. Configuration store 224 may represent one or more etcd databases. Configuration node 230 may be implemented using ng inx.

SDN architecture 200 may provide networking for Openstack and Kubernetes. Openstack uses a plug-in architecture to support networking. The Openstack networking plug-in driver converts the Openstack configuration object into SDN architecture 200 configuration objects (resources) using virtual machine orchestrator 240 as Openstack. The compute node runs Openstack nova to spawn the virtual machine.

For the container orchestrator 242 as Kubernetes, sdn architecture 200 acts as Kubernetes CNI. As described above, kubernetes native resources (pod, service, portal, external load balancer, etc.) may be supported, and SDN architecture 200 may support custom resources of Kubernetes for advanced networking and security of SDN architecture 200.

Configuration node 230 provides REST monitoring to control node 232 to monitor configuration resource changes, which control node 232 functions within the computing infrastructure. Control node 232 receives configuration resource data from configuration node 230 by monitoring resources and builds a complete configuration graph. A given one of the control nodes 232 consumes configuration resource data associated with the control node and distributes the required configuration to the compute nodes (servers 12) via control interface 254 to the control plane aspects of virtual router 21 (i.e., virtual router agents-not shown in fig. 1). Any computing node 232 may receive only a portion of the graph as needed for processing. Control interface 254 may be XMPP. The number of deployed configuration nodes 230 and control nodes 232 may be a function of the number of clusters supported. To support high availability, the configuration plane may include 2n+1 configuration nodes 230 and 2N control nodes 232.

Control node 232 distributes routes among the computing nodes. Control nodes 232 exchange routes between control nodes 232 using iBGP, and control nodes 232 may peer with any external BGP-supported gateway or other router. The control node 232 may use a routing reflector.

Pod 250 and virtual machine 252 are examples of workloads that may be deployed to computing nodes by virtual machine orchestrator 240 or container orchestrator 242 and interconnected by SDN architecture 200 using one or more virtual networks.

User interface 244 may represent an example of UI 60. As described in more detail below, the user interface 244 may present a graphical representation of the network topology. The network may include two or more VNs, such as VN 50A and VN 50N. User interface 244 may represent an example of a graphical user interface (and thus may be referred to as a graphical user interface-GUI-244) that presents various interface elements, such as buttons, sliders, selectors, draggable interface elements (such as graphical icons), etc., through which a VNR, such as VNR 52A, is defined, through which two or more VNs are interconnected.

Fig. 3 is a block diagram illustrating another view of components of SDN architecture 200 in greater detail in accordance with the techniques of this disclosure. Configuration node 230, control node 232, and user interface 244 are illustrated as having their respective component micro services for implementing network controller 24 and SDN architecture 200 as a cloud native SDN architecture. Each of the component microservices may be deployed to a compute node.

Fig. 3 illustrates a single cluster divided into network controller 24, user interface 244, computing (server 12), and telemetry 260 features. As described above, the user interface 244 may represent an example of the UI 60. The configuration node 230 and the control node 230 together form the network controller 24.

Configuration node 230 may include a component micro-service API server 300 (or "Kubernetes API server 300" -corresponding controller 406 is not shown in fig. 3), a custom API server 301, a custom resource controller 302, and an SDN controller manager 303 (sometimes referred to as a "kube-manager" or "SDN kube-manager," where the orchestration platform for network controller 24 is Kubernetes). The Contrail-kube-manager is one example of SDN controller manager 303. The configuration node 230 extends the interface of the API server 300 with the custom API server 301 to form an aggregation layer supporting the data model of the SDN architecture 200. The SDN architecture 200 intent to configure may be a custom resource, as described above.

The control node 232 may include a component micro service control 320 and a core DNS 322. Control 320 performs configuration distribution and route learning and distribution as described above with reference to fig. 2.

The compute nodes are represented by servers 12. Each computing node includes a virtual router agent 316 and a virtual router forwarding element (v router) 318. One or both of virtual router agent 316 and vruter 318 may be component microservices. In general, virtual router agent 316 performs control-related functions. The virtual router agent 316 receives configuration data from the control node 232 and converts the configuration data into forwarding information for the vruter 318. The virtual router agent 316 may also perform firewall rule processing, set up flows of the vruter 318, and interface with orchestration plug-ins (CNI for Kubernetes and Nova plug-ins for Openstack). Virtual router agent 316 generates routes when a workload (pod or VM) is brought onto a computing node, and virtual router 316 exchanges such routes with control nodes 232 for distribution to other computing nodes (control nodes 232 distribute routes between control nodes 232 using BGP). The virtual router agent 316 also drops the route at the termination of the workload. The vruter 318 may support one or more forwarding modes, such as kernel mode, DPDK, smartNIC offload, and so on. In some examples of container architecture or virtual machine workload, the compute nodes may be Kubernetes worker/worker nodes or Openstack nova-compute nodes, depending on the particular orchestrator used.

One or more optional telemetry nodes 260 provide metrics, alarms, logs, and flow analysis. SDN architecture 200 telemetry utilizes cloud native monitoring services such as Promethyus, elastic, fluentd, kinaba stack (EFK) and Influx TSDB. The SDN architecture component microservices of the configuration node 230, the control node 232, the compute nodes, the user interface 244, and the analysis node (not shown) may generate telemetry data. The telemetry data may be consumed by the services of telemetry node 260. Telemetry node 260 may expose REST endpoints for users and may support insight and event correlation.

The optional user interface 244 includes a web User Interface (UI) 306 and UI backend 308 services. In general, the user interface 244 provides configuration, monitoring, visualization, security, and troubleshooting for SDN architecture components. Further, the user interface 244 (which may also be referred to as GUI 244, as described above) may present a graphical representation of the network topology. The network may include two or more VNs, such as VN 50A and VN 50N. GUI 244 may present various interface elements, e.g., buttons, sliders, selectors, draggable interface elements (such as graphical icons), etc., through which VNRs, such as VNR 52A, are defined, through which two or more VNs are interconnected.

GUI 244 may receive input from a user and pass the input to network controller 24, which may process a request to update GUI 244. For example, as described in more detail below, GUI 244 may update GUI 244 to include popups, sub-windows, additional boxes, inputs, dialog boxes, and other interface elements by which to prompt a user for additional configuration details regarding VNR 52A. In some examples, the user may interface with a graphical representation of the topology of the network to select VNs 50A and 50N in response to prompts presented by updates to GUI 244. In any event, GUI 244 may enable a relatively inexperienced user to graphically and intuitively configure VNR 52A without having to specify complex configuration statements that conform to formal configuration languages and/or require extensive knowledge of routing protocols and other low-level network concepts.

Each of telemetry 260, user interface 244, configuration node 230, control node 232, and server 12/compute node may be considered SDN architecture 200 nodes, as each of these nodes is an entity that implements the functionality of the configuration, control, or data plane, or the functionality of the UI and telemetry nodes. Node scaling is configured during "setup" and SDN architecture 200 supports automatic scaling of SDN architecture 200 nodes using orchestration system operators (such as Kubernetes operators).

As described above, SDN architecture 200 intent to configure may be a custom resource. One such custom resource may include a VNR 52 (shown in the example of fig. 1) by which communications are established between two or more VNs 50 in the manner described above. As described above, VNR 52 may represent a logical abstraction of policies for configuring the ingress and egress of routing information between VNR 50, whereby VNR 52 may facilitate the exchange of routing information (referred to as asymmetric or symmetric ingress and egress) using common routing targets established for each of VNR 52. A common route target may be defined and associated with a route instance for implementing VN 50.

An administrator, such as a Kubernetes administrator, may interface with user interface 244 (e.g., web UI 306) to define VNR 52, possibly via a graphical user interface with graphical elements representing pod, VN 50, etc. To define VNR 52, an administrator may associate VNR 52 with one or more labels assigned to VN 50. Using these labels, the VNR 52 can establish ingress and egress policies to and from the common routing targets created when the VNR 52 is instantiated. Web UI 306 may interface with configuration controller 230 to create a common routing target that is installed into one or more virtual routers 318 via control node 232. The Web UI 306 may also define, via the configuration controller 230 and the control nodes 232, route instances for common route targets (to distinguish the common route targets from other common route targets) that are utilized to facilitate interconnection between VNs 50, as will be described in more detail below with respect to a plurality of different networking schemes.

Fig. 4 is a block diagram illustrating exemplary components of an SDN architecture in accordance with the techniques of this disclosure. In this example, SDN architecture 400 extends and uses Kubernetes API server for network configuration objects that implement user intent for network configuration. In Kubernetes terminology, such configuration objects are referred to as custom resources and are simply referred to as objects when persisted in the SDN architecture. The configuration object is mainly a user intention (e.g., virtual network such as VN 50, VNR 52, BGPaaS, network policy, service chain, etc.).

SDN architecture 400 configuration node 230 may use a Kubernetes API server for configuring objects. In kubernetes terminology, these are referred to as custom resources.

Kubernetes provides two ways to add custom resources to a cluster:

custom Resource Definition (CRD) is simple and can be created without any programming.

API aggregation requires programming but allows more control over API behavior, such as how data is stored and transitions between API versions.

The aggregated API is a slave API server located behind a master API server acting as a proxy. This arrangement is known as API Aggregation (AA). For the user, simply show that the Kubernetes API is extended. The CRD allows the user to create a new type of resource without adding another API server. Regardless of how they are installed, the new resources are referred to as Custom Resources (CRs) to distinguish them from the native Kubernetes resources (e.g., pod). CRD was used for the initial Config prototype. The architecture may use an API server builder Alpha library to implement an aggregated API. An API server builder is a collection of libraries and tools for building native Kubernetes syndication extensions.

Typically, each resource in the Kubernetes API requires code to process REST requests and manage persistent storage of objects. The primary Kubernetes API server 300 (implemented with API server micro services 300A-300J) handles native resources and may also handle custom resources generally through CRD. Aggregated API 402 represents an extension of Kubernetes API server 300 to allow for providing an aggregation layer of a specific implementation for custom resources by writing and deploying custom API server 301 (using custom API server micro-services 301A through 301M). The main API server 300 delegates requests for custom resources to the custom API server 301, making these resources available to all its clients.

In this way, API server 300 (e.g., kube-apiserver) receives the Kubernetes configuration object, the native object (pod, service), and the custom resource. The custom resources of SDN architecture 400 may include configuration objects that implement the desired network configuration of SDN architecture 400 when the desired state of the configuration objects in SDN architecture 400 is implemented, including implementing each VNR 52 as one or more ingress policies and/or one or more egress policies and a common routing target (and routing instance). As described above, VNR 52 within SDN architecture 400 may result in an ingress and/or egress policy interconnecting two or more VNs 50 as described in more detail below.

In this regard, the custom resources may correspond to configuration modes that are traditionally defined for network configuration, but in accordance with the techniques of this disclosure, the custom resources are extended to be operable through the aggregated API 402. Such custom resources may alternatively be referred to herein as "custom resources for SDN architecture configuration. These may include VN, VNR, bgp-as-a-service (BGPaaS), a subnet, a virtual router, a service instance, an item, a physical interface, a logical interface, a node, a network ipam, a floating IP, an alarm, an alias IP, an access control list, a firewall policy, a firewall rule, a network policy, a route target, a route instance. Custom resources for SDN architecture configuration may correspond to configuration objects that are typically exposed by SDN controllers, but in accordance with the techniques described herein, the configuration objects are exposed as custom resources and are merged with Kubernetes native/built-in resources to support a unified intent model exposed by aggregated API 402, the model being implemented by Kubernetes controllers 406A-406N and by custom resource controller 302 (shown in fig. 4 with component micro-services 302A-302L) for coordinating the actual state and intent state of a computing infrastructure comprising network elements.

Given the unified nature of exposing custom resources that are consolidated with Kubernetes native/built-in resources, a Kubernetes administrator (or other Kubernetes user) may define VNRs using common Kubernetes semantics, such as VNR 52, which may then be converted into complex policies detailing the introduction and extraction of routing information to facilitate interconnection of VNs 50 without any understanding of BGP and other routing protocols typically required for interconnecting VNs 50. Thus, aspects of the present technique may promote a more uniform user experience, potentially resulting in fewer misconfigurations and trial-and-error, which may improve execution of SDN architecture 400 itself (in terms of utilizing fewer processing cycles, memory, bandwidth, etc., and associated power).

The API server 300 aggregation layer sends API custom resources to their corresponding registered custom API servers 300. There may be multiple custom API servers/custom resource controllers to support different types of custom resources. Custom API server 300 processes custom resources of the SDN architecture configuration and writes to configuration store 304, which may be etcd. Custom API server 300 may be a host and expose SDN controller identifier assignment services that may be needed by custom resource controller 302.

Custom resource controller 302 begins applying business logic to reach a user intent with a user intent configuration. The business logic is implemented as a coordination ring. Fig. 8 is a block diagram illustrating an example of a custom controller for custom resources of an SDN architecture configuration in accordance with the techniques of this disclosure. Custom controller 814 may represent an illustrative instance of custom resource controller 301. In the example shown in FIG. 8, customization controller 814 may be associated with customization resource 818. Custom resources 818 may be any custom resources configured for SDN architecture. Custom controller 814 may include a coordinator 816, which coordinator 816 includes logic for performing a coordination loop, wherein custom controller 814 observes 834 (e.g., monitors) the current state 832 of custom resource 818. In response to determining that the desired state 836 does not match the current state 832, the coordinator 816 may perform actions to adjust 838 the state of the customized resource such that the current state 832 matches the desired state 836. The API server 300 may receive the request and relay the request to the custom API server 301 to change the current state 832 of the custom resource 818 to the desired state 836.

Where API request 301 is a create request for a custom resource, coordinator 816 may act on the creation event of instance data for the custom resource. Coordinator 816 can create instance data for the requested custom resource dependent custom resource. As an example, the edge node custom resources may rely on virtual network custom resources, virtual interface custom resources, and IP address custom resources. In this example, when the coordinator 816 receives a creation event of an edge node custom resource, the coordinator 816 may also create custom resources on which the edge node custom resource depends, such as virtual network custom resources, virtual interface custom resources, and IP address custom resources.

By default, custom resource controller 302 runs an active-passive mode and uses a master selection to achieve consistency. When the controller pod begins, it attempts to create a ConfigMap resource in Kubernetes using the specified keys. If the creation is successful, the pod becomes the master and begins processing coordination requests; otherwise, it would prevent trying to create a ConfigMap in an infinite loop.

Custom resource controller 302 may track the status of custom resources it creates. For example, a Virtual Network (VN) creates a Routing Instance (RI) that creates a Route Target (RT). If creation of the route target fails, the route instance state is downgraded and thus the virtual network state is downgraded as well. Custom resource controller 302 may thus output custom messages indicating the status of these custom resources for troubleshooting. Also, the VNR creates an RI that creates an RT in a similar manner as discussed above with respect to the VN. FIG. 9 illustrates an exemplary flow of creation, monitoring, and reconciliation in custom resource types that have dependencies on different custom resource types.

The configuration plane implemented by the configuration node 230 has high availability. Configuration node 230 may be based on Kubernetes, including Kube-apiserver services (e.g., API server 300) and storage backend etcd, etc. (e.g., configuration store 304). Effectively, the aggregated API 402 implemented by the configuration node 230 operates as the front-end of the control plane implemented by the control node 232. The primary implementation of the API server 300 is a Kube-apiserver, which is designed to scale horizontally by deploying more instances. As shown, several instances of API server 300 may operate to balance API requests and processing loads.

Configuration store 304 may be implemented as etcd. Etcd is a consistent and highly available key value store that is used as a Kubernetes backing store for cluster data.

In the example of fig. 4, servers 12 of SDN architecture 400 each include orchestration agent 420 and a containerized (or "cloud-native") routing protocol daemon 324. These components of SDN architecture 400 are described in further detail below.

SDN controller manager 303 may operate as an interface between Kubernetes core resources (services, namespaces, pod, network policies, network attachment definitions) and extended SDN architecture resources (VirtualNetwork, routingInstance, etc.). SDN controller manager 303 monitors the Kubernetes API for changes on both the Kubernetes core and the custom resources for SDN architecture configuration and may therefore perform CRUD operations on the relevant resources.

In some examples, SDN controller manager 303 is a set of one or more Kubernetes custom controllers. In some examples, in single or multi-cluster deployments, SDN controller manager 303 may run on the Kubernetes cluster it manages.

SDN controller manager 303 listens for create, delete, and update events for the following Kubernetes objects:

·Pod

Service

Node multiport

Inlet port

Endpoint

Namespaces

Deployment

Network policy

When these events are generated, SDN controller manager 303 creates the appropriate SDN architecture object, which in turn is defined as a custom resource for SDN architecture configuration. In response to detecting an event on an instance of a custom resource, whether instantiated by SDN controller manager 303 or by custom API server 301, control node 232 obtains configuration data for the instance of the custom resource and configures a corresponding instance of a configuration object in SDN architecture 400.

For example, SDN controller manager 303 monitors Pod creation events and in response may create the following SDN architecture objects: virtual machine, virtual machine interface, and instant IP. Then, in this case, the control node 232 may instantiate the SDN architecture object in the selected compute node.

As an example, based on the monitoring, control node 232A may detect an event on an instance of a first customized resource exposed by customer API server 301A, where the first customized resource is used to configure some aspect of SDN architecture system 400 and corresponds to a type of configuration object of SDN architecture system 400. For example, the type of configuration object may be a firewall rule corresponding to the first customized resource. In response to this event, control node 232A may obtain configuration data for the firewall rule instance (e.g., firewall rule specification) and provide the firewall rules in the virtual router for server 12A. Configuration node 230 and control node 232 may perform similar operations on other customized resources using corresponding types of configuration objects of the SDN architecture, such as virtual networks, virtual network routers, bgp-as-a-service (BGPaaS), subnets, virtual routers, service instances, items, physical interfaces, logical interfaces, nodes, network ipam, floating ips, alerts, aliases ips, access control lists, firewall policies, firewall rules, network policies, routing targets, routing instances, and the like.

Fig. 5 is a block diagram of an exemplary computing device in accordance with the techniques described in this disclosure. Computing device 500 of fig. 2 may represent a real or virtual server, and may represent an illustrative instance of any server 12, and may be referred to as a computing node, master/work node, or host. In this example, computing device 500 includes a bus 542 that couples hardware components of the hardware environment of computing device 500. Bus 542 couples Network Interface Card (NIC) 530, storage disk 546, and one or more microprocessors 210 (hereinafter, "microprocessor 510"). NIC 530 may have SR-IOV capability. In some cases, a front side bus may couple microprocessor 510 and memory device 524. In some examples, bus 542 may couple memory device 524, microprocessor 510, and NIC 530. Bus 542 may represent a Peripheral Component Interface (PCI) express (PCIe) bus. In some examples, a Direct Memory Access (DMA) controller may control DMA transfers between components coupled to bus 542. In some examples, components coupled to bus 542 control DMA transfers between components coupled to bus 542.

Microprocessor 510 may include one or more processors, each including a separate execution unit to execute instructions conforming to an instruction set architecture, the instructions being stored on a storage medium. The execution units may be implemented as separate Integrated Circuits (ICs), or may be combined within one or more multi-core processors (or "many-core" processors), each implemented using a single IC (i.e., a chip multiprocessor).

Disk 546 represents computer-readable storage media including volatile and/or nonvolatile, removable and/or non-removable media implemented in any method or technology for storage of information such as processor-readable instructions, data structures, program modules, or other data. Computer-readable storage media includes, but is not limited to, random Access Memory (RAM), read Only Memory (ROM), EEPROM, flash memory, CD-ROM, digital Versatile Disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by microprocessor 510.

Main memory 524 includes one or more computer-readable storage media, which may include Random Access Memory (RAM), such as various forms of Dynamic RAM (DRAM), such as DDR2/DDR3 SDRAM, or Static RAM (SRAM), flash memory, or any other form of fixed or removable storage medium that may be used to carry or store desired program code and program data in the form of instructions or data structures and that may be accessed by a computer. Main memory 524 provides a physical address space comprised of addressable memory locations.

A Network Interface Card (NIC) 530 includes one or more interfaces 532 configured to exchange data packets using links of an underlying physical network. Interface 532 may include a port interface card having one or more network ports. NIC 530 may also include on-card memory to, for example, store packet data. Direct memory access transmissions between NIC 530 and other devices coupled to bus 542 may be read from and written to NIC memory.

Memory 524, NIC 530, storage disk 546, and microprocessor 510 may provide an operating environment for a software stack that includes an operating system kernel 580 executing in kernel space. Kernel 580 may represent, for example, linux, berkeley Software Distribution (BSD), another Unix variant kernel, or a Windows server operating system kernel available from microsoft corporation. In some instances, an operating system may execute a hypervisor and one or more virtual machines managed by the hypervisor. Example hypervisors include kernel-based virtual machines (KVM) for Linux kernels, xen, ESXi available from VMware, windows Hyper-V available from Microsoft, and other open source and proprietary hypervisors. The term hypervisor may include a Virtual Machine Manager (VMM). An operating system including kernel 580 provides an execution environment for one or more processes in user space 545.

Kernel 580 includes physical drivers 525 to use network interface card 530. The network interface card 530 may also implement an SR-IOV to enable sharing of physical network functions (I/O) among one or more virtual execution elements, such as container 529A or one or more virtual machines (not shown in fig. 5). A shared virtual device, such as a virtual function, may provide dedicated resources such that each virtual execution element may access the dedicated resources of NIC 530, and thus appear to each virtual execution element as a dedicated NIC. Virtual functions may represent lightweight PCIe functions that share physical resources with physical functions used by physical drivers 525 and with other virtual functions. For an SR-IOV enabled NIC 530, NIC 530 may have thousands of virtual functions available according to the SR-IOV standard, but for I/O intensive applications, the number of virtual functions configured is typically much smaller.

The computing device 500 may be coupled to a physical network switching fabric that includes an overlay network of software or "virtual" routers, including virtual routers 506, that extend the switching fabric from a physical switch to physical servers coupled to the switching fabric. The virtual router may be a process or thread or component thereof executed by a physical server (e.g., server 12 of fig. 1) that dynamically creates and manages one or more virtual networks that are available for communication between virtual network endpoints. In one example, a virtual router implements each virtual network using an overlay network that provides the ability to decouple the virtual address of an endpoint from the physical address (e.g., IP address) of the server on which the endpoint is executing.

Each virtual network may use its own addressing and security scheme and may be considered orthogonal to the physical network and its addressing scheme. Various techniques may be used to transport data packets within and across virtual networks through a physical network. The term "virtual router" as used herein may include OpenvSwitch (OVS), OVS, linux, docker bridges, or other devices and/or software located on a host device and performing switching, bridging, or routing of data packets between virtual network endpoints of one or more virtual networks, where the virtual network endpoints are hosted by one or more servers 12. In the exemplary computing device 500 of fig. 5, the virtual router 506 is executed within the user space as a DPDK-based virtual router, but in various implementations the virtual router 506 may be executed within a hypervisor, host operating system, host application, or virtual machine.

Virtual router 506 may replace and contain virtual routing/bridging functions of the Linux bridge/OVS module typically used for Kubernetes deployment of pod 502. Virtual router 506 may perform bridging (e.g., E-VPN) and routing (e.g., L3VPN, IP-VPN) for virtual networks. Virtual router 506 may perform networking services such as application security policies, NAT, multicasting, mirroring, and load balancing.

The virtual router 506 may be implemented as a kernel module or as a user space DPDK procedure (the virtual router 506 is shown herein in user space 545). The virtual router agent 514 may also execute in user space. In the exemplary computing device 500, the virtual router 506 executes within the user space as a DPDK-based virtual router, but in various implementations the virtual router 506 may execute within a hypervisor, host operating system, host application, or virtual machine. The virtual router agent 514 has a connection to the network controller 24 using a channel for downloading configuration and forwarding information. The virtual router agent 514 programs this forwarding state to the virtual router data (or "forwarding") plane represented by the virtual router 506. Virtual router 506 and virtual router agent 514 may be processes. Virtual router 506 and virtual router agent 514 are containerized/cloud-native.

Virtual router 506 may replace and contain virtual routing/bridging functions of the Linux bridge/OVS module typically used for Kubernetes deployment of pod 502. Virtual router 506 may perform bridging (e.g., E-VPN) and routing (e.g., L3VPN, IP-VPN) for virtual networks. Virtual router 506 may perform networking services such as application security policies, NAT, multicasting, mirroring, and load balancing.

Virtual router 506 may be multi-threaded and execute on one or more processor cores. Virtual router 506 may include multiple queues. Virtual router 506 may implement a packet processing pipeline. The virtual router agent 514 can stitch the pipeline in a manner from the simplest to the most complex, depending on the operation to be applied to the data packet. Virtual router 506 may maintain multiple forwarding base instances. Virtual router 506 may use a Read Copy Update (RCU) lock to access and Update the table.

To send data packets to other computing nodes or switches, virtual router 506 uses one or more physical interfaces 532. Typically, virtual router 506 exchanges overlay packets with a workload such as VM or pod 502. Virtual router 506 has a plurality of virtual network interfaces (e.g., vifs). These interfaces may include a kernel interface vhost0 for exchanging data packets with the host operating system; the interface with the virtual router agent 514, pkt0, is used to obtain forwarding state from the network controller and send the upstream exception packet. There may be one or more virtual network interfaces corresponding to one or more physical network interfaces 532. Other virtual network interfaces of virtual router 506 are used to exchange data packets with the workload.

In a kernel-based deployment of virtual router 506 (not shown), virtual router 506 is installed as a kernel module inside the operating system. Virtual router 506 registers itself with the TCP/IP stack to receive packets from any desired operating system interface it wants. The interfaces may be binding, physical, tap (for VM), veth (for container), etc. In this mode, virtual router 506 relies on the operating system to send and receive data packets from different interfaces. For example, the operating system may expose a tap interface supported by the vhost-net driver to communicate with the VM. Once virtual router 506 registers a packet from the tap interface, the TCP/IP stack sends all packets to it. Virtual router 506 sends the data packets via the operating system interface. In addition, the NIC queues (physical or virtual) are handled by the operating system. Packet processing may operate in an interrupt mode, which generates interrupts and may cause frequent context switches. When high packet rates exist, the overhead associated with frequent interrupts and context switches can overwhelm the operating system and result in poor performance.

In a DPDK based deployment of virtual router 506 (shown in fig. 5), virtual router 506 is installed as a user space 545 application linked to a DPDK library. This may result in faster performance than core-based deployments, especially in the presence of high packet rates. The physical interface 532 is used by the Polling Mode Driver (PMD) of the DPDK instead of the interrupt-based driver of the kernel. Registers of physical interface 532 can be exposed to user space 545 so that the PMD can access; the physical interfaces 532 bound in this manner are no longer managed by or visible to the host operating system, and the DPDK-based virtual router 506 manages the physical interfaces 532. This includes packet polling, packet processing, and packet forwarding. In other words, the user data packet processing step is performed by the virtual router 506DPDK data plane. The nature of this "polling mode" makes virtual router 506DPDK data plane packet processing/forwarding more efficient than interrupt mode when the packet rate is high. There are relatively few interrupts and context switches during packet I/O as compared to the kernel-mode virtual router 506, and in some cases interrupts and context switches during packet I/O may be completely avoided.

In general, each pod 502A-502B can be assigned one or more virtual network addresses for use within a corresponding virtual network, where each virtual network can be associated with a different virtual subnet provided by virtual router 506. pod 502B may be assigned its own virtual layer three (L3) IP address, e.g., for sending and receiving communications, but may not know the IP address of the computing device 500 on which pod 502B is executing. The virtual network address may thus be different from the logical address of the underlying physical computer system (e.g., computing device 500).

The computing device 500 includes a virtual router agent 514 that controls the coverage of the virtual network of the computing device 500 and coordinates the routing of data packets within the computing device 500. Generally, the virtual router agent 514 communicates with the network controller 24 of the virtualization infrastructure, which generates commands to create a virtual network and configure network virtualization endpoints, such as the computing device 500, and more particularly, the virtual router 506, and the virtual network interface 212. By configuring virtual router 506 based on information received from network controller 24, virtual router agent 514 may support configuration network isolation, policy-based security, gateways, source Network Address Translation (SNAT), load balancer, and service chaining capabilities for orchestration.

In one example, network packets generated or consumed by containers 529A-529B within a virtual network domain, such as layer three (L3) IP packets or layer two (L2) ethernet packets, may be encapsulated in another packet (e.g., another IP or ethernet packet) transmitted by a physical network. Packets transmitted in a virtual network may be referred to herein as "inner packets" and physical network packets may be referred to herein as "outer packets" or "tunnel packets. Encapsulation and/or decapsulation of virtual network packets within physical network packets may be performed by virtual router 506. This functionality is referred to herein as tunneling and may be used to create one or more overlay networks. In addition to ipineip, other example tunneling protocols that may be used include multiprotocol label switching (MPLS) over IP, vxLAN, GRE over Generic Routing Encapsulation (GRE), MPLS protocol (UDP) over user datagram, and the like. Virtual router 506 performs tunnel encapsulation/decapsulation of data packets originating from/destined to any container of pod 502, and virtual router 506 exchanges data packets with pod 502 via bus 542 and/or the bridge of NIC 530.

As described above, the network controller 24 may provide a logically centralized controller to facilitate the operation of one or more virtual networks. The network controller 24 may, for example, maintain a routing information base, such as one or more routing tables that store routing information for the physical network and one or more overlay networks. Virtual router 506 implements one or more virtual routing and forwarding instances (VRFs), such as VRF 222A, for the respective virtual networks that virtual router 506 operates as a respective tunnel endpoint. Typically, each VRF stores forwarding information for the corresponding virtual network and identifies where the packet is to be forwarded and whether the packet is to be encapsulated in a tunneling protocol, such as with a tunneling header that may include one or more headers for different layers of the virtual network protocol stack. Each VRF may include a network forwarding table that stores routing and forwarding information for the virtual network.

NIC 530 may receive the tunnel packet. Virtual router 506 processes the tunnel packets to determine the virtual network of source and destination endpoints of the inner packets from the tunnel encapsulation header. Virtual router 506 may peel the layer 2 header and tunnel encapsulation header to internally forward only the inner data packet. The tunnel encapsulation header may include a virtual network identifier, such as a VxLAN label or MPLS label, that indicates the virtual network, e.g., the virtual network corresponding to VRF 222A. VRF 222A may include forwarding information for the inner packet. For example, VRF 222A may map the destination layer 3 address of the inner packet to virtual network interface 212. In response, VRF 222A forwards the inner packet to pod 502A via virtual network interface 212.

Container 529A may also have the inner packet as a source virtual network endpoint. Container 529A may, for example, generate an intra-layer 3 packet destined for a destination virtual network endpoint executed by another computing device (i.e., non-computing device 500) or another of the containers. Container 529A may send the layer 3 intra-layer data packets to virtual router 506 via a virtual network interface attached to VRF 222A.

Virtual router 506 receives the inner data packet and the layer 2 header and determines the virtual network for the inner data packet. Virtual router 506 may determine the virtual network using any of the virtual network interface implementation techniques described above (e.g., macvlan, veth, etc.). Virtual router 506 uses VRF 222A corresponding to the virtual network for the inner data packet to generate an outer header for the inner data packet that includes an outer IP header for the overlay tunnel and a tunnel encapsulation header identifying the virtual network. Virtual router 506 encapsulates the inner packet with an outer header. Virtual router 506 may encapsulate a tunnel packet with a new layer 2 header having a destination layer 2 address associated with a device external to computing device 500 (e.g., one of TOR switch 16 or server 12). If external to computing device 500, virtual router 506 outputs tunnel packets with new layer 2 headers to NIC 530 using physical function 221. NIC 530 outputs the packet on the outbound interface. If the destination is another virtual network endpoint executing on the computing device 500, the virtual router 506 routes the data packet to the appropriate one of the virtual network interfaces 212, 213.

In some examples, a controller of computing device 500 (e.g., network controller 24 of fig. 1) configures a default route in each pod 502 to cause virtual machine 224 to use virtual router 506 as an initial next hop for outbound packets. In some examples, NIC 530 is configured with one or more forwarding rules to cause all data packets received from virtual machine 224 to be exchanged to virtual router 506.

The Pod 502A includes one or more application containers 529A. Pod 502B includes an instance of a containerized routing protocol daemon (containerized routing protocol daemon, cRPD) 560. Container platform 588 includes container runtime 590, orchestration agent 592, service agent 593, and CNI 570.

The container engine 590 includes code executable by the microprocessor 510. The container runtime 590 can be one or more computer processes. The container engine 590 runs containerized applications in the form of containers 529A-529B. The container engine 590 may represent a dock, rk, or other container engine for managing containers. In general, the container engine 590 receives requests and manages objects such as images, containers, networks, and volumes. The image is a template with instructions for creating a container. A container is an executable instance of an image. Based on the instructions from the controller agent 592, the container engine 590 can obtain images and instantiate them as executable containers in the pod 502A-502B.

The service agent 593 includes code executable by the microprocessor 510. The service agent 593 may be one or more computer processes. The service agent 593 monitors the addition and removal of services and endpoint objects and it maintains the network configuration of the computing device 500 to ensure communication between the pod and container, such as using the services. The service agent 593 can also manage the IP table to capture traffic to the virtual IP address and port of the service and redirect the traffic to the agent port of the agent backup pod. The service proxy 593 may represent a kube-proxy for the working nodes of the Kubernetes cluster. In some examples, container platform 588 does not include service agent 593, or service agent 593 is disabled to facilitate configuration of virtual router 506 and pod 502 by CNI 570.

Orchestration agent 592 comprises code executable by microprocessor 510. Orchestration agent 592 can be one or more computer processes. Orchestration agent 592 may represent kubrelet for the working nodes of Kubernetes cluster. Orchestration agent 592 is an agent of an orchestrator (e.g., orchestrator 23 of fig. 1) that receives container specification data for containers and ensures that computing device 500 executes the containers. The container specification data may be in the form of manifest files sent from orchestrator 23 to orchestration agent 592, or indirectly received via a command line interface, HTTP endpoint, or HTTP server. The container specification data can be a container specification of one of the pod 502 of the container (e.g., podSpec-yet another markup language (Yet Another Markup Language, YAML) or JSON object that describes the pod). Based on the container specification data, orchestration agent 592 directs container engine 590 to obtain and instantiate a container image of container 529 to execute container 529 by computing device 500.

Orchestration agent 592 instantiates or otherwise invokes CNI 570 to configure one or more virtual network interfaces for each pod 502. For example, orchestration agent 592 receives container specification data for pod 502A and directs container engine 590 to create pod 502A with container 529A based on the container specification data for pod 502A. Orchestration agent 592 also invokes CNI 570 to configure virtual network interfaces for pod 502A corresponding to the virtual networks of VRF 222A. In this example, pod 502A is a virtual network endpoint of a virtual network corresponding to VRF 222A.

CNI 570 may obtain interface configuration data for configuring virtual network interfaces of pod 502. The virtual router agent 514 operates as a virtual network control plane module for enabling the network controller 24 to configure the virtual router 506. Unlike orchestration control planes (including container platform 588 for the working nodes and master nodes, such as orchestrator 23) that manage the provision, scheduling, and management of virtual execution elements, virtual network control planes (including network controller 24 and virtual router agent 514 for the working nodes) manage the configuration of the virtual network implemented in the data plane in part by virtual router 506 of the working nodes. Virtual router agent 514 communicates interface configuration data for the virtual network interface to CNI 570 to enable orchestration control plane element (i.e., CNI 570) to configure the virtual network interface according to the configuration state determined by network controller 24, bridging the gap between the orchestration control plane and the virtual network control plane. In addition, this may enable CNI 570 to obtain interface configuration data for multiple virtual network interfaces of the pod and configure the multiple virtual network interfaces, which may reduce communication and resource overhead inherent in invoking a separate CNI 570 to configure each virtual network interface.

The containerized routing protocol daemon is described in U.S. application Ser. No. 17/649,632, filed on 1/2/2022, which is incorporated herein by reference in its entirety.

Furthermore, CNI 570 (possibly in conjunction with virtual router agent 514) may configure virtual router 506 to implement VNR 52. The VNR 52 may cause the routing plane to be configured via one or more ingress policies and/or one or more egress policies for exchanging routing information with a common routing target of the VNR 52. These policies result in routing information maintained for VNR 50 being exchanged with (or otherwise leaked between) VNR 52 public routing targets, which in turn are parsed into forwarding information. CNI 570 may obtain configuration data from network controller 24 for installing the forwarding information that interfaces with virtual router agent 514 to install forwarding information for forwarding packets from VN 50.

In other words, creating the common routing target may enable routing information to be imported and exported from one of VNs 50 (e.g., VN 50A) to the common routing target provided by one or more of VNRs 52, and from the common routing target to another of VNs 50 (e.g., VN 50N). The network controller 24 may parse the routing information into the forwarding information described above and install it within the virtual router 506 to enable packet forwarding between the VNs 50A and 50N (in some configurations, such as the mesh configuration described above). In this way, the virtual router 506 may establish a way for routing information in and out between the VNs 50, which may then be used by the VNs 50 to transfer data packets between each of the other VNs in the VN 50.

Fig. 6 is a block diagram of an exemplary computing device operating as a computing node of one or more clusters of an SDN architecture system in accordance with the techniques of this disclosure. Computing device 1300 may represent one or more real or virtual servers. In some examples, computing device 1300 may implement one or more master nodes for a respective cluster or for multiple clusters.

Scheduler 1322, API server 300A, controller 406A, custom API server 301A, custom resource controller 302A, controller manager 1326, SDN controller manager 1325, control node 232A, and configuration store 1328, while illustrated and described as being executed by a single computing device 1300, may be distributed among multiple computing devices that make up a computing system or hardware/server cluster. In other words, each of the plurality of computing devices may provide a hardware operating environment for one or more instances of any one or more of scheduler 1322, API server 300A, controller 406A, custom API server 301A, custom resource controller 302A, network controller manager 1326, network controller 1324, SDN controller manager 1325, control node 232A, or configuration store 1328.

In this example, computing device 1300 includes a bus 1342 that couples the hardware components of the computing device 1300 hardware environment. The bus 1342 couples a Network Interface Card (NIC) 1330, a memory disk 1346, and one or more microprocessors 1310 (hereinafter, "microprocessors 1310"). In some cases, a front side bus may couple microprocessor 1310 and memory device 1344. In some examples, bus 1342 may couple memory device 1344, microprocessor 1310, and NIC 1330. Bus 1342 may represent a Peripheral Component Interface (PCI) express (PCIe) bus. In some examples, a Direct Memory Access (DMA) controller may control DMA transfers between components coupled to bus 242. In some examples, components coupled to bus 1342 control DMA transfers between components coupled to bus 1342.

The microprocessor 1310 may include one or more processors, each including a separate execution unit to execute instructions conforming to an instruction set architecture, the instructions stored to a storage medium. The execution units may be implemented as separate Integrated Circuits (ICs), or may be combined within one or more multi-core processors (or "many-core" processors), each implemented using a single IC (i.e., a chip multiprocessor).

The disk 1346 represents computer-readable storage media including volatile and/or nonvolatile, removable and/or non-removable media implemented in any method or technology for storage of information such as processor-readable instructions, data structures, program modules, or other data. Computer-readable storage media includes, but is not limited to, random Access Memory (RAM), read Only Memory (ROM), EEPROM, flash memory, CD-ROM, digital Versatile Disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by microprocessor 1310.

Main memory 1344 includes one or more computer-readable storage media that can include Random Access Memory (RAM), such as various forms of Dynamic RAM (DRAM), such as DDR2/DDR3 SDRAM, or Static RAM (SRAM), flash memory, or any other form of fixed or removable storage medium that can be used to carry or store desired program code and program data in the form of instructions or data structures and that can be accessed by a computer. Main memory 1344 provides a physical address space comprised of addressable memory locations.

A Network Interface Card (NIC) 1330 includes one or more interfaces 3132 configured to exchange data packets using links of an underlying physical network. Interface 3132 may include a port interface card having one or more network ports. NIC 1330 may also include on-card memory to, for example, store packet data. Direct memory access transmissions between NIC 1330 and other devices coupled to bus 1342 may be read from and written to NIC memory.

The memory 1344, NIC 1330, storage 1346, and microprocessor 1310 may provide an operating environment for a software stack that includes an operating system kernel 1314 executing in kernel space. Kernel 1314 may represent, for example, linux, berkeley Software Distribution (BSD), another Unix variant kernel, or a Windows server operating system kernel available from microsoft corporation. In some instances, an operating system may execute a hypervisor and one or more virtual machines managed by the hypervisor. Example hypervisors include kernel-based virtual machines (KVM) for Linux kernels, xen, ESXi available from VMware, windows Hyper-V available from Microsoft, and other open source and proprietary hypervisors. The term hypervisor may include a Virtual Machine Manager (VMM). An operating system including kernel 1314 provides an execution environment for one or more processes in user space 1345. The kernel 1314 includes a physical driver 1327 to use the network interface card 230.

Computing device 1300 may be coupled to a physical network switch fabric that includes an overlay network of software or virtual routers (such as virtual router 21) that extend the switch fabric from a physical switch to a physical server coupled to the switch fabric. Computing device 1300 can configure the working nodes of the cluster using one or more private virtual networks.

The API server 300A, scheduler 1322, controller 406A, custom API server 301A, custom resource controller 302A, controller manager 1326, and configuration store 1328 may implement the master node of the cluster and may alternatively be referred to as a "master component". The cluster may be a Kubernetes cluster and the master node may be a Kubernetes master node, in which case the master component is a Kubernetes master component.

Each of API server 300A, controller 406A, custom API server 301A, and custom resource controller 302A includes code executable by microprocessor 1310. Custom API server 301A validates and configures data for custom resources of SDN architecture configurations (such as VN 50 and VNR 52). A service may be an abstraction that defines a logical set of pods and policies for accessing the pods. A set of pod implementing the service is selected based on the service definition. The service may be implemented in part as or otherwise include a load balancer. API server 300A and custom API server 301A may implement a representational state transfer (Representational State Transfer, REST) interface to handle REST operations and provide the front end to the shared state of the corresponding cluster stored to configuration store 1328 as part of the configuration plane of the SDN architecture. API server 300A may represent a Kubernetes API server.

Configuration store 1328 is a backing store for all cluster data. The cluster data may include cluster state and configuration data. The configuration data may also provide a back-end for service discovery and/or provide a locking service. Configuration store 1328 may be implemented as a key-value store. Configuration store 1328 may be a central database or a distributed database. Configuration store 1328 may represent etcd storage. Configuration store 1328 may represent a Kubernetes configuration store.

Scheduler 1322 includes code that may be executed by microprocessor 1310. Scheduler 1322 may be one or more computer processes. Scheduler 1322 monitors newly created or requested virtual execution elements (e.g., pod of container) and selects the working node on which the virtual execution element will run. The scheduler 1322 may select a working node based on resource requirements, hardware constraints, software constraints, policy constraints, location, and the like. Scheduler 1322 may represent a Kubernetes scheduler.

In general, the API server 1320 can call the scheduler 1322 to schedule the pod. The scheduler 1322 can select a worker node and return an identifier of the selected worker node to the API server 1320, which can write the identifier to the configuration store 1328 in association with the pod. API server 1320 may invoke orchestration agent 310 for the selected worker node, which may cause container engine 208 for the selected worker node to obtain the pod from the storage server and create a virtual execution unit on the worker node. Orchestration agent 310 for the selected working node may update the state of the pod to API server 1320, which persists this new state to configuration store 1328. Thus, computing device 1300 illustrates a new pod in computing infrastructure 8.

The controller manager 1326 includes code executable by the microprocessor 1310. The controller manager 1326 may be one or more computer processes. The controller manager 1326 may embed a core control loop to monitor the shared state of the cluster by obtaining notifications from the API server 1320. The controller manager 1326 may attempt to move the state of the cluster to a desired state. The example controller 406A and the custom resource controller 302A may be managed by a controller manager 1326. Other controllers may include a copy controller, an endpoint controller, a namespace controller, and a service account controller. The controller manager 1326 can perform lifecycle functions such as namespace creation and lifecycle, event garbage collection, terminated pod garbage collection, cascade delete garbage collection, node garbage collection, and the like. The controller manager 1326 may represent a Kubernetes controller manager for a Kubernetes cluster.

The network controller of the SDN architecture described herein may provide cloud networking for computing architectures running on a network infrastructure. Cloud networking may include private clouds for enterprises or service providers, infrastructure as a service (IaaS), and Virtual Private Clouds (VPC) for Cloud Service Providers (CSP). Private cloud, VPC, and IaaS use cases may involve a multi-tenant virtualized data center, such as described with respect to fig. 1. In such cases, multiple tenants in the data center share the same physical resources (physical servers, physical storage, physical networks). Each tenant is assigned its own logical resources (virtual machines, containers, or other forms of virtual execution elements; virtual storage; virtual networks). These logical resources are isolated from each other unless specifically allowed by the security policy. Virtual networks in the data center may also be interconnected to physical IP VPNs or L2 VPNs.

The network controller (or "network controller") may provide Network Function Virtualization (NFV) to networks such as business edge networks, broadband subscriber management edge networks, and mobile edge networks. NFV involves orchestrating and managing networking functions, such as firewalls, intrusion detection or prevention systems (IDS/IPS), deep Packet Inspection (DPI), caches, wide Area Network (WAN) optimizations, etc., in virtual machines, containers, or other virtual execution elements, rather than on physical hardware devices.

SDN controller manager 1325 includes code executable by microprocessor 1310. SDN controller manager 1325 may be one or more computer processes. SDN controller manager 1325 serves as an interface between orchestration-oriented elements (e.g., scheduler 1322, API server 300A and custom API server 301A, controller manager 1326, and configuration store 1328). Typically, SDN controller manager 1325 monitors the cluster for new Kubernetes native objects (e.g., pod and services). SDN controller manager 1325 may isolate the pod in the virtual network and connect the pod with the service and interconnect the virtual networks using so-called virtual network routers (which should not be confused with virtual routers that implement virtual network routers in the form of various ingress and egress policies to the common routing targets described above to facilitate interconnections between virtual networks).

SDN controller manager 1325 may be executed as a container for the master node of the cluster. In some cases, using SDN controller manager 1325 enables disabling service agents (e.g., kubernetes kube agents) of the working nodes such that all pod connectivity is achieved using virtual routers, as described herein.

The components of the network controller 24 may operate as CNIs of Kubernetes and may support a variety of deployment modes. CNI 17, CNI750 are compute node interfaces of the overall CNI architecture for managing networking of Kubernetes. Deployment modes can be divided into two categories: (1) An SDN architecture cluster that is a CNI integrated into the workload Kubernetes cluster, and (2) an SDN architecture cluster that is a CNI separate from the workload Kubernetes cluster.

Integration with workload Kubernetes clusters

The components of the network controller 24 (e.g., custom API server 301, custom resource controller 302, SDN controller manager 1325, and control node 232) operate in a managed Kubernetes cluster on a master node that is close to the Kubernetes controller components. In this mode, the components of the network controller 24 are actually part of the same Kubernetes cluster as the workload.

Separate from workload Kubernetes clusters

The components of the network controller 24 will be executed by Kubernetes clusters separate from the workload Kubernetes clusters.

SDN controller manager 1325 may use the controller framework for orchestration platform to snoop (or otherwise monitor) changes in objects defined in Kubernetes native API and add annotations to some of these objects. The annotation may be a tag or other identifier (e.g., "virtual network green") that specifies an attribute of the object. SDN controller manager 1325 is a component of the SDN architecture that listens for Kubernetes core resources (such as Pod, networkPolicy, service) events and converts these events to custom resources for SDN architecture configuration as needed. CNI plug-ins (e.g., CNI 17, 570) are SDN architecture components that support the Kubernetes networking plug-in standard: a container network interface.

SDN controller manager 1325 may create a network solution for an application using REST interfaces exposed by aggregated API 402 to define network objects such as virtual networks, virtual network interfaces, and access control policies. The network controller 24 component may implement a network solution in a computing infrastructure by, for example, configuring one or more virtual networks and virtual network interfaces in a virtual router. (this is just one example of an SDN configuration.)

The following example deployment configuration for the application includes a pod and virtual network information for the pod:

this metadata information can be copied to each pod copy created by the controller manager 1326. When SDN controller manager 1325 is notified of these pods, SDN controller manager 1325 may create virtual networks (e.g., pod 202A) as listed in the notes ("red-network", "blue-network", and "default/extns-network" in the example above) and create per-pod copies (e.g., pod 202A) of unique private virtual network addresses with cluster-wide address blocks (e.g., 10.0/16) from the virtual networks for each virtual network.

Contrail is an exemplary network controller architecture. Contrail CNI can be a CNI developed for Contrail. The cloud native Contrail controller may be an example of a network controller (such as network controller 24) described in this disclosure.

Fig. 7A is a block diagram illustrating a control/routing plane for an underlying network and overlay network configuration using an SDN architecture in accordance with the techniques of this disclosure. Fig. 7B is a block diagram illustrating a configured virtual network using a configured tunnel in an underlying network to connect pods in accordance with the techniques of this disclosure.

The network controller 24 for the SDN architecture may use a distributed or centralized routing plane architecture. The SDN architecture may use a containerized routing protocol daemon (process).

From the perspective of network signaling, the routing plane may operate according to a distributed model, where cRPD runs on each compute node in the cluster. This essentially means that intelligence is built into the compute nodes and involves complex configurations at each node. The Route Reflectors (RR) in this model may not make intelligent routing decisions, but rather act as relays to reflect the routing between nodes. A distributed container routing protocol daemon (cRPD) is a routing protocol process that may be used in which each computing node runs its own routing daemon instance. Meanwhile, the centralized cRPD master instance may act as an RR to relay routing information between computing nodes. The routing and configuration intelligence is distributed over nodes with RRs in the central location.

Alternatively, the routing plane may operate according to a more centralized model, where components of the network controller operate centrally and absorb the intelligence required to process configuration information, construct the network topology, and program the forwarding plane into the virtual router. The virtual router agent is a home agent that processes information programmed by the network controller. Such designs facilitate more limited intelligence required at the compute nodes and tend to make for simpler configuration states.

The centralized control plane provides the following:

allowing the proxy routing framework to be simpler and lighter. The complexity and limitations of BGP are hidden from agents. The proxy need not understand concepts such as route-differentiator, route-target, etc. The agents exchange only prefixes and establish their forwarding information accordingly.

Not just the route, which the control node can do. They build on the virtual network concept and can use route replication and re-origination (e.g., to support features such as inter-service chain and VN routing, as well as other use cases) to generate new routes.

Build BUM tree for best broadcast and multicast forwarding.

Note that the control plane has a distributed nature for certain aspects. As a control plane supporting distributed functionality, it allows each local virtual router agent to publish its local routes and subscribe to configurations based on what needs to be known.

It is then interesting to consider the control plane design from the tool POV and to use the hand tool appropriately where they are most suitable. Consider a set of advantages and disadvantages of condil-bgp and cRPD.

The following functions may be provided by the cRPD or the control node of the network controller 24.

Routing daemon/process

Both the control node and cRPD may act as a routing daemon implementing different protocols and having the ability to program routing information in the forwarding plane.

cRPD implements a routing protocol with a rich routing stack including an Interior Gateway Protocol (IGP) (e.g., intermediate system-to-intermediate system (IS-IS)), BGP-LU, BGP-CT, SR-MPLS/SRv, bidirectional Forwarding Detection (BFD), path Computation Element Protocol (PCEP), etc. It can also be deployed to provide control plane only services such as route reflectors and is popular in internet routing use cases due to these capabilities.

Control node 232 also implements the routing protocol, but is primarily BGP-based. The control node 232 understands the overlay networking. The control node 232 provides a rich feature set in the overlay virtualization and satisfies SDN use cases. Overlay features such as virtualization (using an abstraction of a virtual network) and service chaining are very popular in telco and cloud providers. cRPD may not include support for such overlay functions in some cases. However, the rich feature set of CRPD provides strong support for the underlying network.

Network orchestration/automation

The routing function is only a part of the control node 232. An integral part of the overlay networking is orchestration. In addition to providing overlay routing, control node 232 also helps model orchestration functions and provides network automation. Central to the orchestration capability of control node 232 is the ability to model network virtualization using virtual network (and related object) based abstractions, including the VNR described above. Control node 232 interfaces with configuration node 230 to relay configuration information to the control plane and the data plane. Control node 232 also helps to build up a multicast layer 2 and layer 3 overlay tree. For example, the control node may establish a virtual topology of the cluster it uses to achieve this. cRPD typically does not include such orchestration capability.

High availability and horizontal scalability

The control node design is more centralized and cRPD is more decentralized. There is a cRPD worker node running on each compute node. On the other hand, the control node 232 does not run on a computer, and may even run on a remote cluster (i.e., separate from and in some cases geographically remote from the workload cluster). The control node 232 also provides horizontal scalability to the HA and operates in an active-active mode. The computational load is shared between the control nodes 232. cRPD, on the other hand, typically does not provide horizontal scalability. Both control node 232 and cRPD may provide a smooth restart to the HA and may allow data plane operation in headless mode-where the virtual router may operate even if the control plane is restarted.

The control plane should not be just a routing daemon. It should support overlay routing and network orchestration/automation, while cRPD works well as a routing protocol that manages the underlying routes. However, cRPD generally lacks network orchestration capability and does not provide strong support for overlay routing.

Thus, in some examples, the SDN architecture may have cRPD on a compute node, as shown in fig. 7A-7B. Fig. 7A illustrates an SDN architecture 700, which may represent an example embodiment SDN architecture 200 or 400. In SDN architecture 700, cRPD 324 runs on a compute node and provides the underlying routing to the forwarding plane while running a centralized (and horizontally scalable) set of control nodes 232 that provide orchestration and coverage services. In some examples, a default gateway may be used instead of running cRPD 324 on a computing node.

cRPD 324 on a compute node provides a rich underlying route to the forwarding plane by interacting with virtual router agent 514 using interface 540 (which may be a gRPC interface). The virtual router proxy interface may allow programming of routes, configuring virtual network interfaces for overlays, and otherwise configuring virtual router 506. This is described in more detail in U.S. application Ser. No. 17/649,632. At the same time, one or more control nodes 232 operate as separate pod providing overlay services. SDN architecture 700 may thus obtain the rich coverage and orchestration provided by control node 232 and modern underlying routing by cRPD 324 on the compute nodes to supplement control node 232. A separate cRPD controller 720 may be used to configure cRPD 324.cRPD controller 720 may be a device/element management system, network management system, orchestrator, user interface/CLI, or other controller. The cRPD 324 runs routing protocols and exchanges routing protocol messages with routers that include other cRPD 324. Each cRPD 324 may be a containerized routing protocol process and effectively operates as a software-only version of the router control plane.

The enhanced underlying route provided by cRPD 324 may replace the default gateway at the forwarding plane and provide a rich routing stack for supportable use cases. In some examples where cRPD 324 is not used, virtual router 506 will rely on a default gateway for the underlying route. In some examples, cRPD 324 as an underlying routing procedure will be limited to programming only the default inet (6) 0 structure with control plane routing information. In such examples, the non-default overlay VRF may be programmed by the control node 232.

In this case, control node 232 may obtain custom resources defining VNR 52 and instantiate individual routing instances (with corresponding common routing targets). The control node 232 may create ingress and/or egress policies for the common route target that cause ingress and egress of routes from the various virtual networks (e.g., VNs 50A and 50N) to the common route target. The control node 232 may resolve the common routing target to obtain forwarding information, which may then be pushed down to the virtual router 506 via the virtual routing agent 514. Using this forwarding information, virtual router 506 may forward data packets between VNs 50A and 50N (in some interconnectivity schemes, such as the mesh interconnectivity scheme mentioned above).

Fig. 7A to 7B illustrate the dual routing/control plane solution described above. In fig. 7A, cRPD 324 provides the underlying routing/forwarding information to virtual router agent 514, similar in some respects to how the router control plane programs the router forwarding/data plane.

As shown in fig. 7B, cRPD 324 exchanges routing information that can be used to create tunnels for VRFs through the underlying network 702. Tunnel 710 is an example and connects virtual router 506 of server 12A and virtual router 506 of server 12X. Tunnel 710 may represent a Segment Routing (SR) or SRv tunnel, a generic routing encapsulation (Generic Route Encapsulation, GRE) tunnel, an IP-in-IP tunnel, an LSP, or other tunnel. Control node 232 utilizes tunnel 710 to create virtual network 712 connecting pod 22 of server 12A and pod 22 of server 12X, wherein server 12A and server 12X are attached to the VRFs of the virtual network.

As described above, cRPD 324 and virtual router agent 514 may exchange routing information using a gRPC interface, and virtual router agent 5145 may program virtual router 506 with a configuration using a gRPC interface. It is also noted that control node 232 may be used for overriding and orchestration, while cRPD 324 may be used to manage the underlying routing protocols. The virtual router agent 514 may use the gRPC interface with the cRPD 324 while using XMPP to communicate with control nodes and Domain Name Service (DNS).

The gRPC model works well for cRPD 324 because there may be a worker running on each compute node, and virtual router agent 314 acts as a gRPC server exposing services to clients (cRPD 324) for programming routing and configuration information (for the underlying layer). Therefore, gRPC is attractive as a solution when compared to XMPP. In particular, it transmits data as a binary stream and there is no overhead added in the encoded/decoded data to be transmitted through it.

In some examples, control node 232 may interface with virtual router agent 514 using XMPP. Where virtual router agent 514 acts as a gRPC server, cRPD 324 acts as a gRPC client. This means that the client (cRPD) needs to initiate a connection to the server (vruter proxy). In SDN architecture 700, virtual router agent 514 selects a set of subscribed control nodes 232 (because there are multiple control nodes). In this regard, the control node 232 acts as a server and the virtual router proxy 514 connects and subscribes to updates as a client.

Regarding the gRPC, the control node 232 would need to pick the virtual router agent 514 to which it needs to connect and then subscribe as a client. Since the control node 232 does not run on each computing node, this would require an implementation algorithm to select the virtual router agent 514 to which it can subscribe. Furthermore, the control nodes 232 need to synchronize this information with each other. This also complicates the situation when a restart occurs and requires synchronization between the control nodes 232 to pick the agents they serve. Features such as Graceful Restart (GR) and Fast Convergence (Fast Convergence) have been implemented on top of XMPP. XMPP is already lightweight and efficient. Thus, XMPP may be preferred over gRPC for communication of the control node 232 to the virtual router proxy 514.

Additional enhancements to control node 232 and their use are as follows. HA and horizontal scalability with three control nodes. Similar to any routing platform, it is sufficient to have only two control nodes 232 to meet HA requirements. In many cases this is advantageous. (however, one or more control nodes 232 may be used.) for example, it provides a more deterministic infrastructure and is consistent with standard routing best practices. Each virtual router agent 514 is attached to a unique pair of control nodes 232 to avoid randomness. With two control nodes 232, debugging may be simpler. Furthermore, the edge replication for constructing the multicast/broadcast tree may be simplified with only two control annotations 232. Currently, since the vruter proxy 314 is connected to only two of the three control nodes, all control nodes may not have a complete tree picture for a period of time and rely on BGP to synchronize the states between them. This is even more serious in the case of three control nodes 232, since virtual router agent 314 may randomly select two. If there are only two control nodes 232, then each virtual router agent 314 will be connected to the same control node. This in turn means that the control node 232 does not need to rely on BGP to synchronize state and will have the same pictures of the multicast tree.

SDN architecture 200 may provide ingress replication as an alternative to edge replication and provide options to users. Ingress replication may be regarded as a special degradation case that generally covers multicast trees. In practice, however, the signaling of the ingress replication tree is much simpler than the signaling of a typical overlay multicast tree. Through ingress replication, each virtual router 21 ends with a tree that takes itself as the root and every other vruter as the leaf. The virtual router 21 down should not theoretically result in a reconstructed tree. Note that the performance of the ingress replication deteriorates with larger clusters. However, it works well for smaller clusters. Furthermore, multicasting is not a common and popular requirement for many clients. It is mainly limited to transmitting broadcast BUM traffic, which only occurs initially.

Configuration processing module enhancement

In a conventional SDN architecture, a network controller handles orchestration of all usage scenarios. The configuration node converts the intent into configuration objects based on the data model and writes them to a database (e.g., cassandra). In some cases, notifications are sent simultaneously, e.g., via a rabitmq, to all clients waiting for the configuration.

The control node not only acts as BGP speaker, but also has a configuration processing module that reads configuration objects from the database in the following manner. First, when a control node appears (or restarts), it connects to the database and reads all configurations directly from the database. Second, the control node may also be a messaging client. When there is an update to the configuration object, the control node receives a message notification listing the object that has been updated. This in turn causes the configuration processing module to read the object from the database.

The configuration processing module reads configuration objects of the control plane (BGP related configuration) and the vruter forwarding plane. The configuration may be stored as a graph with objects as nodes and relationships as links. The graph may then be downloaded to the client (BGP/cRPD and/or vruter proxy).

In some instances, the conventional configuration API server and messaging service are replaced in some instances by the etcd's Kube API server (API server 300 and custom API server 301) and the previous Cassandra database. With this change, clients interested in configuring objects can directly view the etcd database for updates instead of relying on the RabbitMQ notifications.

Controller orchestration for CRPD

BGP configurations may be provided to cRPD 324. In some examples, cRPD controller 720 may be a Kubernetes controller adapted to develop its own controller that is adapted to Kubernetes space and implements CRDs needed to orchestrate and provision cRPD 324.

Distributed configuration processing

As described earlier in this section, the configuration processing module may be part of the control node 232. It reads the configuration directly from the database, converts the data into JSON format, and uses it as a node, with the relationships between objects stored as linked graphs in its local IFMAP database. The graph is then downloaded via XMPP to the virtual router agent 514 of interest on the computing node. The virtual router agent 514 builds the IFMAP-based dependency graph locally and stores the objects.

The need for IFMAPs and storage dependency graphs as intermediate modules may be avoided by having the virtual router agent 514 directly monitor the etcd servers in the API server 300. The same model may be used by cRPD 324 running on a compute node. This would avoid the need for an IFMAP-XMPP config channel. A Kubernetes configuration client (for control node 232) may be used as part of the configuration. The client may also be used by a virtual router agent.

However, this may increase the number of clients reading the configuration from the etcd server, especially in clusters with hundreds of computing nodes. Adding more observers eventually leads to a decrease in write rate and less than ideal event rate. The etcd's gRPC proxy rebroadcasts from one server observer to multiple client observers. The gRPC agent merges multiple client watchers (c-watchers) on the same key or scope into a single watcher (s-watcher) connected to the etcd server. The agent broadcasts all events from the s-watcher to its c-watcher. Assuming that N clients view the same key, one gRPC proxy can reduce the viewing load on the etcd server from N to 1. A user may deploy multiple gRPC agents to further distribute server load. These clients share a server observer; the agent effectively offloads resource pressure from the core cluster. By adding agents, etcd can service one million events per second.

DNS/naming in SDN architecture

In previous architectures, the DNS service was provided by a process of confil-DNS and confil-named that work together to provide DNS services to VMs in the network. Named DNS server that provides an implementation of the BIND protocol. The con trail-dns receives updates from the vruter-proxy and pushes these records to name.

Four DNS modes are supported in the system and the IPAM configuration can select the desired DNS mode.

1. There is no-no DNS support for VMs.

2. DNS resolution of the default DNS server-VM is done based on the named server configuration in the server infrastructure. When the VM gets a DHCP response, the subnet default gateway is configured as the DNS server of the VM. The DNS request sent by the VM to the default gateway is resolved via a (fabric) naming server configured on the corresponding computing node, and a response is sent back to the VM.

3. Tenant DNS servers-tenants can use this schema to use their own DNS servers. The server list may be configured in the IPAM and then sent in a DHCP response to the VM as a DNS server. The DNS request sent by the VM is routed as any other data packet based on the available routing information.

4. Virtual DNS server-in this mode, the system supports virtual DNS servers, providing DNS servers that resolve DNS requests from VMs. We can define multiple virtual domain name servers under each domain in the system. Each virtual domain name server is an authoritative server for the configured DNS domain.

The SDN architecture described herein is efficient in terms of DNS services it provides. Clients in the cloud native world would benefit from various DNS services. However, with the move to the next generation Kubernetes-based architecture, the SDN architecture may instead use coreDNS for any DNS service.

Data plane

The data plane consists of two parts: the virtual router agent 514 (aka agent) and the virtual router forwarding plane 506 (also referred to as DPDK v router/Kernel v router) in the SDN architecture solution are responsible for managing the data plane components. The agent 514 establishes XMPP neighbor relation with the two control nodes 232 and then exchanges routing information with them. The vruter proxy 514 also dynamically generates flow entries and injects them into the virtual router 506. This gives instructions to virtual router 506 as to how to forward the packet.

Responsibilities of agent 514 include: interfacing with control node 232 to obtain a configuration. The received configuration is converted into a form that the data path can understand (e.g., a data model used to convert the data model from IFMap to the data path). Interfacing with control node 232 to manage routing. And collect and pull statistics from the data path to the monitoring solution.

Virtual router 506 implements data plane functions that may allow virtual network interfaces to be associated with VRFs. Each VRF has its own forwarding and flow tables, while MPLS and VXLAN tables are global within virtual router 506. The forwarding table may contain routes of IP and MAC addresses of the destination and IP-to-MAC associations are used to provide proxy ARP capability. When the VM/container interface is present and only locally important to virtual router 506, the virtual router selects the value of the label in the MPLS table. The VXLAN network identifier is global on all VRFs of the same virtual network in different virtual routers 506 within the domain.

In some examples, each virtual network has a default gateway address assigned to it, and each VM or container interface receives this address in a DHCP response received at initialization. When the workload sends a packet to an address outside its subnet, it will ARP for the MAC corresponding to the gateway's IP address, and virtual router 506 responds with its own MAC address. Thus, virtual router 506 may support fully distributed default gateway functions for all virtual networks.

The following is an example of packet flow forwarding implemented by virtual router 506.

Packets flow between VM/container interfaces in the same subnet.

The worker node may be a VM or container interface. In some examples, packet processing proceeds as follows:

VM 1/container interface needs to send a packet to VM2, so virtual router 506 first looks up its own DNS cache for the IP address, but since this is the first packet, there is no entry.

VM1 sends a DNS request to the DNS server address provided in the DHCP response when its interface appears.

Virtual router 506 captures DNS requests and forwards them to DNS servers running in the SDN architecture controller.

The DNS server in the controller responds with the IP address of VM2

Virtual router 506 sends a DNS response to VM1

VM1 needs to form an ethernet frame and therefore a MAC address for VM2 is needed. It checks its own ARP cache but has no entry because this is the first packet.

VM1 issues an ARP request.

Virtual router 506 captures the ARP request and looks up the MAC address of IP-VM2 in its own forwarding table and finds the association in the L2/L3 route that the controller sends to VM 2.

Virtual router 506 sends an ARP reply with the MAC address of VM2 to VM 1.

TCP timeout occurs in VM 1's network stack

The network stack of VM1 retries to send the packet, this time finds the MAC address of VM2 in the ARP cache, and can form an ethernet frame and send it out.

Virtual router 506 looks up the MAC address of VM2 and finds the encapsulation route. The virtual router 506 builds an outer header and sends the resulting packet to the server S2.

Virtual router 506 on server S2 decapsulates the packet and looks up the MPLS label to identify the virtual interface to which to send the original ethernet frame. Ethernet frames are sent to the interface and received by VM 2.

Packet flow between VMs in different subnets

In some examples, the order in which the data packets are sent to the destinations in the different subnets is similar, except that virtual router 506 responds as a default gateway. VM1 will send the packet in an ethernet frame with the default gateway's MAC address provided in the DHCP response provided by virtual router 506 at the time of VM1 boot. When VM1 makes an ARP request for a gateway IP address, virtual router 506 responds with its own MAC address. When VM1 sends an ethernet frame using the gateway MAC address, virtual router 506 uses the destination IP address of the packet within the frame to look up the forwarding table in the VRF to find the route, which will reach the host on which the destination is running via the encapsulation tunnel.

Fig. 10A-10M are diagrams illustrating examples of graphical user interfaces providing graphical representations of the topology of a network in which virtual network routers may be graphically defined to allow connectivity between different network elements. In the example of FIG. 10A, GUI 2000A may represent one example of GUI 60/244 that presents a graphical representation 2002A of a network topology. The graphical representation 2002A may present the network topology as a graphical data structure comprising graphical nodes representing network elements (e.g., VNs, VNRs, server clusters, etc.) and graphical edges representing connectivity between the graphical nodes.

Each graphical node may use text and graphical icons to specify the type of node. For example, the graphical node 2004A includes a graphical icon representing a virtual network (cloud with display indicia) and text identifying the graphical node 2004A as "VN 1". The graphical representation 2002A may include similar graphical icons 2004B through 2004N, wherein the graphical icons represent virtual networks, and text identifies the graphical nodes 2004B through 2004N as respective "VN2", "VN3", "VN4", … "VN15". As another example, the graph node 2006A includes a graph icon representing a cluster along with text identifying the graph node 2006A as "cluster 2" (which refers to a server cluster).

Each of the graph nodes 2008A-2008C includes a graph icon representing a VNR configured to provide grid connectivity, and text identifying each of the graph nodes 2008A-2008C as a VNR, along with text representing a type of connectivity (e.g., a "grid"). The graphical node 2010A includes graphical icons representing VNRs configured to provide central connectivity, and text identifying the graphical node 2010A as a VNR, also having text representing the type of connectivity (e.g., a "center" supporting central spoke connectivity). The graph node 2012A includes a graph icon representing a VNR configured to provide spoke connectivity, and text identifying the graph node 2012A as a VNR, along with text representing a connectivity type (e.g., a "spoke" supporting center spoke connectivity). Assuming center spoke connectivity provides unidirectional connectivity, where a spoke may forward data only to the center (and not directly to other spokes), the graph edge 2014A includes arrows indicating the unidirectional nature of center spoke connectivity.

GUI 2000A includes an icon 2020A (in this example, a graphical icon with a web representing "topology") that, when selected, causes GUI 2000A to present a graphical representation of topology 2002A. GUI 2000A also includes icons 2020B through 2020G corresponding to different interfaces of "favorites" (icon 2020B), "capabilities" (icon 2020C), "status" (icon 2020D), "configuration" (icon 2020E), "analysis" (icon 2020F), and "settings" (icon 2020G). Each of the icons 2020A-2020G ("icons 2020") may adapt the GUI 2000A to show (or control how) different aspects of the topology are viewed.

GUI 2000A also includes view controls 2030A to 2030E. View control 2030A may expose a legend identifying various icons for depicting different types of graphical nodes, graphical edges, etc. (as described in more detail below with respect to the example of fig. 10B). View control 2030B, when selected, may cause GUI 2000A to reset the view of graphical representation 2002A of the topology (e.g., to the original view when GUI 2000A is first rendered in response to selecting topology icon 2020A). View controls 2030C to 2030E, when selected, may cause GUI 2000A to zoom in and/or out on the view of graphical representation 2002A of the topology.

Further, GUI 2000A includes various mode controls 2040A to 2040D. Mode control 2040A, when selected, may enable GUI 2000A to add graphical elements (e.g., graphical nodes and graphical edges representing VNRs, VNs, etc.) that cause additions and/or changes to the configuration of the underlying topology represented by graphical representation 2002A. Mode control 2040B, when selected, may enable GUI 2000A to edit existing graphical elements, which may enable changes to the configuration of the underlying topology represented by graphical representation 2002A. Mode control 2040C, when selected, may cause GUI 2000A to enable a search function. Mode control 2040D, when selected, may cause GUI 2000A to enable a filtering function that enables a user to enter filtering criteria by which to limit the view of the topology represented by graphical representation 2002A.

GUI 2000A also includes topology/list switch controls 2050A and 2050B. The topology switch control 2050A, when selected, can cause the GUI 2000A to transition to the graphical representation 2002A of the network topology, and the list switch control 2050B, when selected, can cause the GUI 2000A to transition from the graphical representation 2002A of the network topology to a list view of the network topology.

In the example of FIG. 10A, GUI 2000A is configured to present a "topology" view and has enabled the filtering functionality of the reveal filtering control 2060A. The filter control 2060A includes a text box in which a filter criterion (e.g., "namespace = ns 1") is entered or a filter criterion is removed (e.g., "X" next to the entered filter criterion by selection). The filter control 2060A further includes a confirm check mark control that confirms the entered filter criteria, a delete ("X") control that removes the filter criteria, and a save filter control that saves the confirmed filter criteria. Given the filtering criteria of "namespace = ns1", GUI 2000A has filtered the topology of the network to show only the graph nodes associated with the namespace name "ns1" via graphical representation 2002A. Filtering may enable a user to focus only on specific portions of the network topology that are relevant to achieving a given objective and/or function.

Although described herein as having the GUI 2000A perform various functions in response to selecting various icons, controls, etc., it should be appreciated that the GUI 2000A may interface with the network controller 24 to provide various inputs in response to selecting various icons, controls, etc., and the network controller 24 may then update the GUI 2000A to expose and/or perform various functions attributed to the GUI 2000A (as well as other example GUIs described herein).

Referring next to the example of fig. 10B, GUI 2000B shows selecting view control 2030A to show the results of legend box 2070. Legend box 2070 provides a legend associating icons with different types of network elements, such as "virtual network router-grid", "virtual network router-center", "virtual network router-spoke", "virtual network" and "cluster". The legend box 2070 also identifies different topology states, such as "selected", "error", "unidirectional", "connected", "connection selected", "connection error", and "connection pending", linking these different topology states to the graphical representation of each topology state.

Referring next to the example of FIG. 10C, GUI 2000C shows the result of selecting "+" in filter control 2060A to reveal filter criteria prompt box 2080. The filter criteria prompt box 2080 may prompt the user via a drop down box, text box, radio button, check box, etc. to specify filter criteria applicable to the current graphical representation 2002A to further filter out graphical elements (e.g., graphical nodes 2004, graphical edges such as graphical edge 2014A, etc.) from the graphical representation 2002A of the network topology.

In the example of fig. 10D, GUI 2000D shows the result of receiving a selection of edit mode control 2040B (shown in the example of fig. 10A-10C) that causes GUI 2000D to reveal edit mode control panel 2090 including various icons indicating (from left to right, beginning with a check mark icon and ending with a filter icon) "accept", "edit", "view configuration", "cancel", "delete", "search", and "filter". Receiving selection of edit mode control 2040B also causes GUI 2000D to expose an add mode control panel 2092 having icons 2094A through 2094E (where the icons may represent button forms with additional graphical images and/or text).

Icon 2094A, when selected (by a user interfacing with GUI 2000D), may cause GUI 2000D to initiate a series of prompts by which to add VNRs with grid connectivity. Icon 2094B, when selected, may cause GUI 2000D to begin a series of prompts by which to add VNRs with spoke connectivity. Icon 2094C, when selected, may cause GUI 2000D to initiate a series of prompts by which to add VNRs with central connectivity. Icon 2094D, when selected, may cause GUI 2000D to initiate a series of prompts by which to add a VN, while icon 2094E, when selected, may cause GUI 2000D to initiate a series of prompts by which to add a connection (or in other words, a graphical edge) between graphical elements (e.g., VNs and/or VNRs).

Referring next to the example of fig. 10E, GUI 2000E shows the result of selecting icon 2094B, through which icon 2094B a VNR with spoke connectivity is added, which exposes a hint 2100 through which configuration data is defined to add a new VNR with spoke connectivity. The prompt 2100 is shown as a pop-up dialog box with a number of different graphical input elements (such as buttons, drop-down boxes, text boxes, previews, adaptive text boxes (e.g., text completions with underlying configuration data defining a network topology, and other adaptive, pre-filled, and/or other dynamic elements that facilitate user interaction), etc. while described as a pop-up box, the GUI 2000E may present the prompt 2100 in any manner, such as through a pop-up window, a separate GUI, additional tabs, an expanded box, etc.

In any case, the hint 100 shown in the example of FIG. 10E includes mode switches 2102A and 2102B, a namespace drop down box 2104, a VNR naming text box 2106, a descriptive text box 2108, VNR type switches 2110A through 2110C, expandable hint portions 2112A through 2112C, and a preview portion 2114. When switch 2102A is selected (again by the user), GUI 2000E may be caused to present prompt 2100 as shown in the example of fig. 10E, and switch 2102B, when selected, may cause GUI 2000E to present a YAML text editor by which the user may define configuration data for a new VNR using YAML (or in other words, a code conforming to YAML).

The namespace drop down box 2104 may allow the user to select a previously defined namespace in which to add a new VNR with spoke connectivity. The VNR naming text box 2106 may enable the user to enter VNR naming for a new VNR with spoke connectivity. Description text box 2108 may enable a user to enter a description of a new VNR having spoke connectivity. The VNR type switches 2110A to 2100C may enable a user to change the connectivity of a new VNR (which may also enable updating of various elements of the hint 2100, such as the extensible hint portions 2112A to 2112C), where the VNR type switch 2110A may switch connectivity to grid connectivity, the VNR type switch 2110B may switch connectivity to spoke connectivity (which is a default case based on selecting which type of VNR is added 2094C via icon 2094A), and the VNR type switch 2110C may switch connectivity to center connectivity.

The scalable hints sections 2112A to 2112C can be dynamically populated based on which VNR type switching is selected. In the example of fig. 10E, VNR type switching 2110B has been selected, which results in expandable hints sections 2112A through 2112C. However, depending on which VNR type switches 2110A to 2110C have been selected, the hint 2100 may include different, more, or less extensible hint portions specific to the particular VNR type selected via the VNR type switches 2110A to 2110C.

Expandable hint portion 2112A includes various graphical elements through which VNR tabs are specified according to the VNR keys and VNR tabs. The expandable hint portion 2112B includes various graphical elements through which a VN for a new VNR is specified. The expandable hints section 2112C includes various graphical elements by which a new spoke VNR is specified to be connected to a central VNR (in the context of potential VNR labels and/or namespaces) that enables central spoke connectivity.

Preview portion 2114 may represent a portion through which the configuration of a new VNR other than the VNR connectivity type is previewed. Assuming the user does not complete any portion of the prompt 2100, the preview portion 2114 provides a placeholder Fu Tuxing representation 2116 of a new VNR (which may be referred to as a placeholder Fu Futiao VNR 2116) with a corresponding VNR connectivity type (i.e., spoke connectivity in the example of fig. 10E). In this example, the preview portion 2114 also includes enabling preview switching to enable or disable the preview portion 2114.

Fig. 10F is a diagram illustrating the update-hint 2100 shown in the example of fig. 10E to define the results of a new spoke VNR with the name "VNR 11". The namespace drop-down box 2104 has been populated with inputs defining the namespace to which the new spoke VNR belongs as "namespace 1 (NAMESPACE 1)". VNR naming text box 2106 has been populated with inputs specifying the above-described name "VNR 11". Description text box 2108 has been populated with inputs specifying a "VNR for default network" description.

The expandable hint portion 2112A has been populated with inputs defining the key as "station" and the tag in this station as "SJC". The extensible hint portion 2112B has dynamically populated (e.g., via GUI 2000F) various VNs (via naming and namespaces) that are limited by the filtering criteria entered via the filtering control panel 2150 (where such filtering criteria — station = SVL "-indicates all VNs at the SVL station are to be returned).

The GUI 2000F may dynamically update the preview portion 2114 of the prompt 2100 to reflect the entry of each of the input and/or dynamic fill events described above (via the GUI 2000F). In the example of fig. 10F, GUI 2000F may update preview portion 2114 of prompt 2100 to replace placeholder Fu Futiao VNR 2116 with spoke VNR 2118 having a spoke icon and text reflecting the name of the new VNR ("VNR 11") and the connectivity type of the new spoke VNR ("spoke"). The GUI 2000F may also update the preview portion 2114 of the hint 2100 to represent potential connectivity (as defined by the dashed edges) with various matching VNs, shown as a graph node 2120 with a VN icon, text labeled "25VN match" and a label further identifying the number of matching VNs ("25").

Referring next to the example of fig. 10G, GUI 2000G shows the result of adding further filter criteria to filter control panel 2150 along with dynamic filter aid cues 2152 according to combining the previously entered filter criteria ("site=svl") with the newly entered filter criteria ("redisin 19, 50"). Dynamic filter aid hint 2152 can suggest boolean operators such as the examples of "AND" OR "shown in fig. 10G. Although an exemplary boolean operator is suggested, dynamic filter aid hint 2152 can specify any type of filter criteria operator, such as mathematical operators, keywords, values, and the like.

In the example of fig. 10H, GUI 2000H shows the results of expanding expandable hint portion 2112C through which a user may select a center VNR to which a new spoke VNR is to be connected to configure center spoke connectivity. The expandable hint portion 2112C informs the user that a new spoke VNR will connect to all center VNRs that match the VNR label and namespace combination selected below, and then lists various centers, allowing the user to select (via a check box) which of the existing center VNRs that match the VNR label and namespace combination the new spoke VNR should connect to. The user may also interface with the expandable hints portion 2112C to add different center VNRs (via "+" icons), edit listed center VNRs (via pencil icons), and delete various center VNRs (via garbage can icons).

The result of GUI 2000I shown in the example of fig. 10I shows that the center VNR is deselected in the expandable prompt portion 2112C, which causes the GUI 2000I to update the preview portion 2114 of the prompt 2100. The preview portion 2114 updates the preview of the potential topology to add potential connectivity to the three (3) matching hub VNRs, represented by the dashed edges from the graph node 2120 to the new graph node 2122 representing the three potential matching hub VNRs. The graph node 2122 includes an icon associated with the center VNR, text that labels "3 center matches", and a label that emphasizes the number of matching centers ("3").

In this regard, the GUIs 2000F through 2000I are configured by the network controller 24 to dynamically generate graphical elements that include graphical icons associated with the connectivity types of the virtual network routers. Upon dynamically generating graphical elements (e.g., graphical nodes in this example), the network controller 24 may receive input from a user indicating a connectivity type (e.g., spoke connectivity) of the VNR and select a graphical icon associated with the connectivity type of the virtual network router. The network controller 24 may then configure one or more of the GUIs 2000F through 2000I to present graphical elements that include graphical icons associated with the connectivity types of the virtual network routers.

Fig. 10J is a diagram illustrating a dynamic flow of interactions by which a user may interface with GUI 2000J to graphically connect a graph node 2004G representing a VN (which may be referred to as "VN8 2004G") to a graph node 2008C representing a grid VNR (which may be referred to as "grid VNR 5208C"). The dynamic flow of interaction is represented by a numbered dashed circle starting with the number one (1) and ending with the number five (5).

At dynamic flow step 1, the user may initially select (e.g., via left mouse click) to expose an option prompt 2300 that GUI 2000J may be populated with options specific to VN8 2004G (and/or possibly a particular type VN of graphical node 2004G). In the example of fig. 10J, the user selects "edit topology" from the options prompt 2300 instead of "view detailed information. In response to receiving an input selecting "edit topology," GUI 2000J transitions to edit topology prompt 2302 (at dynamic flow step two), where the user selects "add connection" instead of "edit detailed information" or "delete node").

In response to receiving input specifying that "add connection" is selected via editing topology prompt 2302, GUI 2000J may present prompt 2304 at dynamic flow step three, which prompts the user to graphically select the node you want to connect to VN8 2004G. The user may next select grid VNR52008C (as shown in dynamic flow step four), whereupon GUI 2000J may update graphical representation 2002A (in dynamic flow step five) to reflect the pending connection between VN8 2004G and grid VNR52008C in response to the input specifying that grid VNR 5208C was selected.

In the example of fig. 10K, GUI 2000K shows the result of completing a dynamic flow via GUI 2000J described above with reference to the example of fig. 10J. In response to receiving an input indicating that the dynamic flow has completed, GUI 2000K may present prompt 2400, which directs the user to view VN tag additions to VN8 2004G. Prompt 2400 may display existing VN8 labels and add labels to VN8 2004G, where each label includes a check box to include (check) or remove label additions to VN8 2004G (including existing labels as well as label additions).

In the example of FIGS. 10L and 10M, GUI 2000L/2000M shows the result of selecting list switch control 2050B. The GUI 2000L shown in the example of fig. 10L presents a current list of VNRs, where a user may select (via a check box) one or more VNRs in the list. GUI 2000M (shown in the example of fig. 10M) shows the result of selecting VNR 8 from the list shown in GUI 2000L, whereupon GUI 2000M presents an overlay 2500 specifying the detailed information of VNR 8.

Fig. 11 is a diagram illustrating a first example in which a virtual network router may be configured to implement mesh interconnectivity between virtual networks in accordance with aspects of the technology described in the present disclosure. In the example of fig. 11, virtual networks ("VNs") 1500A (also shown as "VN 1") and VN 1500B (also shown as "VN 2") are defined and implemented within Kubernetes (or some other orchestration platform) as custom resources, which are then implemented as VNs within an SDN architecture (e.g., SDN architecture 700). The VNs 1500A, 1500B may be instances of custom resources and have corresponding client API servers 301 and custom resource controllers 302.

The network controller 24 may allow an administrator to define the VN 1500A as a customized resource providing network connectivity to Pod 1502A (also shown as "Pod-1") and the VN 1500B as a customized resource providing network connectivity to Pod 1502B (also shown as "Pod-2").

As further shown in the example of fig. 11, both VNs 1500A and 1500N ("VN 1500") are defined within the same namespace 1504 (also shown as "namespace-1"), which indicates that both VNs 1500 are present within the public network namespace (and thus are not isolated from each other). At least in the context of Kubernetes, the namespace provides a mechanism for isolating resource groups, such as pod 1502A and pod 1502B ("pod 1502") and VN 1500, within a single cluster. As shown in the upper left, VN 1500 is not interconnected and does not provide network connectivity between each other, as indicated by the ping state of pod-2 (from pod 1502A) being "failed" and the ping state of pod-1 (from pod 1502B) being "failed".

An administrator may interconnect the VN 1500 to another by instantiating VNR 1506 within the same namespace 1504 (again shown as "namespace-1") to provide network connectivity between the pod 1502. As described above, VNR 1506 may represent a customized resource that an administrator may instantiate with an example naming "VNR-web" having a network connectivity type "grid". To specify which VNs to interconnect, the administrator may assign a label to each VN 1500 as "VN: web "then specifies a tag with" vn: web "mesh connectivity between VN 1500 of selection tags.

The network controller 24 (e.g., the customized resource controller 302) may process the customized resources to generate the above-described drop and drop policies that indicate that all routes from VN 1500A are to be dropped to VN 1500B via VNR 1506 and that all routes from VN 1500B are to be dropped to VN 1500A via VNR 1506, that all routes from VN 1500B are to be dropped to VN 1500A, and that all routes from VN 1500A are to be dropped to VN 1500B. This ingress/egress strategy is often referred to as symmetric because all routes for each VN are exchanged between each other VN, creating a grid.

The VNR 1506 may be configured by creating a common route target (shown as "VNR-RT" where RT stands for route target) "install" within the context of a common route instance (RI, shown as "VNR-RT") where an ingress/egress policy is defined as all routes that are ingress and egress from each VN 1500 to vnrRT. Once these policies are defined and installed within SDN architecture 700 associated with VN 1500 (e.g., in the routing plane), routes will be brought in and out into the VRFs of each of vnr-RTs and parsed to generate forwarding information. Examples of vnr-RI and vnr-RT may be examples of custom resources for SDN architecture configuration.

Control node 232 may generate forwarding information in the form of configuration data and interface with virtual router agent 514 to provide configuration data to virtual router agent 514, and virtual router agent 514 may install the forwarding information to virtual router 506 to implement a corresponding VRF of VN 1500 (e.g., VRF 222A shown in the example of fig. 5). Once installed, the forwarding information causes the virtual router 506 and other virtual routers in other computing nodes with Pod for the VN 1500 to be properly used for inter-VN traffic. These virtual routers can use the leaked routing information to forward pings correctly between pod1502 as shown in the example of fig. 10 by indicating ping to pod-2 (from pod 1502A) "pass" and ping to pod-1 (from pod 1502B) "pass".

Fig. 12 is a diagram illustrating a second example in which a virtual network router may be configured to implement mesh interconnectivity between virtual networks, in accordance with aspects of the technology described in the present disclosure. The example shown in fig. 12 is constructed from the example shown in fig. 11, where VNR 1506 provides grid interconnectivity between VNs 1500, which allows pod1502 in the same namespace 1504 to communicate symmetrically with each other.

In the example of fig. 12, VN 1500C and VN 1500D (also shown as "VN3" and "VN4", respectively) are defined within the same namespace 1504 and provide networking connectivity for the respective Pod 1502C and Pod 1502D (also shown as "Pod-3" and "Pod-4"). To add VN 1500C and VN 1500D to the grid provided by VNR 1506, the administrator may add "VN: web "markers are added to VN 1500C and VN 1500D.

After adding the labels corresponding to grid VNR 1506, network controller 24 may add VN 1500C and VN 1500D to grid VNR 1506 by generating ingress and egress policies to egress and ingress all routes between VN 1500C and VN 1500A, VN B and VN 1500D via the VNR common route target ("VNR-RT"), and again egress and ingress all routes between VN 1500D and VNs 1500A to 1500C via the VNR common route target ("VNR-RT"). That is, the control node 232 may only need to generate policies that direct routes from vnr-RT to the VNs 1500C and 1500D and route from the VNs 1500C and 1500D to vnr-RT. By using the common routing objective of VNR 1506, in some cases, routes may be automatically led out between all VNs 1500A to 1500D without having to update any existing policies (such as the ingress/egress policies of existing VNs 1500A and 1500B).

In any event, control node 232 may parse the route into forwarding information and interface with virtual router agent 514 to install the forwarding information in a manner similar to that described above with reference to FIG. 10. After forwarding information is installed, each Pod 1502A-1502D may have mesh network connectivity with each other Pod 1502A-1502D, as shown by a "pass" ping state relative to each other Pod 1502A-1502D (e.g., pod 1502A, denoted as "Pod-1", for each Pod 1502B-1502D, having a "pass" ping state, denoted as "Pod-2, pod-3, pod-4", etc.) for one of each other Pod 1502B-1502D. Fig. 12 illustrates in this way that adding an additional VN to a grid of VNs interconnected using VNRs can be achieved by simply adding an appropriate label to the VN instance, here, "VN: web).

Fig. 13A and 13B are diagrams illustrating a third example in which virtual network routers may be configured to implement mesh interconnectivity between virtual networks, in accordance with aspects of the technology described in the present disclosure. In this example, unlike the example of fig. 12, VNs 1500C and 1500D are coupled to VNR 1506B in the example of fig. 13A to provide grid network connectivity between VN 1500C and VN 1500D (right side), where VNR 1506 in the example of fig. 12 is replaced with similar VNR 1506A in the example of fig. 13A to provide grid network connectivity between VN 1500A and VN 1500B (left side).

VNR 1506A has a unique routing instance ("VNR-RI") in the context of namespace 1504 that is different from the routing instance of VNR 1506B (shown as "VNR-2-RI"). VNR 1506A provides VNR-RT and VNR 1506B provides VNR-2-RT (RT may be the same although different/unique routing instances are given). In addition, VNR 1506A has a label selection of "vn: web ", which selects VN 1500A and VN 1500B, and VNR 1506B has label selection" VN: db ", which is the label assigned to VN 1500C and VN 1500D. Thus, pod 1502A and 1502DB do not have network connectivity with pod 1502C and pod 1502D (represented by the "failed" ping state), while pod 1502C and pod 1502D do not have network connectivity with pod 1502A and pod 1502B (represented by the "failed" ping state).

To have mesh network connectivity for pod 1502A-1502D (because VNRs 1506A and 1506B are both "mesh" types), an administrator may add a tag selection statement to select tags for other VNRs 1506A and 1506B. Referring next to the example of fig. 13B, VNR 1506A now includes a tag selection statement "VNR: db ", the network controller 24 translates it into ingress and egress policies for VNR-db-RT, thereby ingress and egress routes from/to VNR-db-RT of VNR 1506B. Similarly, VNR 1506B now includes a tag selection statement "VNR: web ", the network controller 24 may convert it into ingress and egress routes from/to VNR-web-RT of VNR 1506A. All Pod of VNs connected to VNR 1506A ("VNR-web") and VNR 1506B ("VNR-db") may be routed through VNR 1506A, VNR B into/out of each other in such a way that it is possible to communicate with each other using VNR and tags.

Again, the network controller 24 (e.g., the custom resource controller 302) may deploy these policies within a routing plane, which may then perform routing information exchanges. The control node 232 may parse the exchanged routing information into forwarding information and interface with the virtual router agent 512 to install the forwarding information virtual router 506 in a manner similar to that described above with reference to fig. 11. Thus, mesh connectivity between the pods 1502A-1502D is established as evidenced by successful "pass" ping states of each other in the pods 1502A-1502D.

Fig. 14A and 14B are diagrams illustrating a fourth example in which a virtual network router may be configured to implement mesh interconnectivity between virtual networks, in accordance with aspects of the technology described in the present disclosure. In the example of fig. 14A, the general network structure is similar to that shown in the example of fig. 13A, except that VN 1500A and VN 1500B, pod1502 a and pod1502B, and VNR 1506A are in a first namespace 1504A (shown as "namespace-1"), while VN 1500C and VN 1500D, pod 1502C and pod 1502D, and VNR 1506B are in a second, different namespace 1504B (shown as "namespace-2").

Assuming namespaces 1504A and 1504B isolate VN 1500A/1500B, pod 1502A/1502B and VNR 1506A from VN 1500C/1500D, pod 1502C/1502D and VNR 1506B, there must be authorization to introduce routing information across namespaces 1504A and 1504B because each namespace 1504A and 1504B can be managed by a separate group without shared management rights across namespaces 1504A and 1504B. In this way, a request may be sent to an administrator of both namespaces 1504A and 1504B to confirm that routing information is allowed to be introduced and extracted between VNRs 1506A and 1506B.

Furthermore, because of the isolation provided by namespaces 1504A and 1504B relative to other namespaces, VNRs 1506A and 1506B are only allowed to import and export routing information on namespaces 1504A and 1504B. This limitation on inter-namespace routing information enforces the above authorization and avoids misconfigurations that may cause routing information to be exchanged that allows data packets with sensitive or other security information to be transmitted between namespaces 1504A and 1504B (which may cause data compliance to be violated, contractual obligations to be invalidated, or otherwise cause malicious information to violate the isolation provided by namespaces 1504A and 1504B).

Thus, assuming authorization is provided by an administrator of namespaces 1504A and 1504B, control node 232 can translate VNR 1506A and VNR 1506B into the ingress and egress policies described above with respect to the example of fig. 13B that facilitate generating forwarding information to provide grid connectivity between pod1502A to pod 1502D. Fig. 14B shows the results of such translation and installation of policies, as well as parsing exchanged routing information into forwarding information that can be installed in virtual router 506 to achieve such inter-namespace mesh network connectivity.

Fig. 15 is a diagram illustrating a fifth example of aspects of the technology described in the present disclosure, wherein virtual network routers may be configured to implement center spoke interconnectivity between virtual networks. In the example of fig. 15, VNs 1500A to 1500C are within namespace 1504 and provide network connectivity for respective pod1502A to 1502C. VN 1500A and VN 1500B are each assigned a VN tag "VN: web "and VN 1500C is assigned VN tag" VN: db).

In addition, an administrator may interact with network controller 24 to instantiate customized resources in the form of VNR 1516A and VNR 1516B. VNR 1516A is a "center" VNR, which is used as a center in a center spoke network configuration. The center spoke network configuration is an exchange in which the center VNR 1516A introduces all routing information from any spoke VNR coupled to the center VNR 1516A, and the spoke VNR coupled to the center VNR 1516A introduces routing information from the center VNR 1516A. VNR 1516B is configured as a "spoke" VNR, which acts as a spoke in a center spoke network configuration. Spoke VNR 1516B may pull all routing information to central VNR 1516A, but pull all routing information from central VNR 1516A. However, the spoke VN does not receive any routing information from other spoke VNs.

The administrator may define center VNR 1516A to select label "vn: db" and spoke VNR 1516B to select label "vn: web". As described above, the tag selector may indicate which of VNs 1500A-1500C is coupled to which of center VNR 1516A and spoke VNR 1516B.

In the example of fig. 15, control node 232 may generate a center public routing target ("center-RT" or "VNR-center-RT") for center VNR 1516A (along with spoke-RI or VNR-center-RI) and a spoke public routing target ("spoke-RT" or "VNR-spoke-RT" ("spoke-RT" or "VNR-spoke-RT") for spoke VNR 1516B (along with center-RI or VNR-center-RI). The control node 232 may then generate an extraction policy for the central VNR 1516A that directs any VNs 1500A to 1500C, in this example VN 1500C, labeled with the label "VN: db" to extract routing information to VNR-central-RT.

The control node 232 may also generate an output policy for spoke VNR 1516B indicating that spoke VNR 1516B should elicit all routing information to VNR-center-RT. The control node 232 may generate an introduction policy for the central VNR 1516A indicating that the central VNR 1516A should introduce all routing information from any spoke VNR, such as VNR-spoke-RT of the spoke VNR 1516B. The control node 232 may also generate ingress and egress policies indicating that the VNs 1500A and 1500B should offload all routing information to vnr-spoke-RT.

Once the policy is configured, the control node 232 may exchange routing information according to the policy, parse the routing information to generate forwarding information. The control node 232 via the virtual router agent 512 can install forwarding information in the virtual router 506 to enable center spoke interconnectivity, wherein each of the pod 1502C can communicate with each of the pod1502A and pod 1502B (because the center connected pod can reach any spoke pod), and the pod1502A and pod 1502B can communicate with the center pod 1502C but cannot communicate with each other because the spoke pod1502A and pod 1502B cannot communicate with each other under each center spoke network connectivity.

Fig. 16 is a diagram illustrating a sixth example in which virtual network routers may be configured to implement center spoke interconnectivity between virtual networks in accordance with aspects of the technology described in this disclosure. The example shown in fig. 16 is similar to the example shown in fig. 15, except in this example, the center VNR 1516A, VN C and pod 1502C are in a different namespace 1504B (shown as "namespace-2"), while the spoke VNRs 1516B, VN a and VN 1500B and pod1502A and pod 1502B are in the first namespace 1504A (shown as "namespace-1").

In this case, VNs 1500A-1500C may only interconnect with VNRs 1516A and 1516B in the same namespace, and for the reasons described above, only spoke VNR 1516B and center VNR 1516A may communicate across namespaces 1504A and 1504B. Assuming authorization is given by an administrator of the two namespaces 1504A and 1504B, the center spoke connectivity shown at the bottom of fig. 16 may be enabled in a manner similar, if not substantially similar, to that described above with respect to the example of fig. 15.

Fig. 17 is a diagram illustrating a seventh example in which a virtual network router may be configured to implement center spoke interconnectivity between virtual networks in accordance with aspects of the techniques described in this disclosure. The example shown in fig. 17 is similar to the example shown in fig. 15, except that an additional grid VNR 1506 is provided that allows for modified center spoke interconnectivity between VNs 1500A-1500D.

In the example of fig. 17, VNs 1500A-1500D are initially unable to communicate with each other (per "failed" ping state) until central VNR 1516A and spoke VNR 1516B are instantiated, deployed as policies, routing information is exchanged, then parsed, and forwarding information is installed in virtual router 506, as described in more detail above. This enables VNs 1500C and 1500D to elicit routing information to a common center-RT ("vnr-center-RT") while VNs 1500A and 1500B elicit routing information to a common spoke-RT ("vnr-spoke-RT"). Spoke VNR 1516B may output all routing information to VNR-center-RT, which enables center pod 1502C and 1502D to communicate with each other of spoke pod 1502A and 1502B. The spoke pod 1502A and 1502B can communicate with each other center pod 1502C and 1502D, but not with the other of the spoke pod 1502A and pod 1502B.

To enable the central pod 1502C and 1502D to communicate directly with each other, an administrator may instantiate a grid VNR 1506 that enables routing information to be brought out to a separate grid-VNR similar to that described above with respect to the example of fig. 10. In this way, the central pod 1502C and 1502D can communicate directly with each other because all routing information is exchanged between the VN 1500C and the VN 1500D via the mesh-RT.

Fig. 18A-18D are flowcharts illustrating operations performed when one or more virtual network routers are established within an SDN architecture in accordance with aspects of the technology described in the present disclosure. Referring first to the example of fig. 18A, a VNR controller of the network controller 24 (e.g., one of the custom resource controllers 302, such as the custom resource controller 302A shown in the example of fig. 6) may initially identify (or in other words, view the Ectd database to identify) a VN 50A having tags with names VN-1 and vn=web along with a VN 50N having tags with names VN-2 and vn=web be created as a custom resource (e.g., the configuration store 1328 shown in the example of fig. 6) in an Etcd database associated with the network controller 24, wherein a controller manager of the network controller 24 (e.g., the controller manager 1326) invokes a scheduler (e.g., the scheduler 1322) of the network controller 24 to create the VNs 50A and 50N (e.g., 1702, 1704, 1700).

Via kube-API of the network controller 24 (which again may refer to API server 300A and/or custom API server 301A shown in the example of fig. 6), an administrator may instantiate VNR 52A to interconnect VNs 50A and 50N, designating the type as a grid and the label "VN: web". The Kube-API updates the Etcd database to store the new custom resources to create VNR 52A and declares that the creation of VNR 52A is pending in the Etcd database (1706, 1708, 1710).

The dispatcher of the network controller 24 may next receive a notification from the Etcd database that a new customized resource has been defined, which then interfaces with the controller manager to notify the controller manager of the VNR creation request (1712, 1714, 1716). The controller manager may interface with the VNR controller to request VNR coordination, which creates an RI for VNR 52A, provides the new RI to the kube-API (1718), so that the kube-API may update Etcd to reflect the allocation of the new RI to VNR 52A (1720).

The Etcd database may then interface with a scheduler to indicate the presence of a VNR-RI creation event (1722), which in turn interfaces with a controller manager to notify the controller manager of the VNR-RI creation (1724). The controller manager then interfaces with the RI controller of the network controller 24 to request vnr-RI coordination (1726).

Referring next to the example of fig. 18B, the RI controller of network controller 24 may perform coordination of VNR-RI to create RT for VNR-1-RI, reporting VNR-1-RI back to kube-api (1728). Kube-api may update Ectd to note that VNR-1-RT has been created, which may report to Kube-api that VNR-1-RT is pending and then successfully created (1730, 1732). Kube-api may then interface with a VNR controller to indicate that a VNR-1-RI is created, which in turn may obtain the VNR-1-RT from the RI (1734, 1736).

The VNR controller may then interface with Kube-api to identify a list of VNs 50 with labels vn=web (1740), where Kube-api may return VN 50A and 50N designations VN-1 and VN-2 (1742). The VNR controller can again interface with the Kube-api to obtain the RI of VN-1 and VN-2 (1744), which continues to interface with the Kube-api to patch VNR-1-RT in the VN-1-RI, VN-2-RI state (1746). Kube-api then updates Ectd of the VN-RI patch event (1748). Ectd then notifies the scheduler of the patch event (1750), which notifies the controller of the manager patch event (1752).

Referring to the example of fig. 18C, in response to a patch event, the controller manager requests the RI controller to perform VN-RI coordination (1754) on the VN-RI prior to requesting a Kube-api patch VNR-1-RT in the VN-1-RI, VN-2-RI state (1756). In response to the patch request, the Kube-api updates VN-1-RI in Ectd to include policies for ingress and egress to VNR-1-RT (VNR common routing target), and updates VN-2-RI to include policies for ingress and egress to VNR-1-RT (VNR common routing target) (1758). Ectd reports the status of the patch back to Kube-api (1760). Responsive to the patch status of the VN-1-RI and VN-2-RI, the Kube-api interfaces with the RI controller to patch the VNR-1-RT at the VNR-RI (1762), wherein the RI controller reports the patch status back to the Kube-api (1764).

Referring next to the example of fig. 18D, kube-api interfaces with the Etcd to update the Etcd on patch events, which interfaces with the scheduler to notify the scheduler of patch events (1766, 1768). The scheduler then interfaces with the controller manager to notify the controller manager of the VNR-RI patch, which in turn requests VNR-RI coordination from the VNR controller (1770, 1772). The RI controller patches VNR-1-RT at VNR-RI and notifies Kube-api patches (1774, 1776). Kube-api updates Etcd on patch (1778) and interfaces with RI controller to set VNR-1 status to indicate "successful") and then Kube-api updates Ectd when VNR-1 status is successful (1780).

The detailed design of the VNR API model (as YAML file) is provided below:

/>

/>

/>

fig. 19 is another flow chart illustrating operation of the computer architecture shown in the example of fig. 1 in performing aspects of the technology described herein. As described above with respect to the example of fig. 1, the network controller 24 may present the GUI 60. In some examples, GUI 60 may represent a web-based GUI that network controller 24 may remotely host for display via a client device (such as a remote computer or other terminal). The GUI 60 may be configured to graphically represent the topology of the network supported by the Software Defined Network (SDN) architecture system 8. For illustration purposes, it is assumed that the network includes VNs 50A and 50N. However, the network may include any number of VNs 50 or other network elements.

The network controller 24 may maintain configuration data (not shown in the example of fig. 1) defining the topology of the network, which the network controller 24 may transform into a graphical representation of the topology of the network. The graphical representation of the network topology may facilitate viewing of a portion or all of the network, wherein the GUI 60 may provide filtering and other operations to enable a user to view the graphical representation of the network topology at various levels of granularity, as described in more detail below.

GUI 60 may also be configured to dynamically generate a graphical element representing VNR 52A, through which VN 50A and VN 50N are interconnected. GUI 60 may receive input from a user identifying VNRs 52A and VNs 50A and 50N, such as input indicating that a graphical element (e.g., a graphical icon) representing the VNR has been dragged over a network topology that may be proximate, and/or on one or more of VNs 50A and 50N. The GUI 60 may provide these inputs back to the network controller 24, and the network controller 24 may update the GUI 60 to include additional cues for the information about the VNR 52A and VNs 50A and 50N in response to these inputs.

GUI 60 (via network controller 24) may iterate in this manner until VNR 52A has been successfully defined in a manner that achieves connectivity between VNs 50A and 50N. In this regard, the network controller 24 may present a UI 60, which UI 60 is configured to graphically represent the topology of the network and dynamically generate graphical elements representing virtual network routers through which VNs 50A/50N included in the network are interconnected (1800).

The network controller 24 may execute the configuration node 30 to verify the VNR 52A before invoking the control node 32 to configure the VNs 50A and 50N. Once successfully authenticated, control node 32 configures VNs 50A and 50N according to one or more policies to allow one or more of routing information to be introduced and withdrawn between VN 50A and VN 50N via VNR 52A. In other words, network controller 24 may execute a control node configured to configure VNs 50A/50N according to one or more policies represented by VNR 52A that enable one or more of the ingress and egress of routing information between VNs 50A/50N (1802).

In this way, various aspects of the technology may implement the following examples.

Example 1. A network controller for a Software Defined Network (SDN) architecture system, the network controller comprising: a memory configured to store a user interface; processing circuitry configured to present the user interface and execute the control node, wherein the user interface is configured to: graphically representing a topology of a network supported by the Software Defined Network (SDN) architecture system, the network including a first virtual network and a second virtual network; and dynamically generating a graphical element representing a virtual network router for interconnecting the first virtual network and the second virtual network, the virtual network router representing a logical abstraction of one or more policies that cause one or more of ingress and egress of routing information between the first virtual network and the second virtual network, wherein the control node configures the first virtual network and the second virtual network according to the one or more policies to enable one or more of ingress and egress of routing information between the first virtual network and the second virtual network via the virtual network router.

2. The network controller of example 1, wherein the user interface is configured to present a first graphical element representing the first virtual network and a second graphical element representing the second virtual network when graphically representing a topology of the network.

Example 3 the network controller according to any combination of examples 1 and 2, wherein the user interface is configured to, when graphically representing a topology of the network: receiving input from a user indicating a connectivity type of the virtual network router; selecting a graphical icon associated with a connectivity type of the virtual network router; and dynamically generating a graphical element comprising a graphical icon associated with the connectivity type of the virtual network router.

Example 4 the network controller of example 3, wherein the connectivity type includes one of grid connectivity and center spoke connectivity.

Example 5 the network controller of example 3, wherein the connectivity type comprises mesh connectivity, and wherein the one or more policies represented by the mesh virtual network router comprise symmetric ingress and egress policies that cause both the ingress and the egress of the routing information between the first virtual network and the second virtual network.

Example 6. The network controller of example 3, wherein the connectivity type comprises center spoke connectivity; and wherein the one or more policies represented by the central virtual network router include asymmetric ingress and egress policies that cause the routing information to be exported from both the first spoke virtual network and the second spoke virtual network to the virtual network router without the routing information being imported between the first spoke virtual network and the second spoke virtual network.

Example 7 the network controller of any combination of examples 1-6, wherein the user interface is configured to: when graphically representing the topology of the network, receiving input from a user defining a first tag associated with the first virtual network and a second tag associated with the second virtual network, and wherein the processing circuit executes a configuration node that identifies based on the first tag and the second tag: the first virtual network and the second virtual network are configured to configure routing instances corresponding to the virtual network routers according to the one or more policies such that the routing information is introduced and extracted between the first virtual network and the second virtual network.

Example 8 the network controller of any combination of examples 1-7, wherein the first virtual network is associated with a first namespace, wherein the second virtual network is associated with a second namespace, wherein the virtual network router is a first virtual network router, wherein the one or more policies include a first policy and a second policy, wherein the first virtual network router represents a first policy for ingress and egress of routing information between the first virtual network and the second virtual network router, and wherein the second virtual network router represents a second policy for ingress and egress of routing information between the second virtual network and the first virtual network router.

Example 9 the network controller of example 8, wherein the first virtual network router must be authorized to interconnect the first virtual network router to the second virtual network router prior to deploying the first policy.

Example 10. The network controller of any combination of examples 1-9, wherein the network controller supports a container orchestration system.

Example 11 the network controller of any combination of examples 1-10, wherein the virtual network router is an instance of a customized resource for an SDN architecture, the customized resource for an SDN architecture configuration, and wherein the processing circuitry executes the customized resource controller to coordinate a state of the SDN architecture with an expected state of the virtual network router, and wherein the control node is configured to deploy the virtual network router to implement the expected state of the virtual network router.

Example 12 the network controller of any combination of examples 1-11, wherein the virtual network router is implemented using a common routing instance and a common routing target, wherein the first virtual network is implemented using a first routing instance and a first routing target, wherein the second virtual network is implemented using a second routing instance and a second routing target, and wherein the control node is configured to configure the ingress and egress of the first routing instance, the second routing instance, and the common routing instance with the first routing target, the second routing target, and the common routing target to achieve mesh connectivity.

Example 13 the network controller of any combination of examples 1-12, wherein the first virtual network is implemented using one or more first virtual routers of a first plurality of computing nodes, and wherein the second virtual network is implemented using one or more second virtual routers of a second plurality of computing nodes.

Example 14 the network controller of example 13, wherein the first plurality of computing nodes is a first Kubernetes cluster, and wherein the second plurality of computing nodes is a second Kubernetes cluster.

Example 15 the network controller of any combination of examples 1-14, wherein the control node parses the routing information into forwarding information for one or more of the first virtual network and the second virtual network and installs the forwarding information in a virtual router supporting interconnectivity between the first virtual network and the second virtual network.

Example 16. A method, comprising: presenting, by a network controller for a Software Defined Network (SDN) architecture system, a user interface, wherein the user interface is configured to: graphically representing a topology of a network supported by the Software Defined Network (SDN) architecture system, the network including a first virtual network and a second virtual network; and dynamically generating a graphical element representing a virtual network router for interconnecting the first virtual network and the second virtual network, the virtual network router representing a logical abstraction of one or more policies that cause one or more of ingress and egress of routing information between the first virtual network and the second virtual network; and executing, by the network controller, a control node that configures the first virtual network and the second virtual network according to the one or more policies to enable one or more of ingress and egress of routing information between the first virtual network and the second virtual network via the virtual network router.

Example 17 the method of example 16, wherein graphically representing the topology of the network includes presenting a first graphical element representing the first virtual network and a second graphical element representing the second virtual network.

Example 18 the method of any combination of examples 16 and 17, wherein graphically representing the topology of the network comprises: receiving input from a user indicating a connectivity type of the virtual network router; selecting a graphical icon associated with a connectivity type of the virtual network router; and dynamically generating a graphical element comprising a graphical icon associated with the connectivity type of the virtual network router.

Example 19 the method of example 18, wherein the connectivity type includes one of grid connectivity and center spoke connectivity.

Example 20. A non-transitory computer-readable medium comprising instructions to cause processing circuitry of a network controller of a Software Defined Network (SDN) architecture system to: presenting a user interface, wherein the user interface is configured to: graphically representing a topology of a network supported by the Software Defined Network (SDN) architecture system, the network including a first virtual network and a second virtual network; and dynamically generating a graphical element representing a virtual network router through which the first virtual network and the second virtual network are interconnected, the virtual network router representing a logical abstraction of one or more policies that cause one or more of ingress and egress of routing information between the first virtual network and the second virtual network; and executing a control node that configures the first virtual network and the second virtual network according to the one or more policies to enable one or more of ingress and egress of routing information between the first virtual network and the second virtual network via the virtual network router.

If implemented in hardware, the present disclosure may be directed to an apparatus, such as a processor or an integrated circuit device, such as an integrated circuit chip or chipset. Alternatively or additionally, if implemented in software or firmware, the techniques may be realized at least in part by a computer-readable data storage medium comprising instructions that, when executed, cause a processor to perform one or more of the methods described above. For example, a computer-readable data storage medium may store such instructions for execution by a processor.

The computer readable medium may form part of a computer program product, which may include packaging material. The computer-readable medium may include computer data storage media such as Random Access Memory (RAM), read Only Memory (ROM), non-volatile random access memory (NVRAM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory, magnetic or optical data storage media, and the like. In some examples, an article of manufacture may comprise one or more computer-readable storage media.

In some examples, the computer-readable storage medium may include a non-transitory medium. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier wave or propagated signal. In some examples, a non-transitory storage medium may store data (e.g., in RAM or cache) that may change over time.

The code or instructions may be software and/or firmware executed by a processing circuit comprising one or more processors, such as one or more Digital Signal Processors (DSPs), general purpose microprocessors, application Specific Integrated Circuits (ASICs), field Programmable Gate Arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Thus, the term "processor" as used herein may refer to any of the foregoing structures or any other structure suitable for implementation of the techniques described herein. Additionally, in some aspects, the functionality described in the present disclosure may be provided within software modules or hardware modules.

Claims (20)

1. A network controller for a Software Defined Network (SDN) architecture system, the network controller comprising:

a memory configured to store a user interface;

processing circuitry configured to present the user interface and configured to execute a control node, wherein the user interface is configured to:

graphically representing a topology of a network supported by the Software Defined Network (SDN) architecture system, the network including a first virtual network and a second virtual network; and

dynamically generating a graphical element representing a virtual network router, wherein the first virtual network and the second virtual network are interconnected by the virtual network router, the virtual network router representing a logical abstraction of one or more policies that cause one or more of ingress and egress of routing information between the first virtual network and the second virtual network,

Wherein the control node configures the first virtual network and the second virtual network according to the one or more policies to enable one or more of ingress and egress of routing information between the first virtual network and the second virtual network via the virtual network router.

2. The network controller of claim 1, wherein the user interface is configured to: a first graphical element representing the first virtual network and a second graphical element representing the second virtual network are presented when graphically representing the topology of the network.

3. The network controller of claim 1, wherein the user interface is configured to, when graphically representing a topology of the network:

receiving input from a user indicating a connectivity type of the virtual network router;

selecting a graphical icon associated with a connectivity type of the virtual network router; and

dynamically generating the graphical element including the graphical icon associated with a connectivity type of the virtual network router.

4. The network controller of claim 3, wherein the connectivity type comprises one of grid connectivity and center spoke connectivity.

5. The network controller of claim 3,

wherein the connectivity type includes grid connectivity, and

wherein the one or more policies represented by the mesh virtual network router include symmetric ingress and egress policies that cause both ingress and egress of the routing information between the first virtual network and the second virtual network.

6. The network controller of claim 3,

wherein the connectivity type comprises center spoke connectivity; and

wherein the one or more policies represented by the central virtual network router include asymmetric ingress and egress policies that cause the routing information to be exported from both the first spoke virtual network and the second spoke virtual network to the virtual network router, but not imported between the first spoke virtual network and the second spoke virtual network.

7. The network controller of claim 1, wherein the user interface is configured to: receiving input from a user defining a first label associated with the first virtual network and a second label associated with the second virtual network while graphically representing a topology of the network, and

Wherein the processing circuitry executes a configuration node that identifies the first virtual network and the second virtual network based on the first tag and the second tag to configure a routing instance corresponding to the virtual network router according to the one or more policies such that the routing information is introduced and extracted between the first virtual network and the second virtual network.

8. The network controller of claim 1,

wherein the first virtual network is associated with a first namespace,

wherein the second virtual network is associated with a second namespace,

wherein the virtual network router is a first virtual network router,

wherein the one or more policies include a first policy and a second policy,

wherein the first virtual network router represents the first policy for the ingress and egress of the routing information between the first virtual network and a second virtual network router, and

wherein the second virtual network router represents the second policy for the ingress and egress of the routing information between the second virtual network and the first virtual network router.

9. The network controller of claim 8, wherein the first virtual network router must obtain authorization to interconnect the first virtual network router to the second virtual network router prior to deploying the first policy.

10. The network controller of claim 1, wherein the network controller supports a container orchestration system.

11. The network controller of claim 1,

wherein the virtual network router is an instance of a customized resource for an SDN architecture configuration of an SDN architecture,

wherein the processing circuitry executes a custom resource controller to coordinate a state of the SDN architecture to an expected state for the virtual network router, and

wherein the control node is configured to deploy the virtual network router to achieve the desired state for the virtual network router.

12. The network controller of claim 1,

wherein the virtual network router is implemented using a common routing instance and a common routing target,

wherein the first virtual network is implemented using a first routing instance and a first routing target,

wherein the second virtual network is implemented using a second routing instance and a second routing objective, and

Wherein the control node is configured to configure ingress and egress of the first routing instance, the second routing instance, and the common routing instance with the first routing target, the second routing target, and the common routing target to achieve mesh connectivity.

13. The network controller of claim 1,

wherein the first virtual network is implemented using one or more first virtual routers of a first plurality of computing nodes, an

Wherein the second virtual network is implemented using one or more second virtual routers of a second plurality of computing nodes.

14. The network controller of claim 1,

wherein the first plurality of computing nodes is a first Kubernetes cluster, and wherein the second plurality of computing nodes is a second Kubernetes cluster.

15. The network controller of claim 1, wherein the control node parses the routing information into forwarding information for one or more of the first virtual network and the second virtual network and installs the forwarding information in a virtual router that supports interconnectivity between the first virtual network and the second virtual network.

16. A method, comprising:

a user interface is presented by a network controller for a Software Defined Network (SDN) architecture system,

wherein the user interface is configured to:

graphically representing a topology of a network supported by the Software Defined Network (SDN) architecture system, the network including a first virtual network and a second virtual network; and

dynamically generating a graphical element representing a virtual network router, wherein the first virtual network and the second virtual network are interconnected by the virtual network router, the virtual network router representing a logical abstraction of one or more policies that cause one or more of ingress and egress of routing information between the first virtual network and the second virtual network; and

executing, by the network controller, a control node, wherein the control node configures the first virtual network and the second virtual network according to the one or more policies to enable one or more of ingress and egress of routing information between the first virtual network and the second virtual network via the virtual network router.

17. The method of claim 16, wherein graphically representing the topology of the network comprises: a first graphical element representing the first virtual network and a second graphical element representing the second virtual network are presented.

18. The method of claim 16, wherein graphically representing the topology of the network comprises:

receiving input from a user indicating a connectivity type of the virtual network router;

selecting a graphical icon associated with a connectivity type of the virtual network router; and

dynamically generating the graphical element including the graphical icon associated with a connectivity type of the virtual network router.

19. The method of claim 18, wherein the connectivity type comprises one of grid connectivity and center spoke connectivity.

20. A non-transitory computer readable medium comprising instructions for causing a processing circuit of a network controller for a Software Defined Network (SDN) architecture system to:

a user interface is presented with a user interface that,

wherein the user interface is configured to:

graphically representing a topology of a network supported by the Software Defined Network (SDN) architecture system, the network including a first virtual network and a second virtual network; and

Dynamically generating a graphical element representing a virtual network router, wherein the first virtual network and the second virtual network are interconnected by the virtual network router, the virtual network router representing a logical abstraction of one or more policies that cause one or more of ingress and egress of routing information between the first virtual network and the second virtual network; and

executing a control node, wherein the control node configures the first virtual network and the second virtual network according to the one or more policies to enable one or more of ingress and egress of routing information between the first virtual network and the second virtual network via the virtual network router.