JP2022521703A - Virtual service network - Google Patents

Abstract

Some embodiments provide a method for establishing multiple virtual service networks on multiple data centers. This method configures a set of machines distributed throughout a data center for each virtual service network of multiple virtual service networks to provide an ordered set of network services for the virtual service network. .. This method receives a data message, selects one of the virtual service networks for the data message based on the analysis of the content of the data message, and an ordered set of network services for the selected virtual service network. Configure multiple service network selectors in the data center to determine the location in the data center for the machine that implements the first network service and send data messages to the machine that implements the first network service.

Description

[Background technology]
Communication service provider networks receive and process many types of traffic from many different types of equipment, especially advancing. For example, these networks have traffic from mobile phones, IoT (Internet of Things) devices, automobiles, home computers, and the like. Ideally, this traffic is the type of application (eg, streaming video, web browsing, phone calls, etc.), the type of device (eg, data traffic for self-driving cars requires very short latency), and It should be treated differently by network, based on other differentiating factors. Although the 4G and 5G standards have introduced some level of traffic differentiation, more adaptable network slicing that can be generalized to other types of networks is desirable.

Some embodiments provide a method for establishing a virtual service network across a set of data centers. The set of data centers in which a virtual service network is established includes, for example, a software-defined wide area network (SD-WAN) that spans one or more public clouds, public clouds, and private clouds, and a telecommunications service provider access network (eg, wireless access). It can include a combination of networks, edge clouds, and core clouds), or other types of data centers. The virtual service network of some embodiments comprises a large number of network slices in which the data messages assigned to the network slices each provide a different network service.

In some embodiments, when a device (eg, a mobile endpoint device in a telecommunications context) sends a data message over such a network, the network slice selector first processes the data message. This network slice selector assigns a data message to one of the network slices of the virtual service network and processes the service chaining operation so that the data message is processed by the correct set of network services for the assigned slice. In different embodiments, this network slice selector is a virtual machine (VM), a containerized function, a software transfer element running within a VM (eg, a flow-based transfer element), within a container or within the virtualization software of a host computer. A set of modules that run on the host computer, a set of modules that run outside the transfer element within the virtualization software of the host computer (eg, between the VM and the port of the transfer element), a hardware transfer element (eg, a programmable switch), Alternatively, it may be implemented by other implementations.

In some cases, many network slice selectors are configured to implement virtual service networks. In the example of a telecommunications service provider, some embodiments configure a network slice selector for each cellular tower, base station, or other aspect of the access network. Some embodiments of telecommunications service provider access networks include an edge cloud for each cellular tower, each of which constitutes at least one network slice selector. In another example (for example, for SD-WAN traffic that is completely contained within a set of connected data centers), the distributed network slice selector can select network slices for data messages sent from the VM. It is configured to occur on the same host computer (outside the source VM) as the source of the data message or on a specified device (eg, a specific nearby switch or router, a dedicated VM).

Each network slice of a virtual service network is, in some embodiments, a firewall, a load balancer, network address translation, weighing (eg, for billing purposes), a virtual private network (VPN) gateway, a wireless access network (RAN) function ( One or more, such as distributed unit and centralized unit functions), evolved packet core (EPC) functions (eg, subscriber management servers, service gateways, packet data network gateways, mobility management entities), or other types of network functions. Includes network services. These network functions may be implemented as virtual network functions (VNF), physical network functions (PNF), and / or cloud network functions (CNF) in different embodiments.

When the network slice selector assigns a data message to a network slice, in some embodiments, the slice selector performs service chaining to ensure that the data message traverses the network services of the assigned slice in the correct order. Play a role in doing. In some embodiments, the slice selector sends a data message to a first network service (eg, a VM, container, or other data compute node that implements the network service) and maintains contextual information for that data message. .. When the first network service finishes processing the data message, the first network service returns the data message to the slice selector. The slice selector then uses the maintained context information to send the data message to the next network service, etc. In some embodiments, if a complete network slice is implemented across many data centers, a similar service chaining module will handle each service chaining for slices within its own data center. Works in the data center. These service chaining modules may be implemented in the same manner as the network slice selectors in some embodiments (eg, as VMs, as VMs or as transfer elements in virtualization software). The service chaining module of some embodiments receives the data message as it enters the data center and identifies the slice for that data message (eg, the network slice selector or service check of the previous data center). Serve the data message to the next network service in the data center (based on the contextual information provided with the data message by the inning module). Other embodiments do not return a data message to the specified slice selector or service chaining module in each data center (eg, a tag in the packet header indicating the order of services in the selected network slice). Use distributed service chaining (by adding).

In some embodiments, the controller hierarchy constitutes various entities within one or more data centers to implement a virtual service network. High-level controllers (referred to here as virtual service network (VSN) controllers) are virtual from users (eg, telecommunications providers, data center tenants) via interfaces (eg, REST API sets, graphical interfaces, command line interfaces). Receives service network configuration data. This VSN controller aligns a set of other controllers that make up an entity in a data center where VSN is implemented. In some embodiments, each data center has its own suite of lower level controllers. These controllers are for configuring a compute controller (eg, for configuring a VM that implements VNF), a network controller (eg, for configuring a forwarding element for sending data messages between a slice selector and a network service). ), Storage controllers, and SDN controllers (eg, for configuring slice selectors and / or gateways that send data messages between data centers).

In different embodiments, the network slice selector may use different techniques to assign data messages to slices. Slice selection may be based on a combination of Layer 2-Layer 4 (L2-L4) headers and / or (eg, to classify traffic based on the data in Layer 5-Layer 7 (L5-L7) headers. It may also be done by performing a deep packet inspection. For example, slice selection may simply be based on the source device by using the source network layer (eg IP) address, or based on the traffic type / or destination network domain by looking at the higher layer (L5-L7) header. You may. In some embodiments, the network slice selector integrates with other control plane components to collect additional information about the connection (eg, regarding user sessions, device types, or other data) and slices this information. Used as part of the selection process (eg, use only this collected information or combine this information with L2-L4 and / or L5-L7 packet header data). In some embodiments, the network slice selector maintains a state for mapping the connection to the network slice so that it does not need to perform deep packet inspection for each data message of the connection. Also, in some connections, only certain data messages contain the L5-L7 header information needed to perform slice selection.

When performing network slice selection using deep packet inspection, in certain cases the initial data message for the connection may not contain the L5-L7 header information needed for the slice selector to correctly identify the slice. be. For example, endpoint devices (eg, mobile devices such as smartphones or tablets, laptops or desktop computers, IoT devices, self-driving cars, smart cameras belonging to security systems, or other devices) and network domains (eg, www. Connections with (web domains such as netflex.com, www.google.com) often begin with a set of connection initiation messages such as the TCP handshake. When the handshake is complete, the device sends an http get message containing the network domain and the like. Subsequent data messages sent between the device and the network domain may not contain such information.

In some embodiments, the network slice selector acts as a proxy to terminate the connection start message without sending these messages to the intended destination across the virtual service network. That is, this network slice selector exchanges the appropriate set of connection initiation messages with the endpoint device so that the endpoint device behaves as if the connection to the destination domain was properly set up. The network slice selector stores data about these connection start messages for subsequent use. When the network slice selector receives a data message (such as a http get message) from an endpoint device, it performs a deep packet inspection to examine the L5 to L7 headers and select one of the network slices for connection. The network slice selector then exchanges the connection start message with the destination network domain, this time acting as a device rather than the destination domain. These connection start messages are sent through the selected network slice so that all network services on the selected network slice process the connection start message. When this connection is established with the destination domain, the slice selector also forwards the data message to the destination domain via the selected network slice. The slice selector also stores connection information (eg, 5 tuples of connections) that maps connections to selected network slices for use in processing data messages. For subsequent data messages received for a connection, the network slice selector uses the mapping of the connection to the slice to assign the data message to the selected slice without having to perform deep packet inspection. This avoids performing deep packet inspections on all data messages on the connection, as the L5 to L7 headers on all data messages belonging to this connection may not have the required information. , May not be possible. In some embodiments, the egress gateway that processes the data message after the virtual service network completes also stores the connection-to-slice mapping information and sends the data message from the destination domain to the endpoint device (reverse order). Use this mapping information to assign to the correct network slice (along with the network services that run in).

In some embodiments, instead of ending the connection start message with the slice selector, these messages are first passed through the default network slice, then the network slice is selected, and then the correct network slice for the connection. Play the message via. In this procedure, the slice selector stores these messages (or the data needed to recreate the messages) while assigning the initial connection start message to the default slice. In addition, the egress gateway (ie, the gateway that sends the message to an external destination (eg, the Internet) and receives the reverse message from the destination domain) also stores information about the connection start message. Upon receiving the initial data message, the network slice selector performs a deep packet inspection to select the correct slice. However, before sending the data message to the selected slice, the slice selector plays the connection start message through the network slice. Instead of sending these messages from the egress gateway to an external destination domain, the connection start message is sent only between the slice selector and the egress gateway. If these messages were replayed, the network service of the selected slice would have processed the connection start message and would be prepared for subsequent data message traffic between the endpoint device and the destination network domain.

In the case of stateful slice selection, as mentioned above, to save resources / time and many of the subsequent data messages have the information in their L5-L7 headers needed for deep packet inspection to be useful. Subsequent data messages are sent using the state stored by the network slice selector, both because and not. However, mobile devices (eg smartphones, tablets, self-driving cars) are serviced by a second slice selector from one geographical range served by the first slice selector while maintaining one connection. You can move to different geographic areas (eg, when moving from one cell tower to another, or from a WiFi network to a cellular network). In different embodiments, different techniques are used to ensure that the state is maintained without the need for action on some of the endpoint devices.

In some embodiments, the second slice selector (the slice selector for the area where the mobile device moves) sends all the data messages for the connection to the first slice selector (the mobile device when the connection is initiated). Transfer to the slice selector) for the area where was located. That is, the second slice selector receives data indicating that the first slice selector is in the slice mapping state position for the connection, and thus forwards the data traffic for the connection to the first slice selector. .. In a different embodiment, the first slice selector (i) pushes this state position information directly to the second slice selector, or (ii) pushes the state position information to a network controller (eg, the VSN controller described above). Then, the second slice selector retrieves the state position information from there.

In another embodiment, the second slice selector receives this state (ie, mapping the connection to the network slice) and therefore network the data message for the connection without the first network slice selector. Can be transferred to slices. In different embodiments, the second slice selector may receive the state directly from the first slice selector or from a network controller (eg, the VSN controller described above). In some such embodiments, the first slice selector (i) pushes the state directly to the second slice selector (eg, before the device moves to the geographic area of the second slice selector). Or (ii) the second slice selector pushes the state to the network controller that retrieves the state. In other such embodiments, the first slice selector pushes position information about the state to the network controller, the second slice selector retrieves this position information from the network controller, and then the state from the first slice selector. Use that location information to retrieve.

If the first slice selector pushes state information directly to the second slice selector, in some embodiments, the geographic area adjacent to the mobile device that initiates the connection within the geographic area of the first slice selector. When moving to any of, the first slice selector pushes all of its slice mappings to the slice selector for the adjacent geographic area. In other such embodiments, the first slice selector pushes state information to the slice selector for adjacent geographic areas where the device may move, so that the location data of the mobile device (that data is). If enabled) is used.

In some embodiments, the virtual service network is sliced hierarchically. That is, a slice of a virtual service network is itself a virtual service network with a slice selector and multiple network slices. For example, in a telecommunications network, a mobile network operator (MNO) owns the physical infrastructure of access and core networks (ie, RAN and EPC infrastructure), and traffic from devices that subscribe to that MNO is handled by that infrastructure. Will be done. In addition, the MNO can lease its infrastructure to one or more mobile virtual network operators (MVNOs) who also have subscriber devices that use the same infrastructure. These MVNOs may also lease their virtual infrastructure to additional MVNOs or other entities. In addition, a hierarchical layer of slice selection can be implemented on the network for additional reasons other than different telecommunications service providers.

In the telecommunications provider example above, the first slice selector configured by the MNO may assign data messages to network slices based on the source device (eg, by source network address). Therefore, data messages from source devices associated with MNOs are sent to another virtual service network configured by the MNO, and data messages from source devices associated with different MVNOs are virtual services configured by each MVNO. Sent to the network. In some embodiments, a second slice selector for each virtual service network performs additional slice selections based on various aspects of the data message header. If the MVNO leases its virtual infrastructure to one or more additional MVNOs, this second slice selector is based on finer-grained network address analysis (eg, the first MVNO has a pool of IP addresses. Data messages can also be assigned to network slices (if assigned and splitting this pool between its own device and another device for the MVNO). In other cases, the second level slice selector may perform stateful slice selection based on deep packet inspection as described above.

In some embodiments, within a virtual service network (a collection of multiple network slices with different ordered sets of network services), the network slice selector is always the first entity to process a data message. In another embodiment, after the first network slice selector selects one of the slices, this slice (which is a virtual service network) allows the second network slice selector to select a slice within that virtual service network. Prior to selection, it may include network services that apply to the data message. Similarly, in some embodiments, the network service may be applied to the data messages of all network slices in the virtual service network after different services have been applied to a given slice.

The above example of a network that uses slice selection and virtual service networks is a telecommunications provider network (both for hierarchical slice selection and single-level slice selection), but this virtual service network is of another type. It should be understood that networks can also be configured. For example, in the case of a network within a data center or across a large number of data centers, virtualization software (eg, within a host computer hosting a VM or other endpoint in the network) can be a source of data messages sent by the network endpoint. It can be configured to implement slice selection. A type of network that spans multiple data centers is the use of one or more sets of public clouds to connect enterprise data centers (eg, a main office and one or more remote branch offices). In such networks, network slice network services may be implemented in enterprise data centers as well as in public clouds.

The above overview is intended to serve as a brief introduction to some embodiments of the invention. This is not meant to be an introduction or summary of the subject matter of all inventions disclosed herein. The detailed description below and the drawings referenced in the detailed description will further illustrate the embodiments described in the overview as well as other embodiments. Therefore, a thorough review of the outline, detailed description, and drawings is required to understand all the embodiments described herein. Moreover, the claimed subject matter should not be limited by abstracts, detailed descriptions, and exemplary details in the drawings, rather the claimed subject matter does not deviate from the spirit of the subject matter and is in other particular form. It should be defined by the appended claims as it can be embodied in.

The novel features of the invention are described in the appended claims. However, for illustration purposes, some embodiments of the invention are described in the drawings below.

The figure which conceptually shows the virtual service network (VSN) which has a large number of network slice selectors.

A diagram conceptually showing the distribution of services for a single network slice across many data centers.

FIG. 6 is a diagram conceptually showing a route through VSN for a data message received in an edge cloud and assigned to a network slice shown in FIG. 2 by the edge cloud slice selector according to some embodiments.

A diagram conceptually showing a hierarchical set of controllers.

A flow diagram showing a dialogue between an endpoint device, a slice selector, an output gateway, and a network domain for sending a message for a connection in which the slice selector acts as a soft termination proxy.

The figure which conceptually shows the http get message.

The figure which shows the entry of the connection mapping table of some embodiments stored by a slice selector.

A diagram conceptually showing a data message.

FIG. 6 conceptually illustrates the processing of some embodiments for assigning a connection between an endpoint device and a destination network domain to a network slice of a VSN by terminating the connection handshake.

A flow diagram illustrating a dialogue between an endpoint device, a slice selector, an output gateway, and a network domain for the slice selector and output gateway to send a message for a connection to perform handshake playback.

Connection A diagram conceptually illustrating the processing of some embodiments for assigning a connection between an endpoint device and a destination network domain to a network slice of a VSN by playing a connection handshake message.

The figure which conceptually shows the processing of some embodiments for an output gateway in the case of handshake reproduction.

It is a diagram conceptually showing a mobile device moving from a first slice selector area to a second slice selector area, and the second slice selector transfers data traffic from the mobile device to the first slice selector.

The figure which conceptually shows the example of the 1st slice selector which pushes a state position information to a central controller, and the 2nd slice selector which fetches a state position information from a central controller.

The figure which conceptually shows the example which the 1st slice selector pushes the state position information to the 2nd slice selector.

It is a diagram conceptually showing a mobile device moving from a first slice selector area to a second slice selector area, and this second slice selector receives a slice mapping state for connection and displays the slice mapping state. Use to forward data traffic for connections.

The figure which conceptually shows the example of the 1st slice selector which pushes a slice mapping state to a central controller, and the 2nd slice selector which fetches a slice mapping state from a central controller.

,as well as An example of a first slice selector that pushes state position information to the controller and a second slice selector that retrieves state position information and uses that state position information to retrieve the slice mapping state from the first slice selector. The figure which shows conceptually.

The figure which conceptually shows the example which the 1st slice selector pushes the slice mapping state to the 2nd slice selector.

FIG. 6 conceptually illustrates a first slice selector associated with a first geographic area that pushes a slice mapping state to all of its adjacent geographic areas, according to some embodiments.

A mobile device moving within a first geographic area and a slice selector for that area that pushes the slice mapping state for connections initiated by that mobile device only to the adjacent area where the device is moving. A diagram conceptually showing.

The figure which shows the example of a hierarchical VSS conceptually.

A diagram conceptually showing the distribution of providers and tenant slice selectors (as well as network services for network slices) across multiple data centers.

The figure which conceptually shows the branch control of a provider infrastructure and a plurality of tenant VSNs.

The figure which conceptually shows the example of the hierarchical VSN after the network service is inserted between slice selectors and / or the service of different slices is completed.

The figure conceptually showing an example of a hierarchical set of VSNs with three levels of slicing.

The figure which conceptually shows the implementation of VSN in a data center.

A diagram conceptually showing the implementation of VSN to handle WAN communication between two private enterprise data centers over a public cloud.

A diagram conceptually showing that VSN may be implemented to handle communication between guest VMs in a public cloud and public internet traffic within a public cloud or set of public clouds.

The figure which conceptually shows the electronic system in which some embodiments of this invention are implemented.

The following detailed description of the invention describes and describes numerous details, examples, and embodiments of the invention. However, it will be apparent to those skilled in the art that the invention is not limited to the embodiments described and that the invention can be practiced without some of the specific details and examples described. Will.

Some embodiments provide a method for establishing a virtual service network across a set of data centers. The set of data centers in which a virtual service network is established includes, for example, a software-defined wide area network (SD-WAN) that spans one or more public clouds, public clouds, and private clouds, and a telecommunications service provider access network (eg, wireless access). It can include a combination of networks, edge clouds, and core clouds), or other types of data centers. The virtual service network of some embodiments comprises a large number of network slices, each of which provides a different network service, in the data message assigned to the network slice.

In some embodiments, when a device (eg, a mobile endpoint device in a telecommunications context) sends a data message over such a network, the network slice selector first processes the data message. This network slice selector assigns a data message to one of the network slices of the virtual service network and processes the service chaining operation so that the data message is processed by the correct set of network services for the assigned slice. In different embodiments, this network slice selector is a virtual machine (VM), a containerized function, a software transfer element running within a VM (eg, a flow-based transfer element), within a container or within the virtualization software of a host computer. A set of modules that run on the host computer, a set of modules that run outside the transfer element within the virtualization software of the host computer (eg, between the VM and the port of the transfer element), a hardware transfer element (eg, a programmable switch), Alternatively, it may be implemented by other implementations.

In some cases, many network slice selectors are configured to implement virtual service networks. In the example of a telecommunications service provider, some embodiments configure a network slice selector for each cellular tower, base station, or other aspect of the access network. Some embodiments of telecommunications service provider access networks include an edge cloud for each cellular tower, each of which constitutes at least one network slice selector. In another example (for example, for SD-WAN traffic that is completely contained within a set of connected data centers), the distributed network slice selector can select network slices for data messages sent from the VM. It is configured to originate on the same host computer (outside the source VM) as the source of the data message or on a specified device (eg, a specific nearby switch or router, a dedicated VM or container).

FIG. 1 is a diagram conceptually showing a virtual service network (VSN) 100 having a large number of network slice selectors. In this case, the VSS100 performs network services for data messages for devices accessing the Internet (eg, within the telecommunications service provider access network). Which network service this VSS performs for a given data message depends on the slice to which the data message is assigned. As further described below, a given network slice of network services may be implemented in a single data center or set of data centers. In a particular slice, some network services may be distributed across many edge clouds, while later network services are implemented in a central public data center.

As shown in the figure, the virtual service network 100 includes a large number of (N) network slices 105-115. Each of these network slices represents a network path (ie, an ordered set of network services running on the data messages assigned to the slice). These network services include firewalls, load balancers, network address translation, weighing functions (eg, for billing purposes), GBP gateways, wireless access network (RAN) functions (eg, distributed and centralized unit functions), evolutionary packets. It can include core (EPC) functions (eg, subscriber management servers, service gateways, packet data network gateways, mobility management entities), or other types of network functions.

In different embodiments, the network slices of the virtual service network may serve different purposes. Some embodiments are based on the source device (eg, using information that identifies the source network address or device type) or based on the subscriber information (eg, authentication, authorization, and accounting system or). Slice the network (by connecting to the policy system with an interface), while other embodiments slice the network based on the type of traffic (eg, by performing deep packet inspection). A defined quality of service (SLA) can be set for each network slice. For example, network slices for self-driving cars may have very short latency requirements, network slices for streaming video may have high bandwidth requirements, and IoT slices may have a single device. May have less stringent bandwidth or latency requirements, but have greater potential for connectivity.

These network services may be implemented as virtualized network functions (VNFs), physical network functions (PNFs), and / or cloud network functions (CNFs) in different embodiments. VNF is a network service implemented in a virtualized data calculation node such as a virtual machine. This allows, for example, the same network service configuration for a particular slice to be implemented in multiple edge clouds (eg, with multiple slice selectors). CNF is a network service implemented on cloud-native data compute nodes, such as certain types of containers. Finally, PNF is a network service implemented by a physical device (eg, a particular firewall or load balancer device). In general, PNFs are more effectively placed in centralized data centers rather than edge clouds, so there is no need to duplicate the same physical device for each edge cloud.

In this example, the first network slice 105 comprises two VNFAs and Bs, as well as a PNFC. The second network slice 110 has three VNFBs, D, and E and is completely virtual. The final network slice 115 contains the same three network services as slice 105 (VNFA and B, and PNFC), followed by CNFF. In some embodiments, the same VM can implement VNFs for multiple different network slices. In this example, one VM may implement the same VNFB for all three network slices 105-115 shown. If this VNF is located in the edge cloud, a single VM may be instantiated in each edge cloud (eg, for each slice selector). However, in other embodiments, a separate VNF (eg, a separate VM or other data computing node) is instantiated for each VNF, even if the VNF settings are the same for multiple slices. Therefore, in this example, three different VNFs are instantiated for the VNFB for each of slices 105-115. Therefore, if this VNF is located within the edge cloud, each edge cloud will have three different VMs for the VNFC.

Due to the way the device accesses the network 100, some embodiments have a large number of slice selectors 120-130. The device may access the telecommunications service provider network via a base station (eg, cell tower), wireless access point, wired relay (eg, in the home), or other means. For a provider network, slice selectors of some embodiments are implemented near the device so that slice selection can occur before data traffic traverses most of the network. For example, in the case of a 5G wireless network with multi-access edge computation, some embodiments configure slice selectors for each distributed unit (DU). Another embodiment configures a slice selector for each centralized unit (CU) that receives traffic from multiple DUs. In this case, each slice selector has an associated geographic range (ie, the geographic range of the associated DU or CU of that slice selector).

In the situation shown in FIG. 1, each slice selector 120-130 is configured to perform the same slice selection function (ie, act as a single logical slice selector) in some embodiments. That is, each slice selector 1 to K can assign a data message to any of slices 1 to N, and the network slice allocation is the same regardless of which of slice selectors 120 to 130 processes the data message. Will be. In other embodiments, slices are only accessible in certain geographic areas. For example, a network slice associated with a particular application may be available in certain cases in a particular city or other geographic area.

This example shows that many devices can connect to a given slice selector at any particular time. In this example, the smart refrigerator and laptop are attached to the first slice selector 120, the tablet device is attached to the second slice selector 125, and the self-driving car and mobile phone are attached to the last slice selector 130. .. In different embodiments, the network slice selector is a software transfer element (eg, a flow-based transfer element) that operates within a virtual machine (VM), VM or host computer virtualization software, within the host computer virtualization software. A set of modules running outside the transfer element (eg, between the VM and the port of the transfer element), a physical device (eg, a dedicated hardware transfer element, a physical host computer), a container application (eg, running a network service mesh). It may be implemented by a computer system) or other implementation.

FIG. 2 is a diagram conceptually showing the distribution of services for a single network slice 200 across multiple data centers 205-215. As shown in this example, the network slice 200 includes four network services (VNFA-D) applied to specific data traffic from the device 220 assigned to the network slice 200. The first VNFA is implemented in the edge clouds 205 and 207, the second and third VNFBs and Cs are implemented in the core cloud 210, and the fourth VNFD is implemented in the public cloud 215. In networks that use multi-access edge computation (eg, 5G networks), slice selector 225 and arbitrary network services implemented within the edge cloud are instantiated within each edge cloud. Thus, both Edge Cloud 205 and Edge Cloud 207 are each with any network service implemented on the edge for any other VSN implemented across any other slice or network of the same VSN. Similarly) it has an instance of slice selector 225 and VNFA. Further, although not shown, within each edge cloud, some embodiments handle multiple slice selectors (eg, active slice selector and standby slice selector, or all incoming traffic) for high availability reasons. Run a number of active slice selectors) that share the load.

In some embodiments, traffic from the device 220 first passes through a radio access network (RAN) not shown in this figure. In some embodiments, network slicing is performed before the RAN (ie, on the device side of the RAN), but in this example the network slicing is performed after the RAN. The data traffic then reaches the slice selector 225 (edge cloud 205) that analyzes the traffic and allocates the traffic to the network slice 200.

When the slice selector 225 assigns a data message to a network slice 200, in some embodiments, to ensure that the data message traverses the network service of the assigned slice (ie, VNFA-D) in the correct order. This slice selector 225 plays a role of executing service chaining. In some embodiments, the slice selector 225 sends a data message to a first network service (ie, a VM that implements VNFA in the same edge cloud 205) and maintains contextual information for that data message. When the VNFA completes processing the data message, the VNF returns the data message to the slice selector 225. If additional network services for slices are also implemented on Edge Cloud 225 (not for Slice 200), then slice selector 225 is maintained, for example to send data messages to the next network service. Will use the context information.

In this case, the second network service VNFB is implemented in the core cloud 210. In some embodiments, the network slice selector 225 sends a data message to the service chain module of the core cloud (eg, via a wide area network (WAN) gateway interconnecting the clouds 205-215). In some embodiments, if a complete network slice is implemented across many data centers, a similar service chaining module will be in its own data center (eg, both in core cloud 210 and public cloud 215). ) Operates in each data center to handle service chaining for slices. These service chaining modules may be implemented in the same manner as the network slice selectors in some embodiments (eg, as VMs, as VMs or as transfer elements in virtualization software, as containers). Upon completion of the final network service, in some embodiments, the egress gateway 230 sends a data message over the Internet to its destination.

FIG. 3 conceptually shows a route through the VSSon for a data message received by the edge cloud 205 and assigned to the slice 200 by the slice selector 225 in the edge cloud 205 according to some embodiments. As indicated by the circled 1, the endpoint device 220 sends a data message to the telecommunications provider access network, where the data message is processed by the slice selector 225 in the edge cloud 205. In some embodiments, if these parts of the access network are not part of a virtual service network (ie, if the slice selector processes the data message after the RAN and / or EPC), the data message will first be RAN and / or EPC. / Or processed by EPC. The slice selector 225 in the edge cloud 205 assigns a data message to the slice 200 (eg, based on deep packet inspection, L2-L4 headers, or other factors) and the VNFA is (i) the first for this slice. Identify that it is a network service and is also located within (ii) Edge Cloud 205. Therefore, the slice selector 225 sends the data message to the VNFA (circled 2), the VNFA processes the data message, and the data message is represented by the slice selector 225 (circled 3). ).

The slice selector 225 then identifies that the next network service of the selected slice 200 is located within the core cloud 210, and thus, via the WAN gateway (not shown for simplicity), within the core cloud 210. Sends a data message to the service chaining module 310 (indicated by the circled 4) running on. In some embodiments, the service chaining module 310 ensures that reverse traffic is sent to the slice selector 225 in the correct edge cloud 205 (ie, in contrast to the edge cloud 207) of these data messages. Use a learning action (eg MAC learning) to store the source.

The service chaining module 310 in the core cloud 210 receives the data message as it enters the core cloud 210 (after processing by the WAN gateway) and provides it with the data message (eg, by the slice selector 310). Identify slices for data messages (based on contextual information, stored slice mappings for connections, or other factors). The service chaining module 310 provides data messages to network services within the core cloud 210, in this case to VNFB, and then to VNFC. As shown in the figure, this service chaining module sends a data message to the VNFB (indicated by a circled 5) and receives a data message from the VNFB (indicated by a circled 6). , Send a message to the VNFC (indicated by the circled 7) and receive a data message from the VNFC (indicated by the circled 8).

After the data message has been processed by the VNFC, the data message is sent by the service chaining module 310 to another service chaining module 315 (indicated by a circled 9) in the public cloud 215 (eg, the core cloud). It is transmitted (via a WAN gateway that interconnects 210 and the public cloud 215). In some embodiments, the service chaining module 310 operates similarly to the service chaining module 310 in the core cloud 210, using a learning mechanism to store information for processing return traffic. This service chaining module 310 in the public cloud 215 sends a data message to the VNFD (indicated by a circled 10), which executes its network service and sends the data message to the service chaining module 315. Return to.

Finally, the service chaining module 315 determines that the network slicing process for the data message is complete and sends the data message to the output gateway 230 which sends the data message to its destination via the Internet. This example shows connectivity between an endpoint device and an internet domain, but for other virtual service networks, the destination may instead be located in the public cloud and is connected via the WAN. It may be located in another data center. The egress gateway 230 of some embodiments goes to the network slice 200 so that reverse traffic (ie, data messages from the public internet domain) is assigned to the same slice (with network functions running backwards). Stores information that maps the connection of. In another embodiment, the egress gateway 230 allocates data messages in a non-stateful manner (eg, using the destination network address of the data messages). This egress gateway may be implemented in some embodiments with a service chaining module (or with the original slice selector for a virtual service network that spans only a single data center).

Slice selectors, network services (VNF, CNF, PNF, etc.) and various forwarding elements that handle the transmission of data messages between these entities (software forwarding elements that tunnel data messages between host machines, WAN gateways, etc.) It also needs to be set. In some embodiments, a centralized controller allows a user (eg, a network administrator) to provide settings for the entire VSN, and then the controller hierarchy is one or more to implement this VSN. Configure various entities in your data center.

FIG. 4 conceptually shows such a hierarchical set of controllers 400. As shown in this figure, the high level VSN manager 405 receives VSN settings from a network administrator (eg, data center tenant, telecommunications provider). The VSS Manager 405 of some embodiments provides one or more interfaces for the administrator to provide this data (eg, a graphical user interface, a command line interface, a set of REST APIs). In some embodiments, the configuration data for the VSN is a different slice of the VSS, a slice selector configuration (ie, a characteristic for assigning a data message to each of the different slices), and a network service configuration for each network service on the slice. , How each network service will be implemented (eg, as VNF, CNF, or PNF), the location for each network service (eg, edge cloud, core cloud, or other data center), and / Or specify other data.

The VSN controller 410 coordinates centralized storage with the distribution of this information to other controllers in the tier. In some embodiments, a suite of controllers 415 in each data center receives VSN configuration data from the VSN controller 410 and configures the entities in the data center to implement VSN. In some embodiments, each data center has its own suite of these lower level controllers. These controller suites can be the same set of controllers in each data center (eg, a suite of controllers provided by a single company), or different sets of controllers (eg, private edge clouds and private edge clouds compared to public clouds). Can be a different set of controllers for the core cloud).

The controller suite 415 in the first data center 420 includes a software defined networking (SDN) controller 425, a compute controller 430, and a network controller 435. It should be understood that different embodiments may include additional controllers and the functionality of multiple controllers may be combined into a single controller. For example, some embodiments include an orchestrator acting as a layer between the VSN controller 410 and other controllers in the controller suite 415 (eg, an openstack controller), or the SDN controller 425 features a network controller 435. Combined with the features of. In addition, some embodiments include a storage controller for managing VSN-related storage in the data center.

The SDN controller 425 constitutes a slice selector 440. In this example, the single slice selector 440 operates in the data center 420 (eg, in the virtualization software of the host computer 445, as a VM or in a VM on the host computer 445), but in other embodiments the slice selector 440. Please understand that is implemented in a distributed manner within the data center. In some embodiments, the SDN controller 425 correctly assigns data messages to the flow, ensures that the data messages are sent to the correct network services in the correct order within the data center 420, and to perform service chain operations. Configure a slice selector with flow entries or other configuration data. In addition, in a data center that hosts network services but not slice selectors (eg, core cloud, public and / or private cloud for the example of a telecommunications provider), some embodiments of the SDN controller are output gateways. Configure a service chaining module as well (may perform slice selection for reverse data messages).

The compute controller 430 is responsible for instantiating and configuring the VNF (eg, as a VM in this example). In some embodiments, the VM is instantiated on the host computer 450 by a compute controller 430 that configures the VM to implement the specified network service. In some embodiments, the compute controller 430 uses a template for a firewall, load balancer, or other network service to instantiate the VM, and then, as specified by the network administrator, the network service. Provides the VM with specific configuration data for. In addition, the compute controller 430 of some embodiments also serves to constitute any CNF and / or PNF mounted in the data center 420.

The network controller 435 implements a transfer element (eg, a software transfer element 455 or another type of transfer element such as a programmable hardware transfer element) to implement network connectivity between the network service and the slice selector 440. Configure. This configuration is similar to performing encapsulation of data messages to tunnel data messages between entities in the data center, for example, a logical transfer connecting various entities in a slice (slice selectors and network services). Includes plane-following transfers. In addition to the software transfer element 455 shown on the host computer 450 (eg, a virtual switch running in virtualization software), in some embodiments, a similar software transfer element is in the host computer 445 with the slice selector 440. Transfer data messages between and perform encapsulation / decapsulation. In some embodiments (eg, when the slice selector is implemented within the software transfer element or between the software transfer element and the VM in a distributed manner), the network controller 435 also receives the slice selector configuration. Configure the appropriate network entity to implement the slice selector.

In addition to these controllers in controller suite 415, some embodiments also include one or more WAN SDN controllers 460. The WAN SDN controller 460 serves to interconnect the data centers as needed, thereby configuring the WAN gateway 465 within each data center. These WAN gateways may interconnect data centers using MPLS, SD-WAN, or other techniques for inter-data center communication. In many cases, not all data centers require direct communication. For example, in the telecommunications example, edge clouds may not need to communicate with each other because data traffic is sent between the edge cloud and the core cloud rather than between the edge clouds.

In some embodiments, rather than communicating directly with the controller in controller suite 415 and the WAN SDN controller 460, the VSN controller 410 provides data to the agents in each data center and the agents of the WAN SDN controller 460. These agents serve to transform the data from the VSN controller 410 (which may be provided in a uniform format for all controllers) into data that can be used by the various controller suites. In some embodiments, the VSS controller 410 pushes policy-formatted data to a local agent, which sends this data to various SDN controllers, compute controllers, and / or network controllers according to their policies. Convert to data that tells you to set the components. This allows the VSN controller 410 to use a single format to communicate with a variety of different types of controller suites (eg, different public cloud controllers, enterprise data center controller suites). Similarly, for the WAN SDN controller 460, this agent translates the policy into a WAN configuration instruction.

As mentioned above, the network slice selector may assign data messages to slices using different techniques in different embodiments. Slice selection may be based on packet header information including Layer 2 to Layer 4 (L2 to L4) headers and / or traffic (eg, based on data in Layer 5 to Layer 7 (L5 to L7) headers). You may also perform a deep packet inspection (to classify). For example, slice selection may simply be based on the source device by using the source network layer (eg IP) address, or based on the traffic type / or destination network domain by looking at the higher layer (L5-L7) header. You may.

In addition, in some embodiments, the network slice selector integrates with other control plane components to collect additional information about the connection (eg, regarding user sessions, device types, or other data). Use the information as part of the slice selection process (eg, use only this collected information, or combine this information with L2-L4 and / or L5-L7 packet header data). Examples of such control plane components are authentication, authorization, and accounting (AAA) protocols (eg, remote authentication dial-in user service (RADIUS)), policy control and billing rule functions (PCRF), or devices and / or. Includes other such components that can provide user data to the slice selector.

In some embodiments, the network slice selector maintains a state for mapping connections to network slices so that it is not necessary to perform deep packet inspection for each data message on the connection. Also, in some connections, only certain data messages contain the L5-L7 header information needed to perform slice selection.

When performing network slice selection using deep packet inspection, in certain cases the initial data message for the connection may not contain the L5-L7 header information needed for the slice selector to correctly identify the slice. be. For example, endpoint devices (eg mobile devices such as smartphones or tablets, laptops or desktop computers, IoT devices, self-driving cars, smart cameras belonging to security systems) and network domains (eg www.netflex.com, www. Connections with (web domains such as .google.com) often begin with a set of connection initiation messages such as the TCP handshake. When the handshake is complete, the device sends an http get message containing the network domain and the like. Subsequent data messages sent between the device and the network domain may not contain such information.

(I) The connection between the client (eg, the endpoint device) and the server (eg, the web domain) is successfully initiated, and (ii) the network slice cannot be selected based on the first message. Different embodiments use different techniques to identify the correct network slice for a connection, even while ensuring that all of the messages are sent on the correct network slice. In some embodiments, the network slice selector acts as a proxy to terminate the connection start message without sending these messages to the intended destination across the virtual service network. In another embodiment, the slice selector first passes the connection start message to the default network slice, then plays the message on the correct network slice for the connection after the network slice is selected.

FIG. 5 shows a dialogue between an endpoint device 505, a slice selector 510, an output gateway 515, and a network domain 520 (eg, a server) for sending a message for a connection in which the slice selector acts as a soft termination proxy. The flow diagram shown is shown. Although this example shows a connection initiated by the endpoint device 505 using a TCP 3-way handshake, the slice selector 510 of some embodiments provides other types of connection initiation messaging (eg, TLS handshake, UDP). It is also possible to perform a similar termination for the above QUIC connection). In this example, a set of private and public clouds (eg, connected by SD-WAN or MPLS) hosts a virtual service network that is sliced (between slice selector 510 and output gateway 515), but outputs. The gateway connects this VSS to the Internet (and network domain 520).

As shown in the figure, the endpoint device 505 (eg, a smartphone, self-driving car, IoT device) first has a TCP SYN message directed to the network domain 520 (eg, using the destination IP address of the network domain). To send. Such messages may not have valid header information (eg, L5-L7 header information) for the slice selector 510 used to assign the initiated connection to one of the network slices. As mentioned above, in some embodiments, the network slice is application-specified and / or device-specified and is configured to meet latency, bandwidth, jitter, or other requirements of a different application or device type. .. Streaming video, self-driving vehicles, IoT devices, and other devices / applications all have different requirements that can be met by different network slices with different SLAs.

Instead of forwarding the connection start message (SYN message) to the Internet (and ultimately the network domain 520) over the network, the slice selector 510 executes the soft termination of the connection start. The slice selector 510 stores a record of the SYN message (ie, the message itself or enough data to recreate the message) and responds to the endpoint device 505 with a SYN-ACK message. This SYN-ACK message is formatted as if the network domain received the SYN message and responded with the SYN-ACK message. As a result, the endpoint device 505 sends an ACK message and operates as if the connection with the network domain 520 was set up. The slice selector 505 stores data for all connection start messages for subsequent use.

Based on the appearance of the connection being properly set up, the endpoint device 505 then sends a data request message (or other message). This message typically contains L7 information (or other higher layer information such as TLS server name display) that the slice selector needs to properly assign connections to network slices. For example, FIG. 6 conceptually shows the http get message 600. Such messages include an L3 header containing source, destination IP addresses, and transport layer protocol fields (such as the L2 header is not shown for brevity) and an L4 header containing source and destination port numbers. And have. In addition, part of the L7 header of message 600 includes the domain name in the http get command (in this case, www.exampledomain.com).

The slice selector 510 performs deep packet inspection on the data message (data request in this case) to identify the correct network slice for the connection. In some embodiments, this involves inspecting the L5-L7 header (eg, the http data shown in FIG. 6). In addition, the slice selector 510 stores a state that maps the connection to the selected network slice for use in processing subsequent data messages. In some embodiments, the slice selector stores this mapping as a connection 5 taple mapped to a slice identifier (ie, source and destination network addresses, transport layer protocol, source and destination transport layer ports). In addition, some embodiments identify related connections that may require allocation to the same network slice and also store the state for these connections. As an example, a streaming video session will include multiple separate TCP connections (for audio and for different video resolutions) in some embodiments.

FIG. 7 shows entries in the connection mapping table 700 of some embodiments stored by the slice selector. As shown, this table contains 5 tuple entries mapped to slice identifiers. In some embodiments, these entries are generated as flow entries for a flow-based transfer element that implements a slice selector. For each flow entry, the matching condition is 5 tuples of connections and the action is to assign a data message to the specified slice. As shown in Table 700, a single source device (eg, IP1) has multiple ongoing connections (eg, for simultaneous web browsing connections and audio streams) assigned to different network slices. Can have. In addition, the slice selector can assign connections for different devices to the same slice (eg, multiple self-driving cars, two devices streaming video at the same time). In some embodiments, additional information other than 5 tuples is included in the matching condition (eg, VLAN information or other virtual network identifier).

The slice selector 510 then exchanges the connection start message with the destination network domain, this time acting as a device rather than the destination domain. These connection start messages are sent through the selected network slice so that all network services on the selected network slice process the connection start message. As shown, the slice selector 510 sends a SYN message to the network domain 520 via the selected network slice and output gateway 515. The egress gateway 515 also stores the connection to the slice mapping state so that, in some embodiments, return traffic from the network domain 520 can also be sent through the selected network slice. The network domain 520 returns a SYN-ACK message sent by the egress gateway 515 over the selected network slice. This message does not reach the endpoint device 510 because the slice selector 510 recognizes the SYN-ACK message and returns the ACK message to the network domain (via the selected network slice). At this point, the entire 3-way handshake has been completely exchanged and the connection between the endpoint device 505 and the network domain 520 is fully initiated.

The slice selector 510 can then send a data request (ie, the first data message on which deep packet inspection was performed) to the network domain 520 via the selected network slice. Subsequent data messages for the connection are also exchanged between the endpoint device 505 and the network domain 520, and this traffic is selected without performing deep packet inspection (similar to the egress gateway 515 for return traffic). The state in which the slice selector 510 is stored is used to allocate to the network slice. This avoids performing deep packet inspections on all data messages on the connection, as the L5 to L7 headers on all data messages belonging to this connection may not have the required information. , May not be possible. FIG. 8 conceptually shows the data message 700. The data message 800 has the same L3 and L4 headers as those of the data message 600, but the L7 header does not contain the same network domain information. Instead, the L7 portion of this data message contains payload data (eg, video or audio data) that is useless to the slice selectors of some embodiments.

FIG. 9 conceptually illustrates processing 900 of some embodiments for assigning a connection between an endpoint device and a destination network domain to a network slice of a VSN by terminating the connection handshake. In some embodiments, processing 900 is performed by a slice selector such as the slice selector 510 shown in FIG. Similar processing can be performed by slice selectors in other types of networks (eg, intra-datacenter or inter-datacenter communications that do not reach the public Internet).

As shown, processing 900 starts (at 905) by receiving a connection start message from an endpoint device (eg, a telephone, self-driving car, tablet, IoT device). This message may be a TCP SYN message, the first message of the TLS handshake, a QUIC protocol message on UDP, or another connection start message. This process then performs the rest of the connection handshake with the endpoint device (at 910) while storing a copy of the handshake message (or the data needed to recreate the handshake message). For example, in the TCP 3-way handshake example, the slice selector exchanges SYN-ACK and ACK messages with an endpoint device that acts as a server in this handshake. For an optimized single message connection initiation (eg, QUIC protocol), this behavior is not necessary and the slice selector can be used for either the initial message or subsequent messages without performing a handshake with the endpoint device. And perform deep packet inspection.

When this connection is set up on the endpoint device, process 900 receives an initial data message (at 915) for the connection from the endpoint device. This may be a http get message as shown in FIG. 6, or another data message. Often, this data message contains useful information for performing slice selection, so process 900 analyzes the data message to select a network slice for the connection (at 920). This deep packet inspection looks at the name of the domain to be contacted, the particular L7 protocol in use (for example, to identify the type of application initiating the connection), or other information in the upper layer headers of the data message. You may. Process 900 also stores (at 925) the state of mapping the connection to the selected network slice (eg, using 5 tuples). This information may be stored in a connection mapping table as shown in FIG. 7 (eg, by creating a new flow entry for the connection).

Processing 900 then sends a connection handshake message (at 930) to the destination network domain via the selected network slice to establish a connection between the endpoint device and the network domain. In this message exchange, the slice selector acts as an endpoint device rather than a network domain, allowing the network domain server to set up a connection. This also allows the network service of the selected network slice to process the connection start message so that these services are prepared for subsequent data messages (eg, firewalls are often in the manuscript). Set to reject data messages for TCP connections that are not processing the 3-way handshake message).

Process 900 also sends an initial data message (at 935) to the destination network domain via the selected network slice. In addition, process 900 uses the stored connection mapping state to receive and transmit (at 940) subsequent data messages for the connection via the selected network slice. This stored state allows the slice selector to perform a deep packet inspection (which, as mentioned above, may not be possible for many subsequent data messages), for connections from endpoint devices. Each data message can be assigned to a selected network slice.

As mentioned earlier, rather than terminating the connection start message with the slice selector, some embodiments first pass these messages in the direction of the default network slice, and then after the network slice is selected. , Play the message through the correct network slice for the connection. FIG. 10 shows the endpoint device 1005, the slice selector 1010, the output gateway 1015, and the network domain 1020 (eg, a server) for the slice selector and the output gateway to send a message for a connection to perform handshake playback. The flow diagram which shows the dialogue between is illustrated. Although this example shows a connection initiated by the endpoint device 1005 using a TCP 3-way handshake, the slice selector 1010 of some embodiments provides other types of connection initiation messaging (eg, TLS handshake, UDP). It is also possible to perform the same reproduction for the above QUIC connection). In this example, a set of private and public clouds (eg, connected by SD-WAN or MPLS) hosts a virtual service network that is sliced (between slice selector 1010 and output gateway 1015), but outputs. Gateway 1015 connects this VSS to the Internet (and network domain 1020).

As shown in the figure, the endpoint device 1005 (eg, a smartphone, self-driving car, IoT device) first has a TCP SYN message directed to the network domain 1020 (eg, using the destination IP address of the network domain). To send. Such messages do not have valid L5-L7 header information for the slice selector 1010 used to assign the initiated connection to one of the network slices, as described above. In this case, instead of exiting the handshake and responding to the endpoint device, the slice selector 1010 assigns a TCP SYN message to the default network slice and over this network to the network domain 1020 (via the output gateway 1015). Send a message. In addition, both the slice selector 1010 and the output gateway 1015 store a record of the SYN message (ie, the message itself, or enough data to recreate the message). The network domain 1020 responds with a SYN-ACK message that the output gateway 1015 assigns to the default slice, and the slice selector treats the ACK message from the endpoint device 1005 as well.

By setting up the connection between the endpoint device 1005 and the network domain 1020, the endpoint device 1005 then sends a data request message (or other message). This message usually has L5 to L7 information required by the slice selector to properly assign a connection to a usage-specified or device-specified network slice. FIG. 6 above conceptually shows an example of such a message 600.

The slice selector 1010 performs deep packet inspection on the data message (data request in this case) to identify the correct network slice for the connection. In some embodiments, this involves inspecting the L5-L7 header (eg, the http data shown in FIG. 6). In addition, the slice selector 1010 stores a state that maps the connection to the selected network slice for use in processing subsequent data messages. In some embodiments, the slice selector makes this mapping a connection 5 taple (ie, source and destination network address, transport layer protocol, source and destination) mapped to the slice identifier, as shown in FIG. 7 above. Store as a transport layer port). In addition, some embodiments identify related connections that may require allocation to the same network slice and also store the state for these connections. As an example, a streaming video session will include multiple separate TCP connections (for audio and for different video resolutions) in some embodiments.

However, before sending the data message to the selected slice, the slice selector plays the connection start message through the selected network slice. The various network services in the selected slice have not yet processed the connection start message, so if these data messages are sent without playing the handshake, they may reject the connection data message. .. Therefore, as shown, the slice selector 1010 uses its stored data to exchange TCP 3-way handshake messages with the output gateway 515, which also stores information for these messages. The SYN, SYN-ACK, and ACK messages are transmitted between the slice selector 1010 and the output gateway 1015 via the selected network slice. These messages are not sent outside the virtual service network (ie, to either the endpoint device or the public internet) because the connection between the endpoint device 1005 and the network domain 1020 has already been established. The egress gateway 1015 also stores the connection to the slice mapping state so that, in some embodiments, return traffic from the network domain 1020 can be sent over the selected network slice.

If these messages were played, the network service of the selected slice would have processed the connection start message and would be prepared for subsequent data message traffic between the endpoint device and the destination network domain. As shown, the slice selector 1010 sends a data request (ie, the first data message on which deep packet inspection was performed) to the network domain 1020 via the selected network slice. Subsequent data messages for the connection are also exchanged between the endpoint device 1005 and the network domain 1020, and this traffic is selected without performing deep packet inspection (similar to the egress gateway 1015 for return traffic). The state in which the slice selector 1010 is stored is used to allocate to the network slice. This avoids performing deep packet inspections on all data messages on the connection, as the L5 to L7 headers on all data messages belonging to this connection may not have the required information. It may not be possible (as shown by data message 800 in FIG. 8).

FIG. 11 conceptually illustrates process 1100 of some embodiments for assigning a connection between an endpoint device and a destination network domain to a network slice of a VSN by playing a connection handshake message. In some embodiments, the process 1100 is performed by a slice selector such as the slice selector 1010 shown in FIG. Similar processing can be performed by slice selectors in other types of networks (eg, intra-datacenter or inter-datacenter communications that do not reach the public Internet).

As shown, processing 1100 begins by receiving a connection start message (at 1105) from an endpoint device (eg, a telephone, self-driving car, tablet, IoT device). This message may be a TCP SYN message, the first message of the TLS handshake, a QUIC protocol message on UDP, or another connection start message. This process then sends the connection start message (ie, the initial message and any subsequent messages) to the destination network domain via the default network slice in order to establish a connection between the endpoint device and the network domain (ie, the initial message and any subsequent messages). Send (at 1110). Similarly, for connection start handshake return messages sent from the endpoint area, the slice selector receives these messages via the default slice and sends them to the endpoint device.

During the connection initiation handshake, this process stores data (at 1115) about the handshake message (ie, the data needed to copy the message or recreate the message). For example, in the case of the TCP 3-way handshake, the slice selector stores data to send SYN and ACK messages. This behavior is not required for single message connection initiation (eg, QUIC protocol), and some embodiments of the slice selector perform deep packet inspection on the initial message to avoid the need to use the default slice. To execute.

When this connection is set up on the endpoint device, process 1100 receives an initial data message for the connection from the endpoint device (at 1120). This may be a http get message as shown in FIG. 6, or another data message. Often, this data message contains useful information for performing slice selection, so process 1100 analyzes the data message to select a network slice for connection (at 1125). This deep packet inspection looks at the name of the domain to be contacted, the particular L7 protocol in use (for example, to identify the type of application initiating the connection), or other information in the upper layer headers of the data message. You may. Process 1100 also stores (at 1130) the state of mapping the connection to the selected network slice (eg, using 5 tuples). This information may be stored in a connection mapping table as shown in FIG. 7 (eg, by creating a new flow entry for the connection).

Processing 1100 then replays the connection handshake message with the output gateway (at 1135) via the selected network slice. In this message exchange, the slice selector acts as a client (ie, an endpoint device) and the output gateway acts as a server (ie, a network domain). In the example of the TCP 3-way handshake, the slice selector sends a SYN message, receives a SYN-ACK message from the output gateway, and sends the ACK message. This serves the purpose of allowing the network service of the selected network slice to process the connection start message so that these services are prepared for subsequent data messages (eg, firewalls often). Set to reject data messages for TCP connections that are not processing the original 3-way handshake message).

Process 1100 also sends an initial data message (at 1140) to the destination network domain via the selected network slice. In addition, process 900 uses the stored connection mapping state to receive and transmit (at 1145) subsequent data messages for the connection over the selected network slice. This stored state allows the slice selector to perform a deep packet inspection (which, as mentioned above, may not be possible for many subsequent data messages), for connections from endpoint devices. Each data message can be assigned to a selected network slice.

FIG. 12 conceptually shows the process 1200 of some embodiments for the output gateway in the case of handshake reproduction. In the example shown in FIG. 10 (ie, an example in which the VSS is implemented on a telecommunications service provider access network), the output gateway that performs process 1200 is a gateway that connects the VSS to the public Internet.

As shown, process 1200 receives a set of connection handshake messages from the endpoint device through the default network slice and the destination network domain for connection to the endpoint device through the default network slice. Start by sending a return message from (at 1205). This egress gateway receives messages sent from the endpoint device, receives return traffic and sends them to the default network slice for the slice selector (and eventually the endpoint device). Send a message to the destination. In addition, process 1200 stores data (at 1210) about the handshake message (ie, the data needed to copy the message or recreate the message). For example, in the case of the TCP 3-way handshake, the output gateway stores data to send a SYN-ACK message. This behavior is not required for single message connection initiation (eg, QUIC protocol), and the slice selectors of some embodiments are initially to avoid the need to use default slices or perform handshake playback. Perform deep packet inspection on the message.

Once the connection between the endpoint device and the network domain is set up, process 1200 receives playback of the connection start message from the slice selector via the selected network slice (at 1215). At this point, the slice selector receives a data message from the endpoint device, uses deep packet inspection to assign the connection to a specific one of the possible network slices, and then sends the first connection start message to the egress gateway. By doing so, the reproduction process is started. Processing 1200 uses the data stored for these messages to perform a complete replay (at 1220) of the connection handshake with the slice selector via the selected network slice. That is, this output gateway recognizes the initial connection start message as corresponding to the stored set of handshake message data, and uses the stored set of data for playback. For example, in the example of the TCP 3-way handshake, the output gateway receives as a SYN message, sends a SYN-ACK message to the slice selector, and receives the ACK message. This allows the network service of the selected network slice to process the complete set of handshake messages and prepare for the rest of the data belonging to the connection.

Process 1200 stores (at 1225) the state of mapping the connection to the selected network slice (eg, using 5 tuples). This information may be stored in a connection mapping table as shown in FIG. 7 (eg, by creating a new flow entry for the connection if the gateway is a flow-based forwarding element). Using this stored state, process 1200 receives and sends subsequent return data messages belonging to the connection to the endpoint device via the selected network slice (ie, from the public network domain) (at 1230). Send.

Other embodiments may use other techniques to build a state that maps connections to network slices. In some embodiments, the slice selector integrates a registered connection-slice mapping table (eg, via a control plane channel) with an external component that provides the slice selector, which performs a stateful slice selection. Use this external component to do so (so that the slice selector avoids the requirement to perform deep packet inspection). In different embodiments, the external component may provide a fully pre-registered connection-slice mapping table when the endpoint device initiates a connection, or may gradually register updates in the table. .. As an example, the slice selectors of some embodiments can be integrated with the 5G Network Slice Selection Function (NSSF), allowing the NSSF to define connection-slice mappings. In some such embodiments, the NSSF provides a mapping state for the slice selector, which uses that state to select the correct slice for the data packet. That is, the offline external component provides a connection-slice mapping state, and the slice selector enforces this state in the data plane.

Stateful slice selection is the same slice selector (and) where the first data message is inspected to select the network slice for the connection and subsequent data messages are assigned to the network slice based on the state stored by the slice selector. It works as long as the egress gateway) handles all the data traffic for the connection. However, in a distributed network with a large number of slice selectors associated with different geographic areas (eg, telecommunications service provider access networks), mobile devices (eg, smartphones, tablets, self-driving cars) remain connected while maintaining connectivity. , (For example, when moving from one base station to another, when moving from a WiFi network to a cellular network between groups of base stations serving traffic to the same centralized unit), provided by the first slice selector. You may move from one geographic range to another geographic range provided by the second slice selector. In different embodiments, different techniques are used to ensure that the state is maintained without the need for action on some of the endpoint devices.

In some embodiments, the second slice selector (the slice selector for the area where the mobile device moves) sends all the data messages for the connection to the first slice selector (the mobile device when the connection is initiated). Transfer to the slice selector) for the area where was located. That is, the second slice selector receives data indicating that the first slice selector is in the slice mapping state position for the connection, and thus forwards the data traffic for the connection to the first slice selector. ..

FIG. 13 conceptually shows a mobile device 1300 moving from a first slice selector region to a second slice selector region, the second slice selector being a mobile device via two stages 1305-1310. Forward data traffic from 1300 to the first slice selector. As shown in step 1305, the mobile device 1300 is connected to a public network domain (not shown) while located in the first geographic area 1315 serviced by the first slice selector 1320. To start. Adjacent (possibly partially overlapping) geographic regions 1325 are serviced by a second slice selector 1330. In some embodiments, each slice selector is located within an edge cloud that corresponds to a 5G centralized unit (CU) that includes multiple distributed unit (DU) ranges (ie, multiple cell towers).

When the mobile device 1300 initiates a connection (which may be only one of a plurality of connections initiated by the device (eg, in a single PDU session)), the first slice selector 1320 determines the access network. Assign a connection to slice 1335, one of several slices of the virtual service network implemented via. As shown, the network slice 1335 contains three VNFAs-Cs prior to transmitting data to the Internet via an output gateway (not shown). The first slice selector 1320 performs state data that maps the connection (in this case, the TCP connection between IP1 and IP2) to the selected network slice after performing a deep packet inspection to select the network slice. Store. As mentioned above, this state data may be stored as a flow entry (or set of flow entries), as an entry in the connection table, or in another way. For subsequent traffic from mobile device 1300 belonging to this connection, slice selector 1320 allocates traffic to the selected network slice 1335 (other connections from device 1300 may be assigned to other slices). The return traffic for the connection is received from the Internet at the egress gateway, which uses a similar stored state to allocate this traffic to the same network slice 1335. This return traffic is processed in reverse order by the VNF of the network slice 1335 and then sent from the slice selector 1300 to the mobile device 1300.

However, in the second stage, the mobile device 1300 moves to the second geographic area 1325 and therefore no longer connects to the first slice selector 1320 (ie, the mobile device 1300 is not the first slice selector 1320). Connected to another base station that supplies traffic to the second slice selector 1330). The second slice selector 1330 does not have a connection-slice mapping state for allocating this data traffic from device 1300 to the correct network slice, and in many cases the data message is a slice selector 1330 for allocating a connection to the network slice. Does not contain the necessary data in the L5-L7 header for. Therefore, the second slice selector 1330 forwards this traffic to the first slice selector 1320, which uses its stored state information to allocate the traffic to the selected network slice 1335. do. New connections initiated by device 1300 while in the second geographic area 1325 will be assigned to the correct slice by the second slice selector 1330.

In some embodiments, the second slice selector 1330 packets through a routable network between the two slice selectors in order for the second slice selector 1330 to send data traffic to the first slice selector 1320. To send. That is, in such an embodiment, there is a routable network between the two edge clouds where the slice selector is implemented, and this network can be used to send data traffic between the two slice selectors. can. In other embodiments, data traffic can be sent over the core cloud or other WAN connection (provided that the two edge clouds connect to the same core cloud) or through the VSN controller (provided that it is). This solution is not optimal if a large amount of traffic is sent between slice selectors).

In some embodiments, the reverse (return) traffic of an ongoing connection is different because the slice selector does not require a connection state to handle the return traffic and send this return traffic to device 1300. They are treated differently in form. However, in many cases at least one of the network services is stateful and is implemented in the same location as the slice selector (eg, the same edge cloud) and therefore the same implementation of those network services (ie, a second). Return traffic needs to be sent to that edge cloud because of the VM in the first edge cloud with the first slice selector 1320, not the VM in the second edge cloud with the slice selector 1330. be. The first slice selector 1320 then forwards this return traffic to the second slice selector 1330 for the second slice selector 1330 to forward data to the mobile device 1300 (eg, via RAN). In some embodiments, the service chaining module in the core cloud has its learning function (in order to automatically send the traffic back to the first slice selector 1320 that received the traffic originating on the mobile device 1300). For example, the MAC learning function) is used. Further, in some embodiments, the first slice selector 1320 directs the traffic for connection from the second slice selector 1330 so that the return traffic is automatically forwarded onto the network between the two slice selectors. A similar learning function is used when receiving (as a result, the traffic returns to the second slice selector 1330). For example, if there is a routable network between the two slice selectors, the first slice selector 1320 can use the stored MAC address to forward traffic back to this router. Stores the MAC address of the router that received the traffic from the slice selector 1330. Another embodiment uses a separate ingress gateway function on the slice (ie, before the first network service), which serves to send the traffic back to the correct slice selector.

In order for the second slice selector 1330 to forward data traffic for a particular connection to the first slice selector 1320, the second slice selector has state information for the first slice selector 1320 to connect. It is necessary to receive data indicating that. In a different embodiment, the first slice selector (i) pushes the state position information to a network controller (eg, the VSN controller described above) from which the second slice selector retrieves the state position information, or (ii). Push the state position information to the second slice selector.

FIG. 14 shows an example of a first slice selector 1420 that pushes the state position information 1400 to the central controller 1425 and a second slice selector 1430 that retrieves the state position information from the central controller 1425 via three stages 1405-1415. Is shown conceptually. As shown in the first stage 1405, as in the example of FIG. 13, the mobile device 1435 is located in the first geographic area 1440 associated with the first slice selector 1420 while in the public network. Initiate a connection with the domain. The first slice selector assigns the connection to the network slice 1445, directs data traffic from mobile devices 1440 belonging to this connection to this slice (ie, to the network service of this slice), and connects to the selected network slice. Stores the connection state that maps to.

Further, the first slice selector 1420 pushes information to the network controller 1425 specifying that the first slice selector is the position of the slice mapping state for this connection. This network controller is, in some embodiments, a VSN controller that provides VSN configuration data to controllers in a plurality of data centers in which VSN is implemented. Specifically, in some embodiments, the first slice selector 1420 transfers slice mapping state position data to one of the controllers local to that data center (eg, an SDN controller that constitutes the slice selector). And then pass the state position data to the VSS controller so that the state position data can be accessed by slice selectors in other data centers.

In the second stage 1410, the mobile device 1435 was moved to the second geographic range 1450 associated with the second slice selector 1430. Upon receiving a data message from device 1435 regarding an in-progress connection that the second slice selector 1430 does not recognize, the slice selector 1430 (eg, makes such a request to one of the controllers local to that data center). And the data center sends the request to the controller 1425 (by sending the request to the VSS controller). The controller 1425 stores this state position information 1400 and therefore returns the information 1400 to the second slice selector 1430 (eg, via a controller local to the data center of the second slice selector 1430).

Based on this state position information, in the third stage 1415, the second slice selector 1430 transfers the data message for this connection (and the subsequent data message for the connection) to the first slice selector 1420. It can transfer data onto the selected network slice 1445. In some embodiments, data center connections (ie, routable networks) exist between the edge clouds, and in other embodiments this traffic is separate from one slice selector over the core cloud or other network. Passed to the slice selector of.

In another embodiment, the connected slice selector has the state location information available to other slice selectors when the connected mobile device moves to a new geographic area. Push location information to other slice selectors (eg, geographically adjacent slice selectors). FIG. 15 conceptually shows an example in which the first slice selector 1515 pushes the state position information 1500 to the second slice selector 1520 via two stages 1505 to 1515. As shown in the first stage 1505, similar to the example of FIG. 13, the mobile device 1525 is located in the first geographic area 1530 associated with the first slice selector 1515 while being located in the public network. Initiate a connection with the domain. The first slice selector 1515 assigns the connection to the network slice 1535 and forwards data traffic from the mobile device 1525 belonging to this connection to this slice (ie, to the network service of this slice) to the selected network slice. Stores the connection state that maps the connection.

Further, the first slice selector 1515 pushes information specifying that the first slice selector 1515 is the position of the slice mapping state for this connection to the second slice selector 1520. Different embodiments transmit state location information in different ways. In some embodiments, this information is transmitted over the data network (eg, routing) for data traffic transmitted between the two slice selectors (but as control plane data between the control plane interfaces of the slice selectors). It is transmitted over a possible data center-to-data center network (over the edge cloud), while in other embodiments state location information is pushed to the controller (ie, as shown in FIG. 14). The controller then automatically pushes the state position information to the second slice selector 1520. This state location information, in different embodiments, is a particular slice selector with adjacent geographic extents, all slice selectors for a particular network (eg, a particular network service provider), or other combinations of slice selectors. May be pushed to.

In the second stage 1510, the mobile device 1525 has moved to the second geographic range 1540 associated with the second slice selector 1520. Upon receiving data traffic from the device 1525 for an in-progress connection, the second slice selector 1520 maps the data traffic to the state location data already stored and sends the data message to the first slice selector 1515. It can be transferred and the data transferred onto the selected network slice 1535. In some embodiments, data center connections (ie, routable networks) exist between the edge clouds, and in other embodiments this traffic is separate from one slice selector over the core cloud or other network. Passed to the slice selector of.

Instead of always transferring the data for the connection to the original slice selector where the mobile device initiated the connection, another embodiment transfers the slice mapping state for the connection to another slice selector to which the mobile device moves. And provide. The second slice selector (ie, the slice selector to the range to which the mobile device moves) receives the slice mapping state for the connection and therefore without the first network slice selector (where the connection was initiated). , Data messages for connection can be forwarded to network slices.

FIG. 16 conceptually illustrates a mobile device 1600 moving from a first slice selector region to a second slice selector region, the second slice selector being connected via two stages 1605-1610. Receives the slice mapping state for and uses that slice mapping state to forward data traffic for the connection. As shown in the first stage 1605, the mobile device 1600 and the public network domain (not shown) while located in the first geographic area 1615 serviced by the first slice selector 1620. To start the connection. Adjacent (possibly partially overlapping) geographic regions 1625 are serviced by a second slice selector 1630. In some embodiments, each slice selector is located within an edge cloud that corresponds to a 5G centralized unit (CU) that includes multiple distributed unit (DU) ranges (ie, multiple cell towers).

When the mobile device 1600 initiates a connection (which may be only one of a plurality of connections initiated by the device (eg, in a single PDU session)), the first slice selector 1620 determines the access network. Implemented via and assigns a connection to slice 1635, one of several slices of VSN. As shown, the network slice 1635 contains three VNFAs-Cs before transmitting data to the Internet via an output gateway (not shown). The first slice selector 1620 performs state data that maps the connection (in this case, the TCP connection between IP1 and IP2) to the selected network slice after performing a deep packet inspection to select the network slice. Store. As mentioned above, this data may be stored as a flow entry (or set of flow entries), as an entry in a connection table, or in another way. For subsequent traffic from mobile device 1600 belonging to this connection, slice selector 1620 allocates traffic to the selected network slice 1635 (other connections from device 1600 may be assigned to other slices). The return traffic of the connection is received from the Internet at the egress gateway, which egress gateway uses a similar stored state to allocate this traffic to the same network slice 1635. This return traffic is processed in reverse order by the VNF of network slice 1635 and then sent from the slice selector 1600 to the mobile device 1600.

However, in the second stage, the mobile device 1600 moves to the second geographic area 1625 and therefore no longer connects to the first slice selector 1620 (ie, the mobile device 1600 is not the first slice selector 1620). Connected to another base station that supplies traffic to the second slice selector 1630). In this case, instead of transferring data from the mobile device 1600 to the first slice selector 1620, the first slice selector 1620 provides the slice mapping state for connection to the second slice selector 1630. .. Therefore, the second slice selector 1630 can transfer this data directly to the network slice 1635 selected for the connection without having to perform deep packet inspection.

In some embodiments, one or more network services for slicing are stateful and implemented in the edge cloud with a slice selector. If the services are stateless, then when the traffic moves to the second slice selector 1630, the instances of these services in the new edge cloud can handle the traffic without any problems. However, if the network service in the edge cloud is stateful, some embodiments will be from an instance of the service in the edge cloud with the first slice selector 1620 to in the edge cloud with the second slice selector 1630. Transfer state to an instance of a network service. Another option utilized by some embodiments is to migrate the network service instance from the first edge cloud to the second edge cloud. However, if the network service instance is handling traffic for a large number of connections, this option then has the disadvantage of interrupting other connections. In some other embodiments, if one of the network services for the selected slice is stateful and implemented in the edge cloud using the slice selector, then the slice mapping state for the connection is the first. It is not provided to the second slice selector and instead, as described above in FIGS. 13-15, the data traffic is forwarded to the first slice selector.

In different embodiments, the second slice selector 1630 may receive the state directly from the first slice selector or from a network controller (eg, the VSN controller described above). In some such embodiments, the first slice selector (i) pushes the state directly to the second slice selector (eg, before the device moves to the geographic area of the second slice selector). Or (ii) the second slice selector pushes the state to the network controller that retrieves the state. In other such embodiments, the first slice selector pushes position information about the state to the network controller, the second slice selector retrieves this position information from the network controller, and then the state from the first slice selector. Use that location information to retrieve.

FIG. 17 is an example of a first slice selector 1720 that pushes the slice mapping state 1700 to the central controller 1725 and a second slice selector 1730 that retrieves the slice mapping state from the central controller 1725 via three stages 1705 to 1715. Shown conceptually. As shown in the first stage 1705, similar to the example of FIG. 16, the mobile device 1735 is located in the first geographic area 1740 associated with the first slice selector 1720 while being located in the public network. Initiate a connection with the domain. The first slice selector 1720 assigns the connection to the network slice 1745, forwards data traffic from the mobile device 1740 belonging to this connection to the selected slice (ie, to the network service of this slice), and the selected network. Stores the connection state 1700 that maps the connection to the slice.

Further, the first slice selector 1720 pushes the connection to the slice mapping state 1700 to the network controller 1725 so that other slice selectors can retrieve this state as needed. This network controller is, in some embodiments, a VSN controller that provides VSN configuration data to controllers in a plurality of data centers in which VSNs are implemented. Specifically, in some embodiments, the first slice selector 1720 brings the slice mapping state 1700 to one of the controllers local to its data center (eg, the SDN controller that constitutes the slice selector). It provides and then passes the state to the VSN controller so that the state can be accessed by slice selectors in other data centers.

In the second stage 1710, the mobile device 1735 has moved to the second geographic range 1750 associated with the second slice selector 1730. Upon receiving a data message from device 1735 regarding an in-progress connection that the second slice selector 1730 does not recognize, the slice selector 1730 (eg, makes such a request to one of the controllers local to that data center). And the data center sends the request to the controller 1725 (by sending the request to the VSS controller). The controller 1700 stores slice mapping information 1400 for the connection specified in the request and therefore the second slice of state 1700 (eg, via a controller local to the data center of the second slice selector 1730). Return to selector 1730.

Based on this slice mapping state, in the third stage 1715, the second slice selector 1730 processes the data message received from the mobile device 1735 (as well as the subsequent data message for this connection). This data message can be forwarded onto the selected network slice (ie, the slice specified in the slice mapping state for the connection).

In another embodiment, the slice selector where the connection is initiated provides only the state position information to the controller, the other slice selector retrieves the state position information and uses it to directly from the first slice selector the slice mapping state. Allows you to retrieve. 18A-B take out the first slice selector 1825 for pushing the state position information 1840 to the controller 1830 and the state position information 1840 for the four stages 1805 to 1820, and slice mapping state 1800 from the first slice selector 1830. An example is conceptually shown with a second slice selector 1835 that uses its state position information 1840 to retrieve. As shown in step 1805, similar to the example in FIG. 16, the mobile device 1845 is located in the first geographic area 1850 associated with the first slice selector 1825 while being located in the public network. Initiate a connection with the domain. The first slice selector 1825 assigns a connection to network slice 1855, forwards data traffic from mobile devices 1845 belonging to this connection to selected network slice 1855 (ie, to network services in this slice) and is selected. Stores the connection state 1800 that maps the connection to the network slice.

Further, the first slice selector 1825 pushes state position information 1840 to the network controller 1830, which specifies that the first slice selector is the position of the slice mapping state for this connection. This network controller is, in some embodiments, a VSN controller that provides VSN configuration data to controllers in a plurality of data centers in which VSN is implemented. Specifically, in some embodiments, the first slice selector 1825 uses the slice mapping state position data 1840 as one of the controllers local to the data center (eg, an SDN controller that constitutes the slice selector). ), And then pass the state position data to the VSS controller so that the state position data can be accessed by the slice selectors of other data centers.

In the second stage 1810, the mobile device 1845 was moved to the second geographic range 1860 associated with the second slice selector 1835. Upon receiving a data message from device 1845 about an in-progress connection that the second slice selector 1835 does not recognize, this slice selector 1835 (eg, makes such a request to one of the controllers local to that data center). And the data center sends the request to the controller 1830 (by sending the request to the VSS controller). The controller 1830 stores this state position information 1840 and therefore returns the information 1840 to the second slice selector 1835 (eg, via a controller local to the data center of the second slice selector 1835).

Based on this state position information, in the third stage 1815, the second slice selector 1830 sends a request for a slice mapping state for connection to the first slice selector 1825. This request, in some embodiments, specifies a connection (eg, by 5 tuples) and is formatted in a particular manner as recognized by the first slice selector 1825 as a request for slice mapping state. In response, the first slice selector 1825 sends the slice mapping state 1800 for connection to the second slice selector 1835. In some embodiments, routable data center connections exist between the edge clouds, and in other embodiments, requests and subsequent responses are sent from one slice selector to another via the core cloud or other network. Passed to.

In the fourth stage 1820, the second slice selector 1835 processes the data message received from the mobile device 1845 (as well as the subsequent data message for this connection) and selects the network slice 1855 (ie, the selected network slice 1855). This data message can be forwarded on the slice specified in the slice mapping state for connection.

In yet another embodiment, the connection-initiated slice selector will have a slice mapping state for the connection available to other slice selectors when the initiating mobile device moves to a new geographic area. Pushes its slice mapping state to another slice selector (eg, a geographically adjacent slice selector). FIG. 19 conceptually shows an example of a first slice selector 1915 that pushes the slice mapping state 1900 to the second slice selector 1920 over two stages 1905-1910. As shown in the first stage 1905, similar to the example of FIG. 16, the mobile device 1925 is located in the first geographic area 1930 associated with the first slice selector 1915 while being located in the public network. Initiate a connection with the domain. The first slice selector 1915 assigns the connection to the network slice 1935 and forwards data traffic from mobile devices 1925 belonging to this connection to this slice (ie, to the network service of this slice) to the selected network slice. Stores the connection state that maps the connection.

Further, the first slice selector 1915 pushes the slice mapping state 1900 for the connection to the second slice selector 1920, indicating that the connection is assigned to the network slice 1935. Different embodiments transmit slice mapping states in different ways. In some embodiments, this state is transmitted over a data network (eg, over a routable network between data centers, over an edge cloud), while in other embodiments, this state is Pushed to the controller (ie, as shown in FIG. 17), the controller then automatically pushes this state to the second slice selector 1920. This slice mapping state, in different embodiments, is a particular slice selector with adjacent geographic extents, all slice selectors for a particular network (eg, a particular network service provider), or other combinations of slice selectors. May be pushed to.

In the second stage 1910, the mobile device 1925 has moved to the second geographic range 1940 associated with the second slice selector 1920. Upon receiving data traffic from device 1925 for an in-progress connection, the second slice selector 1920 processes the data message received from mobile device 1925 (as well as subsequent data messages for this connection). , This data message can be forwarded onto the selected network slice 1935 (ie, the slice specified in the slice mapping state for connection).

In some of the above examples, the first slice selector pushes the slice mapping state (or state position information) to the second controller. In some embodiments, if a mobile device that initiates a connection within the geographic area of the first slice selector moves to one of the adjacent geographic areas, the first slice selector will be in its slice mapping state ( Or push all of their connection state location information) to the slice selector in the adjacent geographic area. In other such embodiments, the first slice selector pushes state information to the slice selector for adjacent geographic areas where the device may move, so that the location data of the mobile device (that data is). If available) is used.

FIG. 20 conceptually shows a first slice selector 2000 associated with a first geographic region that pushes slice mapping states to all of its adjacent geographic regions 2005, according to some embodiments. In this example, the first geographic area 2005 has six adjacent geographic areas 2010-2035. These geographic regions 2005-2035 are all circular and of equal size in this example, but the actual geographic regions are for a variety of reasons (eg, different slice selectors are associated with different numbers of base stations and different bases. It should be understood that the size and shape can vary due to the station having different associated geographic areas). When the connection is initiated by a mobile device located in the first geographic region 2005, the slice selector 2000 associated with this region will change the slice mapping state to the slice selector associated with the adjacent geographic region 2010-2035. Push to all.

Some embodiments only push the slice mapping state (or state location information) into a directly adjacent area (ie, a region that partially overlaps or is adjacent to the region where the connection is initiated), and others. The embodiment pushes the state to an additional area (eg, all areas, areas adjacent to all adjacent areas of the area where the connection is initiated). In some embodiments, the slice selector consists of a list of all areas that push slice mapping states (or state location information), which (eg, by sending information over a connection between data centers). Push the state directly to the slice selector in another area. When a mobile device moves to a different area and the slice selector for that area uses the slice mapping state to handle data traffic for connections from the mobile device, in some embodiments it is for a new area. The slice selector pushes its state to the slice selector for its adjoining area as the mobile device continues to move.

The slice selector of the other embodiment pushes the state to a central controller (eg, a VSN controller) that automatically distributes the state to the slice selectors in the adjacent area, in which case the slice selector is processed by the controller. , Does not need to be configured with a list of slice selectors that push that state.

As mentioned above, some embodiments use more accurate location data for mobile devices to push slice mapping states (or state location information) to specific adjacent areas in a centralized computer-managed manner. .. FIG. 21 pushes the mobile device 2125 moving within the first geographic area 2105 and the slice mapping state for the connection initiated by the mobile device only to the adjacent area where the device 2125 is moving. A slice selector 2100 for a region is conceptually shown. As shown in the figure, the mobile device 2125 has moved from a position closer to the center of the area 2105 to a position closer to the overlap between the area 2105 and its adjacent area 2115. In addition, the movement vector of the mobile device indicates that the device can quickly move to region 2110. Therefore, based on this location information, the first slice selector 2100 pushes the slice mapping state for any connection initiated by the mobile device 2125 to the slice selectors for regions 2110 and 2115 (but not). Do not push to other slice selectors for the illustrated adjacent regions 2120). Different embodiments may use different heuristics regarding when to push the slice mapping state (or state position information) to a particular adjacent region (eg, using an absolute position within a threshold distance of the neighborhood region, neighborhood). Use a direction vector that indicates the movement towards the region, or use other heuristics).

All of the above examples show a single virtual service network implemented on a physical infrastructure (eg, a telecommunications service provider access network). However, in some embodiments, the virtual service network is hierarchically sliced. That is, a slice of a virtual service network is itself a virtual service network with a slice selector and multiple network slices.

FIG. 22 conceptually shows an example of such a hierarchical virtual service network. Specifically, this figure shows a provider infrastructure 2200 with a slice selector 2205 to choose from, two separate virtual service networks 2210 and 2215, each with a plurality of slices. The provider infrastructure 2200 receives data traffic from various devices 2220 (eg computers, smartphones, tablets, self-driving cars, IoT devices) and transfers this data traffic out of two different lower level virtual service networks 2210 and 2215. It is its own top-level virtual service network with a slice selector 2205 assigned to one of the above.

For example, in some embodiments of a telecommunications service provider network, a device in which a mobile network operator (MNO) owns and subscribes to an access network and core network physical infrastructure 2200 (ie, RAN and EPC infrastructure). Configure slice selector 2205 to handle traffic from. In addition, the MNO can lease its physical infrastructure to one or more mobile virtual network operators (MVNOs) who also have subscriber devices that use the same infrastructure. These MVNOs may also lease their virtual infrastructure to additional MVNOs or other entities. In the example of FIG. 22, the slice selector 2205 is configured such that the MNO chooses from tenant A's VSS2210 (for its own subscriber device) and tenant B's VSS2215 (for the MVNO's subscriber device). be able to.

For example, the slice selector 2205 configured by the MNO assigns a data message to either VSN2210 or VSN2215 based on the source device (eg, by source network address). Therefore, the data message from the source device related to the MVNO is transmitted to the VSS2210, and the data message from the source device related to the MVNO is transmitted to the VSS2215 configured by the MVNO. If the additional MVNO also leases the infrastructure, then the slice selector 2205 will have an additional VSN of choice (each MVNO will have a slice selector and a set of network services for slicing its own VSN. Can be configured).

Each of VSN 2210 and 2215 also has its own slice selectors 2225 and 2230, respectively. In this example, each of these slice selectors 2225 and 2230 chooses from two possible network slices, just as the provider infrastructure can have as many VSNs as the top-level slice selector 2205 chooses. It should be understood that each of the above will often contain a large number of slices. In some embodiments, these slice selectors 2210 and 2215 for the tenant VSN perform additional slice selections based on various aspects of the data message header. For example, in some embodiments, the top-level slice selector 2205 selects VSNs based on the source device network address, while the lower-level slice selectors 2210 and 2215 can assign data messages to slices in the stateful manner described above. (For example, use deep packet inspection to assign connections to slices in an application-aware manner).

FIG. 23 conceptually shows the distribution of providers and tenant slice selectors (as well as network services for network slices) across multiple data centers 2305-2320. As shown, in this example, both the provider slice selector 2325 and the tenant slice selector 2330 are implemented in the edge clouds 2305 and 2310, respectively. In addition, although not shown, each other tenant slice selector is implemented in each of the edge clouds (unless other tenant slice selectors are implemented in the core cloud) (in some embodiments, those tenants). Allows for none of the VSN slices or any of the network services to be instantiated in the edge cloud). Further, as shown in FIG. 2, the network services (VNFA to D) of the illustrated network slice 2300 are distributed among the edge clouds 2305 and 2310, the core cloud 2315, and the public cloud 2320.

A single-level slice selector runs differently (eg, as a flow-based transfer element operating within a VM or virtualization software, as a field programmable gate array physical transfer element, between a VM and a port of a software transfer element. Different embodiments implement the multi-level slice selectors 2325 and 2330 in different ways, just as they may be implemented as separate sets of modules). If the form factor of the slice selector is a VM or a transfer element performed within the VM, some embodiments of each instance of the provider slice selector 2325 and the tenant slice selector 2330 (and any other tenant slice selector). Use a separate VM for each instance. This allows, for example, the provider administrator to configure the VM and the transfer element for the provider slice selector 2325 separately from the transfer element for each of the VM and the tenant slice selector.

In this case, when the access network receives the data message, the message is first sent to the provider slice selector 2325 (eg, after any preprocessing via the RAN). After the provider slice selector transfer element selects one of the tenant VSNs (or the provider's own VSN, effectively another tenant VSN), the provider slice selector 2325 is the same edge cloud (ie, in this example, the edge cloud 2305). ) Sends a data message to the slice selector 2330 for the selected tenant VSN. In some embodiments, the provider slice selector 2325 uses service chaining technology to send a data message to the tenant slice selector 2330, in other embodiments the provider slice selector 2325 is a data message at this point. It is configured to finish processing and simply send a data message to the appropriate tenant slice selector (eg, slice selector 2330).

The tenant slice selector 2330 receives a data message, performs slice selection and service chaining on the selected slice (ie, in the same way as shown in FIG. 3), and then data via the output gateway. Send a message. If the network is distributed across multiple data centers (ie, as shown in this example), the tenant VSN implementation will include a service chain module within each data center in some embodiments. In some such embodiments, the provider slice selector 2325 does not perform service chaining (ie, the tenant slice selector 2330 and / or the service chaining module returns data traffic to the provider slice selector after the tenant network slice is complete. Therefore, the provider service chaining module is not needed in other data centers).

In the example of FIG. 23, the mapping of the provider slice selector to the tenant slice selector is 1: 1. However, in other embodiments, the top-level (provider) slice selector may be more distributed than the lower-level (tenant) slice selector. For example, in a 5G access network, provider slice selectors in some embodiments may be implemented in each DU and slice selectors for various tenants may be implemented in each CU. In some such embodiments, the tenant slice selector uses MAC learning to determine to which provider slice selector return traffic should be sent. In many cases, only tenant slice selectors use stateful connections to slice mappings, so only movements between regions associated with different tenant slice selectors are state sharing or state location sharing as described above with reference to FIGS. 13-19. Causes application application (ie, if the provider slice selector assigns a data message to a network slice based on another value based on the source network address or source device, stateful mapping is not required). In this situation, the tenant slice selector uses the learned MAC address to send the traffic back to the correct provider slice selector, and the provider slice selector sends the traffic from one provider slice selector to another. It will be the correct provider slice selector for the current location of the device as it does not need to be.

In some embodiments, the lower level (tenant) slice selector is implemented in the same VM and / or transfer element as the top level (provider) slice selector, rather than implementing different levels of slice selectors separately. For example, in some such embodiments, a first set of flow entries implements a provider slice selector and a separate set of flow entries implements each of the tenant slice selectors. Which of these separate sets of flow entries is evaluated (ie, which of the tenant slice selectors evaluates the data message) is the first of the first set of flow entries. It depends on whether it is matched by the slice selector of (that is, which tenant VSN is assigned the data message).

In the service insertion model for slice selectors where slice selection is performed as a service related to the port of the software transfer element, some embodiments have both top-level (provider) slice selection and lower-level (tenant) slice selection. , Run one after another as separate services. That is, the data message is first intercepted by the provider slice selector, and then the data message is intercepted by one of the tenant slice selectors based on which tenant VSN was selected.

FIG. 24 conceptually shows the branch control of the provider infrastructure 2200 and the plurality of tenant VSN 2210 and 2215 according to some embodiments. As shown in FIG. 4, the VSN manager and controller 2400 (hereinafter referred to as VSN controller) is a centralized top layer of control for the entire network. In some embodiments, the VSN controller has a separate instance of the provider VSN controller 2405 and the tenant VSN controllers 2410 and 2415 corresponding to each of the tenant VSNs.

Separate controller instances 2405 to 2415 are, in some embodiments, accessed by logins with different administrative privileges (ie, using role-based access control). That is, some embodiments of VSN controller interfaces (CLI, GUI, API) provide different logins to provider administrator accounts and separate accounts for each tenant administrator. These different accounts can provide different configuration datasets to the VSN controller to configure their respective VSNs. For example, in some embodiments, the provider VSN controller 2405 provides a top-level slice selector, chaining between any top-level service and selected tenant VSNs, and physical networks and gateways between data centers. Allows to be configured. In addition, the provider VSN controller 2405 manages the life cycle of the provider VSN (eg, instantiating slice selectors, gateways, and other components) in some embodiments, and / or provider VSN entities and various. Includes the ability to monitor and optimize the tenant VSN.

In some embodiments, each of the tenant VSN controllers 2410 and 2415 allows each tenant to configure their respective VSNs separately. By using different logins for the administrator, controller 2400 separates the tenant administrator to configure only its own VSN without configuring any of the other tenant VSNs or provider VSNs. In some embodiments, each tenant administrator sets up its own slice selector, network services for their various slices, chaining between slice selectors and network services, and other aspects of tenant configuration. Can be done. Further, in some embodiments, the tenant VSN controllers 2410 and 2415 monitor and / or monitor various network services (and slices as a whole) for managing the life cycle of the tenant VSN and various network services, respectively. Includes optimization capabilities.

As mentioned above with respect to FIG. 4, in some embodiments, each data center has its own set of controllers. In some embodiments, these controllers do not distinguish between top-level configuration data and low-level configuration data. Instead, the VSS controller 2400 uses configuration data (network configuration data, slice selector configuration data, network service configuration data, network services to configure forwarding elements to tunnel data messages between slice selectors and network services. Configuration data) is provided to these controllers, which configure different entities in the same way for different levels. For example, in the example of FIG. 24, the provider VSN controller 2405 provides slice selector settings that the SDN controller uses to configure the provider slice selector, but provides VNF configuration data for the compute controller to configure the VNF. do not. Instead, this data about the compute controller is provided by a variety of different tenant VSS controller instances 2410 and 2415. These VSN controller instances also provide slice selector configuration data for use by SDN controllers to configure slice selectors. In some embodiments, the WAN configuration for transmitting data traffic between data centers is provided only to the SDN controller that manages these gateways by the provider VSN controller 2405 (ie, the tenant manages the physical infrastructure). Not to do).

In the above example of FIG. 22, the lower level slice selectors 2225 and 2230 are the first entity to process the data messages in the VSNs 2210 and 2215, respectively. Some embodiments require the slice selector to be the first entity in VSN in order to process the data message. However, in other embodiments, after the first network slice selector selects one of the VSNs, this VSN (which is a slice of the top level VSN) has the lower level slice selector sliced within its lower level VSN. May include network services applied to the data message before performing the action of selecting. Similarly, in some embodiments, the network service may be applied to the data messages of all network slices in the virtual service network after different services have been applied to a given slice.

FIG. 25 conceptually shows an example of a hierarchical VSN after a network service has been inserted between slice selectors and / or services for different slices have completed. This figure shows a provider infrastructure VSS2500 with a slice selector 2505 to choose from two tenants VSS2510 and 2515.

The first tenant VSS2510 includes a slice selector 2520 to select between the two slices 2525 and 2530, each of which has a different set of network services. Further, as shown, regardless of which slice the data message is assigned to, the data message is sent to the VNFE for processing after either slice 2525 or slice 2530 is completed. For example, regardless of the type of data transmitted over the connection, the tenant may want a single weighing service that processes all of the connection for billing purposes. In this case, the slice selector (or the service chain module of the final data center where the slice is implemented) sends the data message to the VM that implements VNFE, regardless of which slice the data message is assigned to. However, in other embodiments, if all of these connections are sent to the same VNF, it would be more difficult to control the QoS parameters of all the connections and would not allow such network services outside the slice. In this case, separate instances of VNFE will be contained within slices 2525 and 2530, respectively.

When the provider slice selector 2505 assigns a data message to the second tenant VSS2515, in this example, the provider slice selector first sends the data message to an instance of the network service VNFA, and then the slice of the second tenant VSS2515. Send a data message to selector 2535. This allows the provider to provide network services to apply to all data traffic sent to a particular tenant VSN (for billing purposes, to provide RAN or EPC functionality, or for other purposes). Can be configured. However, another embodiment requires that the lower level slice selector be the first entity to which the data message is sent after the upper level slice selector and that all of the network services be centralized in the lowest layer of the network slice. do.

The above example shows two levels of slice selection, eg, an MNO that owns the infrastructure and one or more MVNOs that lease the infrastructure from the MNO. In many cases, the MVNO will also sublease those virtual infrastructures to one or more additional MVNOs and a third level of slice selection will be used. In this case, the MNO slice selector can assign a data message to the tenant VSS based on the source network address analysis, and then the first level of the MVNO slice selector is that data message based on a finer-grained network address analysis. (For example, all source devices with IP addresses within the / 24 subnet are assigned to the first level tenant VSS, after which it is subdivided among multiple source / 28 subnets). .. The third level of slice selection may then perform stateful slice selection based on deep packet inspection as described above.

FIG. 26 conceptually shows an example of a hierarchical set of VSNs with three levels of slicing. As in the example above, the provider infrastructure VSN2600 has a slice selector 2605 that assigns data messages to two different tenants VSN2610 and 2615. The first tenant VSS2610 has a slice selector 2620 that assigns data messages to two slices with different sets of network services.

On the other hand, the second tenant VSN2615 allocates a data message to either (i) a third level VSN2630 or (ii) a network slice 2635 that sparsely sets the VNFs that are part of the configuration of VSN2615. It has a selector 2625. For example, if the VSN2615 is managed by a first MVNO, the VSN2630 may be managed by a second MVNO that leases part of the virtual infrastructure, while the network slice 2635 joins the first MVNO. It is for data traffic to and from the device. The VSS2630 is a third level slice selector configured by a second MVNO to choose from two slices 2645 and 2650 (eg, based on the L2-L4 header or by using other factors in an application recognition method). It has 2640.

Further, if the first MVNO performs an application-recognized slice selection, the slice selector 2625 can choose from the VSS2630 and a plurality of different network slices for different applications. In this case, the slice selector 2625 sends data traffic to the VSS2630 that matches a particular set of source network destinations, and then slices the data traffic for other source network destinations based on the application layer data. Can be configured. In the implementation of the flow-based transfer element of the slice selector 2625, the flow entry of the VSS2630 (matching based on the source address) has a higher priority than the flow entry of other network slices, otherwise it is in the application-aware flow entry. The matching second MVNO data traffic will not be sent to one of the network slices.

However, other embodiments do not allow the slice selector to select from VSNs and network slices that are not further subdivided. In this case, the slice selector 2625 uses a slice selector to select from the slice 2635 configured for the first MVNO and any other slice, and another VSN configured by the first MVNO as well as the VSS2630. Will be configured to choose from.

The above example of a VSN that uses slice selection is a telecommunications provider network (both for hierarchical slice selection and single level slice selection), but this virtual service network is also configured for other types of networks. Please understand what you can do. For example, in the case of a network within a data center or across a large number of data centers, the virtualization software (eg, within the host computer hosting the VM or other endpoints of the network) or VM is the data transmitted by the network endpoint. It can be configured to implement message slice selection.

FIG. 27 conceptually shows the implementation of VSN in the data center 2700. In a different embodiment, the data center 2700 may be a public cloud (eg, using slice selection in a virtual private cloud of a public data center) or a private data center (eg, a private data center). May be good. In this example, data traffic is transmitted from the first guest VM2705 and the second guest VM2710.

In some embodiments, these two virtual machines 2705 and 2710 belong to the same logical network (eg, they are connected to the same logical switch and they are connected via one or more logical routers. Connected to a different logical switch). In some embodiments, the first guest VM2705 sends a data message processed by the slice selector 2715 running in the same host computer 2720 as the guest VM2705. In some embodiments, the slice selector 2715 is implemented by a software transfer element running on the host computer (eg, a software virtual switch running on virtualization software). In another embodiment, this slice selector is implemented as part of a service insertion layer between the guest VM2705 and a software transfer element (not shown). In some embodiments, this service insertion layer has L5-L7 header information determined from the characteristics of the data message (eg, source and / or destination address, deep packet inspection) in order to assign the data message to a particular slice. , Or other factors).

In this example, data traffic is assigned to the first slice with three services implemented as VNF2725-2735 in three VMs on different host computers. At least one other network slice (ie, a different set of ordered network services) will have different traffic from the first guest VM2705, possibly including different traffic to the same second guest VM2710. Implemented separately in the data center (as indicated by the dashed line) as handled by a set of network services. In some embodiments with slice selectors and / or service chaining modules implemented on a host computer with all guest VMs, data traffic from different different source VMs will be the same service from different origin host computers. It is transmitted along the route (ie, to the same VNF in the same order).

Some embodiments use the same service chaining method as described above for the telecommunications service provider network, where the data message returns to the slice selector 2715 after each service in the network slice has completed its processing. In this case, the traffic does not follow a linear flow through the chain of services, as shown in the figure, but rather repeatedly returns to the host computer 2720.

Another embodiment is a distributed service check such that the forwarding element on the same host computer as the first VNF2725 automatically forwards the data traffic to the second VNF2730 instead of returning the data traffic to the slice selector 2715. Use innings. Some such embodiments achieve this by automatically forwarding traffic received from an interface where one VNF connects to the next VNF in the service chain, while other such embodiments. The slice selector adds a tag or set of tags to the data message used by forwarding elements along the service chain to forward the message to the next VNF in the service chain.

When the data traffic is received by the host 2745 of the destination guest VM2710, the reverse slice selector 2740 stores connection information about the traffic before providing the data to the guest VM2710. In some embodiments, the reverse slice selector 2740 is such that the reverse slice selector 2740 stores a connection to the slice mapping (eg, maps 5 tuples to the slice selected by the slice selector 2715). Similar to the output gateway of. When the guest VM2710 sends return traffic to guest VM2705, the reverse slice selector 2740 uses this stored slice mapping for the connection to allocate the return traffic to the same slice.

In some embodiments, slice selection and reverse slice selection functions are performed by the same components within hosts 2720 and 2745 (eg, software transfer elements, service insertion layers, or other components). The slice selector and the reverse slice selector are distributed in such an embodiment, and the components thereof have a VM connected to each host computer (for example, VSN) to perform both the slice selection function and the reverse slice selection function. Each host computer) is configured on.

When the VM initiates a connection (that is, acts as a client) and sends traffic without stored connection mappings, this component uses (L2-L4 header fields, deep packet inspection, or other factors). ) Performs slice selection and sends traffic to the slice's network services in the configured order (in this case VNF2725, VNF2730, VNF2735). When a host receives incoming traffic for a new connection (ie, if the VM on that host acts as a server), its component acts as an inverse slice selector and stores the slice mapping data for the connection. When the VM returns traffic for a connection initiated elsewhere, this component uses the stored slice mapping data for the connection and reverses the order to the network service of the selected slice (in this case). It acts as a reverse slice selector that sends traffic on VNF2735, then VNF2730, then VNF2735).

FIG. 28 conceptually illustrates the implementation of VSN to handle WAN communication between two private enterprise data centers (ie, main office 2805 and branch office 2810) via the public cloud 2815. In this example, communication between the main office and the branch office will be described, but as a similar VSS between two branch offices, between mobile users connecting to a corporate data center via a VPN, or as any of the above endpoints and software. It should be understood that it may be configured to handle other WAN examples, such as communication with a Software as a Service (Software as a Service) provider data center. Moreover, in some embodiments, the VSN is fully configured within a network of private data centers rather than including one or more public data centers as in this case.

In this example, data traffic is sent from the VM2820 running on the host computer in the main office 2805 to the VM2825 running on the host computer in the branch office 2810. Similar to the previous example in FIG. 27, the distributed slice selector 2830 is implemented on the same host computer as the source VM2825. The distributed slice selector 2830 may be implemented by a software transfer element running on the host computer, by a service insertion layer between the VM and the software transfer element, or by other components as described above.

Data traffic from the VM2820 is allocated to a first network slice with four network services implemented as a VNF. As shown, the first VNF2835 is mounted in the main office data center 2805. For example, in some embodiments, a firewall can be used to filter the data traffic in the initial data center before sending the data traffic over the WAN to other data centers. Data traffic sent to the public cloud via WAN gateways (MPLS gateways, SD-WAN gateways, etc.) after initial VNF2835 processing, where additional network services 2840-2850 for the selected network slices are implemented. To. After processing by the three network services 2840-2850, this data traffic is sent to the branch office via another set of WAN gateways, where the data is delivered to the host at the destination VM2825. The reverse slice selector 2860 in this host (implemented by, for example, a software transfer element, service insertion layer, or other component) is to slice mapping information for use in allocating return traffic to the same network slice. Store the connection.

In some embodiments, as described above, the slice selector 2830 (up to at least data traffic sent to another data center at which the service chaining module in that data center handles service chaining). Handle service chaining. Thus, the slice selector 2830 not only determines which network slice the data message is assigned to, but also the location of the next network service within the selected network slice (eg VM, container in the current data center). , Or a physical device, another data center). For example, in the example shown in FIG. 28, the slice selector 2830 assigns the data message from the VM2820 to the first network slice, sends the data message to the first VNF2835, and then receives the data message after processing by the VNF. return. Next, the slice selector determines that the next network service is located in the public cloud 2815 and sends the data message to the WAN gateway so that the data message can be sent to the public cloud 2815. For distributed slice selectors (eg, with slice selectors and / or service chaining implemented in the service insertion layer, or these features implemented by software transfer elements on each host), the data message is actually the host. Instead of being returned to the slice selector on the computer, the service chaining module running on the same host as the first VNF2835 determines that the next service in the selected slice is located in the public cloud 2815 and the data Note that the message is sent to the public cloud 2815.

It should also be noted that in the example of FIG. 28, the second network slice (not selected for the data traffic shown) is at least partially implemented in a different public cloud 2855. That is, if the slice selector 2830 allocates data traffic from the VM2820 to a second network slice, as indicated by the dashed line, this traffic is in the public cloud 2855 via the WAN gateway before being transmitted to the branch office 2810. Sent to network services.

FIG. 29 conceptually illustrates that VSNs may be implemented to handle communication between guest VMs and public internet traffic in a public cloud within a public cloud or set of public clouds. In some cases, traffic between endpoint devices (eg, mobile devices, IoT devices, laptops or desktop computers) is the first in the telecommunications service provider access network (as shown in the various figures above). It is processed by the VSS, accesses the public cloud where the destination web server is located via the Internet, and is processed by the second VSS in the public cloud before reaching the web server. Similarly, return traffic is processed by the second VSN (in the manner shown in FIG. 29), routed over the Internet, and then processed by the first VSN before being delivered to the endpoint device.

As shown in the figure, the guest VM2905 in the host computer in the first public cloud 2910 sends a data message to a public internet destination. This destination may be, for example, a user endpoint device (eg, VM2905 acting as a server in response to a data request from a client device), or another destination in a public or private cloud. In this case, the data message is sent to the slice selector 2915 implemented on a host computer different from VM2905. This slice selector assigns the data message to the first network slice and sends the data message to the first two network services 2920 and 2925 in the first data center. In this example, the slice selector 2915 is centralized (within the data center 2910) rather than distributed across all host computers hosting guest VMs connected to the VSS. In some embodiments, the guest VM2905 is configured to use the slice selector 2915 as its IP gateway so that all traffic is first sent to the slice selector 2915 for processing. Different embodiments may use a single slice selector for a data center as shown in this figure (eg, as a transfer element performed in a VM or VM), as shown in FIGS. 27 and 28. Distributed slice selector may be used.

As mentioned above, in different embodiments, the data message is returned to the slice selector 2915, or distributed service chaining is used, and the data message is sent directly from VNF2920 to VNF2925 (ie, to the host on which the slice selector 2915 is implemented). Passed (via a transfer element on these host computers) without returning.

Next, since the third and fourth network services 2930 and 2935 for the selected network slice are implemented on the host computer in the second public cloud 2940, the data message is sent to this data center via the WAN gateway. Sent to 2940. There, the service chaining module (or distributed service chaining layer) sends data messages to network services 2930 and 2935, which are then implemented on the same host computer. Finally, after the network slice is complete, the data message is sent to the public internet via the output gateway 2945. This egress gateway, in some embodiments, behaves like the egress gateway 230 and stores connections to slice mapping states for use in allocating reverse traffic received from the Internet.

In some embodiments, the VSN consists of one or more data centers in the same manner as shown above for the telecommunications service provider network of FIG. 4 (eg, as shown in FIGS. 27-29). To. That is, the administrator has a slice selector, various network slices and implementations for each service (each network service should be implemented in a data center), how different data centers are connected, and configurations. Access to top-level VSS managers / controllers to provide configurations for other embodiments. The VSS controller uses multiple controllers in each data center to provide VNFs (or other form factors for network services if used), slice selectors and / or service chain modules (if required), data. Configure forwarding elements that tunnel traffic within the center, WAN gateways that send traffic between data centers, and other components.

FIG. 30 conceptually shows an electronic system 3000 in which some embodiments of the present invention are implemented. The electronic system 3000 may be a computer (eg, a desktop computer, a personal computer, a tablet computer, a server computer, a mainframe, a blade computer, etc.), a telephone, a PDA, or any other type of electronic device. Such electronic systems include various types of computer readable media and interfaces for various other types of computer readable media. The electronic system 3000 includes a bus 3005, a processing unit 3010, a system memory 3025, a read-only memory 3030, a permanent storage 3035, an input device 3040, and an output device 3045.

Bus 3005 collectively represents all system buses, peripheral buses, and chipset buses that communicably connect a large number of internal devices of the electronic system 3000. For example, the bus 3005 connects the processing unit 3010 communicably with the read-only memory 3030, the system memory 3025, and the permanent storage 3035.

From these various memory units, the processing unit 3010 extracts an instruction to be executed and data to be processed in order to execute the processing of the present invention. The processing unit may be a single processor or a multi-core processor in different embodiments.

The read-only memory (ROM) 3030 stores static data and instructions required by the processing unit 3010 and other modules of the electronic system. On the other hand, the permanent storage 3035 is a literate storage device. This device is a non-volatile memory unit that stores instructions and data even when the electronic system 3000 is off. Some embodiments of the present invention use large capacity storage devices (such as magnetic or optical discs and their corresponding disk drives) as permanent storage 3035.

In another embodiment, a removable storage device (floppy disk (registered trademark) disk, flash drive, etc.) is used as permanent storage. Like the permanent storage 3035, the system memory 3025 is a literate storage device. However, unlike the storage device 3035, the system memory is a volatile read / write memory such as a random access memory. The system memory stores some of the instructions and data that the processor needs at run time. In some embodiments, the process of the invention is stored in system memory 3025, permanent storage 3035, and / or read-only memory 3030. From these various memory units, the processing unit 3010 extracts an instruction to be executed and data to be processed in order to execute the processing of some embodiments.

The bus 3005 also connects to the input / output devices 3040 and 3045. The input device allows the user to convey information and select commands to the electronic system. The input device 3040 includes an alphanumerical keyboard and a pointing device (also referred to as a "cursor control device"). The output device 3045 displays an image generated by the electronic system. Output devices include a printer and a display device such as a cathode ray tube (CRT) or liquid crystal display (LCD). Some embodiments include devices such as touch screens that serve as both input and output devices.

Finally, as shown in FIG. 30, the bus 3005 also couples the electronic system 3000 to the network 3065 via a network adapter (not shown). In this way, the computer may be a network of computers (local area network (“LAN”), wide area network (“WAN”), or part of an intranet, or a network of networks such as the Internet. Any or all components of the electronic system 3000 can be used in conjunction with the present invention.

Some embodiments are microprocessors, storage devices and devices that store computer program instructions on a machine-readable or computer-readable medium (alternatively referred to as a computer-readable storage medium, machine-readable medium, or machine-readable storage medium). Includes electronic components such as memory. Some examples of such computer-readable media include RAM, ROM, read-only compact discs (CD-ROMs), recordable compact discs (CD-R), rewritable compact discs (CD-RW), and read-only. Digital general-purpose discs (eg, DVD-ROM, double-layer DVD-ROM), various recordable / rewritable DVDs (eg, DVD-RAM, DVD-RW, DVD + RW, etc.), flash memory (eg, SD card, mini SD). Cards, micro SD cards, etc.), magnetic and / or solid state hard drives, read-only and recordable Blu-ray® discs, ultra-dense optical discs, any other optical or magnetic medium, and floppy disks. The computer-readable medium may be executed by at least one processing unit and may contain a computer program containing a set of instructions for performing various operations. Examples of computer programs or computer code include machine code as generated by a compiler and files containing higher level code executed by a computer, electronic component, or microprocessor using an interpreter.

The above discussion primarily refers to microprocessors or multi-core processors running software, but some embodiments may be one such as an application specific integrated circuit (ASIC) or field programmable gate array (FPGA). It is executed by multiple integrated circuits. In some embodiments, such integrated circuits execute instructions stored in the circuit itself.

As used herein, the terms "computer," "server," "processor," and "memory" all refer to electronic devices or other technical devices, and these terms exclude people or groups of people. Is. For the purposes of this specification, the term display means to be displayed on a display or electronic device. As used herein, the terms "computer-readable medium", "computer-readable medium", and "machine-readable medium" are entirely limited to tangible physical objects that store information in a computer-readable format. .. These terms exclude all radio signals, wired download signals, and other ephemeral signals.

As a whole, the present specification refers to a computing environment and a network environment including a virtual machine (VM). However, a virtual machine is just one example of a data calculation node (DCN) or data calculation end node, also called an addressable node. The DCN may include a non-virtualized physical host, a virtual machine, a container running on a host operating system that does not require a hypervisor or a separate operating system, and a hypervisor kernel network interface module.

VMs, in some embodiments, use host resources virtualized by virtualization software (eg, hypervisors, virtual machine monitors, etc.) to run on their own guest operating system on the host. The tenant (ie, the owner of the VM) can choose which application to run on the guest operating system. On the other hand, some containers are constructs that run on the host operating system without the need for a hypervisor or a separate guest operating system. In some embodiments, the host operating system uses namespaces to separate containers from each other, thus providing operating system-level isolation for different groups of applications running within different containers. This isolation is similar to the VM isolation provided in a hypervisor virtualization environment that virtualizes system hardware and can therefore be viewed as the formation of virtualization that isolates different groups of applications running in different containers. can. Such containers are lighter than VMs.

The hypervisor kernel network interface module is, in some embodiments, a non-VM DCN comprising a hypervisor kernel network interface and a network stack having receive / transmit threads. One example of a hypervisor kernel network interface module is the VMknic module, which is part of VMware, Inc's ESX hypervisor.

While the specification refers to VMs, it should be understood that given examples may be any type of DCN including physical hosts, VMs, non-VM containers, and hypervisor kernel network interface modules. Is. In fact, the exemplary network can include different types of DCN combinations in some embodiments.

Having described the invention with reference to a number of specific details, one of ordinary skill in the art will appreciate that the invention can be practiced in other particular embodiments without departing from the spirit of the invention. .. In addition, some figures (including FIGS. 9, 11, and 12) conceptually illustrate the process. Certain actions of these processes do not have to be performed in the exact order shown and described. The specific operation may not be performed in one continuous series of operations, and different specific operations may be performed in different embodiments. In addition, this process can be performed using some sub-processes or as part of a larger macro process. Accordingly, one of ordinary skill in the art will appreciate that the invention should not be limited by the above exemplary details, but rather should be defined by the appended claims.

Claims (23)

A method of establishing multiple virtual service networks through multiple data centers.
For each virtual service network among the plurality of virtual service networks, to configure a set of machines distributed across the plurality of data centers in order to implement an ordered set of network services for the virtual service network. ,
Receiving a data message, selecting one of the plurality of virtual service networks for the data message based on analysis of the content of the data message, said network service for the selected virtual service network. Determining a location within the plurality of data centers for a machine that implements the first network service of the sequence set, and sending the data message to the machine that implements the first network service. By configuring multiple service network selectors running within the multiple data centers,
Including, how. The method according to claim 1, wherein the plurality of data centers include a plurality of public cloud data centers. The method of claim 1, wherein the plurality of data centers comprises at least one public cloud data center and at least one private data center. The plurality of virtual service networks are (i) a first virtual service network comprising a first sequence set of network services implemented by a first set of machines operating in the plurality of data centers, and (ii). ) The method of claim 1, comprising at least a second virtual service network comprising a second sequence set of network services implemented by a second set of machines operating in the plurality of data centers. The method is performed by a virtual service network controller.
Configuring a particular set of machines is for the set of machines to a first controller for instantiating at least one of the virtual machines and containers on the host computer in at least one of the data centers. Including providing configuration data for
Configuring the plurality of service network selectors includes providing configuration data for a service network selection operation to a second controller for configuring the service network selector in at least one of the data centers. , The method according to claim 1. It is specific to (i) provide the configuration data for the set of machines to the first controller and (ii) provide the configuration data for the service network selection operation to the second controller. Including providing a virtual service network policy to an agent running in a data center of the agent, the agent instantiates and configures the virtual service network policy, (i) the first controller instantiates and configures the set of machines. 5. The method of claim 5, wherein the second controller converts data for configuring the service network selector to perform (ii) the service network selection operation. (I) The data message is received from an external endpoint device in a particular service network selector and (ii) processed by the network service sequence set for the selected virtual service network, and then the data message. 1 is the method according to claim 1, wherein is transmitted from the data center to a public network. The particular service network selector runs in the first data center, receives data messages from external endpoint devices located within the first geographic range, and additional service network selectors run in other data centers. 7. The method of claim 7, wherein the data message is received from an external endpoint device located within a second geographic range. The method of claim 1, wherein the data message is received by a particular service network selector running on a particular host computer in the particular data center of the plurality of data centers. The data message is received from the data compute node, the data compute node also runs on the particular host computer, and an additional set of service network selectors runs on the additional host computer in the particular data center. The method according to claim 9. The machine running the first network service runs on an additional host computer in the particular data center, and the service chaining module running on the additional host computer is the selected virtual. The location within the plurality of data centers for a machine that implements the second network service of the sequence set of network services for a service network and the data message that implements the second network service. The method of claim 10, which is transmitted to the machine. The data compute node is a first data compute node that runs on the first host computer, and after being processed by the network service sequence set for the virtual service network, the data message is the data. 10. The method of claim 10, which is transmitted to a second data compute node running on a second host computer in one of the centers. The first data compute node and the second data compute node run on host computers in two different private data centers, and at least a subset of the network services are on host computers in at least one public data center. 12. The method of claim 12. The particular service network selector is the first service network selector.
The second service network selector is executed on the second host computer.
The second service network selector stores data about the data message used in selecting the same virtual service network for the return data message from the second data compute node.
The method according to claim 12. A non-temporary machine-readable medium that stores a program that establishes multiple virtual service networks through multiple data centers when executed by at least one processing unit.
For each virtual service network among the plurality of virtual service networks, to configure a set of machines distributed across the plurality of data centers in order to implement an ordered set of network services for the virtual service network. ,
Receiving a data message, selecting one of the plurality of virtual service networks for the data message based on analysis of the content of the data message, said network service for the selected virtual service network. Determining a location within the plurality of data centers for a machine that implements the first network service of the sequence set, and sending the data message to the machine that implements the first network service. By configuring multiple service network selectors running within the multiple data centers,
A non-temporary machine-readable medium containing a set of instructions. The plurality of virtual service networks are (i) a first virtual service network comprising a first sequence set of network services implemented by a first set of machines operating in the plurality of data centers, and (ii). ) The non-temporary according to claim 15, comprising at least a second virtual service network comprising a second sequence set of network services implemented by a second set of machines operating in the plurality of data centers. Machine readable medium. The program is a virtual service network controller
The set of instructions that make up a particular set of machines has configuration data for the set of machines in a first controller for instantiating a virtual machine on a host computer in at least one of the data centers. Includes a set of instructions to provide
The set of instructions constituting the plurality of service network selectors provides configuration data for a service network selection operation to a second controller for configuring the service network selector in at least one data center. Including a set of instructions,
The non-temporary machine-readable medium of claim 15. A set of instructions that (i) provide the configuration data for the set of machines to the first controller and (ii) provide the configuration data for the service network selection operation to the second controller. Includes the set of instructions that provide virtual service network policies to agents running in a particular data center, where the agents set the virtual service network policies, and (i) the first controller sets the machines. 17. The data for instantiating and configuring the service network selector and (ii) the data for the second controller to configure the service network selector in order to execute the service network selection operation. Non-temporary machine-readable medium. The data message is received from an external endpoint device at a particular service network selector and
After being processed by the network service sequence set for the selected virtual service network, the data message is sent from the data center to the public network.
The particular service network selector runs in the first data center, receives data messages from external endpoint devices located within the first geographic range, and additional service network selectors run in other data centers. And receive data messages from external endpoint devices located within a second geographic range,
The non-temporary machine-readable medium of claim 15. The data message is received by a particular service network selector running on a particular host computer in a particular data center in the plurality of data centers.
The data message is received from the data compute node, which is also executed on the particular host computer.
A set of additional service network selectors is run on the additional host computers in the particular data center.
The machine running the first network service is run on an additional host computer in the particular data center.
The service chaining module running on the additional host computer is the plurality of data centers for a machine that implements a second network service of the network service sequence set for the selected virtual service network. Determines its location within and sends the data message to the machine implementing the second network service.
The non-temporary machine-readable medium of claim 15. A set of host computers running in multiple data centers,
A computing device that executes a virtual service network controller to establish a plurality of virtual service networks through the plurality of data centers.
For each virtual service network out of the plurality of virtual service networks, a set of machines running on the host computers in the plurality of data centers to implement a sequence of network services for the virtual service network. To configure and
Receiving a data message, selecting one of the plurality of virtual service networks for the data message based on analysis of the content of the data message, said network service for the selected virtual service network. Determining a location within the plurality of data centers for a machine that implements the first network service of the sequence set, and sending the data message to the machine that implements the first network service. To configure multiple service network selectors to run on the host computers in the multiple data centers,
Because of the computing device and
The system. The plurality of virtual service networks may include (i) a first sequence of network services implemented by a first set of machines running on a first subset of the host computers in the plurality of data centers. A second sequence of network services implemented by a first virtual service network comprising and (ii) a second set of machines running on a second subset of said host computers in said plurality of data centers. 21. The system of claim 21, comprising at least a second virtual service network comprising. (I) The data message is received from an external endpoint device in a particular service network selector and (ii) processed by the network service sequence set for the selected virtual service network, and then the data message. 21. The system according to claim 21, wherein is transmitted from the data center to a public network.

Images