DE112020004639T5 - MANAGEMENT OF DISTRIBUTED ENDPOINTS - Google Patents

Abstract

A system and method for managing communications for components in a global accelerator system is provided. The global accelerator system includes client devices that communicate with a global access point over a public network to access various endpoints that can host services global access point, in turn, communicates over a private network with various endpoints that are organized in different data centers. To facilitate exchanges on behalf of the different client devices, the global access points can characterize groupings of endpoints into subgroups or meshes. Individual mesh communications can be encapsulated and are routed in packets in the data center by network equipment using 5-tuple information. The components within individual meshes can carry status information via management messages broadcast or multicast to ensure failover or mitigate double processing.

Description

BACKGROUND ART

Generally described, computing devices utilize a communications network or series of communications networks to exchange data. Businesses and organizations operate computer networks that connect a variety of computing devices to support operations or provide services to others. The computing systems may be located in a single geographic location or located in multiple different geographic locations (e.g., interconnected via private or public communications networks). In particular, a data center or data processing center, collectively referred to herein as a "data center," may include a number of interconnected computing systems to provide computing resources to users of the data center. The data centers can be private data centers operated on behalf of an organization or public data centers operated on behalf of or for the benefit of the general public.

Because the resources of each individual computing device are limited, a variety of techniques are known to attempt to balance resource usage between devices. For example, a "load balancer" device may be placed between client devices requesting use of computing resources and servers providing such resources. The load balancer can attempt to distribute the requests across the servers, allowing the servers to work together to provide such resources. A major disadvantage of using such a load balancer is that it can create a single point of failure such that if the load balancer fails, requests to the servers will fail and resources will not be made available.

Another type of load balancing known in the art is Domain Name System (DNS) load balancing. DNS generally refers to a network of devices that operate to translate human-recognizable domain names into network addresses on a network such as the Internet. To enable load balancing using DNS, a DNS server is populated with network addresses of multiple computing devices offering a particular resource. When responding to requests to resolve a domain name associated with that resource, the DNS server alternates which addresses are served, causing requests for the resources to be split among the addresses served. Because the DNS server does not act as a conduit for traffic to the resources, and DNS generally provides highly redundant operation, DNS load balancing generally avoids a single point of failure. However, a major disadvantage of DNS load balancing is the delay required to change a load balancing scheme. DNS queries generally flow through a series of DNS resolvers, each of which can cache previous DNS results for a period of time. Therefore, changes made to a DNS server to change how the load is distributed between servers can take a long time to propagate. Especially in cases where a server is down, these delays can cause significant errors in network communications. In addition, DNS records can be difficult to keep up-to-date as they generally need to be modified as new devices are added or removed from a load balancing scheme.

Yet another type of load balancing known in the art is the use of "anycast" network addresses. In a network such as the Internet, different Autonomous Systems ("AS") provide devices with different network addresses. Each AS notifies its neighboring AS of the addresses available in its network by "offering" the addresses. Most typically, each address is associated with a single location (e.g., a single device or a group of devices). In an anycast configuration, multiple devices, often in multiple ASs, advertise the same network address. Depending on the configuration of neighboring ASs, client requests to access the address can then be directed to any of the multiple devices, thereby sharing the load between the devices. A major disadvantage of using anycast when attempting load balancing is that routing to the anycast address is generally controlled by neighboring networks. These networks are often controlled by other entities. Thus, it is difficult or impossible to fully control how requests are distributed between devices with an anycast network address. In addition, when the configuration of neighboring networks changes, the distribution of requests can also change, leading to volatility in the load-balancing scheme.

character list

• 1 12 is a block diagram depicting an illustrative logical network that includes multiple client devices and data centers, and a set of global access points that provide load-balanced access to the data centers from a set of global network addresses.
• 2 FIG. 14 is a block diagram showing an illustrative configuration of a data center 1 represents.
• 3 Fig. 12 is a block diagram showing an illustrative configuration of a global access point 1 represents.
• 4 Figure 12 is a block diagram depicting an illustrative configuration of a data flow manager server that hosts one or more data flow managers in a global access point 1 implemented.
• 5 provides illustrative interactions for routing a request from a client device addressed to a global network address to a global access point 1 represent.
• 6 provides illustrative interactions for routing a request from a global access point to a data center 1 based, at least in part, on load-balancing requests between data centers.
• 7 provides illustrative interactions for propagating endpoint information from a data center to global access points 1 so that the access points can correctly route client device traffic addressed to global network addresses.
• 8th Figure 12 provides an illustrative routine for increasing resiliency of global network addresses by selecting different neighboring devices that have different global network addresses from the access points of the 1 are to be offered.
• 9 provides an example routine for routing traffic addressed to a global network address associated with a service provided by endpoints within data centers using the access points of FIG 1 represent.
• 10 provides an illustrative routine for updating information at the global access points of the 1 in relation to data center endpoints that provide a network accessible service.
• 11 Figure 12 illustrates illustrative interactions for establishing a Transmission Control Protocol (TCP) session at a global access point 1 and handing off the TCP session to an endpoint to allow a client device and the endpoint to communicate over the TCP session.
• 12 Figure 12 provides an illustrative routine for establishing a Transmission Control Protocol (TCP) session at a global access point 1 and handing off the TCP session to an endpoint to allow a client device and the endpoint to communicate over the TCP session.

DETAILED DESCRIPTION

Broadly described, aspects of the present disclosure relate to providing load-balanced access to a pool of computing devices distributed across multiple geographic locations using one or more global network addresses. More particularly, aspects of the present disclosure relate to providing a distributed set of access points reachable via global network addresses, where the access points select and route requests to endpoint devices within the pool based at least in part on load balancing the requests. In one embodiment, the access points use anycast routing techniques to advertise the availability of global network addresses associated with the pool of computing devices, thereby attracting traffic addressed to those addresses. Upon receiving a request to access the pool, an access point can select an appropriate endpoint within the pool based on a distribution algorithm that allows data packets to be distributed to different groupings of endpoints, commonly referred to as an endpoint group. The access point can then act as a proxy, routing the request to the endpoint and enabling further communications between the endpoint and a requesting device. The access point as disclosed in this document, can implement a variety of techniques to provide resilient, efficient access to the pool of endpoint devices. As disclosed in this document, the access points can be distributed over a wide geographic area, thereby eliminating single points of failure within the system. Additionally, by using anycast offerings, the access points can distribute requests among the pool even if the requests are addressed to a single network address, avoiding complications and delays associated with other techniques such as DNS-based load balancing. By acting as a proxy between clients and the device pool, rather than providing resources directly, the access points can control the distribution of requests to the pool regardless of how external devices route the requests to the anycast address, avoiding disadvantages associated with traditional Anycast networks are linked. Thus, embodiments disclosed herein greatly improve upon known load balancing techniques.

Illustratively, embodiments of the present disclosure may be implemented in a wide geographic area. In one embodiment, the present disclosure is implemented on the global Internet and provides global Internet Protocol (IP) addresses, such as IP version 4 (IPv4) or IP version 6 (IPv6) addresses. Different data centers may exist in different geographic locations, and each data center may include one or more endpoint devices that provide access to a network-based service. Examples of such services include, but are not limited to, web hosting, data storage, on-demand computing services, and the like. The resources in each data center may be limited, and therefore an operator of the network-based service may wish to load-balance such services. To simplify the operation of the network-based service (e.g. to avoid the complications associated with DNS-based load balancing), it may be desirable to provide the operator with a single set of relatively static network addresses for the service, whose network addresses are independent of the individual endpoints that provide access to the service. Such network addresses are generally referred to in this document as "global" network addresses.

As used herein, the term "global" is intended to refer to the scope of the network address related to the Service (e.g., that the network address applies to the entire Service and not to individual devices) and does not necessarily imply that a such network address is accessible worldwide. Nonetheless, embodiments of the present disclosure can be implemented to provide global network addresses that are generally accessible from a worldwide network, such as the Internet.

In order to provide global network addresses for a service, a system is disclosed that provides a set of geographically distributed global access points. As noted above, in this context, the term "global" access point generally refers to the access point that provides access to the Service generally, as opposed to individual endpoints, and does not necessarily imply that such access points exist worldwide (although such a configuration is possible). In general, however, the access points are geographically distributed. In one embodiment, the global access points are in different and more numerous geographic locations than the data centers that provide endpoints for the service, which reduces the average network distance between the global access points and client devices attempting to access the service.

Each global access point can use anycast techniques to advertise the availability of the service via the one or more global network addresses associated with the service. As an illustration, each access point can use the Border Gateway Protocol ("BGP") to advertise the global network addresses by including a BGP "speaker" to announce the availability of the global network address to neighboring networks. The global access point can thus attract traffic addressed to the global network addresses. As will be disclosed in more detail below, in some cases the global access points may "shape" BGP advertisements to increase the resiliency of a network-based service against network disruptions. For example, the global access points can divide their advertised global network addresses into two groups and allocate at least one network address from each group to a service. The global access points can then advertise each group of addresses to another neighboring network. In this way, each access point can effectively provide two paths to reach the point: through a first neighbor using an address from the first group, or through a second neighbor using an address from a second group. So if a neighboring network fails in any way, an old one is native path exists to reach the access point. In addition, since any access point can operate in this way, if one access point fails completely, traffic can be automatically routed to another access point using traditional anycast routing mechanisms. While examples are provided in this document that relate to two groups, any number of groups may be provided.

In some cases, to provide even more resiliency, each access point may segregate traffic according to a specific subset of the offered global network addresses and use different devices or program threads to process traffic addressed to each subset. In this way, if the traffic of one subset is problematic for the operation of the access point (e.g. due to a misconfiguration with respect to a global network address of the subset), the operation of one device or thread handling traffic of another is unlikely Subset is affected. In one embodiment, the access point uses a combination of different network address groups and separate processing of subsets of each network group. In addition, services can be "reordered" between the subsets of each group, such that if two services are assigned network addresses within a common subset of one group, the two services are likely to be assigned network addresses within different subsets of a second group. With this configuration, if the configuration of a particular service causes problems in the operation of an access point with respect to a subset of a group, other services would probably still be reachable through an alternative group (where their addresses are probably not in the same subset as the problematic service) . Increasing the number of address groups and subsets can gradually reduce the total number of services affected by a problematic service.

Upon receiving a request to access a service, a global access point can be configured to route traffic to an appropriate endpoint within a data center providing the service. To do this, the global access point may need to know a network address of the endpoint. In one embodiment, each data center is configured with a resource manager device that maintains a list of endpoints that provide access to a service. The resource manager can inform the endpoints' access points, including network address information for the endpoints. Thus, upon receiving a request for access to a service, each access point may be configured to select an available endpoint for the service and route the request to that endpoint. As discussed below, the endpoint may be selected based on a distribution algorithm that selects a set of endpoints based on a product of geographic criteria and distribution criteria. For example, the global access point can apply a percentage distribution using a coin toss algorithm to different geographic subsets of a data center. Additionally, the global access point can then apply a selection algorithm to select individual endpoints based on processing attributes of the request (e.g., 5-tuple information) to ensure that a selected endpoint is consistently selected. For example, the global access point may implement a scoring algorithm that enables consistent selection of an endpoint based on a highest score.

In one embodiment, to route traffic to an endpoint, each access point is configured to utilize network address translation ("NAT"). NAT techniques are known in the art and are therefore not described in detail in this document. In general, however, NAT techniques allow a device to act as a middleman between two devices while aspects of each data packet, such as a source and destination network address, are rewritten to facilitate communication between the devices. According to embodiments of the present disclosure, each access point is operable to replace a requesting device's source network address with its own network address (e.g., a unicast network address that uniquely identifies the global access point) and the destination network address (e.g., the global network address of a service) with a network address of an endpoint that provides this service. The access point can then route the data packet to the endpoint, receive a response from the endpoint (if any), perform a reverse translation of the source and destination (e.g. the source of the response by the global network address and the destination of the response by the network address replace the requesting device) and return the data packet to the requesting device. In one embodiment, the access point may use port translation (known in the art) to help distinguish traffic flows (series of contiguous packets) when using NAT to ensure correct translation of addresses when processing traffic from multiple requesting devices .

Advantageously, the use of NAT allows the access point to route traffic received from a client device to an endpoint with minimal disruption to a connection between the client device and the endpoint. For example, using NAT eliminates the need for an access point to act as an "endpoint" for certain network protocols, such as Transmission Control Protocol (TCP). For example, alternative routing techniques may cause an access point to accept a TCP connection from a client device and initiate a second TCP connection to an endpoint. Maintaining multiple such connections (and equivalents between them) can significantly increase an access point's resource utilization, reducing its capacity to route traffic addressed to global network addresses. Thus, the use of NAT can offer advantages over such techniques.

In addition, the use of NAT at an access point allows traffic to be directed to a global network address that is different from any address used by an endpoint, thereby preventing disclosure of the actual address used by the endpoint. Thus, the endpoint can be protected from receiving traffic directly. This, in turn, can reduce or eliminate the endpoint's vulnerability to network attacks, such as denial-of-service (DoS) attacks. In this way, the global access points can provide benefits similar to those provided by traditional content delivery networks (CDN). However, unlike CDN, the access points do not have to replicate the functionality of the service (e.g. by implementing a web server at each dial-up node within the CDN). Rather, the access points of the present disclosure may allow traffic to flow through the access point, thereby allowing data packets from a requesting device to reach an endpoint providing the service. For this reason, among others, the global access points disclosed in this document can provide a greater range of functions than CDN, which enables load balancing and distribution to any number of different network-based services.

In another embodiment, each access point is configured to use encapsulation to route traffic to an endpoint. Encapsulation is a well-known network technology and is therefore not described in detail in this document. In general, however, encapsulation can be used to add additional information (often in the form of a header, sometimes in the form of a trailer) to a data packet, thereby "wrapping" or encapsulating the data packet to obtain an encapsulated data packet. In the context of the present disclosure, encapsulation can be used to provide a "network tunnel" between each access point and each endpoint. In a specific example, the global access point may establish a set of virtual LAN ("VLAN") channels between subsets of endpoints to enable more secure communication over encapsulated tunnels. Broadly described, VLAN corresponds to a broadcast domain that is partitioned and isolated at the data link layer in a computer network. The global access point can implement different VLANs to communicate with individual endpoints or subsets of endpoints by applying tags to network frames and manipulating these tags in network systems, e.g. B. Packet processing by network components. The global access points can therefore implement different VLANs to keep network communication separate across client devices and endpoints that implement applications, even though they are connected to the same physical network. Accordingly, when a data packet is received at an access point from a client device and addressed to a network address of a service provided by an endpoint, the access point can encapsulate the data packet with additional information that allows the data packet to be routed to a selected endpoint , such as a network address of the endpoint. The encapsulated data packet can then travel over a physical network to the endpoint via a network component such as a top-of-rack ("TOR") switch that can use 5-tuple information to distribute incoming data packets to different assigned VLANs be routed. The endpoint can "decapsulate" the encapsulated data packet to remove the additional information and process the deencapsulated data packet as if it were received directly from the client device. In still other aspects of the present application, the endpoint devices may be assigned different subsets of packet processing responsibilities, commonly referred to as fabrics, implemented on different VLANs, facilitating resiliency in state recovery. More specifically, individual endpoints associated with a particular VLAN can exchange notification information, such as Path MTU discovery (“PMTUD”) messages, to ensure communications to endpoints are not processed redundantly, or to provide status information for a replace failsafe. The exchanged notification information may be multicast within a particular fabric or broadcast across multiple fabrics. For example, other endpoints be configured to ignore or filter broadcast communications not associated with an assigned fabric.

In one embodiment, the endpoint may respond to the client device directly over a physical network connecting the endpoint to the client device. In another embodiment, the endpoint may respond to the client device by encapsulating a response packet and transmitting the encapsulated response packet back to the access point. In turn, the access point can decapsulate the response packet and return the response packet to the client. Returning responses directly from an endpoint to a client device may advantageously reduce the workload of an access point and may also reduce traffic on a physical network connecting the access point to the endpoint. However, direct return of responses may be required to utilize a physical network connecting the endpoint to the client device, which may not be as robust as the physical network connecting the access point to the endpoint. For example, if the endpoint and access point are connected through a private network, while the access point, endpoint, and client device are connected through a public network (such as the Internet), returning responses through the access point (rather than a direct return) may be preferred as it may be preferred to minimize the total distance traveled in the public network (e.g. when the access point is closer to the client device than the endpoint).

In general, encapsulating packets between access points and endpoints can increase the computational resource utilization of access points and endpoints compared to using NAT to route packets from access points to endpoints. However, encapsulation can also provide advantages over a NAT implementation. For example, encapsulation can provide resiliency of a TCP session in case data packets of a client device are routed to different global access points during the TCP session. Such routing may be done, for example, based on the operation of networks between the client device and the different access points due to potential changes in how the networks process anycast offers from the different access points. When an access point uses NAT to route data packets to an endpoint, each access point can independently manage NAT information. If client device data packets are routed to a different access point, that second access point may not have sufficient information to properly handle the client's communication (e.g., the second access point's transformed data packets may not be identical to what generated at the original access point). This can cause the client device and endpoint TCP session to be broken, requiring the client device to reestablish the TCP connection.

In contrast, encapsulation can allow for proper handling of the redirection of client requests between access points. With either NAT or encapsulation implementations, each access point can be configured to choose an endpoint in a consistent manner. As described above, the access points may apply a selection algorithm that generates a score that indicates a desired data center. Such selection algorithms may include, but are not limited to, weighted rendezvous hashing or weighted consistent hashing algorithms. Thus, client device packets would be expected to be routed to the same endpoint regardless of the access point to which they were originally sent. Furthermore, by using encapsulation, no data of the client device's initial data packets need to be modified when the data packets are routed to the endpoint. Rather, this data may be encapsulated (e.g., with additional information enabling routing to the endpoint) and retrieved at the endpoint after de-encapsulation. Because of this, the final data available to the endpoint after decapsulation can be the same even if client data packets are routed to different access points, allowing the endpoint to maintain a TCP session with the client device even if Data flows through different access points.

To further improve performance in implementations that use encapsulation, in some embodiments each access point may be configured to assist endpoints in establishing connection-oriented communication sessions, such as TCP sessions, with client devices. Connection-oriented communication sessions generally require a session initialization phase in which the two participants in the session communicate to establish a mutual understanding of the communication status. For example, TCP sessions use a "three-way handshake." The TCP three-way handshake is well known in the art and is therefore not described in detail in this document. In short, however, a TCP session is established when a first participant initiates a synchronisa communication packet ("SYN") to a second party (including a first party's sequence number to be used during the session), the second party with a synchronization acknowledgment ("SYN-ACK") packet (confirming the first party's sequence number and a second party's sequence number) and the first party responds to the SYN-ACK data packet with an acknowledgment ("ACK") packet (acknowledging the second party's sequence number). Since the three-way handshake requires three separate communications between the participants, an increase in latency between the participants during the handshake is magnified by a factor of three, commonly referred to as "first byte latency". For example, in the context of the present disclosure, if communications between a client device and an endpoint incur a 100 millisecond (ms) latency, the three-way handshake can be expected to last at least 300 milliseconds. Since such a handshake is required prior to the transmission of data over a TCP session, it is advantageous to reduce the time required to initialize the TCP session, e.g. B. to reduce the latency of the first byte. As described below, in other embodiments, the TCP session between the global access point may further include the TCP session between the global access point and an intermediate device located on the selected endpoint, and a third TCP session (new) between the located one include intermediate device and the selected endpoint. Accordingly, the first byte latency may be further reduced by further limiting the latency between the endpoint and the located intermediate device.

Accordingly, in embodiments of the present disclosure, each global access point may be configured to perform a TCP three-way handshake with a client device and then "drop off" the TCP connection to an endpoint. In particular, each access point may be configured to "listen" for incoming SYN data packets from client devices and respond to such data packets by performing the TCP three-way handshake. After the handshake is complete, the access point may transmit TCP session information (e.g., the sequence numbers of the first and second participants in the TCP session) to an endpoint selected to provide a requested service to the client device. Upon receiving the TCP session information, the endpoint can "accept" the TCP session and process data packets from the client device as if the TCP session had been established between the client device and the endpoint. Because the access point can be considered to be "closer" to the client device in terms of latency, the time to establish the TCP session is reduced without impacting the ability of the client device and endpoint to get over a common TCP session to communicate. While examples are provided in this document in relation to TCP handshakes, stateful communication sessions generally each require an initialization phase. Embodiments described herein may be used to perform such an initialization phase at an access point while passing context for the connection-oriented communication session to an endpoint to allow a client device and the endpoint to communicate over the connection-oriented communication session.

In addition, the global access points can be configured to modify or select different TCP-based configurations to deploy higher-bandwidth connections between the access points and the data centers. More specifically, according to aspects of the present application, TCP-based communications typically include various settings or parameters that can control the level of data transferred. Such parameters may be associated with a transmitted data set, commonly referred to as a windowing parameter. The parameters may also include a congestion control parameter, such as a slow start strategy used by TCP in conjunction with other algorithms to avoid sending more data than the network can carry, ie to avoid network congestion. For traditional TCP communications, including public network connections, data windowing and congestion control parameters are typically set at lower levels in the event multiple TCP connections share a common network connection, and may be increased when data throughput is measured and determined that larger amounts of data throughput are supported. This has the effect of increasing the latency of the communication, commonly referred to as last byte latency. According to aspects of the present application, the client computing device and the global access point have a first TCP connection configured with conventional approaches for congestion control parameters and windowing parameters (e.g., initial lower values that can be increased based on measured attributes of the connection). are. The global access point and endpoint have a second TCP connection that is not configured with the traditional approach because the global access point and endpoint use a second network communicate over a factory connection, such as a private network, which may be configured to support higher data throughput with less likelihood of congestion or less effective bandwidth. Accordingly, the global access point can set the congestion control parameters and windowing parameters to a maximum or higher initial value, which directly leads to greater data throughput. Thus, overall data throughput (e.g., last byte latency can be increased).

As will be apparent to those skilled in the art in light of the present disclosure, the embodiments disclosed herein improve the ability of computing systems to provide network accessible services. In particular, embodiments of the present disclosure improve on previous load balancing techniques by providing scalable, resilient, and responsive load balancing across a common network address, thereby improving on known load balancing techniques. In addition, the presently disclosed embodiments address technical issues inherent in computing systems, particularly the limited nature of the computing resources that can be used to provide network-accessible services and the difficulties in handling requests for such services in a scalable, resilient, and responsive manner for load balancing. These technical problems are addressed through the various technical solutions described in this paper, including the use of a distributed set of access points mapped to a common network address, each configured to receive requests for a service and the requests to endpoints of the service route based at least in part on load balancing the requests between the endpoints. Thus, the present disclosure represents an improvement on existing network load balancing systems and computing systems in general.

The foregoing aspects and many of the attendant advantages of this disclosure will become more apparent as a better understanding thereof is developed by reference to the following description in connection with the accompanying drawings.

1 10 is a block diagram depicting an example logical environment 100 that includes multiple client devices 102 in communication with a set of global access points 106A-N over a first network 104, where the global access points 106A-N are connected to a set of data centers 110A-N communicate via a second network 108 . As client devices 102, global access points 106, and data centers 110 reside in 1 located in clusters, client devices 102, global access points 106, and data centers 110 may be geographically distant and independently owned or operated. For example, client devices 102 could represent a variety of users in different global, continental, or regional locations accessing network-accessible services provided by data centers 110, which data centers may also be in different global, continental, or regional locations Locations can be distributed. The global access points 106 may be similarly distributed. In one embodiment, data centers 110 represent devices in locations controlled by a single entity, such as a "cloud computing" provider, while global access points 106 represent devices in shared locations, such as network "point-of-sale" nodes. or Internet Exchange Points (IXP). Global access points 106 may generally outnumber data centers 110 and be in different physical locations. However, in other embodiments, one or more of the access points 106 may be located in one or more data centers 110 . Accordingly, the groupings of client devices 102, access points 106, and data centers 110 in 1 represent a logical rather than a physical grouping.

The networks 104 and 108 can be any wired network, wireless network, or a combination thereof. Additionally, networks 104 and 108 may be a personal network, local area network, wide area network, cable network, satellite network, cellular network, or a combination thereof. In the exemplary environment of 1 For example, network 104 is a global network (Global Area Network - GAN), such as the Internet, while network 108 is a private network dedicated to traffic associated with an entity that provides data centers 110 and access points 106 . Protocols and components for communicating over the other aforementioned types of communication networks are well known to those skilled in the computer communications arts and therefore need not be described in further detail herein.

While each of the client devices 102 and each of the access points 106 is shown as having a single connection to the network 104, individual components of the client devices 102 and Access points 106 may be connected to network 104 at various points (e.g., via different neighboring networks within network 104). In some embodiments, data centers 110 may additionally or alternatively be connected to network 104 . Similarly, while each of the access points 106 and each of the data centers 110 are shown as having a single connection to the network 108, individual components of the access points 106 and data centers 110 may be connected to the network 108 at various points. Accordingly, communication times and capabilities between the components of the 1 vary. The network configuration of the 1 FIG. 1 is intended to illustrate a communication path in embodiments of the present disclosure and is not intended to represent all possible communication paths.

Client devices 102 may include any number of different computing devices capable of communicating with global access points 106 . For example, individual client devices 102 may be a laptop or tablet computer, a personal computer, a portable computer, a server, a personal digital assistant (PDA), a hybrid PDA-cellular phone, a cellular phone, an e-book reader, a set-top box, a camera, a digital media player, and the like. In some cases, client devices 102 are operated by end users. In other cases, the client devices 102 themselves provide network-accessible services that interact with the global access points 106 to access other network-accessible services.

The data centers 110 of the 1 10 illustratively include endpoint computing devices that provide one or more network accessible services on behalf of one or more service providers. Illustratively, the data centers 110 may be operated by a “cloud computing” provider that makes host computing devices within the data center available to service providers to provide their services. The cloud computing deployment in general can manage the operations of the data center while providing various mechanisms for the server providers to configure their respective endpoints. An illustrative configuration of a data center 110 is provided below with reference to FIG 2 provided.

According to embodiments of the present disclosure, the cloud computing provider may allow service providers to associate their endpoints with one or more global network addresses addressable in the network 104 to interact with the data centers 110 in a load-balanced manner. The cloud computing provider may also allow service providers to specify how such load balancing should be accomplished, such as by specifying a percentage of requests to be routed to each data center 110 . The cloud computing provider may also allow service providers to change the configuration of endpoints independently of global network addresses, such that changing the specific endpoints providing a service does not require reconfiguration of network addresses. The use of global network addresses can greatly simplify the operation of network services since any client device 102 wishing to connect to the service can simply transmit a request to a global network address of the service. Changes to the endpoints providing the service can then be made without, for example, having to change DNS entries for the service. As will be described below, in some cases these changes can be made automatically such that no user action is required even if endpoints within the data centers 110 change.

To enable global network addresses, a set of global access points 106A-N is provided. Each access point may generally include one or more computing devices configured to receive requests from client devices 102 to interact with services and to route such requests to an endpoint within a data center 110 based at least in part of load balancing requests across data centers 110 is selected. The access points 106 may also act as a sort of proxy for the endpoints, allowing traffic to flow between the client devices 102 and the data centers 110 through the access points 106 . The operation of access points 106 is discussed in more detail below. Briefly, however, they may use anycast techniques to broadcast global network address availability to neighboring network devices within network 104, which in one embodiment includes devices that are not under the control of a common entity that provides access points 106A. The access points 106 can thus attract traffic addressed to the global network addresses. The access points 106 can then select an endpoint to direct traffic to based on factors such as endpoint availability, load balancing between the data centers 110 and performance criteria between the access point 106 and the different data centers 110.

After selecting a data center 110, an access point 106 can route the request to the endpoint. In one embodiment, access point 106 uses NAT translation or encapsulation (e.g., virtual private networks) to redirect the request over network 108 to the endpoint, thereby preventing disclosure of an endpoint's network address to client devices 102 becomes. When connection-oriented communication sessions are employed between client devices 102 and an endpoint, the access point 106 according to the present embodiments is operable to perform an initialization phase of the communication session on behalf of the endpoint. In cases where the network 108 is a private network, the global access points 106 can also act as an "offload point" for traffic to the endpoints by moving that traffic from a public network (e.g., network 104) to the private network 108 becomes. In general, such a private network would be expected to have higher performance than a public network, and therefore such offloading can further increase the communication speed between client devices 102 and endpoints. As described above, in some embodiments, the global access point 106 may select different congestion control parameters and windowing parameters for TCP communications between the access point 106 and the endpoint to increase data throughput by enabling higher performance of the private network 108 compared to traditional or smaller values for the congestion control parameters and windowing parameters for the TCP communication between the client computing devices 102 and the access point.

As noted above, access points 106 may implement a variety of techniques to ensure resiliency of a network service using a global network address. Illustratively, the use of anycast to advertise access points 106 may provide resiliency between the access points 106 since, in general, failure of a single access point 106 may be expected to cause network 104 devices to route requests to another access point 106 . In addition, to address potential network 104 malfunctions, each access point 106 may be configured to control its advertisement of global network addresses in the network 104, thereby providing multiple routing paths to the access point 106 for each service. Further details regarding such control of advertisements to provide routing paths are provided below. Furthermore, to address potential malfunctions within an access point 106, each access point 106 may be configured to include multiple data flow managers that handle different traffic flows addressed to global network addresses. The data flow managers may be distributed logically, such as across program threads, and/or physically, such as across processors or computing devices. Thus, the failure of one data flow manager may have little or no impact on other data flow managers within an access point 106, limiting the impact of partial failures within an access point 106. An illustrative configuration of an access point 106 is described below with reference to FIG 4 discussed.

2 11 is a block diagram showing an illustrative configuration of a data center 110 of FIG 1 represents. As in 2 As shown, the data center 110 includes an endpoint pool 201 containing a set of endpoints 202A-N. Each endpoint 202 illustratively represents a computing device configured to provide access to a network accessible service. In one embodiment, endpoints 202 are individual physical computing devices. In another embodiment, endpoints 202 are virtualized computing devices running on physical computing devices. In yet another embodiment, endpoints 202 are collections of computing devices (physical or virtualized) configured together to provide access to a network accessible service. For example, each endpoint 202 may be a collection of devices that is a load balancer configured to distribute load balancing requests to the endpoint 202 among the collection of devices. Each endpoint 202 communicates with network 108 and is thus network 108 addressable. The number of endpoints 202 may vary depending on the capacity requirements of the network accessible service, for example. Illustratively, a service provider for such a service may contract with an operator of the data center 110 (e.g., a cloud computing provider) to generate and provision the endpoints 202 .

In one embodiment, the number of endpoints 202 may vary according to current or past demand for the network accessible service. To accommodate a varying number of endpoints 202, the data center may include a resource manager 206 (e.g., implemented directly on a physical computing device or as a virtualized device on a host physical computing device) that monitors the load of the endpoints 202 (e.g., in terms of requests per second, computing resource usage, response time, etc.) and dynamically adjusts the number of endpoints 202 to keep the load within the thresholds. For example, if the endpoints 202 are implemented as virtualized devices, the resource manager 206 can generate and provision new virtualized devices when a current set of endpoints 202 has usage metrics that exceed desired upper usage parameters, and "shut down" and remove virtualized devices when the metrics fall under lower usage parameters. The resource manager 206 may be further configured upon modifying a number of endpoints 202 within the pool 201 to notify the global access points 106 of network address information for endpoints 202 within the pool 201 so that the access points 106 can address traffic to the endpoints 202.

In addition, the data center includes 110 of the 2 a health checker 204 configured to monitor the health of endpoints 202 within pool 201 . To illustrate, the health checker 204 may periodically (e.g., every n seconds) transmit a query to the endpoints 202 and monitor whether an appropriate response is received. If so, health checker 204 may consider endpoint 202 healthy. If not, health checker 204 may consider endpoint 202 to be unhealthy. Illustratively, the health checker 204 may notify the global access points 106 of unhealthy endpoints 202 to cause the access points 106 to reduce or eliminate routing of traffic to the endpoint 202 as long as the endpoint 202 is unhealthy. In some cases, the health checker 204 may be further configured to check the health of the global access points 106, the health of endpoints 202 in other data centers 110, or the health of a network path between the health checker 204 and the access points 106. For example, health check device 204 may periodically transmit information to access point 106 and monitor a response from access point 106, network metrics related to the response (e.g., latency), or the like. Health check device 204 may report this information to access points 106 to facilitate routing of traffic to endpoints 202A within data center 110 . Illustratively, health checker 204 may report health information for endpoints 202 to configuration data store 210 (e.g., a database in data store 210), which may be provided by operation of configuration manager 208 to access points 106 in the manner described below (e.g., as part of a configuration package or a configuration broadcast) can be distributed.

The data center 110 of 2 further includes a configuration manager 208 configured to allow service providers to configure operation of data centers 110 and global access points 106. Illustratively, the configuration manager 208 can provide an interface through which users can specify the endpoints 202 that provide a network accessible service, configure those endpoints 202, and configure the resource manager 206 to scale up or down endpoints. The configuration manager 208 may also allow service providers to assign global network addresses to these endpoints and set load balancing parameters for routing traffic addressed to the global network address to different data centers 110 . The configurations created by service providers may be stored in a configuration data store 210, which may correspond to any persistent or substantially persistent storage device (e.g., hard drives, solid state drives, network attached storage devices, etc.). In some cases, configuration data store 210 may include multiple representations of a service provider's configuration. For example, to facilitate rapid reconfiguration of global access points 106, configuration data store 210 may include a database (such as a relational database) that is modified each time a service provider confirms a change to its configuration. The configuration manager 208 may periodically (e.g., every 100 milliseconds, 1 second, 2 seconds, 5 seconds, 30 seconds, etc.) determine whether changes have been made to the database, and if so, a new configuration package for the global access points 106 generate, where the configuration package includes the changes to the database (and thus the configuration of the service provider) relative to a previous configuration package. The configuration manager 208 can then store the configuration package in the configuration data store 210 for retrieval by the global access points 106 . In an out 1, each global access point 106 is configured to periodically (e.g., every 100 milliseconds, 1 second, 2 seconds, 5 seconds, 30 seconds, etc.) poll the configuration data store 210 to determine if a new configuration package is present, and if so, retrieve and implement the data packet. In some cases, a configuration package may be broken down into "shipments" that are part of the configuration. The global access points 106 may be configured to retrieve only those shows that have been modified relative to an existing show. Changes can be tracked based on the versions of programs or a data package, for example. Additionally, in some embodiments, data packets or broadcasts may be stored in the data store 210 as differences or "deltas" from a previous version, such that an access point 106 can only retrieve changes since a previous version of the broadcast, thereby eliminating the data transfer necessary to update the data packet or the shipment is required is reduced. In one embodiment, the configuration manager 208 may periodically (e.g., every 100 milliseconds, 500 milliseconds, etc.) "check" data packets or broadcasts by collecting any changes since a previous check and saving the data packet or broadcast as a standalone version. Such verification may facilitate rapid reconfiguration in the event that a global access point 106 does not have a reference frame from a previous data packet or transmission.

In one embodiment, only one data center 110 of data centers 110A-N includes the 1 the configuration manager 208, the manager 208 propagating each service provider's configuration to the other data centers 110 and access points 106. In another embodiment, each data center 110 includes a configuration manager 208, and the managers 208 of each data center 110 can communicate to synchronize configurations between the data centers 110 and with the access points 106. In addition, the data centers 110A-N may further include various network routing components 214, such as top-of-rack ("TOR") switches, which may be configured to route data packets to different endpoints 202. More specifically, in one embodiment, TOR switches may use data packets included to route data packets to endpoints 202 . For example, a TOR can use 5-tuple information, namely a source IP address, a source port address, a destination IP address, a destination port address, and a routing protocol, to automatically route data packets an assigned endpoint 202 to route. This facilitates use in a set of multiple VLANs or fabrics securing transmissions to different endpoints 202. Additionally, in other aspects of the present application, endpoints 202 may utilize MTU path discovery messages to exchange information. According to aspects of the present application, status information can be exchanged to increase resilience in the event of a change in the data packets, in order to enable the further processing of services/requests by the computer center 110 . More specifically, the endpoints can use multicast messaging to a set of endpoints 202 that are associated with a specific VLAN or fabric that allows only those endpoints to receive the multicast message, such as to relay status information in the event that an endpoint has failed and is attempting to maintain service continuity by providing status information to another endpoint. In other embodiments, endpoints may use broadcast messages that are broadcast to all endpoints in data center 210 . Because the broadcast message is received at all endpoints (or a larger portion of them), individual endpoints 202 in a data center can filter or exclude the messages that do not apply.

While only some components of data center 110 are shown communicating with network 108 , other components may additionally communicate with network 108 and/or network 104 . The lines in 2 are not intended to represent all actual or potential network connections, but rather to illustrate a possible flow of service-related traffic to the endpoints 202.

Furthermore, while shown within a data center 110 in one embodiment, global access points 106 may also include a configuration manager 208 that allows for direct configuration of access point 106 . In another embodiment, the global access points 106 exclude any configuration manager 208 and data store 210 . For example, if the access points 106 are implemented in shared environments (e.g., not operated by or accessible to anyone other than an operator of the access points 106), the access points 106 may be configured to exclude any persistent storage and instead configure information from a data center 110 upon initialization of the access point 106. In this way, the security of the access points 106 can be increased since it would be expected that shutting down the access point 106 will result in the loss of any sensitive data that may be on the access point 106.

While data center 110 is shown as including an endpoint pool 201 corresponding to a network accessible service, data center 110 may host numerous pools 201, each corresponding to a different service. Thus, multiple service providers can use a data center 110. Additionally, as noted above, any network accessible service may be provided by endpoints 202 across multiple data centers 110 . Accordingly, the global access points of the 1 Distribute traffic to such a service across data centers 110.

In accordance with embodiments of the present disclosure, the data center 110 further includes a session handover manager 212 configured to facilitate acceptance of connection-oriented communication sessions with clients 102 initialized at a global access point 106 . As discussed above, global access points 106 may in some cases be configured to complete an initialization phase of a connection-oriented communication session, such as the TCP three-way handshake, and then hand off the communication session to an endpoint 202 . In some cases, an endpoint 202 may be configured to accept such a handoff by receiving context of the connection-oriented communication session (e.g., sequence numbers of both ends of the connection) and generating local status information that includes that context. The endpoint 202 may be configured, for example, by modifying network protocol handlers within an operating system of an endpoint 202 (e.g., by modifying a kernel to accept and accept TCP context information from a global access point 106). However, it may be preferred that endpoints 202 do not have to be configured to allow many different endpoints 202 to use global access points 106 . To enable this, the data center 110 may also include a session handoff manager 212 configured to accept and accept a connection-oriented communication session from a global access point 106 . The session handoff manager 212 can then establish a separate connection-oriented communication session with the endpoint 202 selected by the access point 106 to provide service to a client device 102 and act as a "middleman" between the two sessions. Since the session handover manager 212 and the endpoint 202 can generally be co-located within the data center 110, the creation of a second connection-oriented communication session can be expected to introduce a minimal delay in the communication between the client device 102 and the endpoint 202.

3 Fig. 12 is a block diagram showing an illustrative configuration of a global access point 1 represents. As in 3 As shown, each global access point 106 communicates with the network 104 via a router 302 . while in 2 only a single router 302 is shown, access points 106 may include multiple routers 302 . Additionally, although a single connection to network 104 is shown, each router 302 may include multiple connections to network 104, possibly to multiple different neighboring devices within network 104, each of which may correspond to different sub-networks (e.g., autonomous systems (AS) within the network 104).

As mentioned above, global access points 106 may be configured to use anycast techniques to pull traffic to global network addresses associated with network accessible services. Thus, router 302 is illustratively configured to advertise the global network addresses to neighboring devices on network 104 . In one embodiment, such offers are BGP offers. Such offers may cause the router 302 to attract traffic addressed to the global network addresses, since the offers may cause devices on the network 104 to send traffic addressed to the addresses according to the operation of anycast techniques to the router 302 to route.

As discussed above, global access point 106 may implement a variety of techniques to increase access point 106 resilience. In one embodiment, the global network addresses offered by access point 106 are divided into multiple address groups. To reduce the potential impact of malfunctions on the network 104, the router 302 (or multiple routers 302) may be configured to transmit BGP advertisements to different neighboring devices in the network 104 (e.g., different AS) for each address group . A network-accessible service can be associated with addresses from multiple address groups, each of which may be provided to client devices 102 as an address at which the service may be accessed. Since addresses from different groups are advertised differently in network 104, different routing paths in network 104 can be expected for addresses of each group. For example, data packets addressed to addresses within a first group can reach router 302 via a first AS of network 104, while data packets addressed to addresses within a second group can reach router 302 via a second AS. Thus, if a malfunction occurs within the first AS (or a downstream AS connected to the first AS), data packets addressed to addresses within the second group can still be expected to reach the router 302 and vice versa. Thus, dividing global network addresses into multiple groups may increase the resilience of access points 106 to failures within network 104 .

Upon receiving a data packet addressed to a global network address, router 302 may route the data packet to a data flow manager 304 or other similarly functioning component of a set of flow managers 304A-N. While an access point 106 may implement a single data flow manager 304, it may be advantageous for an access point to implement multiple data flow managers 304 to provide redundant operation of such data flow managers 304. Router 302 may use any number of known load-balancing techniques to distribute data packets to data flow managers 304, such as round-robin election. In one embodiment, router 302 uses consistent hashing to distribute data packets. Consistent hashing is well known in the art and is therefore not described in detail in this document. For example, consistent hashing may be beneficial to increase the chances that multiple data packets with the same properties (e.g., source network address, source network port, destination network address, destination network port, protocol) are routed to the same data flow manager 304 . Advantageously, this may allow data flow managers 304 to maintain status information related to data packet flows between a client device 102 and a destination endpoint 202 . In some cases, such status information may be required, for example, to implement NAT techniques or to perform an initialization phase of a connection-oriented communication session. In another embodiment, equal-cost multi-path (ECMP) load balancing is used to route traffic to data flow managers 304A-N. ECMP load balancing is well known in the art and is therefore not described in detail in this document.

In one embodiment, ECMP load balancing is applied to route data packets to data flow managers 304A based on a global network address to which the data packet is addressed. To illustrate, each data flow manager can handle data packets addressed to a subset of addresses within a set of global network addresses. For example, a first flow manager 304 may preferentially handle a first quartile of addresses within a group, a second manager 304 may preferentially handle a second quartile, and so on that are affected by improper operation of a data flow manager 304, such as due to a misconfiguration of a service associated with an address managed by that data flow manager 304. When multiple sets of global network addresses are used, services can be "shuffled" between different sets, making it unlikely that two services with addresses that share a subset under one address set will have addresses that share a subset under a different address set share. Such rearrangement can reduce the overall percentage of services rendered completely unavailable by an access point 106 due to a malfunction of a single data flow manager 304A.

To further facilitate the use of status information on the flow managers 304, in one embodiment, the router 302 implements a flow hash routing algorithm or a similar algorithm to identify related data packets that constitute a "traffic flow" between a client 102 and a Form endpoint 202. Many different data flow identification algorithms are known in the art and are therefore not described in detail in this document. Illustratively, the router 302 may implement a flow identification algorithm to identify flows of data packets and route data packets of the same flow to the same data flow manager 304 consistently. In one embodiment, router 302 applies flow-based routing to a data packet before otherwise distributing data packets to data flow managers 304 such that when the data packet is recognized as part of a data flow, it is distributed to the same data flow manager 304 that previously handled data packets of that data flow Has. If the data package in such a version If the data packet is not recognized as part of the data flow, additional load balancing algorithms (e.g., consistent hashing) are applied to the data packet to determine a data flow manager 304 to which the data packet should be routed.

Upon receiving a data packet, a data flow manager 304 can determine a data center 110 to which the data packet is to be routed and an end point 202 within the data center 110 . In one embodiment, data flow manager 304 may apply a combination of criteria to select a data center 110 to route a data packet to, including network performance criteria and load balancing criteria. In a first aspect, for a given data packet, a data flow manager 304 may initially select a data center 110 based on network or geographic criteria between the global access point 106 and the various available data centers 110 or groups of endpoints 202. The network or geographic criteria may correspond to a measure of network distance (e.g., across network 108) from an identified access point 106. The network or geographic criteria may further include or include, at least in part, performance criteria such as latency, number of hops, bandwidth, or a combination thereof. In general, routing a data packet to a data center 110 with a maximum network performance criterion may advantageously increase the communication speed between a client device 102 and the data center 110 . Because network performance criteria are unlikely to shift rapidly between an access point 106 and a data center 110, simply routing each data packet to a data center 110 with a maximum expected performance criteria may not achieve the load balancing requested by a service provider.

In addition to measured or determined network or geographic criteria, each data flow manager 304 may also modify or incorporate distribution criteria as necessary to achieve a service provider's desired load balancing. More specifically, in some embodiments, each grouping of endpoints 202 (e.g., data centers 110) may be divided into different regions. Individual data centers 110 may associate subsets of endpoints 202 with different sub-regions or other groupings of a region. Accordingly, a customer may specify distribution criteria that identify a measured distribution or method of calculating a measured distribution of data packets provided to a data center 110 . Illustratively, the distribution may be specified as a percentage of traffic, total number of data packets (e.g., total amount of data, cost allocated or billed to individual endpoints, and the like). A system administrator may illustratively use a software tool or interface (e.g., an API) to provide allocations, as described in various examples herein. In turn, the data flow manager 304 may implement an algorithm, such as a coin toss algorithm, to implement the percentage selections. The algorithm can also take into account additional factors such as geographic or network criteria.

In a set of illustrative examples, assume that a data center 110 assigned to a region may be further assigned to three different sub-regions ("Part 1", "Part 2", and "Part 3"). A tool allows a system administrator to set allocation on a percentage basis. With regard to the distribution of illustrative 100 packages, various iterations will now be described: subregion allocation Number of data packets Part 1 10% 10 data packages Part 2 20% 18 (20% of 90 remaining) part 3 100% 72 (100% of 72 remaining)

In the above example, the data flow manager 304 can apply the distribution using the coin toss algorithm so that the data packets are allocated to one of the sub-regions. In the following example, a system administrator may not have full allocation, as follows: subregion allocation Number of data packets Part 1 10% 10 data packages Part 2 20% 18 (20% of 90 remaining) part 3 10% 7 (10% of 72 remaining)

In the example above, the illustrative allocations do not consider the full set of data packets. Accordingly, in one embodiment, data flow manager 304 may randomize among the set of sub-regions. In other embodiments, data flow manager 304 may allocate based on standard distribution algorithms, such as a closest geographic sub-region or a manually selected distribution. In yet another embodiment, the data flow manager 304 may further consider an auto linear ramp to allocate additional traffic as follows: subregion allocation Number of data packets Part 1 10% 10 data packages Part 2 20% 18 (20% of 90 remaining) part 3 10% 7 (10% of 72 remaining)

Autolinear fit for unallocated 65: subregion allocation Number of data packets Part 1 90% 58 data packets Part 2 100% 17 (20% of 17 remaining) part 3 90% 0 (10% of 0 remaining)

In another embodiment, global access points 106 may be configured to initially select a data center 110 based on performance criteria between the client device 102 from which traffic was received and the originally selected data center 110, rather than initially selecting a data center 110 based on geography or select network criteria between the access point 106 and the data center 110. Advantageously, the use of network performance criteria between client devices 102 and data centers 110 can result in a consistent selection of a data center 110 for a given client device 102, regardless of the access point 106 to which the client device's traffic is routed. After selecting an initial data center 110 (e.g., closest to a client device 102 in terms of distance or minimized latency), the access point 106 may modify that selection as needed to achieve load balancing specified by a service provider. For example, if the service provider desires no more than 70% of traffic to be routed to the initially selected data center 110, and the percentage of traffic over a past period of time exceeds that percentage, the access point 106 may select a different data center (e.g., a Data center 110 with the next performance in relation to the client device 102). The access point 106 can then determine whether the request could be routed to that other data center 110 based on a fraction of previous traffic that was routed to that other data center 110 . This process may continue until the access point 106 locates a data center 110 that is acceptable based on the service provider's load balancing specifications.

In some embodiments, load balancing is implemented locally at each data flow manager 304 . In other embodiments, load balancing is implemented across all data flow managers 304 of an access point 106 . In still other embodiments, load balancing is implemented across data flow managers 304 of multiple access points 106 . In general, a localized load balancer is expected to be less resource intensive because less communication is required between distributed components. However, less localized load balancing can result in load balancing criteria more similar to those desired by a service provider.

In some cases, data flow managers 304 may implement a mix of localized and non-localized load balancing. For example, each data flow manager 304 may implement localized load balancing (e.g., localized to each manager 304 or each access point 106) and periodically negotiate with other access points 106 to adjust the weights applied when selecting a data center 110. For example, if a service provider requests that traffic be split evenly between data centers 110, a localized load can occur equal cause each of the two access points 106 to share traffic evenly between the data centers 110. This can result in sub-optimal routing, since half of the traffic at each access point 106 may be routed to a non-nearest data center 110. Accordingly, in this scenario, the access points 106 can communicate regarding the routing of their traffic, and hypothetically assuming that the volume of traffic at each point 106 is the same (and that only two access points 106 are considered), each access point 106 can begin to forward all of its data packets into one to route the nearest data center 110. Such a split would still result in an even split of traffic between the data centers 110 and, moreover, would beneficially increase the average network performance metric for each data packet flow.

In one embodiment, a desired fraction or volume of traffic routed to a given data center 110 may be statically selected by a service provider. For example, a service provider may require that no more than 70% of requests at an access point 106 be routed to a given data center 110. In another embodiment, the desired proportion or volume may be dynamic. For example, a service provider may specify that a desired fraction or volume of traffic routed to a given data center 110 should increase from a first time to a second time according to a given rate of change (e.g., linear, exponential , logarithmic, etc.) increases or decreases.

After selecting a data center 110 to route traffic to, a data flow manager 304 may select an endpoint 202 within data center 110 to route traffic to. The endpoint 202 can be selected according to any load balancing algorithm. In one embodiment, data flow manager 304 may use consistent hashing to select endpoint 202 .

As discussed above with respect to data flow managers 304, it may be desirable for traffic of a given data flow to be consistently routed to the same endpoint 202 so that endpoint 202 can maintain status information regarding that data flow. Thus, each data flow manager 304 may use a data flow selection algorithm to identify subsequent data packets within a data flow previously routed to an endpoint 202 by data flow manager 302A and to route such data packets to the same endpoint 202. In an embodiment in which a data packet is identified by the data flow manager 304 as part of an already routed data flow, the data flow manager 304 may omit the selection of a data center 110 and an endpoint 202 for the data packet in favor of routing the data packet to the endpoint 202 that previously used for previous data packets of the data flow. In particular, the data flow manager 304 may implement a selection algorithm that attempts to identify an endpoint 202 consistently. To illustrate, the selection algorithm may generate a score associated with various endpoints 202 that is applied to attributes of each endpoint or communication. For example, the selection algorithm can be computed in terms of the 5 tuples (e.g., namely, a source IP address, a source port address, a destination IP address, a destination port address, and a routing protocol, to automatically route data packets to an assigned endpoint 202). The selection algorithm can then process the generated scores for a given communication request (e.g., a connection to a service) by selecting an endpoint with a "highest" score. Subsequent communications may be routed to the selected endpoint, assuming that the selection algorithm consistently generates the same "high" score. In one embodiment, the selection algorithm may correspond to a weighted rendezvous hashing algorithm. In another embodiment, the selection algorithm may be a weighted consistent hashing algorithm.

In still other embodiments, the data flow manager 304 may further optimize the generation of scores by organizing the calculation of scores into hierarchies. In one embodiment, the endpoints 202 are grouped into pairs that enable a score to be calculated based on cumulative attributes. The pairs are further combined in successive combinations until only two combinations remain, creating a hierarchy akin to a pyramid. Rather than requiring computation and comparison scores for each endpoint 202 in a group, the resulting pyramid allows for a binary split of the score and comparison, making endpoint 202 selection even more efficient.

After selecting an endpoint 202 to route a data packet to, a data flow manager 304 may modify the data packet to facilitate routing to the endpoint 202. For example, a destination network address of the data packet when received at router 302 may be a global network address. The data flow manager 304 may therefore modify the data packet to replace the destination network address with an endpoint 202 network address. In one embodiment, each data flow manager 304 implements NAT techniques to modify data packets addressed to global network addresses. For example, each data flow manager 304 for data packets destined for endpoints 202 can replace a global network address with a network address of an endpoint 202 as the destination network address of the data packet and a network address of the client device 102 with an address of the access point 106 as the source address. A similar translation can be done for data packets from endpoints 202 to be routed to client devices 102 according to NAT techniques. Flow managers 304 may illustratively use port translation (a known NAT technique) to distinguish between translated flows. After translation, data flow manager 304 may return the data packet to router 302 for transmission to selected endpoint 202 over network 108 .

In another embodiment, a data flow manager 304 may use encapsulation to route a data packet to an endpoint 202. Illustratively, each data flow manager 304 can generate an IP "tunnel" to a device within a data center 110, such as the session handoff manager 212 or a router within the data center 110. To route data packets to an endpoint 202, a data flow manager 304 can encapsulate the data packet and transmit the data packet over the tunnel to the receiving device. The receiving device can then decapsulate the data packet and transmit the data packet to endpoint 202 . In one embodiment, the data flow manager 304 replaces a destination address of the data packet (e.g., a global network address of the service) with a destination address of the selected endpoint 202 to facilitate transmission of the data packet 202 to the endpoint. Encapsulating packets over a tunnel can provide benefits such as preserving the network address of a client device 102 that transmitted the data packet. As previously described, in one specific example, data flow manager 304 may establish a set of VLANs that utilize encapsulated communications to different subsets of endpoints. Thus, for example, encapsulation may allow a client device 102 and an endpoint 202 to maintain connection-oriented communication sessions even when data packets of a client device 102 are routed through different access points 106 . In some cases, data flow managers 304 and endpoints 202 (or session handover managers 212) may be configured to authenticate communications between themselves to ensure the authenticity of packets transmitted between the devices. For example, each of data flow manager 304 and endpoint 202 (or session handover manager 212) may be configured to generate a digital signature that authenticates the respective devices, and to include such digital signature in a header of encapsulated packets sent between the devices flow so that each device can authenticate the digital signature to verify that the data packets were generated at a known device. As another example, flow managers 304 and endpoints 202 (or session handoff managers 212) may be configured to communicate via known secure protocols, such as Transport Layer Security (TLS) protocols.

As noted above, in some cases, access points 106 may be configured to perform an initialization phase of a connection-oriented communication session to reduce the time required to perform such phase (e.g., due to access point 106's relative proximity to the client device 102 relative to data center 110). To enable such functionality, each data flow manager 304 may implement the initialization phase at a client and provide information related to an initialized session to a device within the data center 110, such as an endpoint 202 or a session handoff manager 212. The information may, for example, via the tunnels are provided. Illustratively, a data flow manager 304 may generate a tunnel to the session handover manager 212 and pass within the tunnel a "TCP handover" command that includes TCP session state information. For example, the status information can include a TCP "five-tuple" (five values that often define a TCP connection: source address, source port, destination address, destination port, and the protocol used) or part of it and/or sequence numbers of the TCP connection. Upon receiving the TCP handoff command, the session handoff manager 212 may generate a corresponding entry in its own stateful session table (e.g., TCP table) and thus "accept" the connection-oriented communication session. The data flow manager 304 can then receive data packets from the client front direction 102 are transmitted over the tunnel, which may be decapsulated and processed at session handoff manager 212 as part of the accepted session. In embodiments where a session handover manager 212 is used to achieve stateful session handover, the data flow managers 304 may not need to select an endpoint 202 to send a data packet to. Instead, data flow managers 304 may be configured to consistently select an appropriate session handover manager 212 within a selected data center 110 as the destination for client device 102 data packets. In turn, the session handoff manager 212 may select an endpoint 202 within the data center 110 . Thus, using a session handoff manager 212 may offload the responsibility for selecting an endpoint 202 from the data flow manager 304 to the session handoff manager 212 . The selection of an endpoint 202 at a session handoff manager 212 may be similar to such a selection at a data flow manager 304 .

In addition to one or more routers 302 and flow managers 304, the global access point 106 includes the 2 a health check device 306. Health check device 306 may illustratively transmit health checks to data flow managers 304 to determine whether such data flow managers 304 are malfunctioning. The transmission of health check data may be similar to the transmission of health check data by a health check device 204 within a data center 110, as discussed above. Additionally or alternatively, health check device 306 may transmit health check data to endpoints 202 and/or client devices 102 . Health check data received with respect to endpoints 202 may facilitate routing decisions to endpoint 202 . For example, if the health checker 306 is unable to communicate with an endpoint 202, a data flow manager 304 may stop selecting the endpoint 202 as a destination for traffic, regardless of whether a health checker 204 local to the endpoint 202 communicates with the endpoint 202 reported as error-free. Illustratively, health check data collected with respect to client devices 102 may be used to modify availability advertisements (e.g., BGP advertisements) to network 104 devices. For example, an operator of global access points 106 may review client device 102-related health check data to adjust advertisements to attempt to direct traffic from some devices 102 to a different (e.g., closer) access point 106 than the one to which currently being routed in the network 104.

The global access point 106 of the 3 further includes a configuration manager 308 configured to receive configuration information related to services associated with global network addresses and to configure operation of global access point 106 to implement such configuration. For example, configuration manager 308 may receive information about how router 302 should offer global network addresses to network 104, information that maps services provided at data centers 110 to global network addresses that are routed through access points 106, information , identifying data centers 110 and endpoints 202 of such services, information specifying a desired load balance between such data centers 110, and the like. In one embodiment, the configuration manager 308 retrieves configuration information from a configuration data store 210 in a data center 110 . For example, the configuration manager 308 may periodically query the data store 210 for new configuration information. As another example, a data center 110 may "push" configuration changes to the configuration manager 308 using a variety of known push notification techniques. For example, a data center 110 may implement a publish-subscribe ("pub/sub") notification system, and a configuration manager 208 may publish changes to a service provider's configuration to the system. The system can then notify the configuration managers 308 of the access points 106 of such changes. Therefore, when service providers modify the configuration of their services and global network addresses in a data center 110, such modifications can be propagated to the access points 106. In one embodiment, access points 106 do not store configuration information in persistent storage (and may not have such persistent storage) to reduce the likelihood that such information could be obtained from access points 106 .

While examples were discussed above in relation to one network accessible service, each access point server 402 may be associated with multiple services. For example, if each data flow manager 304 is tasked with handling data packets addressed to a subset of global network addresses within a group of such addresses, each data flow mana ger 304 will therefore handle traffic addressed to any service associated with a global network address within that subset.

While only some components of access points 106 are shown communicating with networks 104 and 108, other components may additionally communicate with network 108 and/or network 104. The lines in 3 are not intended to represent all actual or potential network connections, but rather to illustrate a possible flow of service-related traffic to the access points 106.

The data center 110 of 2 and the global access point 106 of the 3 operate in a distributed computing environment that includes one or more computer systems that are interconnected using one or more computer networks (not in the respective figures). The data center 110 of 2 and the global access point 106 of the 3 could also be operated within a computing environment with fewer or more devices than illustrated in the respective figures. Thus, the depictions of the data center 110 of the 2 and the global access point 106 of the 3 are to be understood as illustrative and not limited to the present disclosure. The data center 110 of 2 , the global access point 106 of 3 or various portions thereof could implement various web service components, hosted or "cloud" computing environments, and/or peer-to-peer network configurations to implement at least some of the processes described in this specification.

4 10 depicts a general architecture of an example computing system (referred to as access point server 402) that operates to implement the data flow managers 304 of an access point 106. FIG. The general architecture of the in 4 The illustrated access point server 402 includes an arrangement of computer hardware and software modules that can be used to implement aspects of the present disclosure. The hardware modules can be implemented with physical electronic devices, as discussed in more detail below. Access point server 402 may include many more (or fewer) elements than those in 4 shown. However, it is not necessary that all of these commonly used elements be shown to provide an enabling disclosure. In addition, the in 4 illustrated general architecture can be used to implement one or more of the other components described in the 2 and 3 are illustrated. As illustrated, access point server 402 includes one or more processing units 490, one or more network interfaces 492, and one or more computer-readable media drives 494, all of which can communicate with one another via a communications bus. Network interfaces 492 may provide connectivity to one or more networks or computing systems, such as router 302 (which may correspond, for example, to a commercially available router device). Processing units 490 may thus receive information and instructions from other computing systems or services over networks, such as network 104 or 108. Processing unit 490 can also communicate to and from memory 480 .

Memory 480 may contain computer program instructions (grouped as modules in some embodiments) that processing unit 490 executes to implement one or more aspects of the present disclosure. Memory 480 generally includes random access memory (RAM), read only memory (ROM), and/or other persistent, supplemental, or non-transitory computer-readable media. The memory 480 may store an operating system 482 that provides computer program instructions for use by the processing unit 490 in the general administration and operation of the access point server 402. Memory 480 may also include computer program instructions and other information for implementing aspects of the present disclosure. For example, in one embodiment, memory 480 includes one or more data flow manager units 483, each representing code that is executed to provide a data flow manager 304 of the 3 to implement. Each data flow manager entity 483 may illustratively be isolated from other entities 483 on server 402 . For example, each entity can represent a separate virtual machine or an isolated software container. In some cases, each unit 483 may be associated with separate processing units 490, interfaces 492, or drives 494, thereby minimizing the possibility of the operation of one unit 483 affecting the operation of another unit 483. Each unit 483 illustratively includes an endpoint selection unit 484 representing code executable to select an endpoint 202 to which a data packet addressed to a global network address is to be routed, a flow table 486 representing a table of information to select data flows from to map packets to endpoints 202, a NAT unit 488 representing code executable to NAT packets addressed to global network addresses, or to perform replies to such data packets from endpoints 202, and a session handoff unit 489 representing code executable to perform an initialization phase of a connection-oriented communication session and handoff that session to a receiving device. Even if this in 4 not shown, the memory 480 illustratively further includes an encapsulation unit that is code executable to generate a tunnel connection to another device that enables the transmission of encapsulated data packets and to perform encapsulation/de-encapsulation to facilitate such transmission facilitate.

The memory 480 may further include a state check unit 496 corresponding to instructions executable to implement the state check device 306 and a configuration manager unit 498 corresponding to instructions executable to implement the configuration manager 308 . In some embodiments, health checker 306 and configuration manager 308 may be implemented as standalone devices rather than as part of access point server 402 . Additionally, while router 302 is shown as separate from access point server 402, in some cases it may be integrated with server 402 (e.g., by including in memory 480 software executable to implement routing functionality). .

In one embodiment, access point server 402 lacks any non-volatile memory and is configured to operate solely on volatile memory, such as RAM. Illustratively, the access point server 402 may be configured to use a pre-boot execution environment (PXE) such that the access point server 402 upon initialization from a network-accessible location (e.g., the configuration data store 210) at least retrieves some of the contents of memory 480, such as the operating system 482, the health check unit 496, and the configuration manager unit 498. The configuration manager unit 498 can then retrieve additional configuration information from a network-accessible storage location, such as the configuration for individual services and associated global network addresses , and use this additional information to generate the data flow manager entities 483 . To prevent unauthorized disclosure of the contents of storage 480, authentication of server 402 at the storage location may be linked, at least in part, to a network location of server 402 (e.g., at an access point 106) such that attempts to physically relocate server 402 are prevented , result in an error retrieving the contents of memory 408.

While 4 Illustrating a single server 402 and a single router 302, a global access point 106 may be implemented by multiple servers 402 and/or routers 302 in some cases. In some cases, such servers 402 or routers 302 may be physically or logically isolated to avoid propagation of errors between such servers 402 or routers 302. Illustratively, if an access point 106 handles multiple pools of network addresses, each pool can be handled by a standalone server 402 and router 302 . Thus, should a router 302 and/or server 402 fail, it would be expected that only services associated with the pool handled by that router 302 and/or server 402 would be affected.

With reference to 5 describes illustrative interactions that show how a single global access point 106A can operate resiliently by providing multiple routes to reach a network accessible service using global network addresses. 5 10 illustrates an environment 500, which in turn is an embodiment of a portion of the environment 100 of FIG 1 can represent. In particular, in environment 500, network 104 is divided into networks 104A-E, each of which may represent, for example, an autonomous system. Networks 104A-C are illustrated as communicating with client devices 102. These networks 104A-C may represent Internet service providers (ISP) of client devices 102, for example. Networks 104D and E represent other ASs to which global access point 106A has a network connection. While networks 104D and E in 5 are shown connected to client devices 102, such networks may also act as an ISP for client devices 102. Each of the networks 104A-E is in 5 shown connected to other networks 104A-E. This configuration is illustrative and may vary by implementation.

In the illustration of 5 For example, client devices 102 generally have two routes to reach global access point 106A: network 104D and 104E. To increase resiliency of access to global access point 106A, access point 106A may selectively transmit availability advertisements to networks 104D and 104E. In particular, as in 5 shown, a global access point 106A at (1) at least two sets of global network ads identifiers, such as two contiguous ranges of network addresses. Thereafter, at (2), global access point 106A transmits an advertisement (e.g., a BGP advertisement) for the first network group to network 104D, rather than offering all global network addresses to both networks 104D and E . Similarly, at (3), global access point 106A transmits an advertisement (e.g., a BGP advertisement) for the second network group to network 104E. In this manner, traffic addressed to addresses within the first network group is likely to arrive at access point 106A via network 104D and traffic addressed to addresses within the second network group is likely to arrive at access point 106A via network 104E. Each service can be associated with global network addresses from at least two groups. Thus, if failures or problems occur in either network 104D or E, a client device 102 can use an alternative global network address of a service to access global access point 106A over the remaining network.

With reference to 6 describes illustrative interactions for operating a global access point 106 to facilitate communication between a client device 102 and a data center 110 that provides endpoints 202 for a network accessible service. The interactions in 6 begin at (1) where a client device 102 transmits to global access point 106A a data packet addressed to a global network address associated with global access point 106A. The data packet can be formatted according to the Transmission Control Protocol (TCP), the User Datagram Protocol (UDP) or the Internet Control Message Protocol (ICMP), for example. The data packet may be transmitted over the network 104 and routed to the global access point 106A via the operation of the network 104 based on offers from the global access point 106A indicating that the global network address to which the data packet is addressed is via the global access point 106A is available. For example, global access point 106A may be a closest (e.g., in terms of network distance) global access point 106 to client device 102 .

Upon receiving the data packet, a router 302 within the global access point 106A assigns the data packet to a data flow manager 304 within the access point 106. If the data packet is not associated with an existing data flow, the router 302 can use ECMP load balancing to assign the data packet to a data flow manager 304 based on the network address to which the data packet is addressed. If the data packet is associated with an existing data flow, the router 302 can route the data packet to the same data flow manager 304 to which previous data packets within the data flow were routed.

At (2), the data flow manager 304 within the access point 106A allocates the data packet to a data center 110 . If the data packet is not associated with an existing data flow, the data flow manager 304 may apply a load balancing algorithm to the data packet to select a data center 110 from the set of available data centers 110 . For example, as described above, the selection algorithm may include initially selecting a data center 110 that includes a combination of network and geographic criteria and a selected distribution algorithm. As discussed above, the distribution algorithm may correspond to distribution based on distribution criteria, such as percentage allocation. In this way, even if data packets of a client device 102 are routed to a different access point 106, they can be routed to the same data center 110. If the data packet is associated with an existing data flow, the data flow manager 304 can route the data packet to the same data center 110 to which previous data packets within the data flow were routed.

After selecting a data center 110 , the data flow manager 304 may select an endpoint 202 within the selected data center 120 . Illustratively for a given data center 110, data flow manager 304 may manage a set of endpoints 202 that provide a network accessible service associated with the global network address to which the data packet was addressed. In one embodiment, this set of endpoints 202 may be identified based on information received from a resource manager 206 within data center 110 . Data flow manager 304 may also maintain information regarding the apparent state of endpoints 202 as obtained from a health checker 204 within data center 110, a health checker 306 within access point 106A, or both. The data flow manager 304 can thus select from a set of endpoints 202 an error-free endpoint for the service that is associated with the network address to which a data packet was addressed. The data flow manager 304 can be any of a number of known ones Use load balancing algorithms, such as consistent hashing, to distribute data packets between endpoints 202. More specifically, as described above, in one embodiment, the data flow manager 304 may implement a weighted rendezvous hashing or weighted consistent hashing algorithm based on the 5-tuple information of the data packet. The data flow manager 304 can then select endpoints 202 based on an endpoint with the "highest" rating to enable consistent selection for the entire data flow. When a data packet is associated with an existing data packet flow, the data flow manager 304 may select the same endpoint 202 to which previous data packets in the data flow were routed.

After selecting an endpoint, data flow manager 304 within access point 106A modifies the data packet as necessary to route the data packet to the selected endpoint 202. In particular, in the embodiment of FIG 6 in (5) NAT techniques to the data packet, for example by replacing source and destination network addresses of the data packet. Thereafter, at (6), the data flow manager 304 transmits the data packet to the selected endpoint 202 within the selected data center 110.

At (7), the endpoint 202 processes the data packet according to the network accessible service. As described, data centers 110 may implement network equipment, such as top-of-rack, that may allocate data packet traffic to endpoints based on allocated hops or braids. More specifically, in one aspect, the TOR may use 5-tuple information included in the communication to route data packets based on arbitration slices. In addition, the communications can be encapsulated, for example in VLAN, to provide further security. In the event of a malfunction, the network endpoints 202 associated with a cluster may send MTU Path Identifier (“PMTUD”) messages to other endpoints within an allocated subset or subsection to obtain status information or to determine which subsections will process service requests to duplicate to avoid processing/assignment. This allows endpoints to quickly receive information and avoid double processing. Endpoint 202 then returns a response packet to global access point 106A. Because the response packet is associated with an existing data flow, router 302 within access point 106A routes the response packet to the same data flow manager 304 as discussed above. The data flow manager 304 reverses the previous NAT transformation to address the data packet to the client device 102 . The data flow manager 304 then returns the data packet to the client 102. About the interactions of 6 For example, a client device 102 may address a data packet to a global network address and be routed (via a global access point 106) to an endpoint that provides a network accessible service related to the network address. The access point 106 may apply load balancing to data packets so that the load of the service is distributed among the data centers 110 according to a desired configuration of a service provider.

With reference to 7 An example routine 700 for increasing resiliency of global network addresses by selecting offers of addresses to different neighboring devices is described. Routine 700 may illustratively be implemented by a global access point 106 (e.g., during access point 106 initialization). In one embodiment, an instance of routine 700 is implemented by each access point 106 .

Routine 700 begins at block 704 where global network addresses to be serviced via access point 106 are obtained. This information may illustratively be obtained as part of an access point 106 configuration, such as within a configuration file stored in a configuration data store 210 and obtained during access point 106 initialization. The global network addresses can illustratively be divided into different addressing pools in the configuration. For example, each pool may contain a different "block" of network addresses, such as a contiguous range of addresses. For example, the areas can represent “subnets”. In one embodiment, ranges are represented by a "prefix" denoting the first n bits of network addresses in the range. For example, the prefix 192.0.2.0/24 (expressed in Classless Inter-Domain Routing or CIDR) can represent the first 24 bits of an IPv4 address, which corresponds to addresses in the range 192.0.2.0 to 192.0.2.255.

At block 708, access point 106 selects neighboring devices to be offered each pool of network addresses. In general, the available pools can be divided among available neighboring devices to provide different network paths from which to reach the access point. In one embodiment, such a selection is based on a specific tion within the configuration of the access point 106. In another embodiment, such a selection is determined by the access point 106. To illustrate, the access point 106 can divide the pools evenly among available neighboring devices (e.g., which can be found through typical routing protocols). In some cases, neighboring devices may additionally be determined based on a preferred routing of the traffic. For example, an access point 106 may be configured not to advertise a given pool of network addresses on a given neighboring device to cause that neighboring device to route requests for addresses within the pool to an alternative access point 106 .

Thereafter, at block 710, the global access point 106 transmits availability advertisements for the respective pools of network addresses to the relevant neighboring devices. The advertisements may be BGP protocol advertisements, for example. The transmission of BGP protocol advertisements is well known in the art and is therefore not described in detail in this document.

As described above, dividing the global network address into pools and advertising such pools to different neighboring devices may advantageously increase the resilience of the access points 106, particularly to failures or failures in neighboring networks. In particular, since each network-accessible service may be associated with different pools of global network addresses (e.g., one for each pool), client devices 102 of a service may be made aware (e.g., via DNS) of multiple routes to reach the service will. This technique may further increase the resilience of the access points 106 to limited failures at the access points 106, such as failures of a network interface or a router connected to a neighboring network. As noted above, in one embodiment, the resilience of each service is further increased by dividing each pool into subsets of network addresses, which in turn may be distributed among data flow managers 304 of an access point 106 . The associations between services and each subset can be "reordered" between pools. For example, the global network address for a service in a given pool can be chosen at random or can be chosen via a varied selection mechanism for each pool. This can result in the groupings of services becoming "disorganized" within subsets between the pools. In this way, if a single service's configuration is problematic for other services within the subset, it is likely that the other services affected by that misconfiguration will vary between pools. Because each service can be expected to be accessible through addresses in multiple pools, client devices 102 can connect to any other service through an alternate address of an alternate pool, thereby bypassing the problematic subset and pool. This reordering mechanism can therefore greatly increase the resilience of services using global network addresses.

With reference to 8th An illustrative routine 800 for routing traffic addressed to a global network address associated with a service provided by endpoints 202 within data centers 110 is described. The routine 800 may illustratively be implemented by a data flow manager 304 in an access point 106 . In one embodiment, an instance of routine 800 is implemented by each data flow manager 304 of each access point 106 .

Routine 800 begins at block 802 with data flow manager 304 receiving a data packet addressed to global network addresses advertised as accessible at access point 106 . The data packet can, for example, represent a request for access to a service that is provided via the network address and is accessible via the network. In an example, the data packet is formatted as a TCP or UDP data packet.

At block 803, the routine 800 varies depending on whether the data packet is associated with an existing data packet flow, as determined at block 803. Illustratively, the access point 106 can compare attributes of the data packet to attributes of previous data packets to determine whether the new data packet is within the same communication flow as a previous data packet. Any number of data flow identification techniques, a variety of which are known in the art, can be used to determine whether a data packet is in an existing data flow.

If the data packet is part of an existing data flow, the routine 800 proceeds to block 813 where the data flow manager 304 selects as the endpoint 202 for the data packet the same endpoint 202 selected for previous data packets within the data flow. In one embodiment, can each data flow manager 304 maintains a cache in memory that maps data flows to endpoints 202, facilitating this selection.

If the data packet is not part of an existing data flow, the routine 800 proceeds to block 804 where the data flow manager 304 identifies the network accessible service associated with the network address to which the data packet is addressed. Illustratively, data flow manager 304 may access information associating network addresses with associated services (e.g., as maintained in access point 106 memory) to identify the service associated with the address.

Thereafter, at block 806, the access point 106 selects a data center 110 that will provide the network accessible service. In one embodiment, data center 110 is selected based at least in part on geographic and network performance criteria between access point 106 and data center 110, and a load balancing algorithm for the network accessible service. Additionally, in one embodiment, the maximum desired threshold may be specified by a service provider, such as a desired percentage of traffic to be routed to each data center 110 . In another embodiment, the maximum desired threshold may be determined collectively by multiple access points 106 such that the combined thresholds of each access point 106 achieve the desired allocation of a service provider between data centers 110 . For example, access points 106 may implement a selection algorithm that aggregates traffic volumes into a service across access points 106 and optimally routes each data packet based on the access point 106 to which the data packet was addressed (and corresponding network performance criteria for each data center 110), definitely. The algorithm can then modify the optimal routing for each access point 106 as needed to implement a globally optimized routing, resulting in an individualized desired ratio between the data centers 110 for each access point 106. If so, the access point can change the initial selection to an alternative data center, such as a closest data center. In one embodiment, the implementation of block 806 repeats for each selected data center 110 to ensure that no data center receives more than a desired maximum fraction or volume of data packets.

After selecting a data center 110, the data flow manager 304 selects an endpoint 202 within the data center 110 to which the data packet is to be routed. In one embodiment, data flow manager 304 may use consistent hashing to select an endpoint 202 based on an attribute of the data packet (e.g., source IP address). As described above, in one embodiment, the data flow manager 304 may implement a weighted rendezvous hashing or weighted consistent hashing algorithm based on the 5-tuple information of the data packet. The data flow manager 304 can then select endpoints 202 based on an endpoint with the "highest" rating to enable consistent selection for the entire data flow.

Thereafter, at block 810, the data flow manager 304 applies NAT to the data packet. Illustratively, data flow manager 304 may replace a source address of the data packet with a global access point 106 address (e.g., a unicast address as opposed to a global network address) and a destination address with an endpoint 202 address (which is also a unicast address of the may be endpoint 202). The data flow manager 304 can also, for example, modify a port number of the data packet to facilitate later application of NAT techniques to data packets within the data flow. The data flow manager 304 can then transmit the data packet to the selected endpoint 202. The routine 800 may then end at block 814.

Various modifications of routine 800 are contemplated throughout this document. For example, while routine 800 includes three blocks related to selecting an endpoint 202 for data packets not associated with an existing data flow (blocks 804, 806, or 808), some implementations of routine 800 may exclude one or more of these blocks. Illustratively, rather than separately identifying a service and data centers related to that service, a data flow manager 304 may maintain a data association of a network address with the data centers 110 and endpoints 202 associated with the address. The data flow manager 304 can then select a data center 110 based on the criteria described above without having to directly identify the service associated with the address. In other embodiments, data flow manager 304 may only manage data corresponding to end network addresses points 202, and can directly select an endpoint 202 for a data packet without first identifying the service associated with the address or a data center 110 providing the service. For example, each endpoint 202 may be selected in a manner similar to the data centers 110 described above by selecting an endpoint 202 with a best performance criteria value and modifying that selection if the endpoint 202 receives more than a desired fraction of the data volume. Also, while the routine is 800 in 8th as shown ending at block 814 , a data flow manager 304 may perform other actions subsequent to routine 800 , such as acting as a NAT device for communication between client devices 102 and endpoints 202 . Thus, the representation of routine 800 is in 8th an embodiment of the concepts disclosed herein.

With reference to 9 an example routine 900 for selecting a data center 110 at the global access points 106 that provide a network accessible service is described. The routine 900 may be implemented by a data flow manager 304 in a global access point, for illustration. In one embodiment, an instance of routine 900 is implemented by each resource manager 206 of each data center.

The routine 900 begins at block 902 where the data flow manager 304 of a data center 110A obtains routing configuration information. Illustratively, the routing configuration information may include allocation or distribution of regions/sub-regions. At block 904, the data flow manager determines geographic proximity or other performance criteria. Illustratively, the distribution may be specified as a percentage of traffic, total number of data packets (e.g., total amount of data, cost allocated or billed to individual endpoints, and the like). A system administrator may illustratively use a software tool or interface (e.g., an API) to provide allocations, as described in various examples herein.

At block 906, the data flow manager forms the consideration set based on a product of geographic proximity and configuration. As described above, the data flow manager 304 may implement an algorithm, such as a coin toss algorithm, to implement the percentage selections. The algorithm can also take into account additional factors such as geographic or network criteria. At decision block 908, a test is performed to determine if there is a full allocation. If so, the routine sets the configuration at block 912 and the routine ends at block 914. Alternatively, data flow manager 304 may process using standard processes or autolinear processing.

With reference to 10 Illustrative interactions for performing an initialization phase of a connection-oriented communication session at a global access point 106A and handing off the session to a device within a data center 110 are described, thereby reducing the time required to establish such a session. The interactions in 10 may be implemented by a data flow manager 304 of an access point 106, for example. The interactions in 10 are described in relation to a single access point 106A and data center 110A. However, similar interactions with respect to other access points 106 and data centers 110 may occur simultaneously. In addition, the interactions in 10 described in relation to a particular type of connection-oriented communication session: a TCP session. Similar interactions can occur with respect to other connection-oriented communication sessions. 11 flanked 10 to illustrate the handshake aspect.

In addition, TCP transmission throughput may be limited by two considerations, congestion-based limitations and data-processing-based limitations. The congestion limitations in TCP try to manage the transmission so that it does not exceed the capacity of the network (congestion control). The data processing limitations in TCP attempt to manage the receiver's capacity to process data (data flow control). For illustration, individual TCP segments contain values for various TCP settings that allow congestion limits or data processing limits to be configured and implemented.

The interactions in 11 begin at (1) where a client device 102 attempts to initiate a connection-oriented communication session with a network service by transmitting a TCP SYN data packet addressed to a global network address of the service. In accordance with the functionalities described above, the TCP SYN data packet is sent to the global access point 106A transferred as in 11 shown. The TCP SYN data packet illustratively includes a sequence number for the client device 102. Also, in some embodiments, the TCP protocol implements a congestion avoidance algorithm that steadily increases a configuration of the number of data packets that are forwarded and the number of packets based on of data throughput increased linearly. In TCP, such values can have the maximum segment size (MMS). In this embodiment, the global access point can take advantage of the additional throughput between the global access point 206 and the endpoint 202 by increasing the initial value. This allows immediate use of the greater available bandwidth.

Similarly, global access point 206 may also set data processing limitations, namely maximum window size, to a larger value or largest value as global access point 206 has greater data processing capability.

At (2), a data flow manager 304 of global access point 106A continues the TCP three-way handshake by returning a TCP SYN-ACK data packet to client device 102, illustratively containing a sequence number for access point 106A (as well as confirming the sequence number 102 of the client device). At (3), the client device 102 continues the TCP three-way handshake by returning a TCP ACK data packet to the access point 106A, thereby confirming the sequence number of the access point 106A. Upon receipt of the TCP ACK data packet, a TCP session is initiated between client device 102 and global access point 106A. Since the access point 106A is expected to be close to the client device 102 (e.g., in terms of latency), it is expected that the interactions (1)-(3) relative to performing the initialization phase between of the client device 102 and an endpoint 202 are completed quickly.

At (4), the client device 102 (understanding that the TCP session has been initialized with the service) transmits a data packet within the TCP session addressed to the global network address of the service. The data packet is routed to the global access point 106A, which at (5) selects a data center 110 to which the data packet is to be routed. The selection of a data center can be done in the same way as described above (e.g. as interaction (3) in 6 ). The access point 106 also at (6) encapsulates the data packet for transmission to the selected data center 110 (in this case data center 110A) via a tunnel. In the interactions in 11 For example, assume that access point 106A has previously established a tunnel to data center 110A (e.g., to a session handover manager 212 of data center 110A). For example, access point 106A may maintain one or more idle TCP tunnels to data center 110A for use in transmitting data packets to data center 110. However, an additional interaction may be included where the global access point 106A establishes a tunnel to the data center 110A (e.g., a UDP or TCP tunnel). Illustratively, to facilitate handoff of the established TCP session, the encapsulated data packet also includes TCP session state information, such as the five-tuple for the TCP session and sequence numbers of the session. In the exemplary in 11 In the illustrated embodiment, the session state information is included as header information for the encapsulated data packet. At (7), the global access point 106A transmits the encapsulated data packet to the data center 110A (e.g., to a session delivery manager 212 within the data center 110A).

At (8), upon receiving the encapsulated data packet, the device within data center 110A (e.g., session handoff manager 212, or in some cases an endpoint 202) accepts the TCP session by using information specified in the encapsulated data packet (e.g., e.g. within a header) into their own TCP session state table. The device then decapsulates at (9) the data packet from the client device 102 and processes it at (10) as if it had been received directly at the device. For example, if the device is an endpoint 202, the endpoint 202 may process the data of the data packet according to the service requested by the client device 102. If the device is a session handoff manager 212, the manager 212 may process the data packet by identifying an endpoint 202 to service the data packet (e.g., in a manner similar to interaction (4) in 6 ), and use endpoint 202 to service the data packet. For example, since the session handover manager 212 (and in this case not the endpoint 202) is involved in the TCP connection with the client device 102, the manager 212 can initiate a second TCP connection with the endpoint 202 and the data within the data packet of the Client devices 102 forward to endpoint 202 over the second TCP session. Manager 212 may continue to operate as a proxy between client device 102 and endpoint 202 for future communications. For example, manager 212 may receive a response from endpoint 202 and enable transmission of the response to client device 102 over the TCP session with client device 102 .

At (11), the device within the data center 110A transmits a response packet to the global access point 106A, such as over the tunnel. Interaction (11) may involve, for example, encapsulating the response packet for transmission over the tunnel. The global access point 106A then forwards the response packet to the client device 102 at (12). Interaction (12) may include decapsulating the response packet for transmission to client device 102, for example. Other communications between client device 102 and an endpoint 202 within data center 110A may occur in a manner similar to interactions (4)-(7) and (9)-(12) above. Thus, the client device 102 can communicate with the data center 110A via a TCP session without actually having to complete an initialization phase of the session with a data center device 110A. 11 Figure 12 illustrates the interaction described above, illustrating the arbitration of two different TCP handshaking algorithms.

While 10 Figure 12 depicts illustrative interactions, such interactions may vary by embodiment. For example, while in 10 For example, while a response packet is illustrated as passing through global access point 106A, in some cases an endpoint 202 may be configured to respond directly to a client device 102 without the response passing through access point 106A. For example, instead of encapsulating a response packet and transmitting it to access point 106A, an endpoint 202 (or manager 212) may transmit the response packet directly to client device 102 (e.g., over network 104), eliminating the need for encapsulation is avoided. As another variant, while session information is in 11 described as being included in an encapsulated data packet, in other embodiments session state information may be included in a separate data packet. For example, a global access point 106A may be configured to separately transmit session state information to a data center 110A after an initialization phase of a communication session has been performed, such as in a "session accept" command to the data center 110A. As yet another variation, while manager 212 was discussed above as selecting an endpoint 202 within data center 110A to serve client device 102, in some cases an access point 106 may select such endpoint 202 even if a manager 212 is used. For example, an access point 106 can select an endpoint 202 and inform the manager 212 which endpoint has been selected (e.g., in headers of an encapsulated data packet). In cases where the global access point 106 selects an endpoint 202 to route a data packet to, the access point 106 may modify the data packet to facilitate routing to the endpoint 202. For example, within a destination address field of the data packet, the access point 106 may substitute a network address of the endpoint 202 (e.g., a "unicast" address) for a global network address for the service. Thus, the interactions in 10 be illustrative in nature.

While the interactions in 10 Illustrating interactions of a client device 102 with a single global access point 106A, a client device 102 may interact with multiple global access points 106 in some cases. As noted above, each global access point 106 may be configured to advertise global network addresses to a public network (e.g., network 104) that are not under the global access points 106 control. Thus, devices in such a network can generally determine the access point 106 to which data packets of a client device 102 addressed to a global network address are routed. In some configurations, redirecting packets within a given data packet flow to a different global access point 106 may adversely affect client device 102 communications. For example, when a client device 102 establishes a TCP connection that requires status information to be maintained at a single access point 106 (e.g., a TCP connection to the access point 106 or via NAT through the access point 106), a redirect may be used of client device 102 communicating with another access point 106 undesirably break that connection.

The interactions in 10 address this scenario by allowing TCP connections to be maintained between a client device 102 and the endpoint 202 (or session handoff manager 212) even when redirection of client device 102 packets to another access point 106 occurs . In particular, each access point 106 may be configured to apply the same load balancing criteria when selecting a data center 110 or endpoint 202 to which a client device 102 data packet is to be routed.

Such load balancing criteria may be independent of the access point 106 (e.g., immutably independent of the access point 106 on which they are applied). For example, the load balancing criteria may reflect the latency between a client device 102 and the data center 110, as well as the state of the data center 110 (or the endpoints 202 therein) independent of the latency to or from the access point 106. Therefore, each access point 106 is expected to route the data packets of a given client device 102 to the same data center 110 . For example, if the client device 102 were to transmit an additional data packet to a second access point 106, the second access point 106 would apply the same load balancing criteria to select the data center 110A as the destination for the data packet. The second access point 106 would then route the data packet to endpoint 202 (or session handoff manager 212), which would process the data packet in the same manner as if it had been routed through access point 106A. Since the interactions in 10 do not require status information to be maintained at access point 106A, and since the encapsulation mechanism of the 10 maintains a source network address of client device 102 within encapsulated data packets, no interruption of a connection-oriented communication session (e.g., a TCP session) would occur. Thus, the interactions in 10 Rerouting problems that would arise when using anycast technologies to design the routing of connection-oriented communication sessions.

With reference to 12 An illustrative routine 1200 for initiating a connection-oriented communication session at a global access point 106A and handing off the session to a manager 212 within a data center 110 is described. The routine 1200 is illustratively implemented jointly between the global access point 106A and the session to a manager 212, and thus the fields of the routine 1200 are partitioned among such devices. While portions of routine 1200 are shown as being implemented by a manager 212, in some cases these portions may alternatively be implemented directly in an endpoint 202.

The routine 1200 begins at block 1202 where the access point 106 (e.g., a data flow manager 304) receives a request directed to a global network address of a service to initiate a connection-oriented communication session with the service. The request can be a TCP SYN data packet, for example.

At block 1204, access point 106 completes an initialization phase of the session according to the particular protocol being used for the session. For example, if the protocol is TCP, the initialization phase may involve performing a three-way handshake with client device 102 .

At block 1206, access point 106 receives a data packet from client device 102 within the session. The data packet can be a user data packet within a TCP session, for example.

At block 1208, access point 106 encapsulates the data packet for transmission to manager 212 over a network tunnel. The access point 106 also includes in the encapsulated data packet (e.g., as header information for the encapsulated data packet) session context for the communication session, such as a TCP five-tuple and sequence numbers. The access point 106 then, at block 1210, sends the data packet to the handoff manager 212 as a handoff request for the communication session.

At block 1212, the handover manager 212 receives the encapsulated data packet and at block 1214 establishes a communication session within its local data based on the context information from the access point 106 . The manager 212 thus accepts the session, allowing subsequent intra-session communications between the client device 102 and the manager 212 . At block 1216, the handover manager 212 decapsulates the data packet from the client device 102 and processes the data packet within the session. For example, the manager 212 may select an endpoint 202 to process the request and transmit the contents of the data packet to the endpoint 202 over another communication session. The routine 1200 then ends at block 1218. Thus, the client device 102 and the manager 212 can communicate over the stateful session, requiring the client device 102 and the manager 212 to communicate in order to establish the session.

Routine 1200 may include additional or alternative blocks to those described above. For example, prior to sending an encapsulated data packet as a handoff request, the access point 106 may select a data center to receive the handoff request at block 1210, similarly as in choosing a data center that is in 9 was discussed. Additionally, while routine 1200 is shown as ending after block 1216, access point 106 and manager 212 may continue to operate to facilitate communications from or to a client device 120 within the session, as discussed above. Thus, the number and arrangement of the blocks are in 12 illustrative in nature.

All of the methods and processes described above can be embodied in, and fully automated through, software code modules executed by one or more general purpose computers or processors. The code modules may be stored on any type of non-transitory computer-readable medium or other computer storage device. Alternatively, some or all of the methods may be performed in specialized computer hardware.

Conditional language such as “may”, “could”, “would” or “would” among others, shall, unless specifically stated otherwise, is otherwise to be construed within the context as commonly used to convey that certain embodiments may include certain features, elements, and/or steps while other embodiments do not. Thus, such conditional language is not intended to generally imply that features, elements, and/or steps are in any way required for one or more embodiments, or that one or more embodiments necessarily include logic for deciding, with or without user input or prompting, whether those features, Elements and/or steps are included or intended to be performed in a specific embodiment.

Disjunctive language such as the phrase "at least one of X, Y, or Z," unless expressly stated otherwise, is otherwise to be understood with the context as it is commonly used to convey that an element, term, etc. is either X , Y or Z or any combination thereof (e.g. X, Y and/or Z). Thus, such disjunctive language is not generally intended and is not intended to imply that certain implementations require that at least one of X, at least one of Y, and at least one of Z be present.

Unless expressly stated otherwise, articles such as "a" or "an" should generally be interpreted as including one or more items being described. Accordingly, phrases such as "a device configured to be" are intended to include one or more named devices. One or more devices mentioned can also be configured together to carry out the statements mentioned. For example, "a processor configured to execute statements A, B, and C" may include a first processor configured to execute statement A operating in conjunction with a second processor configured to do so to carry out statements B and C.

All routine descriptions, elements or blocks in the flowcharts described in this specification and/or shown in the accompanying figures are to be understood as potentially representing modules, segments or sections of code that contain one or more executable instructions for implementing specific logical functions or elements in included in the routine. All alternative implementations are included within the scope of the embodiments described herein, wherein elements or functions may be deleted or performed out of the order shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as will occur to those skilled in the art is known.

It is to be emphasized that many variations and modifications can be made to the embodiments described above, the elements of which are to be understood as part of other acceptable examples. All such modifications and variations within the scope of this disclosure are intended to be included in this specification and protected by the following claims.

Examples of embodiments of the present disclosure may be described in terms of the following paragraphs.

Paragraph 1. A system comprising: a plurality of endpoints, each endpoint being at a different geographic location and including at least one server computing device configured to provide a network accessible service associated with a network address ; a plurality of global access points associated with the network accessible service, wherein individual global access points of the plurality of global access points include a processor, associated with different geographic locations, and configured to: use anycast methodology to advertise a network prefix of a network address as reachable via the access point; receiving a network packet addressed to the network address from a client device; selecting a data center having a plurality of endpoints based on an implementation of a distribution algorithm that corresponds to a product of network and geographic criteria and distribution criteria; selecting an endpoint from the plurality of endpoints included in the selected data center to which the network packet is to be routed based on the implementation of a selection algorithm according to a consistent hashing algorithm; and routing the transformed data packet to the selected endpoint.

Paragraph 2. The system of paragraph 1, wherein the data center corresponds to a region and wherein the plurality of endpoints correspond to two or more sub-regions.

Paragraph 3. The system of paragraph 2, wherein the distribution criteria of the distribution algorithm conform to a specification for the two or more subregions.

Paragraph 4. The system of paragraph 1, wherein the distribution criteria correspond to a full allocation of a percentage of network traffic to a region.

Paragraph 5. The system of paragraph 1, wherein the consistent hashing algorithm is one of a weighted rendezvous hashing algorithm or a weighted consistent hashing algorithm.

Paragraph 6. A method implemented at a plurality of access points to a network accessible service distributed across a geographic area, the method comprising: receiving a network packet from a client device at a global access point, wherein the network packet is addressed to the network address; selecting, by the global access point, a data center containing a plurality of endpoints based on an implementation of a distribution algorithm that corresponds to a product of network and geographic criteria and distribution criteria; selecting, by the global access point, an endpoint from the plurality of endpoints included in the selected data center to which the network packet is to be routed based on implementation of a corresponding selection algorithm that determines a rating for the network packet; and routing the transformed data packet to the selected endpoint.

Paragraph 7. The method of paragraph 6, further comprising offering a network address of the network accessible service as reachable via each access point.

Paragraph 8. The method of paragraph 6, wherein the data center corresponds to a region and wherein the plurality of endpoints correspond to two or more sub-regions.

Paragraph 9. The method of paragraph 8, wherein the distribution criteria of the distribution algorithm conform to a specification for the two or more sub-regions.

Paragraph 10. The method of paragraph 8, wherein the distribution criteria correspond to a full allocation of a percentage of network traffic to a region.

Paragraph 11. The method of paragraph 8, wherein the distribution criteria correspond to a partial allocation of a percentage of network traffic to a region.

Paragraph 12. The method of paragraph 11, wherein the allocation criteria include a default allocation for remaining portions of a sub-region.

Paragraph 13. Procedures in accordance with Paragraph 12, where the default allocation for remaining portions of the sub-region includes geographic proximity.

Paragraph 14. The method of paragraph 6, wherein the rating of the network packet corresponds to a rating based on at least one of a source IP address, a source port address, a destination IP address, a destination port address, and a protocol.

Paragraph 15. The method of paragraph 14, wherein selecting an endpoint from the plurality of endpoints corresponds to selecting the endpoint with the highest score.

Section 16. A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by each of a plurality of access points to a network-accessible service distributed across a geographic area, configure each access point to: receive a network packet from a client device addressed to a network address of the network accessible service; Selecting a data center from a plurality of available data centers and an associated endpoint from a plurality of endpoints associated with individual data centers to which the network packet is to be routed based on an implementation of a combination of based on an implementation of a distribution algorithm according to a product network and geographic criteria and distribution criteria and a selection algorithm corresponding to a consistent hashing algorithm that determines a score for the network packet; and routing the transformed data packet to the selected endpoint.

Paragraph 17. The non-transitory computer-readable medium of paragraph 16, wherein the data center corresponds to a region and wherein the plurality of endpoints correspond to two or more sub-regions.

Paragraph 18. The non-transitory computer-readable medium of paragraph 16, wherein the distribution criteria conform to a scalable allocation.

Paragraph 19. The non-transitory computer-readable medium of paragraph 16, wherein the rating of the network packet is a rating based on at least one of a source IP address, a source port address, a destination IP address, a destination port address, and a protocol is equivalent to.

Paragraph 20. The non-transitory computer-readable medium of paragraph 16, wherein the distribution criteria correspond to performance criteria and the endpoint is based at least in part on an apparent state of the endpoint to the access point.

Paragraph 21. A system comprising: a plurality of endpoints, each endpoint being at a different geographic location and including at least one server computing device configured to provide a network accessible service associated with a network address ; at least one global access point of the network accessible service, the at least one global access point including a processor and configured to: receive a network packet addressed to the network address from a client device; selecting a data center containing a plurality of endpoints based on an implementation of a distribution algorithm; selecting an endpoint from the plurality of endpoints to which the network packet is to be routed based on the implementation of a selection algorithm; and routing the received data packet to the selected endpoint via encapsulated communications; wherein the plurality of endpoints are assigned to subgroups that share a common encapsulated tunnel, and wherein each of the endpoints in the subgroup is configured to transmit communication messages to other subgroups to forward client data packets and/or to provide fragmentation information.

Paragraph 22. The system of paragraph 21, wherein the encapsulation tunnel corresponds to a virtual local area network.

Paragraph 23. The system of paragraph 21, wherein the plurality of endpoints include network components for processing the subgroup messages using five-tuple communication.

Paragraph 24. The system of paragraph 21, wherein the communication information messages include path maximum transmission size unit identification packets.

Paragraph 25. The system of paragraph 21, wherein each of the endpoints transmits the communication messages via a multicast message.

Paragraph 26. The system of paragraph 21, wherein each of the endpoints transmits the communication messages via a broadcast message.

Paragraph 27. A method implemented at a plurality of access points to a network accessible service distributed across a geographic area, the method comprising: receiving a network packet from a client device at an access point, the network packet is addressed to a network address associated with a service; selecting, by the access point, a data center containing a plurality of endpoints based on an implementation of a distribution algorithm; selecting, by the access point, an endpoint from the plurality of endpoints to which the network packet is to be routed based on the implementation of a selection algorithm; and routing the received data packet through the access point to the selected endpoint via an encapsulated tunnel, wherein the plurality of endpoints are assigned to subgroups that share a common encapsulated tunnel, and wherein each of the endpoints in the subgroup is configured to transmit management messages to other endpoints in the subgroup to forward client data packets and/or to provide fragmentation information.

Paragraph 28. The method of paragraph 27, wherein the encapsulation tunnel corresponds to a virtual local area network.

Paragraph 29. The method of paragraph 27, wherein the plurality of endpoints include network components for routing received data packets using five-tuple communication.

Paragraph 30. The method of paragraph 27, wherein the five-tuple communication includes a source IP address, a source port address, a destination IP address, a destination port address, and a protocol.

Paragraph 31. The method of paragraph 27, wherein the management messages include path maximum transfer size unit identification packets.

Paragraph 32. The method of paragraph 27, wherein each of the endpoints transmits the management messages via a multicast communication.

Paragraph 33. The method of paragraph 32, wherein individual endpoints are registered with a single multicast communication channel based on an associated subset.

Paragraph 34. The method of paragraph 27, wherein each of the endpoints transmits the management messages via a broadcast communication.

Paragraph 35. The method of paragraph 34, wherein each individual endpoint is registered to broadcast communication channels and to filter information packets that are not associated with an assigned subgroup.

Section 36. A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by each of a plurality of access points to a network-accessible service distributed across a geographic area, configure each access point to: receive a network packet from a client device addressed to a network address of the network accessible service; selecting a data center and an associated endpoint from the plurality of endpoints associated with the selected data center based on a combination of a distribution algorithm based on a product of network and geographic criteria and distribution criteria and a selection algorithm based on message attributes; and routing the transformed data packets to the endpoint via an encapsulated tunnel associated with an assigned subgroup, wherein the plurality of endpoints are assigned to subgroups that share a common encapsulated tunnel, and wherein each of the endpoints in the subgroup is configured to transmit messages to other subgroups to forward client data packets and/or provide fragmentation information.

Paragraph 37. The non-transitory computer-readable medium of paragraph 36, wherein the plurality of endpoints includes network components for routing network packets using five-tuple communication.

Paragraph 38. The non-transitory computer-readable medium of paragraph 36, wherein the messages exchanged by endpoints in a subset include unit identifier packets of the maximum transmission size.

Section 39. The non-transitory computer-readable medium of Section 36, wherein individual endpoints transmit messages via a multicast transmission.

Paragraph 40. The non-transitory computer-readable medium of paragraph 39, wherein individual endpoints are registered with multicast communication channels based on an associated subset.

Section 41. The non-transitory computer-readable medium of Section 36, wherein individual endpoints transmit messages via a broadcast transmission.

Paragraph 42. The non-transitory computer-readable medium of paragraph 41, wherein each individual endpoint is registered to broadcast communication channels and filter messages not associated with an assigned subgroup.

Paragraph 43. A system comprising: a plurality of endpoint systems, each endpoint system located in a different geographic location and including at least one endpoint computing device configured to provide a network accessible service associated with a network address; at least one access point associated with the network accessible service, the at least one access point comprising a processor and configured to: receive a request from a client device to initiate a first Transmission Control Protocol (TCP) session with the network accessible service; performing a first initialization phase of the first TCP session to create context information for the TCP session including at least one sequence number; receiving a data packet from the client device as part of the first TCP session; selecting an endpoint system from the plurality of endpoint systems to which the network packet is to be routed; performing a second phase of initialization of a second TCP session to establish a TCP session with the selected endpoint system, the second TCP session being independent of the first TCP session; and transmitting the received data packet as part of the second TCP session.

Paragraph 44. The system of paragraph 43, wherein the first TCP session is associated with a first sequence number and a second sequence number assigned by the access point.

Paragraph 45. The system of paragraph 44, wherein the second TCP session is associated with the first sequence number and a third sequence number assigned by the selected endpoint.

Paragraph 46. The system of paragraph 45, wherein the access point is configured to translate the third sequence number and the second sequence number.

Paragraph 47. The system of paragraph 43, wherein the access point is configured to perform the second initialization phase of the second TCP session with a regional access point.

Paragraph 48. A method implemented at each of a plurality of access points to a network accessible service distributed across a geographic area, the method comprising: at a first access point of the plurality of access points: receiving a request from a client device for initiating a connection-oriented communication session with the network accessible service; performing a first initialization phase of the connection-oriented communication session to create context information for the connection-oriented communication session; applying selection criteria to select an endpoint system from a plurality of endpoint systems for the network accessible service to which the network packet is to be routed; performing a second connection-oriented communication session initialization phase to establish context information for a connection-oriented communication session with the selected endpoint system; and transmitting the data packet and the context information for the connection-oriented communication session to the endpoint system.

Paragraph 49. The method of paragraph 48, wherein the connection-oriented communication sessions each correspond to a Transmission Control Protocol (TCP) session.

Paragraph 50. The method of paragraph 49, wherein the first TCP session is associated with a first sequence number and a second sequence number assigned by the access point.

Paragraph 51. The method of paragraph 50, wherein the second TCP session is associated with the first sequence number and a third sequence number assigned by the selected endpoint.

Paragraph 52. The method of paragraph 51, further comprising translating the third sequence number and the second sequence number.

Paragraph 53. The method of paragraph 48, wherein performing a second initialization phase in a connection-oriented manner includes performing the second initialization phase directly with the selected endpoint.

Paragraph 54. The method of paragraph 48, wherein performing a second initialization phase in a connection-oriented manner includes performing the second initialization phase directly with a regional access point.

Paragraph 55. The method of paragraph 48, wherein the regional access point performs a third initialization phase of the connection-oriented communication session to establish context information for the connection-oriented communication session with the selected endpoint.

Section 56. A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by each of a plurality of access points to a network-accessible service distributed across a geographic area, configure each access point to: receive a request from a client apparatus for initiating a first connection-oriented communication session with the network accessible service; performing a first initialization phase of a TCP communication session to create context information for the first connection-oriented communication session; applying selection criteria to select an endpoint system from a plurality of endpoint systems for the network accessible service to which the network packet is to be routed; and performing a second phase of initialization of an independent TCP communication session to establish context information for a second connection-oriented communication session with the selected endpoint system.

Paragraph 57. The non-transitory computer-readable medium of paragraph 56, wherein the first TCP session is associated with a first sequence number and a second sequence number assigned by the access point.

Paragraph 58. The non-transitory computer-readable medium of paragraph 56, wherein the second TCP session is associated with a first sequence number and a third sequence number assigned by the selected endpoint.

Paragraph 59. The non-transitory computer-readable medium of paragraph 56, wherein the instructions are further operable to effect translation of sequence numbers between the first and second communication sessions.

Paragraph 60. The non-transitory computer-readable medium of paragraph 56, wherein performing a second phase of initialization in a connection-oriented manner includes performing the second phase of initialization directly with the selected endpoint.

Paragraph 61. The non-transitory computer-readable medium of paragraph 56, wherein performing the second phase initialization of an independent TCP communication includes performing the second phase initialization directly with a regional access point.

Paragraph 62. The non-transitory computer-readable medium of paragraph 56, wherein the instructions are further operable to cause transmission of the received data packet in accordance with the second connection-oriented communication session.

Paragraph 63. A system comprising: a plurality of endpoint systems, each endpoint system located in a different geographic location and including at least one endpoint computing device configured to provide a network accessible service associated with a network address; at least one access point associated with the network accessible service, each of the at least one access point comprising a processor and configured to: establish a first TCP session to receive context information for a first establish a TCP session between the at least one access point and a client device, the first TCP session including a specification of a congestion control parameter and a data validation parameter; selecting an endpoint system from the plurality of endpoint systems to which the network packet is to be routed; and establishing a second TCP session to establish a TCP session with the selected endpoint system, the second TCP session being independent of the first TCP session and the second TCP session including a specification of a congestion control parameter and a data validation parameter.

Paragraph 64. The system of paragraph 63, wherein the first TCP session and the second TCP session are associated with a different congestion parameter.

Paragraph 65. The system of paragraph 64, wherein the congestion parameter corresponds to a selection from a set of values, and wherein the first TCP session corresponds to a lower value in the set of values.

Paragraph 66. The system of paragraph 63, wherein the first TCP session and the second TCP session are associated with a different data validation parameter.

Paragraph 67. The system of paragraph 66, wherein the data validation parameter corresponds to a selection from a set of values, and wherein the second TCP session corresponds to a maximum value in the set of values.

Paragraph 68. A method implemented at each of a plurality of access points to a network accessible service distributed across a geographic area, the method comprising: receiving a request from a client device to initiate a first session with transmission control protocol (TCP) with the service accessible over the network; establishing a first TCP session to establish context information for the first TCP session including a specification of a congestion control parameter and a data validation parameter; selecting an endpoint system from the plurality of endpoint systems to which the network packet is to be routed; and establishing a second TCP session to establish a second TCP session with the selected endpoint system, the second TCP session being independent of the first TCP session and the second TCP session including a specification of a congestion control parameter and a data validation parameter.

Paragraph 69. The method of paragraph 68, wherein the first TCP session and the second TCP session are associated with a different congestion parameter.

Paragraph 70. The method of paragraph 69, wherein the congestion parameter corresponds to a selection from a set of values, and wherein the first TCP session corresponds to a lower value in the set of values.

Paragraph 71. The method of paragraph 69, wherein the congestion parameter corresponds to a selection from a set of values, and wherein the second TCP session corresponds to a highest value in the set of values.

Paragraph 72. The method of paragraph 68, wherein the first TCP session and the second TCP session are associated with a different data validation parameter.

Paragraph 73. The method of paragraph 72, wherein the data validation parameter corresponds to a selection from a set of values, and wherein the second TCP session corresponds to a maximum value in the set of values.

Paragraph 74. The method of paragraph 72, wherein the data validation parameter corresponds to a selection from a set of values, and wherein the first TCP session corresponds to a minimum value in the set of values.

Paragraph 75. The method of paragraph 68, wherein the congestion parameter corresponds to a maximum segment size specification.

Paragraph 76. The method of paragraph 68, wherein the data validation parameter corresponds to a specification of a window scaling value.

Section 77. A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by each of a plurality of access points to a network-accessible service that is distributed over a geographic area, configure each access point to: receive a request from a client apparatus for initiating a first Transmission Control Protocol (TCP) session with the network accessible service; establishing a first TCP session to establish context information for the TCP session including a specification of a congestion control parameter and a data validation parameter; selecting an endpoint system from the plurality of endpoint systems to which the network packet is to be routed; and establishing a second TCP session to establish a second TCP session with the selected endpoint system, the second TCP session being independent of the first TCP session and the second TCP session including a specification of a congestion control parameter and a data validation parameter.

Paragraph 78. The non-transitory computer-readable medium of paragraph 77, wherein the first TCP session and the second TCP session are associated with a different congestion parameter.

Paragraph 79. The non-transitory computer-readable medium of paragraph 77, wherein the congestion parameter corresponds to a selection from a set of values, and wherein the first TCP session corresponds to a lowest value in the set of values.

Paragraph 80. The non-transitory computer-readable medium of paragraph 77, wherein the first TCP session and the second TCP session are associated with a different data validation parameter.

Paragraph 81. The non-transitory computer-readable medium of paragraph 77, wherein the data validation parameter corresponds to a selection from a set of values, and wherein the second TCP session corresponds to a maximum value in the set of values.

Paragraph 82. The non-transitory computer-readable medium of paragraph 77, wherein the data validation parameter corresponds to a selection from a set of values, and wherein the first TCP session corresponds to a minimum value in the set of values.

Claims (15)

System that includes: a plurality of endpoints, each endpoint being in a different geographic location and including at least one server computing device configured to provide a network accessible service associated with a network address; a plurality of global access points associated with the network accessible service, wherein individual global access points of the plurality of global access points include a processor, are associated with different geographic locations, and are configured to: receiving a network packet addressed to the network address from a client device; selecting a data center containing a plurality of endpoints based on an implementation of a distribution algorithm; selecting an endpoint from the plurality of endpoints to which the network packet is to be routed based on the implementation of a selection algorithm; and Route the received data packet to the selected endpoint via encapsulated communications. system after claim 1 , where each global access point is configured to use anycast methodology to advertise a network prefix of the network address as reachable through the access point. system after claim 1 , wherein each global access point is configured to select a data center containing a plurality of endpoints based on the implementation of a distribution algorithm that is a product of network and geographic criteria and distribution criteria. system after claim 1 wherein the plurality of endpoints are assigned to subgroups that share a common encapsulated tunnel, and wherein each of the endpoints in the subgroup is configured to transmit communication messages to other subgroups to forward client data packets and/or to provide fragmentation information. system after claim 1 , wherein each global access point is configured to select an endpoint from the plurality of endpoints included in the selected data center to which the network packet is to be routed based on implementation of a selection algorithm that corresponds to a consistent hashing algorithm . system after claim 1 , where the distribution criteria correspond to a full allocation of a percentage of network traffic to a region. system after claim 1 , where the consistent hashing algorithm is one of a weighted rendezvous hashing algorithm or a weighted consistent hashing algorithm. system after claim 1 , where the encapsulation tunnel corresponds to a virtual local area network. system after claim 1 wherein the plurality of endpoints includes network components for processing the subgroup messages using five-tuple communication. system after claim 1 wherein the communication information messages include path maximum transmission size unit identification packets. system after claim 1 , wherein each global access point is configured to: establish a first TCP session to establish context information for a first TCP session between the at least one access point and a client device, the first TCP session including a specification of a congestion control parameter and a data validation parameter; selecting an endpoint system from the plurality of endpoint systems to which the network packet is to be routed; and establishing a second TCP session to establish a TCP session with the selected endpoint system, the second TCP session being independent of the first TCP session and the second TCP session including a specification of a congestion control parameter and a data validation parameter. system after claim 1 wherein each global access point is configured to: receive a request from a client device to initiate a first Transmission Control Protocol (TCP) session with the network accessible service; performing a first initialization phase of the first TCP session to create context information for the TCP session including at least one sequence number; receiving a data packet from the client device as part of the first TCP session; selecting an endpoint system from the plurality of endpoint systems to which the network packet is to be routed; performing a second phase of initialization of a second TCP session to establish a TCP session with the selected endpoint system, the second TCP session being independent of the first TCP session; and transmitting the received data packet as part of the second TCP session. A method implemented at a plurality of access points to a network accessible service distributed over a geographic area, the method comprising: receiving a network packet from a client device at an access point, the network packet being sent to a a network address associated with a service is addressed, the access point selecting a data center containing a plurality of endpoints based on an implementation of a distribution algorithm; selecting an endpoint from the plurality of endpoints to which the network packet is to be routed based on the access point's implementation of a selection algorithm; and Routing the received data packet through the access point to the selected endpoint via an encapsulated tunnel. procedure after Claim 13 wherein the plurality of endpoints are assigned to subgroups that share a common encapsulated tunnel, and wherein each of the endpoints in the subgroup is configured to transmit management messages to other endpoints in the subgroups to forward client data packets and/or to provide fragmentation information . procedure after Claim 13 , wherein routing the received data packet includes routing the received data packet through the access point to the selected endpoint via an encapsulated tunnel, wherein the plurality of endpoints are assigned to subgroups that share a common encapsulated tunnel, and wherein each of the endpoints is in the subgroup configured to transmit management messages to other endpoints in the subgroup to forward client data packets and/or provide fragmentation information.

Images