KR102304416B1 - Automatic Tuning of Hybrid WAN Links by Adaptive Replication of Packets on Alternate Links - Google Patents

Abstract

Embodiments of the present disclosure describe systems and methods for dynamically replicating network packets over a reliable link. A network device may be connected by a primary link and a secondary link. As the network quality of the primary link degrades, the system can adaptively replicate network packets to the secondary link. As a high priority link, the secondary link may have less total bandwidth compared to the primary link. Thus, all traffic traversing the primary link may not be able to be replicated on the secondary link to maintain network reliability when the quality of the primary link degrades. This solution can effectively manage the bandwidth of the secondary link and set the replication rate to control the replication of packets to the secondary link.

Description

Automatic Tuning of Hybrid WAN Links by Adaptive Replication of Packets on Alternate Links

Related applications

This patent application claims priority to U.S. Patent Application Serial No. 15/463,635, entitled "Automatic Tuning of Hybrid WAN Links by Adaptive Replication of Packets on Alternate Links," filed March 20, 2017, and The content is incorporated herein by reference for all purposes in its entirety.

initiation field

BACKGROUND This application relates generally to data communication networks. In particular, this application relates to systems and methods for dynamically replicating network packets.

The network devices may be connected through multiple virtual wide area networks (WANs) or other networks. Various connections between network devices can experience reduced network performance and other network issues that can lead to packet loss and increased packet latency. Packet loss and packet latency can result in network outages between network devices.

The present solution describes a system and method for dynamically replicating network packets to a reliable link. A network device may be connected by a primary link and a secondary link. In some cases, the secondary link may be a high priority link. As the network quality of the primary link degrades, the system can adaptively replicate network packets to the secondary link. As a high priority link, the secondary link may have less total bandwidth compared to the primary link. Thus, all traffic traversing the primary link may not be able to be replicated on the secondary link to maintain network reliability when the quality of the primary link degrades. This solution can effectively manage the bandwidth of the secondary link and set the replication rate to control the replication of packets to the secondary link. The replication rate may be set based on the link quality of the primary link such that the effective packet loss rate (or packet delay rate) at the receiving device remains above a threshold.

According to an aspect of the present disclosure, there is provided a method for transmitting a duplicate packet based on a network state, wherein, by a first device, a metric of a network state of a first link established between a first device and a second device is a threshold value. determining whether it is within The method generates, in response to determining, by the first device, that a metric of a network state of the first link is within a threshold, duplicate packets of packets to be transmitted from a first device to a second device over the first link. may include the step of The method may include transmitting, by the first device, packets to be transmitted from the first device to the second device on the first link. The method may include transmitting, by the device, duplicate packets on a second link established between the first device and the second device.

The method may include determining, by the first device, a packet replication rate based on a difference between a metric and a threshold value of a network condition of the first link. The method may include generating, by the first device, the copy packets based on the determined packet copy rate. The threshold may be a first threshold. The method may include monitoring, by the first device, a network state of the first link. The method may include determining, by the first device, whether a metric of the network condition is within a second threshold. The method includes, in response to determining, by the first device, that the metric of a network condition of the first link is within a second threshold, a packet replication rate that may correspond to a rate at which duplicate packets are generated and transmitted on the second link. may include modifying

The method may include determining, by the first device, a bandwidth of the second link. The method may include determining, by the first device, a packet duplication rate based on the determined metric of a bandwidth of the second link and a network state of the first link. Generating duplicate packets of packets to be transmitted from the first device to the second device via the first link may include generating duplicate packets based on a packet copy rate.

In some embodiments, determining the metric of the network state of the first link established between the first device and the second device comprises determining one of a packet loss rate of the first link or a packet delay rate of the first link. may include.

In some embodiments, the first link may be an Internet broadband link and the second link may be a multiprotocol label switching link. The first device and the second device may be virtual wide area network devices.

The method may include determining, by the first device, a first packet loss rate of the first link corresponding to a number of packets lost on the first link. The method may include determining, by the first device, a packet acceptance rate of the second link based on a number of duplicate packets accepted for lost packets on the first link. The method may include determining, by the first device, an effective packet loss rate of the first link based on a difference between the first packet loss rate and the packet acceptance rate. The metric of the network state of the first link may be an effective packet loss rate of the first link.

The method may include determining, by the first device, a first packet delay rate of the first link corresponding to a number of packets delayed on the first link. The method may include determining, by the first device, a packet acceptance rate of the second link based on a number of duplicate packets accepted for delayed packets on the first link. The method may include determining, by the first device, an effective packet delay rate of the first link based on a difference between the first packet delay rate and the packet acceptance rate. The metric of the network state of the first link is the effective packet delay rate of the first link.

The method may include monitoring, by the first device, a network status of the first link to update a metric of the network status. The method may include determining, by the first device, in response to transmitting the duplicate packets on the second link, whether an updated metric of a network state of the first link is within a threshold. The method may include increasing, by the first device, a packet replication rate in response to determining, by the first device, that the updated metric of the network condition is within the threshold.

The method may include determining, by the first device, whether a current metric of a network state of the first link is within a threshold value. The method may include determining, by the first device, whether a bandwidth usage metric of the second link is within a predetermined maximum bandwidth usage value assigned to the second link. The method comprises, in response to determining, by the first device, that a current metric of a network state of the first link is within a threshold, increasing a packet replication rate to increase the number of replicated packets generated per unit time. may include steps. The method may include transmitting, by the first device, some of the duplicate packets on the first link in response to determining that the bandwidth usage metric of the second link is within a predetermined maximum bandwidth usage value.

According to another aspect of the present disclosure, a system for transmitting duplicate packets based on a network condition includes a first device. The first device may be configured to establish a first link with the second device and establish a second link with the second device. The first device may be configured to determine a metric of a network state of a first link established between the first device and the second device. In response to determining that the metric of the network state of the first link is within the threshold, the first device may generate duplicate packets of packets to be transmitted from the first device to the second device over the first link. The first device may transmit packets to be transmitted from the first device to the second device on the first link. The first device may transmit duplicate packets on a second link established between the first device and the second device.

In some embodiments, the first device may determine the packet replication rate based on a difference between the metric and the threshold of the network state of the first link. The first device may also generate duplicate packets based on the determined packet duplication rate.

In some embodiments, the threshold may be a first threshold. The first device may be configured to monitor a network status of the first link. The first device may be configured to determine whether the metric of the network condition is within a second threshold. The first device may be configured to, in response to determining that the metric of a network state of the first link is within a second threshold, modify a packet replication rate corresponding to a rate at which duplicate packets are generated and transmitted on the second link. have.

The first device may be configured to determine a bandwidth of the second link. The first device may be configured to determine the packet replication rate based on the determined metric of a bandwidth of the second link and a network state of the first link. The first device may be configured to generate duplicate packets based on the packet duplication rate.

To determine a metric of a network state of a first link established between the first device and the second device, the first device is configured to determine one of a packet loss rate of the first link or a packet delay rate of the first link . The first link may be an Internet broadband link, and the second link may be a multiprotocol label switching link. The first device and the second device may be virtual wide area network devices.

The first device may be configured to determine a first packet loss rate of the first link that may correspond to a number of packets lost on the first link. The first device may determine the packet acceptance rate of the second link based on the number of duplicate packets accepted for lost packets on the first link. The first device may determine an effective packet loss rate of the first link based on a difference between the first packet loss rate and the packet acceptance rate. The metric of the network state of the first link may be an effective packet loss rate of the first link.

The first device may be configured to determine a first packet delay rate of the first link corresponding to a number of packets delayed on the first link. The first device may be configured to determine a packet acceptance rate of the second link based on a number of duplicate packets accepted for delayed packets on the first link. The first device may be configured to determine an effective packet delay rate of the first link based on a difference between the first packet delay rate and the packet acceptance rate. The metric of the network state of the first link is the effective packet delay rate of the first link.

The first device may monitor the network status of the first link to update a metric of the network status. In response to transmitting the duplicate packets on the second link, the first device may determine whether an updated metric of a network state of the first link is within a threshold. The first device may increase the packet replication rate in response to determining that the updated metric of the network condition is within the threshold.

The first device may determine whether a current metric of a network state of the first link is within a threshold value. The first device may determine whether a bandwidth usage metric of the second link is within a predetermined maximum bandwidth usage value assigned to the second link. In response to determining that the current metric of the network state of the first link is within the threshold, the first device may increase the packet replication rate to increase the number of replicated packets generated per unit time. The first device may be configured to transmit the portion of the duplicate packets on the first link in response to determining that the bandwidth usage metric of the second link is within a predetermined maximum bandwidth usage value.

The above and other objects, aspects, features and advantages of the present solution will become more apparent and better understood by reference to the following description taken in conjunction with the accompanying drawings.
1A is a block diagram of an embodiment of a network environment for a client to access a server via a device.
1B is a block diagram of an embodiment of an environment for transferring a computing environment from a server to a client via a device.
1C is a block diagram of another embodiment of an environment for transferring a computing environment from a server to a client via a device.
1D is a block diagram of another embodiment of an environment for transferring a computing environment from a server to a client via a device.
1E-1H are block diagrams of an embodiment of a computing device.
2A is a block diagram of an embodiment of a device for processing communications between a client and a server;
2B is a block diagram of another embodiment of a device for optimizing, accelerating, load-balancing and routing communications between a client and a server;
3 is a block diagram of an embodiment of a client for communicating with a server via a device;
4A is a block diagram of an embodiment of a virtualized environment;
4B is a block diagram of another embodiment of a virtualized environment.
4C is a block diagram of an embodiment of a virtualized device.
5A is a block diagram of an embodiment of an approach for implementing parallelism in a multi-core system.
5B is a block diagram of an embodiment of a system utilizing a multi-core system.
5C is a block diagram of another embodiment of an aspect of a multi-core system.
6 is a block diagram of an embodiment of a cluster system.
7 shows a block diagram of an exemplary intermediate device.
Fig. 8a shows duplication of packets based on effective packet loss rate.
Fig. 8b shows duplication of packets based on effective packet delay rate.
9 shows an exemplary flow diagram of an exemplary method for transmitting a duplicate packet based on a network condition.
BRIEF DESCRIPTION OF THE DRAWINGS The features and advantages of the present invention will become more apparent from the detailed description set forth below in connection with the drawings, in which like reference characters identify corresponding elements throughout. In the drawings, like reference numbers generally indicate identical, functionally similar, and/or structurally similar elements.

In order to read the description of the various embodiments that follow, the following description of the sections of this specification and their respective contents may be helpful:

Section A describes network environments and computing environments that may be useful in practicing the embodiments described herein;

Section B describes embodiments of systems and methods for delivering a computing environment to a remote user;

Section C describes embodiments of systems and methods for accelerating communication between clients and servers;

Section D describes an embodiment of a system and method for virtualizing an application delivery controller;

Section E describes embodiments of systems and methods for providing multi-core architectures and environments;

Section F describes embodiments of systems and methods for providing a clustered device architecture environment; and

Section G describes an embodiment of a system and method for adaptive packet replication.

A. Network and Computing Environments

Before discussing the specifics of embodiments of systems and methods of devices and/or clients, it may be helpful to discuss the network and computing environments in which such embodiments may be deployed. Referring now to FIG. 1A, an embodiment of a network environment is illustrated. In a brief overview, a network environment includes one or more servers 106a - 106n (generally server(s) 106 or remote machines via one or more networks 104 , 104 ′ (generally referred to as network 104 ). one or more clients 102a - 102n (also commonly referred to as local machine(s) 102 or client(s) 102 ) in communication with ( also referred to as (s) 106 ). In some embodiments, the client 102 communicates with the server 106 via the device 200 .

Although FIG. 1A shows a network 104 and a network 104 ′ between a client 102 and a server 106 , the client 102 and the server 106 may be on the same network 104 . Networks 104 and 104' may be the same type of network or different types of networks. Network 104 and/or network 104' may be a corporate intranet, a local area network (LAN), such as a Metropolitan Area Network (MAN), or a wide area network (WAN), such as the Internet or the World Wide Web. . In one embodiment, network 104' may be a private network and network 104 may be a public network. In some embodiments, network 104 may be a private network and network 104' may be a public network. In other embodiments, networks 104 and 104' may both be private networks. In some embodiments, the client 102 may be located at a branch of the corporate enterprise that communicates via a WAN connection with a server 106 located in the corporate data center via the network 104 .

Network 104 and/or 104' may be any type and/or type of network, and may be a point-to-point network, a broadcast network, a wide area network, a local area network, a telecommunications network, a data communication network, a computer network, an ATM It may include any of an Asynchronous Transfer Mode (Asynchronous Transfer Mode) network, a Synchronous Optical Network (SONET) network, a Synchronous Digital Hierarchy (SDH) network, a wireless network, and a wired network. In some embodiments, network 104 may include a wireless link such as an infrared channel or satellite band. The topology of networks 104 and/or 104' may be a bus, star or ring network topology. Networks 104 and/or 104' and network topologies may be any such networks or network topologies known to those of ordinary skill in the art capable of supporting the operations described herein.

As shown in FIG. 1A , a device 200 , which may also be referred to as an interface unit 200 or gateway 200 , is shown between networks 104 and 104 ′. In some embodiments, device 200 may be located on network 104 . For example, a branch of a company may place the device 200 at the branch. In other embodiments, device 200 may be located on network 104 ′. For example, device 200 may be located in a corporate data center. In another embodiment, a plurality of devices 200 may be deployed on the network 104 . In some embodiments, a plurality of devices 200 may be deployed on a network 104 ′. In one embodiment, the first device 200 communicates with the second device 200 ′. In other embodiments, device 200 may be part of any client 102 or server 106 on the same or different network 104 , 104 ′ as client 102 . One or more devices 200 may be located at any point in the network or in the network communication path between the client 102 and the server 106 .

In some embodiments, appliance 200 is referred to as a NetScaler® device, Ft. includes any network device manufactured by Citrix Systems, Inc. of Lauderdale Florida. In another embodiment, device 200 includes any product embodiment referred to as BigIP and WebAccelerator manufactured by F5 Networks, Inc. of Seattle, Washington. In another embodiment, appliance 200 is any DX acceleration device platform and/or SSL VPN device series, such as SA 700, SA 2000, SA 4000 and SA 6000 devices manufactured by Juniper Networks, Inc. of Sunnyvale, California. includes In another embodiment, the device 200 accelerates any application manufactured by Cisco Systems, Inc. of San Jose, California, such as the Cisco ACE Application Control Engine Module service software and network module and the Cisco AVS Series Application Velocity System. and/or security related devices and/or software.

In one embodiment, the system may include a plurality of logically-grouped servers 106 . In these embodiments, a logical grouping of servers may be referred to as a server farm 38 . In some of these embodiments, the servers 106 may be geographically distributed. In some cases, the farm 38 may be managed as a single entity. In another embodiment, server farm 38 includes a plurality of server farms 38 . In one embodiment, the server farm runs one or more applications on behalf of one or more clients 102 .

The servers 106 in each farm 38 may be heterogeneous. One or more servers 106 may operate according to one type of operating system platform (eg, WINDOWS NT manufactured by Microsoft Corp. of Redmond, Washington), while one or more of the other servers 106 may be It can operate on different types of operating system platforms (eg Unix or Linux). A server 106 in each farm 38 need not be in physical proximity to another server 106 in the same farm 38 . Thus, groups of servers 106 logically grouped as farms 38 may be interconnected using wide-area network (WAN) connections or medium-area network (MAN) connections. . For example, farm 38 may include servers 106 physically located on different continents or on different continents, countries, states, cities, campuses, or different regions of the room. When the servers 106 are connected using a local-area network (LAN) connection or some form of direct connection, the data transfer rate between the servers 106 within the farm 38 can be increased. .

Server 106 may be referred to as a file server, application server, web server, proxy server, or gateway server. In some embodiments, server 106 may have the ability to function as an application server or a master application server. In one embodiment, server 106 may include Active Directory. Client 102 may also be referred to as a client node or endpoint. In some embodiments, client 102 has the ability to function both as a client node seeking access to applications on the server and as an application server providing access to hosted applications for other clients 102a-102n.

In some embodiments, the client 102 communicates with the server 106 . In one embodiment, the client 102 communicates directly with one of the servers 106 in the farm 38 . In another embodiment, the client 102 runs a program neighbor application to communicate with the server 106 of the farm 38 . In another embodiment, the server 106 provides the functionality of a master node. In some embodiments, client 102 communicates with server 106 in farm 38 via network 104 . Via network 104 , client 102 may request execution of various applications hosted by servers 106a - 106n in farm 38 , for example, and receive output of the results of application execution for display. can do. In some embodiments, only the master node provides the functionality necessary to identify and provide address information associated with the server 106' hosting the requested application.

In one embodiment, server 106 provides the functionality of a web server. In another embodiment, server 106a receives a request from client 102 and forwards the request to second server 106b and to client 102 in response to the request from second server 106b. respond to requests by In another embodiment, the server 106 obtains a list of applications available to the client 102 and address information associated with the server 106 hosting the application identified by the list of applications. In another embodiment, the server 106 presents a response to the request to the client 102 using a web interface. In one embodiment, the client 102 communicates directly with the server 106 to access the identified application. In another embodiment, the client 102 receives application output data, such as display data, generated by the execution of the identified application on the server 106 .

Hereinafter, referring to FIG. 1B , an embodiment of a network environment in which a plurality of devices 200 are disposed is illustrated. The first device 200 may be disposed on the first network 104 and the second device 200 ′ may be disposed on the second network 104 ′. For example, a company company may deploy a first device 200 in a branch office and a second device 200 ′ in a data center. In another embodiment, the first device 200 and the second device 200' are disposed on the same network 104 or network 104'. For example, the first device 200 may be disposed in the first server farm 38 , and the second device 200 may be disposed in the second server farm 38 ′. In another example, the first device 200 may be disposed at a first point, and the second device 200 ′ may be disposed at a second point. In some embodiments, the first device 200 and the second device 200 ′ cooperate or operate in conjunction with each other to accelerate network traffic or transfer of applications and data between a client and a server.

Referring now to FIG. 1C , another embodiment of a network environment for deploying devices 200 using one or more other types of devices, such as one or more WAN optimization devices 205 and 205', is shown. For example, a first WAN optimization device 205 may be represented between the networks 104 and 104 ′ and a second WAN optimization device 205 ′ may be disposed between the device 200 and the one or more servers 106 . can For example, a company enterprise may deploy a first WAN optimization device 205 in a branch office and a second WAN optimization device 205 ′ in a data center. In some embodiments, device 205 may be located on network 104 ′. In another embodiment, the device 205 ′ may be located on the network 104 . In some embodiments, device 205' may be located on network 104' or network 104". In one embodiment, devices 205 and 205' are on the same network. Other embodiments , the devices 205 and 205' are on different networks In another example, a first WAN optimizer 205 is deployed for a first server farm 38 and a second WAN optimizer 205' It may be deployed relative to the second server farm 38'.

In one embodiment, the appliance 205 is a device for accelerating, optimizing, or otherwise improving the performance, operation or quality of service of any type and form of network traffic, such as traffic to and/or from a WAN connection. am. In some embodiments, device 205 is a performance enhancing proxy. In other embodiments, appliance 205 is any type and form of WAN optimization or acceleration device, sometimes also referred to as a WAN optimization controller. In one embodiment, the device 205 uses Ft. Any product example referred to as CloudBridge® manufactured by Citrix Systems, Inc. of Lauderdale, Florida. In another embodiment, appliance 205 includes a BIG-IP link controller manufactured by F5 Networks, Inc. of Seattle, Washington and any product embodiment referred to as WANjet. In another embodiment, the appliance 205 includes any of the WX and WXC WAN acceleration device platforms manufactured by Juniper Networks, Inc. of Sunnyvale, California. In some embodiments, appliance 205 includes any of the steelhead lines of any WAN optimization appliance manufactured by Riverbed Technology of San Francisco, California. In another embodiment, the appliance 205 comprises any of the WAN related devices manufactured by Expand Networks Inc. of Roseland, New Jersey. In one embodiment, device 205 includes any WAN related device manufactured by Packeteer Inc. of Cupertino, California, such as the PacketShaper, iShared, and SkyX product embodiments provided by Packeteer. In another embodiment, device 205 includes any WAN related device manufactured by Cisco Systems, Inc. of San Jose, California, such as Cisco Wide Area Network Application services software and network modules and Wide Area Network Engine Appliance, and / or software.

In one embodiment, the device 205 provides application and data acceleration services to a branch or remote office. In one embodiment, the device 205 includes optimization of Wide Area File Services (WAFS). In another embodiment, the device 205 accelerates the transfer of files, such as via the Common Internet File System (CIFS) protocol. In another embodiment, the device 205 provides caching in memory and/or storage to accelerate delivery of applications and data. In one embodiment, the device 205 provides compression of network traffic at any level of the network stack or at any protocol or network layer. In another embodiment, the device 205 provides transport layer protocol optimization, flow control, performance enhancement or modification and/or management to accelerate the delivery of applications and data over a WAN connection. For example, in one embodiment, the device 205 provides a Transport Control Protocol (TCP) optimization. In other embodiments, the device 205 provides optimization, flow control, performance enhancement or modification and/or management for any session or application layer protocol.

In other embodiments, the device 205 may transmit any type and form of data or information to other devices 205' in custom or standard TCP and/or IP header fields or options in network packets to inform other devices 205' of their presence, function or capability. Encoded as a field. In other embodiments, a device 205' may communicate with another device 205' using data encoded in both TCP and/or IP header fields or options. For example, the device may use TCP option(s) or IP header fields or options to pass one or more parameters to be used by the devices 205, 205' to perform a function such as WAN acceleration or to operate in conjunction with each other. have.

In some embodiments, device 200 preserves any of the information encoded in TCP and/or IP headers and/or option fields passed between devices 205 and 205'. For example, the device 200 may terminate a transport layer connection that traverses the device 200, such as a transport layer connection between the client and server traversing devices 205 and 205'. In one embodiment, the device 200 identifies and preserves any encoded information in the transport layer packet transmitted by the first device 205 over the first transport layer connection, and the transport layer packet with the encoded information. is transmitted to the second device 205' through the second transport layer connection.

Referring now to FIG. 1D , a network environment for delivering and/or operating a computing environment on a client 102 is illustrated. In some embodiments, server 106 includes an application delivery system 190 for delivering a computing environment or application and/or data files to one or more clients 102 . In brief overview, client 102 is communicating with server 106 via network 104 , 104 ′ and device 200 . For example, client 102 may reside in a remote office, eg, a branch office, of a company, and server 106 may reside in a company data center. The client 102 includes a client agent 120 and a computing environment 15 . Computing environment 15 may run or operate applications that access, process, or use data files. Computing environment 15 , applications and/or data files may be delivered via device 200 and/or server 106 .

In some embodiments, device 200 accelerates delivery of computing environment 15 or any portion thereof to client 102 . In one embodiment, device 200 accelerates delivery of computing environment 15 by application delivery system 190 . For example, embodiments described herein may be used to accelerate the delivery of streaming applications and data files processable by applications from a central corporate data center to a remote user location, such as a branch of a corporation. In another embodiment, the device 200 accelerates transport layer traffic between the client 102 and the server 106 . The device 200 is configured to perform such functions as 1) transport layer connection pooling, 2) transport layer connection multiplexing, 3) transport control protocol buffering, 4) compression, and 5) caching, from the server 106 to the client 102 ) can provide an acceleration technique to accelerate any transport layer payload to In some embodiments, device 200 provides load balancing of servers 106 in response to requests from clients 102 . In other embodiments, the device 200 acts as a proxy or access server to provide access to one or more servers 106 . In another embodiment, the device 200 provides a secure virtual private network connection from the first network 104 of the client 102 to the second network 104' of the server 106, such as an SSL VPN connection. In another embodiment, device 200 provides application firewall security, control and management of connections and communications between client 102 and server 106 .

In some embodiments, the application delivery management system 190 delivers the computing environment to the user's desktop remotely or otherwise based on a plurality of execution methods and based on any authentication and authorization policies applied through the policy engine 195 . Provides application delivery technology for These techniques allow remote users to obtain a computing environment and access server-stored applications and data files from any network-connected device 100 . In one embodiment, the application delivery system 190 may reside or run on the server 106 . In other embodiments, the application delivery system 190 may reside or run on a plurality of servers 106a - 106n. In some embodiments, application delivery system 190 may run on server farm 38 . In one embodiment, the server 106 running the application delivery system 190 may also store or serve application and data files. In other embodiments, a first set of one or more servers 106 may run the application delivery system 190 and other servers 106n may store or serve application and data files. In some embodiments, each application delivery system 190, application, and data files may reside or be located on different servers. In yet another embodiment, any portion of the application delivery system 190 may reside, run, stored, or distributed on the device 200 or a plurality of devices.

The client 102 may include a computing environment 15 for executing applications that use or process data files. Client 102 via network 104 , 104 ′ and device 200 may request application and data files from server 106 . In one embodiment, device 200 may forward a request from client 102 to server 106 . For example, client 102 may not have locally stored or accessible application and data files. In response to the request, the application delivery system 190 and/or the server 106 may deliver the application and data files to the client 102 . For example, in one embodiment, the server 106 may transmit the application as an application stream to operate in the computing environment 15 on the client 102 .

In some embodiments, application delivery system 190 is any part of Citrix Workspace Suite™ by Citrix Systems, Inc., such as XenApp® or XenDesktop®, and/or any of Microsoft® Windows Terminal Services manufactured by Microsoft Corporation. include that In one embodiment, the application delivery system 190 may deliver one or more applications to the client 102 or user via a remote-display protocol or otherwise remote-based or server-based computing. In another embodiment, the application delivery system 190 may deliver one or more applications to a client or a user through streaming of the application.

In one embodiment, the application delivery system 190 includes a policy engine 195 for controlling and managing the delivery of applications and selection of access to application execution methods. In some embodiments, policy engine 195 determines one or more applications that a user or client 102 may access. In another embodiment, the policy engine 195 determines how the application should be delivered to the user or client 102 , eg, how to run it. In some embodiments, the application delivery system 190 provides a plurality of delivery technologies for selecting an application execution method, such as server-based computing for local execution, streaming or delivering the application locally to the client 120 . .

In one embodiment, the client 102 requests execution of the application program, and the application delivery system 190 including the server 106 selects a method for executing the application program. In some embodiments, server 106 receives a certificate from client 102 . In another embodiment, the server 106 receives a request from the client 102 for a list of available applications. In one embodiment, in response to requesting or receiving a certificate, application delivery system 190 enumerates a plurality of application programs available to client 102 . The application delivery system 190 receives a request to run the enumerated application. The application delivery system 190 selects one of a predetermined number of methods for executing the enumerated application, for example in response to a policy of the policy engine 195 . The application delivery system 190 may select an application execution method that enables the client 102 to receive application output data generated by the execution of the application program on the server 106 . After searching for a plurality of application files including applications, the application delivery system 190 may select an application execution method that enables the local machine 10 to locally execute an application program. In another embodiment, the application delivery system 190 may select an execution method of the application for streaming the application to the client 102 over the network 104 .

Client 102 may be any type and/or form of software, program, or web browser, web-based client, client-server application, thin-client computing client, ActiveX control of any type and/or form. or execute, operate, or otherwise provide an application that may be executable instructions, such as a Java applet, or any other type and/or form of executable instructions that may be executed on the client 102 . In some embodiments, the application may be a server-based or remote-based application that runs on behalf of the client 102 on the server 106 . In one embodiment, the server 106 is Ft. any thin-client or remote-display protocol, such as the Independent Computing Architecture (ICA) protocol manufactured by Citrix Systems, Inc. of Lauderdale, Florida or the Remote Desktop Protocol (RDP) manufactured by Microsoft Corporation of Redmond, Washington. can be used to display the output to the client 102 . The application may use any type of protocol, for example it may be an HTTP client, an FTP client, an Oscar client, or a Telnet client. In another embodiment, the application includes any type of software related to VoIP communication, such as a soft IP phone. In another embodiment, the application includes any application related to real-time data communication, such as an application for streaming video and/or audio.

In some embodiments, server 106 or server farm 38 may run one or more applications, such as applications that provide thin-client computing or remote display presentation applications. In one embodiment, server 106 or server farm 38 is any portion of Citrix Workspace Suite™ from Citrix Systems, Inc. such as XenApp® or XenDesktop® as an application and/or Microsoft® manufactured by Microsoft Corporation. Run any of Windows Terminal Services. In one embodiment, the application is an ICA client developed by Citrix Systems, Inc. of Fort Lauderdale, Florida. In another embodiment, the application comprises a Remote Desktop (RDP) client developed by Microsoft Corporation of Redmond, Washington. Server 106 may also run an application, which may be, for example, an application server providing email services, such as Microsoft Exchange manufactured by Microsoft Corporation of Redmond, Washington, a web or Internet server, or a desktop sharing server or It may be a collaboration server. In some embodiments, any applications are GoToMeeting™ provided by Citrix Systems, Inc. of Fort Lauderdale, Florida, WebEx™ provided by Cisco Systems, Inc. of San Jose, California, or Microsoft Corporation of Redmond, Washington. may include any type of hosted service or product, such as Microsoft Office Live Meeting provided by

Still referring to FIG. 1D , an embodiment of a network environment may include a monitoring server 106A. Monitoring server 106A may include any type and form of performance monitoring service 198 . The performance monitoring service 198 may include monitoring, measurement and/or management software and/or hardware including data collection, aggregation, analysis, management and reporting. In one embodiment, the performance monitoring service 198 includes one or more monitoring agents 197 . Monitoring agent 197 includes any software, hardware, or combination thereof for performing monitoring, measurement, and data collection activities on a device such as client 102 , server 106 or instrument 200 , 205 . In some embodiments, the monitoring agent 197 includes any type and form of script, such as a Visual Basic script or Javascript. In one embodiment, the monitoring agent 197 runs transparently to any application and/or user of the device. In some embodiments, monitoring agent 197 is installed and operated inconspicuously to applications or clients 102 . In another embodiment, the monitoring agent 197 is installed and operated without any instrumentation on the application or device.

In some embodiments, the monitoring agent 197 monitors, measures, and collects data at a predetermined frequency. In other embodiments, monitoring agent 197 monitors, measures, and collects data based on detection of any type and form of event. For example, the monitoring agent 197 may collect data when it detects the receipt of a request for a web page or an HTTP response. In another example, monitoring agent 197 may collect data upon detection of any user input event, such as a mouse click. The monitoring agent 197 may report or provide any monitored, measured, or collected data to the monitoring service 198 . In one embodiment, the monitoring agent 197 sends information to the monitoring service 198 according to a schedule or predetermined frequency. In another embodiment, the monitoring agent 197 sends information to the monitoring service 198 upon detection of an event.

In some embodiments, monitoring service 198 and/or monitoring agent 197 may be any such as client 102 , server 106 , server farm 38 , device 200 , device 205 , or a network connection. Monitoring and performance measurement of network resources or network infrastructure elements of In one embodiment, monitoring service 198 and/or monitoring agent 197 performs monitoring and performance measurement of any transport layer connection, such as a TCP or UDP connection. In another embodiment, monitoring service 198 and/or monitoring agent 197 monitors and measures network latency. In another embodiment, monitoring service 198 and/or monitoring agent 197 monitors and measures bandwidth usage.

In other embodiments, monitoring service 198 and/or monitoring agent 197 monitors and measures end-user response times. In some embodiments, monitoring service 198 performs monitoring and performance measurement of applications. In another embodiment, monitoring service 198 and/or monitoring agent 197 performs monitoring and performance measurement of any session or connection to an application. In one embodiment, monitoring service 198 and/or monitoring agent 197 monitors and measures the performance of the browser. In another embodiment, monitoring service 198 and/or monitoring agent 197 monitors and measures the performance of HTTP-based transactions. In some embodiments, monitoring service 198 and/or monitoring agent 197 monitors and measures the performance of a Voice over IP (VoIP) application or session. In another embodiment, monitoring service 198 and/or monitoring agent 197 monitors and measures the performance of remote display protocol applications, such as ICA clients or RDP clients. In another embodiment, monitoring service 198 and/or monitoring agent 197 monitors and measures the performance of any type and form of streaming media. In another embodiment, monitoring service 198 and/or monitoring agent 197 monitors and measures the performance of hosted applications or Software-As-A-Service (SaaS) delivery models.

In some embodiments, monitoring service 198 and/or monitoring agent 197 performs monitoring and performance measurement of one or more transactions, requests, or responses related to the application. In another embodiment, monitoring service 198 and/or monitoring agent 197 monitors and measures any portion of the application layer stack, such as any .NET or J2EE call. In one embodiment, monitoring service 198 and/or monitoring agent 197 monitors and measures database or SQL transactions. In another embodiment, monitoring service 198 and/or monitoring agent 197 monitors and measures any method, function, or application programming interface (API) call.

In one embodiment, the monitoring service 198 and/or the monitoring agent 197 may provide applications and /or perform monitoring and performance measurement of the transfer of data. In some embodiments, monitoring service 198 and/or monitoring agent 197 monitors and measures the delivery performance of virtualized applications. In another embodiment, monitoring service 198 and/or monitoring agent 197 monitors and measures the delivery performance of streaming applications. In another embodiment, the monitoring service 198 and/or the monitoring agent 197 monitors and measures the performance of the delivery of the desktop application to the client 102 and/or the execution of the desktop application on the client 102 . In another embodiment, monitoring service 198 and/or monitoring agent 197 monitors and measures the performance of client/server applications.

In one embodiment, monitoring service 198 and/or monitoring agent 197 is designed and configured to provide application performance management for application delivery system 190 . For example, the monitoring service 198 and/or the monitoring agent 197 may monitor, measure, and manage the delivery performance of the application through Citrix Presentation Server. In this example, monitoring service 198 and/or monitoring agent 197 monitors individual ICA sessions. Monitoring service 198 and/or monitoring agent 197 may measure application and networking performance, as well as overall and per-session system resource usage. Monitoring service 198 and/or monitoring agent 197 may identify active servers for a given user and/or user session. In some embodiments, monitoring service 198 and/or monitoring agent 197 monitors back-end connections between application delivery system 190 and application and/or database servers. Monitoring service 198 and/or monitoring agent 197 may measure network latency, latency, and volume per user-session or ICA session.

In some embodiments, monitoring service 198 and/or monitoring agent 197 measures and monitors memory usage for application delivery system 190, such as total memory usage, per user session and/or per process. In other embodiments, monitoring service 198 and/or monitoring agent 197 measures and monitors CPU usage of application delivery system 190, such as total CPU usage, per user session and/or per process. In another embodiment, monitoring service 198 and/or monitoring agent 197 measures and monitors the time required to log in to an application, server 106, or application delivery system 190, such as Citrix Presentation Server. do. In one embodiment, monitoring service 198 and/or monitoring agent 197 measures and monitors how long a user has logged into an application, server 106 or application delivery system 190 . In some embodiments, monitoring service 198 and/or monitoring agent 197 measures and monitors active and inactive session counts for application, server 106, or application delivery system 190 sessions. In another embodiment, monitoring service 198 and/or monitoring agent 197 measures and monitors user session latency.

In another embodiment, monitoring service 198 and/or monitoring agent 197 measures and monitors any type and form of server metrics. In one embodiment, monitoring service 198 and/or monitoring agent 197 measures and monitors metrics related to system memory, CPU usage, and disk storage. In another embodiment, monitoring service 198 and/or monitoring agent 197 measures and monitors metrics related to page faults, such as page faults per second. In another embodiment, monitoring service 198 and/or monitoring agent 197 measures and monitors round trip time metrics. In another embodiment, monitoring service 198 and/or monitoring agent 197 measures and monitors metrics related to application crashes, errors, and/or hangs.

In some embodiments, monitoring service 198 and monitoring agent 197 may include Ft. Any product example referred to as EdgeSight manufactured by Citrix Systems, Inc. of Lauderdale, Florida is included. In other embodiments, performance monitoring service 198 and/or monitoring agent 197 comprises any portion of a product embodiment referred to as the True View suite manufactured by Symphoniq Corporation of Palo Alto, California. In one embodiment, performance monitoring service 198 and/or monitoring agent 197 comprises any portion of a product embodiment referred to as the TeaLeaf CX family manufactured by TeaLeaf Technology Inc. of San Francisco, California. In another embodiment, the performance monitoring service 198 and/or the monitoring agent 197 may manage any part of a business service management product, such as the BMC Performance Manager and Patrol product manufactured by BMC Software, Inc. of Houston, Texas. include

Client 102 , server 106 , and appliance 200 can communicate over any type and form of network and can perform the operations described herein of any type and form, such as a computer, network device, or appliance. may be deployed as and/or executed on a computing device of 1E and 1F show block diagrams of a computing device 100 useful for practicing embodiments of a client 102 , a server 106 , or an appliance 200 . 1E and 1F , each computing device 100 includes a central processing unit 101 and a main memory unit 122 . As shown in FIG. 1E , computing device 100 may include a visual display device 124 , a keyboard 126 , and/or a pointing device 127 such as a mouse. Each computing device 100 also includes one or more input/output devices 130a - 130b (referred to generally using reference numeral 130 ) and additional, such as cache memory 140 in communication with central processing unit 101 . It may contain optional elements.

The central processing unit 101 is any logic circuit that responds to and processes commands fetched from the main memory unit 122 . In many embodiments, central processing unit 101 is manufactured by Intel Corporation of Mountain View, California; manufactured by Motorola Corporation of Schaumburg, Illinois; manufactured by Transmeta Corporation of Santa Clara, California; RS/6000 processor manufactured by International Business Machines of White Plains, New York; or by a microprocessor unit such as manufactured by Advanced Micro Devices of Sunnyvale, California. Computing device 100 may be based on any of these processors or any other processor capable of operating as described herein.

Main memory unit 122 is SRAM (Static Random Access Memory), Burst SRAM or SynchBurst SRAM (BSRAM), Dynamic Random Access Memory (DRAM), FPM DRAM (Fast Page Mode DRAM), EDRAM (Enhanced DRAM), EDO RAM ( Extended Data Output RAM), EDO DRAM (Extended Data Output DRAM), BEDO DRAM (Burst Extended Data Output DRAM), EDRAM (Enhanced DRAM), SDRAM (synchronous DRAM), JEDEC SRAM, PC100 SDRAM, DDR SDRAM (Double Data Rate SDRAM) ), ESDRAM (Enhanced SDRAM), SLDRAM (SyncLink DRAM), DRDRAM (Direct Rambus DRAM), or FRAM (Ferroelectric RAM) to store data and allow any storage location to be directly accessed by the microprocessor 101 . may be one or more memory chips. Main memory 122 may be based on any of the memory chips described above or any other available memory chip capable of operating as described herein. In the embodiment shown in FIG. 1E , the processor 101 communicates with the main memory 122 via the system bus 150 (described in more detail below). 1F illustrates an embodiment of computing device 100 in which processor 101 communicates directly with main memory 122 via memory port 103 . For example, the main memory 122 in FIG. 1F may be a DRDRAM.

1F shows an embodiment in which the main processor 101 communicates directly with the cache memory 140 via a secondary bus, sometimes also referred to as a backside bus. In another embodiment, main processor 101 communicates with cache memory 140 using system bus 150 . Cache memory 140 typically has a faster response time than main memory unit 122 and is typically provided by SRAM, BSRAM, or EDRAM. In the embodiment shown in FIG. 1F , processor 101 communicates with various I/O devices 130 via local system bus 150 . Various buses, including VESA VL bus, ISA bus, EISA bus, MicroChannel Architecture (MCA) bus, PCI bus, PCI-X bus, PCI-Express bus, or NuBus, connect the central processing unit 101 to any I/O device. 130 may be used. In embodiments where the I/O device 130 is a video display 124 , the processor 101 may use an Advanced Graphics Port (AGP) to communicate with the display 124 . 1F illustrates an embodiment of a computing device 100 in which main processor 101 communicates directly with I/O device 130b via HyperTransport, Rapid I/O, or InfiniBand. shows Figure If also shows an embodiment in which direct communication with a local bus is mixed: processor 101 communicates directly with I/O device 130a while using a local interconnect bus to communicate with I/O device 130b. .

Computing device 100 may include a floppy disk drive, a CD-ROM drive, a CD-R/RW drive, a DVD-ROM drive, a tape drive of various formats, for accommodating a floppy disk such as a 3.5 inch, 5.25 inch disk or a ZIP disk; Any suitable installation device 116 may be supported, such as a USB device, a hard-drive, or any other device suitable for installing software and programs, such as any client agent 120 or portions thereof. Computing device 100 is a storage device such as one or more hard disk drives or redundant arrays of independent disks for storing operating systems and other related software and for storing application software programs such as any programs associated with client agent 120 ( 128) may be further included. Optionally, any installation device 116 may also be used as the storage device 128 . The operating system and software can also be run from bootable media, for example a bootable CD such as KNOPPIX®, a bootable CD for GNU/Linux available as a GNU/Linux distribution at knoppix.net.

In addition, the computing device 100 may include a standard telephone line, LAN or WAN link (eg, 802.11, T1, T3, 56kb, X.25), broadband access (eg, ISDN, Frame Relay), ATM ), a wireless connection, or a network interface for interfacing to the Internet (Local Area Network), Wide Area Network (WAN) or the Internet via various connections including, but not limited to, some combination of any or all of the foregoing. 118) may be included. Network interface 118 may be capable of performing the operations described herein and communicating with an embedded network adapter, network interface card, PCMCIA network card, Card Bus network adapter, wireless network adapter, USB network adapter, modem, or computing device 100 . It may include any other device suitable for interfacing to any type of network.

A wide variety of I/O devices 130a - 130n may be present in computing device 100 . Input devices include keyboards, mice, trackpads, trackballs, microphones, and drawing tablets. Output devices include video displays, speakers, inkjet printers, laser printers, and dye-sublimation printers. I/O device 130 may be controlled by I/O controller 123 as shown in FIG. 1E . I/O controller 123 may control one or more I/O devices, such as keyboard 126 and pointing device 127 , for example, a mouse or optical pen. In addition, the I/O device may also provide storage 128 and/or installation media 116 for the computing device 100 . In another embodiment, computing device 100 may provide a USB connection to accommodate a portable USB storage device, such as a USB Flash Drive line of devices manufactured by Twintech Industry, Inc. of Los Alamitos, California.

In some embodiments, computing device 100 may include or be connected to a plurality of display devices 124a - 124n , each of which may be the same or a different type and/or form. As such, any of the I/O devices 130a - 130n and/or the I/O controller 123 supports, enables, and uses for the connection and use of the plurality of display devices 124a - 124n by the computing device 100 . It may include any type and/or form of suitable hardware, software, or combination of hardware and software for enabling or providing. For example, computing device 100 may include any type and/or form of video adapters, video cards, drivers, and/or libraries for interfacing, communicating, connecting, or otherwise using display devices 124a - 124n. can In one embodiment, the video adapter may include a plurality of connectors for interfacing to a plurality of display devices 124a-124n. In another embodiment, computing device 100 may include a plurality of video adapters, each video adapter connected to one or more display devices 124a - 124n . In some embodiments, any portion of the operating system of computing device 100 may be configured to use multiple displays 124a - 124n . In other embodiments, one or more display devices 124a - 124n may be provided by one or more other computing devices, such as, for example, computing devices 100a and 100b connected to computing device 100 via a network. . These embodiments may include any type of software designed and configured to use another computer's display device as the second display device 124a for the computing device 100 . Those skilled in the art will recognize and understand the various ways and embodiments in which computing device 100 may be configured to have a plurality of display devices 124a - 124n .

In a further embodiment, I/O device 130 is a USB bus, Apple Desktop bus, RS-232 serial connection, SCSI bus, FireWire bus, FireWire 800 bus, Ethernet bus, AppleTalk ( AppleTalk) bus, Gigabit Ethernet bus, Asynchronous Transfer Mode bus, HIPPI bus, Super HIPPI bus, SerialPlus bus, SCI/LAMP bus, FiberChannel bus or Serial Attached small computer system interface bus; It may be a bridge 170 between the external communication bus and the system bus 150 .

A computing device 100 of the kind shown in FIGS. 1E and 1F typically operates under the control of an operating system that controls scheduling of tasks and access to system resources. Computing device 100 can be any version of the Microsoft® Windows operating system, any other release of Unix and Linux operating systems, any version of Mac OS® for Macintosh computers, any embedded operating system, any real-time operating system, any may run any operating system, such as an open source operating system, any proprietary operating system, any operating system for mobile computing devices, or any other operating system that runs on a computing device and is capable of performing the operations described herein. . Common operating systems include WINDOWS 3.x, WINDOWS 95, WINDOWS 98, WINDOWS 2000, WINDOWS NT 3.51, WINDOWS NT 4.0, WINDOWS CE and WINDOWS XP, all manufactured by Microsoft Corporation of Redmond, Washington, among others; MacOS manufactured by Apple Computer of Cupertino, California; OS/2 manufactured by International Business Machines of Armonk, New York; and Linux, a freely available operating system distributed by Caldera Corp. of Salt Lake City, Utah, or a UNIX operating system of any type and/or form.

In other embodiments, computing device 100 may have different processors, operating systems, and input devices that match the device. For example, in one embodiment, computer 100 is a Treo 180, 270, 1060, 600 or 650 smart phone manufactured by Palm, Inc. In this embodiment, the Treo smartphone is operated under the control of the PalmOS operating system and includes a stylus input device as well as a 5-way navigator device. In addition, computing device 100 may be any workstation, desktop computer, laptop or notebook computer, server, handheld computer, mobile phone, capable of communication and having sufficient processing capability and memory capacity to perform the operations described herein. , any other computer, or other form of computing or telecommunications device.

As shown in FIG. 1G , computing device 100 may include a plurality of processors and may provide functionality for concurrent execution of instructions or concurrent execution of one instruction for more than one piece of data. In some embodiments, computing device 100 may include a parallel processor with one or more cores. In one of these embodiments, computing device 100 is a shared memory parallel device having multiple processors and/or multiple processor cores, accessing all available memory as a single global address space. In another of these embodiments, computing device 100 is a distributed memory parallel device having multiple processors each accessing only local memory. In another of these embodiments, computing device 100 has both some memory shared and some memory that can only be accessed by a particular processor or subset of processors. In another of these embodiments, computing device 100, such as a multi-core microprocessor, combines two or more independent processors into a single package, often a single integrated circuit (IC). In yet another of these embodiments, the computing device 100 has a CELL BROADBAND ENGINE architecture and is a power processor that is linked together by a power processor element and a plurality of synergistic processing elements, an internal high-speed bus, which may be referred to as an element interconnect bus. a chip comprising an element and a plurality of synergistic processing elements.

In some embodiments, the processor provides functionality for concurrently executing a single instruction on a plurality of data SIMDs. In another embodiment, the processor provides a function for concurrently executing a plurality of instructions on a plurality of data MIMD. In another embodiment, the processor may use any combination of SIMD and MIMD cores in a single device.

In some embodiments, computing device 100 may include a graphics processing unit. In one of these embodiments shown in FIG. 1H , the computing device 100 includes at least one central processing unit 101 and at least one graphics processing unit 101 ′. In another of these embodiments, computing device 100 includes at least one parallel processing unit and at least one graphics processing unit 101 ′. In another of these embodiments, computing device 100 includes a plurality of processing units of any type, one of the plurality of processing units including graphics processing unit 101 ′.

In some embodiments, first computing device 100a executes an application on behalf of a user of client computing device 100b . In another embodiment, computing device 100a executes a virtual machine, which provides an execution session in which an application executes on behalf of a user or client computing device 100b. In one of these embodiments, the launch session is a hosted desktop session. In another of these embodiments, the computing device 100 executes a terminal service session. The terminal service session may provide a hosted desktop environment. In yet another of these embodiments, the execution session provides access to a computing environment, the computing environment comprising one or more of an application, a plurality of applications, a desktop application, and a desktop session in which the one or more applications may be executed. can

B. Instrument Architecture

2A shows an exemplary embodiment of a device 200 . The device 200 of FIG. 2A is provided by way of example only and is not intended to be limiting. As shown in FIG. 2 , the device 200 includes a hardware layer 206 and a software layer divided into a user space 202 and a kernel space 204 .

Hardware layer 206 provides the hardware elements on which programs and services in kernel space 204 and user space 202 run. Hardware layer 206 also provides structures and elements that allow programs and services in kernel space 204 and user space 202 to communicate data both internally and externally to device 200 . As shown in Figure 2, the hardware layer 206 includes a processing unit 262 for executing software programs and services, a memory 264 for storing software and data, and a network for transmitting and receiving data via a network. port 266 and a cryptographic processor 260 for performing functions related to Secure Socket Layer processing of data transmitted and received over the network. In some embodiments, central processing unit 262 may perform the functions of cryptographic processor 260 in a single processor. Additionally, the hardware layer 206 may include a plurality of processors for each of the processing unit 262 and the cryptographic processor 260 . The processor 262 may include any of the processors 101 described above with respect to FIGS. 1E and 1F . For example, in one embodiment, the device 200 includes a first processor 262 and a second processor 262 ′. In another embodiment, the processor 262 or 262' comprises a multi-core processor.

While the hardware layer 206 of the device 200 is generally represented by a cryptographic processor 260 , the processor 260 provides functionality related to any cryptographic protocol, such as a Secure Socket Layer (SSL) or Transport Layer Security (TLS) protocol. It may be a processor for performing In some embodiments, the processor 260 may be a general purpose processor (GPP), and in further embodiments may have executable instructions to perform processing of any security related protocol.

Although the hardware layer 206 of the device 200 is shown with certain elements in FIG. 2 , the hardware portion or component of the device 200 is a computing device shown and discussed herein with respect to FIGS. 1E and 1F . It may include elements, hardware or software of any type and form of a computing device such as 100 . In some embodiments, device 200 may include a server, gateway, router, switch, bridge, or other type of computing or network device, and may have any hardware and/or software elements associated therewith.

The operating system of the device 200 allocates, manages, or otherwise separates the available system memory into kernel space 204 and user space 202 . In the example software architecture 200, the operating system may be any type and/or type of UNIX operating system, although the described embodiments are not limited. As such, device 200 may include any version of the Microsoft® Windows operating system, any other release of Unix and Linux operating systems, any version of Mac OS® for Macintosh computers, any embedded operating system, any network operating system; Any real-time operating system, any open source operating system, any proprietary operating system, any operating system or device 200 for a mobile computing device or network device and capable of performing the operations described herein and running on the appliance 200 . It can run any operating system like any other operating system in

Kernel space 204 is reserved for executing kernel 230, including any device drivers, kernel extensions, or other kernel-related software. As is known to those skilled in the art, the kernel 230 is the core of the operating system and provides access, control, and management of resources and hardware-related elements of the application 104 . According to an embodiment of the device 200 , the kernel space 204 includes a number of network services or processes that operate in conjunction with a cache manager 232 , also sometimes referred to as a unified cache, the benefits of which are described in more detail herein. explained. Further, the embodiment of the kernel 230 will depend on the embodiment of the operating system installed, configured, or otherwise used by the device 200 .

In one embodiment, device 200 includes one network stack 267 , such as a TCP/IP based stack, for communicating with client 102 and/or server 106 . In one embodiment, the network stack 267 is used to communicate with a first network such as network 108 and a second network 110 . In some embodiments, device 200 terminates a first transport layer connection, such as a TCP connection of client 102 , and establishes a second transport layer connection to server 106 for use by client 102 . and, for example, the second transport layer connection is terminated at the device 200 and the server 106 . The first and second transport layer connections may be established via a single network stack 267 . In another embodiment, the device 200 may include multiple network stacks, such as, for example, 267 and 267', a first transport layer connection in one network stack 267, and a second transport layer A connection may be established or terminated in the second network stack 267'. For example, one network stack may be for receiving and transmitting network packets on a first network, and another network stack may be for receiving and transmitting network packets on a second network. In one embodiment, the network stack 267 includes a buffer 243 for queuing one or more network packets for transmission by the device 200 .

As shown in FIG. 2 , kernel space 204 includes cache manager 232 , high-speed layer 2-7 unified packet engine 240 , encryption engine 234 , policy engine 236 , and multi-protocol compression logic 238 . ) is included. Running these components or processes 232 , 240 , 234 , 236 and 238 in kernel space 204 or kernel mode instead of user space 202 improves the performance of each of these components, alone and in combination. Kernel operation means that these components or processes 232 , 240 , 234 , 236 and 238 run in the core address space of the operating system of the device 200 . For example, running the encryption engine 234 in kernel mode improves encryption performance by moving encryption and decryption operations to the kernel, thereby improving the memory space in kernel mode or the memory space in kernel thread and user mode or Reduces the number of transitions between threads. For example, data obtained in kernel mode may not need to be passed or copied from a kernel level data structure to a user level data structure to a process or thread running in user mode. In another aspect, the number of context switches between kernel mode and user mode is also reduced. In addition, synchronization and communication between arbitrary components or processes 232 , 240 , 235 , 236 and 238 in kernel space 204 may be performed more efficiently.

In some embodiments, any portion of components 232 , 240 , 234 , 236 and 238 may execute or operate in kernel space 204 , while these components 232 , 240 , 234 , 236 and 238 may be operable. ) may execute or operate in user space 202 . In one embodiment, device 200 is a kernel that provides access to one or more network packets, eg, any portion of a network packet including a request from a client 102 or a response from a server 106 . -Use a level data structure. In some embodiments, the kernel-level data structure may be obtained by the packet engine 240 via a transport layer driver interface or filter to the network stack 267 . Kernel-level data structures may include any interfaces and/or data accessible through kernel space 204 associated with network stack 267 , network traffic or packets received or transmitted by network stack 267 . . In other embodiments, kernel-level data structures may be used by any component or process 232 , 240 , 234 , 236 and 238 to perform the desired operation of the component or process. In one embodiment, components 232, 240, 234, 236, and 238 execute in kernel mode when using kernel-level data structures, while in another embodiment components 232, 240, 234, 236 and 238) runs in user mode when using kernel-level data structures. In some embodiments, the kernel-level data structure may be copied or passed to a second kernel-level data structure, or any desired user-level data structure.

The cache manager 232 is a software, hardware or set of software and hardware components to provide cache access, control and management of content of any type and form, such as objects provided by the originating server 106 or dynamically created objects. It may include any combination. The data, objects, or content processed and stored by the cache manager 232 may include data in any format, such as a markup language, or be delivered via any protocol. In some embodiments, cache manager 232 duplicates original data stored elsewhere or previously computed, generated, or transmitted data, wherein the original data is fetched, computed, or otherwise obtained for reading a cache memory element. Longer access times may be required. Once data is stored in the cache memory element, future use can be made by accessing the cached copy instead of refetching or recalculating the original data, thereby reducing access time. In some embodiments, cache memory elements may contain data objects in memory 264 of device 200 . In another embodiment, the cache memory element may include a memory that has a faster access time than the memory 264 . In other embodiments, the cache memory element may include any type and form of storage element of the device 200, such as part of a hard disk. In some embodiments, processing unit 262 may provide cache memory for use by cache manager 232 . In yet another embodiment, cache manager 232 may use any portion and combination of memory, storage, or processing unit to cache data, objects, and other content.

Additionally, cache manager 232 includes any logic, function, rule, or action for performing any embodiment of the technology of device 200 described herein. For example, cache manager 232 includes logic or functionality to invalidate objects based on expiration of an invalidation period or receipt of an invalidation command from client 102 or server 106 . The cache manager 232 may operate as a program, service, process, or task executing in kernel space 204 in some embodiments and in user space 202 in other embodiments. In one embodiment, a first portion of cache manager 232 runs in user space 202 , while a second portion runs in kernel space 204 . In some embodiments, cache manager 232 is any type of general purpose processor (GPP), or any other type, such as a Field Programmable Gate Array (FPGA), Programmable Logic Device (PLC), or Application Specific Integrated Circuit (ASIC). may include an integrated circuit of

Policy engine 236 may include, for example, an intelligent statistics engine or other programmable application(s). In one embodiment, the policy engine 236 provides a configuration mechanism that allows a user to identify, specify, define, or configure a caching policy. In some embodiments, policy engine 236 also has access to memory to support data structures such as lookup tables or hash tables to facilitate user-selected caching policy decisions. In other embodiments, the policy engine 236 may be configured by the device 200 in addition to access, control and management of security, network traffic, network access, compression, or any other function or action performed by the device 200 . It may include any logic, rule, function, or action for determining and providing access, control, and management of objects, data, or content being cached. Additional examples of specific caching policies are further described herein.

Cryptographic engine 234 includes any logic, business rules, functions or actions for handling the processing of or any function related to any security related protocol, such as SSL or TLS. For example, the encryption engine 234 encrypts and decrypts network packets communicated via the device 200 or any portion thereof. Cryptographic engine 234 may also establish or establish SSL or TLS connections on behalf of clients 102a - 102n , servers 106a - 106n , or device 200 . As such, the encryption engine 234 provides offloading and acceleration of SSL processing. In one embodiment, encryption engine 234 provides a virtual private network between clients 102a-102n and servers 106a-106n using a tunneling protocol. In some embodiments, the cryptographic engine 234 is in communication with the cryptographic processor 260 . In another embodiment, cryptographic engine 234 includes executable instructions executing on cryptographic processor 260 .

The multi-protocol compression engine 238 includes any logic, business rules, functions, or actions for compressing one or more protocols of a network packet, such as any protocol used by the network stack 267 of the device 200 . do. In one embodiment, multi-protocol compression engine 238 is bidirectional between clients 102a-102n and servers 106a-106n, Messaging Application Programming Interface (MAPI) (email), File Transfer Protocol (FTP), HTTP (HyperText Transfer Protocol), Common Internet File System (CIFS) protocol (file transfer), Independent Computing Architecture (ICA) protocol, Remote Desktop Protocol (RDP), Wireless Application Protocol (WAP), Mobile IP protocol, and Voice over IP (VoIP) protocol. ) to compress any TCP/IP-based protocol. In other embodiments, the multi-protocol compression engine 238 provides compression of Hypertext Markup Language (HTML) based protocols, and in some embodiments, compression of any markup language, such as Extensible Markup Language (XML). do. In one embodiment, the multi-protocol compression engine 238 provides compression of any high performance protocol, such as any protocol designed for device 200 to device 200 communication. In another embodiment, the multi-protocol compression engine 238 is a transactional TCP (T/TCP), TCP with selective acknowledgment (TCP-SACK), TCP with large window (TCP-LW), TCP- It compresses any payload or any communication using a congestion prediction protocol such as the Vegas protocol and a modified transmission control protocol such as the TCP spoofing protocol.

As such, the multi-protocol compression engine 238 is a desktop client, for example, a non-web thin, such as Microsoft Outlook and any client launched by popular enterprise applications such as Oracle, SAP and Siebel. Accelerates performance for users accessing applications through clients, even mobile clients such as Pocket PCs. In some embodiments, the multi-protocol compression engine 238 by integrating with the packet processing engine 240 running in kernel mode and accessing the network stack 267 is carried by a TCP/IP protocol, such as any application layer protocol. It can compress any protocol that can be used.

High-speed Layer 2-7 Unified Packet Engine 240 , also commonly referred to as packet processing engine or packet engine, manages kernel-level processing of packets received and transmitted by device 200 over network port 266 . play a role The high-speed layer 2-7 aggregation packet engine 240 may include a buffer 243 for queuing one or more network packets during processing, such as reception of network packets or transmission of network packets. In addition, the high-speed layer 2-7 aggregation packet engine 240 communicates with one or more network stacks 267 to transmit and receive network packets through a network port 266 . The high-speed layer 2-7 unified packet engine 240 operates in conjunction with the encryption engine 234 , the cache manager 232 , the policy engine 236 and the multi-protocol compression logic 238 . In particular, the encryption engine 234 is configured to perform SSL processing of packets, and the policy engine 236 is configured to perform functions related to traffic management, such as request-level content switching and request-level cache redirection, and multi- Protocol compression logic 238 is configured to perform functions related to compression and decompression of data.

The high-speed layer 2-7 aggregation packet engine 240 includes a packet processing timer 242 . In one embodiment, packet processing timer 242 provides one or more time intervals for triggering ingress, ie, reception or egress, ie, processing of a transmitted network packet. In some embodiments, the fast layer 2-7 aggregation packet engine 240 processes the network packets in response to the timer 242 . Packet processing timer 242 provides signals of any type and form to packet engine 240 to notify, trigger, or communicate time-related events, intervals, or occurrences. In many embodiments, the packet processing timer 242 operates on the order of milliseconds, such as, for example, 100 ms, 50 ms, or 25 ms. For example, in some embodiments, the packet processing timer 242 provides a time interval or otherwise causes network packets to be processed by the high-speed layer 2-7 unified packet engine 240 at 10 ms time intervals; In an embodiment it is a 5 ms time interval, in another embodiment as short as a 3, 2 or 1 ms time interval. The high-speed layer 2-7 unified packet engine 240 may interface, integrate, or communicate with the encryption engine 234 , the cache manager 232 , the policy engine 236 and the multi-protocol compression engine 238 during operation. As such, any logic, function, or operation of the encryption engine 234 , the cache manager 232 , the policy engine 236 and the multi-protocol compression logic 238 is dependent on the packet processing timer 242 and/or the packet engine ( 240) may be performed. Accordingly, any logic, function, or operation of the encryption engine 234 , the cache manager 232 , the policy engine 236 , and the multi-protocol compression logic 238 is dependent on the time interval provided via the packet processing timer 242 . The degree of subdivision, for example, may be performed at a time interval of 10 ms or less. For example, in one embodiment, cache manager 232 may perform invalidation of any cached objects in response to fast layer 2-7 unified packet engine 240 and/or packet processing timer 242 . . In another embodiment, the expiration or invalidation time of a cached object may be set to the same granularity as the time interval of the packet processing timer 242, such as every 10 ms.

In contrast to kernel space 204, user space 202 is a memory area or portion of an operating system used by user mode applications or programs that otherwise run in user mode. User mode applications may not have direct access to kernel space 204 and use service calls to access kernel services. As shown in FIG. 2 , the user space 202 of the device 200 includes a graphical user interface (GUI) 210 , a command line interface (CLI) 212 , a shell service 214 , and a health monitoring program. 216 and a daemon service 218 . GUI 210 and CLI 212 provide a means for a system administrator or other user to interact with and control the operation of device 200 , such as through the operating system of device 200 . GUI 210 or CLI 212 may include code executing in user space 202 or kernel space 204 . GUI 210 may be any type and form of graphical user interface, and may be presented as text, graphics, or otherwise by any type of program or application, such as a browser. CLI 212 may be a command line or text-based interface of any type and form, such as a command line provided by an operating system. For example, CLI 212 may include a shell, which is a tool that allows a user to interact with an operating system. In some embodiments, CLI 212 may be provided through a bash, csh, tcsh, or ksh type shell. Shell service 214 includes programs, services, tasks, processes, or executable commands to support interaction with device 200 or an operating system by a user via GUI 210 and/or CLI 212 . do.

Health monitoring program 216 is used to monitor, verify, report and ensure that network systems are functioning properly and that users are receiving requested content over the network. The health monitoring program 216 includes one or more programs, services, tasks, processes, or executable instructions for providing logic, rules, functions, or actions for monitoring any activity of the device 200 . In some embodiments, health monitoring program 216 intercepts and inspects any network traffic passing through device 200 . In another embodiment, health monitoring program 216 may include encryption engine 234 , cache manager 232 , policy engine 236 , multi-protocol compression logic 238 , packets by any suitable means and/or mechanism. interfaces with one or more of engine 240 , daemon service 218 , and shell service 214 . As such, the health monitoring program 216 may call any application programming interface (API) to determine the state, situation, or health of any part of the device 200 . For example, health monitoring program 216 may periodically ping or send status queries to determine if a program, process, service, or task is active and currently running. In another example, the health monitoring program 216 checks any status, error, or historical log provided by any program, process, service, or task to determine any status, condition, or error of any part of the device 200 . can be decided

Daemon service 218 is a program that runs continuously or in the background and handles periodic service requests received by device 200 . In some embodiments, the daemon service may forward requests to other programs or processes, such as other suitable daemon services 218 . As will be known to those skilled in the art, daemon services 218 may run unattended to perform continuous or periodic system-wide functions, such as network control, or to perform any desired task. In some embodiments, one or more daemon services 218 run in user space 202 , while in other embodiments one or more daemon services 218 run in kernel space 204 .

Referring now to FIG. 2B , another embodiment of the device 200 is illustrated. In brief overview, device 200 includes SSL VPN connection 280 , switching/load balancing 284 , domain name service resolution 286 , acceleration 288 and one or more clients 102 and one or more servers 106 . Provides a service, function, or operation of one or more of the application firewalls 290 for communication between them. Each server 106 may provide one or more network related services 270a - 270n (referred to as services 270 ). For example, server 106 may provide http service 270 . Device 200 includes one or more virtual servers or virtual Internet Protocol servers referred to as vServers, VIP servers, or merely VIPs 275a - 275n (also referred to herein as vServers 275 ). vServer 275 receives, intercepts, or otherwise processes communications between client 102 and server 106 depending on the configuration and operation of device 200 .

vServer 275 may include software, hardware, or any combination of software and hardware. vServer 275 may include any type and form of programs, services, tasks, processes, or executable instructions that operate in user mode, kernel mode, or any combination thereof on device 200 . vServer 275 may be any logic, function, rule or described herein, such as SSL VPN 280 , Switching/Load Balancing 284 , Domain Name Service Resolution 286 , Acceleration 288 and Application Firewall 290 . and performing any embodiment of the technology. In some embodiments, vServer 275 establishes a connection to service 270 of server 106 . Services 270 may include any program, application, process, task, or set of executable instructions that can connect and communicate with device 200 , client 102 , or vServer 275 . For example, service 270 may include a web server, http server, ftp, email or database server. In some embodiments, service 270 is a daemon process or network driver for listening, receiving, and/or sending communications to applications such as email, databases, or enterprise applications. In some embodiments, services 270 may communicate over a specific IP address, or IP address and port.

In some embodiments, vServer 275 applies one or more policies of policy engine 236 to network communications between client 102 and server 106 . In one embodiment, the policy is associated with vServer 275 . In another embodiment, the policy is based on a user or group of users. In another embodiment, the policy is global and applies to any user or group of users communicating through one or more vServers 275a - 275n and device 200 . In some embodiments, the policy of the policy engine 236 is an Internet protocol address, port, protocol type, header or field in a packet, or user, user group, vServer 275, transport layer contact, and/or client 102 . or a state in which the policy is applied based on any content of the communication, such as the content of the communication, such as an identification or attribute of the server 106 .

In another embodiment, the device 200 may use a policy to determine authentication and/or authorization of a remote user or remote client 102 to access the computing environment 15 , applications and/or data files from the server 106 . It communicates with or interfaces with the engine 236 . In another embodiment, the device 200 determines the authentication and/or authorization of the remote user or remote client 102 for the application delivery system 190 to deliver one or more computing environments 15 , applications and/or data files. communicates or interfaces with the policy engine 236 to In another embodiment, the device 200 establishes a VPN or SSL VPN connection based on authentication and/or authorization of the policy engine 236 of the remote user or remote client 102 . In one embodiment, the device 200 controls the flow of network traffic and communication sessions based on the policies of the policy engine 236 . For example, device 200 may control access to computing environment 15 , applications, or data files based on policy engine 236 .

In some embodiments, vServer 275 establishes a transport layer connection, such as a TCP or UDP connection, with client 102 via client agent 120 . In one embodiment, vServer 275 listens for and receives communications from clients 102 . In another embodiment, vServer 275 establishes a transport layer connection, such as a TCP or UDP connection, with client server 106 . In one embodiment, vServer 275 establishes a transport layer connection to the Internet Protocol address and port of server 270 running on server 106 . In another embodiment, vServer 275 associates a first transport layer connection to client 102 with a second transport layer connection to server 106 . In some embodiments, vServer 275 establishes a pool of transport layer connections to server 106 and multiplexes client requests over the pooled transport layer connections.

In some embodiments, device 200 provides an SSL VPN connection 280 between client 102 and server 106 . For example, a client 102 on a first network 104 requests establishment of a connection with a server 106 on a second network 104'. In some embodiments, the second network 104 ′ is not routable from the first network 104 . In another embodiment, the client 102 is on a public network 104 and the server 106 is on a private network 104', such as a corporate network. In one embodiment, the client agent 120 intercepts communications of the client 102 over the first network 104 , encrypts the communications, and sends the communications to the device 200 over the first transport layer connection. . The device 200 associates a first transport layer connection on the first network 104 with a second transport layer connection to the server 106 on the second network 104'. The device 200 receives the intercepted communication from the client agent 102 , decrypts the communication, and sends the communication to the server 106 on the second network 104 ′ via a second transport layer connection. The second transport layer connection may be a pooled transport layer connection. As such, the device 200 provides an end-to-end secure transport layer connection for the client 102 between the two networks 104 and 104'.

In one embodiment, device 200 hosts an Intranet Internet Protocol or IntranetIP 282 address of client 102 on virtual private network 104 . The client 102 has a local network identifier, such as an Internet Protocol (IP) address and/or a host name on the first network 104 . When connected to the second network 104' via the device 200, the device 200 provides an IntranetIP network identifier such as an IP address and/or hostname for the client 102 on the second network 104'. The address 282 is established, assigned, or otherwise provided. The device 200 listens and receives on the second or private network 104 ′ for any communication directed towards the client 102 using the client's established IntranetIP 282 . In one embodiment, the device 200 acts as or on behalf of the client 102 on the second private network 104 . For example, in another embodiment, vServer 275 listens for and responds to client 102's IntranetIP 282 communication. In some embodiments, when computing device 100 on second network 104 ′ sends a request, device 200 processes the request as if it were client 102 . For example, the device 200 may respond to a ping of the client's IntranetIP 282 . In another example, the device may establish a connection, such as a TCP or UDP connection, and the computing device 100 on the second network 104 ′ requests a connection with the client's IntranetIP 282 .

In some embodiments, device 200 may include: 1) compression; 2) decompress; 3) transmission control protocol pooling; 4) Transmission control protocol multiplexing; 5) Buffering Transmission Control Protocol; and 6) an acceleration technique 288 , such as caching, for communication between the client 102 and the server 106 .

In one embodiment, the device 200 opens one or more transport layer connections with each server 106 and maintains these connections to allow repeated data access by the client over the Internet, thereby transmitting to the client 102 . Offloading much of the processing load on the server 106 caused by repeatedly opening and closing hierarchical connections. This technique is referred to herein as “connection pooling”.

In some embodiments, the device 200 modifies the sequence number and acknowledgment number at the transport layer protocol level to seamlessly splice communication from the client 102 to the server 106 over the pooled transport layer connection. Translates or multiplexes communications by doing so. This is referred to as "connection multiplexing". In some embodiments, application layer protocol interaction is not required. For example, in the case of an in-bound packet (ie, a packet received from the client 102 ), the source network address of the packet is changed to the address of the output port of the device 200 , and the destination network address is changed to the address of the intended server. For an outbound packet (ie, a packet received from the server 106 ), the source network address is changed from the address of the server 106 to the address of the output port of the device 200 and the destination address is the address of the device 200 . ) is changed from the address of the requesting client 102 to the address of the requesting client 102 . The sequence number and acknowledgment number of the packet are also translated into the sequence number and acknowledgment number expected by the client 102 in the transport layer connection of the device 200 to the client 102 . In some embodiments, the packet checksum of the transport layer protocol is recomputed to account for these translations.

In another embodiment, the device 200 provides a switching or load-balancing function 284 for communication between the client 102 and the server 106 . In some embodiments, device 200 distributes traffic and directs client requests to server 106 based on layer 4 or application-layer request data. In one embodiment, the network layer or layer 2 of the network packet identifies the destination server 106, but the device 200 determines the server that will distribute the network packet with application information and data carried as the payload of the transport layer packet. 106) is determined. In one embodiment, the health monitoring program 216 of the device 200 monitors the health of the server to determine which server 106 to distribute the client's request to. In some embodiments, when device 200 detects that server 106 is unavailable or has a load that exceeds a predetermined threshold, device 200 directs the client request to another server 106 . or it can be distributed.

In some embodiments, device 200 acts as a domain name service (DNS) resolver or otherwise provides resolution of DNS requests from clients 102 . In some embodiments, the device intercepts DNS requests sent by the client 102 . In one embodiment, the device 200 responds to the client's DNS request with the IP address of the device 200 or the IP address hosted by the device 200 . In this embodiment, the client 102 sends a network communication for the domain name to the device 200 . In another embodiment, the device 200 responds to the client's DNS request with the IP address of the second device 200' or the IP address hosted by the second device 200'. In some embodiments, device 200 responds to the client's DNS request with the IP address of server 106 determined by device 200 .

In another embodiment, the device 200 provides an application firewall function 290 for communication between the client 102 and the server 106 . In one embodiment, the policy engine 236 provides rules for detecting and blocking illegal requests. In some embodiments, application firewall 290 protects against denial of service (DoS) attacks. In another embodiment, the device 200 examines the content of the intercepted request to identify and block application-based attacks. In some embodiments, the rules/policy engine 236 includes one or more application firewall or security control policies to provide protection against various classes and types of web or Internet based vulnerabilities, such as one or more of: 1) Buffer overflow, 2) CGI-BIN parameter manipulation, 3) form/hidden field manipulation, 4) strong browsing, 5) cookie or session poisoning, 6) broken access control list (ACL) or weak password, 7) Cross-site scripting (XSS), 8) command injection, 9) SQL injection, 10) errors triggering sensitive information leak, 11) insecure use of encryption, 12) server misconfiguration, 13) backdoor and Debug options, 14) website compromise, 15) platform or operating system vulnerabilities, and 16) zero-day exploits. In one embodiment, application firewall 290 provides HTML form field protection in the form of inspecting or parsing network communications for one or more of the following: 1) requested fields returned, 2) added fields allowed not, 3) read-only and hidden field enforcement, 4) drop-down list and radio button field conformance, and 5) form-field max-length enforcement. In some embodiments, application firewall 290 ensures that cookies are not modified. In another embodiment, application firewall 290 prevents strong browsing by enforcing legal URLs.

In another embodiment, application firewall 290 protects any confidential information contained in network communications. The application firewall 290 may inspect or analyze any network communication to identify any confidential information in any field of the network packet according to the rules or policies of the engine 236 . In some embodiments, application firewall 290 identifies one or more occurrences of a credit card number, password, social security number, name, patient code, contact information, and age in network communications. The encoded portion of the network communication may contain such generated or confidential information. Based on these occurrences, in one embodiment, application firewall 290 may take policy actions on network communications, such as preventing the transmission of network communications. In other embodiments, the application firewall 290 may rewrite, remove, or otherwise mask such identified occurrence or confidential information.

Still referring to FIG. 2B , the device 200 may include a performance monitoring agent 197 as described above with respect to FIG. 1D . In one embodiment, device 200 receives monitoring agent 197 from monitoring service 198 or monitoring server 106 , as shown in FIG. 1D . In some embodiments, the device 200 stores the monitoring agent 197 in storage, such as a disk, for delivery to any client 102 or server 106 that communicates with the device 200 . For example, in one embodiment, device 200 sends monitoring agent 197 to client 102 upon receiving a request to establish a transport layer connection. In another embodiment, the device 200 sends the monitoring agent 197 upon establishing a transport layer connection with the client 102 . In another embodiment, the device 200 sends a monitoring agent 197 to the client 102 upon intercepting or detecting a request for a web page. In another embodiment, device 200 sends monitoring agent 197 to client 102 or server 106 in response to a request from monitoring server 198 . In one embodiment, the device 200 sends the monitoring agent 197 to the second device 200 ′ or device 205 .

In another embodiment, the device 200 runs the monitoring agent 197 . In one embodiment, the monitoring agent 197 measures and monitors the performance of any application, program, process, service, task or thread running on the device 200 . For example, the monitoring agent 197 may monitor and measure the performance and operation of the vServers 275A-275N. In another embodiment, the monitoring agent 197 measures and monitors the performance of any transport layer connection of the device 200 . In some embodiments, monitoring agent 197 measures and monitors the performance of any user sessions traversing device 200 . In one embodiment, the monitoring agent 197 measures and monitors the performance of any virtual private network connection and/or session traversing the device 200, such as an SSL VPN session. In another embodiment, the monitoring agent 197 measures and monitors memory, CPU and disk usage and performance of the device 200 . In another embodiment, the monitoring agent 197 measures and monitors the performance of any acceleration techniques 288 performed by the device 200, such as SSL offloading, connection pooling and multiplexing, caching and compression. In some embodiments, the monitoring agent 197 measures and monitors the performance of any load balancing and/or content switching 284 performed by the device 200 . In another embodiment, the monitoring agent 197 measures and monitors the performance of the application firewall 290 protection and processing performed by the device 200 .

C. Client Agent

Referring now to Figure 3, an embodiment of a client agent 120 is shown. Client 102 includes a client agent 120 for establishing and exchanging communication with device 200 and/or server 106 over network 104 . In a brief overview, a client 102 operates on a computing device 100 having an operating system with kernel mode 302 and user mode 303 and a network stack 310 with one or more layers 310a-310b. . Client 102 may install and/or run one or more applications. In some embodiments, one or more applications may be delivered to the network 104 via the network stack 310 . One of the applications, such as a web browser, may also include a first program 322 . For example, the first program 322 may be used to install and/or run the client agent 120 or any portion thereof in some embodiments. The client agent 120 includes an interception mechanism or interceptor 350 for intercepting network communications from the network stack 310 from one or more applications.

The network stack 310 of the client 102 may include any type and form of software, or hardware, or any combination thereof, for providing connectivity and communication with a network. In one embodiment, the network stack 310 includes a software embodiment for a network protocol suite. The network stack 310 may include one or more network layers, such as any network layer of the Open Systems Interconnection (OSI) communication model recognized and understood by those of ordinary skill in the art. As such, the network stack 310 may include any type and type of protocol for any of the following layers of the OSI model: 1) a physical link layer, 2) a data link layer, 3) a network layer, 4) transport layer, 5) session layer, 6) presentation layer and 7) application layer. In one embodiment, network stack 310 may include Transmission Control Protocol (TCP) over the Network Layer Protocol of Internet Protocol (IP), commonly referred to as TCP/IP. In some embodiments, the TCP/IP protocol may be carried over an Ethernet protocol, which is equivalent to an IEEE wide-area-network (WAN) or local-area-network (LAN) protocol, such as these protocols covered by IEEE 802.3. It can include any of the families. In some embodiments, network stack 310 includes any type and form of wireless protocol, such as IEEE 802.11 and/or Mobile Internet Protocol.

From the perspective of a TCP/IP-based network, MAPI (Messaging Application Programming Interface) (email), FTP (File Transfer Protocol), HTTP (HyperText Transfer Protocol), CIFS (Common Internet File System) protocol (file transfer), ICA (Independent) Any TCP/IP based protocol may be used, including Computing Architecture) protocol, Remote Desktop Protocol (RDP), Wireless Application Protocol (WAP), Mobile IP protocol, and Voice over IP (VoIP) protocol. In another embodiment, the network stack 310 implements a modified transmission control protocol, e.g., transactional TCP (T/TCP), TCP with selective acknowledgment (TCP-SACK), TCP with large window (TCP-LW). ), congestion prediction protocols such as the TCP-Vegas protocol, and transmission control protocols of any type and form, such as the TCP spoofing protocol. In other embodiments, any type and form of User Datagram Protocol (UDP), such as UDP over IP, may be used by the network stack 310, such as for voice communications or real-time data communications.

Additionally, the network stack 310 may include one or more network drivers that support one or more layers, such as a TCP driver or a network layer driver. The network driver may be included as part of the operating system of the computing device 100 or as part of any network interface card or other network access component of the computing device 100 . In some embodiments, any network drivers of the network stack 310 may be customized, modified, or adapted to provide custom or modified portions of the network stack 310 that support any of the technologies described herein. In another embodiment, the acceleration program 302 is designed and configured to operate with or in conjunction with the network stack 310 installed by or otherwise provided by the operating system of the client 102 .

Network stack 310 includes any type and form of interfaces for receiving, obtaining, providing, or otherwise accessing any information and data related to network communications of client 102 . In one embodiment, the interface to the network stack 310 includes an application programming interface (API). Interfaces may also include any function calls, hooking or filtering mechanisms, event or callback mechanisms, or any type of interface technology. The network stack 310 via the interface may receive or provide data structures of any type and form, such as objects related to the function or operation of the network stack 310 . For example, a data structure may include information and data related to a network packet or one or more network packets. In some embodiments, the data structure includes a portion of a network packet that is processed at the protocol layer of the network stack 310 , such as a network packet at the transport layer. In some embodiments, data structure 325 includes a kernel-level data structure, while in other embodiments, data structure 325 includes a user-mode data structure. A kernel-level data structure is a data structure associated with or obtained from a portion of the network stack 310 operating in kernel mode 302 , or a network driver or other software running in kernel-mode 302 , or the kernel of the operating system. may contain any data structure obtained or received by a service, process, task, thread, or other executable instruction executing or operating in a -mode.

Additionally, some portions of the network stack 310 may run or operate in kernel mode 302 , for example, the data link or network layer, while other portions may be user-like, such as the application layer of the network stack 310 . It can run or operate in mode 303 . For example, a first portion 310a of the network stack 310 provides user-mode access of the network stack 310 to an application, while a second portion 310a of the network stack 310 accesses the network. provides In some embodiments, the first portion 310a of the network stack 310 may include one or more upper layers of the network stack 310, such as any of layers 5-7. In another embodiment, the second portion 310b of the network stack 310 includes one or more lower layers, such as any of layers 1-4. Each of the first portion 310a and the second portion 310b of the network stack 310 is at any one or more network layers in user-mode 303 , kernel-mode 302 , or a combination thereof, or a network may include any portion of the network stack 310 at any portion of the layer or interface point to the network layer or any portion of or interface point to user-mode 303 and kernel-mode 302 . have.

Interceptor 350 may include software, hardware, or any combination of software and hardware. In one embodiment, interceptor 350 intercepts network communications at any point in network stack 310 and directs network communications to destinations desired, managed, or controlled by interceptor 350 or client agent 120 . redirect or send For example, the interceptor 350 may intercept the network communication of the network stack 310 of the first network and transmit the network communication to the device 200 for transmission on the second network 104 . In some embodiments, interceptor 350 , including any type of interceptor 350 , includes a driver, such as a network driver, configured and designed to interface and operate with network stack 310 . In some embodiments, client agent 120 and/or interceptor 350 operate at one or more layers of network stack 310, such as at the transport layer. In one embodiment, interceptor 350 includes any form and type of suitable network driver interface that interfaces to the transport layer of the network stack, such as through a filter driver, hooking mechanism, or transport driver interface (TDI). In some embodiments, interceptor 350 interfaces to any layer above the transport protocol layer, eg, a transport layer, such as an application protocol layer, and a first protocol layer, such as another protocol layer. In one embodiment, the interceptor 350 may include a driver or NDIS driver conforming to the Network Driver Interface Specification (NDIS). In another embodiment, the interceptor 350 may include a mini-filter or a mini-port driver. In one embodiment, interceptor 350 , or a portion thereof, operates in kernel-mode 302 . In another embodiment, interceptor 350 , or a portion thereof, operates in user-mode 303 . In some embodiments, portions of interceptor 350 operate in kernel-mode 302 , while other portions of interceptor 350 operate in user-mode 303 . In another embodiment, the client agent 120 operates in user mode 303 but a kernel-mode driver, process, service, task, such as to obtain a kernel-level data structure 225 via the interceptor 350 . Or interface to part of the operating system. In a further embodiment, the interceptor 350 is a user-mode application or program, such as an application.

In one embodiment, interceptor 350 intercepts any transport layer connection request. In these embodiments, interceptor 350 executes a transport layer application programming interface (API) call to set destination information, such as destination IP address and/or port, to the desired location for that location. In this manner, interceptor 350 intercepts and redirects transport layer connections to IP addresses and ports controlled or managed by interceptor 350 or client agent 120 . In one embodiment, the interceptor 350 sets the incoming information for the connection to the local IP address and port of the client 102 on which the client agent 120 is listening. For example, the client agent 120 may include a proxy service that listens on a local IP address and port for redirected transport layer communications. In some embodiments, the client agent 120 then forwards the redirected transport layer communication to the device 200 .

In some embodiments, interceptor 350 intercepts Domain Name Service (DNS) requests. In one embodiment, client agent 120 and/or interceptor 350 parses the DNS request. In another embodiment, the interceptor sends the intercepted DNS request to the device 200 for DNS resolution. In one embodiment, the device 200 parses the DNS request and forwards the DNS response to the client agent 120 . In some embodiments, device 200 resolves DNS requests via another device 200 ′ or DNS server 106 .

In another embodiment, the client agent 120 may include two agents 120 and 120'. In one embodiment, the first agent 120 may include an interceptor 350 operating in the network layer of the network stack 310 . In some embodiments, first agent 120 intercepts network layer requests, such as Internet Control Message Protocol (ICMP) requests (eg, pings and traceroute). In another embodiment, the second agent 120' may operate at the transport layer and intercept transport layer communications. In some embodiments, first agent 120 intercepts communications in one layer of network stack 210 and forwards the intercepted communications to or interfaces with second agent 120'.

The client agent 120 and/or the interceptor 350 may operate at or interface with the protocol layer in a manner that is transparent to any other protocol layer of the network stack 310 . For example, in one embodiment, the interceptor 350 is transparent to any protocol layer below the transport layer, such as a network layer, and any protocol layer, above the transport layer, such as a session, presentation, or application layer protocol, the network stack. It interfaces with or operates with the transport layer of 310 . This allows the other protocol layers of the network stack 310 to operate as desired and without modification to use the interceptor 350 . As such, the client agent 120 and/or interceptor 350 interfaces with the transport layer to receive any communication provided over any protocol carried by the transport layer, such as any application layer protocol over TCP/IP. It can be secured, optimized, accelerated, routed or load-balanced.

In addition, the client agent 120 and/or interceptor 350 can be configured to network stack 310 in a manner transparent to any application, user of client 102 and any other computing device, such as a server in communication with client 102 . can operate on or interface with The client agent 120 and/or the interceptor 350 may be installed and/or executed on the client 102 without modification of the application. In some embodiments, the client 102 or the user of the computing device communicating with the client 102 is unaware of the presence, execution, or operation of the client agent 120 and/or interceptor 350 . As such, in some embodiments, the client agent 120 and/or the interceptor 350 is above and above the protocol layer interfaced by the application, the user of the client 102 , another computing device such as a server, or the interceptor 350 . It is transparently installed, executed and/or operated on any protocol layer below.

The client agent 120 includes an acceleration program 302 , a streaming client 306 , an aggregation agent 304 and/or a monitoring agent 197 . In one embodiment, client agent 120 includes an Independent Computing Architecture (ICA) client, or a portion thereof, developed by Citrix Systems, Inc. of Fort Lauderdale, Florida, also referred to as an ICA client. In some embodiments, client 120 includes an application streaming client 306 for streaming applications from server 106 to client 102 . In some embodiments, the client agent 120 includes an accelerator program 302 for accelerating communication between the client 102 and the server 106 . In another embodiment, client agent 120 includes aggregation agent 304 for performing endpoint detection/scanning and collecting endpoint information for device 200 and/or server 106 .

In some embodiments, the accelerator program 302 accelerates, enhances, or otherwise accelerates, enhances, or otherwise accesses a client's communication with the server 106 and/or access to the server 106 , such as accessing applications provided by the server 106 . and a client-side acceleration program for performing one or more acceleration techniques to improve upon. The logic, functions, and/or operations of the executable instructions of the acceleration program 302 may perform one or more of the following acceleration techniques: 1) multi-protocol compression, 2) transmission control protocol pooling, 3) transmission control protocol multiplexing. , 4) Transmission Control Protocol Buffering and 5) Caching via Cache Manager. Additionally, the acceleration program 302 may perform encryption and/or decryption of any communications received and/or transmitted by the client 102 . In some embodiments, the acceleration program 302 performs one or more of the acceleration techniques in an integrated manner or modality. In addition, the acceleration program 302 may perform compression for any protocol or multi-protocol carried as a payload of a network packet of a transport layer protocol.

The streaming client 306 includes applications, programs, processes, services, tasks, or executable instructions for receiving and executing applications streamed from the server 106 . The server 106 may stream one or more application data files to the streaming client 306 for playback, execution, or otherwise causing the application on the client 102 to be executed. In some embodiments, server 106 sends a set of compressed or packaged application data files to streaming client 306 . In some embodiments, the plurality of application files are compressed and stored on a file server in an archive file such as a CAB, ZIP, SIT, TAR, JAR, or other archive. In one embodiment, the server 106 decompresses, unpackages, or de-archives the application file and sends the file to the client 102 . In another embodiment, the client 102 decompresses, unpackages, or de-archives the application files. The streaming client 306 dynamically installs an application or a portion thereof and executes the application. In one embodiment, the streaming client 306 may be an executable program. In some embodiments, streaming client 306 may launch other executable programs.

The aggregation agent 304 includes an application, program, process, service, task, or executable instruction for identifying, obtaining, and/or collecting information about the client 102 . In some embodiments, device 200 sends aggregation agent 304 to client 102 or client agent 120 . The aggregation agent 304 may be configured according to one or more policies of the policy engine 236 of the device 200 . In another embodiment, the aggregation agent 304 sends the collected information about the client 102 to the device 200 . In one embodiment, the policy engine 236 of the device 200 uses the collected information to determine and provide access, authentication, and admission control of the client's connection to the network 104 .

In one embodiment, the aggregation agent 304 includes an endpoint detection and scanning mechanism that identifies and determines one or more attributes or characteristics of the client. For example, the aggregation agent 304 may identify and determine any one or more of the following client-side attributes: 1) the operating system and/or version of the operating system, 2) the service pack of the operating system, 3) Running services, 4) running processes, and 5) files. The aggregation agent 304 also identifies and determines the presence or version of any one or more of the following on the client: 1) anti-virus software, 2) personal firewall software, 3) anti-spam software, and 4) Internet security software. The policy engine 236 may have one or more policies based on any one or more of a client-side attribute or an attribute or characteristic of the client.

In some embodiments, client agent 120 includes monitoring agent 197 discussed with respect to FIGS. 1D and 2B . The monitoring agent 197 may be any type and type of script, such as Visual Basic or Java script. In one embodiment, monitoring agent 197 monitors and measures the performance of any portion of client agent 120 . For example, in some embodiments, the monitoring agent 197 monitors and measures the performance of the acceleration program 302 . In another embodiment, the monitoring agent 197 monitors and measures the performance of the streaming client 306 . In another embodiment, the monitoring agent 197 monitors and measures the performance of the aggregation agent 304 . In another embodiment, the monitoring agent 197 monitors and measures the performance of the interceptor 350 . In some embodiments, the monitoring agent 197 monitors and measures any resources of the client 102, such as memory, CPU, and disk.

The monitoring agent 197 may monitor and measure the performance of any application of the client. In one embodiment, the monitoring agent 197 monitors and measures the performance of the browser on the client 102 . In some embodiments, monitoring agent 197 monitors and measures the performance of any applications communicated through client agent 120 . In another embodiment, the monitoring agent 197 measures and monitors end user response times for applications such as web-based or HTTP response times. The monitoring agent 197 may monitor and measure the performance of the ICA or RDP client. In another embodiment, the monitoring agent 197 measures and monitors metrics for user sessions or application sessions. In some embodiments, monitoring agent 197 measures and monitors ICA or RDP sessions. In one embodiment, the monitoring agent 197 measures and monitors the performance of the device 200 in accelerating the delivery of applications and/or data to the client 102 .

3 , in some embodiments still referring to FIG. 3 , first program 322 automatically, silently, transparently or otherwise installs and/or executes a portion thereof, such as client agent 120 or interceptor 350 . can be used to In one embodiment, first program 322 includes plug-in components such as ActiveX controls or Java controls or scripts that are loaded into and executed by applications. For example, the first program 322 includes an ActiveX control that is loaded and executed by a web browser application, such as a memory space or context of an application. In another embodiment, the first program 322 includes a set of executable instructions that are loaded into and executed by an application, such as a browser. In one embodiment, the first program 322 includes a program designed and configured to install the client agent 120 . In some embodiments, first program 322 obtains, downloads, or receives client agent 120 over a network from another computing device. In another embodiment, the first program 322 is an installer program or plug and play manager for installing a program, such as a network driver, into the operating system of the client 102 .

D. Systems and Methods for Providing Virtualized Application Delivery Controllers

Referring now to FIG. 4A , a block diagram illustrates one embodiment of a virtualized environment 400 . In a brief overview, computing device 100 includes a hypervisor layer, a virtualization layer, and a hardware layer. The hypervisor layer allocates access to multiple physical resources (eg, processor(s) 421 and disk(s) 428 ) of the hardware layer by at least one virtual machine executing in the virtualization layer, and and a managing hypervisor 401 (also referred to as a virtualization manager). The virtualization layer includes at least one operating system 410 and a plurality of virtual resources allocated to the at least one operating system 410 . Virtual resources include, without limitation, multiple virtual processors 432a, 432b, 432c (typically 432) and virtual disks 442a, 442b, 442c (typically 442), as well as virtual resources such as virtual memory and virtual network interfaces. can do. The plurality of virtual resources and operating systems 410 may be referred to as virtual machines 406 . The virtual machine 406 may include a controlling operating system 405 that communicates with the hypervisor 401 and is used to run applications on the computing device 100 to manage and configure other virtual machines.

More specifically, hypervisor 401 may provide virtual resources to an operating system in any manner that simulates an operating system accessing a physical device. Hypervisor 401 may provide virtual resources to any number of guest operating systems 410a, 410b (generally 410). In some embodiments, computing device 100 executes one or more types of hypervisors. In these embodiments, a hypervisor may be used to run a virtual machine that emulates virtual hardware, partitions the physical hardware, virtualizes the physical hardware, and provides access to the computing environment. The hypervisor is manufactured by VMWare, Inc. of Palo Alto, California; XEN Hypervisor, an open source product overseen in development by the open source Xen.org community; may include HyperV, VirtualServer, or Virtual PC hypervisors provided by Microsoft or other vendors. In some embodiments, a computing device 100 running a hypervisor that creates a virtual machine platform on which a guest operating system may run is referred to as a host server. In one of these embodiments, for example, computing device 100 is a XEN SERVER provided by Citrix Systems, Inc. of Fort Lauderdale, FL.

In some embodiments, hypervisor 401 runs within an operating system running on the computing device. In one of these embodiments, the computing device running the operating system and hypervisor 401 includes a host operating system (an operating system running on the computing device) and a guest operating system (computing resource partitions provided by hypervisor 401 ). It can be said to have an operating system running within it. In another embodiment, the hypervisor 401 interacts directly with hardware on the computing device instead of running on the host operating system. In one of these embodiments, the hypervisor 401 may be said to run on “bare metal” with reference to the hardware comprising the computing device.

In some embodiments, hypervisor 401 may create virtual machines 406a - c (generally 406 ) on which operating system 410 runs. In one of these embodiments, for example, hypervisor 401 loads a virtual machine image to create virtual machine 406 . In another of these embodiments, hypervisor 401 runs operating system 410 within virtual machine 406 . In another of these embodiments, virtual machine 406 runs operating system 410 .

In some embodiments, hypervisor 401 controls processor scheduling and memory partitioning for virtual machine 406 executing on computing device 100 . In one of these embodiments, the hypervisor 401 controls the execution of the at least one virtual machine 406 . In another of these embodiments, the hypervisor 401 presents an abstraction of the at least one hardware resource provided by the computing device 100 to the at least one virtual machine 406 . In another embodiment, the hypervisor 401 controls whether and how physical processor capabilities are provided to the virtual machine 406 .

The controlling operating system 405 may execute at least one application for managing and configuring the guest operating system. In one embodiment, the controlling operating system 405 controls the execution of the virtual machine, including functions for executing the virtual machine, terminating the execution of the virtual machine, or identifying the type of physical resource for allocation to the virtual machine. A management application may be executed, such as an application comprising a user interface that provides an administrator with access to functionality for management. In another embodiment, the hypervisor 401 runs the controlling operating system 405 within the virtual machine 406 created by the hypervisor 401 . In yet another embodiment, the controlling operating system 405 runs on a virtual machine 406 having direct access to physical resources on the computing device 100 . In some embodiments, the controlling operating system 405a on the computing device 100a exchanges data with the controlling operating system 405b on the computing device 100b via communication between the hypervisor 401a and the hypervisor 401b. can do. In this manner, one or more computing devices 100 may exchange data with one or more other computing devices 100 regarding processors and other physical resources available in the pool of resources. In one of these embodiments, this functionality enables the hypervisor to manage a pool of resources distributed across multiple physical computing devices. In another of these embodiments, the plurality of hypervisors manage one or more guest operating systems running on one of the computing devices 100 .

In one embodiment, the controlling operating system 405 runs in a virtual machine 406 authorized to interact with at least one guest operating system 410 . In another embodiment, the guest operating system 410 communicates with the controlling operating system 405 via the hypervisor 401 to request access to a disk or network. In another embodiment, the guest operating system 410 and the controlling operating system 405 communicate with the hypervisor 401 , such as via a plurality of shared memory pages made available by the hypervisor 401 , for example. It can communicate through the communication channel established by

In some embodiments, the controlling operating system 405 includes a network back-end driver for communicating directly with the networking hardware provided by the computing device 100 . In one of these embodiments, the network back-end driver processes at least one virtual machine request from at least one guest operating system 110 . In another embodiment, the controlling operating system 405 includes a block back-end driver for communicating with a storage element on the computing device 100 . In one of these embodiments, the block back-end driver reads and writes data from the storage element based on at least one request received from the guest operating system 410 .

In one embodiment, the controlling operating system 405 includes a tool stack 404 . In another embodiment, the tool stack 404 interacts with the hypervisor 401 , communicates with another controlling operating system 405 (eg, on the second computing device 100b ), and the computing device 100 . ) provides a function for managing the virtual machines 406b and 406c. In another embodiment, the tool stack 404 includes custom applications for providing enhanced management capabilities to administrators of a virtual machine farm. In some embodiments, at least one of the tool stack 404 and the controlling operating system 405 includes a management API that provides an interface for remotely configuring and controlling the virtual machine 406 running on the computing device 100 . do. In another embodiment, the controlling operating system 405 communicates with the hypervisor 401 via the tool stack 404 .

In one embodiment, hypervisor 401 runs a guest operating system 410 within virtual machine 406 created by hypervisor 401 . In another embodiment, the guest operating system 410 provides a user of the computing device 100 with access to resources within the computing environment. In another embodiment, a resource is a program, application, document, file, plurality of applications, plurality of files, executable program file, desktop environment, computing environment, or other resource made available to a user of computing device 100 . includes In another embodiment, the resource is provided by conventional direct installation onto the computing device 100 , delivery of the computing device 100 via a method for application streaming, execution of the resource on the second computing device 100 ′. Output generated by delivery of output data to computing device 100 that is generated and passed to computing device 100 via a presentation layer protocol, execution of a resource via a virtual machine running on second computing device 100' Limited to transfer of data to computing device 100 , or from a removable storage device connected to computing device 100 , such as a USB device, or via a virtual machine running on computing device 100 and generating output data. However, it may be transmitted to the computing device 100 through a plurality of access methods including the same.

In one embodiment, the guest operating system 410 forms a fully-virtualized virtual machine that is unaware that it is a virtual machine in association with the virtual machine in which it runs; Such a machine may be referred to as a “Domain U Hardware Virtual Machine (HVM) virtual machine”. In another embodiment, a fully-virtualized machine includes software that emulates a Basic Input/Output System (BIOS) to run an operating system within the fully-virtualized machine. In another embodiment, a fully-virtualized machine may include a driver that provides functionality by communicating with the hypervisor 401 . In such an embodiment, the driver may recognize that it runs within a virtualized environment. In another embodiment, the guest operating system 410 forms a paravirtualized virtual machine in association with the virtual machine in which it runs, recognizing that it is a virtual machine; Such a machine may be referred to as a “Domain U PV Virtual Machine”. In other embodiments, the paravirtualized machine includes additional drivers that the fully-virtualized machine does not include. In another embodiment, the paravirtualized machine includes a network back-end driver and a block back-end driver included in the control operating system 405, as described above.

Referring now to FIG. 4B , a block diagram illustrates one embodiment of a plurality of networked computing devices in a system in which at least one physical host executes a virtual machine. In a brief overview, the system includes a management component 404 and a hypervisor 401 . The system includes a plurality of computing devices 100 , a plurality of virtual machines 406 , a plurality of hypervisors 401 , a plurality of management components, variously referred to as a tool stack 404 or management component 404 , and Physical resources 421 and 428 are included. A plurality of physical machines 100 may be provided as computing devices 100 , as described above with respect to FIGS. 1E-1H and 4A , respectively.

More specifically, physical disk 428 is provided by computing device 100 and stores at least a portion of virtual disk 442 . In some embodiments, virtual disk 442 is associated with a plurality of physical disks 428 . In one of these embodiments, one or more computing devices 100 may exchange data with one or more other computing devices 100 regarding the processors and other physical resources available in the pool of resources, such that the hypervisor may Allows management of a pool of resources distributed across computing devices. In some embodiments, the computing device 100 on which the virtual machine 406 runs is referred to as the physical host 100 or host machine 100 .

Hypervisor 401 runs on a processor on computing device 100 . The hypervisor 401 allocates the amount of access to the physical disk to the virtual disk. In one embodiment, the hypervisor 401 allocates an amount of space on the physical disk. In another embodiment, the hypervisor 401 allocates a plurality of pages on the physical disk. In some embodiments, hypervisor 401 provides virtual disk 442 as part of the process of initializing and running virtual machine 450 .

In one embodiment, the management component 404a is referred to as the pool management component 404a. In another embodiment, the management operating system 405a, which may be referred to as the controlling operating system 405a, includes a management component. In some embodiments, the management component is referred to as a tool stack. In one of these embodiments, the management component is the tool stack 404 described above with respect to FIG. 4A . In another embodiment, the management component 404 provides a user interface for receiving an identification of the virtual machine 406 being provided and/or running from a user, such as an administrator. In another embodiment, the management component 404 provides a user interface for receiving a request to move a virtual machine 406b from one physical machine 100 to another physical machine 100 from a user, such as an administrator. do. In a further embodiment, the management component 404a identifies the computing device 100b to execute the requested virtual machine 406d and executes the identified virtual machine to the hypervisor 401b on the identified computing device 100b. instructed to do; Such a management component may be referred to as a pool management component.

Referring now to FIG. 4C , an embodiment of a virtual application delivery controller or virtual appliance 450 is illustrated. In a brief overview, any of the functions and/or embodiments of the device 200 (eg, an application delivery controller) described above with respect to FIGS. 2A and 2B may be virtualized as described above with respect to FIGS. 4A and 4B . It can be deployed in any embodiment of the environment. Instead of the functionality of the application delivery controller being deployed in the form of appliance 200 , such functionality is implemented in virtualized environment 400 on client 102 , server 106 , or any computing device 100 , such as appliance 200 . ) can be placed in

Referring now to FIG. 4C , a diagram of an embodiment of a virtual device 450 operating on a hypervisor 401 of a server 106 is shown. Similar to device 200 of FIGS. 2A and 2B , virtual device 450 may provide functionality for availability, performance, offload, and security. For availability, the virtual appliance 450 performs load balancing between layers 4 and 7 of the network and may also perform intelligent service health monitoring. To improve performance through network traffic acceleration, the virtual device 450 may perform caching and compression. To offload processing of any server, the virtual machine 450 may perform connection multiplexing and pooling and/or SSL processing. For security, the virtual device 450 may perform any application firewall function and SSL VPN function of the device 200 .

Any module of device 200 as described in connection with FIG. 2A may include one or more software modules executable in virtualized environment 300 or non-virtualized environment on any server, such as an off the shelf server, or It may be packaged, combined, designed, or configured in the form of a virtualized device delivery controller 450 that may be deployed as a component. For example, the virtual device 450 may be provided in the form of an installation package for installation on a computing device. Referring to FIG. 2A , cache manager 232 , policy engine 236 , compression 238 , encryption engine 234 , packet engine 240 , GUI 210 , CLI 212 , shell service 214 . and any of the health monitoring programs 216 may be designed and configured as software components or modules to run on the computing device and/or any operating system of the virtualized environment 300 . Instead of using the cryptographic processor 260 , processor 262 , memory 264 and network stack 267 of appliance 200 , virtual appliance 450 may be provided by or otherwise provided by virtualized environment 400 . Any of these resources available on the server 106 may be used.

Still referring to FIG. 4C , in a brief overview, any one or more vServers 275a - 275n may be operated or executed in the virtualized environment 400 of any type of computing device 100 , such as any server 106 . can Any module or function of the device 200 described with respect to FIG. 2B may be designed and configured to operate in either a virtualized or non-virtualized environment of a server. Any vServer (275), SSL VPN (280), Intranet UP (282), Switching (284), DNS (286), Acceleration (288), AppFW (290), and Monitoring Agents (197) may contain one or more software modules or It may be packaged, combined, designed, or configured in the form of a deployable application delivery controller 450 as a component executable on the device and/or virtualized environment 400 .

In some embodiments, the server may run a plurality of virtual machines 406a - 406n in a virtual environment, each virtual machine running the same or a different embodiment of the virtual application delivery controller 450 . In some embodiments, the server may run one or more virtual machines 450 on one or more virtual machines on the core of the multi-core processing system. In some embodiments, the server may execute one or more virtual machines 450 on one or more virtual machines on each processor of the plurality of processor devices.

E. Systems and Methods for Providing Multi-Core Architectures

According to Moore's Law, the number of transistors that can be placed in an integrated circuit can double approximately every two years. However, CPU speed increases can peak, for example, CPU speeds have been in the approximate 3.5-4 GHz range since 2005. In some cases, CPU manufacturers may not rely on increasing CPU speed to gain additional performance. Some CPU manufacturers may add additional cores to their processors to provide additional performance. Products such as those from software and networking vendors that rely on CPUs for performance gains can take advantage of these multi-core CPUs to improve their performance. Software designed and configured for a single CPU may be redesigned and/or rewritten to take advantage of multi-threaded, parallel architectures or otherwise multi-core architectures.

The multi-core architecture of device 200 , referred to as nCore or multi-core technology, in some embodiments allows the device to break single-core performance barriers and utilize the power of multi-core CPUs. In the previous architecture described with respect to FIG. 2A, a single network or packet engine is running. The multiple cores of the nCore technology and architecture allow multiple packet engines to run concurrently and/or in parallel. By running the packet engine on each core, the device architecture utilizes the processing capacity of the additional cores. In some embodiments, this provides up to 7x performance increase and scalability.

5A illustrates some embodiments of distribution of operations, tasks, loads, or network traffic across one or more processor cores according to a type of parallel processing or parallel computing scheme, such as functional parallelism, data parallelism, or flow-based data parallelism. have. In brief overview, FIG. 5A depicts an embodiment of a multi-core system, such as device 200', with n-cores, a total number of cores of 1 to N. In one embodiment, the task, load or network traffic is a first core 505A, a second core 505B, a third core 505C, a fourth core 505D, a fifth core 505E, a sixth core. 505F, the seventh core 505G, etc. may be distributed over all or two or more of the n cores 505N (hereinafter collectively referred to as core 505 ). There may be multiple VIPs 275 each running on each of the multiple cores. There may be a plurality of packet engines 240 each running on each of the plurality of cores. Any approach used may lead to a different, variable or similar workload or performance level 515 across any core. For a functional parallel processing approach, each core may execute another of the functions provided by the packet engine, VIP 275 or device 200 . In the data parallel processing approach, data may be parallelized or distributed through cores based on a network interface card (NIC) or VIP 275 that receives the data. In another data parallel processing approach, processing may be distributed across cores by distributing the data flow to each core.

In further detail with respect to FIG. 5A , in some embodiments, loads, tasks, or network traffic may be distributed among cores 505 according to functional parallelism 500 . Functional parallelism 500 may be based on each core performing one or more respective functions. In some embodiments, the first core performs a first function, while the second core performs a second function. In the functional parallel processing approach, the functions to be performed by a multi-core system are divided and distributed to each core according to the function. In some embodiments, functional parallelism 500 may be referred to as task parallelism, and may be achieved when each processor or core executes a different process or function on the same or different data. A core or processor may execute the same or different code. In some cases, different threads of execution or code may communicate with each other as they operate. As part of a workflow, communication may occur to pass data from one thread to the next.

In some embodiments, distributing tasks across cores 505 according to functional parallelism 500 may include network input/output management (NW I/O) 510A, secure socket layer (SSL) encryption and decryption 510B. ), and distributing network traffic according to a specific function, such as a Transmission Control Protocol (TCP) function 510C. This may lead to a workload, performance, or computing load 515 based on the volume or level of functionality being used. In some embodiments, distributing tasks across cores 505 according to data parallelism 540 may include distributing workload 515 based on distributed data associated with particular hardware or software components. In some embodiments, distributing work across cores 505 according to flow-based data parallelism 520 may result in a similar, substantially equal, or relatively evenly distributed workload 515A-N on each core. may include distributing data based on context or flow so that

For a functionally parallel approach, each core may be configured to execute one or more of a plurality of functions provided by the packet engine or the device's VIP. For example, core 1 may perform network I/O processing for device 200', while core 2 performs TCP connection management for device 200'. Similarly, core 3 may perform SSL offloading, while core 4 may perform layer 7 or application layer processing and traffic management. Each core may perform the same function or a different function. Each core may perform more than one function. Any core may perform any function or portion thereof identified and/or described with respect to FIGS. 2A and 2B . In this approach, work across the core can be partitioned by function in either a coarse or fine-grained manner. In some cases, as shown in FIG. 5A , partitioning by function may lead to different cores running at different levels of performance or load 515 .

For a functionally parallel processing approach, each core may be configured to execute one or more of a plurality of functions provided by the device's packet engine. For example, core 1 may perform network I/O processing for device 200', while core 2 performs TCP connection management for device 200'. Similarly, core 3 may perform SSL offloading, while core 4 may perform layer 7 or application layer processing and traffic management. Each core may perform the same function or a different function. Each core may perform more than one function. Any core may perform any function or portion thereof identified and/or described with respect to FIGS. 2A and 2B . In this approach, work across the core can be partitioned by function in either a coarse or fine-grained manner. In some cases, as shown in FIG. 5A , partitioning by function may lead to different cores running at different levels of load or performance.

Functions or tasks may be distributed in any arrangement and scheme. For example, FIG. 5B illustrates Core 1 505A, which is a first core processing applications and processes associated with network I/O function 51OA. In some embodiments, network traffic associated with network I/O may be associated with a particular port number. Thus, outgoing and ingress packets having a port destination associated with the NW I/O 51OA will be destined for core 1 505A dedicated to handling all network traffic associated with the NW I/O port. Similarly, core 2 505B may be dedicated to handling functions associated with SSL processing, and core 4 505D may be dedicated to handling all TCP level processing and functions.

5A shows functions such as network I/O, SSL and TCP, but other functions may be assigned to the core. Such other functions may include any one or more of the functions or operations described herein. For example, any of the functions described with respect to FIGS. 2A and 2B may be distributed across cores on a functional basis. In some cases, a first VIP 275A may run on a first core, while a second VIP 275B with a different configuration may run on a second core. In some embodiments, each core 505 may handle a particular function, such that each core 505 may handle processing associated with that particular function. For example, core 2 505B may handle SSL offloading, while core 4 505D may handle application layer processing and traffic management.

In other embodiments, work, load, or network traffic may be distributed among cores 505 according to any type and form of data parallelism 540 . In some embodiments, data parallelism 540 may be achieved in a multi-core system with each core performing the same task or function on different pieces of data distributed. In some embodiments, a single thread of execution or code controls operations on all data fragments. In other embodiments, different threads or instructions may control the operation, but execute the same code. In some embodiments, data parallel processing 540 is included in or included in device 200 , packet engine, vServer (VIP) 275A-C, network interface card (NIC) 542D-E, and/or device 200 . in terms of any other networking hardware or software associated with it. For example, each core may run the same packet engine or VIP code or configuration, but may operate on a different set of distributed data. Each networking hardware or software configuration may receive a different, varying, or substantially the same amount of data and, consequently, may have a varying, different, or relatively equal amount of load 515 .

For the data parallel processing approach, tasks may be partitioned and distributed based on VIPs, NICs and/or data flows from VIPs or NICs. In either of these approaches, the tasks of a multi-core system can be split or distributed among VIPs by having each VIP task for a distributed data set. For example, each core may be configured to run one or more VIPs. Network traffic can be distributed to a core for each VIP that handles that traffic. In another of these approaches, the work of a device may be partitioned or distributed among cores based on which NIC receives the network traffic. For example, the network traffic of the first NIC may be distributed to the first core, while the network traffic of the second NIC may be distributed to the second core. In some cases, a core may process data from multiple NICs.

5A shows a single vServer associated with a single core 505, as in the case of VIP1 275A, VIP2 275B and VIP3 275C. In some embodiments, a single vServer may be associated with more than one core 505 . In contrast, more than one vServer may be associated with a single core 505 . Associating a vServer with the core 505 may include the core 505 processing all functions associated with a particular vServer. In some embodiments, each core runs a VIP with the same code and configuration. In another embodiment, each core runs a VIP with the same code but different configurations. In some embodiments, each core runs a VIP with different code and the same or different configuration.

Like a vServer, a NIC may also be associated with a particular core 505 . In many embodiments, a NIC may be coupled to one or more cores 505 such that when the NIC receives or transmits data packets, a particular core 505 handles processing related to the reception and transmission of data packets. In one embodiment, as in the case of NIC1 542D and NIC2 542E, a single NIC may be associated with a single core 505 . In other embodiments, more than one NIC may be associated with a single core 505 . In other embodiments, a single NIC may be associated with more than one core 505 . In these embodiments, the load may be distributed among one or more cores 505 such that each core 505 processes a substantially similar amount of load. A core 505 associated with a NIC may process all functions and/or data associated with a particular NIC.

Distributing tasks across cores based on data from VIPs or NICs may have some degree of independence, but in some embodiments, this may lead to disproportionate use of cores, as indicated by variable load 515 in FIG. 5A . can lead

In some embodiments, loads, tasks, or network traffic may be distributed among cores 505 based on any type and form of data flow. In another of these approaches, work may be partitioned and distributed among cores based on data flow. For example, network traffic between a client and a server traversing a device may be distributed and processed by one of a plurality of cores. In some cases, the core that initially establishes the session or connection may be the core to which network traffic for that session or connection is distributed. In some embodiments, data flow is based on any unit or portion of network traffic, such as traffic originating from a transaction, request/response communication, or application on a client. In this manner and in some embodiments, the data flow between the client and server traversing the device 200 ′ may be distributed in a more balanced manner than other approaches.

In flow-based data parallelism 520, distribution of data involves any type of data flow, such as request/response pairing, transaction, session, connection, or application communication. For example, network traffic between a client and a server crossing a device may be distributed to one core among a plurality of cores for processing. In some cases, the core that initially establishes the session or connection may be the core to which network traffic for that session or connection is distributed. The distribution of data flows may be such that each core 505 carries a substantially equal or relatively evenly distributed amount of load, data, or network traffic.

In some embodiments, data flow is based on any unit or portion of network traffic, such as traffic originating from a transaction, request/response communication, or application on a client. In this manner and in some embodiments, the data flow between the client and server traversing the device 200 ′ may be distributed in a more balanced manner than other approaches. In one embodiment, the data flow may be distributed based on a transaction or series of transactions. In some embodiments, this transaction may be between a client and a server, and may be characterized by an IP address or other packet identifier. For example, since core 1 505A may be dedicated to transactions between specific clients and specific servers, load 515A on core 1 505A may consist of network traffic associated with transactions between specific clients and servers. Allocating network traffic to core 1 505A may be accomplished by routing all data packets originating from a particular client or server to core 1 505A.

The work or load may be distributed among the cores based in part on transactions, but in other embodiments, the load or task may be assigned per packet. In these embodiments, the device 200 may intercept the data packet and assign it to the core 505 with the least amount of load. For example, the device 200 sends the first incoming data packet to the core 1 505A because the load 515A on core 1 505A is less than the load 515B-N on the remaining cores 505B-N. can be assigned Once the first data packet is assigned to core 1 505A, the load 515A on core 1 505A is increased proportionally to the amount of processing resources needed to process the first data packet. When device 200 intercepts the second data packet, device 200 will assign a load to core 4 505D because core 4 505D has the second least amount of load. Allocating data packets to the cores with the least amount of load may, in some embodiments, ensure that the loads 515A-N distributed to each core 505 remain substantially the same.

In another embodiment, the load may be allocated on a per-unit basis in which a section of network traffic is allocated to a particular core 505 . The above example shows load balancing on a per-packet basis. In other embodiments, the load may be assigned based on the number of packets such that every 10, 100, or 1000 packets is assigned to the core 505 with the least amount of load. The number of packets allocated to the core 505 may be a number determined by an application, user, or administrator, and may be any number greater than zero. In another embodiment, loads may be assigned based on time metrics such that packets are distributed to specific cores 505 for a predetermined amount of time. In these embodiments, a packet may be distributed to a particular core 505 for 5 milliseconds or for any period of time determined by a user, program, system, administrator, or otherwise. After a predetermined time has elapsed, the data packet is transmitted to the other core 505 for a predetermined period of time.

A flow-based data parallel processing method for distributing work, load, or network traffic among one or more cores 505 may include any combination of the above-described embodiments. This method may be performed by any part of the device 200 , by an application or set of executable instructions executing on one of the cores 505 , such as a packet engine, or in communication with any application, program or device 200 . may be performed by an agent running on a computing device.

The functional and data parallel processing computing scheme shown in FIG. 5A is a hybrid parallel processing or distributed processing comprising functional parallel processing 500 , data parallel processing 540 , flow-based data parallel processing 520 , or any portion thereof. They can be combined in any way to create a scheme. In some cases, a multi-core system may use any type and form of load balancing scheme for distributing the load among one or more cores 505 . The load balancing scheme may be used in any combination with any functional and data parallel processing scheme or combinations thereof.

5B shows an embodiment of a multi-core system 545, which may be one or more systems, appliances, devices, or components of any type and form. This system 545 may, in some embodiments, be included within a device 200 having one or more processing cores 505A-N. System 545 may further include one or more packet engines (PEs) or packet processing engines (PPEs) 548A-N in communication with memory bus 556 . A memory bus may be used to communicate with one or more processing cores 505A-N. Also included within the system 545 may be a flow distributor 550 that may further communicate with one or more network interface cards (NICs) 552 and one or more processing cores 505A-N. The flow distributor 550 may include a Receive Side Scaler (RSS) or a Receive Side Scaling (RSS) module.

Referring further to FIG. 5B and more particularly, in one embodiment packet engine(s) 548A-N are device 200 described herein, such as any part of the device described in FIGS. 2A and 2B . may include any part of Packet engine(s) 548A-N may, in some embodiments, include any of the following elements: packet engine 240 , network stack 267 ; cache manager 232; policy engine 236; compression engine 238; encryption engine 234; GUI 210; CLI (212); shell service 214; monitoring program 216; and any other software or hardware element capable of receiving data packets from either the memory bus 556 or one or more cores 505A-N. In some embodiments, packet engine(s) 548A-N may include one or more vServers 275A-N, or any portion thereof. In other embodiments, packet engine(s) 548A-N may provide any combination of the following functions: SSL VPN 280; intranet UP 282; switching 284; DNS (286); packet acceleration 288; App FW (290); monitoring, such as monitoring provided by monitoring agent 197; Functions associated with functions as a TCP stack; load balancing; SSL offloading and processing; content switching; policy evaluation; caching; compression; encoding; decompress; decoding; application firewall function; XML processing and acceleration; and SSL VPN connections.

Packet engine(s) 548A-N may be associated with a particular server, user, client, or network in some embodiments. When a packet engine 548 is associated with a particular entity, the packet engine 548 may process the data packet associated with that entity. For example, if a packet engine 548 is to be associated with a first user, that packet engine 548 may process and it will work Similarly, the packet engine 548 may choose not to associate with a particular entity, so that the packet engine 548 may process and otherwise operate on any data packets not generated by or destined for that entity. .

In some examples, packet engine(s) 548A-N may be configured to perform any of the functional and/or data parallel processing schemes shown in FIG. 5A . In this example, the packet engine(s) 548A-N may distribute the function or data among the processing cores 505A-N such that the distribution conforms to a parallel processing or distribution scheme. In some embodiments, a single packet engine(s) 548A-N performs a load balancing scheme, while in other embodiments one or more packet engine(s) 548A-N performs a load balancing scheme. In one embodiment, each core 505A-N may be associated with a particular packet engine 548 such that load balancing may be performed by the packet engine. Load balancing may, in this embodiment, allow each packet engine 548A-N associated with a core 505 to communicate with other packet engines associated with the core to collectively determine where packet engines 548A-N distribute the load. you can request to have One embodiment of this process may include a moderator receiving a vote from each packet engine for a load. The moderator may distribute the load to each packet engine 548A-N based in part on a priority value associated with the engine's voting age and in some cases the current load on the engine's associated core 505 .

Any packet engine running on the core may run in user mode, kernel, or any combination thereof. In some embodiments, the packet engine operates as if the running application or program is the user or application space. In these embodiments, the packet engine may use any type and form of interface to access any functionality provided by the kernel. In some embodiments, the packet engine operates in kernel mode or as part of the kernel. In some embodiments, a first portion of the packet engine operates in user mode, while a second portion of the packet engine operates in kernel mode. In some embodiments, the first packet engine on the first core runs in kernel mode, while the second packet engine on the second core runs in user mode. In some embodiments, the packet engine or any portion thereof operates on or in conjunction with the NIC or any driver thereof.

In some embodiments, memory bus 556 may be any type and form of memory or computer bus. Although a single memory bus 556 is shown in FIG. 5B , the system 545 may include any number of memory buses 556 . In one embodiment, each packet engine 548 may be associated with one or more separate memory buses 556 .

The NIC 552 may in some embodiments be any of the network interface cards or mechanisms described herein. The NIC 552 may have any number of ports. The NIC 552 may be designed and configured to connect to any type and type of network 104 . Although a single NIC 552 is shown, the system 545 may include any number of NICs 552 . In some embodiments, each core 505A-N may be associated with one or more single NICs 552 . Thus, each core 505 may be associated with a single NIC 552 dedicated to a particular core 505 .

Cores 505A-N may include any of the processors described herein. Further, the cores 505A-N may be configured according to any of the core 505 configurations described herein. Additionally, cores 505A-N may have any of the core 505 functions described herein. 5B shows seven cores 505A-G, any number of cores 505 may be included in system 545 . In particular, system 545 may include “N” cores, where “N” is an integer greater than zero.

The core 505 may have or use memory allocated or allocated for use in the core 505 . Memory is considered private or local memory of that core 505 and can only be accessed by that core 505 . The core 505 may have or use memory shared or allocated to multiple cores 505 . Memory may be considered public or shared memory that may be accessed by more than one core 505 . Core 505 may use any combination of private and public memory. Using a separate address space for each core 505 eliminates some coordination from using the same address space. If a separate address space is used, the core 505 can perform operations on information and data in the core 505's own address space without worrying about collisions with other cores 505 . Each packet engine may have a separate memory pool for TCP and/or SSL connections.

With further reference to FIG. 5B , any of the functions and/or embodiments of the core 505 described above with respect to FIG. 5A may be deployed in any embodiment of the virtualized environment described above with respect to FIGS. 4A and 4B . can be Instead of the functionality of the core 505 being deployed in the form of a physical processor 505 , such functionality is implemented in a virtualized environment on any computing device 100 , such as the client 102 , server 106 or appliance 200 . 400 may be disposed. In other embodiments, instead of the functionality of the core 505 being deployed in the form of a device or a single device, the functionality may be deployed across a plurality of devices in any arrangement. For example, one device may include two or more cores 505 and another device may include two or more cores 505 . For example, a multi-core system may include a cluster of computing devices, a server farm, or a network of computing devices. In some embodiments, instead of the functionality of the core 505 being deployed in core form, the functionality may be deployed on multiple processors, such as multiple single core processors.

In one embodiment, core 505 may be any type and type of processor. In some embodiments, core 505 may function substantially similar to any processor or central processing unit described herein. In some embodiments, core 505 may include any portion of any processor described herein. 5A shows seven cores, there may be any “N” cores 505A-N in device 200 , where “N” is any integer greater than one. In some embodiments, the core 505 may be installed within a common device 200 , while in other embodiments the core 505 may be installed within one or more device(s) 200 communicatively connected to each other. have. Core 505 may include graphics processing software in some embodiments, while in other embodiments core 505 provides general processing functions. The cores 505 may be physically installed close to each other and/or may be communicatively connected to each other. The core 505 may be any type and/or physically and/or communicatively coupled to the core 505 for passing data to, from, and/or between the cores 505 . It can be connected by a bus or subsystem of the form.

Each core 505 may include software for communicating with other cores 505 , although in some embodiments a core manager (not shown) may facilitate communication between each core 505 . In some embodiments, the kernel may provide core management. The cores 505 may interface or communicate with each other using various interface mechanisms. In some embodiments, core-to-core messaging may be used for communication between cores 505 , such as a first core sending messages or data to a second core via a bus or subsystem that connects the cores 505 . . In some embodiments, core 505 may communicate via any type and form of shared memory interface. In one embodiment, there may be more than one memory location shared among all cores 505 . In some embodiments, each core 505 may have a separate memory location shared with each other and core 505 . For example, the first core 505A may have a first shared memory with the second core 505B and a second shared memory with the third core 505C. In some embodiments, the core 505 may communicate via any type of programming or API, such as a function call via the kernel. In some embodiments, the operating system may recognize and support multiple core devices and provide interfaces and APIs for inter-core communication.

Flow distributor 550 may be any application, program, library, script, task, service, process, or executable instruction of any type and form running on any type and type of hardware. In some embodiments, flow distributor 550 may be any design and configuration of circuitry for performing any of the operations and functions described herein. In some embodiments, flow distributor 550 distributes, forwards, routes, controls and/or manages distribution of data packets between core 505 and/or packet engines or VIPs running on core 505 . In some embodiments, flow distributor 550 may be referred to as an interface master. In one embodiment, flow distributor 550 includes a set of executable instructions that are executed on a processor or core 505 of device 200 . In another embodiment, flow distributor 550 includes a set of executable instructions that are executed on a computing machine in communication with device 200 . In some embodiments, flow distributor 550 includes a set of executable instructions that are executed on NIC 552, such as firmware. In another embodiment, flow distributor 550 includes any combination of software and hardware for distributing data packets between cores 505 or processors. In one embodiment, flow distributors 550 run on at least one of the cores 505A-N, while in another embodiment, individual flow distributors 550 assigned to each core 505A-N are associated with an associated core. (505A-N). The flow distributor 550 may use any type and form of statistical or probabilistic algorithm or decision making to balance the flow across the core 505 . Hardware of a device, such as NIC 552 or kernel, may be designed and configured to support sequential operation across NIC 552 and/or core 505 .

In embodiments where system 545 includes one or more flow distributors 550 , each flow distributor 550 may be associated with a processor 505 or packet engine 548 . Flow distributors 550 may include an interface mechanism that enables each flow distributor 550 to communicate with other flow distributors 550 executing within system 545 . In one example, one or more flow distributors 550 may communicate with each other to determine how to balance the load. This process may operate substantially similar to the process described above for submitting votes to a moderator, whereupon the moderator determines which flow distributor 550 should receive the load. In another embodiment, the first flow distributor 550 ′ identifies a load on the associated core 505 and determines whether to forward the first data packet to the associated core 505 based on any of the following criteria. may: the load on the associated core 505 exceeds a predetermined threshold; the load on the associated core 505 is below a predetermined threshold; the load on the associated core 505 is less than the load on the other core 505; or any other metric that may be used to determine where to forward the data packet based in part on the amount of load on the processor.

The flow distributor 550 may distribute network traffic among the cores 505 according to a distribution, computing, or load balancing scheme as described herein. In one embodiment, the flow distributor 550 is a functional parallel processing distribution scheme 550 , a data parallel processing load distribution scheme 540 , a flow-based data parallel processing distribution scheme 520 , or any combination of these distribution schemes. or distribute the network traffic according to any one of any load balancing scheme for distributing the load among the plurality of processors. Thus, flow divider 550 may act as a load divider by taking data packets and distributing them across processors according to an operating load balancing or distribution scheme. In one embodiment, flow distributor 550 may include one or more operations, functions, or logic for determining how to distribute packets, tasks, or loads accordingly. In yet another embodiment, flow distributor 550 may include one or more sub-operations, functions, or logic capable of identifying source addresses and destination addresses associated with data packets and distributing packets accordingly.

In some embodiments, flow distributor 550 may include receive-side scaling (RSS) network drivers, modules, or executable instructions of any type and form to distribute data packets between one or more cores 505 . . The RSS module may include any combination of hardware and software. In some embodiments, the RSS module operates in conjunction with flow distributor 550 to distribute data packets across cores 505A-N or among multiple processors of a multi-processor network. The RSS module may run within the NIC 552 in some embodiments, and may run on any one of the cores 505 in other embodiments.

In some embodiments, the RSS module uses a MICROSOFT receive-side-scaling (RSS) scheme. In one embodiment, RSS is a Microsoft Scalable Networking initiative technology that allows incoming processing to be balanced across multiple processors in a system while maintaining in-order delivery of data. RSS may use any type and form of hashing scheme to determine which core or processor to process a network packet.

The RSS module can apply a hash function of any type and form, such as the Toeplitz hash function. A hash function can be applied to a hash type or any sequence of values. The hash function may be a secure hash of any security level, or otherwise cryptographically secure. A hash function can use a hash key. The size of the key depends on the hash function. For the Toeplitz hash, the size can be 40 bytes for IPv6 and 16 bytes for IPv4.

The hash function may be designed and constructed based on one or more criteria or design goals. In some embodiments, hash functions may be used that provide uniform distribution of hash results for different hash inputs and different hash types, including TCP/IPv4, TCP/IPv6, IPv4 and IPv6 headers. In some embodiments, a hash function may be used that provides a uniformly distributed hash result when there are a small number (eg, two or four) of buckets. In some embodiments, a hash function may be used that provides a randomly distributed hash result when there is a large number of buckets (eg, 64 buckets). In some embodiments, the hash function is determined based on computational level or resource usage. In some embodiments, the hash function is determined based on the ease or difficulty of implementing the hash in hardware. In some embodiments, the hash function is determined based on the ease or difficulty of a malicious remote host sending a packet that it all hashes into the same bucket.

RSS can generate hashes from inputs of any type and form, such as a series of values. This set of values may include any part of a network packet, such as any header, field, or payload of the network packet or a portion thereof. In some embodiments, the input to the hash may be referred to as a hash type and may include any tuple of information associated with a network packet or data flow, such as any of the following: at least two IP addresses and two 4 tuples containing ports; 4 tuples containing any set of 4 values; 6 tuples; 2 tuples; and/or any other series of numbers or values. The following are examples of hash types that can be used by RSS:

A 4-tuple of source TCP port, source IP version 4 (IPv4) address, destination TCP port, and destination IPv4 address.

A 4-tuple of the source TCP port, the source IP version 6 (IPv6) address, the destination TCP port, and the destination IPv6 address.

A 2-tuple of the source IPv4 address and destination IPv4 address.

A 2-tuple of the source IPv6 address and destination IPv6 address.

2-tuple of source IPv6 address and destination IPv6 address, including support for parsing IPv6 extended headers.

The hash result, or any portion thereof, may be used to identify a core 505 or entity, such as a VIP or a packet engine for distributing network packets. In some embodiments, one or more hash bits or masks are applied to the hash result. A hash bit or mask can be any number of bits or bytes. The NIC 552 may support any number of bits, such as 7 bits. The network stack can set the actual number of bits to be used during initialization. The number will be between 1 and 7, inclusive of 1 and 7.

The hash result may be used to identify the core 505 or entity via any type and type of table, such as a bucket table or an indirection table. In some embodiments, the number of hash-result bits is used to index into the table. The range of the hash mask can effectively define the size of the direction table. The hash result, or part of the hash result itself, can be used to index the direction table. A value in the table may identify any core 505 or processor, such as by a core 505 or processor identifier. In some embodiments, all cores 505 of a multi-core system are identified in the table. In another embodiment, the ports of the core 505 of the multi-core system are identified in the table. The direction table may contain any number of buckets that may be indexed by a hash mask, for example from 2 to 128 buckets. Each bucket may contain an index value of a range that identifies the core 505 or processor. In some embodiments, the flow controller and/or RSS module may rebalance the network load by changing the direction table.

In some embodiments, the multi-core system 545 does not include an RSS driver or RSS module. In some of these embodiments, a software steering module (not shown) or software embodiment of an RSS module in the system operates in conjunction with or as part of a flow distributor 550 to operate as a core ( 505) to steer the packet.

The flow distributor 550 in some embodiments within any module or program on the appliance 200 , any one of the cores 505 and any one of the devices or components included within the multi-core system 545 . run on top In some embodiments, the flow splitter 550' may run on the first core 505A, while in other embodiments the flow splitter 550" may run on the NIC 552. In another embodiment, An instance of flow distributor 550' may run on each core 505 included in multi-core system 545. In this embodiment, each instance of flow distributor 550' is ) to pass packets back and forth across core 505. There may be situations where a response to a request packet may not be processed by the same core, i.e., first core 505A While processing the request, the second core 505B processes the response In this situation, an instance of the flow distributor 550' can intercept the packet and forward it to the desired, or correct, core 505; That is, the flow distributor instance 550' may communicate the response to the first core 505A. The plurality of instances of the flow distributor 550' may include any number of cores 505 and any combination of cores 505. ) can be executed on

Flow distributor 550 may operate in response to any one or more rules or policies. A rule may identify a core or packet processing engine receiving a network packet, data, or data flow. A rule may identify any type and form of tuple information associated with a network packet, such as a 4-tuple of source and destination IP addresses and source and destination ports. Based on the received packets matching the tuple specified by the rule, the flow distributor 550 may forward the packet to the core 505 or the packet engine. In some embodiments, packets are delivered to the core via shared memory and/or core-to-core messaging.

5B illustrates flow distributor 550 running within multi-core system 545 , in some embodiments, flow distributor 550 is a computing device or appliance located remotely from multi-core system 545 . can be run on In such an embodiment, the flow distributor 550 may communicate with the multi-core system 545 to take data packets and distribute the packets across one or more cores 505 . In one embodiment, flow distributor 550 receives data packets destined for device 200 , applies a distribution scheme to the received data packets, and applies the data packets to one or more cores 505 of multi-core system 545 . ) is distributed to In one embodiment, the flow distributor 550 allows the router to target a particular core 505 by changing the metadata associated with each packet such that each packet is targeted towards a sub-node of the multi-core system 545 . It can be included in a router or other device to In such an embodiment, CISCO's vn-tag mechanism may be used to alter or tag each packet with the appropriate metadata.

5C shows an embodiment of a multi-core system 575 including one or more processing cores 505A-N. In a brief overview, one of the cores 505 may be designated as the control core 505A and may be used as the control plane 570 for the other core 505 . The other core 505 may be a secondary core operating in the data plane while the control core 505A provides the control plane 570 . Cores 505A-N may share a global cache 580 . The control core 505A provides the control plane 570 , while the other cores 505 of the multi-core system 575 form or provide the data plane. These cores 505 perform data processing functions for network traffic, while control provides initialization, configuration, and control of the multi-core system 575 .

Referring further to FIG. 5C , and more particularly, control core 505A as well as cores 505A-N may be any of the processors described herein. Further, cores 505A-N and control core 505A may be any processor capable of functioning within system 575 described in FIG. 5C . Also, cores 505A-N and control core 505A may be any core or group of cores described herein. The control core 505A may be a different type of core or processor than the other cores 505 . In some embodiments, the control may operate a different packet engine or have a packet engine configured differently from the packet engine of the other core 505 .

Any portion of the memory of each core 505 may be allocated to or used for the global cache 580 shared by the cores 505 . In a brief overview, a predetermined percentage or predetermined amount of each of the memory of each core 505 may be used for the global cache 580 . For example, 50% of each memory of each core 505 may be dedicated or allocated to the shared global cache 580 . That is, in the embodiment shown, 2 GB of each core 505 except for the control plane core or core 1 may be used to form a 28 GB shared global cache 580 . Configuration of the control plane 570 , such as via a configuration service, may determine the amount of memory used for the shared global cache 580 . In some embodiments, each core 505 may provide a different amount of memory for use by the global cache 580 . In other embodiments, any one core 505 may not provide any memory or use the global cache 580 . In some embodiments, any core 505 may also have a local cache of memory that is not allocated to global shared memory. Each core 505 may store any portion of network traffic in a global shared cache 580 . Each core 505 may check the cache for any content for use in a request or response. Any core 505 may obtain content from the global shared cache 580 for use in a data flow, request or response.

Global cache 580 may be any type and form of memory or storage element, such as any memory or storage element described herein. In some embodiments, core 505 may access a predetermined amount of memory (ie, 32 GB or any other amount of memory corresponding to system 575 ). Global cache 580 may be allocated from a predetermined amount of memory, while the remaining available memory may be allocated between cores 505 . In other embodiments, each core 505 may have a predetermined amount of memory. The global cache 580 may contain the amount of memory allocated to each core 505 . This amount of memory can be measured in bytes, or it can be measured as a percentage of the memory allocated to each core 505 . Accordingly, the global cache 580 may include 1 GB of memory from the memory associated with each core 505 , or may include 20% or half of the memory associated with each core 505 . In some embodiments, only a portion of core 505 provides memory to global cache 580 , and in other embodiments, global cache 580 may include memory that is not allocated to core 505 .

Each core 505 may use a global cache 580 to store network traffic or cache data. In some embodiments, the packet engine of the core 505 uses the global cache 580 to cache and use data stored by a plurality of packet engines. For example, the cache manager of FIG. 2A and the cache function of FIG. 2B may share data for acceleration using the global cache 580 . For example, each packet engine may store a response, such as HTML data, in the global cache 580 . Any cache manager running on core 505 may access global cache 580 to cache server cache responses to client requests.

In some embodiments, core 505 may use global cache 580 to store port allocation tables that may be used to determine data flow based in part on ports. In another embodiment, the core 505 uses the global cache 580 to store an address lookup table or any other table or list that can be used by the flow distributor to determine where incoming and outgoing data packets are destined for. can Core 505 may in some embodiments read and write to or from global cache 580 , while in other embodiments core 505 may read from or write to global cache 580 or global cache 580 . ) can only be read or written to. The core 505 may perform core-to-core communication using the global cache 580 .

The global cache 580 may be partitioned into separate memory sections, each section of which may be dedicated to a particular core 505 . In one embodiment, the control core 505A may receive a larger amount of available cache, while the other core 505 may receive various amounts or access the global cache 580 .

In some embodiments, system 575 may include a control core 505A. 5C shows core 1 505A as the control core, the control core may be the device 200 or any core in a multi-core system. Also, although only a single control core is shown, system 575 may include one or more control cores, each having a level of control for system 575 . In some embodiments, one or more control cores may each control a particular aspect of system 575 . For example, one core may control determining which distribution scheme to use, while the other may determine the size of the global cache 580 .

The control plane of the multi-core system 575 may be a dedicated management core or the designation and configuration of the core 505 as a master core. This control plane core may provide control, management, and coordination of the operations and functions of the plurality of cores 505 in the multi-core system 575 . This control plane core may provide control, management and coordination of memory allocation and usage of the system among the plurality of cores 505 of the multi-core system 575 , including initialization and configuration of the multi-core system 575 . can In some embodiments, the control plane includes a flow distributor for controlling allocation of data flows to core 505 and distribution of network packets to core 505 based on data flows. In some embodiments, the control plane core runs the packet engine, and in other embodiments, the control plane core is dedicated to managing and controlling other cores of the system 575 .

The control core 505A controls the other cores 505, such as determining how much memory should be allocated to each core 505 or determining which cores 505 should be allocated to process a particular function or hardware/software entity. ) can exercise a level of control for In some embodiments, the control cores 505A may exercise control over these cores 505 within the control plane 570 . Accordingly, the processor may exist outside the control plane 570 that is not controlled by the control core 505A. Determining the boundaries of the control plane 570 may include maintaining a list of these cores 505 controlled by the control core 505A by the control core 505A or an agent executing within the system 575. can Control core 505A may control any of the following: initialization of core 505; determine when the core 505 is unavailable; redistribution of load to other cores 505 when one core 505 fails; Decide which distribution scheme to implement; determining the core 505 that should receive network traffic; determining how much cache should be allocated to each core 505; determining whether to assign a specific function or element to a specific core 505; determining whether to enable the cores 505 to communicate with each other; determine the size of the global cache 580; and any other determination of the function, configuration, or operation of the core 505 within the system 575 .

F. Systems and Methods for Providing a Distributed Cluster Architecture

As discussed in the previous section, to overcome limitations on transistor spacing and CPU speed increases, many CPU manufacturers have integrated multi-core CPUs to improve performance that can outperform even a single higher speed CPU. Similar or additional performance gains may be achieved by operating multiple devices, either single or multi-core, together as distributed or clustered devices. An individual computing device or appliance may be referred to as a node of a cluster. A centralized management system may perform load balancing, distribution, configuration, or other tasks, allowing nodes to work together as a single computing system. A cluster may be viewed as a single virtual device or computing device, although in many embodiments it outperforms typical individual appliances for external or other devices, including servers and clients.

Referring now to FIG. 6 , an embodiment of a computing device cluster or appliance cluster 600 is illustrated. A plurality of appliances 200a-200n, or other computing devices, sometimes referred to as nodes, such as desktop computers, servers, rack mount servers, blade servers, or any other type and type of computing device, is a single appliance cluster 600 ) can be combined. Although referred to as an appliance cluster, in many embodiments, the cluster may operate without limitation as an application server, network storage server, backup service, or any other type of computing device. In many embodiments, the appliance cluster 600 may be used to perform many of the functions of the appliance 200 , a WAN optimization device, a network accelerator appliance, or other devices described above.

In some embodiments, appliance cluster 600 may include a set of homogeneous computing devices, such as the same appliance, blade servers in one or more chassis, desktop or rack mount computing devices, or other devices. In other embodiments, device cluster 600 may include heterogeneous or mixed device sets including sets of different models of devices, mixed devices and servers, or any other computing devices. This may allow the appliance cluster 600 to expand or upgrade over time, for example, with new models or devices.

In some embodiments, each computing device or device 200 of the device cluster 600 may include a multi-core device as described above. In many of these embodiments, the core management and flow distribution methods described above may be utilized by each individual appliance in addition to the node management and distribution methods discussed herein. This can be thought of as a two-tier distribution system, where one device contains and distributes data to multiple nodes, and each node contains and distributes data to multiple cores for processing. Thus, in this embodiment, the node distribution system does not need to manage the distribution of flows to individual cores, which may be handled by the master or control core as described above.

In many embodiments, appliance cluster 600 may be physically grouped, such as multiple blade servers within a chassis or multiple rack mount devices in a single rack, although in other embodiments, appliance cluster 600 may include multiple chassis , multiple racks, multiple rooms in a data center, multiple data centers, or any other physical arrangement. Accordingly, the device cluster 600 may not be regarded as a physical group, but as a virtual device grouped through a common configuration, management, and purpose.

In some embodiments, the device cluster 600 may be connected to one or more networks 104 , 104 ′. For example, and referring briefly back to FIG. 1A , in some embodiments, the device 200 may be configured between a network 104 coupled to one or more clients 102 and a network 104 ′ coupled to one or more servers 106 . can be placed. Device cluster 600 may be similarly arranged to operate as a single device. In many embodiments, this may not require any network topology changes outside of the device cluster 600 , allowing for easy installation and scalability from a single device scenario. In other embodiments, the device cluster 600 may be arranged similarly as shown in FIGS. 1B-1D or described above. In another embodiment, the device cluster 600 may include a plurality of virtual machines or processes executed by one or more servers. For example, in one such embodiment, a server farm may run a plurality of virtual machines, each virtual machine configured with a device 200 , wherein the plurality of virtual machines operate cooperatively as a device cluster 600 . In another embodiment, device cluster 600 may include device 200 or a mixture of virtual machines configured as device 200 . In some embodiments, the device cluster 600 may be geographically distributed without a plurality of devices 200 co-located. For example, referring back to FIG. 6 , in one such embodiment, the first device 200a may be located at a first site, such as a data center, and the second device 200b may be located at a central office or corporate headquarters. It may be located on a second site such as In further embodiments, such geographic remote devices may include a dedicated network, such as a T1 or T3 point-to-point connection; VPN; or by any other type and type of network. Thus, there may be additional communication latency compared to co-located devices 200a-200b, but may have reliability advantages in case of site power failure or communication interruption, scalability or other benefits. In some embodiments, latency issues may be reduced through geographic or network-based distribution of data flows. For example, although configured as a device cluster 600, communications from clients and servers located at the company headquarters may be directed to devices 200b located at the site, and load balancing may be weighted by location; Similar steps can be taken to mitigate latency.

Still referring to FIG. 6 , the device cluster 600 may be connected to a network via a client data plane 602 . In some embodiments, the client data plane 602 may include a communications network, such as the network 104 that passes data between the client and the device cluster 600 . In some embodiments, the client data plane 602 may include a switch, hub, router, or other network device that bridges the external network 104 and the plurality of devices 200a - 200n of the device cluster 600 . . For example, in this embodiment, the router is connected to the external network 104 and may be connected to the network interface of each device 200a-200n. In some embodiments, such a router or switch may be referred to as an interface manager and may be further configured to evenly distribute traffic across nodes within the device cluster 600 . Thus, in many embodiments, the interface master 608 may include a flow distributor external to the instrument cluster 600 . In another embodiment, the interface master 608 may include one of the devices 200a-200n. For example, the first device 200a may serve as the interface master 608 receiving ingress traffic to the device cluster 600 and distributing the traffic across each device 200b - 200n . In some embodiments, return traffic may similarly flow from each device 200b - 200n via the first device 200a serving as the interface master 608 . In other embodiments, the return traffic from each device 200b - 200n may be transmitted directly to the networks 104 , 104 ′ or via an external router, switch, or other device. In some embodiments, devices 200 of device cluster 600 that do not function as interface masters 608 may be referred to as interface slaves 610a - 610n.

Interface master 608 may perform load balancing or traffic flow distribution in any of a variety of ways. For example, in some embodiments, the interface master 608 configures a router to perform equal-cost multi-path (ECMP) routing with the next hop configured of devices or nodes in the cluster. may include The interface master 608 may use open-shortest path first (OSPF). In some embodiments, the interface master 608 may use a stateless hash-based mechanism for traffic distribution, such as a hash or other packet information tuple based on an IP address, as described above. A hash key and/or salt may be selected for even distribution across nodes. In other embodiments, interface master 608 may perform flow distribution through a link aggregation (LAG) protocol, or any other type and form of flow distribution, load balancing, and routing.

In some embodiments, device cluster 600 may be connected to a network via server data plane 604 . Similar to client data plane 602 , server data plane 604 may include a communications network, such as network 104 ′, that carries data between servers and device cluster 600 . In some embodiments, the server data plane 604 may include a switch, hub, router, or other network device bridging the external network 104 ′ and the plurality of devices 200a - 200n of the device cluster 600 . have. For example, in this embodiment, the router may be connected to the external network 104', and may be connected to the network interface of each device 200a-200n. In many embodiments, each device 200a - 200n may include a plurality of network interfaces having a first network interface connected to a client data plane 602 and a second network interface connected to a server data plane 604 . can This may provide additional security and prevent direct interface of the client and server networks by having the appliance cluster 600 server as an intermediate device. In other embodiments, the client data plane 602 and the server data plane 604 may be merged or combined. For example, device cluster 600 may be deployed as a non-intermediary node on a network with client 102 and server 106 . As noted above, in many embodiments, the interface master 608 is on the server data plane 604 for routing and distribution of communications from the servers and networks 104 ′ to each device 200 in the device cluster 600 . can be placed in In many embodiments, the interface master 608 to the client data plane 602 and the interface master 608 to the server data plane 604 may be similarly configured by performing ECMP or LAG protocols as described above. .

In some embodiments, each device 200a - 200n in device cluster 600 may be connected via an internal communication network or backplane 606 . The backplane 606 may include a communications network for inter-node or inter-device control and configuration messages, and for inter-node delivery of traffic. For example, in one embodiment in which the first device 200a communicates with the client over the network 104 and the second device 200b communicates with the server over the network 104', communication between the client and the server can flow from the client to the first device 200a, from the first device 200a to the second device 200b via the backplane 606, and from the second device 200b to the server or vice versa. In another embodiment, the back plane 606 may include an interface pause or reset command; policy updates, such as filtering or compression policies; status messages such as buffer status, throughput, or error messages; or any other type and form of inter-node communication, such as configuration messages. In some embodiments, the RSS key or hash key may be shared by all nodes within the cluster 600 and communicated over the backplane 606 . For example, a first node or master node may select an RSS key such as startup or boot and distribute this key for use by other nodes. In some embodiments, the backplane 606 may include a network between the network interfaces of each device 200, and may include a router, switch, or other network device (not shown). Thus, in some embodiments and as described above, a router for the client data plane 602 may be placed between the device cluster 600 and the network 104 , and the router for the server data plane 604 is a device It may be deployed between cluster 600 and network 104 ′, and a router to backplane 606 may be deployed as part of device cluster 600 . Each router may be connected to a different network interface of each device 200 . In other embodiments, more than one plane 602-606 may be combined, or a router or switch may be divided into multiple LANs or VLANs to access different interfaces of devices 200a-200n and provide multiple routing functions simultaneously. This reduces complexity or eliminates additional devices from the system.

In some embodiments, a control plane (not shown) may control configuration and control traffic from administrators or users to device cluster 600 . In some embodiments, the control plane may be the fourth physical network, while in other embodiments the control plane may include a VPN, tunnel or communication through one of the planes 602-606. Thus, in some embodiments, the control plane may be considered a virtual communication plane. In other embodiments, the administrator may include a separate interface such as a serial communication interface such as RS-232; USB communication interface; or provide configuration and control through any other type and form of communication. In some embodiments, device 200 includes a front panel with buttons and a display; a web server for configuration via network 104, 104' or backplane 606; or interfaces for management, such as interfaces of any other type and form.

In some embodiments, as described above, the device cluster 600 may include an internal flow distribution. For example, this can be done to allow nodes to connect/emit transparently to external devices. A node or device may act as an interface master 608 or distributor to steer network packets to the correct node within cluster 600 so that external flow distributors do not have to be reconfigured repeatedly for such changes. For example, in some embodiments, if a node leaves the cluster (eg, fails, resets, or the like), an external ECMP router can identify the node's change and re-hash all flows to redistribute traffic. This can lead to all connections being lost and reset. The same disconnects and resets can occur when a node is reconnected. In some embodiments, for reliability reasons, two devices or nodes in device cluster 600 may receive communications from external routers via connection mirroring.

In many embodiments, the distribution of flows between the nodes of the device cluster 600 may use any of the methods described above for distribution of flows between the cores of a device. For example, in one embodiment, the master device, master node, or interface master 608 may compute an RSS hash, such as a Toeplitz hash, for incoming traffic and refer to a preference list or distribution table for the hash. In many embodiments, the flow distributor may provide the hash to the recipient device when forwarding the traffic. This can eliminate the need for nodes to recompute hashes to distribute flows to nodes. In many such embodiments, the RSS key used to compute the hash for distribution between devices may include the same key used to compute the hash for distribution among cores, which is a global RSS key. can be referred to, and enables reuse of computed hashes. In some embodiments, the hash may include a transport layer header including a port number, an Internet layer header including an IP address; or as an input tuple of any other packet header information. In some embodiments, packet body information may be used for hashing. For example, in one embodiment where traffic of one protocol is encapsulated within traffic of another protocol, such as lossy UDP traffic that is encapsulated via a lossless TCP header, the flow splitter is an encapsulating protocol (e.g., a TCP header) The hash may be calculated based on a header (eg, UDP) of an encapsulated protocol instead of . Similarly, in some embodiments where packets are encapsulated and encrypted or compressed, the flow distributor may compute a hash based on the header of the payload packet after decryption or decompression. In another embodiment, a node may have an internal IP address, such as for configuration or management purposes. Traffic to these IP addresses need not be hashed and distributed, but can be forwarded to the node that owns the destination address. For example, a device may have a web server or other server running for configuration or management purposes at an IP address of 1.2.3.4, and in some embodiments, it may register this address as an internal IP address with a flow distributor. In another embodiment, the flow distributor may assign an internal IP address to each node within the device cluster 600 . Traffic arriving from external clients or servers, such as workstations, used by the administrator and destined for the device's internal IP address (1.2.3.4) can be forwarded directly without the need for hashing.

G. Systems and Methods for Adaptive Packet Replication

This disclosure describes a solution for using a lossy Internet broadband link in a hybrid WAN link environment using adaptive packet replication of priority traffic on a secondary reliable link. The secondary link may be a multiprotocol label switching (MPLS) link. This solution can determine the link quality status based on the overall network reliability rather than the quality of a single link by calculating the effective packet loss in the entire network. The present solution can also adaptively duplicate packets when needed to reduce bandwidth. The present solution can also increase reliability by adaptively replicating packets over the secondary link. The solution can also identify priority traffic and duplicate packets from the priority traffic to the secondary link.

This solution can provide increased reliability by using the Internet broadband link in a complementary manner without frequent changes in link conditions, even with higher rates of packet loss and delay. By replicating packets over the second link, the present solution can increase throughput and improve latency by reducing packet retransmissions for accepted packets. The adaptive packet replication technique of the solution may also use less bandwidth when compared to a system that replicates all traffic to the second link. The present solution can also reduce the overhead added by the adaptive forward error correction (FEC) scheme and improve the capacity of the network compared to the fixed FEC and single link. This solution can also simplify Packet Order Correction (POC).

7 shows a block diagram of an exemplary intermediate device 700 . The intermediate device 700 may include a packet engine 548 . Packets from packet engine 548 may be sent to packet replicator 702 . The packet replicator 702 may replicate the packet according to the replication policy applied by the replication policy selector 704 based on the network conditions. The intermediate device 700 may also include a link quality estimator 706 that may provide an indication of the network status to the replication policy selector 704 .

The intermediate device 700 may include a link quality estimator 706 . Link quality estimator 706 may include an application, service, routine, daemon, or other executable logic for measuring and/or determining a quality of one or more network links between two network devices. For example, the intermediate device 700 may communicate with a second device via first and second links. A link quality estimator 706 may determine the quality of the first and second links. Link quality may also be referred to as a network metric. In some embodiments, link quality estimator 706 may transmit control packets over different links coupled with packet replicator 702 to determine the link quality of each link. Link quality may include or be based on metrics such as, but not limited to, link utilization, link latency, link jitter, link drop rate, or any combination thereof. Link quality estimator 706 may also use packets that have already traversed the link (eg, not specifically generated control packets) to make periodic measurements of link quality. In one example, the link quality estimator 706 may transmit a plurality of TCP packets and measure the round trip time until receiving an acknowledgment of the packets from the remote device. Link quality estimator 706 may determine the loss rate (or drop rate) by identifying packets that were not acknowledged or acknowledged negative (nACK) packets. Link quality estimator 706 may also measure the bandwidth of the connection, maximum segment size, window size, congestion, or any other characteristic of the network link. Link quality estimator 706 may measure latency based on round-trip time for the packet. The round trip time may be the amount of time from transmission of a packet to receipt of a packet acknowledgment. Link quality estimator 706 provides a link quality score and measures of bandwidth, utilization, jitter, latency (or delay), packet drop rate and other network characteristics for each link to replication policy selector 704 . can do. In some embodiments, link quality estimator 706 calculates one or more measurements of bandwidth, utilization, jitter, latency (or delay), packet drop rate, and other network characteristics for each link to replication policy selector 704 . can be used to generate or calculate link quality scores.

Link quality estimator 706 may determine network metrics of a plurality of links coupled with packet replicator 702 . A plurality of links may be connected to a remote device and may provide a plurality of paths between the packet replicator 702 and the remote device. Multiple links may provide paths to remote WANs or data centers. The packet replicator 702 may determine the network state of the link established between the remote device and the intermediate device 700 . The network status may include or be based on one or more network metrics. In some embodiments, the first link may be an Internet broadband link and the second link may be a multiprotocol label switching link. The remote device may be a virtual wide area network device.

The intermediate device 700 may include a packet replicator 702 . Packet replicator 702 may include applications, services, routines, daemons, or other executable logic for replicating or otherwise replicating data packets to one or more links. As shown, the packet replicator 702 is coupled to two links; The packet replicator 702 may be coupled to more than two links. Different links may provide different paths to remote devices. Links may traverse different networks, network types, and may be provided by different carriers. In one example, the first link may be an MPLS link and the second link may be an Internet connection provided by a cable or DSL carrier. The rate at which the packet replicator 702 replicates packets is controlled by the replication policy. The replication rate may indicate the number of replicated packets per unit time—eg, the number of replicated packets per second. In another implementation, the replication rate may be the number of replicated packets per packet received, eg, 20 packets are replicated per 100 received packets. A replication policy selector 704 , described in greater detail below, may determine which replication policy to implement and provide the replication policy to the packet replicator 702 . A replication policy may include a set of rules that may establish replication based on which packets are replicated, such as, for example, link quality or other network metrics. A replication policy may include one or more thresholds. A replication rate may be associated with each threshold. The threshold may be based on a metric of the network, such as link quality, or any measure used to determine link quality, such as packet drop rate and bandwidth utilization. For example, when the link quality estimator 706 determines that a first threshold of the replication policy is crossed, the packet replicator 702 may replicate the packet at the first rate; When the link quality estimator 706 determines that the second threshold is crossed, the packet replicator 702 may replicate the packet at a second rate as dictated by the replication policy.

In some embodiments, the original packet from packet engine 548 includes routing information in a header that packet replicator 702 can read to determine which link to send the original packet to. The packet replicator 702 may transmit the duplicate packet over a second, different link than the link through which the original packet is transmitted. For example, if the measured network metric (eg, link quality) is less than (or not meeting the threshold) a first threshold, the packet replicator 702 may only transmit the original packet over the first link. can If the network metric meets the threshold (eg, the latency increases above the latency threshold), the packet replicator 702 may transmit the duplicate packet over the second link. Packet replicator 702 may replicate packets at a replication rate associated with threshold crossing. The replication rate may indicate how many packets the packet replicator 702 replicates per received original packet or the number of packets the packet replicator 702 replicates during a given time interval.

In some embodiments, when replicating a packet over the second link consumes a predetermined amount of bandwidth of the second link, packet replicator 702 may start transmitting a portion of the duplicated packet over the first link. . For example, the link quality estimator 706 may determine that the bandwidth utilization value of the second link meets a predetermined threshold (eg, 80% of the total bandwidth of the link). The packet replicator 702 may transmit an amount of duplicate packets of the second link that meets a predetermined threshold, and may transmit duplicate packets above the predetermined threshold on the first link.

In some embodiments, the packet replicator 702 only transfers the packet to the first (original packet) Copy over link. For example, a receiving device may experience a network condition in which a large number of packets are dropped in the transmission of packets from the sending device to the receiving device. The packet replicator 702 may transmit the duplicate packet to the receiving device along the second link to reduce the effective packet loss rate. If the effective packet drop rate has not improved to an acceptable range when the replica packets reach the maximum bandwidth utilization (or threshold) of the second link, the packet replicator 702 may start replicating packets onto the first link. have.

The intermediate device 700 may include a replication policy selector 704 . Replication policy selector 704 may include an application, service, routine, daemon, or other executable logic for determining and selecting a replication policy. The replication policy selector 704 may receive network metrics from the link quality estimator 706 and select a replication policy based on the metrics of one or more links connected to the packet replicator 702 . The replication policy selector 704 may determine or set the packet replication rate based on the network metric. The replication policy selector 704 may determine the packet replication rate based on a difference between a predetermined network metric and a threshold. For example, the replication policy selector 704 may monitor the latency and packet loss rate of the first link. When the latency and packet loss rate are too high and cross a threshold, the replication policy selector 704 may instruct the packet replicator 702 to replicate the packet on the second link. The replication policy selector 704 may continue to receive network metric information from the link quality estimator 706 and instruct the packet replicator 702 to stop replicating packets when the latency and packet loss rate fall below threshold values.

The replication policy selector 704 may also set a replication rate at which the packet replicator 702 replicates packets. The replication policy selector 704 may receive the network metric of the first link, and when the network condition crosses the threshold, the replication policy selector 704 directs the packet replicator 702 to replicate the packet over the second link. can be instructed to start. The rate at which the replication policy selector 704 instructs the packet replicator 702 to replicate packets may be based on a network metric. For example, if the replication policy selector 704 determines that the network metric is deteriorating (eg, latency is increasing or the packet loss rate is increasing), the replication policy selector 704 increases the replication rate. can do it As the network metric degrades, the replication policy selector 704 may increase the replication rate to compensate for the degradation of the first link.

The replication policy selector 704 may set the replication rate based on a network metric of the first link alone or a network metric of one or more links. When determining the replication rate based on the network metrics of the plurality of links, the same or different types of network metrics may be used. In an example where different network metrics are used, the packet replication rate may be based on network metrics such as the determined bandwidth of the second link and latency or packet loss of the first link.

The replication policy selector 704 may set the replication rate based on the effective packet loss rate. Fig. 8a shows duplication of packets based on effective packet loss rate. The effective packet loss rate may be a packet loss rate that takes into account a packet loss rate over the first link and an acceptance rate of duplicate packets transmitted over the second link. For example, referring also to FIG. 8A , the first virtual WAN device 800(a) may communicate with the second virtual WAN device 800(b). The first virtual WAN device 800(a) may communicate with the second virtual WAN device 800(b) through the first link 802 and the second link 804 . Packet 806 is transmitted over first link 802 and second link 804 . The shaded packets 806 ′ represent packets that are dropped during transmission from the first virtual WAN device 800(a) to the second virtual WAN device 800(b) along the first link 802 . The first link 802 may be a link over the Internet, such as provided by a cable or DSL provider. The second link 804 may be an MPLS link.

As shown in Figure 8a. Seven packets 806 ′ are dropped on the first link 802 . Six packets 806 are replicated on the second link 804 . Without adaptive replication, the loss rate along the first link 802 is 7 packets. Calculating the effective packet loss rate may include determining a packet acceptance rate on the second link 804 . In some embodiments, the packet acceptance rate may be an acceptance rate of the total number of duplicate packets received at the receiving device. For example, in the example shown in FIG. 8A, the acceptance rate over the time frame shown is 6. In another embodiment, the packet acceptance rate may be based on a number of duplicate packets received on the second link 804 corresponding to packets lost on the first link 802 during a given time frame. For example, a packet 806(11) transmitted over the second link 804 corresponds to an accepted packet 806(11) transmitted over the first link 802, and the second link 804 Packet 806 ( 10 ) transmitted via ? corresponds to lost packet 806 ( 10 ) sent via first link 802 . Packet 806 on second link 804 corresponding to packet 806 on first link 802 may indicate that packet 806 is a duplicate of each other. For example, packet 806 ( 8 ) on second link 804 is a duplicate of packet 806 ( 8 ) on first link 802 . The effective packet loss rate may be based on the difference between the first packet loss rate over the first link 802 and the packet acceptance rate over the second link 804 . In the example shown in FIG. 8A , the first packet loss rate over the first link 802 is 7 packets during the time frame shown, and the shown (for packets corresponding to the lost packets) over the second link 804 . The packet acceptance rate during the time frame is 4. In this example, the effective loss rate for the indicated time frame is three. In some embodiments, the replication policy selector 704 may continue to increase the replication rate when the effective loss rate is below a predetermined threshold. For example, if the threshold was set to an effective loss rate of 2 during the time frame shown in the example above, the replication policy selector 704 will increase the replication rate because the effective loss rate is calculated to be 3.

The first link 802 and the second link 804 may include a plurality of paths between the first virtual WAN device 800(a) and the second virtual WAN device 800(b). Different paths may have different numbers of hops between the first virtual WAN device 800(a) and the second virtual WAN device 800(b). Other paths may traverse different physical connections along the link. The first device may mark one of the paths along the link as invalid if a predetermined number of packets transmitted along that path are dropped or delayed. In one example, when three consecutive packets are dropped while transmitting along the path of the first link 802 , the path may be marked invalid and the path may not be used for subsequent packets. The marking of the invalid path may be based on the drop rate or delay rate of the first link 802 and not on the combined valid drop or delay rate of the first link 802 and the second link 804. have.

Referring again to FIG. 7 , in some embodiments, the replication policy selector 704 may set the replication rate based on a delay rate of packets traversing the first link. The replication policy selector 704 may also set the replication rate based on the effective delay rate of the packet. Fig. 8b shows duplication of packets based on effective delay rate. 8B shows an exemplary system similar to that shown in FIG. 8A. 8B shows a system in which packets are replicated based on a delay rate. In FIG. 8B , packet 806 is transmitted between a first virtual WAN device 800(a) and a second virtual WAN device 800(b).

A data stream comprising packets 806 ( 1 ) - 806 ( 6 ) is transmitted along a first link 802 . Duplicate packets 806(2), 806(4) and 806(6) are transmitted along the second link 804. In the system shown in Fig. 8B, the shaded packet 806' is a delayed packet. For example, packet 806(2) on first link 802 is delayed from packet 806(1). The effective delay rate determines the first packet delay rate of the first link 802 and determines the packet acceptance rate of the second link 804 which may be based on the number of duplicate packets accepted for the delayed packets on the first link. can be calculated by the replication policy selector 704 by The effective packet delay rate may be based on a difference between the first packet delay rate and the packet acceptance rate. The effective packet delay rate may indicate effective silence between packets received by the second virtual WAN device 800(b). For example, virtual WAN device 800(a) receives packet 806(1) at time t=t ₁ , and packet 806(2) on first link 802 at t=t _n and receive packet 806(2) on second link 804 at t=t _m , where m is less than n. The silence between the reception of packets 806(1) and 806(2) on the first link 802 is (t _n -t ₁ ) . Since m is less than n , the effective silence rate of packet reception at virtual WAN device 800(b) is (t _m -t ₁ ) . In some embodiments, the replication policy selector 704 may increase the packet replication rate until the effective silencing rate is less than a predetermined threshold. For example, the packet replication rate may be increased such that additional packets are transmitted over the second link 804 . After transmitting the replication packet over the second link 804 at the increased replication rate for a predetermined amount of time (eg, 30 seconds), the link quality estimator 706 may reconfirm the effective silencing rate. The packet replicator 702 and the link quality estimator 706 may iteratively increase the replication rate and check the effective silencing rate until the effective silencing period is less than a predetermined threshold.

9 shows a block diagram of an example method 900 for transmitting a duplicate packet based on a network condition. The method 900 may include determining that a network state metric of a first link established between the first device and the second device is within a threshold (block 902 ). Method 900 may include generating, based on determining that the network state metric is within a threshold, a duplicate packet of the original packet to be transmitted from the first device to the second device along the first link ( block 904). The method 900 may include transmitting, by the first device, the original packet to the second device (block 906). Method 900 may include transmitting, by the first device, a copy packet to a second device on a second link established between the first device and the second device (block 908 ).

At block 902 , the method 900 may include determining, by the first device, whether a network state metric of a first link between the first device and the second device is within a threshold value. The first and second devices may be virtual WAN devices. The network health metric can be a packet loss rate over the first link, a packet delay time over the first link, an effective packet loss rate over the first and second links, or an effective packet delay time over the first and second links. have. The threshold may be a range of network status metrics set by the replication policy. The threshold may be set below a network health metric value at which a user or computer system begins to experience network outages. For example, if the packet loss rate rises to x, the computer system may begin to experience network outages. The threshold can be set to x/2 so that the system starts replicating packets before the user or system experiences a significant network outage. The first link may be an Internet broadband link and the second link may be a multiprotocol label switching link.

At block 904 , the method 900 may include generating a duplicate packet of the original packet to be transmitted between the first device and the second device. Upon determining that the network state metric is within the threshold, the packet replicator may begin intercepting and replicating packets transmitted from the packet engine to the second device over the first link. Duplicating the packet may include creating one or more copies of the original packet. Which packets are copied may be based on the replication rate. For example, the replication rate may be that every fifth packet is copied or a predetermined number of packets are copied every second. The packet duplication rate may be set based on a duplication policy. The replication rate may be based on a network health metric. For example, when the network status metric deteriorates to meet a first threshold, the replication rate may be set to a first value, and when the network status metric deteriorates further to meet a second threshold, the replication rate is It may be set to a second value greater than the first value. As the network condition metric deteriorates, setting the replication rate higher can counteract the effect of worsening network condition. For example, as more packets are dropped along the first link, the packet replicator creates a larger number of duplicate packets transmitted along the second link to counteract the effect of dropped packets on the first link. .

The replication rate may be based on the difference between a metric and a threshold of network conditions. For example, the replication rate may increase as the difference between the packet loss rate and the threshold for the packet loss rate increases. In some embodiments, a replication policy may include multiple thresholds. When the metric of the network condition meets the first threshold, the replication rate may be set to the first rate. When the metric of the network condition meets the second threshold, the replication rate may be set to the second rate. In some embodiments, the replication rate may be based on the bandwidth of the second link. For example, when the second link has a high bandwidth capacity, the replication rate may be set higher than that of the second link having a low bandwidth capacity.

At block 906 , the method 900 may include transmitting the original packet to the second device over the first link. The first link may be an Internet broadband link. The first link may be a primary link between the first device and the second device. The first link may include a plurality of paths between the first device and the second device.

At block 908 , the method 900 may include transmitting the duplicate packet to the second device via the second link. The second link may be an MPLS link. The second link may include a plurality of paths between the first device and the second device. In some embodiments, when the duplicate packets fill the maximum bandwidth utilization of the second link, the packet replicator may transmit the duplicate packets on the first link. The rate or amount of duplicate packets transmitted over the second link to the second device may be based on network state metrics such as effective packet delay rate and effective packet loss rate.

In some embodiments, portions of method 900 may be repeated and repeated after a predetermined amount of time. For example, the system may have a target effective packet loss rate or a target effective delay rate. The system may determine the current effective packet loss rate or effective delay rate. The system can then determine if the current rate is lower than the target rate. If there is a difference between the target rates such that the current effective packet loss rate or the current effective delay rate is less than the target rate, the system may increase the replication rate. After a predetermined time after the increase in the replication rate, the system may compare the updated rate to the target rate. The system may continue to increase the replication rate until the updated rate meets the target rate.

It should be understood that the system described above may provide any or a plurality of each of these components, and that these components may be provided either on a standalone machine or, in some embodiments, on either a plurality of machines in a distributed system. . The systems and methods described above may be implemented as methods, devices, or articles of manufacture using programming and/or engineering techniques to create software, firmware, hardware, or any combination thereof. In addition, the systems and methods described above may be provided as one or more computer readable programs embodied on or in one or more articles of manufacture. As used herein, the term “article of manufacture” refers to one or more computer readable devices, firmware, programmable logic, memory devices (eg, EEPROM, ROM, PROM, RAM, SRAM, etc.), hardware (eg, integrated circuits). Accessible from chips, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc.), electronic devices, computer-readable non-volatile storage units (eg CD-ROMs, floppy disks, hard disk drives, etc.) and is intended to include code or logic embedded therein. The article of manufacture may be accessed from a file server that provides access to the computer readable program via network transmission lines, wireless transmission media, signals propagating through space, radio waves, infrared signals, and the like. The article of manufacture may be a flash memory card or magnetic tape. The article of manufacture includes hardware logic as well as software or programmable code embodied in a computer readable medium executed by a processor. In general, a computer readable program may be implemented in any programming language such as LISP, PERL, C, C++, C#, PROLOG, or any byte code language such as JAVA. A software program may be stored as object code on or in one or more articles of manufacture.

While various embodiments of methods and systems have been described, these embodiments are illustrative and not limiting of the scope of the described methods or systems. Those skilled in the art can make changes in the form and details of the described methods and systems without departing from the broadest scope of the described methods and systems. Accordingly, the scope of the methods and systems described herein should not be limited by any of the exemplary embodiments, but should be defined in accordance with the appended claims and their equivalents.

Claims (20)

A method for transmitting a duplicate packet based on a network condition, comprising:
determining, by a first device, whether a metric of a network status of a first link established between the first device and a second device is within a threshold, wherein the metric of a network status of the first link is determined by the first is the packet delay rate of the link or the packet loss rate of the first link;
in response to determining, by the first device, that the metric of the network state of the first link is within the threshold, via a second link established between the first device and the second device; generating duplicate packets of packets to be transmitted from the first device to the second device, the duplicate packets being based on a loss rate or delay rate of the first link and a target effective packet acceptance rate of the second link created -;
transmitting, by the first device, the packets to be transmitted from the first device to the second device on the first link; and
transmitting, by the first device, the duplicate packets on the second link. delete The method of claim 1, wherein the threshold is a first threshold,
monitoring, by the first device, the network status of the first link;
determining, by the first device, whether the metric of the network status is within a second threshold; and
In response to determining, by the first device, that the metric of the network condition of the first link is within the second threshold, a packet corresponding to a rate at which duplicate packets are generated and transmitted on the second link. The method further comprising modifying the replication rate. According to claim 1,
determining, by the first device, a bandwidth of the second link; and
determining, by the first device, a packet replication rate based on the determined bandwidth of the second link and the metric of the network state of the first link,
and generating duplicate packets of packets to be transmitted from the first device to the second device over the first link comprises generating the duplicate packets based on the packet copy rate. 2. The method of claim 1, wherein determining the metric of the network state of the first link established between the first device and the second device comprises: a packet loss rate of the first link or a packet loss rate of the first link. determining one of the delay rates. The method of claim 1 , wherein the first link is an Internet broadband link and the second link is a multiprotocol label switched link and the first device and the second device are virtual wide area network devices. According to claim 1,
determining, by the first device, a first packet loss rate of the first link corresponding to a number of packets lost on the first link;
determining, by the first device, a packet acceptance rate of the second link based on a number of duplicate packets accepted for lost packets on the first link; and
determining, by the first device, an effective packet loss rate of the first link based on a difference between the first packet loss rate and the packet acceptance rate;
and the metric of the network state of the first link is the effective packet loss rate of the first link. According to claim 1,
determining, by the first device, a first packet delay rate of the first link corresponding to a number of packets delayed on the first link;
determining, by the first device, a packet acceptance rate of the second link based on a number of duplicate packets accepted for delayed packets on the first link; and
determining, by the first device, an effective packet delay rate of the first link based on a difference between the first packet delay rate and the packet acceptance rate;
and the metric of the network state of the first link is the effective packet delay rate of the first link. According to claim 1,
monitoring, by the first device, the network status of the first link to update the metric of the network status;
determining, by the first device, whether the updated metric of the network status of the first link is within the threshold in response to transmitting the duplicate packets on the second link; and
increasing, by the first device, a packet replication rate in response to determining, by the first device, that the updated metric of the network condition is within the threshold. According to claim 1,
determining, by the first device, whether a current metric of the network state of the first link is within the threshold;
determining, by the first device, whether a bandwidth usage metric of the second link is within a predetermined maximum bandwidth usage value assigned to the second link;
In response, by the first device, determining that the current metric of the network state of the first link is within the threshold, a packet replication rate to increase the number of replicated packets generated per unit time. increasing; and
in response to determining, by the first device, that the bandwidth usage metric of the second link is within the predetermined maximum bandwidth usage value, transmitting, by the first device, some of the duplicate packets on the first link. How to. A system for transmitting duplicate data packets based on network conditions, comprising:
A first device configured to establish a first link with a second device and establish a second link with the second device, the first device comprising:
determine that a metric of a network state of the first link established between the first device and the second device is within a threshold, wherein the metric of a network state of the first link is a packet delay rate of the first link or a packet loss rate of the first link;
in response to determining that the metric of the network state of the first link is within the threshold, from the first device to the second device via a second link established between the first device and the second device generate duplicate packets of packets to be transmitted to , the duplicate packets being generated based on a packet loss rate or delay rate of the first link and a target effective packet acceptance rate of the second link;
transmit the packets to be transmitted from the first device to the second device on the first link; and
and transmit the duplicate packets on the second link. delete The method of claim 11 , wherein the threshold is a first threshold, and the first device comprises:
monitor the network status of the first link;
determine whether the metric of the network condition is within a second threshold;
and in response to determining that the metric of the network condition of the first link is within the second threshold, modify a packet replication rate corresponding to a rate at which duplicate packets are generated and transmitted on the second link. constituted, system. 12. The method of claim 11, wherein the first device,
determine a bandwidth of the second link;
further configured to determine a packet replication rate based on the determined metric of the determined bandwidth of the second link and the network state of the first link,
and the first device is configured to generate the duplicate packets based on the packet copy rate to generate duplicate packets of packets to be transmitted from the first device to the second device over the first link. 12. The method of claim 11, wherein to determine the metric of the network state of the first link established between the first device and the second device, the first device is configured to: and determine one of a packet delay rate of the first link. The system of claim 11 , wherein the first link is an Internet broadband link and the second link is a multiprotocol label switched link and the first device and the second device are virtual wide area network devices. 12. The method of claim 11, wherein the first device,
determine a first packet loss rate of the first link corresponding to a number of packets lost on the first link;
determine a packet acceptance rate of the second link based on a number of duplicate packets accepted for lost packets on the first link; and
and determine an effective packet loss rate of the first link based on a difference between the first packet loss rate and the packet acceptance rate;
and the metric of the network state of the first link is the effective packet loss rate of the first link. 12. The method of claim 11, wherein the first device,
determine a first packet delay rate of the first link corresponding to a number of packets delayed on the first link;
determine a packet acceptance rate of the second link based on a number of duplicate packets accepted for delayed packets on the first link; and
and determine an effective packet delay rate of the first link based on a difference between the first packet delay rate and the packet acceptance rate;
and the metric of the network state of the first link is the effective packet delay rate of the first link. 12. The method of claim 11, wherein the first device,
monitor the network status of the first link to update the metric of the network status;
in response to transmitting the duplicate packets on the second link, determine whether the updated metric of the network status of the first link is within the threshold; and
and increase a packet replication rate in response to determining that the updated metric of the network condition is within the threshold. 12. The method of claim 11, wherein the first device,
determine whether a current metric of the network state of the first link is within the threshold;
determine whether a bandwidth usage metric of the second link is within a predetermined maximum bandwidth usage value assigned to the second link;
in response to determining that the current metric of the network condition of the first link is within the threshold, increase a packet replication rate to increase the number of replicated packets generated per unit time; and
and in response to determining that the bandwidth usage metric of the second link is within the predetermined maximum bandwidth usage value, transmit some of the duplicate packets on the first link.

Images