CN110912798B

CN110912798B - Method and system for transmitting data through aggregated connections

Info

Publication number: CN110912798B
Application number: CN201911234144.2A
Authority: CN
Inventors: 宋浩伟; 陈浩明; 吴锦超
Original assignee: Pismo Labs Technology Ltd
Current assignee: Pismo Labs Technology Ltd
Priority date: 2014-08-08
Filing date: 2014-08-08
Publication date: 2021-12-21
Anticipated expiration: 2034-08-08
Also published as: CN106576073A; WO2016020727A1; GB2532587B; CN106576073B; CN110912798A; GB201517507D0; GB2532587A; HK1232353A1

Abstract

Methods and systems for processing data packets received at a first network node and processing encapsulation packets received at a second network node are disclosed. The first network node receives a data packet from its network interface. It then selects the first tunnel according to a selection policy and does not select or selects the at least one second tunnel. The Original Encapsulation Packet (OEP) is transmitted through the first tunnel to the second network node and the at least one Duplicate Encapsulation Packet (DEP) is transmitted through the at least one second tunnel. The second network node receives an encapsulated packet with a Global Sequence Number (GSN) over the aggregated connection. The second network node determines whether one or more data packets corresponding to the encapsulation packet have been previously received. Subsequently, the second network node may determine whether to forward the one or more data packets.

Description

Method and system for transmitting data through aggregated connections

The present application is a divisional application of patent application having application number 2014800810806, application date 2014 8/8 entitled "method and system for transmitting data over aggregated connections".

Technical Field

The present invention relates generally to the field of computer networks. More particularly, the present invention relates to a method and system for processing data packets, encapsulating packets, and replicating encapsulating packets transmitted over an aggregated connection having a plurality of tunnels.

Background

A multi-Wide Area Network (WAN) site-to-site VPN router is a router that supports aggregating the bandwidth of multiple interconnects, such as WAN connections for accessing one or more remote private networks. In some embodiments, each TCP/IP session is routed to only one WAN. In this configuration, a single TCP file transfer session can only utilize the bandwidth of one WAN connection on each end. For example, in a session-based site-to-site Virtual Private Network (VPN) connection, VPN traffic is routed to multiple WAN connections between two sites (e.g., site a and site B).

In one embodiment, M × N tunnels are initially formed between WAN connections, where M and N are the number of WAN network connections for site a and site B, respectively. Subsequently, the application TCP/IP session is routed over a different tunnel. It is worth noting, however, that while a session-based site-to-site VPN can use different tunnels for different sessions, a single download session in this type of connection can only utilize one tunnel.

In wireless communications, the quality of packet transmission may be unpredictable, and the packet loss rate may change frequently. This can degrade the quality of the overall packet transmission. Even if the bandwidth limit per tunnel is high, the packet loss rate may not improve. There is a need for a solution to utilize multiple tunnels to increase the chances of successful transmission of data, which can be achieved by using duplicate packets.

Disclosure of Invention

Methods and systems for processing a data packet received at a first network node are disclosed. When the first network node receives a data packet from a network interface of the first network node, the first network node selects the first tunnel according to a selection policy and also does not select or selects the at least one second tunnel according to the selection policy. Subsequently, the first network node transmits an Original Encapsulation Packet (OEP) through the first tunnel. The OEPs encapsulate the data packets and each OEP has an original encapsulation packet global sequence number (OEP-GSN). The OEP-GSN is stored in a field of each OEP. The first network node also transmits at least one Duplicate Encapsulation Packet (DEP) over the at least one second tunnel when the at least one second tunnel is selected. The at least one DEP encapsulates at least one of the data packets. Each of the at least one DEP has a duplicate encapsulation packet global sequence number (DEP-GSN) stored in a field of each of the at least one DEP.

The first tunnel and the at least one second tunnel may be comprised in an aggregated connection. The selection strategy is based on one or more of the following criteria: user selection, performance of multiple tunnels, service provider, usage restrictions, location, time, usage price, security, user identity, internet protocol address range, communication protocol, communication technology, application and device. According to one of the embodiments, the performance of the first tunnel is determined to be better than the performance of the second tunnel, and the performance is substantially based on the delay and bandwidth of the tunnels.

According to one embodiment, the OEP-GSN of the OEP is the same as the DEP-GSN of the at least one DEP.

According to one embodiment, the at least one DEP comprises a list of OEP-GSNs. The list of OEP-GSNs comprises at least one OEP-GSN.

According to one of the embodiments, when transmitting a plurality of DEPs for each OEP, each of the plurality of DEPs is transmitted through a different tunnel of the aggregated connection.

Methods and systems for processing encapsulated packets received from a first network node over an aggregated connection at a second network node are also disclosed. The second network node receives the encapsulation packet through one of the tunnels of the aggregated connection. The encapsulation packet may be an OEP or a DEP. When the encapsulation packet is an OEP, it encapsulates the data packet. Alternatively, when the encapsulation packet is a DEP, it encapsulates packet information that is substantially based on the data packet. The data packet may originate from the first network node or be received by the first network node. The second network node determines whether a data packet has been previously received over the aggregated connection. The determination is substantially based on a record of missing Global Sequence Numbers (GSNs). When the second network node determines to forward the data packet, the second network node decapsulates the data packet from the encapsulation packet if the encapsulation packet is the OEP. If the encapsulation packet is a DEP, the second network node essentially recreates the data packet based on the data packet information. The second network node then forwards the data packet to its destination. The destination is indicated in the header of the data packet. Subsequently, the second network node may update the record of the missing GSN.

According to one embodiment, when the encapsulation packet is a DEP and the data packet information holds a plurality of encapsulated packets or error correction information, the encapsulation packet includes a list of OEP-GSNs.

According to one embodiment, the second network node determines whether the GSN corresponding to the data packet is in the record of the missing GSN. The second network node determines not to forward the packet if the GSN is not in the record of the missing GSN and instead determines to forward the packet if the GSN is in the record of the missing GSN.

According to one of the embodiments, the second network node may further update the expected global sequence number after determining whether to forward the data packet.

Drawings

FIG. 1A illustrates an overall system for optimizing throughput for a plurality of variable bandwidth connections according to an embodiment of the present invention;

FIG. 1B illustrates a network environment in accordance with various embodiments of the invention;

fig. 1C illustrates a system 100 adapted according to an embodiment configured to optimize throughput of a plurality of variable bandwidth connections being bonded;

FIG. 2A shows a flow diagram depicting a method for improving throughput of bonded connections according to an embodiment of the invention;

FIG. 2B shows a flow diagram depicting a method for improving throughput of a bonded connection according to an embodiment of the invention;

FIG. 3 is an exemplary embodiment illustrating the types of information that may be encapsulated in a transmitted IP packet according to embodiments of the invention;

FIG. 4A is an exemplary embodiment illustrating the types of information that may be encapsulated in a feedback packet according to embodiments of the invention;

FIG. 4B is a chart showing possible values for fields of the feedback packet of FIG. 4A;

FIG. 5 depicts a block diagram of a processing system suitable for implementing the present invention;

FIG. 6 is a flow diagram illustrating a process for transmitting encapsulating packets over an aggregated connection in accordance with various embodiments of the invention;

FIG. 7 is a flow chart illustrating a process for processing a received encapsulation packet;

FIG. 8A illustrates a structure of an Original Encapsulation Package (OEP) in accordance with various embodiments;

FIG. 8B illustrates the structure of a Duplicate Encapsulation Packet (DEP) in accordance with one of the embodiments of the present invention;

FIG. 8C shows the structure of a DEP according to one of the embodiments of the invention;

FIG. 9 is a flow diagram illustrating a process for transmitting DEPs according to one of the embodiments of the invention;

FIG. 10 is a flow diagram illustrating a process according to one of the embodiments of the invention;

FIG. 11 is a flow chart illustrating a process of processing encapsulated packets received over an aggregated connection;

FIG. 12A shows the contents of a DEP in accordance with an exemplary embodiment;

FIG. 12B illustrates the contents of a DEP in accordance with one of the embodiments;

FIG. 12C illustrates the contents of a DEP in accordance with one of the embodiments; and

FIG. 13 is a flow diagram illustrating a process for processing DEPs according to one of the embodiments.

Detailed Description

Fig. 1A shows a system 101 adapted according to an embodiment configured to optimize the throughput of a plurality of variable bandwidth connections bundled by adjusting a tunnel bandwidth weighting pattern during a data transmission session. System 101 includes a plurality of

stations

102 and 104 that each include at least one network node. The network node may be referred to as a communication router. However, the scope of the present invention is not limited to communication routers, such that the present invention can be performed at a gateway, router, server, or any other type of network node. For simplicity, fig. 1A shows: station 102 includes a communications router 106 and station 104 includes a communications router 108. Communication router 106 and communication router 108 may be implemented as multiple WAN routers supporting a bandwidth that aggregates multiple internet connections. The communication router 106 and the communication router 108 are connected by a network 110. Network 110 may include a LAN, MAN, WAN, wireless network, PSTN, Internet, intranet, extranet, or the like.

Site 102 and router 106 may include M connections 112 and site 104 and router 108 may include N connections 114. Connection 112 and connection 114 are data connections used to communicate information in network 110 between station 102 and station 104. In the illustrated embodiment, M equals 3 and N equals 2; however, these values may vary depending on the desired router and configuration. Connection 112 and connection 114 may have similar or different bandwidth capabilities. Further,

connections

112 and 114 may include different types of WAN connections, such as WiFi, cable, DSL, Tl, 3G, 4G, satellite connections, and so forth. It should also be noted that both station 102 and station 104 may be considered senders or receivers, and that discussion of the functionality of either station may be implemented on the other station. In other words, system 101 may be implemented as a symmetric network.

FIG. 1B illustrates a network environment in accordance with one of the embodiments of the invention.

Tunnels

103A, 103B, and 103C are established between communication router 106 and communication router 108.

Tunnels

103A, 103B, and 103C may be bound to form an aggregated connection.

According to one embodiment, communication router 106 and communication router 108 may have multiple network interfaces. Communication router 106 establishes

tunnels

103A, 103B, and 103C with one or more network interfaces of communication router 108 via one or more of its multiple network interfaces.

The

communication devices

106 and 108 may function as gateways, routers, switches, access points, hubs, bridges, and the like.

Fig. 1C illustrates a system 100 adapted according to an embodiment configured to optimize throughput of a plurality of variable bandwidth connections being bonded. System 100 is similar to system 101 except for MxN virtual tunnels 116. When a bonded connection is created between site 102 and site 104, such as by implementing a bonded site-to-site VPN connection, MxN tunnels 116 may be created. Tunnel 116 corresponds to a unique arrangement of network connections for station 102 and station 104. An aggregated connection may be formed between communication router 106 and communication router 108. Tunnel 116 may be a virtual tunnel.

Multiple established tunnels 116 may be aggregated, combined, or bound together to form one aggregated connection. Those skilled in the art will appreciate that there are an infinite number of ways to aggregate, combine or bind multiple established tunnels to form an aggregated tunnel. An aggregated connection is treated as a tunnel by the session or application that is using it. The aggregated connection may be an end-to-end connection, a virtual private network connection, or a connection-less connection. For example, the aggregated connection may be a TCP connection or a UDP connection. In another example, the aggregated connection is an aggregation of multiple tunnels, and each tunnel is linked between communication router 106 and communication router 108. In another example, the aggregated connection may be a VPN tunnel that includes a plurality of established tunnels, and each established tunnel is linked between communication router 106 and communication router 108.

Fig. 2A illustrates a high-level flow diagram depicting operation of system 100 of a method 200 for improving throughput of a bonded connection. It should be appreciated that the specific functions, sequences of functions, etc. provided in fig. 2A are intended as exemplary operations according to the inventive concept. Thus, the concepts herein may be implemented in a variety of ways other than the illustrated embodiments.

At block 201 of the illustrated embodiment, when a bonded connection is established between site 102 and site 104, such as by a site-to-site VPN connection implementing the bonding, MxN virtual tunnels 116 as shown in fig. 1C may be created. Virtual tunnel 116 corresponds to a unique arrangement of the network connections of station 102 and the network connections of station 104.

At block 202 of the illustrated embodiment, a default weight for the tunnel is determined and/or assigned. To determine the default weights, embodiments exchange uplink bandwidth data and downlink bandwidth data for

connections

112 and 114 between station 102 and station 104. Using such bandwidth data, default weights may be calculated according to the following method: assume that the downlink bandwidth of connection 1 to connection m of station 102 is d1, d2.. dm, and the uplink bandwidth of connection 1 to connection n of station 104 is U1, U2.. Un; the default weight for the tunnel between connection X of station 102 and connection Y of station 104 may be defined as DW (X, Y), where DW (X, Y) ═ dx.dy. Using the above method to calculate the default weights, if connection 112-1 to connection 112-3 are WAN connections of a multi-WAN router with corresponding uplink/downlink bandwidths of 10M/6M, 8M/4M and 6M/6M, and connection 114-1 to connection 114-2 are WAN connections of a multi-WAN router with corresponding uplink/downlink bandwidths of 7M/5M and 9M/3M, the respective default weights for each tunnel will be as follows:

[ TABLE 001 ]

Site 102	Site 104
		DW(1,1)＝6*7＝42	DW(1,1)＝5*10＝50
DW(1,2)＝6*9＝54	DW(1,2)＝5*8＝40
		DW(2,1)＝4*7＝28	DW(1,3)＝5*6＝30
DW(2,2)＝4*9＝36	DW(2,1)＝3*10＝30
		DW(3,1)＝6*7＝42	DW(2,2)＝3*8＝24
DW(3,2)＝6*9＝54	DW(2,3)＝3*6＝18

Note that other ways to calculate default weights are contemplated, and thus the above are merely implementation examples of embodiments of the present invention. Note that many different weighting patterns may be used to define the initial bandwidth of the tunnel. For example, it may be desirable to weight a tunnel in one direction using only the downlink capacity of the receiving station and the uplink capacity of the transmitting station. Any weighting pattern for characterizing the capacity of a tunnel when establishing a bonded connection may be used for the purposes of the present invention.

When a packet is routed from station 102 to station 104 according to an embodiment, the packet is assigned to a tunnel at a rate according to the effective weight EW (x, y). Initially, the effective weight of an embodiment is set equal to the default weight, EW (x, y) ═ DW (x, y), and is optimal for packet distribution if the bandwidth of tunnel 116 remains unchanged from the initial setting. However, if one or more tunnels drop packets while a user is downloading a file over a bonded network connection in a TCP session, the overall throughput of the session may be substantially reduced. This is because, in part, the packet loss will cause TCP to continue retransmission, and although the tunnel without the packet loss is not fully occupied, the flow control of TCP will maintain a low throughput.

An effective way to increase throughput is to avoid such packet loss. To this end, embodiments of the present invention identify at block 203 of the illustrated embodiment when the tunnel is experiencing an increase or decrease in packet loss rate. Embodiments also provide for modifying the effective weights of tunnels that are experiencing or have experienced a change in packet loss rate at block 204. The packet loss rate information may be continuously monitored or monitored based on a specific time period. Upon determining that the tunnel is experiencing an unacceptable rate of packet loss (block 204-1), the illustrated embodiment block reduces the effective weight of the tunnel at 204-2. In some embodiments, not acceptable may mean that the packet loss rate is a non-zero amount, while other embodiments may determine that the unacceptable rate is any rate that exceeds a predefined threshold. Embodiments implement effective weight reduction in a stepwise manner, in a continuous manner, in a one-time reduction proportional to an increase in packet loss rate, and the like. While decrementing in a step-wise manner, embodiments may continue to monitor the tunnel to optimize the amount of decrementing achieved.

Tunnel 116 may be established or monitored by sending heartbeat packets from router 106 or router 108 via each tunnel. In some embodiments, when the receiving end fails to receive a heartbeat packet from a tunnel for a period of time, it treats the tunnel as closed and the tunnel will not be used to route traffic. If the heartbeat packet begins to be received again, the tunnel may be re-established and weighted along with other tunnels. Thus, embodiments may utilize heartbeat packets to monitor and re-establish a connection in the event that all packets are lost in a tunnel and the effective weight of the tunnel is reduced to zero.

Further, when a tunnel recovers all or a portion of its respective bandwidth, e.g., a drop in packet loss rate is determined (block 204-3), the illustrated embodiment operates to increase the effective weight of such a tunnel (block 204-4) in order to fully or more fully utilize the bandwidth. Some embodiments use a predetermined step size to increase the effective weight of the tunnel until the exact effective weight is regained. Other embodiments may increase the effective weight in proportion to the re-measured bandwidth corresponding to the re-measured packet loss rate. Further, embodiments may increase the effective weight of the tunnel based on a predetermined linear or exponential scale.

After the effective weights of the tunnels are adjusted, or when it is determined that no adjustment is needed, then the weighting pattern of the system is updated at block 205 of the illustrated embodiment. The updating may include storing information for any processing, using such information in further processing, causing the system to take no action, etc. For example, the processing performed with respect to block 205 may be operated to average the weighted pattern over a period of time, e.g., to reduce errors associated with highly transient anomalies. Further, as discussed with respect to fig. 2B, the updated information may be used on the system 100 to modify the packet allocation for the data transfer session. The system 100 may continue to perform steps 203 through 205 continuously or periodically during a data transfer session.

Fig. 2B illustrates an embodiment in which packets are assigned based at least in part on the modified tunnel weights after the weighting method 200 is implemented. In particular, block 206 of the illustrated embodiment operates to distribute packets across the tunnel according to a weighting pattern determined by the operation of method 200. In some embodiments, this allocation will change during a data transfer session, and thus, the steps of fig. 2B are shown as a loop. Some embodiments change the packet allocation each time the system is updated at block 205. Further, block 205 may cause the change to be implemented periodically in response to certain loss rate change thresholds or the like. It should be appreciated that determining the weights by operation of the method 200 and applying the determined weights to the packet assignments at block 206 may have different periodicity. For example, the method 200 may operate to provide updates of weighting pattern information using relatively short iteration cycles, while changing the assignment of packets based on such weighting pattern information using longer iteration cycles.

To monitor the bandwidth of each tunnel 116, some embodiments of the present invention have each transmitted IP packet be populated with various information. Fig. 3 illustrates an example embodiment showing information types 300 that may be encapsulated in transmitted IP packets. The transmitted IP packets may be referred to as encapsulation packets, i.e., original encapsulation packets or duplicate encapsulation packets. Details of the original encapsulation packet and the duplicate encapsulation packet are shown in fig. 8A, 8B, and 8C. The version field 302 may contain information about the protocol version being used and the protocol type field 303 may contain the protocol type of the payload packet. Typically, the value of this field will correspond to the ethernet protocol type of the packet. However, additional values may be defined in other documents. Tunnel ID field 304 may be a 32-bit field and may contain an identifier for identifying the current tunnel of the IP packet. Advanced Encryption Standard (AES) initialization vector field 306 may be a 32-bit field and may contain an initialization vector for AES encryption. The global sequence number field 308 may be a 32-bit field and may contain a sequence number that is used to reorder the individual packets of the various sessions into the appropriate order as they emerge from their respective tunnels. Each tunnel sequence number field 310 may be a 32-bit field that may represent a sequence number assigned to each packet routed to a particular tunnel. The AES encrypted payload field 312 may be used to transport the payload of the IP packet. For higher security of the payload, AES encryption may be applied to prevent attacks from third parties.

Each tunnel sequence number discussed above may be used to monitor packets lost in the tunnel. In one embodiment, the router at the receiving end calculates the packet loss rate DR (x, y) per tunnel every f seconds by monitoring the per-tunnel sequence number of the received packet. DR (x, y) may be characterized as the missing sequence number divided by the sequence number increase over time period f. The length of the time period f is variable and in one embodiment f equals 5 seconds.

Other methods may also be used to monitor for lost packets, such as: the sender may periodically inform the receiving end how many data packets it has sent, the sender sends heartbeat packets to the receiving end every constant period of time and the receiving end can estimate the overall loss rate by monitoring the loss rate of the heartbeat packets, by obtaining loss rate data from the physical interface/device/layer, etc.

The receiving end may feed back to the sending router the loss rate, effective weight, or other bandwidth indicator for a particular tunnel. When the sender receives information about packet loss, some embodiments reduce the effective weight of the tunnel, EW (x, y), by EW (x, y) · DR (x, y). Other metrics may be used to modify the effective weights of the tunnels. In some embodiments, the sender may receive feedback and the effective weight may be reduced by a number greater or less than the packet loss rate. This difference can be configured according to the specific requirements of the communication system. The above example presents a metric that attempts to reduce the effective weight of the tunnel to a weight that maximizes the amount of available bandwidth of the tunnel while preventing further packet loss. Any metric that finds such a balance may be preferred.

Fig. 4A illustrates an example embodiment of information types 400 that may be encapsulated in a feedback packet sent to a transmitting router to report packet loss rate or other bandwidth-related data received at a receiving router. The type field 402 may include data related to the type of data to be included in the data 1 field 404 and the data 2 field 406. Data 1 field 404 and data 2 field 406 may contain any information that may be used to assist a router in determining tunnel information regarding the number of tunnels, the bandwidth of the tunnels, the number of packets lost in the tunnels, etc. An example of possible values for type field 402 in

data fields

404 and 406 is shown in the chart of FIG. 4B.

Information encapsulated in transmitted IP data packets, such as those shown in fig. 3 and 4, may also be used for packet buffering and reordering. Since the delay of each tunnel may be different, when two consecutive packets of the same TCP session are sent to a VPN peer through a bonded VPN tunnel, they may arrive out of order as they are routed via two different tunnels. If a TCP session receives out-of-order packets from a VPN, the TCP session will be slowed down by TCP retransmissions. Therefore, the receiving end should buffer packets that arrive too early until slower packets arrive or the expiration time has been exceeded. With this buffering, delayed packets arriving before the expiration time will be forwarded in-order to the target device. This buffering helps to optimize end-to-end throughput.

Note that the embodiments described herein are sometimes discussed in the context of VPN connections. These discussions are presented to illustrate example embodiments of binding connections. The inventive concepts claimed herein are not limited to such a connection. Indeed, any connection in which sufficient data is available and exchanged to dynamically monitor the bandwidth of multiple communication paths used in a data transfer session may be implemented with embodiments of the present invention.

As described above, each packet may be assigned two different sequence numbers, a Global Sequence Number (GSN), and a Per Tunnel Sequence Number (PTSN). These numbers may be used to facilitate packet buffering and reordering operations. After the packet is passed to the upper layer, the receiver may update the next expected per-tunnel sequence number (NE-PTSN) and the next expected global sequence number (NE-GSN)

A method of how packets may be buffered or forwarded to a target device after being received and decrypted will be described below.

1. If the GSN of the packet is equal to 0, it is immediately forwarded to the target device.

2. It is checked whether the packet's PTSN is equal to the NE-PTSN. If not, all packets having a GSN less than that of the packet are dequeued in order (forwarded to the target device). Keeping the packet unprocessed.

3. NE-PTSN is updated (i.e., NE-PTSN is set to PTSN + 1).

4. If the GSN is less than the NE-GSN, forwarding to the target device.

5. If the GSN of the packet is equal to NE-GSN, then NE-GSN is updated (i.e., set to GSN +1) and forwarded to the target device. If the GSN of the header is equal to the new NE-GSN, the NE-GSN is repeatedly updated and the buffered header is dequeued from the buffer.

6. Otherwise (GSN is greater than NE-GSN), enqueue the packet in order of GSN.

7. If the packet is in the queue longer than a fixed amount of time, NE-GSN is set to the GSN +1 of the packet and all packets having a GSN less than the GSN of the packet are dequeued in order.

Thus, the encapsulation packet information discussed in fig. 2 and 3 may include such information: this information optimizes the overall throughput of the data transmission system (e.g., system 100) by facilitating optimization of tunnel bandwidth and efficient ordering of packets received through the ancillary data transmission sessions in response to monitoring packet loss rates.

Fig. 5 illustrates an exemplary processor-based system 500 that may be used to implement systems, apparatus, and methods according to some embodiments. The processor-based system 500 may represent an architecture of the communication router 106 and the communication router 108. A Central Processing Unit (CPU)501 is coupled to the system bus 502. CPU 501 may be any general purpose CPU or may be a special purpose CPU designed to implement the teachings described above. The present disclosure is not limited by the architecture of CPU 501 (or other components of exemplary system 500) as long as CPU 501 (and other components of system 500) supports the inventive operations described herein. The CPU 501 may execute the various logical instructions described herein. For example, CPU 501 may execute machine-level instructions according to the exemplary operational flow described above in connection with FIG. 2. When executing instructions representing the operational steps illustrated in fig. 2, CPU 501 becomes a special-purpose processor of a special-purpose computing platform that is specially configured to operate in accordance with various embodiments of the teachings described herein.

The system 500 also includes Random Access Memory (RAM), which may be SRAM, DRAM, SDRAM, or the like. RAM 503 may be secondary memory that stores program instructions executable by CPU 501. The system 500 includes Read Only Memory (ROM)504, which may be PROM, EPROM, EEPROM, etc. The RAM 503 and ROM 504 store user and system data and programs, as is well known in the art.

System 500 also includes input/output (I/O) adapter 505, communications adapter 511, user interface adapter 508, and display adapter 509. In certain embodiments, I/O adapter 505, user interface adapter 508, and/or communications adapter 511 may enable a user to interact with system 500 to input information.

I/O adapter 505 connects storage devices 506 (such as one or more of a hard disk drive, Compact Disk (CD) drive, floppy disk drive, tape drive, etc.) to system 500. Storage devices are utilized in addition to the RAM 503 for the storage requirements associated with performing the operations discussed in the embodiments above. The communication adapter 511 is adapted to couple the system 500 to a network 512, which may enable input and/or output of information to and/or from the system 500 via such network 512 (e.g., the internet or other wide area network, a local area network, a public or private switched telephone network, a wireless network, any combination of the preceding).

The communication adapter 511 may be considered a network interface, and the system 500 may include a plurality of communication adapters 511. The user interface adapter 508 couples user input devices (e.g., keyboard 513, pointing device 507, and microphone 514) and/or output devices (e.g., speaker 515) to the system 500. The display adapter 509 is driven by the CPU 501 to control display on the display device 510. The display adapter 509 transmits instructions for converting or manipulating the state of various numbers of pixels used by the display device 510 to visually present desired information to a user. Such instructions include instructions for changing states from on to off, setting a particular color, intensity, duration, etc. Each such instruction constitutes a rendering instruction that controls how and what is displayed on the display device 510.

Fig. 6 is a flow diagram illustrating a process for transmitting encapsulated packets from communication router 106 to communication router 108 over an aggregated connection according to various embodiments of the invention. The aggregated connection may include a plurality of tunnels. In step 601, the cpu 501 of the communication router 106 selects a tunnel among a plurality of tunnels for transmitting an Original Encapsulation Packet (OEP). The selected tunnel may be referred to as a first tunnel. Subsequently, the communication router 106 transmits the OEP through the first tunnel. In step 602, the CPU 501 selects another tunnel among the plurality of tunnels for transmitting a Duplicate Encapsulation Packet (DEP). The tunnel selected in step 602 may be referred to as a second tunnel. Subsequently, the communication router 106 transmits the DEP through the second tunnel.

In one variation, the process of FIG. 6 may be performed periodically. This process may preferably be performed every few seconds. Instead, the process of FIG. 6 is performed upon receipt of a trigger. For example, the trigger is received when the CPU 501 determines that no acknowledgement is received or an acknowledgement is delayed from being received through the first tunnel and/or the second tunnel. This may indicate that the delay of the first tunnel and/or the second tunnel has become very high. Subsequently, the CPU 501 executes the steps of fig. 6 again to select the first tunnel and the second tunnel again. In another example, the trigger may be received when a user or administrator manually initiates the process of FIG. 6.

The first tunnel and the second tunnel may be selected according to a selection policy. In one variation, the selection policy may be based on one or more of the following criteria: performance of tunnels, service providers, usage restrictions, location, time, usage price, security, user identity, internet protocol address range, communication protocols, communication technologies, applications and devices. When the selection policy is based on the performance of the tunnel, the selection may be performed according to performance metrics such as throughput, error rate, packet delay, packet jitter, symbol jitter, quality of service, bandwidth, bit error rate, packet error rate, frame error rate, packet loss rate, queuing delay, cycle time, capacity, signal level, interference level, bandwidth delay product, handover delay time, signal-to-interface ratio, and signal-to-noise ratio.

The user or administrator may configure the selection policy. For example, the user may configure the selection policy to be based only on the bandwidth of each of the plurality of tunnels. Therefore, the tunnel having the best bandwidth is selected as the first tunnel through which the OEP is transmitted. Another tunnel with a second best bandwidth is selected as the second tunnel through which the DEP is transmitted. In another embodiment, the user may configure the selection policy to be based only on the delay of each of the plurality of tunnels. Therefore, the tunnel with the lowest delay is selected as the first tunnel through which the OEP is transmitted. Another tunnel with the next lowest latency is selected as the second tunnel through which the DEP is transmitted.

In another example, the user may configure the selection policy to be based on usage restrictions and packet loss rates. The tunnel with the highest usage limit and/or lowest packet loss rate is selected as the first tunnel. And selecting the tunnel with the next highest use limit and/or the next lowest packet loss rate as the second tunnel.

In another variation, the selection policy is based on user selection. A user or administrator may configure the communication router 106 to use a certain tunnel as a first tunnel and another certain tunnel as a second tunnel.

A user or administrator may configure the selection policy of the communication router 106 by sending the configuration locally or remotely through a web interface, Application Program Interface (API), command line interface, or console.

Those skilled in the art will appreciate that in wireless communications, packet transmission quality may be unpredictable and that delay may be high in some cases. The packet loss rate may also be higher. Transmitting the DEP increases the chances that the communication router 108 will receive the data because the OEP may be lost. However, bandwidth usage can be significantly improved by the transmission of DEPs. DEP may also be used for Forward Error Correction (FEC), as will be explained in more detail below.

Referring to fig. 8A, 8B and 8C, the types of information contained in the OEP800 and the DEPs 810 and 820 may be similar to those shown in fig. 3.

The encapsulation packet encapsulates the data packet. Therefore, when the data packet is encapsulated by the encapsulation packet, the data packet becomes the encapsulation packet. The data packets may be received by the communication router 106 through one or more of its network interfaces from hosts or nodes in the site 102 accessible through the communication router 106. The host or node may be located in the LAN of the communication router 106. The source and destination of the data packet may be indicated in the header of the data packet.

Each encapsulation packet may be assigned two sequence numbers, namely a global sequence number and each tunnel sequence number. The global sequence number may be used to assist in packet buffering and reordering operations. When the encapsulation packet is assigned with the global sequence number, the designated host or node arranges the encapsulation packet arriving at the designated host or node according to the global sequence number corresponding to the encapsulation packet. Each tunnel sequence number indicates through which of a plurality of tunnels the encapsulation packet is transmitted.

Fig. 8A illustrates a structure of an OEP according to various embodiments. OEP800 includes an IP header 801, an original encapsulation packet global sequence number (OEP-GSN)802, a per tunnel sequence number 803, other information 804, and an encapsulated packet 805. OEP-GSN 802 holds the OEP-GSN of OEP 800. The IP header 801 may include a version field 302 and a protocol type field 303. Each tunnel sequence number 803 may be optional. In one variation, each tunnel sequence number 803 and tunnel ID field 304 may be used to compare tunnel performance and reallocate packets according to tunnel performance. In another variation, when each tunnel sequence number 803 and tunnel ID field 304 are not included, a database may need to be maintained to store information of which tunnel is used to transmit encapsulation packets of which GSN. By performing a lookup on the database, the tunnel performance can be compared. Thus, the inclusion of each tunnel sequence number 803 and tunnel ID field 304 may assist in determining latency without the need to maintain and look up a database of all GSNs. Other information 804 may include AES initialization vector field 306 and tunnel ID field 304. In one variation, the OEP-GSN 802, per-tunnel sequence number 803, and other information 804 are included in an option field of the IP header 801, and the encapsulated packet 805 is included in the AES encrypted payload field 312. The benefit of including the OEP-GSN 802, per-tunnel sequence number 803 and other information 804 in the selection field of the IP header 801 is: information can be obtained in these fields without decapsulating OEP800, which can significantly reduce processing time. However, when the OEP800 is tunneled, since routers and devices through which the OEP800 passes may change the IP header 801, if they are included in the selection fields of the IP header 801, information in these fields may be lost. Thus, instead, the OEP-GSN 802, per-tunnel sequence number 803, other information 804, and encapsulated packet 805 may all be included in the AES encrypted payload field 312 so that information is not lost when the OEP800 passes through routers and devices that may change the IP header 801.

FIG. 8B illustrates the structure of a DEP in accordance with one of the embodiments of the present invention. The DEP810 includes an IP header 811, a duplicate encapsulated packet global sequence number (DEP-GSN)812, per-tunnel sequence numbers 813, other information 814, and encapsulated packets 815. The IP header 811 may include a version field 302 and a protocol type field 303. Each tunnel sequence number 813 and other information 814 may be optional. Other information 814 may include AES initialization vector field 306 and tunnel ID field 304. In one variation, the DEP-GSN 812, per-tunnel sequence number 813, and other information 814 are included in an option field of the IP header 811, and the encapsulated packet 815 is included in the AES encrypted payload field 312. Instead, the DEP-GSN 812, each tunnel sequence number 813, other information 814, and the encapsulated packet 815 are included in the AES encrypted payload field 312. Since DEP810 comprises the same encapsulated packets as OEP800, DEP810 can be used to recreate information in OEP800 in the event that DEP 800 is lost, or delayed during transmission, and DEP810 is received before receiving OEP 800.

OEP-GSN 802 and DEP-GSN 812 may be identical. The encapsulated

packets

805 and 815 may be identical, such that the OEPs 800 and DEPs 810 encapsulate the same data packet, which becomes the encapsulated

packets

805 and 815, respectively. When communications router 106 generates DEP810 for transmission, its payload (i.e., the encapsulated packet) is the same as the payload of OEP 800. Since OEP800 and DEP810 have the same content, their global sequence numbers are also the same.

FIG. 8C illustrates the structure of a DEP in accordance with one of the embodiments of the present invention. The DEP 820 includes an IP header 821, DEP-GSN 822, per tunnel sequence number 823, list 826 of m OEP-GSNs, other information 824, and encapsulated packet 825. Because the payload of the DEP 820 contains information of the last few OEPs that have been transmitted over the aggregated connection, it can be used to recreate the missing encapsulated packets corresponding to the OEPs. This is illustrated in more detail in fig. 12A, 12B and 12C.

The DEP 820 may be transmitted periodically. In one variation, the time period between transmission of each DEP 820 may be predefined. When the number of OEPs transmitted in the time period is n, the DEP 820 may include m encapsulated packets corresponding to the n OEPs, where m is less than or equal to n, and the value of m is at least 1. The m encapsulated packets are identical to the encapsulated packets in the m OEPs of the n OEPs. M encapsulated packets, such as encapsulated packets 825-1, 825-2 through 825-m, may be included in the AES encrypted payload field 312 of the DEP 820. The list 826 of m OEP-GSNs includes OEP-GSNs for OEPs corresponding to the m encapsulated packets. For example, 10 OEPs are transmitted in a predefined time period. The first DEP 820 may comprise 6 encapsulated packets corresponding to 6 of the 10 OEPs, and the second DEP 820 may comprise 4 encapsulated packets corresponding to the remaining 4 OEPs. After a predefined period of time, the first DEP 820 and the second DEP 820 are transmitted. The first DEP 820 and the second DEP 820 may be transmitted through the same tunnel or different tunnels. Alternatively, the first DEP 820 may comprise 10 encapsulated packets corresponding to 10 OEPs, respectively. After a predefined period of time, the first DEP 820 is transmitted. The DEP 820 cannot include more than 10 encapsulated packets because each encapsulated packet must correspond to each OEP.

In another variation, the number of OEPs before transmission of one DEP 820 is predefined. Thus, the predefined number is n. The DEP 820 may include m encapsulated packets corresponding to n OEPs, where m is less than or equal to n. The m encapsulated packets may be the same as the encapsulated packets in the m OEPs of the n OEPs. M encapsulated packets, such as encapsulated packets 825-1, 825-2 through 825-m, may be included in the AES encrypted payload field 312 of the DEP 820. For example, when n is defined as 5, the DEP 820 is transmitted after every 5 OEPs are transmitted, and the DEP 820 includes 5 encapsulated packets corresponding to the 5 OEPs. The time period between transmission of the DEPs 820 may not be predefined. The list 826 of m OEP-GSNs includes 5 OEP's OEP-GSNs corresponding to the encapsulated packet. Alternatively, when n is defined as 5, the first DEP 820 and the second DEP 820 are transmitted after every 5 OEPs are transmitted. The first DEP 820 may comprise encapsulated packets corresponding to 2 of the 5 OEPs, and the second DEP 820 may comprise encapsulated packets corresponding to the remaining 3 of the 5 OEPs.

DEP 820 can be used to correct any errors that may have occurred in the transmission of n OEPs because encapsulated packet 825 contains information from the data packets encapsulated in the OEPs. In one variation, the DEP-GSN 822, per-tunnel sequence number 823, list 826 of m OEP-GSNs, and other information 824 may be included in an option field of the IP header 821 and the encapsulated packet 825 included in the AES encrypted payload field 312. Instead, the DEP-GSN 822, each tunnel sequence number 823, a list 826 of m OEP-GSNs, other information 824, and the encapsulated packet 825 are included in the AES encrypted payload field 312. It is possible that one or more of the OEPs transmitted over the aggregated connection have been lost, missed, or delayed. Even when an OEP is received, there may be errors in the OEP. The DEP 820 can be used to recreate an encapsulated packet in an OEP and also to check for any errors. Details regarding the transmission of the DEP 820 and the re-creation of the encapsulated packets are shown in fig. 12A, 12B, and 12C.

OEP800 is transported through a different tunnel than the tunnel used to transport DEP810 or DEP 820. Thus, each tunnel sequence number 803 may be different from each tunnel sequence number 813 or each tunnel sequence number 823. If the OEP800 and DEP are transmitted through the same tunnel and the performance of the tunnel is degraded, it is possible that neither the OEP800 nor the DEP may be successfully received by the communication router 108. Accordingly, to increase the chances of at least one of the OEP800 or DEP being received, the OEP800 is transmitted through a different tunnel than the tunnel used to transmit the DEP810 or DEP 820.

The source address indicated in the

IP headers

801, 811, and 812 may be the IP address of one of the network interfaces of the communication router 106. The destination address indicated in the

IP headers

801, 811, and 812 may be an IP address of one of the network interfaces of the communications router 108. The source address of the encapsulated

packets

805, 815, and 825 may be the IP address of a host or node in the site 102 that is accessible through the communications router 106. The destination address of the encapsulated

packets

805, 815, and 825 may be the IP address of a host or node in the station 104 accessible through the communications router 108.

OEP-GSN 802, DEP-GSN 812, and DEP-GSN 822 may correspond to global sequence number field 308. Each

tunnel sequence number

803, 813, and 823 may correspond to each tunnel sequence number field 310.

Because the encapsulation packets may arrive out of order, one of the purposes of encapsulating the data packets within the encapsulation packets is to reorder the data packets as they are received at the other end of the aggregate connection. The data packets may also have different protocols and may be encapsulated in an encapsulation packet to meet the protocol requirements of the aggregated connection.

FIG. 9 is a flow diagram illustrating a process for transmitting DEPs according to one of the embodiments of the invention. The CPU 501 of the communication router 106 determines the number of DEPs to be transmitted in step 901. In step 902, the CPU 501 determines which tunnel(s) has not been used to transmit the OEP or DEP, and selects the determined tunnel(s) for transmitting the DEP. In step 903, the CPU 501 determines whether the bandwidth limit of the selected tunnel is approached, and the communication router 106 does not transmit the DEP through the tunnel whose bandwidth limit is being approached in step 904. If the bandwidth limits are not close, the communications router 106 transmits the DEP through the selected tunnel in step 905. In step 906, the process ends. Monitoring when bandwidth limits are approached may help reduce congestion and prevent tunnel flooding. Those skilled in the art will appreciate that as the bandwidth limit of the tunnel approaches, more and more packets may be lost and the performance of the tunnel degrades. Therefore, it is preferable to stop transmitting DEPs through tunnels that are approaching their bandwidth limits.

According to one embodiment of the invention, the communications router 106 may transmit multiple DEPs for one OEP. Each of the plurality of DEPs is transported through a different tunnel. The number of DEPs to be transmitted may be defined by a user or administrator of the communication router 106. However, the number of DEPs transmitted may not be higher than the number of tunnels available for transmitting DEPs. This is because tunneling multiple DEPs for each OEP may not be beneficial. When the performance of the tunnel degrades, most of the plurality of DEPs may not be successfully received. Thus, this may consume more bandwidth unnecessarily without increasing the chance of successfully receiving at least one of the DEPs. When a user or administrator defines the number of DEPs to be higher than the number of tunnels available for transmitting DEPs, the communication router 106 does not transmit the number of DEPs defined by the user or administrator. Instead, the communications router 106 transmits DEPs equal to the number of available tunnels, i.e., one DEP per tunnel. At least one tunnel may be reserved for transmitting OEP and the remaining tunnels may be used for transmitting DEP. For illustration purposes, the user defines the number of DEPs to be transmitted as 5. The communication router 106 has only 4 tunnels established with the communication router 108, i.e., a first tunnel, a second tunnel, a third tunnel, and a fourth tunnel. Each OEP is transmitted through the first tunnel. One DEP corresponding to each OEP is transmitted through each of the second, third, and fourth tunnels, respectively. Thus, only 3 DEPs are transmitted for each OEP, not 5.

In one example, if a user or administrator defines the number of DEPs to be less than the number of available tunnels, the DEPs may be transmitted according to the performance of the tunnels. For illustration purposes, the user defines the number of DEPs to be transmitted as 2. The communication router 106 has 4 tunnels established with the communication router 108, i.e., a first tunnel, a second tunnel, a third tunnel, and a fourth tunnel. The OEP is transmitted through the first tunnel with the best performance. If the second tunnel and the third tunnel have better performance than the fourth tunnel, one DEP is transported through the second tunnel and another DEP is transported through the third tunnel.

In another example, when there are more than two available tunnels, the OEP is transmitted through the first tunnel with the best performance. The DEP is transported through the remaining tunnels using load balancing techniques. Alternatively, the DEPs may also be transmitted through the remaining tunnels in a round-robin fashion, particularly when the number of DEPs to be transmitted is less than the number of remaining tunnels.

Fig. 7 is a flowchart illustrating a process for processing a received encapsulation packet. First, the processing environment that exists is one that: an aggregated connection including a plurality of tunnels is established between the communication router 106 and the communication router 108, and the communication router 106 sends the encapsulation packet to the communication router 108 through the aggregated connection. The communication router 108 receives the encapsulated packet from the communication router 106 over the aggregated connection. The encapsulated packets may include encapsulated packets destined for a host or node accessible through the communication router 108.

The communication router 108 may receive a plurality of encapsulated packets encapsulated in an OEP and/or DEP. OEP may arrive earlier than DEP and vice versa. Therefore, it is necessary to know whether an encapsulated packet, whether it is encapsulated in an OEP or a DEP, has been received before.

In step 701, communication router 108 receives an encapsulation packet from communication router 106. In step 702, the CPU of the communication router 108 determines whether an encapsulated packet encapsulated in an encapsulated packet has been previously received. If the global sequence number of the encapsulating packet is the same as another global sequence number of another encapsulating packet that has been received before, it is determined that the encapsulated packet has been received before. If an encapsulated packet has been received before, in step 703, communications router 108 does not forward the encapsulated packet inside the encapsulated packet to the destination and discards the encapsulated packet. If the encapsulated packet has not been received before, the encapsulated packet is forwarded to the destination in step 704. In step 705, the process ends. When it is determined that an encapsulated packet has been previously received, then it may be assumed that the encapsulated packet has been forwarded. For this reason, the communication router 108 does not forward the encapsulated packet in the encapsulated packet again, and discards the encapsulated packet.

The encapsulation packet may be an OEP800 or a DEP 810. Referring to step 702, there may be various methods of determining whether an encapsulated packet has been received. As described above, one way to determine this is by examining the GSN of the encapsulated packet. The communication router 108 determines that an encapsulated packet has been received if any other encapsulated packet with the same global sequence number has been previously received over the same aggregated connection. The same encapsulated packet need not then be forwarded.

Another way to determine whether an encapsulated packet has been received is by checking a hash code. When the encapsulated packet arrives at communication router 108, the CPU of communication router 108 may apply a hash function to the payload (i.e., the encapsulated packet) after decapsulating the encapsulated packet. The hash code generated by applying the hash function may be stored in a storage medium. The storage medium may be a local storage medium (e.g., RAM 503) or a remote storage medium (e.g., a remote server). When the hash function is applied to the same encapsulated packet that has been received before, the hash code will be the same. The CPU of the communications router 108 may then determine whether the generated hash code is unique or whether it has been previously stored. If the hash code is unique, it is determined that the encapsulated packet has not been received previously. If the hash code has been previously stored, it is determined that the encapsulated packet has been received. While the hash function or seed may change, it is more likely to remain unchanged for several minutes. Thus, the uniqueness of the hash code can be used to determine whether an encapsulated packet has been received in the past few minutes.

Fig. 10 is a flowchart showing processing performed at the communication router 108 when the communication router 108 receives an encapsulated packet from the communication router 106. The encapsulated packet is assigned a global sequence number and is then transmitted by the communication router 106 to the communication router 108 over the aggregated connection. The encapsulation packet may be an OEP800 or a DEP 810.

In step 1000, communications router 108 receives an encapsulated packet assigned a first global sequence number (GSN-1). In step 1001, the CPU of the communications router 108 determines an expected global sequence number (E-GSN). Subsequently, in step 1002, GSN-1 is compared to E-GSN. The E-GSN may be used to estimate what the GSN for the next encapsulation packet should be.

If it is determined in step 1002 that the E-GSN is greater than GSN-1, the encapsulated packet is considered to have delayed arrival. In step 1003, the CPU of communications router 108 determines whether GSN-1 is identified in the record of the missing GSN. If GSN-1 is identified in the record of the missing GSN, then in step 1005, the record of the missing GSN is updated to remove GSN-1 from the record of the missing GSN. Subsequently, communications router 108 decapsulates the encapsulated packet to retrieve the encapsulated packet, and forwards the encapsulated packet to its destination in step 1009. The destination may be a host or node in site 104 and may be accessed through communications router 108. Alternatively, if GSN-1 is not identified in the record of the missing GSN, then in step 1004, the encapsulated packet is discarded and not forwarded to the destination. The encapsulated packet is lost because it is considered to contain encapsulated packets that have been received before. Since GSN-1 is not found in the record of the missing GSN, this indicates that another encapsulated packet with the same GSN-1 has been received before and may have been forwarded to the destination.

If, in step 1002, the E-GSN is not determined to be greater than GSN-1, then, in step 1006, the CPU of communications router 108 determines whether the E-GSN is equal to GSN-1. If the E-GSN is equal to GSN-1, then in step 1007 the E-GSN may be increased if necessary. The encapsulated packet is decapsulated to retrieve the encapsulated packet, which is then forwarded to the destination in step 1009. If the E-GSN is not equal to GSN-1 (i.e., the E-GSN is less than GSN-1), the encapsulation packet is considered to have arrived early. Thus, in step 1008, the encapsulation packet is stored in the buffer for a predefined period of time. When the predefined time period has expired, the encapsulated packet may then be removed from the buffer and decapsulated to retrieve the encapsulated packet. Subsequently, in step 1009, the encapsulated packet is forwarded to the destination. Alternatively, when the E-GSN has become equal to GSN-1, then the encapsulated packet may be removed from the buffer and de-encapsulated. The encapsulated packet is then forwarded to the destination in step 1009. The E-GSN may become equal to GSN-1 because the E-GSN is incremented by the CPU of the communications router 108 based on the GSN of the received encapsulated packet. In one variation, the encapsulated packet may first be decapsulated to retrieve the encapsulated packet, and then stored in a buffer at step 1008. When the predefined time period has expired or when the E-GSN becomes equal to GSN-1, then the encapsulated packet may be removed from the buffer and forwarded to the destination. In step 1010, the process ends.

The benefits of storing the encapsulated packets in the buffer are: the encapsulated packet need not be decapsulated prior to storage, thereby saving computational resources for decapsulation. However, encapsulating packets may consume more space in the buffer than encapsulated packets, which may not be satisfactory. When encapsulated packets are stored in the buffer, a table indicating the corresponding GSN is also stored with the encapsulated packets.

The E-GSN may be increased even when several encapsulated packets with smaller global sequence numbers may be missing (i.e., delayed, lost, or missed). The E-GSN continues to increase because the buffer space for storing the encapsulated packets is limited and the encapsulated packets should be removed from the buffer once their GSN becomes equal to the E-GSN.

Detailed processing corresponding to

steps

1007 and 1008 and a METHOD FOR calculating E-GSN is disclosed in the publication entitled "METHOD AND SYSTEM FOR redundancy OF TIME VARIANCE OF PACKETS RECEIVED FROM bundled COMMUNICATIONs LINKS" ("METHOD and system FOR reducing time variance OF packets received FROM BONDED COMMUNICATIONs LINKS") published on 11.4.2013, international publication No. W02013/049960 Al. In one variation, step 1008 may be performed in accordance with the process of FIG. 13.

The communication router 108 stores the GSN of the encapsulated packets received over the aggregated connection. The missing GSN may be recorded in a record of the missing GSN. For example, communication router 108 has received 5 encapsulated packets with GSN 01, GSN 02, GSN 03, GSN 05, and GSN 06 within a time period. A packet loss with GSN 04 is determined. Thus, GSN 04 is recorded in the record of the missing GSN.

In one variation, communication router 108 stores the GSN of each encapsulated packet whose corresponding encapsulated packet has been forwarded to the target host or node. The GSN may be recorded in a record of the forwarded GSN. The encapsulation packet may be an OEP800 or a DEP 810. When determining whether to forward an encapsulated packet corresponding to the encapsulated packet, communication router 108 checks the record of the forwarded GSN. If the GSN encapsulating the packet is found in the record of forwarded GSNs, the encapsulated packet is not forwarded to the target host or node because it has been forwarded previously.

Fig. 11 is a flow chart illustrating a process of processing encapsulated packets received over an aggregated connection. For example, the communication router 108 receives the encapsulation packet from the communication router 106 over an aggregated connection that includes a plurality of tunnels. The encapsulated packets may include one or more encapsulated packets destined for a host or node accessible through the communication router 108.

In step 1100, the communications router 108 receives an encapsulation packet. In step 1101, the CPU of the communication router 106 determines whether the encapsulation packet includes a list of OEP-GSNs. If the encapsulation packet includes the list of OEP-GSNs, it may be the DEP 820, and if the encapsulation packet does not include the list of OEP-GSNs, it may be the OEP 800.

When it is determined that the encapsulating packet is the OEP800, the CPU of the communication router 108 determines whether the GSN (i.e., the OEP-GSN 802) of the encapsulating packet is recorded in the record of the missing GSN in step 1110. If the GSN is recorded in the record of the missing GSN, then in step 1112, the encapsulated packet encapsulated in an encapsulation packet (e.g., encapsulated packet 805) is forwarded to the target host or node. If the GSN is not recorded in the record of the missing GSN, then it is determined that the encapsulated packet has been previously received and that the encapsulated packet has been forwarded, and therefore, in step 1111, the encapsulated packet is no longer forwarded to the destination.

Upon determining that the encapsulation packet is the DEP 820, in step 1120, the CPU of the communications router 108 determines whether at least one OEP-GSN in the list 826 of m OEP-GSNs is recorded in the record of the missing GSN. If not, in step 1121, the encapsulated packet 825 is not forwarded to the target host or node. If at least one OEP-GSN is found in the record of the missing GSN, then in step 1122, the communication router 106 forwards the encapsulated packet corresponding to the at least one OEP-GSN in the list of m OEP-GSNs recorded in the record of the missing GSN. For example, if an OEP-GSN is found in the record of the missing GSN that corresponds only to encapsulated packets 825-2 and 825-3, then in step 1122, only encapsulated packets 825-2 and 825-3 are forwarded to the destination, while the remaining encapsulated packets are not forwarded to the destination. In step 1130, the process ends.

According to one embodiment, when the communication router 108 receives an encapsulation packet, it may determine whether the encapsulation packet is an OEP or a DEP by checking the tunnel ID. The communication router 106 can notify the communication router 108 to select a first tunnel for transmitting the OEP and a second tunnel for transmitting the DEP. Thus, the communications router 108 may determine that any packets received through the first tunnel are OEPs and any packets received through the second tunnel are DEPs. However, in some cases, the second tunnel may also be used to transmit other packets, such as management packets, health check packets, and the like. In this case, the encapsulation packet may have an additional field indicating whether the encapsulation packet is an OEP or a DEP. Alternatively, the information indicating whether the encapsulation packet is an OEP or a DEP may be included in other information fields of the encapsulation packet. Instead of performing step 1101 to determine whether the encapsulation packet is an OEP or a DEP, checking the tunnel ID or checking additional fields may be performed.

FIG. 12A shows the contents of a DEP 820 created based on n OEPs in accordance with an exemplary embodiment. For ease of viewing, the IP header, each tunnel sequence number, and other information are not shown in OEP 800-1, OEP 800-2, OEP 800-3, OEP 800-4, and DEP 820-1. Although the IP header, per-tunnel sequence number, and other information are not shown, each of the OEPs and DEPs contains those fields.

Communication router 106 first receives packet 1, packet 2, packet 3, and packet 4 from a host or node in station 102. These data packets are encapsulated in the OEP. Data packet 1 becomes encapsulated packet 1805-1 encapsulated in OEP 800-1. Data packet 2 becomes encapsulated packet 2805-2 encapsulated in OEP 800-2. Data packet 3 becomes encapsulated packet 3805-3 encapsulated in OEP 800-3. Data packet 4 becomes encapsulated packet 4805-4 encapsulated in OEP 800-4. The fields OEP-GSN 802-1, OEP-GSN 802-2, OEP-GSN 802-3 and OEP-GSN 802-4 include OEP-GSN 1, OEP-GSN 2, OEP-GSN 3 and OEP-GSN 4, respectively. The communication router 106 transmits the OEP 800-1, OEP 800-2, OEP 800-3, and OEP 800-4 to the communication router 108 through the first tunnel of the aggregated connection. OEP 800-1, OEP 800-2, OEP 800-3 and OEP 800-4 are the n OEPs transmitted and their GSNs are OEP-GSN 1, OEP-GSN 2, OEP-GSN 3 and OEP-GSN 4, respectively. The communication router 106 then transmits the DEP 820-1 to the communication router 108 through a second tunnel for the four OEPs 800-1, 800-2, 800-3 and 800-4. The list 827 of m OEP-GSNs includes OEP-GSN 1, OEP-GSN 2, OEP-GSN 3, and OEP-GSN 4. As shown in fig. 12B and 12C, information (packet information) 1225 is based on the encapsulated packets 1 to 4 of the OEP-GSN. One of the purposes of transmitting DEP 820-1 is to recover any information that may be lost during transmission of OEP. If any of the four OEPs are not received by the communications router 108, or there is an error in any of the four OEPs, the DEP 820-1 may be used to recreate any of the four OEPs because it contains information from the four OEPs. Information 1225 may or may not contain encapsulated packets 1 through 4. However, information 1225 may be used to recreate the encapsulated packages 1-4.

FIG. 12B shows the contents of DEPs with respect to the four OEPs of FIG. 12A, in accordance with one of the embodiments. For ease of viewing, the IP header, per-tunnel sequence number, and other information are not shown in the DEP 820-2. The DEP 820-2 may be transmitted to the communications router 108 through a second tunnel for the four OEPs 800-1, 800-2, 800-3, and 800-4. In DEP 820-2, information 1225 includes encapsulated packets 1805-1, encapsulated packets 2805-2, encapsulated packets 3805-3, and encapsulated packets 4805-4. In this way, information 1225 is based on encapsulated packets 1 through 4, and thus, encapsulated packets 1 through 4 may be recreated by communications router 108 using information 1225 in DEP 820-2.

FIG. 12C shows the contents of DEPs with respect to the four OEPs of FIG. 12A, according to an alternative embodiment. For ease of viewing, the IP header, per-tunnel sequence number, and other information are not shown in the DEP 820-3. The DEP 820-3 may be transmitted to the communications router 108 through a second tunnel for the four OEPs 800-1, 800-2, 800-3, and 800-4. In DEP 820-3, information 1225 contains information 805-12 and information 805-34. Information 805-12 and information 805-34 may be generated by applying operations to the encapsulated packets 1 through 4. For example, the operation is an exclusive or (XOR) operation. Information 805-12 is generated by applying an XOR operation on encapsulated packet 1 and encapsulated packet 2. Thus, information 805-12 may be used to recreate encapsulated package 1 and/or encapsulated package 2. Information 805-34 is generated by applying an XOR operation on encapsulated packet 3 and encapsulated packet 4. Thus, information 805-34 may be used to recreate encapsulated package 3 and/or encapsulated package 4. When the communications router 108 receives the DEP 820-3, it can use the information 805-12 and the information 805-34 to recreate any missing encapsulated packets 1 through 4 or correct any errors in any of the encapsulated packets 1 through 4. Different combinations of the encapsulated packets 1 through 4 may be used together to generate other types of information in the information 1225 of the DEP 820-3 by performing XOR operations. The scope of the invention is not limited to operation as an XOR operation, such that it can be any operation suitable for performing forward error correction. One of the benefits of transmitting the DEP 820-3 instead of the DEP 820-2 is that the transmission of the DEP 820-3 may consume less bandwidth due to reduced overhead.

FIG. 13 is a flow diagram illustrating a process for processing a DEP (e.g., DEP 820-3) received at the communications router 108. In step 1301, the communication router 108 receives the DEP 820-3 over the aggregated connection. In step 1302, the CPU of the communication router 108 determines the OEP-GSN in the list 827 of OEP-GSNs. In step 1303, the CPU of the communication router 108 determines whether an OEP corresponding to at least one of the OEP-GSNs in the list 827 of OEP-GSNs has been received. More precisely, in step 1303, it is determined whether at least one of OEP 800-1, OEP 800-2, OEP 800-3, and OEP 800-4 has been received. If none of OEP 800-1, OEP 800-2, OEP 800-3, and OEP 800-4 are received, then in step 1304, information 805-12 and information 805-34 may be stored in a buffer for later processing. If at least one of OEP 800-1, OEP 800-2, OEP 800-3 and OEP 800-4 has been received and after the delay in step 1309, if it is determined in step 1305 that all of OEP 800-1, OEP 800-2, OEP 800-3 and OEP 800-4 have been received, then DEP 820-3 is discarded in step 1308. If at least one but not all of the OEPs 800-1, 800-2, 800-3, 800-4 have been received, then in step 1306 the communications router 108 may use the stored information 805-12 and/or information 805-34 to recreate one or more data packets corresponding to one or more OEPs that were not previously received, if possible. In step 1307, the recreated packet may be forwarded to its destination.

A delay is introduced between step 1303 and step 1305 because all OEPs may take some time to reach the communication router 108. Therefore, after receiving one of the OEPs, all the OEPs may not be received immediately.

For purposes of illustration, if it is determined in step 1303 that OEP 800-1 was previously received, but OEP 800-2 was not received, then in step 1306, packet 2 may be recreated using information 805-12. This is possible because both information 805-12 and packet 1 are available and there is enough information to create packet 2. If both OEP 800-3 and OEP 800-4 have not been previously received, then in step 1306, DEP 820-3 and information 805-34 may not be used to recreate packet 3 or packet 4 because there is not enough information even if OEP 800-1 and/or OEP 800-2 have been previously received.

In another example, when the information included in the DEP 820-3 is generated by performing an XOR operation on three data packets and an OEP corresponding to only one of the three data packets is previously received, it may not be possible to recreate the other two data packets.

In one variation, in step 1304, the entire DEP 820-3 is stored in the buffer instead of only the information 805-12 and the information 805-34. An advantage of storing the entire DEP 820-3 is that no decapsulation is required before storing the DEP 820-3. If the DEP 820-3 is eventually discarded, computational resources for decapsulation may be saved. However, the DEP 820-3 may consume more space in the buffer than the information 805-12 and the information 805-34, which may not be satisfactory.

In one variation, the communication router 108 stores the packet in a buffer for a predefined period of time before forwarding the packet at step 1307. When the predefined time period expires, then the packet is forwarded to the destination. Alternatively, the packet may be stored in a buffer until the E-GSN becomes equal to the OEP-GSN corresponding to the packet, and then the packet is forwarded to the destination.

In one example, the communications router 106 establishes an aggregated connection with the communications router 108 that includes three tunnels (i.e., a first tunnel, a second tunnel, and a third tunnel). A first tunnel is selected to transmit OEP and a second tunnel and a third tunnel are selected to transmit DEP. A user or administrator may configure the communications router 106 to transmit one type of DEP810 and DEP 820, or both types of DEP810 and DEP 820. When the communications router 106 is configured to transmit DEPs 810, then at least one DEP810 is transmitted for each OEP 800. As discussed above, the number of DEPs 810 to be transmitted corresponding to each OEP800 may be configured.

When the communication router 106 is configured to transmit a DEP 820, one DEP 820 may be transmitted for n OEPs 800. The number n can be configured. For purposes of illustration, n is 5, and the communications router 106 is configured to: one DEP 820 is transmitted for every 5 transmitted OEPs 800 through the first tunnel. Thus, the communication router 106 transmits one DEP 820 through the second tunnel or the third tunnel after transmitting 5 OEPs 800 through the first tunnel. If the first, second, third, fourth and

fifth OEPs

800, 800 and 800 are transmitted and the DEP 820 includes information corresponding to the encapsulated packets of the first, second, third, fourth and fifth OEPs 800, the list 826 of m OEP-GSNs includes the OEP-GSN of the first OEP800, the OEP-GSN of the second OEP800, the OEP-GSN of the third OEP800, the OEP-GSN of the fourth OEP800 and the OEP-GSN of the fifth OEP 800. DEP-GSN 822 is not identical to OEP-GSN 802 of any OEP 800. The encapsulated packets 825 include encapsulated packets 825-1, encapsulated packets 825-2, encapsulated packets 825-3, encapsulated packets 825-4, and encapsulated packets 825-5 corresponding to the first OEP800, the second OEP800, the third OEP800, the fourth OEP800, and the fifth OEP800, respectively.

When the communications router 106 is configured to transmit both the DEP810 and the DEP 820, it may or may not use the same tunnel for transmission. For purposes of illustration, the communication router 106 is configured to transmit two DEPs 810 per OEP800 and one DEP 820 per 5 OEPs 800. Two DEPs 810 corresponding to the same OEP800 do not transmit through the same tunnel. The first DEP810 corresponding to each OEP800 may be transported through a second tunnel, and the second DEP810 corresponding to each OEP800 may be transported through a third tunnel. The DEP 820 may be transported through the second tunnel or the third tunnel. The DEP 820 may be transmitted through the second tunnel and the third tunnel in a polling manner. Alternatively, when the communication router 106 is configured not to transmit

DEPs

810 and 820 through the same tunnel, it may select a second tunnel to transmit one DEP810 for each OEP800 and a third tunnel to transmit one DEP 820 for each 5 OEPs 800.

In some cases, it may be preferable not to transmit DEP, as it may have a detrimental effect on the transmission of OEP. For example, transmitting the DEP through the second tunnel may extend the delay experienced by the OEP through the first tunnel when the first tunnel and the second tunnel are established using the same network interface of the communication device 106 and/or using the same network interface of the communication device 108. The throughput of OEP can also be reduced. Similar effects may be observed when the first tunnel and the second tunnel are established over the same WAN in site 102 and/or the same WAN in site 104. Similar effects may also be observed when the first tunnel and the second tunnel are established using networks provided by the same operator, since the base stations to which the tunnels are connected may be the same. Furthermore, the computational resources consumed to generate and transmit DEP may slow the transmission of OEP. Therefore, in these cases, it may be preferable not to transmit DEP or not to transmit a large number of DEP corresponding to each OEP.

The above-described process may also be applied to encapsulate packets transmitted from communication router 108 to communication router 106.

It should be understood that the present disclosure is not limited to the architecture of system 500. For example, any suitable processor-based device may be used to implement the teachings above, including but not limited to routers, personal computers, laptops, computer workstations, multi-processor servers, and even mobile phones. Furthermore, some embodiments may be implemented on an Application Specific Integrated Circuit (ASIC) or a very large scale integrated circuit (VLSI). Indeed, one of ordinary skill in the art may utilize any number of suitable structures capable of performing logical operations in accordance with the embodiments.

Although embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, modifications and substitutions may be made herein without departing from the spirit and scope of the invention as defined by the appended claims. Moreover, the scope of the present application is not intended to be limited to the particular embodiments of the process, machine, manufacture, composition of matter, means, methods and steps described in the specification. As one of ordinary skill in the art will readily appreciate from the disclosure of the present invention, processes, machines, manufacture, compositions of matter, means, methods, or steps, presently existing or later to be developed that perform substantially the same function or achieve substantially the same result as the corresponding embodiments described herein may be utilized according to the present invention. Accordingly, the appended claims are intended to include within their scope such processes, machines, manufacture, compositions of matter, means, methods, or steps.

Claims

1. A method for forwarding data packets at a first network node through a plurality of encrypted tunnels, comprising the steps of:

a. selecting a first encryption tunnel according to user configuration;

b. selecting a second encryption tunnel according to a selection strategy;

c. transmitting an original encapsulation packet through the first encryption tunnel; wherein the original encapsulation packet encapsulates all of the data packets;

d. transmitting a duplicate encapsulation packet through the second encryption tunnel; wherein the duplicate encapsulation packet encapsulates at least one of the data packets;

wherein the second encrypted tunnel is protected by encryption;

when a first original encapsulation packet and a first duplicate encapsulation packet encapsulate a first data packet, the first original encapsulation packet and the first duplicate encapsulation packet have the same global sequence number and different tunnel sequence numbers.

2. The method of claim 1, wherein the plurality of encrypted tunnels are aggregated connected tunnels.

3. The method of claim 1, wherein the selection policy is based on one or more of the following criteria: user selection, performance of multiple tunnels, service provider, usage restrictions, location, time, usage price, security, user identity, internet protocol address range, communication protocol, communication technology, application and device.

4. The method of claim 1, wherein performance of the first encrypted tunnel is better than performance of the second encrypted tunnel; wherein the performance is substantially based on bandwidth and/or latency.

5. The method of claim 1, further comprising the steps of:

e. determining whether to transmit the duplicate encapsulation packet;

f. when it is determined that the duplicate encapsulated packet is not transmitted, not performing step (d);

wherein the performing of step (e) is performed before performing step (d).

6. The method according to claim 5, wherein the determining is based on whether the second network node does not receive the one or more original encapsulation packets.

7. The method of claim 1, wherein the duplicate encapsulation packet encapsulates a plurality of data packets.

8. The method of claim 1, wherein the duplicate encapsulated packet encapsulates error correction information.

9. The method of claim 1, further comprising:

transmitting the duplicate encapsulation packet through a third encryption tunnel; wherein the third encrypted tunnel is an aggregated connected tunnel.

10. The method of claim 9, wherein the duplicate encapsulation packet encapsulates the data packet; the data packet is encapsulated in the duplicate encapsulation packet in step (d).

11. A first network node for forwarding data packets through a plurality of encrypted tunnels, comprising:

at least one network interface;

at least one processing unit;

at least one non-transitory computer-readable storage medium stores program instructions executable by at least one processing unit to:

a. selecting a first encryption tunnel according to user configuration;

b. selecting a second encryption tunnel according to a selection strategy;

wherein the second encrypted tunnel is protected by encryption;

12. The first network node of claim 11, wherein the plurality of encrypted tunnels are aggregated-connected tunnels.

13. The first network node of claim 11, wherein the selection policy is based on one or more of the following criteria: user selection, performance of multiple tunnels, service provider, usage restrictions, location, time, usage price, security, user identity, internet protocol address range, communication protocol, communication technology, application and device.

14. The first network node of claim 11, wherein performance of the first encryption tunnel is better than performance of the second encryption tunnel; wherein the performance is substantially based on bandwidth or latency.

15. The first network node of claim 11, wherein the at least one non-transitory computer-readable storage medium further stores program instructions executable by at least one processing unit to:

e. determining whether to transmit the duplicate encapsulation packet;

f. when it is determined that the duplicate encapsulation packet is not transmitted, not performing step (d);

wherein the performing of step (e) is performed before performing step (d).

16. The first network node of claim 15, wherein the determination is based on whether the second network node did not receive the one or more original encapsulation packets.

17. The first network node of claim 11, wherein the duplicate encapsulation packet encapsulates a plurality of data packets.

18. The first network node of claim 11, wherein the duplicate encapsulation packet encapsulates error correction information.

19. The first network node of claim 11, wherein the at least one non-transitory computer-readable storage medium further stores program instructions executable by the at least one processing unit to transmit the duplicate encapsulated packet over a third encrypted tunnel, wherein the third encrypted tunnel is a tunnel of an aggregated connection.

20. The first network node of claim 19, wherein the duplicate encapsulation packet encapsulates the data packet; the data packet is encapsulated in the duplicate encapsulation packet in step (d).