CA2628788A1 - Transmission control protocol with performance enhancing proxy for degraded communication channels - Google Patents

Abstract

The integration of an improved retransmission protocol into a performance enhancing proxy (PEP) for degraded communication links. Various embodiments of the invention include congestion control, window size adjustment algorithms, connection negotiation features, and connection establishment acceleration features.

Description

TRANSMISSION CONTROL PROTOCOL WITH PERFORMANCE
ENHANCING PROXY FOR DEGRADED COMMUNICATION CHANNELS
Field of the Invention [01] The invention generally relates to network communications over wireless links that are subject to congestion and accompanying degradation and the protocols that may be used for effective and reliable transport of packets.
More specifically, the invention relates to the enhancement of reliable communications between mobile terminals that provide "communications on the move" (COTM) and geostationary satellites of the passive (bent pipe) and intelligent (on-board processing) types.

Background of Invention [02) A number of protocols have been developed to provide reliable packet conununications between two user entities, especially where there is an unreliable communications link. The Time Sequence Protocol (TSP) is a high performance data transport protocol that is suitable for use as a link layer protocol. for packet conununications. The TSP is particularly suited for use over a link between two end users in a point-to-point communication environment where the link is subject to severe degradation due to interference. Such interference may be caused by environmental conditions, such as the weather, or physical impediments, such as buildings, bridges or the like, that prevent a clear link between a COTM
terminal and another terminal, whether a satellite, other COTM terminal or stationary terminal. For example, in the communications system 100 illustrated in Fig. 1, a satellite 101 in geostationary orbit may be in communication with a plurality of COTM terniiiials COTMI-COTMQ- as well as fixed site terminals FS-1, FS2,.the latter being coupled to the Internet 102. The fixed site terminals FSl and FS2 are shown with large dish antennas because this is typical practice, although one of ordinary skill would understand that the fixed sites can employ small antennas if the link budget allows. The mobile terminals generally require steerable antennas to compensate for the motion of the platform. However, buildings B 1 and B2 may serve to block one or more of the links between the satellite 101 and COTM
terminals and create an outage, as depicted by the dashed lines_ The TSP uses retransmission strategies to achieve a desired throughput and delay performance where packets are lost due to interference with an established point-to-point link.
However, TSP is not applicable to network communications where links are controllably opened and closed, nor is it concerned with congestion on links since each link is established point-to-point. Moreover, even though point-to-point, end point applications communicate indirectly, via the link layer, the TSP cannot provide direct application to application communication.

[031 The transmission control protocol (TCP) is well known standard for Internet applications and is represented by published standards that are incorporated herein by reference. The TCP protocol is combined with the Internet protocol (IP) to establish a general basis for network communications (TCP/IP).
In a TCP-based system, end users enter iiito and. depart from the network, for example by signing on to the Internet at a workstation, and the links that provide the comrnunication between end users can open and close frequently as transmitted packets are switched. While such flexibility provides advantages, it also creates a problem with congestion on links as demand for capacity rises.

1041 An example of a TCP based network is illustrated in Fig. 2 where a system 200 includes a satellite 201 that is in wireless" communication with terrestrial satellite terminals 210, 220 over links 202, 203 so that end user workstations or servers 230, 240 may communicate with each other. The satellite terminals 210, 220, which may act as IP routers as would be known to those skilled in the art, are coupled to the satellite 201 via links 202, 203 through a satellite physical layer 211, 221, which in turn couples pairs of IP layers 212, 213 and 222, 223 to a respective Ethernet layer 214, 224 for communication via a local area network.
The Ethernet tayer 214 may couple directly via link 216 to a similar layer 231 in the workstation server 230 or to a similar layer 241 via links 251, 252 through the Internet 250. Each workstation or server has appropriate IP layers 232, 242, TCP
layers 233, 243 and applications 234, 244. The applications may talk directly to each other via the transport layer, thereby providing a significant advantage over the TSP-based system, which only uses the link layer for cornmunications. In the illustrated system, TCP flow control and acknowledge message exchange is provided end-to-end.

[05] Since network links necessarily extend into the wireless domain, appropriate consideration must be given to the unique problems created when the TCP protocol is applied to a wireless network. Applying a standard transmission control protocol directly over a long delay link, such as those encountered in a ground to satellite link for example, results in poor performance due to many factors related to interference or the like. If a comm.unication link is also subject to operational degradations, such as service outages, then network protocol performance is further diminished. For example, in a TCP protocol communication system supporting COTM terminals in moving vehicles, outages or blockages will occur when the vehicles pass under bridges or other obstructions. Thus, in COTM applications of TCP, the packet losses tend to be high, on the order of 15%-55% depending on whether the environment is open, rural or urban, and will lead to retransmissions, thereby causing the TCP-based protocol to reduce its transfer rate. Indeed, most TCP/IP based applications, such as web access,, email and ftp, as well as other applications related to voice and video, do not perform well over communication channels that link COTM
terminals with fixed terminals, whether terrestrial or geostationary.

[06] According to the conventional TCP protocol, when packets are lost, retransmissions are required according to certain rules and cause TCP to reduce its transfer rate. In addition, TCP uses a window management scheme in order to prevent congestion in the network and manage throughput. Throughput is defined as the window size "W" divided by the round trip time ("RTT"). Thus, at start-up, an initial window size "W" is set to be small, equal to one or two packets, and for a given RTT the throughput will be small. The window size W is increased exponentially (e.g., 50%) every round trip time as confidence in the capability of the link is developed, and the throughput increases accordingly. However, if a packet is lost, W is reduced exponentially (e.g., by 50%) and then increased linearly (e.g., by 0.5 packets) every round trip time, so that through put is decreased and then slowly increased. After multiple packet losses, W may be reduced to a minimum of 1 packet and then increased exponentially every round trip time until the window equals a percentage of the original window, and then is increased linearly every round trip time. For a COTM channel with periodic losses every 1-4 seconds, the instantaneous throughput is low since W stays close to 1. An example of conventional TCP performance over a satellite link is illustrated in Fig. 3. Time flows down the page, and the originating terminal is designated Tx (ori the left) and the receiving terrainal is on the right (Rx).
Here a window of 8 kbytes and an RTR of 500 ms yield a throughput of 128 kbps, given the fact that all eight 1-kbyte packets of data require an acknowledgment (ACK).
The throughput is severely limited, even though a 2 Mbps channel is used.

[071 In the 'industry, signal outages often are compensated through the use of ARQ (Automatic Retransnlission request) techniques.. Link layer ARQ-S
provides for retransmission of lost packets between the COTM terminal and the payload, e.g.,'a TCM satellite payload. In such case, the payload must maintain a separate ARQ session with every COTM terminal and the COTM terminal must maintain one ARQ session with the payload. Link layer ARQ-T requires retransmission of lost packets between COTM terminal and a fixed or other COTM terminal. In ARQ-T, the ARQ must be embedded in the terminal IP layer, so that payload can route ARQ packets. The fixed terminal must maintain a separate ARQ session with every destination COTM terminal and each COTM
terminal must maintain a separate ARQ session for every one of its destination terminals.

[081 Notwithstanding the potential benefits of ARQ techniques, ARQ interacts poorly with end-to-end TCP protocols, and even proxies. ARQ causes packets to get delayed and causes TCP to time out, reduce window sizes and invoke its own retransmissions. This unfavorable interaction is exacerbated when employed over long delay links, due to the long delay necessary for each retransmission.
Moreover, ARQ will not fix end-to-end TCP limitations such as small TCP
windows, small initial windows, slow windows increases, and lack of support for ECN (Explicit Congestion Notification), which is a TCP/IP standard that uses two bits in the IP header.

[09] The TCP-ECN standard, particularly as detailed in RFC3168 (see Ramakrishnan et al, The Addition of ECN to IP, September 2001), specifies the use of two bits in the IP header of data packets to signal among routers and connection endpoints as to the existence of congestion. Congestion on a TCP
network causes loss of packets. In order to identify congestion as a cause of packet loss, a sending TCP terminal can set an ECN-capable bit in the IP
header of data packets, and intermediate routers can set the ECN bit in the IP header when they experience congestion while. forwarding a transmitted packet. If the ECN bit is set,. the receiver TCP terminal extracts the ECN bit and sends it to the sending TCP ternninal in the TCP header of a subsequent ACK packet. The sendiing TCP terminal uses that information to slow down the transmission rate.

[10] The number of packets of unacknowledged data that are permitted to be outstanding at any given time, before a congestion or failure condition is established, is specified by a value CWND. The value CWND serves as a congestion window and is decreased under certain conditions, such as the receipt of an ECN bit. According to the TCP standard, the use of ECN is always in combination with retransmission protocols that are engaged to overcome blockages and interference. However, the incompatibility of ECN with ARQ
leaves a need for a solution to the combined congestion and interference problem.

[11] Forward error correction also can be employed to simply 'code through' any outage length with the proper selection of code, code length and interleaving technique. This technique is not acceptable because of the inherent delay and traffic rate reduction required to compensate for losses due to long periods of degradation_ For example, in a system subject to degradations of variable duration, the system designer would need to select a coding technique robust enough to compensate for the longest period of degradation. This coding, however, will result in unnecessary delay and/or lowered data rate during the times in which only short periods of degradation occur.

[12] One solution to the foregoing problems lies in a TCP-based technique having a performance enhancing proxy (PEP) that serves to increase the perfonnance of network protocols over degraded communication links. That TCP-PEP uses an effective retransmission protocol that is integrated into a performance . enhancing proxy in . order - to-. --compensate for degraded=
cornmunication links. The TCP-PEP retransmission technique is modified and supplemented by additional features related to control of congestion, window size, connection negotiation, and accelerated establishment.

Summary of Invention [13] The invention concerns the integration of an improved retransmission protocol into a performance enhancing proxy (PEP) for degraded communication links. Various embodiments of the invention include congestion control, wir-dow size algorithms, connection negotiation, and connection establishment acceleration.

[14] In particular, the invention concerns a method of implementing a retransmission protocol in a TCP network having plurality of wireless links, including at least one link between a first terminal and a second terminal, where each of said terminals supports a first TCP retransmission protocol and at least one of said first and second terminals also supports a second TCP
retransmission protocol. which is different from the first retransmission protocol. The method comprises at each station, making a first determination of whether the first protocol is supported, and at each station supporting the first protocol, making a second determination of whether said second protocol is to be utilized. Then, at each station, at least information regarding the first determination is inserted into a call setup packet. At each of the first and second terminals, the call setup packet is transmitted to the other of said first and second terminals and at the other first 'and second station, a third determination is made as to whether the second TCP
retransmission protocol is supported by both first and second terminals.
Finally, in response to the third determination, one of the first and second TCP
protocols is implemented.

[15] The invention also involves a method of implementing a retransmission protocol in a TCP network having a plurality of wireless links including at least one link between a first terminal and a second terminal, where each terminal supports a TCP retransmission protocol having a variable size window. The window size is varied according to a size variation protocol. The method compiises establishing a window size increment value and changing the increment value on the basis of detected congestion and indicia of successful transmission and reception of packets.

[16] In accordance with still further features of the invention, the method comprises increasing or decreasing one or both of the window size and increment value on the basis of transmission history_ [17] In accordance with yet another feature of the invention, the method comprises use of a reorder count that perrnits a specified number of out of order packets to be outstanding before remedial action, including retransmission and adjustment of window size and increment value, is undertaken.

Brief Description of the Drawings [18] The present invention is illustrated by way of example, particularly a preferred eznbodiment best mode, and not by way of limitation. Further objects, features and advantages of the present invention will become more clear from the following detailed description of a preferred embodiment and best mode of implementing 'the invention, as shown in the figures of the accompanying drawing, in which like reference numerals refer to similar elements, and in which:

[19] Figure 1 is an illustration of a wireless communications system to which the invention is applicable.

[20] Figure 2 is an illustration of a conventional satellite based TCP/IP
cornmunication system_ [21] Figure 3 is an illustration of signal flows involved in packet transmission and acknowledgment for a wireless communications link via satellite.

[22] Fig. 4 is an illustration of an exemplary satellite based TCP/IP
communication network, having plural satellite terminals that are in communication with a satellite over wireless links and are adapted to use performance enhancing proxy techniques.

[23] Fig. 5 is an illustration of signal flow among transmitters and receivers at workstations and satellite terminals in a satellite based TCP/IP communication system using performance enhancing proxy techniques.

[24] Fig. 6 is a time sequence diagram that illustrates signal flow in a conventional TCP network using a conventional retransmission technique for missing packets.

[25] Fig. 7 is a time sequence diagram that illustrates signal flow in an improved TCP network using an improved retransmission technique for missing packets in accordance with the present invention.

[26] Fig. 8 illustrates a flow chart for a process that adjusts window size CWND using the improvcd TCP strategy.

[27]. Fig. 9 is a chart illustrating the ramp up of CWND according to the present invention.

[28] Fig. 10A an illustration of a TCP SYN packet used during connection setup and Fig. lOB is an illustration of a flowchart for determining whether to use the improved or conventional TCP protocol.

[29] Fig. 11 an illustration of another exemplary satellite based TCP/IP
communication network, having plural satellite terminals that are in communication with a satellite that has performance enhancing proxy equipment on board.

[30] Fig. 12 an illustration of yet another exemplary satellite based TCP/IP
communication network, having plural satellite terminals that are in communication with a satellite over wireless links and at least one is coupled to a host having performance enhancing proxy equipment.

[31] Fig. 13 illustrates a TCP-PBP host with enhanced features according to the present invention.

Description of Exemplary Embodiments of the Invention [32] In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the broader aspects of the present invention, as well as to appreciate the advantages of the specific details themselves according to the more narrow aspects of the present invention. It is apparent, however, to one skilled in the art, that the broader aspects of the present invention may be practiced without these specific details or with equivalents determined explicitly herein or in accordance witli the guides set forth herein. Well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the present invention with unnecessary details.

[33] The present invention is also capable of other and different embodiments, and its several details can be modified in various respects, all without departing from the spirit and scope of the present invention. The drawing and description are illustrative and not restrictive. In the following description of the exemplary embodiments, the same reference numbers or characters may be used in different figures where the structures or features are the same and a common description of such structures or features would be applicable to all. However, the use of different numbers for a similar structure or feature in one figure does not necessarily mean that it is different from a similar structure or feature in another figure.

[34] The present invention is appropriate for a wide range of network protocols and degraded channels, but for purposes of providing an exemplary and currently applicable embodiment, it will be described with respect to the Internet's Transmission Control Protocol (TCP). The exemplary embodiment assumes the existence of system of the type illustrated in Fig. I where there is one or more degraded wireless conununication links that rely on a satellite link through a geostationary satellite. The exemplary signal degradation will be outages due to the use of Communication on the Move (COTM) terminals, but it need not be limited to such outages, as they can be the result of weather and RF or other interference. Outage characteristics of different types of environments (open and urban) are described in "F-HF Satellite Communications-on-the-Move Blockage Channel Modeling" H. Yao, MIT Lincoln Laboratory Technical Report 1098, 2 November 2004. While a single satellite is depicted for clarity, the application of the invention to a system with multiple satellites and inter-satellite links is also contemplated. The satellites can be simple relay (bent pipe) or more complex processing satellites.

[35] Figure 4 illustrates an exemplary IP wireless network 400, which provides communication via satellite 401 between satellite terminals 410, 420, 430 via links 402, 403, 404, respectively. Each satellite terminal 410, 420, 430 is coupled to a representative user workstation or server 440, 450, 460, respectively, to enable workstations and servers in a network to communicate among each other directly or via the Internet 470. A network as illustrated may support multiple satellite terminaVuser terminal combinations, as indicated by the illustration of the combination of satellite terminal 430 and user workstation/server 460 with a dotted line. As with the terminals illustrated in Fig. 2, user applications communicate bi-directionally across the network to server applications through layered network connections. The satellite terminals use the IP layer to interface the satellite physical layer to the Ethernet physical layer. Specifically, the satellite terminals 410, 420, 430 have satellite physical layers 411, 421, 431 that couple to a respective link 402-404, IP layers 412, 422, 432 that couple to the satellite physical layers, and IP layers 413, 423, 433 that couple to Ethernet layers 416, 426, 436. The workstations 440, 450, 460 similarly have Ethernet layers 231, 241, 251, IP layers 232, 242, 252, TCP layers 233, 243, 253, and Application layers 234, 244, 254. Within the satellite terminal, however, between the IP
layers, a TCP,satellite layer 425A, 435A, 445A and TCP terrestrial layer 425B, 435B, 445B is linked by a TCP performance enhancing proxy (PEP) layer 425C, 435C, 445C.

[36] The TCP-PEP feature can be installed in a satellite hub, integrated into the terminal as shown, or separately implemented between the satellite terminal and the Internet and coupled to a router. The TCP-PEP feature substitutes for the conventional ARQ-T or ARQ-S and retransmits lost packets between terminals, particularly where one 'or both terminals are COTM terminals. The. TCP-PEP
feature uses an enlarged window over tiie satellite network and uses TCP with RFC1323 extensions as the protocol over the satellite link. TCP-PEP also provides local TCP acknowledgements to a workstation or host PC and uses an optimized TCP protocol over the satellite network. The TCP-PEP feature is used between every pair of terminals, COTM or fixed, and a TCP-PEP session is dynamically established for each user.

[37] In an operation of the TCP-PEP as applied to a satellite network as illustrated in Fig. 4, the end-to-end TCP connection is broken ix,to 3 concatenated transport layer connections. Data from the user TCP instance is acknowledged by the TCP instance in the local satellite terminal. Data is transported across the satellite network using a separate TCP-like transport layer connection, which typically uses a window that is sized for the satellite bandwidth-delay product.

[38] The result of a TCP-PEP implementation in a satellite system of Fig. 4 may be understood from the illustration in Fig. 5, where in contrast to the standard system in Fig. 2, a 2 Mbps TCP throughput is achieved using TCP-PEP, even when end-user TCP implementations use 8 kbyte windows. As in Fig. 2, data transmissions are in 1 kbyte packets with eight packets being transmitted within approximately one roundtrip time. Column A in Fig. 5 represents the communication between the workstation and the satellite terminal, possibly over the Internet, and utilizes a window of 8kbytes with an ACK round trip time of ins, yielding a throughput of 2 Mbps. Column B in Fig. 5 represents the communication between the satellite terminals, and utilizes a window of 128 Kbytes with an ACK round trip time of 500 ms, as is typical for geostationary satellite communicati.ons. The throughput is also 2 Mbps. Column C in Fig. 5 represents the communication between the satellite terminal and work station, and as in column A, has a throughput of 2 Mbps. Indeed, with the insertion of TCP-PEP functions at the satellite terminal, without requiring any change in the end user equipment or software, the throughput can be as high as the satellite link rate, e.g., up to 100 Mbps.

[39] Figure 6 is a time sequence diagram that illustrates the operation of the TCP-PEP on a channel that experiences a disruption that causes the loss of four 1-kbyte packets. Time flows down the page, and the originating terminal is designated Tx (on the left) and the receiving terminal is on the right (Rx).
The transmitting terminal starts the transfer by sending data packets with transmit sequence (txseq) numbers 0-9. The delay time for these packets across the satellite link is illustrated by the downward sloped lines, and the interrupted lines for txseq=l - txseq =4 indicate that these packets are lost on route to the Rx terminal. Note that Tx continues to send packets txseq=5 up to txseq=9, iinplying that a congestion window CWND value at this initial point is greater than or equal to 9. Packets with txseq=5 to txseq=9 are all successful (in this illustxative example, the channel impairment has disappeared by this time).

[40] Once Rx receives txseq=0, it replies with an acknowledgement ACK
rxseq=l, indicating that the next packet it is ready to receive is txseq=l.
The line for this responding ACK for packet txseq=0 reaches the Tx, indicating that txseq=0 has been received. However, since none of packets txseq =1 to txseq=4 generate ACK packets at the Rx, based on the standard TCP protocol, after the packet with txseq=9 is transmitted, Tx experiences a TCP timeout. It notes the last rxseq that it received (rxseq-1) and thus retransmits packet txseq=l. The TCP
congestion window CWND value is now reduced to.1 packet, so Tx must wait for an ACK for the retransmitted packet corresponding to txseq=l before it transinits again.

[41] In the meantime, Rx can receive successful packets txseq=5 - txseq=9.
Each of these packet receptions causes an ACK to be returned, but the next packet in sequence needed by Rx still remains txseq=l, so the same ACK is transmitted back to Tx for each of the received packets txseq=5 to txseq=9. Tx ignores the ACKs corresponding to packets txseq=6 - txseq=9, since all also have rxseq=l.
Only after the successful reception of packet txseq=l does Rx transmit an ACK
with rxseq=2.

[42] As illustrated, the packet txseq=l is successfully received at Rx and an ACK with rxseq=2 is generated. Since the value of CWND is only 1 due to the reduction caused by the lost packet, Tx waits for the ACK rxseq=2 before sending any other packets. Upon receipt of ACK rxseq=2, the value of CWND is increased to 2 and Tx sends the next two of the lost packets, txseq=2 and txseq'3.

It is assumed here that these ACKs arrive within the timeout period of the TCP, so that the congestion window CWND is increased to 3 and packets txseq=4 and txseq=5 are subsequently transmitted. The figure does not show any further communications, although it is implicit that these next packets will be acknowledged and the data transfer will continue. Note that packets with txseq=5 to 9 will all be retransmitted even though they were successfully received the first time.

[43] The foregoing TCP-based retransmission protocol does not provide a desired leve] of efficiency, since there are readily apparent delays_..in waiting. for returned ACKs and because the congestion window CWND is subject to iznmediate reduction to a minimum level of 1 packet upon detection of a loss and because the increase in the window size is slow, being performed only on the basis of successful completion of round trips of a packet transmission and an acknowledgment of receipt.

[441 The foregoing approach is useful where there is an assumption that there are not too many packet losses. Such assumption is valid where the network connections are over reliable terrestrial links or even over satellite links where there is little interference, as with fixed station to satellite communications.
Improvements can be made to the performance of the system by using a large initial window, supported by large storage capability to collect out of sequence packets, and b.y increasing the window faster as ACK packets are successfully returned, based on confidence in the reliability of the system. However, this approach may not be suitable where there are mobile COTM terminals and where congestion is possible.

[45] An improvement to the above retransmission protocol can overcome these problems and achieve important goals, particularly where both congestion and interference problems are likely to be encountered. First, the improved retransmission protocol enables the communicating nodes to determine quickly if packets have become lost in transit. Second, the improved retransmission protocol avoids unnecessary retransmissions to minimize latency and keep the channel loading to a minimum. With the improved protocol, there are no spurious retransmissions and no RTT timeout based retransmissions. Also, the transmissions are insensitive to delay and delay-variance, and insensitive to link layer ARQ, if any. The average throughput that can be achieved approaches 1-packet_I oss__rate.

[46] In achieving these goals, the retransmission protocol uses ACK (positive acknowledgment) and NACK (negative acknowledgment) signaling selectively, rather than using such signaling for each packet as in Fig. 6. It also allows use of a large number of ACK/NACK signals per ACK packet. In particular, every packet is assigned a send sequence number txseq, as in conventional TCP, but a time sequence value Tseq is also maintained at the transmitter. This sequence number is incremented whenever a packet is transmitted or retransmitted, and the current time sequence value is saved locally for each packet whenever it is transmitted or retransmitted. The receiver sends acknowledgement packets, using the same rules as conventional TCP. However, there are additional packets generated and transmitted in order to efficiently convey transmission and reception information.

[471 For example, every N packets, where N is an integer (e.g., N=4), a STATREQ packet is sent by the transmitter to the receiver. The STATREQ

packet contains (1) the time sequence number, txTseq, and (2) the highest sequence number sent, txSseq plus 1. The STATREQ packet is also sent whenever TstatReql seconds (e.g., 0.1 seconds) have elapsed since the last STATREQ packet was sent, and some data remains unacknowledged. The STATREQ packet is also sent whenever TstatReq2 seconds (e.g.,.5 seconds) have elapsed since the last STATREQ packet was sent, and no data remains unacknowledged. The STATREQ information can be piggybacked on a normal data packet, when possible. Thus, within a roundtrip time RTT, several STATREQ packets may be sent_ The values of TstatReql and TstatReq2 are systeni design parameters that may be set on the basis of one round trip time and/or other system parameters as would be understood by those skilled in the art.

[48] The following features are implemented as rules for the retransmission protocol.

[49] Whenever a receiving terminal receives an out of sequence packet whose sequence number is larger than the largest sequence number received, the receiving terminal sends a USTAT packet to the transmitting terminal, identifying the sequence numbers of the missing packets, the sequence numbers of the received packet and the normal acknowledgment sequence number. This step may optionally be skipped if the network is expected to re-order packets.

[50] Whenever, the receiving terminal receives a STATREQ packet, it sends to the transmitting terminal a STAT packet, which contains the received time sequence number, the sequence number for all the missing packets, the sequence numbers of all received packets and the normal acknowledgement sequence number.

[51] Whenever the transmitting terminal receives a USTAT packet, it frees- up acknowledged packets and retransmits to the receiving terminal the missing packets identified in the USTAT packet.

[52] Whenever the transmitting terminal receives a STAT packet, it frees up acknowledged packets and retransmits to the receiving terminal the missing packets identified in the STAT packet, if the time sequence value saved for the packet is less than the time sequence value received in the STAT packet. If the network'is expected to re-order packets, then the last comparison is done with the time sequence value received in the STAT packet minus- a REORDERCOUNT
value, which represents the number of packets that may be received out of order (and surely reassembled into a proper order), before it is determined that a packet loss has occurred and that further transmission may be halted. A value of REORDERCOUNT = 3 is recommended and is similar to that which is used in standard TCP.

[53] In accordance with an exemplary embodiment of the subject invention, every packet is assigned a send sequence number txseq, as is done in normal TCP.
A time sequence txTseq counter is maintained at the transmitter, which is incremented whenever a packet is transinitted or retransmitted. The time sequence number txTseq will start at zero for the first packet of a transmission and will increment every time a packet is transmitted, whether or not that packet was an original transmission or a retransmission. Thus, if no packets need to be retransmitted, txTseq will always be the same as txseq. The receiver sends traditional acknowledgement packets, using the same rules as normal TCP_ [54] In normal TCP, the retransmission timer is based on the round trip time RTT and is doubled on every timeout. Retransmissions are done whenever the retransmission timer expires. However, according to the exemplary embodiment of the invention, the TCP retransmission timer is used only during connection setup. For data retransmissions, the periodic STATREQ packets are used, which are not related to round-trip-time.

[55] Figure 7 is a time sequence diagram that illustrates the operation of an exemplary embodiment of the retransmission aspect of the present invention for the same degraded channel as used in Figure 6. The transmitting terminal in this example sends a STATREQ every 4 packets starting with packet txseq=4.

[56] The transmitting terminal Tx starts the transmission by sending packet txseq=O and the transmission is received' at the receiving terminal Rx and acknowledged by transmission of ACK and rxseq=l to Tx. As in Fig. 6, subsequent data packets txseq=l to txseq=4 are sent, but the transmission is blocked by interference. One difference is that the latter transmission of txseq=4 is accompanied by a STATREQ with txSseq = 4 (indicating that the highest packet sequence number txseq, sent so far is txseq=4) and txTseq=4 (indicating that a total of 4. packet transmissions - original (4 transmissions) or retransmissions (0 transmissions) - have occurred).

[57] The transmitting terminal Tx then sends data packets with transmit sequence (txseq) numbers 5-8 as in the previous case of Figure 6. Packet txseq=8 follows txseq=7 and now also contains the STATREQ with txSseq=8 (indicating that the highest packet sequence number, txseq, sent so far is txseq=8) and txTseq=B (indicating that a total of txTseq+l = 8 packet transmissions -original or retransmissibns - have occurred). The broken lines for txseq=l - txseq=4 (including the STATREQ) indicate that these packets are again lost on route to the receiving terminal Rx. Notwithstanding the loss of packets, the transmitting terminal Tx continues to send new packets up to txseq=l1.

[58] Once Rx receives txseq=O, it replies with an acknowledgement ACK with rxseq=l, indicating that it has successfully received the packet sequence up to txseq=0.

[591 The next successful packet at the receiver is txseq=5.. This out of order packet causes Rx to issue a USTAT packet with rxseq=l (indicating that it has successfully received the packet sequence up to txseq=O). It also issues a spanlist [1,5,6] that, as will be detailed later, indicates that packets with txseq=l to txseq=4--are missing, and the packet with txseq=5 has been received. The listing of 6 does not indicate that 6 has been received but merely serves as marker for the end of the span list.

[60] Next, Rx successfully receives packets txseq=6 and txseq=7. The next packet in sequence needed by Rx still remains txseq=l. Thus, the identical ACK
is transmitted back to Tx (with rxseq=l) in response to both packet receptions for txseq=6 and txseq=7. Next, Rx receives data packet txseq=8 with STATREQ, txSseq=8 and txTseq=8. Rx responds with a STAT message with spanlist [1,5,9], indicating that it has successfully received packets up to txseq=8, but is still missing packets with txseq=l to txseq=4. This STAT message also has rxseq=l, since the next packet in sequence needed by Rx still remains txseq=l.

[61] Rx subsequently receives packets with txseq= 9 to txseq=ll. The next packet in sequence needed by Rx still remains txseq=l upon the reception of these packets, so again an ACK identical to the previous ACK (with rxseq=l) is transmitted back to Tx in response to these packet receptions for txseq=9 to txseq=ll.

[62j After the packet with txseq=ll is transmitted, Tx receives the USTAT
identifying which packets have been lost (I to 4). It notes the first packet that has been lost and. retransmits that packet, txseq=l. As well, txTseq is now 12, a multiple of 4, so Tx piggybacks a STATREQ with the data packet. For this STATREQ, txSseq -is 11, since the transmit packet with the highest sequence number already transmitted has txseq=l1.

[63) Tx subsequently retransmits packets with txseq=2 to txseq=4. Meanwhile, Tx receives and -ignores the ACKs corresponding to packets txseq=6 and txseq=7 (both have rxseq=l). It further receives and ignores the STAT resulting from the STATREQ of packet with txseq=8, since the spanlist [1, 5, 9] gives no new information about unsuccessful packets. Likewise, Tx receives and ignores the ACKs corresponding to packets txseq=9 to txseq=ll, since they all still have rxseq=l.

[64) After Tx has re-transmitted the last of the known lost packets (txseq=4), it continues transmitting the next packet in order, txseq=12. This transmission packet coincidently has txTseq=16 (indicating that a total of txTseq+1 - 17 packet transmissions -original or retransmissions - have occurred). Since txTseq is a multiple of N, where N=4, a STATREQ is transmitted along with data packet txseq=12. This STATREQ has txSseq=12 (indicating that the highest packet sequence number, txseq, sent so far is 12) and txTseq=16. Tx finishes its transmissions in this example by subsequently sending packets with txseq=13 and txseq=l4, respectively (not shown).

[65] When Rx receives the retransmission of data packet txseq=l and its STATREQ with txTseq =11 and txSseq =12, it notes that it has now completely received the transmission sequence up to txseq=1 and thus will set rxseq=2 and respond with a STAT containing spanlist [2, 5, 12] since the kziown packet gap is now only from txseq=2 to txseq=4, and packets txseq=5 to txseq=lI have been successfully received_ As Rx receives the retransmitted packets txseq=2 to txseq-4, it replies with ACKs containing rxseq=2 to rxseq=4, respectively.

[66] When Rx receives packet txseq=12 and its STATREQ, it notes that it has now completely received the transmission sequence up to txseq=12 and thus sets rxseq=13 and responds with a STAT containing..spanlist [J. õWhile not shown, it responds to the last two receptions of packet txseq=13 and txseq=14 with ACKs containing rxseq=14 and rxseq=l5, respectively.

[67] In the foregoing description of the retransmission protocol that embodies an aspect of the present invention, a listing of received packets that bound missing packets was used and were called "spanlists." Spanlists embody one technique to specify which packets have (or have not) been successfully received. If all packets known (at the receiver) to be transmitted have been received, then the spanlist will be null. If a single gap is known, then a two element spanlist [a,b]
will indicate that packets with txseq=a to b-1 are known to be lost. If another element is added to the spanlist, resulting in [a, b, c), then there still is an indication that packets with txseq=a to txseq=b-I are known to be lost.
Additionally, it is asserted that packets with txseq=b to txseq=c-1 have been successfully received. Each added element to the spanlist will alternatively identify a known missing' block or a'known received' block. While this is one technique for representing missing or received blocks, there are many equivalent techniques such as sending the sequence numbers of all successfully received packets, although that alternate technique can result in large lists being sent that could dramatically increase the system overhead.

[68] As noted previously with regard to Fig. 6, the standard TCP uses a window CWND that is a value retained at the transmitter and represents the number of packets that may have been transinitted and remain unacknowledged, before further transmission is halted. The size of the window correlates to the size of available memory to store transmitted packets, since those packets may have to be retransmitted if ultimately it is deternmined that they are lost. Where memory resources are_limited, or where packet losses are expected due to interference or congestion, the CWND value or window size is initially small, and the reduction in CWND value upon loss of a packet is rapid, with the increase in CWND for successful round trip communications being deliberately slow. Specifically, the TCP-standard algorithm increases CWND by only one packet every RT3'. This is especially problematic for high delay links, because the RTT is long because of the delay, independent of any congestion related delay. Even if there is an exponential increase in window size, which could increase the speed of any increase, the result would not be fair, in that older connections would be favored over younger connections.

[69] The present invention dramatically modifies the standard TCP approach'by incrementing window size rapidly on the basis of a determination that congestion is low. The modification in window size is designed to be fair across multiple TCP connections. According to an exemplary embodiment for this feature of the invention, as illustrated with respect to the exemplary flow diagram in Fig.
8, at step S80 (START) the initial window size (CWND) is set (e.g., 2 or 8 packets).
A

window size increment variable (CWNDINC) is also set (e.g. 1 packet). At step S81, a determination is made at a transmitting terminal as to whether an acknowledgment by a receiving terminal of a transmitted packet (e.g., in the form of an ACK segment) has been received. The invention is not limited to the use of an ACK signal, and other ways of determining a successful receipt of a packet at a receiving terminal are known in the art. Upon determination that there has be a successful receipt (YES), a determination'is made in step S82 as to whether (1) an ECN has been received at the transmitting terminal, indicating the existence of congestion in a manner known in the art, and whether CWND has not been decremented within the last round trip tinie. If the two conditions are met (YES), the process proceeds to step S85 and the window size (CWND) is reduced. In addition, the incrernent for window size increase (CWNDIINC) is reset or, in an altemative approach, reduced by a predetermined amount. If the result of the determination in step S82 is NO, then a further determination is made in step as to whether a round trip time has elapsed. If there is no congestion (step S82) and a round trip time has elapsed (step S83), then an aggressive operation of the systenn may be justified. In this case, CWND is additively increased by the then set value of CWNDINC, at step S84. Moreover, in the same step, CWNDINC is multiplicatively increased, for example, by a factor of Fl (e.g, 1.5). The process is repeated thereafter with the value of CWND and the value of CWNDINC being changed every round trip time, unless congestion is encountered or an ACK
segment is not received on time.

[70) As noted, rather than relying on the receipt of an ACK packet, these increment rules can also be implemented using number of bytes instead of number of packets. Also, the rules can be based upon computations that are done on receipt of every ACK packet, instead of once per RTT. The algorithm provides a much faster ramp up of CWND than the TCp-standard algorithm. This is especially valuable for high delay links. Examples of CWND for RTTs from 0-10, with CWND increases until there is a reduction due to the existence of an ECN, followed by a renewed ramp up, are provided in the table illustrated in Fig.
9.

[71] As. already noted, the present invention increases network throughput by taking advantage of the fact that, in a degraded channel environment, packet loss may be attributed to channel impairment instead of being an absolute indicator of congestion. At any given time, up to CWND packets of unacknowledged data are allowed to be outstanding. It should be noted that acknowledged data is not counted in CWND for this purpose. Whenever a packet is retransmitted, CWND
is not automatically changed. Thus, a CWND decrease on packet loss is suppressed. However, CWND is decreased on the basis of ECN and other congestion indicators. Optionally, CWND may be decreased upon packet retransmission, after which it is not decreased for one round trip time.

[72] Specifically, since a transmitter or router may set an ECN bit in an IP
header when congestion is detected, and an ECN notification is sent back to the transmitter by the receiver to advise of the existence of congestion, the transmitter can then adjust CWND according to a rule that causes it to be decreased by a multiplicative factor F(e.g, 0.5). However, CWND is not decreased subsequently for one round trip time RTT. Window size may also be decreased by a multiplicative factor F, if congestion is detected by other means, such as an increase in round-trip delay. CWND is not decreased subsequently for one round trip time. CWND is increased on receipt of an ACK packet according to applicable window increase rules.

[73] If multiple packets are re-transinitted, they may be "paced" t seconds apart, where t = round-trip-time / CWND (in packets). Round-trip-time (RTT) is measured using normal TCP procedures (e.g., using the timestamp option in every data packet). Alternatively, if the improved retransmission protocol as exemplified by the illustration in Fig. 7 is used and a greater efficiency is achieved, round-trip-time may be measured using STATREQ and STAT packets only.

[74] According to another exemplary feature of the present invention, each instance of the improved TCP (designated by the shorthand notation "TCP-XL") according to the present invention is configured with information related to whether (1) it can support the features of the invention and (2) whether it prefers the features of the invention. An illustration of these bits is provided in Fig. ZOA.
These two bits are exchanged in TCP call setup (SYN) packets. An algorithm for detenmining the use of the invention, based upon the input of the users at each end of a communication link, is illustrated in Fig. 10B and is initiated for each instance, as shown in step S 100. The process for connection set up using the SYN
packets proceeds at step S101 with an insertion into the SYN packet of two preference codes of the type illustrated in Fig. 10A. The codes are exchanged in step S102 and each user evaluates the information from the other user. In step S103, it is deterinined at a user terminal whether both sides of a communications link support the advanced retransmission protocol_ If not, the normal TCP
protocol is used at step S105 and the features and procedures of the advanced protocol are disabled. If both sides do support the protocol, and it is determined that at least one side prefers use of the advanced protocol in step SI04, then the advanced protocol procedures are enabled in step S 106. A user that has the capability to use the advanced procedure may choose not to do so for a variety of reasons. For example, a terminal that normally is mobile may be temporarily fixed. In this case the terminal would indicate that it did not prefer use of the enhanced TCP features during the time it was stationary, to minimize CPU time, channel overhead, etc. For recently used connections, the destination IP
address and the results of the negotiation over use of the invention is saved in a local cache_ This historical information may later be used at startup to implement the advanced protocol set up and connection procedures. Finally, the determination of whether to use the first or second protocol may be based upon congestion infonnation, detennined by the receipt of ECN information, delays in round trip time or other indicia of congestion.

[75] In regular TCP, connection setup is done by a first terminal and a second terminal exchanging SYN packets. Where SYN packets are determined to be lost, due to the absence of a reply, the lost SYN packets are retransmitted after a timeout. The timeout value is set to 3 seconds initially, and doubled after every timeout. As a result, connection set-up time can get prolonged substantially on degraded links. Since data packets cannot be sent until connection set up and establishment is complete, the performance of the system is degraded. The advanced protocol uses an improved TCP connection setup procedure that reduces connection setup time since, if it not known that the destination supports the improved TCP procedure, then the conventional TCP procedure is used.
However, if it known that the destination supports the improved TCP,procedure (e.g., based on the content of a cache or based on configuration), then the timeout value is set to a small duration (e.g., 0.1 seconds) and not changed for every retransmission. As a result, multiple SYN packets may be outstanding at any given time. If any of them are delivered to the receiver, then the connection establishment procedure can proceed to its next step.

[76] Figure, l 1 is a block - diagram of another exemplary system that may implement the improved TCP protocol, which is implemented in' relevant terminals by a module designated as TCP-PEP-XL. In Figure 11, the improved TCP features 'are contained in the COTM terminal that directly communicates over the outage prone (in this case, satellite) channel. The relay satellite in this case is a processing satellite that also contains an implementation of the improved TCP features. The satellite communicates with a network interface on the ground that, in this case, uses a traditional TCP-PEP, since this exemplary network interface terminal is fixed and thus not subject to outages or channel degradations in its direct path to the satellite.

[77] Figure 12 is an example of a bent-pipe, or siYriple relay satellite.
Additionally, the improved TCP feature of the,COT'M is implemented in a stand-alone host communicating via Ethernet to the satellite terminal. In this case, the fixed (network interface) terminal must also implement the improved TCP
feature to enable the enhanced protocol to operate. Although not shown, the fixed terminal could also implement the improved TCP feature in a separate host as illustrated in the COTM terminal in this figure.

[78] Figure 13 shows an example stand-alone implementation of the TCP-PEP-XL host 1300 of Figure 12. The protocol is implemented on a LTNUX or UNIX
host computer 1301 with at least two Ethernet adaptors, ethernetO and ethernetl, connected respectively to the terrestrial network and the satellite terminal.
At the IP Software layer 1302, non-TCP packets are transparently transferred per normal industrial techniques. Two tunnels, tun0 and tunl, connect the TCP-PEP-XL
module 1303 (which sits at the same level as TCP) to the rP level 1304 for ethernetO and ethernetl, respectively.

[79] While the present invention has been described in connection with a number of embodiments, implementations, modifications and variations that have advantages specific to them, the present invention is not necessarily so limited, but covers various obvious modifications and equivalent arrangements according to the broader aspects, all according to the spirit and scope of the following claims.

Claims (26)

1. A method of implementing a retransmission protocol in a TCP

network having plurality of wireless links, including at least one link between a first terminal and a second terminal, each of said terminals supporting a first TCP
retransmission protocol and at least one of said first and second terminals also being adapted to support a second TCP retransmission protocol which is different from said first retransmission protocol; said method comprising:

at each station, making a first determination of whether said first protocol is supported, at each station supporting said first protocol, making a second determination of whether said second protocol is to be utilized;

at each station, inserting into a call setup packet at least information regarding said first determination;

at each of said first and second terminals, transmitting said call setup packet to the other of said first and second terminals;

at said other first and second station, making a third determination of whether said second TCP retransmission protocol is supported by both first and second terminals; and in response to said third determination, implementing one of said first and said second TCP protocols.

2. The method of claim 1, further comprising:

at said other first and second terminal, making a fourth determination of whether said second TCP protocol is to be utilized; and in response to said third and fourth determinations, implementing one of said first and said second TCP protocols.

3. The method of claim 2, where historical information concerning a previous implementation of said second protocol is stored and said second protocol is implemented on the basis of said stored historical information.

4. The method of claim 2, wherein said second TCP protocol is implemented if only one of said first and said second terminals selects said second protocol.

5. The method of claim 1, wherein said second protocol comprises:
transmitting a plurality of original data packets from said first terminal to said second terminal for reception by said second terminal, each original packet comprising a sequence number;

upon reception of each original data packet at said receiving terminal in a transmitted sequence, returning an acknowledged packet and a sequence number;
upon reception of an original data packet at said receiving terminal out of sequence, returning to said transmitting terminal a status packet indicating an out of sequence reception and lost information enabling an identification of a last successfully received original packet and a lost packet;

receiving at said transmitting terminal said status packets and lost information and, in response thereto, retransmitting said lost packet.

6. The method of claim 5 wherein the retransmission of packets is paced.

7. The method of claim 5, further comprising tracking the number of packets originally transmitted and the number of packets retransmitted at the transmitting terminal and forwarding such information to the receiving terminal.

8. The method of claim 7, further comprising:

transmitting a sequence request packet from said first terminal to said second terminal;

receiving at said second terminal said sequence request packet; and in response thereto, retransmitting from said second terminal to said first terminal a status response together with an identification of lost packets.

9. The method of claim 8, wherein said sequence request comprises at least one of a time sequence number and a highest sequence number sent.

10. The method of claim 8, wherein the transmitting of a sequence request is performed on the basis of one of a transmission of each N packets, where N is an integer, or a period of time.

11. The method of claim 8, wherein the retransmission of packets is paced on the basis of round trip time divided by window size.

12. The method of claim 1, wherein a quantity of lost transmission packets is determined by a window, wherein said window is incremented for successful return trip times of original packets but is not adjusted for successful return times of retransmitted packets.

13. The method of claim 1, wherein said first protocol is a standard TCP retransmission protocol and said second protocol is a non-standard TCP
retransmission protocol.

14. The method of claim 1, wherein implementing of one of said first TCP protocol or said second TCP protocol is based on a determination of congestion.

15. The method of claim 1, wherein said call setup packet is a TCP
synchronization packet.

16. The method of claim 15, wherein plural of said call setup packets are transmitted without waiting for a reply for connection setup.

17. A method of implementing a retransmission protocol in a TCP
network having a plurality of wireless links including at least one link between a first terminal and a second terminal, each of said terminals supporting a TCP
retransmission protocol having a variable size window, said window size being varied according to a size variation protocol, said method comprising:

establishing a window size increment value; and changing said increment value on the basis of detected congestion and indicia of successful transmission and reception of packets.

18. The method of claim 17, wherein said changing step comprises increasing said window size by said increment value in response to indicia of successful transmissions, and increasing said increment value by a multiplication factor in response to indicia of successful transmission and reception of packets.

19. The method of claim 17 wherein said protocol comprises:

(a) setting an initial window size and an initial increment value variable;

(b) determining whether a packet transmission and reception time has passed; and (c) upon receipt of an acknowledgment packet, increasing the window size by the amount of said increment value and increasing said increment value by a multiplication factor.

20. A method as set forth in claim 19, wherein steps (b) and (c) are repeated a plurality of times.

21. The method of claim 19, further comprising decrementing the window size if there are congestion indications.

22. The method of claim 19, further comprising decrementing the window size if there are packet losses.

23. The method of claim 19, further comprising at least one of resetting and reducing the increment variable if there are packet losses.

24. The method of claim 19, further comprising resetting or reducing the increment variable if there are congestion indications.

25. The method of claim 18, wherein the window size is not reduced if there is a retransmission of packets.

26. The method of claim 21 wherein, if congestion is reduced, the window size is increased.