CA2351351A1 - Fair flow controlling method in packet networks - Google Patents

Abstract

There is provided a fair flow controlling method in a packet switching network. The packet switching network has a plurality of nodes and each node is connected to a plurality of sources which transmit/receive data and a dat a queue related with a current queue length and a target queue length for storing data received from the sources. In the fair flow controlling method, each node estimates the number of locally bottlenecked virtual circuits (VCs ) using the explicit rate (ER) and the minimum cell rate (MCR) of a corresponding source. The node allocates an ER to each source based on the difference between the current queue length and the target queue length of t he node, the derivative of the current queue length, and the estimated number o f the locally bottlenecked VCs, then conveys the ER to each source through a feedback signal.

Description

wo om~os Pc~rncROOioio9s FAIR FLOW CONTROLLING METHOD IN PACKET NETWORKS
BACKGROUND OF THE INVENTION
The present invention relates generally to a packet switching network, and in particular, to a fair flow controlling method.
~ Descri ion of ~rlle Related Art Packet switching networks include ATM (Asynchronous Transfer Mode) networks and the Internet. A fair flow control has significant effects in the information transmission over a packet switching network. In particular, a fair flow control of the ATM networks is related to the ABR (Available Bit Rate) service.
The ATM layer provides the following four services: CBR (Constant Bit Rate), VBR (Variable Bit Rate), UBR (Unspecified Bit Rate), and ABR. In the ABR
service, the source dynamically changes its bit rate within the available bandwidth from the network to adapt the changing network conditions when transmitting the data.
The ABR
service was introduced to the ATM network in order to support data applications that were not efficiently supported by a guaranteed bandwidth service, such as the VBR
service. For details, see S. Sathaye, ATM forum Traffic Management Specification, Version 4.0, Feb. 1996, F. Bonomi and K. W Fendick "The Rate-Based Flow Control Framework for the Available Bit Rate ATM Service", IEEE Network, vol. 9, no.

2, pp.
25-39, 1995, and R. Jain "Congestion Control and Traffic Management in ATM
Networks: Recent Advances and Survey", Computer Network and ISDN Systems, vol.
28, no. 13, pp. 1723-1738, 1996.
Most data applications are highly bursty and thus hard to predict the data tragic.
To make successful data transmission, the network has to work comply with a certain condition satisfying the predeftned cell loss requirements, time-varying tolerance requirements, and cell delays. Due to these characteristics, the network has to modify their data transfer rates according to the network loading conditions. Thus, the notion of elastic traffic services was introduced in which the data transfer rates are adjusted based on the bandwidth available from the network. A representative example of the elastic traffic services is the ABR service of ATM networks.

The ABR service, in principle, does not require a complex traffic characterization and call admission control, respectively, at the source and at the switches.
Due to this simplicity, it has been expected that the implementation and the deployment of the ABR service would be much easier than those of the bandwidth-guaranteed services, i.e. CBR or VBR service. In reality, however, the implementation of ABR-capable switches appears to be much more difficult than originally expected. The di~culty mainly lies in the designing of a simple, scalable, and stable ABR flow algorithm, more specifically an ER (Explicit Rate) allocation algorithm in an asynchronous and distributed network environment.
The ATM forum has selected a closed-loop rate-based approach for the ABR
flow control. The closed-loop rate-based flow control approach, as its name implies, uses feedback information from the network to control the data rate at which each source can I S transmit a number of cells into the network. The feedback information is conveyed to the source through a specific control cell known as a resource management (RM) cell. There are three mechanisms for a switch to write its congestion status onto a RM
cell: Explicit Forward Congestion Indication (EFCI) marking, Relative Rate (RR) marking, and ER
marking, at least one of which has to be implemented on a switch for the rate-based flow control.
Meanwhile, the long and diverse round trip delays (RTDs) involved in the closed loop and the distributed bottle neck locations of the ABR VCs (Virtual Circuits) make it difficult to design a high performance ER allocation algorithm. ABR queues in the network can hardly be stabilized when the transmission rates of ABR sources are determined based on the network state information at different times. In particular, if only a binary feed-back mechanism (either EFCI or RR marking, or both) is employed, ABR queues in a steady state inevitably exhibit persistent oscillation with its magnitude and period being an increasing function of the delay-bandwidth product. For details, see E. Hernandez-Valencia, et al., "Rate Control Algorithms for the ATM ABR
Service", European Transactions on Telecommunications, vol. 8, no. 1, pp. 7-20, 1997, F.
Bonomi, D. Mitra, and J. B. Serry "Adaptive Algorithms for Feedback-Based Flow Control in High-Speed, Wide-Area ATM Networks", IEEE J. Select. Areas on Communications, vol.
13, no. 7, pp. 1267-1283, 1995, and K. K. Ratmarkrishnan and Jain "A Binary Feedback Scheme for Congestion Avoidance in Computer Networks with a Connectionless Network Layer", Proc. ACM SIGCOMM'88, pp. 303-313, 1988.

Such an oscillatory behavior of the ABR queues will increase the likelihood of the cell loss and the link under-utilization due to the periodic buffer overflow and underflow. An ABR flow control scheme using the ER marking has been introduced to realize an asymptotic stability of the ABR queues, hence overcome the drawback of the binary feedback mechanisms. Still, designing an asymptotically stable ER
allocation algorithm, particularly in a simple form, is a difficult task. This problem naturally falls into the feedback control problem having a delay.
L. Benmohanmed and S. M. Meerkov formulated the rate-based flow control problem as a discrete-time feedback control problem with a delay, and derived an ER
allocation algorithm that achieves asymptotic stability and allows arbitrary control of the closed-loop performance. This information is disclosed in "Feedback Control of Congestion in Packet Switching Networks: The Case of Single Congested Node", IEEE/ACM Trans. On Networking, vol. 1, no. 6, pp. 693-708, 1993, and "Feedback Control of Congestion in Packet Switching Networks: The Case of Multiple Congested Nodes", International Journal of Communication Systems, vol. 10, no. 5, pp.
227-246, 1997. Their ER allocation algorithm is formed as:
_~
r[k+ 1]= r[k]- ~ ar(q[k- I]- 9r)- ~ ~;r[k-.I] . . . . . (1) =n ~=o where r[k] is the ER calculated by the switch at the discrete time k, q[k] is per-class ABR
queue length at the time k, qT is a target queue length, ai and (3j are controller gains, imax is the largest RTD of an ABR VC, and I is an arbitrary constant greater than 0.
Despite its solid theoretical foundation, the practical use of the algorithm (1) is limited by its high degree of the implementation complexity. The shortcomings and the limitations of the algorithm are described in A. Kolarov and G. Ramamurthy, "A
Control Theoretic Approach to the Design of Close Loop Rate Based Flow Control for High Speed ATM Networks", Proc., IEEE INFOCOM'97, vol. 1, pp. 293-301, 1997. In the ER allocation algorithm, the ER terms should be maintained at present and in the past, up to time lags Tmax, and a number of floating point multiplications should be performed in every discrete time slot.
At the same time, S. Chong has proposed a simpler control-theoretic ER
allocation algorithm ("Second-Order RateOBased Flow Control with Dynamic Queue Threshold for High-Speed Wide-Area ATM Networks", preprint 1997). A. Elwalid ("Analysis of Adaptive RateOBased Congestion Control for High-Speed Wide-Area Networks", Proc. IEEE ICC'95, pp. 1948-1953, 1995) has proposed a continuous-time ER allocation algorithm, given by i-(t) _ -Ar(t)- B(q(t)- qT), A,B > 0 . . . . . (2), and obtained the necessary and sufficient condition for the closed-loop system to be asymptotically stable when the RTDs of all VCs are identical. Chong has extended the stability analysis of this algorithm to the general case having arbitrary RTDs.
Meanwhile, S. Chong, R. Nagarajan, and Y. T. Wang showed that a more simplified ER allocation algorithm ("Designing Stable ABR Flow Control with Rate Feedback and Open-Loop Control: First-Order Control Case", Performance Evaluation, vol. 34, no. 4) can readily achieve an asymptotically stable system, expressed by:
r(t) ° [-K(q(t)-qT)]+, K > 0 . . . . . (3), where [X]' = max[x, 0] indicates that the greater of x and 0 should be selected.
The ER allocation algorithm (3) is disclosed as U.S. Patent No. 5,864,538 entitled "First-Order Rate-Based Flow Control with Adaptive Queue Threshold for ATM
Networks", January 26, 1999.
In the algorithm (2), two other stabilization conditions have been induced.
One of them is a suWcient condition for the general case with heterogeneous RTDs and the other is a necessary and su~cient condition for a particular case with homogeneous RTDs.
A common drawback of the algorithms (2) and (3), as compared to the algorithm ( 1 ), is that unless the controller gains and the queue length threshold are properly chosen according to the instantaneous knowledge on the bandwidth available to ABR traffic and the fraction of the available bandwidth utilized by remotely bottlenecked VCs, the ABR queue length can converge to zero which is undesirable because, at this equilibrium point, the link cannot be fully utilized.
The remotely bottlenecked VCs cannot fairly share the link since the phenomenon of bottle neck occurs at another link if the transmission rates of the VCs are not limited by their PCRs (Peak Cell Rates). In contrast, if the algorithm (1) is applied, there exists no such undesirable equilibrium point.
SUMMARY OF TAE INVENTION
It is, therefore, an object of the present invention to provide a method of guaranteeing the maximal link utilization and minimal cell loss regardless of RTDs in an ABR loop.
It is another object of the present invention to provide a method of minimizing ABR queue size requirements by ensuring asymptotical stabilization of the ABR
queues.
It is a further object of the present invention to provide a method of guaranteeing MAX-MIN fairness based on the ATM forum standards by ensuring a fair share of an available bandwidth to each ABR service user.
It is still another object of the present invention to provide a method of increasing responsiveness and transient control performance to a communication network environmental change, such as changes in the number of ABR users and an ABR
bandwidth.
It is yet another object of the present invention to provide a method of providing all functions including EFCI, RR, and ER markings as specified in the ATM
forum traffic management specification.
It is another object of the present invention to provide a method of achieving high utilization, low cell loss, and MAX-MIN fair rate allocation through the existence of an asymptotically stable operating point.
It is further object of the present invention to provide a method of increasing the responsiveness to the network loading changes at multiple time scales, that is, at the cell level rate changes of the VBR and ABR VCs and at the cell level arrivals and departures of the VBR and ABR VCs.
It is yet another object of the present invention to provide a method of minimizing the number of operations required to compute an ABR service algorithm and to achieve low and scalable degrees of the implementation complexity, by virtually removing per-VC operations including per-VC queuing, per-VC accounting, and per-VC
table access.
The above object of the present invention can be achieved by providing a fair flow controlling method in a packet switching network. The packet switching network includes a plurality of nodes and each node is connected to a plurality of sources which transmits/receives data, and includes a data queue related with a current queue length and a target queue length for storing data received from the sources. In the fair flow controlling method, each node estimates the number of locally bottlenecked VCs using the ER and the MCR to be guaranteed of a corresponding source. The node allocates an ER to each source based on the difference between the current queue length and the target queue length of the node, the derivative of the current queue length, and the estimated number of the locally bottlenecked VCs, then conveys the ER to each source through a feedback signal.
BRIEF DESCRIPTION OF THE DRAWINGS
The above and other objects, features, and advantages of the present invention will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which:
FIG. 1 illustrates a network configuration to provide the ABR service;
FIG. 2 illustrates fields in an RM cell;
FIG. 3 schematically illustrates a switch fabric for the ABR service having I/O
ports connected to I/O cards;
FIG. 4 is a detailed block diagram of an I/O card shown in FIG. 3;
FIG. 5 is a block diagram of an ABR service engine according to the preferred embodiment of the present invention;
FIG. 6 illustrates a network model with a node of interest;
FIG. 7 is a graph showing a stable region with respect to U and V according to the preferred embodiment of the present invention;
FIG. 8 illustrates an asymptotic decay rate a as a function of U and V;
FIG. 9 illustrates queue length thresholds for ER marking and RR marking;
FIG. 10 illustrates a peer to peer configuration;
FIGS. I lA to 11D illustrate simulation results in the peer to peer configuration with ER marking only and with no VBR background traffic, wherein FIG. 11A
shows a source transmission rate ai(t) of VCs with PCR = ISOMbps; FIG. 11B shows a source transmission rate ai(t) of VCs with PCR = 25Mbps; FIG. 11 C shows a queue length at a switch SWI; and FIG. I1D shows the estimated number ~Q~avg(t) of locally bottlenecked VCs at the switch SW1;
FIGS. 12A to 12D illustrate simulation results in the peer to peer configuration with ER marking and VBR background traffc, wherein FIG. 12A shows a source transmission rate ai(t) of VCs with PCR = 150Mbps; FIG. 12B shows a source transmission rate ai(t) of VCs with PCR = 25Mbps; FIG. 12C shows a queue length at the switch SW1; and FIG. 12D shows the estimated number ~Q~avg(t) of locally bottlenecked VCs at the switch SWI;
FIG. 13 illustrates a parking lot configuration;
FIGs. 14A to 14F illustrate simulation results in the parking lot configuration with ER marking only and no VBR background traffic, wherein FIG. 14A shows a source transmission rate ai(t) of VCs with PCR = 150Mbps; FIG. 14B shows a source transmission rate ai(t) of VCs with PCR = 25Mbps; FIG. 14C shows a queue length at a switch SW3; FIG. 14D shows a queue length at a switch SW4; FIG. 14E shows the estimated number ~Q~avg(t) of locally bottlenecked VCs at the switch SW3; and FIG. 14F
shows the estimated number (Q~avg{t) of locally bottlenecked VCs at the switch SW4;
FIGS. I SA to 15F illustrate simulation results in the parking lot configuration with the ER marking and VBR background traffic, wherein FIG. 15A shows a source transmission rate ai(t) of VCs with PCR = 150Mbps; FIG. 15B shows a source transmission rate ai(t) of VCs with PCR = 25Mbps; FIG. 15C shows a queue length at the switch SW3; FIG. 14D shows a queue length at the switch SW4; FIG. I SE
shows the estimated number ~Q~avg(t) of locally bottlenecked VCs at the switch SW3; and FIG. 15F
shows the estimated number ~Q~avg(t) of locally bottlenecked VCs at the switch SW4;
and FIG. 16 is a flowchart illustrating the ER allocation control operation in the ABR service engine according to the preferred embodiment of the present invention.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT
A preferred embodiment of the present invention will be described hereinbelow with reference to the accompanying drawings. For the purpose of clarity, well-known functions or constructions are not described in detail as they would obscure the invention in unnecessary detail.
FIG. 1 illustrates a network configuration for implementing the ABR service.
With reference to FIG. 1, a network 2 for providing the ABR service includes a plurality of nodes 4, 6, 8, and 10. The nodes conceptually represent switches and will be referred _g_ to as switches hereinafter. Each switch is connected to a plurality of sources. In the drawing, the switch 4 is connected to sources A and C as indicated by reference numerals 12 and 14. The switch 8 is connected to sources B and D as indicated by reference numerals 16 and 18. Each source transmits/receives data through the switch 4 or 8 connected to the source. Data transmitted from the source reaches a destination through a so-called VC path having a plurality of nodes. For example, data from the source A 12 reaches the destination, the source B I 6, through a VC path having the switches 4, 6, and 8.
In the ABR service, the bandwidth availability of the network is conveyed to the source through special cells called RM (Resource Management) cells. In FIG. 1, RMc denotes an RM cell. While RM cells are generated in the sources I2, 14, 16, and 18 or the switches 4, 6, 8, and 10, the following description is confined to a source-generated RM cell.
A source-generated RM cell is transmitted to a destination through the switches 4, 6, and 8 through a VC path. The transmission direction of the cell is forward. Upon receiving the forward RM cell, the destination returns the cell (backward) hack to the source. In FIG. l, reference numerals 12 and 14 denote sources, and reference numerals 16 and 18 denote destinations. With respect to the sources and destinations, the forward and backward directions are defined as shown. The switches 8, 6, and 4 write their allowed bandwidth information in the backward RM cell. The source 12 or 14 adapts its rate according to the changing network conditions based on the received bandwidth information.
An RM cell contains information that includes fields of CCR (Current Cell Rate), MCR (Minimum Cell Rate), ER, NI (No Increase), and CI (Congestion Indication), as shown in FIG. 2. In FIG. 2, an S-R (Source-Receive) field provides information related to a source and a destination information. The CCR field is set by the source to its current ACR (Allowed Cell Rate) when the source generates the RM cell. The MCR
field indicates the minimum bandwidth allocated to each VC when the source generates the RM cell. ER is an available bandwidth written in a backward source-generated RM
cell by the ABR service engine of a switch as the RM cell passes through the switch.
Only if a calculated available bandwidth of the ABR service engine is less than the existing available bandwidth, the former value is written in the ER field.
Thus, the smallest available bandwidth in the VC path is conveyed to the source. To this end, the source writes its PCR in the ER field when it generates the RN cell. CI and M
are fields used to control RR. CI notifies the source that the network is much congested and thus WO 01/26305 PCTlKR00101098 the bandwidth of the source is to be reduced, and IVI is used to prevent a source from increasing its ACR.
A switch calculates a bandwidth available for the ABR service and writes the available bandwidth information in a backward RM cell, using a switch algorithm. What is intended from the switch algorithm is to obtain a bandwidth allowable to a corresponding VC by the switch. This available bandwidth is conveyed to the source so that the source adapts its rate to provide the ABR service reliably.
As shown in FIG. 3, a switch fabric 20 has I/O ports each connected to an I/O
port card 22 in each switch shown in FIG. 1. Each I/O port card includes an I/O buffer management unit 30, an ABR service engine 32, and an output interface 34, as shown in FIG. 4. The I/O buffer management unit 30 is connected to the switch fabric 20 and responsible for I/O queuing. The I/O buffer management unit 30 includes ABR
queues 36. The ABR service engine 32 implements an ABR algorithm and its related operations for the ABR service while an RM cell is passing through the ABR engine 32 according to the embodiment of the present invention. The output interface 34 acts as a user network interface at the ATM layer.
FIG. 5 is a block diagram of the ABR service engine 32 shown in FIG. 4. With reference to FIG. 5, the ABR service engine 32 is comprised of an EFCI marker 40, a ~Q~
estimation unit 42, an ER engine 44, an average queue calculator 46, and a backward RM
cell writer 48.
At EFCI congestion, the EFCI marker 40 marks an EFCI bit in an input forward data cell to indicate the EFCI congestion. A queue read signal is generated when a cell is removed from a queue and a queue write signal is generated when a new cell enters the queue. Upon receipt of the queue write signal, the average queue calculator 46 increases a queue instantaneous variable by one, and upon receipt of the queue read signal, it decreases the queue instantaneous variable by one, to calculate an average queue length within a predetermined period interval for the ER computation. The average queue calculator 46 feeds the average queue length q[k] to the ER engine 44. The average queue calculator 46 decides as to whether the EFCI congestion has occurred using the queue read signal and the queue write signal and in case of the EFCI
congestion, it outputs an EFCI congestion signal EFCI~CG to the EFCI marker 40. According to the embodiment of the present invention, the average queue calculator 46 also outputs a signal CG representative of "congested" condition and a signal VCG
representative of "very congested" condition to the backward RM cell writer 48. The congested state and WO 01/26305 PCTlKR00/01098 the very congested state are determined based on a predetermined lowest queue length threshold qLT and a predetermined highest queue length threshold qHT for an ABR
queue. The signals CG and VCG are used to control RR according to the embodiment of the present invention.
S
The ~Q~ estimation unit 42 estimates the number of locally bottlenecked VCs by reading CCR and MCR from an RM cell and comparing CCR-MCR with a periodically calculated ER r(t). Interval, W, at which the estimated value IQ is calculated will be described later in detail in "(5) Discrete Time ER Algorithm and ~Q~
Estimation". The IQ
is provided to the ER engine 44.
The ER engine 44 updates the ER through a periodical ER calculation and outputs the latest ER r(t) to the backward RM cell writer 48 upon arrival of a backward RM cell.
The backward RM cell writer 48 writes r(t) in the ER field of the backward RM
cell. More specifically, the backward RM cell writer 48 compares the ER of the received RM cell with r(t) + the MCR of the RM cell (r(t)+mi) and writes r(t)+mi in the RM cell only when r(t)+mi is less than the ER of the RM cell. The backward RM cell writer 48 writes a binary logic state bit in the 1\TI and CI fields of the RM cell according to the signals CG and VCG received from the average queue calculator 46, for control of RR.
FIG. 16 is a flowchart illustrating an ER allocation control operation in the ABR
seance engine.
With reference to FIGS. 5 and 16, the ~Q~ estimation unit 42 estimates Q in step 100 and the ER engine 44 updates ER through a periodical calculation in step 102. Upon arrival of an RM cell in the backward RM cell writer 48 in step 104, the ABR
service engine 32 proceeds to step 106; otherwise, it returns to step 100. In step 106, the backward RM cell writer 48 reads the latest ER r(t) from the ER engine 44 and the MCR
mi from the received RM cell. The backward RM cell writer 48 calculates an ER
allocation value ri f t) for VCi (an i"' VC) by r(t)+mi in step 108 and writes ri f t) in the ER
field of the backward RM cell in step 110.
The purpose of the switch algorithm suggested in the present invention is to converge the AVR queue 36 to a target queue length qT and maintain a queue length at a WO 01126305 PCTlKR00/01098 steady state at the target queue length qT, which implies that the input traffic is equal to the output traffic of the ABR queue 36. A transient period is generated in the process of achieving the target queue length qT. In case the input traffic is momentarily greater than the output traffic, the ABR queue 36 is overloaded. As a result, unexpected cell loss might occur. Accordingly, a switch algorithm is implemented considering the buffer capacity and queue length variation of the ABR queue 36 in the embodiment of the present invention.
In order to keep a controller as simple as possible and at the same time remove undesirable non-equilibrium, a continuous-time ER allocation algorithm is proposed in accordance with the embodiment of the present invention, r(t) = A g(t) B (q(t) - qr), A, B > 0 . . . . . (4) where r(t) is an ER calculated by the ER engine, >~(t) is the derivative of r(t), Q is the set of locally bottlenecked VCs, ~Q~ is the cardinality of Q, q(t) is the length of the ABR
queue 36 at a time t, and A and B are controller gains that are varied depending on the length of the ABR queue 36 for asymptotic stability in the embodiment of the present invention.
It is to be noted that the ER allocation algorithm (4) uses q(t) instead of r(t) in the damping term (the first term in the right-hand side) as compared to the algorithm (2).
q(t) is the derivative of q(t), which will be described in relation with Eq.
(21) and Eq.
(23).
This change indeed removes the aforementioned undesirable non-equilibrium in a closed-loop system; consequently, the ABR queue 36 always converges to the target queue length qT irrespective of the available bandwidth and the fraction of the available bandwidth used by remotely bottlenecked VCs, which implies that full utilization of the available bandwidth is always guaranteed in the steady state.
Another notable feature of the proposed algorithm is the normalization of the controller gains A and B by the number of locally bottlenecked VCs, ~Q~. This normalization makes the asymptotic decay rate of the closed-loop system to be independent of (Q~.

Estimation of ~Q~ has been an important research subject in the rate-based ABR
flow control research (for reference, see M. K. Wong and F. Bonomi, "A Novel Explicit Rate Congestion Control Algorithm", it Proc. IEEE GLOBECOM'98, vol. 4, pp.

2439, 1998, L. Kalampoukas, A. Varma and K. K. Ramarkrishnan, "An Efficient Rate Allocation Algorithm for ATM Networks Providing MAX-MIN Fairness", Technical Report UCSC-CRL-95-29, Computer Engineering Dept., University of California, Santa Cruz, June 1995, A Charny, K. K. Ramakrishnam and A. Lauck, "Time Scale Analysis and Scalability Issue for Explicit Rate Allocation in ATM Networks", IEEE/ACM
Trans.
On Networking, vol. 4, pp. 569-581, 1996, and R. Jain et. al., "ERICA Switch Algorithm: A Complete Description", ATM Forum/96-1172, 1996).
The difficulty involved in that ~Q~ estimation lies in that the dynamics of any ~Q~
estimation process are coupled with those of the ER allocation process, i.e., ER updates affect ~Q~ updates and vice versa until the closed-loop system reaches the steady state so that improper design of an ~Q~ estimation algorithm can make the closed-loop system unstable. In view of the foregoing, a stable yet scalable ~Q~ estimation algorithm is proposed in relation with the ER allocation algorithm according to the embodiment of the present invention.
The control-theoretic ER allocation algorithm (4) of the present invention achieves MAX-MIN fair rate allocation according to the principle as described below.
An identical ER (common ER) is allocated to all VCs, based on Eq. (4), sharing the same link. Thus, the minimum allocated ER in the path is conveyed to a source through an RM
cell and the source transmits data at the minimum ER. If other VCs in a different location are bottlenecked and the source transmits data at a rate less than the switch-allocated common ER, the ABR queue 36 will experience time delay and the delay will be decayed.
In case the ABR queue 36 gets below the target queue length qT, the switch increases the common ER until the ABR queue 36 reaches the target queue length qT in the algorithm (4). Therefore, each locally bottlenecked VC can get a fair share of the bandwidth that is not used by remotely bottlenecked VCs.
The algorithm (4) can be approximated to the following discrete time algorithm when it applies in reality.
r[k + 1] = r[k]- ~ (g[k]- q[k - 1]) - ~ (q[k]- gT), A, B > 0 . . . . . (5) where T is the duration of a discrete time slot. The algorithm (5) is a special case of the algorithm ( 1 }. That is, with I= I , a I = - ~ , a0+a 1= ~ , and pj=0, 'dj, the algorithm ( I ) is reduced to the algorithm (5). In contrast to the algorithm (1), the reduced algorithm no longer allows for arbitrary control of the closed-loop dynamics. However, the present inventors argue that having the capability of arbitrary control is too expensive in terms of implementation cost and may not be necessary for the ABR flow control design.
To support this argument, it will be shown that the reduced algorithm (5) in fact allows for an acceptable level of control for the closed-loop performance.
IO In the control-theoretic ER allocation algorithms (4) and (S), the queue length control is a primary concern, and the fair rate allocation is a by-product of the queue length control. There is another class of ER allocation algorithm ( 14) to ( 18) where the fair rate allocation is a primary concern, and the queue length control, if any, is supplementary. In this type of algorithms, it is necessary for each switch to track the available bandwidth, the fair share of remotely bottlenecked VCs at their own bottleneck link, and the number of locally bottlenecked VCs. Based on this information, the switch has to update per-VC ER allocation in such a way that the MAX-MIN fair rate allocation and target link utilization are asymptotically achieved.
( 1 ) ER Rate Flow Control The ER allocation algorithm of the present invention will be modeled mathematically and it will be shown that the ER allocation algorithm guarantees an MCR
required by each source and at the same time satisfies the MAX-MIN fairness.
FIG. 6 illustrates a network model with a node of interest. In FIG. 6, a plurality of VCs are matched with a plurality of corresponding sources, VCi 54 to source i 50 and VCj 56 to source j 52. Reference numeral 36 denotes the ABR queue, qT is the target queue length, and p denotes the available bandwidth of a link shared by the plurality of VCs. Tif and lib are forward and backward path delays, respectively, in VCi 54, and the sum of Tif and lib is the RTD ii of VCi 54. Tjf and ijb are forward and backward path delays, respectively, in VCi 56, and the sum of tjf and tjb is the RTD ij of VCi 56.
The network model is analyzed based on the following assumptions:
A.1. Traffic is viewed as a deterministic fluid flow, and the network queuing process and the flow control mechanism are continuous in time. This assumption enables the closed-loop system to be modeled by a differential equation.
A.2. The RTD zi of VCi 54 is the sum of the forward path delay iif and the backward path delay Tib, which includes propagation, queuing, transmission, and processing time. Here, the RTD is assumed to be constant.
A.3. The sources are persistent until the system reaches a steady state. The term "persistent" means that the sources always have enough data to transmit at the allocated rate.
A.4. There are no arrivals and departures of VCs until the system reaches the steady state.
A.S. The available bandwidth ~ at the link is constant until the system reaches the steady state. Also, the buffer size at the link is assumed infinite.
Let ai(t) and ri f t) respectively denote the rate at which source i transmits data at source time t and the ER of VCi computed by the node of interest at node time t. Also, let (3i(t) and pi(t) respectively denote the latest minimum of ERs allocated to VCi by the nodes along the VCi's path, except the one allocated by the node of interest and the PCR
constraint of VCi.
Neglecting the linear increase and exponential decrease operation based on binary feedbacks, the source behavior can be modeled as:
ai(t) = min[ri(t-iib), bi(t), piJ, di E N . . . . . (6), where N is the set of all the VCs whose route includes the node of interest.
This model implies that source i 54 transmits data at the smallest value among an ER
allocated by the node of interest before the backward path delay Tib, ri f t-Tib), a minimum ER
on the route except the node of interest, bi(t), and the PCR of the VC, pi.
The dynamics of the per-class ABR queue 36 of interest are given by:

q(t ) _ ~'EN a~ (l - zf ) - f~ ; q(t) > 0 . . . . . . (7).
[~~Erra~(i- zr )-f~] ~ q(t)= 0 The ER allocation algorithm according to the embodiment of the present invention is a distributed algorithm which runs at each switch based on the current network state including the queue length q(t), the derivative of the queue length, q(t) , and the estimate of the number of locally bottlenecked VCs, Q , given by:
ri f t) = r(t) + mi, 'di E N . . . . . (8), and - Q q(~) - Q (R'(t ) - qr )~ r(t ) > 0 r(t) = 1 1 I I . . . . . (9), ~- A q(t)- Q (R'(t)- 9T)~+~ r(r) = 0 IQI I
where A, B > 0 and mi denotes the MCR which the node is required to guarantee during the entire life of VCi. It is assumed that mi _< pi, di E N, and there exists a call admission control which guarantees the following condition:
m; < ,u . . . . . ( 10), BEN
where r(t) is the common part ~/i of per-VC ER allocations, ri f t) and requires most of the computation of the algorithm. The only per-VC computation required is the addition of mi to the common ER, r(t) according to the embodiment of the present invention. This is why this algorithm is scalable in terms of computation complexity with an increasing number of VCs. Each RM cell of VCi 54 carries the value mi in the round trip.
The embodiment of the present invention is characterized in that the node keeps updating the common ER in the background. "Background" calculation refers to the calculation of the common ER periodically regardless of the arrival of RM
cells. The benefit of the background calculation is that the latest common ER, r(t) that is preliminarily updated is directly provided upon arrival of an RM cell in the corresponding node.

The node, that has updated the common ER, r(t) according to Eq. (9) by background calculation, reads mi from the MCR field of a passing-by RM cell in VCi 54, calculates an ER ri f t) to be allocated to VCi 54 by adding mi to the latest common ER, r(t) in Eq. (8), and writes ri f t) in the ER field of the RM cell.
Another notable feature of the ER allocation algorithm according to the embodiment of the present invention is the normalization of the controller gains, A and B
according to the estimate of the number of locally bottlenecked VCs, ~QI. The normalization is optional, that is, it is not absolutely necessary but it is recommended since, as it will be described in "(4) Principal Root and Asymptotic Decay Rate", it makes the asymptotic decay rate of the closed-loop system to be independent of the number of locally bottlenecked VCs.
The terms, remotely bottlenecked VC and locally bottlenecked VC, are defined in the steady state for a given network loading. Remotely bottlenecked VCs in a given link are defined as those VCs which cannot achieve their fair share at the given link because either their transfer rate is limited by their PCR or they represent bottlenecked at some other link in the path. In the same manner, locally bottlenecked VCs at a link are defined as those VCs which achieve their fair share at the given link.

Let ais = lima~(t), ris = limr(t), and bis = limbl(t) in order to define the locally bottlenecked VCs and the remotely bottlenecked VCs in mathematical terms.
Here, ais, ris, and bis represent, respectively, ai(t), ri(t), and bi(t) in the steady state.
Then, the set of all the locally bottlenecked VCs, Q is given by:
Q = {iii E N and ais = risk . . . . . (11), and the set of all the remotely bottlenecked VCs, N-Q, is given by:
N-Q = {iii E N and ais = min[bis, pi]~ . . . . . (12).
In "(5) Discrete-Time ER Algorithm and ~Q~ Estimation", a ~Q( estimation algorithm is presented which is not only effective in that IQI rapidly converges to ~Q~, but also robust in that IQI tends to be greater than ~Q~ during the transient period. Such a robustness property is necessary for the stability of the closed-loop control.

(2) Steady State and Fairness A description will be made of the steady state characteristics of the closed-loop dynamics when the ER allocation algorithm according to the embodiment of the present invention is applied. That is, there will be given analysis result according to the embodiment of the present invention. It is assumed that the closed-loop dynamics have an equilibrium point at which the derivatives of the system variables are zero, that is, li~mq(t) = 0 and li~mr(t) = 0. Then, from Eq. (6), Eq. (8), and Eq. (9), ais - min[ris, bis, pi], 'diE N, . . . . . (13), ris = rs + mi, diE N, . . . . . (14), and qs = qT . . . . . (15), where qs =limq(t) and the other notations were defined previously. Since qs=qT
> 0, r--~ ~o Eq. ( 17) implies that ~ai.s - ~ ~ . . . . (1V).
BEN
By combining Eq. ( 13 ), Eq. ( 14), and Eq. ( 16) and the definitions of Eq. ( 11 ) and Eq. (12), the following Eq. (17) is obtained:
~rs+ ~m, + min[b;S,P~]= ~ . . . . . (17), iEQ iEQ iEN-Q
which implies that ~ - ~rEN min[bs,Pr]- ~ Qm -Q ~E _ . (Ig).
rs IQI
As described above, the embodiment of the present invention has the following characteristics; and a result, Eq. (19) is obtained.

For ~;EN m; < ~ , there exists a unique steady state solution (equilibrium point) at which (i) the queue length is equal to the target queue length (qs=qT), (ii) the available bandwidth at the link is fully utilized (~~ENa;., _ ~ ), and (iii) individual MCRs are guaranteed at the link and the bandwidth subtracted by the sum of MCRs, ~ -~~ENm; , is fairly shared in the MAX-MIN fair sense. That is, ~- ~rEN-Qmm[b;S,P~]- ~;EQm~
+m iEQ
. (19).
aa.s= ....
min[b;s, p; ], i E N - O
The above characteristics imply that when the ER allocation algorithm according to the embodiment of the present invention is applied, the ABR closed-loop system has a unique oscillation-free operating point at which the MAX-MIN fairness with NCR
guarantee is achieved, and the queue length is equal to the target value qT no matter what the network loading is. That is, an equilibrium point is independent of the available bandwidth for the ABR traffic and the fraction of the available bandwidth utilized by remotely bottlenecked VCs including PCR-constrained VCs.
A notable point about the embodiment of the present invention is that the ER
allocation algorithm results in the unique equilibrium point but the computation is very simple. The ER algorithm requires q(t) and q(I) only, not any other measurement or monitoring result. The estimate IQ , used to normalize the gains A and B, is introduced to accelerate the convergence of the queue length to the target queue length and has little to do with the equilibrium point. The Benmohamed's algorithm in Eq. (1) also results in a unique system equilibrium point with the same properties as the ER
allocation algorithm of the present invention but at the cost of high-degree implementation complexity. In contrast, if the algorithms in Eq. (2) and Eq. (3) are applied, the closed-loop system possesses an equilibrium point that varies depending on the available bandwidth and the fraction of the available bandwidth utilized by the remotely bottlenecked VCs and can converge to zero at worst scenario. For details, refer to S. Chong, Nagarijan, and Y. T.
Wang, "Design Stable ABR Flow Control with Rate Feedback and Open-Loop Control:
First-Order Control Case", Performance Evaluation, vol. 34, no. 4, pp. 189-206, 1998, S.
Chong, "Second-Order Rate-Based Flow Control with Dynamic Queue Threshold for High-Speed Wide-Area ATM Networks", preprint 1997, and A. Elwalid, "Analysis of Adaptive Rate-Based Congestion Control for High-Speed Wide-Area Networks", Proc.

IEEE ICC'95, pp. 1948-1953, 1995.
(3) Asymptotic Stability In this part, there will be given a description of the stability of the ER
allocation algorithm according to the embodiment of the present invention. In general, the stability properties of the equilibrium point given in Eq. ( 19) could be investigated in the case of multiple nodes. However, due to the complex nature of the coupled dynamics between nodes, the analysis could be so complicated that there is no attempt to solve the global stability problem in the coupled mufti-node setting in the embodiment of the present invention. This will be only investigated by simulations in "(6) Simulation Results". On the other hand, Benmohamed and Meerkov showed that under a special service discipline there exists a neighborhood of the equilibrium point in which node-to-node dynamics are decoupled ("Feedback Control of Congestion in Packet Switching Networks: The Case of Multiple Congested Nodes", International Journal of Communication System, vol. 10, no. 5, pp. 227-246, 1997). Moreover, they showed through simulations that the local stability condition derived in the neighborhood works well for the FCFS
service discipline as well. By appealing to this result, it is assumed that a neighborhood R with such properties exists in the embodiment of the present invention. A subset of the neighborhood R is considered, which satisfies:
a) bi(t) = bis, dI E N, i.e., the dynamics of the other nodes are in a steady state;
b) {iii E N and ai(t) = rift-rib)} = Q and {iii E N and ai(t) = min[bis, pi]}
= N
Q, i. e., the locally bottlenecked VCs transmit data at ri f t-rib) and the remotely bottlenecked VCs transmit data at min[bis, pi];
c) the saturation nonlinearities in Eq. (7) and Eq. (8) are not activated, i.e., both q(t) and r(t) are positive-valuef; and d) the ~Q~ estimation process is in the steady state, i.e., QI is constant.
In this neighborhood of the equilibrium point, the dynamic equations (6), (7), and (8) can be simplified to:
r, (t-z;b), i~Q
( 20 a; 1 ) min[b;s, p; ], i E N - p ~ . . . . ( ) g(t)= ~ a; (t- z;f )-~ . . . . . (21), ~ E ~1' and i'(t ) _ - O q(t ) - 0 (q(t ) - qr ) . . . . . (22).
By combining Eq. (20) and Eq. (21 ), q(t)= ~ min[bis,pi]+~r,.(1-z;)-~ .....(23), iEN-Q ieQ
Let an error function be defined as e(t) = q(t)-qT. BY combining Eq. (22), the differentiation of Eq. (23), and the differentiation of Eq. (8), the following closed-loop equation is obtained:
e(t ) + A ~ e(t - zi ) + B ~ e(t - ri ) = 0 . . . . . (24), IOI iEQ IO iEQ
which is a second-order retarded differential equation. The characteristic equation of the I S closed-loop equation is given by:
A _B
D(s) - s2 + /1 ~ se-°'~ + ~ e-Sr~ = 0 . . . . . (25), yI iEQ IQ iEQ
which has an infinite number of roots. For this asymptotic stability of the closed-loop equation (24), all the roots of the characteristic equation (25) must have negative real parts (see R. Bellman and K. L: Cooke, "Differential-Difference Equations", Academic Press, New York, 1963 and G. Stepan, "Retarded Dynamical Systems: Stability and Characteristic Functions", Longman Scientific & Technical, 1998).
To find the necessary and sufficient condition for D(s) = 0, one can appeal to Pontryagin's criterion assuming the discrete delays of rational ratios (see R.
Bellman and K. L:. Cooke, "Differential-Difference Equations", Academic Press, New York, 1963 and S. J. Bhatt and C. S. Hsu, "Stability Criteria for Second-Order Dynamical Systems with Time Lag", Journal of Applied Mechanics, pp. 113-118, 1996). For general cases with continuous delays or discrete delays of irrational ratios, Stepan's criterion provides a way to construct the necessary and sufficient condition (see "Retarded Dynamical Systems:
Stability and Characteristic Functions", Longman Scientific & Technical, 1989).
However, constructing such a condition in an explicit form is extremely complicated particularly for the case with a large number of heterogeneous RTDs.
Instead, the necessary and sufficient condition for the asymptotic stability is derived in the case that all the RTDs are identical. Let zi = T, di. Then, the closed-loop equation 924) becomes:
e(t)+ ~~~ Ae(1- z)+ ~~ Be(t - z) = 0 . . . . . (26).
This new equation may be normalized as the time lag i becomes unity. Let t =
T~. In terms of the new variable ~, Eq. (26) becomes:
e(~) + Ue(C - 1) + he(~ - 1) = 0 . . . . . (27), where U = IQI A z and V = ~ B z'' . The characteristic equation of Eq. (27) is:
H(z) = z2ez + Uz + V = 0 . . . . . (28).
To find the necessary and sufficient condition that all the roots of H(z) = 0 have negative real parts, one can appeal to the aforementioned Pontrygin's criterion.
O
The above theorem yields the following result. Let U = ~ ~~ Az and V =
Q
~~~ B z ' . Then, the closed-loop equation (26) is asymptotically stable only if:
Q
0< U<-0 < V < ~,2 I- (~ )Z . . . . . (29), where ~ 1 is the unique solution of U = c~sinw in the interval (0, 2 ). From Eq. (29), the controller gains A and B can be set. FIG. 7 illustrates a stable region with respect to U

and V according to the embodiment of the present invention.
(4) Principal Root and Asymptotic Decay Rate In this part, there will be described determination of the rate at which the stable closed-loop system approaches a steady state. Any solution to the normalized closed-loop equation (27) can be represented by series:
e(~) _ ~ Pn(~)eZn' . . . . . (30) n=1 where pn(~) is a suitable polynomial and zn, do are the roots of the corresponding characteristic equation (28). Consider the principal root, denoted by z*, which is the root having the largest real part. Let z~ _ -a ~ j(3, a > 0, ~i > 0. It follows from Eq. (30) that e(~) ~ Ce'~E . . . . . (31) where C is a constant depending on the initial conditions of Eq. (27) and x(~)~y(~) indicates that x(~) asymptotically approaches 1. Also from Eq. (30), .v(~) ( e(~) <_ ce-a' for large ~ . . . . . (32) where ~~y) denotes the Euclidean norm and c is a constant depending on the initial conditions of Eq. (27). In terms of the original variable t(=r~), Eq. (32) can be rewritten as, _Q
e(t)li <_ ce r 1 for large t . . . . . (33).
Note that a is the asymptotic decay rate at which the original system tends to T
the equilibrium point. Hence the inverse of it, a , is the time constant of the original T
closed-loop system, i.e., the time it takes for a small perturbation around the equilibrium point to decrease by a factor of a '. Similarly, a and a ' are the asymptotic decay rate and the time constant of the normalized system.
FIG. 8 illustrates the asymptotic decay rate a as a function of U and V With reference to FIG. 8, the asymptotic decay rate a is a concave function with respect to U
and V with its maximum being approximately 0.3 at (U, V) _ (0, 6, 0, 1 ). The contour line at a corresponds to the boundary of the stable region shown in FIG. 7.
Once a is determined for a given (U, V) pair, (3 can readily be determined by substituting z = -a +
j(3 in the characteristic equation (28) and equating the real and imaginary parts.
(S) Discrete-Time ER Algorithm and ~Q~ Estimation A continuous-time model of the MAX-MIN flow control problem has been dealt with so far. In reality, however, feedback information is relayed in RM cells, and thus available not in continuous time, but rather in a sampled form. Moreover, the feedback information is not periodic because the RM cells themselves have to complete the link bandwidth along the round trip path so that the inter-arrival time of the feedback RM
cells are time-varying.
The ER allocation algorithm expressed in Eq. (8) and Eq. (9) can be implemented at a switch in discrete time as follows: Update the common ER
periodically with a period T by:
r[k+ I]= [r[k]- ~ (q[k]- 9[k- 1])- O (q[k]- qT)]', A,B > 0 . . . . (34), where q[k] denotes the low-pass filtered queue length, i.e., an average queue length.
Particularly, a periodic averaging filter is used in order to obtain q[k] in the embodiment of the present invention such that q [k] = T f ~kT ~~T q(t' )dt' .
It is to be noted that the ER update equation (34) corresponds to Eq. {9) as T
~
oo. In contrast to the periodic computation r[k+1] of the common part of ER, r{t), per-VC ER allocation is performed periodically upon arrival of the corresponding RM cells in either a forward direction or a backward direction. That is, upon arrival of a VCi's RM
cell at time t, the switch computes:

ri f t) = r(t) + mi . . . . . (35), and writes this result on that RM cell. In Eq. (35), r(t) denotes the latest value, r[k+1], of the common ER, r[k], which is being updated periodically in the background according to the embodiment of the present invention. The value of mi is available from the RM cell that has arrived or the per-VC MCR table being maintained in the switch and updated upon arrival or departure of an RM cell. If it is determined that the value of mi is to be taken from the RM cell that has arrived, the per-VC MCR table and the access to it are no longer necessary. Therefore, the only per-VC operation required in the ER
allocation algorithm of the present invention is the addition in Eq. (35).
Meanwhile, many schemes have been proposed for estimating or tracking the number of locally bottlenecked VCs. Some of those schemes can be encountered in M. K.
Wong and F. Bonomi, "A Novel Explicit Rate Congestion Control Algorithm", it Proc.
IEEE GLOBECOM'98, vol. 4, pp. 2432-2439, 1998, L. Kalampoukas, A. Varma and K.
K. Ramarkrishnan, "An Efficient Rate Allocation Algorithm for ATM Networks Providing MAX-MIN Fairness", Technical Report UCSC-CRL-95-29, Computer Engineering Dept., University of California, Santa Cruz, June 1995, A Charny, K. K.
Ramakrishnam and A. Lauck, "Time Scale Analysis and Scalability Issue for Explicit Rate Allocation in ATM Networks", IEEE/ACM Trans. On Networking, vol. 4, pp. 569-581, 1996, and R. Jain et. al., "ERICA Switch Algorithm: A Complete Description", ATM
Forum/96-1172, 1996. Each of the above schemes varies in the degree of implementation complexity. The basic idea of Su et. al.'s algorithm (in C. F. Su, G. de Veciana, and J.
Warlrand, "Explicit Rate Flow Control for ABR Services in ATM Networks", preprint, 1997), which estimates the number of "on" sources sharing a link, is attractive since it does not require per-VC accounting.
The above algorithm is modified to estimate the number of locally bottlenecked VCs without doing per-VC accounting in the embodiment of the present invention.
Suppose that a j"' RM cell arrives at a switch at a switch time t'. According to the ABR specification, if the j'k' RM cell happens to be an RM cell of VCi, it carries the value ai(t'-iif) in the CCR field and the value mi in the MCR field. The switch monitors the RM cell arrivals in a synchronous fashion over fixed length intervals of W
seconds.
For an I'" interval, the number of locally bottlenecked VCs can be approximated by NRM+ 1. 1{CCR(t')- MCR(t') >_ S~r(t')}, 0< S < 1 . . . . . (36), t~ e(JW,(!+1)W) W ~ CCR(t ) where 1 { ~ } is the indicator function, CCR(t') and MCR(t') respectively denote the value in the CCR field and the value of the MCR field of the j'" RM cell, and r(t') is the latest value of the common ER at time t'. Upon arrival of the j'" RM cell, if the CCR
subtracted S by the MCR is greater than or equal to the latest value of the common ER at the switch, the VC to which the j'" RM cell belongs is counted as a locally bottlenecked VC.
Otherwise, it is treated as a remotely bottlenecked VC. 8 is the margin to avoid the underestimation of the number of locally bottlenecked VCs particularly near the steady state. As the system approaches the steady state, the CCR of a locally bottlenecked VC
stays around the sum of the MCR and the common ER. Thus, without the margin b the VC could be counted wrongly as a remotely bottlenecked VC even for a small perturbation in the CCR. By having this margin, however, this type of underestimation can effectively be avoided. Through simulations, it is found out that 8 = 0.9 is the recommended choice. Also, it is to be noted that the value of the indicator function is normalized by the expected number of RM cell arrivals of the VC within W
seconds, W~CCR
NRM+ 1, so that the summation of these values over a W-second interval gives a correct estimate of the number of locally bottlenecked VCs. Based on this estimate for each interval, the recursive estimate is computed at the end of every interval by:
QI~((I+ 1)W)= satiNi[~,IQI~(lT~+ {1- ~.)QI,], 0< ~. < 1 . . . . . (37) and QI ({l + 1)W) = int[IQI~ {(l + 1) W)] . . . . . (38), where 7~ is an averaging factor, int[a] denotes the smallest positive integer greater than or equal to a, and the saturation function Bata[b] is defined as:
a, b > a satQ[b] = h~ otherwise ' ' ~ ~ ~ (39).
Note that 1 _< IO (t) _< ~N~ for all t. The actual number of locally bottlenecked VCs can be zero for some network loading but the value of its estimate is lower-bound intentionally to avoid the divisian by near-zero value.

Now the question is how to choose the interval W and the averaging factor ~,.
As the number of VCs sharing a link increases or the available bandwidth decreased for a given W, the inter-arrival time of RM cells of a VC increases so that the switch begins to fluctuate largely. To solve this problem, W may be adjusted according to the changes of the number of VCs sharing the link and the available bandwidth but this is not easy to implement. Instead, a large 7~ close to 1 can be selected in the hope that the periodic averaging operation in Eq. (37) will effectively filter out the variability of the ~Q~I.
Through simulations, it is found that ~, = 0.98 yields stable and effective estimation of ~Q~
for a wide range of number of VCs sharing a link and the available bandwidth, irrespective of the choice of W
We further improve the ~Q~ estimation algorithm for the stability purpose.
Suppose that the controller gains A and B are selected so that A = U , B = T . . . . . (40) and the (U, V) pair resides inside the stable region depicted in FIG. 7. This selection is made in such a way that the ~Q~ estimation is accurate enough that I Q' ~ ~QI
for all times.
However, since IOI ~ IQ in general, the actual or effective values of U and V, say, U' and V', governing the normalized closed-loop equation (27) are given by:
I~ ~ _ ~~ U, V ~ = I Q V . . . . . (41 ), by substituting Eq. (40) in Eq. (26). If IQI is less than 1, the point (U', V') resides Q
somewhere on the straight line connecting the point (U, V) and the origin (0, 0) and thus remains inside the stable region. In contrast, as ~=~ increases beyond 1, the point (U', Q
V') moves upward along the straight line including the origin (0, 0) and the point (U, V) and eventually gets out of the stable region. In short, overestimation of ~Q~
is tolerable since it does not affect the stability of the system but it only changes the asymptotic decay rate whereas underestimation of ~Q) should be avoided since it can make the system unstable. This is why the margin 8 is necessary in Eq. (36). However, 8 does not resolve the whole situations, particularly the situation that a new VC joins with a small initial cell rate (ICR). If a new VC joins with a small ICR, the VC is very likely to be counted as a remotely bottlenecked VC during the transient period since CCR(t')-MCR(t') is very likely to be less than 8~r(t') in Eq. 36 since the margin 8 is small enough.
In this state, if the new VC is actually a remotely bottlenecked VC, it is not a problem but if the new VC
is actually a locally bottlenecked VC, it results in an initial underestimation of ~QI, which could make the system diverge. To resolve this problem, the IQI estimation algorithm is improved as follows in the embodiment of the present invention.
Upon arrival of a new VC at time t, the estimate is updated on purpose by:
IQlavg(t) = satlNl [IQlavg(t)+1] . . . . . (42) and Q(t)= int[f Q~~(t)J . . . . . (43) as if the VC is a locally bottlenecked VC. By doing so, the initial underestimation of ~Q) is avoided so that IOI converges to IQI from Eq. (42) and Eq. (43), which is a highly desirable property for the system stability as stated before. The performance of the IQI
estimation algorithm proposed in conjunction with the ER allocation algorithm according to the embodiment of the present invention will be verified through simulations in "(6) Simulation Results".
Next, it will be considered the case where RR marking is applied in conjunction with ER marking. The ER marking plays the major role to achieve the MAX-MIN
fairness with MCR guarantee in an asymptotically stable manner whereas the RR
marking plays a supplementary role to limit the transient overshooting of the queue length and hence minimize the transient cell loss.
FIG. 9 illustrates the suggested design of the queue length thresholds for the ER
plus RR marking. In FIG. 9, qT is the target queue length, qLT is the lowest queue length threshold for the ABR queue, and qHT is the highest queue length threshold for the ABR queue. qT is related with ER marking and qLT and qHT are related with RR
marking. qLT is a threshold from which the NI bit of a NI field is set and qHT
is one from which the CI bit of a CI field. If the queue length is greater than qHT, the switch sets the CI bit of the backward RM cell to 1 to indicate its congestion. If the queue length is between qHT and qLT, the switch sets the NI bit of the backward RM
cell to 1 to prevent the source from increasing its bandwidth. By placing qT far below qLT, however, the RR marking is hardly activated since the queue length would stay in the neighborhood of qT by the control of the ER marking unless queue surge due to abrupt changes in the network loading occur. Once the queue length overshoots qHT by some loading changes, the linear increase and exponential decrease mode is triggered and remains active until the ER marking re-takes the control of the queue length.
In "(6) Simulation Results", it will be shown through simulations that the RR
marking effectively limits the worst-case queue length in the transient period and the ER
marking indeed takes the control back so that it recovers the queue length from the oscillatory mode to the asymptotically stable mode.
(6) Simulation Results In this part, simulation results will be presented to verify the above-described analysis and demonstrate the excellent performance of the ER allocation algorithm according to the embodiment of the present invention. The simulation model used is developed on the KIST ATM simulator platform.
Two different network topologies were considered: the peer-to-peer configuration and the parking lot configuration, which are fairly standard.
The recommended values for the design parameters in the switch algorithm suggested in the present invention are summarized in Table 1 to be used in the following simulation studies.
(Table 1 ) ER allocation algorithm ~Q~ estimation algorithm A B qT T W 8 0.6/tmax 0.1/imax 250cells 3200 0.9 0.98 (imax = max{Ti, i E N}, d = one cell transmission time) First the peer-to-peer configuration will be considered, shown in FIG. 10 where 20 ABR VCs having an identical path are contained and the capacity of the all links is set equally at 600Mbps. To represent a WAN environment, the distance between each of the sources from s 1 to s20 and a first switch SW 1 in the path is set at 1,OOOkm.
If it is assumed that a signal propagation speed is 2.0 x lOSKm/sec and that the queuing time are negligible, imax is roughly l Omsecs. VC models used in this simulation configuration are listed in Table 2 and all the sources are assumed to be persistent. Note that PCR, MCR, ICR, arrival time, and departure time are measured in units of Mbps.
(Table 2) Source PCR MCR ICR Arrival Fair Rate Time Departure Time 0~ 1 ~2 2~3 3 1 ~oo sl-s4 150 0 10 0 ~e 36.133.3 30 32.5 s5-s9 150 10 10 0 00 46.143.3 40 42.5 s10 150 0 10 2 00 30 32.5 sll-sl4 25 0 25 0 oc 25 25 25 25 sly-sl9 25 10 25 0 00 25 25 25 25 s20 25 0 25 1 3 25 25 It is to be noted that the PCR value, the MCR value, the ICR value, and the arrival and departure times of the VCs are varied in order to investigate the impacts of the PCR-constrained sources, the difference in MCR and ICR, and the call activities on the network performance. For the purpose of comparison, the theoretical fair rates satisfying the MAX-MIN fairness with MCR guarantee are calculated and the results are listed in Table 2. Referring to Table 2 and FIG. 10, it is observed that the fair rate of each VC
varies in time according to the arrivals and departures of the other VCs and that the sources s 11 to s 10 are bottlenecked at the switch SW 1, and the sources s 11 to s20 are bottlenecked at the access by its PCR constraint. For example, the ABR sources sl to s4 should transmit data at 36.1Mbps in O~lsec, 30Mbps in 2~3 sec, and 32.SMbps in sec to be Max-MIN fair. In accordance with the present invention, 32 data cells are generated between two adjacent forward RM cells, i.e., NRM = 32.
FIGS. 11A to 11D illustrate simulation results in the peer-to-peer configuration with ER marking only and without no VBR background traffic. FIGs. 12A to 12D
illustrate simulation results in the peer-to-peer configuration with ER
marking and VBR
background traffic.
FIGS. 11 A and 11 B illustrate the source transmission rates ai(t} of VCs, respectively with PCR=1 SOMbps and PCR=25Mbps. From FIGs. 11 A and 11 B, it is observed that the actual source transmission rates perfectly agree with the theoretical fair rates given in Table 2. The transmission rates of the sources sl to s4 and s10 are equal to the common ER, r(t) computed by the switch SW 1 since their MCR is OMbps, the transmission rates of the sources s5 to s9 are greater than the common ER, r(t) since their MCR is lOMbps, and the sources sl l to s20 are PCR-constrained irrespective of their MCR values. The initial transient behavior is due to the initial condition that r(0) = 0 at both switches SW 1 and SW2. That is, it takes a time for the common ER value to ramp up to the operating point, which is, however, a phenomenon that hardly occurs during the ordinary operation. FIG. 11 C illustrates a queue length at the bottleneck node SW 1. The joins of the source s20 at 1 sec and the source s 10 at 2sec result in the surge of the queue length and the leave of the source s20 at 3sec results in the sudden drop of the queue length. The ER allocation algorithm suggested in the present invention, however, rapidly re-stabilizes the queue length at qT = 250 cells. FIG. 1 I D illustrates the estimate of the number of locally bottlenecked VCs, ~Q~avg(t), at the switch SWI. By casting the estimate to the smallest positive integer greater than or equal to it by Eq.
(38), the integer estimate of ~Q~ can finally be obtained in the way described below. Here, symbol "[x"
indicates "x<_" and symbol "y]" indicates "y>". The integer estimate is 18 in [0, 1]sec, 9 [l, 2]sec, and 18 at [2, oo]sec, as shown in FIG. 11D. It is to be noted that except the initial time interval [0, 1 ]sec this integer estimate perfectly agrees with the true value of ~Q~ which is shown to be 9 in [0, 2]sec and 10 in [2, oo]sec in Table 2. The difference in the interval [0, 1]sec is due to the intentional increase of the estimate ~Q~avg(t) by 1 upon every arrival of a new VC as in Eq. (42). In the given simulation scenario, 18 VCs arrive at Osec, which is why the estimate ~Q~avg(t) has the value around 18 initially.
With VBR background traffic the results are virtually unchanged as shown in FIGs. 12A to 12D. That is, the macroscopic (time-averaged) behavior is identical to that of the case with no VBR traffic disturbance. The VBR background traffic was generated by a deterministic on/off source with the peak rate and the lengths of on and off periods are, respectively, l OMbps, 20imaxsecs, and 20imaxsecs. It is noted from FIGS.
12A and 12B that the transmission rate of the sources (except the PCR-constrained sources) almost perfectly follows the on/off pattern of the VBR background traffic. The on/off behavior of the VBR background traffic causes the repeated surges and drops of the ABR
queue length. The ER allocation algorithm according to the embodiment of the present invention, however, rapidly recovers the queue length to the target value as shown in FIG.
12C.
Next the parking lot configuration shown in FIG. 13 will be considered to study the case of multiple bottleneck nodes and VCs with different RTDs. 16 ABR VCs with different source locations are contained and the capacity of the links is set equally at 600Mbps except that the link between the switch SW3 and the switch SW4 and the link between the switch SW4 and the switch SW5 are 300Mbps. The VC models used in this simulation configuration are summarized in Table 3 and all the sources are assumed to be persistent.
(Table 3) Source PCR MCR ICR Arr. Time Depart. Fair rateBottle.
Time Locat.

s 1, s5, 1 0 10 0 00 21.67 SW3 s9 SO

s13 150 0 10 0 00 120 SW4 s2, s6, s10 150 10 10 0 oc 31.67 SW~

s14 25 0 25 0 oc 130 SW4 s3, s7, sll 25 10 25 0 00 21.67 SW3 s15 25 0 25 0 00 25 PCR

s4, s8, s12,25 10 25 0 00 25 PCR
s16 For comparison purposes, the theoretical fair rates satisfying the MAX-MIN
fairness with MCR guarantee was computed for the given simulation scenario and the results are included in Table 3. To further clarify the scenario, the theoretical bottleneck location of each VC, which is the location at which each individual fair share is determined, is included in the table. FIGs. 14A to 14F illustrate the simulation results with ER marking only and no VBR background traffic. FIGS. 14A and 14B
illustrate the source transmission rates ai(t) of VCs, respectively with PCR=150Mbps and PCR=25Mbps. FIG. 14C illustrates a queue length at the switch SW3, FIG. 14D a queue length at the switch SW4, FIG. 14E illustrates the estimate of the number of locally bottlenecked VCs, ~Q~avg(t), at the switch SW3, and FIG. 14F illustrates the estimate of the number of locally bottlenecked VCs, ~Q~avg(t), at the switch SW4. It is observed from FIGS. 14A and 14B that the actual source transmission rates in the steady state perfectly agree with the theoretical fair rates given in Table 3, irrespective of their RTDs and the bottleneck locations. The initial transient behavior is due to the initial condition that r(0) = 0 at all the switches, which is again a phenomenon that hardly occurs during the ordinary operation. In the given simulation scenario, there are two congested nodes, SW3 and SW4. As expected, the queue length at the congested nodes converges to the target value, 250 cells, which is shown in FIGS. 14C and 14D. It is observed that in the steady state the estimates stay around 9 and 2 at the switches Sw3 and SW4, respectively, which agrees with the data in Table 3.
Finally, application of RR marking in conjunction with ER marking will be studied. The queue length thresholds for the RR marking are set at qLT=750 cells and qHT = 1,000 cells on conjunction with qT = 250 cells. This choice of the queue length thresholds was made in the expectation that in the normal operation condition the ER
marking plays the major role to guarantee the asymptotically stable MAX-MIN
flow control whereas the RR marking plays a supplementary role to limit the transient overshooting of the queue length if any. The same parking lot configuration scenario with VBR background tragic is used in the embodiment of the present invention.
Again the VBR background traffic was generated by a deterministic on/off source with the peak rate and the lengths of on and off periods are, respectively, lOMbps, 20imaxsecs, and 20imaxsecs. The results are shown in FIGS. 1 SA to 1 SF. FIGS. 1 SA to 1 SF
illustrate simulation results in the parking lot configuration with ER marking and VCR
background traffic. FIGS. 15A and 15B illustrate the source transmission rates ai(t) of VCs, respectively with PCR=150Mbps and PCR=25Mbps.FIG. FIG. 15C illustrate a queue length at the switch SW3 and FIG. 15D illustrates a queue length at the switch SW4.
FIG. 15D illustrates the estimate of the number of locally bottlenecked VCs, ~Q~avg(t), at the switch SW3 and FIG. 15E illustrates the estimate of the number of locally bottlenecked VCs, ~Q~avg(t), at the switch SW4. Compared to the previous case shown in FIGs. 14A to 14F, the transient overshooting of the ABR queue length of the switch SW3 was substantially reduced with its maximum decreasing from 6,000 cells to 3,200 cells. Obviously, this gain comes at the cost of temporary oscillation of the queue length and the source transmission rates as observed in FIGs. 15A, 15B, and 15C. It is also observed that the ER allocation algorithm according to the embodiment of the present invention effectively takes the control back and hence recovers the queue length behavior from the oscillatory mode to the asymptotic mode.
As described above, the ER allocation algorithm according to the preferred embodiment of the present invention is advantageous in that (1) maximal link utilization and minimal cell loss are guaranteed regardless of RTDs in an ABR closed loop;
(2) ABR
queue size requirements are minimized by ensuring asymptotical stabilization of ABR
queues: (3) the MAX-MIN fairness based on the ATM forum standards is guaranteed by ensuring a fair share of an available bandwidth to each user; (4) communication network environmental change is fast reacted to such as changes in the number of ABR
users and the ABR bandwidth; {5) all functions including EFCI, RR, and ER markings are provided as specified in the ATM forum traffic management specification; (6) high utilization, low cell loss, and the MAX-MIN fair rate allocation are achieved through existence of an asymptotically stable operating point; (7) high responsiveness and transient control performance to network loading changes is achieved at multiple time scales, i.e., at the cell level rate changes of VBR and ABR VCs and at the cell level arrivals and departures of VBR and ABR VCs; (8) the number of operations required to compute the algorithm is minimized; and (9) low and scalable degree of implementation complexity is achieved by virtually removing per-VC operations including per-VC queuing, per-VC
accounting, and per-VC table access.
While the invention has been shown and described with reference to a certain preferred embodiment thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and the scope of the invention as defined by the appended claims.

Claims (12)

WHAT IS CLAIMED IS:

1. A fair flow controlling method in a packet switching network of the type having a plurality of nodes, each node being connected to a plurality of sources which transmit/receive data, an explicit rate (ER) engine, and a data queue with a current queue length and a target queue length for storing the data received from the sources, the method comprising the steps of:
generating a first signal and a second signal;
periodically updating an explicit rate (ER) by the ER engine in response to the first signal at one of the sources;
upon arrival of a forward cell at one of the sources, estimating a number of locally bottlenecked virtual circuits (¦Q¦) based on the updated explicit rate (ER) from the ER engine;
in response to the second signal, determining a new ER based on the difference between a current queue length and a target queue length of the node, the estimated number of the locally bottlenecked virtual circuits (¦Q¦), and a derivative of the current queue length; and upon detecting an arrival of a backward cell at one of the sources, writing the new ER in an ER field of the detected backward cell.

2. The method of claim 1, further comprising the step of retrieving a minimum cell rate (MCR) from the backward cell upon detecting the arrival of the backward cell at one of the sources.

3. The method of claim 1, wherein the step of writing the new ER in the ER field of the backward cell comprises the steps of:
comparing the ER of the backward cell with a sum of the newly determined ER
and the MCR of the backward cell, and writing the new ER in the ER field of the backward cell if the sum less than the ER retrieved from the backward cell.

4. The method of claim 1, wherein the number of the locally bottled necked VCs is estimated by comparing a difference between a Current Cell Rate (CCR) and a minimum cell rate (MCR) of the forward cell and the updated ER received from the ER engine.

5. The method of claim 4, wherein the number of the locally bottlenecked VCs is estimated if a difference between the CCR and the MCR is greater than or equal to the updated ER.

6. The method of claim 1, wherein the forward and backward cells comprised of a resource management cell.

7. The method of claim 1, wherein the updated ER by the ER engine is determined at a predetermined time interval.

8. The method of claim 1, further comprising the steps of:
determining whether congestion occurs in one of the sources upon detecting a data cell, and transmitting a first congestion signal or a second congestion signal if the congestion is detected;
wherein the second congestion signal indicates substantially more congested state.

9. The method of claim 8, further comprising the step of controlling a relative rate (RR) of the backward cell, wherein the RR is defined by the first signal and the second signal.

10. The method of claim 8, wherein the first signal is transmitted if the difference between the current queue length and the target queue length of the node exceeds a low predetermined threshold value, and wherein the second signal is transmitted if the difference between the current queue length and the target queue length of the node exceeds a high predetermined threshold value.

11. The method of claim 1, wherein the number of the bottlenecked virtual circuits (¦Q¦) is determined if a difference between a Current Cell Rate (CCR) and a Minimum Cell Rate (MCR) retrieved from the forward cell is less than the updated ER
transmitted from the ER engine.

12. The method of claim 3, wherein the new ER is calculated by subtracting (((average queue length - previous average queue length) x first gain) ~
estimated ¦Q¦) +
((average queue length - target queue length) x ((second gain x second signal period) ~
calculated ¦Q¦)) from a previous ER in response to the second signal.