Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L12/00—Data switching networks
  • H04L12/02—Details
  • H04L12/14—Charging, metering or billing arrangements for data wireline or wireless communications
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
  • H04L41/02—Standardisation; Integration
  • H04L41/0213—Standardised network management protocols, e.g. simple network management protocol [SNMP]
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
  • H04L41/08—Configuration management of networks or network elements
  • H04L41/0803—Configuration setting
  • H04L41/0823—Configuration setting characterised by the purposes of a change of settings, e.g. optimising configuration for enhancing reliability
  • H04L41/0826—Configuration setting characterised by the purposes of a change of settings, e.g. optimising configuration for enhancing reliability for reduction of network costs
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
  • H04L41/50—Network service management, e.g. ensuring proper service fulfilment according to agreements
  • H04L41/5003—Managing SLA; Interaction between SLA and QoS
  • H04L41/5009—Determining service level performance parameters or violations of service level contracts, e.g. violations of agreed response time or mean time between failures [MTBF]
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
  • H04L41/50—Network service management, e.g. ensuring proper service fulfilment according to agreements
  • H04L41/5003—Managing SLA; Interaction between SLA and QoS
  • H04L41/5019—Ensuring fulfilment of SLA
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
  • H04L41/50—Network service management, e.g. ensuring proper service fulfilment according to agreements
  • H04L41/5029—Service quality level-based billing, e.g. dependent on measured service level customer is charged more or less
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L43/00—Arrangements for monitoring or testing data switching networks
  • H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
  • H04L43/0876—Network utilisation, e.g. volume of load or congestion level
  • H04L43/0882—Utilisation of link capacity
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L47/00—Traffic control in data switching networks
  • H04L47/10—Flow control; Congestion control
  • H04L47/12—Avoiding congestion; Recovering from congestion
  • H04L47/125—Avoiding congestion; Recovering from congestion by balancing the load, e.g. traffic engineering
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L67/00—Network arrangements or protocols for supporting network services or applications
  • H04L67/01—Protocols
  • H04L67/10—Protocols in which an application is distributed across nodes in the network
  • H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers

Definitions

• a network user can decrease the cost of using the network, or otherwise enhance the load distribution ofthe network.
• One approach to decreasing the cost of using the network is for a person to periodically intervene and adjust the forwarding decisions ofthe network.
• Some embodiments ofthe invention control load in a network. Some embodiments of this invention reduce the monetary cost of operating the network. Some embodiments include at least part of one or more of: > Monitoring at least a first utilization of a first subset of two or more links in the network
• the criteria could be based at least partly one or more monetary billing structures of a second subset of two or more links, wherein: o at least one ofthe one or more monetary billing structures receives as input at least a second utilization ofthe second subset of two or more links, o at least one ofthe one or more monetary billing structures includes variable cost, and o the first utilization ofthe first subset of two or more links is at least partly indicative ofthe second utilization ofthe second subset of two or more links
• the steps of monitoring, assessing, and adjusting are independent - in such embodiments, no causal relationship exists between the steps of monitoring, assessing, and adjusting.
• adjustments can be made as to control load without excessively compromising performance.
• the assessment of suboptimality is based at least partly on the monitoring, hence providing a closed loop system, (e.g., in such embodiments ofthe invention, the adjusting could affect load; the reading ofthe monitoring could then be reflected by the consequent changes in load, resulting in a modification in the results ofthe assessment, which in turn provokes new adjustments.)
• the assessment of suboptimality is not necessarily based on the monitoring.
• the steps of monitoring, assessing, and adjusting are continually repeated so that the latest information provided by the monitoring can be used in adjusting the forwarding decisions.
• Figure 1 illustrates a computer programmed from program media.
• Figure 2 illustrates a computer programmed from a network.
• Figure 3 illustrates a network with nodes and links that are adjusted, links that are assessed, and links that are monitored.
• Figure 4 illustrates a network with links that are both assessed and monitored.
• Figure 5 illustrates a network with links that are both assessed and adjusted.
• Figure 6 illustrates a network with links that are both assessed and monitored, links that are assessed but not monitored, and links that are monitored but not assessed.
• Figure 7 illustrates an example of a first degree of unacceptability function.
• Figure 8 illustrates an example of monetary billing structures.
• Various embodiments ofthe invention include methods, software, hardware, and/or a combination.
• the software can be on any of various program media, such as an optical medium
• the software can also be in a transitory medium, such as an optical signal, magnetic signal, electrical signal, or some combination, such as an electromagnetic wave.
• the software can also be stored on a computer, such as on long term storage or short term storage, such as in volatile or nonvolatile memory.
• the hardware can be any of various mechanisms, such as a computer, personal digital assistant, cell phone, or embedded device.
• the hardware may be implemented on program media such as an integrated circuit or chip that can be added to a computer.
• Some embodiments are a combination of hardware and software, such as hardware with some ofthe instructions implemented in the hardware, combined with software for some ofthe instructions executing on the hardware.
• Figure 1 illustrates a computer 110, which is programmed by code stored on program media 120.
• the program media 120 is used to place code on the computer 110.
• Figure 2 illustrates a computer 210, which is programmed by code from a network 230.
• the network 230 is used to place code on the computer 210
• these mechanisms will be used to reduce the monetary cost of operating the network.
• Some embodiments include at least part of one or more of: Monitoring at least a first utilization of a first subset of two or more links in the network > Assessing the degree of suboptimality with respect to some criteria.
• the criteria could be based at least partly one or more monetary billing structures of a second subset of two or more links, wherein: o at least one ofthe one or more monetary billing structures receives as input at least a second utilization ofthe second subset of two or more links, o at least one ofthe one or more monetary billing structures includes variable cost, and o the first utilization ofthe first subset of two or more links is at least partly indicative ofthe second utilization ofthe second subset of two or more links Adjusting automatically a subset ofthe forwarding decisions of one or more forwarding nodes in the network based at least partly on the assessing, wherein the adjusting attempts to reduce the degree of suboptimality.
• the steps of monitoring, assessing, and adjusting are independent - in such embodiments, no causal relationship exists between the steps of monitoring, assessing, and adjusting.
• adjustments can be made as to control load without excessively compromising performance.
• the assessment of suboptimality is based at least partly on the monitoring, hence providing a closed loop system. (E.g., in such embodiments ofthe invention, the adjusting could affect load; the reading ofthe monitoring could then be reflected by the consequent changes in load, resulting in a modification in the results ofthe assessment, which in turn provokes new adjustments.)
• the assessment of suboptimality is not necessarily based on the monitoring.
• the steps of monitoring, assessing, and adjusting are continually repeated so that the latest information provided by the monitoring can be used in adjusting the forwarding decisions.
• load and utilization can be inter-related.
• Load can include a measure of traffic, for example, in bits per second, flowing across a resource.
• Utilization can include a measure ofthe load portion of resource capacity.
• utilization can include an absolute portion without reference to the resource capacity, such as a load, rather than a relative portion with reference to the resource capacity.
• utilization can include a relative portion of another value besides the resource capacity.
• monitoring is used to provide load information upon which, in some systems, the assessing will partly be based.
• the monitoring uses the Simple Network Monitoring Protocol (SNMP); in other embodiments, the monitoring is based partly on flow information export.
• SNMP Simple Network Monitoring Protocol
• flow information export is NetFlow.
• monitoring is based at least partly on a source external to the subset of forwarding decisions used in the adjusting. In some embodiments of this invention, the monitoring is based at least partly on span port.
• systems are included to deal with the case where monitoring is done for a subset of set of two or more links, but not for another subset ofthe two or more links.
• systems are included to deal with timeouts in SNMP polling.
• monitoring can be done using bye counts over a time interval of specified length. In other embodiments, monitoring can be done using rates.
• a minimum limit is imposed on the number of utilization samples obtained from the monitoring before assessing can proceed.
• the method takes into account the load corresponding to subsets ofthe objects.
• the subsets of objects correspond to one or more prefixes.
• This information can be obtained through monitoring systems that will be recognized by the skilled in the field. Such mechanisms include NetFlow, RMONI/II, span port, and other external monitoring sources. Such monitoring systems can also include systems based at least partly on web server logs; for example, rate of requests per destination can be counted for different applications.
• the subsets of objects include one or more prefixes, one can also use the size ofthe prefix as an estimate ofthe contribution of that prefix to the total utilization. For example, a /8 would be estimated to have twice the traffic than a 19, itself having twice the traffic of a 110.
• the monitoring combines the utilization samples in some fashion. In some embodiments of this invention, the monitoring estimates a percentile of load samples. In some embodiments, an estimation ofthe n th percentile includes, given a sampling rate r and a billing period b, storing the largest (l-n)*b*r samples during a billing period.
• the assessing is done on a set of two or more links that, in some embodiments of this invention, are the same as the set of two or more links being monitoredln some embodiments, the two sets are equal; In some embodiments, the two sets may overlap; yet in other embodiments, they can be different.
• the load utilization of the set of links used for the assessing can be deduced from the load utilization ofthe set of links that are used for the monitoring. For example, in some embodiments of this invention, the utilization on the links that are monitored can be equal to the utilization on the links that are assessed.
• forwarding decisions are adjusted as to control load. In some embodiments of this invention, forwarding decisions are adjusted as to strike an adequate balance between load control and performance.
• assessing includes at least partly an assessment of load and/or an assessment of performance.
• load and performance information can be combined in a metric that can be used to rate one or more ofthe two or more links in the network.
• metrics can be computed for one or more links for objects controlled by forwarding decisions based at least partly on performance information for these objects on the one or more links; the metric for each of these links can then be penalized by an amount that is based, at least partly on the desired utilization ofthe one or more links.
• the penalty associated for at least one ofthe one or more links can be at least partly fixed; in other embodiments, at least one ofthe one or more penalty values corresponding to the one or more links can be at least partly variable.
• the objects controlled by the forwarding are prefixes. In some embodiments ofthe invention, the objects controlled by the forwarding are flows. In some embodiments ofthe invention, the objects controlled by the forwarding are network applications. In some embodiments of this invention, computing the object penalties ofthe one or more links is based at least partly on the amount the corresponding metric needs to be degraded by so that the metric on this link is deemed unacceptable. In some embodiments, the standard of unacceptability is based at least partly on the concept of a winner set, the width of this set including metric values that are deemed acceptable. First degree of unacceptability functions
• the assessing includes generating one or more sets of functions, wherein at least one function in the one or more sets of functions gives a first degree of unacceptability of at least one link from the first subset of two or more links, wherein the first degree of unacceptability is based at least partly on utilization ofthe at least one link in the network.
• At least one function in the one or more sets of functions outputs at least a varying value. In some embodiments, at least one function in the one or more sets of functions is continuous or piecewise continuous with respect to utilization. In some embodiments, the at least one function in the one or more sets of functions is non-decreasing with respect to load.
• At least one degree of unacceptability function in the at least one set of degree of unacceptability functions receives at least one input, the at least one input at least partly depending on load, wherein the at least one degree of unacceptability function outputs at least:
• the first degree of unacceptability function can be computed as follows: (We denote the first degree of unacceptability p.)
• Figure 7 illustrates an example of a first degree of unacceptability function.
• At least one degree of unacceptability function in the at least one set of functions receives at least one input, the at least one input at least partly depending on load , wherein the at least one degree of unacceptability function outputs at least: 1) a first constant value for values ofthe at least one input up to a threshold value
• the first degree of unacceptability function can be computed as follows: (We denote the first degree of unacceptability p.)
• the load value is based at least partly on the monitoring. In some instances ofthe invention, the load value is based at least partly on inbound utilization. In some instances ofthe invention, the load value is based at least partly on outbound utilization. In some embodiments ofthe invention, load value is based at least partly on max(inbound,outbound); in some instances ofthe invention, load value is based at least partly on avg(inbound,outbound); in some instances ofthe invention, the load value is based at least partly on inbound + outbound. In some instances ofthe invention, the load value can be based on the instantaneous load values that result from the monitoring.
• the load values are based at least partly on a percentile of a subset of load values that result from the monitoring. In some instances ofthe invention, the load values are based at least partly on the average of a subset of load values that result from the monitoring.
• different first degrees of unacceptability curves are applied to different forwarding decisions. More than one degree of unacceptability can exist. Selection of a set of functions can be done per forwarding decision. In some embodiments ofthe invention, no degree of unacceptability is applied to at least one link for at least one forwarding decision. For example, not all functions that are being assessed must have one or more sets of functions assigned to them.
• the assessing also includes the computation of a second degree of unacceptability for a link that can be dependent at least partly on the first degree of unacceptability.
• determining ofthe second degree of unacceptability includes treating the first degree of unacceptability as a probability value, and assigning, using the probability value, one of a plurality of states to the second degree of unacceptability.
• the second degree of unacceptability can be assigned two states, that we denote here "hot” and "cold” based at least partly on the result of a random selection based at least partly on the first degree of unacceptability.
• the winner sets are constructed in an ordered list of one or more winner sets, where the elements of a winner set are links from the set of two or more links.
• the elements of a winner set are comparable in quality for an object influenced by the forwarding decisions.
• links that have a second degree of unacceptability that is large enough are not included in at least one winner set.
• hot links are removed from at least one winner set in a list of one or more winner sets.
• the ordered list of one or more winner sets includes two winner sets, denoted the basic winner set and the extended winner set. If such instances also include a second degree of unacceptability that includes two states, "hot" and "cold", and if, for an object, the basic winner set is empty and the extended winner set is non-empty, then the forwarding decision that influences this object is adjusted to point to at least one ofthe one or more links in the extended winner set.
• all winner sets are empty in the ordered list of winner sets, no adjustment is done for this object, and an attempted adjustment may be done to the following object. In other embodiments, an adjustment is performed that is based solely on performance.
• a new ordered list of winner sets is constructed, based on a new set of first degree of unacceptability functions for each link. (See the section on more than one set of functions.).
• one or more links in the set of two or more links can be chosen using a probabilistic approach.
• one link in the set of two or more links can be chosen randomly among the various links in the set of two or more links.
• the probability density function used for the random selection can be biased towards some links and away from other links, based at least partly on the monetary cost ofthe one or more links.
• assessing is based at least partly on monitoring a degree of suboptimality with respect to one or more monetary billing structures of a subset of two or more links in the network, wherein:
• At least one ofthe one or more monetary billing structures receives as input at least a utilization ofthe subset of two or more links, and - at least one ofthe one or more monetary billing structures includes at least variable cost.
• the monetary billing structures are applied to a set of two or more links that, in some embodiments of this invention, are related to the set of two or more links being assessed.
• Monetary billing structures can include one or more rules which determine a monetary bill resulting from the use of network links.
• the two sets are at least partly equal and/or unequal; in some embodiments, the load utilization ofthe set of links on which the monetary billing structures are based can be deduced from the load utilization ofthe set of links that are used for the assessing.
• the utilization on the links that are monitored can be equal to the utilization on the links on which the monetary billing structures are based. In some embodiments, the utilization of the links that are monitored overlap the utilization on the links on which the monetary billing structures are based. In yet other embodiments, the utilization ofthe links that are monitored are different from the utilization ofthe links on which the monetary billing structures are based.
• Suboptimality can mean the existence of a state, and/or can mean the degree of a state,, respect to one or more ofthe monetary billing structures, such that the cost of operating the network, as given by the monetary billing structures, is not minimized.
• At least one ofthe one or more monetary billing structures receives as input at least a utilization of at least one link from the second subset of two or more links, wherein the utilization may be determined over time.
• the utilization is computed at least partly from at least one of: la) a maximum and lb) an average, of at least one of: 2a) one or more percentiles and 2b) one or more averages, of one or more sets of utilization samples ofthe at least one link from the second subset of two or more links.
• the billing structure is based on some amount such as a percentage, e.g. 95%, ofthe link utilization, measured over a billing period.
• the billing period is equal to a regular period, such as a month, week, day, hour, or fraction or multiple thereof.
• load is controlled by taking into account, at least partly, the same formula used in utilization for billing. For example, in the instance where the billing structure is based on the 95% of a link utilization, some embodiments ofthe invention can choose to only react when some estimation ofthe 95% ofthe link utilization is about to jump beyond a value that could cause in an increase in the bill. In some such embodiments, this can be achieved by having the first degree of unacceptability only increase once such thresholds are reached.
• the billing structures are based at least partly on the 95 th percentile of a function of both the inbound and outbound load ofthe at least one link.
• the function of both the inbound and outbound load is a combining function, such as the averaging function.
• the billing structures are based at least partly on a function of both the 95' percentile ofthe inbound load and the 95 th percentile ofthe outbound load.
• the function of both the 95 th percentile ofthe inbound load and the 95 th percentile ofthe outbound load is the averaging function; in some embodiments, the function of both the 95 th percentile ofthe inbound load and the 95 th percentile ofthe outbound load is the max function.
• the 95 th percentile value is illustrative. Other values in the range of 0-100%, or an absolute, non-percentage-based value, can be used.
• the assessing is done using more than one set of functions.
• the system would select, for a given object, a first set of functions from the one or more sets of functions; if the first degree of unacceptability fails a threshold of acceptable unacceptability for all functions in the set of functions, then a second set is chosen.
• one example of a degree of unacceptability can be a degree of unacceptability.
• one example of a threshold of acceptable unacceptability can be a threshold of unacceptability.
• examples of failing a threshold of acceptable unacceptability can include any of: passing a threshold of unacceptable unacceptability, failing a threshold of unacceptable acceptability, and/or passing a threshold of acceptable acceptability.
• the assessing further includes selecting at least one object from the one or more objects, selecting at least one set of functions from the one or more sets of functions, and constructing one or more winner sets for the at least one object and the at least one set of functions, wherein each winner set from the one or more winner sets includes a corresponding quality characterization threshold, wherein constructing includes:
• an example a quality characterization can indicate quality and/or lack of quality.
• an example of failing a quality characterization threshold can be passing a quality characterization.
• the links that are selected are from the a non-empty winner set from the one or more winner sets, wherein the non-empty winner set has a low corresponding quality characterization threshold (such as a lowest corresponding quality characterization threshold) from all corresponding quality characterization thresholds included by all winner sets from the one or more winner sets.
• a low corresponding quality characterization threshold such as a lowest corresponding quality characterization threshold
• the excluding, from the at least one or more winner sets, links for which the quality characterizations ofthe at least one object fails the corresponding quality characterization threshold included by each winner set from the one or more winner sets can include: identifying at least one best link from the one or more links from the third subset of two or more links, wherein the at least one best link has a high quality characterization from at least one ofthe one or more links from the third subset of two or more links, and- determining the corresponding quality characterization threshold based at least partly on the high quality characterization.
• the selection of a second set can also occur when the constructing ofthe first one or more winner sets corresponding to the first set of functions yields all empty winner sets.
• the one or more sets of functions are ordered into an ordered list of, for example, functions that are nontrivial to the embodiment.
• the first and second set of functions referred to above are adjacent in the ordered list of the one or more sets of functions. Adjacent functions can have in between one or more functions that are trivial to the embodiment.
• the ordering includes the following steps:
• a level for each set of functions in the one or more sets of functions, a level, wherein a level is based at least partly on a sum of maxAvoidance values across the one or more functions in each set of functions
• the approach above is combined in a table that we denote the threshold table.
• the table consists of multiple rows, wherein each row in the table includes information regarding one set of functions, i.e., corresponding to one level. For each set of functions, the parameters corresponding to each function are described. If the functions include a minAvoidance and maxAvoidance as described above, then the minAvoidance and maxAvoidance parameters are included in the row for each function. In addition, if assessing is based at least on a second degree of acceptability, then in some embodiments, the value ofthe second degree of acceptability can also be stored along with each function.
• Each set of functions includes functions for a number of links in the network.
• one level is selected at any one time.
• the selection includes the following steps:
• the example below applies: if the total load is 90, the probability of rejection for link LI will be computed using start- avoidance 40, max- avoidance 44.
• the (x, y) pairs represent the minAvoidance and maxAvoidance for each function for each set of functions corresponding to each level.
• a function for at least a link receive for input at least one ofthe values of outbound loads for the at least one link. In some embodiments of this invention, a function for at least a link receive for input at least one ofthe values of inbound loads for the at least one link. In some embodiments of this invention, a function for at least a link receive for input at least one ofthe values of a combination of inbound loads and outbound loads for the at least one link.
• the system upon receipt of a new load sample on a link, can do the following:
• Some embodiments of this invention have different sets of functions for different objects.
• the assessing when the monitoring results in a new load sample that triggers a change in the active level, the assessing also includes re-computing the first degree of unacceptability based at least partly on the new level.
• the assessing includes at least one ofthe following steps:
• the system selects another object in the list.
• a selection of a link based solely or primarily on the quality characterization ofthe links is done.
• at least one ofthe following steps is included: o For those links in the performance-only winner set for which the probability is less then one, reevaluate the probabilities until at least one links' second degree of unacceptability is assigned the "cold" state, o Select at least one link from the one or more links that are assigned the "cold" state.
• o For those links in the performance-only winner set for which the probability is less then one, reevaluate the probabilities until at least one of links' second degree of unacceptability is assigned the "cold" state, o Select at least one link from the one or more links that are assigned the "cold” state. - In some embodiments, select from any subset ofthe links at random
• a second probability based on a first degree of unacceptability assigned to each link, wherein the second probability is based at least partly on the distance between one and the value ofthe first degree of unacceptability.
• the second probability corresponds to the probability for the link to be "cold".
• the set of functions from which one derives the first degree of unacceptability based at least partly on the monetary billing structures is not limited.
• assessing includes generating, from at least one ofthe one or more monetary billing structures, one or more sets of functions, wherein at least one function in the one or more sets of functions gives a first degree of unacceptability of at least one link from a subset of two or more links, wherein the first degree of unacceptability is based at least partly on a utilization ofthe at least one link from the subset of two or more links.
• the generating ofthe sets of functions includes
• determining the optimal utilization involves solving for the minimum monetary cost of operating the network, with respect to the at least one ofthe one or more monetary billing structures
• determining the optimal utilization involves a steepest descent strategy with respect to the at least one ofthe one or more monetary billing structures. (See example on steepest descent approach.)
• the determining ofthe adequate set of functions includes at least one ofthe following steps:
• Step 4 can be repeated for all links of interest. In some embodiments, if the number of links that include first degree of unacceptability functions is N, then we have N+l levels.
• the number of links that include first degree of unacceptability functions is N, then we have at most N levels.
• startAvoidance and maxAvoidance are related as follows:
• the problem of finding an optimal load distribution can be posed as a linear programming problem. That is, given:
• linear programming techniques can be applied to solve this problem.
• the fundamental theorem of linear programming states that optimal points in an optimization problem are extreme points ofthe feasible regions, that is the regions where a valid solution can be found.
• a valid solution is a combination of load values such that the cost is optimal, for a given total load.
• Linear programming algorithms such as the simplex algorithm speed up the calculation of solutions by restricting the search for optimal values on the set of extreme points only.
• the problem can be converted into a table lookup using a heuristic approach.
• a table of optimal solutions is stored, wherein the table of optimal solutions includes the combinations of load values that lead to optimal cost.
• the appropriate row is retrieved each time a new load sample comes in.
• the choice ofthe optimal solution is based on a proximity factor, wherein the proximity factor selects the optimal solution that minimizes the load changes among links, for the current combination of individual loads that lead to the total load that's being looked up.
• the proximity factor can be based on at least one ofthe following functions:
• PF(OPj) sum i ( current_load_i - target_loadJ_i ) 2 square error
• OP minj PF(OPj) least square error
• computing this table is a one-time effort. In some embodiments, the computation of this table is done off-line. In some embodiments, the computation of this table is done periodically. In some embodiments of this invention, the computation of this table is triggered by an external event.
• determining the optimal utilization involves a steepest descent strategy with respect to the at least one ofthe one or more monetary billing structures.
• the one or more sets of function that give a first degree of unacceptability use at least one ofthe following:
• Steps 2 and 3 are repeated. In some embodiments of this invention, Steps 2 and 3 are repeated until the maximum cost tier is reached for all providers. In some embodiments, the maximum cost tier constitutes the physical link capacity
• a set of function in the one or more sets of functions that give the first degree of unacceptability is set at the actual level of transition, wherein the actual level of transition is based at least partly on the provider's billing model. In some embodiments, it is not necessary to cautiously set thresholds lower than the actual provider bandwidth tiers.
• the maxAvoidance is set to the actual transition levels for all links.
• startAvoidance is set to an amount, such as 10% lower than the true threshold. In some embodiments, a value for startAvoidance is selected automatically. For this example, we will assume that the enterprise has active links to three service providers, who bill according to the following utilization tiers: usage level cost
• Figure 8 illustrates an example of monetary billing structures.
• Tier 2 30 10 15 5 Tier 3 65 10 15 40 T ⁇ er 5 105 20 45 40
• the first tier is configured to make optimal use ofthe minimum commit level of each provider.
• the level value is simply the sum of all provider thresholds.
• Tier 2 In some embodiments of this invention, the second tier is configured to use provider 2 for any traffic that exceeds the minimum commit levels of tier (1). In some embodiments, Provider 2 was selected by comparing the incremental cost increase of all three providers at the next utilization level, and selecting the cheapest:
• provider 2 is utilized to its full capacity at the next cost tier. In this example, provider 2 is used until that link approaches 15 mbps.
• Tier 3 In some embodiments, if bandwidth utilization exceeds the 30 mbps aggregate of tier (2), the same heuristic is used to determine the next provider to bear an increase on tier (3):
• provider 3 will be the next link utilized.
• Provider 3 is utilized to its full capacity at this cost level, which is 40 mbps.
• Tier 4 In this example, at tier (4), there is a tie among the cost increments:
• the provider that provides the most capacity at the next billing level is selected.
• Provider 2's cost remains at this cost level from 15 mbps - 45 mbps, which is the longest duration ofthe three.
• Tier 5 In this example, at tier (5), Provider 1 is selected using the same logic as tier (4).
• Tier 6 In this example, note that although provider 1 is again selected at tier (6), this tier is not combined with tier (5).
• Tier 7 In this example, the last tier represents the full link capacity of each provider. Adjusting can be done automatically to a subset ofthe forwarding decisions of one or more forwarding nodes in the network based at least partly on the assessing, wherein:
• a minimum limit can be imposed on the interval separating consecutive reevaluations of one or more of their first and second degrees of unacceptability for an object.
• a minimum limit can be imposed on the interval separating consecutive hot/cold reevaluations.
• the reevaluation interval can be chosen randomly with respect to some probability distribution function. In some embodiments ofthe invention, the reevaluation interval is chosen as to be larger than the minimum interval between two consecutive monitoring actions. In some such embodiments in which the second degree of unacceptability includes the states "hot” and "cold", the probability distribution functions in respect to which the reevaluation interval are computed can be chosen differently for hot to cold transitions, and cold to hot transitions, respectively. In some such embodiments, the probability distribution function for cold to hot transitions has a lower median than the probability distribution function for hot to cold transitions.
• the probability distribution function with respect to which the reevaluation interval is computed can include an exponential distribution function.
• a minimum limit can be imposed on the range of values that is allowed by the distribution.
• a maximum limit can be imposed on the range of values allowed by the distribution.
• the subset of two or more forwarding decisions in the network that are to be adjusted automatically does not consist of all forwarding decisions. Load often varies randomly in unpredictable ways. Computing a target that provides an optimal solution to the problem, and adjusting the forwarding decisions to meet this target seldom leads to the optimal solution, because the conditions at the time when the target was computed, and at the time the forwarding decisions were adjusted are not the same.
• the incremental approach is used, wherein a subset ofthe forwarding decisions are selected for adjustment at any one time.
• continuously monitoring and assessing, and continuously adjusting in an incremental fashion a subset ofthe forwarding decisions allows for stable load movements towards the optimal load distribution.
• the subset ofthe forwarding decisions of one or more forwarding nodes is done automatically.
• the selecting ofthe subset ofthe forwarding decisions is random In some embodiments, the selecting ofthe subset ofthe forwarding decisions is independent from the assessing.
• the selecting ofthe subset ofthe forwarding decisions uses a flow monitoring device
• At least one forwarding decision from the subset ofthe forwarding decisions at least partly influences one or more objects, wherein the one or more objects includes at least one of a prefix, a flow, and a network application; in some such embodiments, the assessing is further based at least partly on quality characterizations ofthe one or more objects, wherein the quality characterizations are with respect to at least one link from the third subset of two or more links. In some such embodiments, the selecting ofthe subset ofthe forwarding decisions is based at least partly on a measuring ofthe quality characterizations ofthe one or more objects.
• the selecting ofthe subset ofthe forwarding decisions is based at least partly on a source external to the third subset of two or more links.
• the forwarding decisions ofthe one or more forwarding nodes are described at least partly by at least one Layer 3 Protocol
• At least one ofthe forwarding decisions ofthe one or more forwarding nodes are described at least partly by at least one Internet Protocol
• the forwarding decisions ofthe one or more forwarding nodes are described at least partly by at least one Layer 2 Protocol
• the adjusting is described at least partly by at least one Border Gateway Protocol (BGP)
• Border Gateway Protocol (BGP) Version 1 In some embodiments of this invention, the adjusting is described at least partly by Border Gateway Protocol (BGP) Version 1
• the adjusting is described at least partly by Border Gateway Protocol (BGP) Version 2
• the adjusting is described at least partly by Border Gateway Protocol (BGP) Version 3
• the adjusting is described at least partly by Border Gateway Protocol (BGP) Version 4.
• Border Gateway Protocol (BGP) Version 4 Border Gateway Protocol (BGP) Version 4

Applications Claiming Priority (3)

Publications (2)

Family

ID=27734398

Family Applications (1)

Country Status (5)

Citations (3)

Family Cites Families (4)

• 2003
  • 2003-02-04 WO PCT/US2003/003297 patent/WO2003067731A2/en not_active Application Discontinuation
  • 2003-02-04 AU AU2003214992A patent/AU2003214992A1/en not_active Abandoned
  • 2003-02-04 CA CA2704318A patent/CA2704318C/en not_active Expired - Fee Related
  • 2003-02-04 EP EP03710842A patent/EP1479009A4/de not_active Ceased
  • 2003-02-04 CA CA2474501A patent/CA2474501C/en not_active Expired - Fee Related
• 2004
  • 2004-07-22 IL IL163148A patent/IL163148A/en not_active IP Right Cessation

Patent Citations (3)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events