US8213323B1 - System and method for network performance monitoring - Google Patents
System and method for network performance monitoring Download PDFInfo
- Publication number
- US8213323B1 US8213323B1 US10/737,922 US73792203A US8213323B1 US 8213323 B1 US8213323 B1 US 8213323B1 US 73792203 A US73792203 A US 73792203A US 8213323 B1 US8213323 B1 US 8213323B1
- Authority
- US
- United States
- Prior art keywords
- link
- utilization
- router
- network
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active, expires
Links
- 238000000034 method Methods 0.000 title claims abstract description 73
- 238000012544 monitoring process Methods 0.000 title claims abstract description 62
- 208000027744 congestion Diseases 0.000 claims description 60
- 238000005259 measurement Methods 0.000 claims description 47
- 230000001934 delay Effects 0.000 claims description 36
- 238000004364 calculation method Methods 0.000 claims description 12
- 230000005540 biological transmission Effects 0.000 claims description 9
- 238000013480 data collection Methods 0.000 claims description 6
- 238000012545 processing Methods 0.000 claims description 6
- 238000004458 analytical method Methods 0.000 description 14
- 230000000694 effects Effects 0.000 description 10
- 238000007726 management method Methods 0.000 description 9
- 230000003287 optical effect Effects 0.000 description 6
- 230000006399 behavior Effects 0.000 description 5
- 238000013459 approach Methods 0.000 description 3
- 230000015556 catabolic process Effects 0.000 description 3
- 238000006731 degradation reaction Methods 0.000 description 3
- 230000000135 prohibitive effect Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000002085 persistent effect Effects 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 230000002411 adverse Effects 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000003139 buffering effect Effects 0.000 description 1
- 239000000969 carrier Substances 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000002045 lasting effect Effects 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 238000011002 quantification Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0876—Network utilisation, e.g. volume of load or congestion level
- H04L43/0882—Utilisation of link capacity
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/16—Threshold monitoring
Definitions
- the present invention relates to a method and system for network performance monitoring and in particular to identification of micro-congestion episodes in a network.
- IP networks feature hundreds of routers and thousands of links. Monitoring of a network-wide state is usually achieved through the use of the Simple Network Management Protocol (SNMP). Routers collect measurements and report the measurements to a specific location at regular intervals. The regular intervals are typically set to five minutes. The selection of the time interval may be based on the size of the network, since the interval has to be large enough so that it allows the polling of the entire network during a single interval. Additionally, the interval should not be too small to avoid overloading the polled network elements.
- SNMP Simple Network Management Protocol
- the SNMP statistics that are usually collected over the five-minute intervals correspond to average activity of IP links and network elements for the duration of the interval.
- the system collects link utilization from every link inside a network. The utilization information is gathered and moved to a network management system in a central location.
- MIB Management Information Base
- routers count how many bytes each link sends.
- the management station collects byte measurements and subtracts neighboring measurements to determine how many bytes were sent over the five-minute interval. This technique doesn't provide an adequate understanding of what happens during each time interval.
- each network provider usually aims for network-wide link utilizations that do not exceed the “acceptable utilization levels”. Those levels are specific to each network provider and are frequently set to around 50%. If each link in the network has a utilization level of less than 50%, it is capable of carrying the traffic of any equal-capacity link in its neighborhood in case of failure. Previous analytical work has shown that links having a utilization level of less than 50% introduce minimal queuing delays.
- Network management systems typically use the above-identified measurements to determine if a link is overloaded. If the link is overloaded, it may be dropping customer packets or experiencing delays that are likely to violate service level agreements.
- conventional reports of traffic performance across periods of minutes can mask out performance degradation due to short-lived events, such as micro-congestion episodes, that manifest themselves at smaller time scales. Cases may exist in which utilization during the overall time period appears low according to accepted standards, yet packets may be dropped during smaller intervals. During the five-minute intervals, the system may at times be operating at full capacity. No technique is currently available for showing whether links experience small periods of high utilization that lead to increased delays and possibly packet drops within the five-minute intervals.
- the network management system may continue to gather data at the standard interval. Since the management system collects measurements from an entire network, the collection is asynchronous and collection from different links occurs at different times. If the network management system collects data more frequently, too much traffic will result. Yet, traffic counters collected every five minutes mask out micro-congestion episodes that occur at time scales of milliseconds. Accordingly, a new solution is needed for identifying the time scales in various networks over which micro-congestion episodes occur and for taking the measurements in accordance with the identified time scales.
- the present invention is directed to a method for monitoring network performance in order to reveal micro-behavior of network traffic.
- the method includes setting at least two distinct durations for monitoring link utilization and setting at least one link utilization threshold for each duration.
- the method further includes monitoring link utilization at a router and providing a number of durations during which the at least one link utilization threshold has been exceeded.
- a method for monitoring network performance in order to reveal micro-behavior of network traffic includes setting at least one distinct duration for monitoring link utilization and setting a collection interval for collecting monitored data, wherein the collection interval is larger than the at least one distinct duration.
- the method further includes analyzing the collected data.
- a system for monitoring network performance in order to reveal micro-behavior of network traffic.
- This system is positioned inside a router for the efficient collection of fine-grained traffic measurements.
- the system relies on the router monitoring link utilization during at least two distinct durations and a threshold monitor for determining whether link utilization during the at least two durations exceeds the at least two distinct thresholds.
- the invention includes a method for determining parameters for monitoring micro-congestion episodes based on detailed packet information captured on selected links attached to the same router, which may be accurately time-stamped through the Global Positioning System (GPS), inside an operational Internet Protocol (IP) network.
- the method includes collecting data over at least one time interval having a specified duration, the data including at least a number of packets entering a router and a number of packets exiting the router.
- the method further includes determining link throughput based on the collected data, calculating delays for each time interval, and relating link throughput with calculated delays.
- the method further includes selecting at least one monitoring time interval based on the relationship between throughput and calculated delays.
- FIG. 1 is a block diagram showing an embodiment of a system of the invention
- FIG. 2 is a block diagram showing details of a monitoring system of the invention
- FIG. 3 is a flow chart illustrating a method for monitoring network traffic in accordance with an embodiment of the invention
- FIG. 4 is a graph illustrating a relationship between output link utilization and single-hop delay for a first interval in a first network
- FIG. 5 is a graph showing a relationship between output link utilization and the delay distribution for different intervals in the first network
- FIG. 6 is a graph illustrating a relationship between output link utilization and single hop delay during the first interval in a second network
- FIG. 7 is a graph illustrating a relationship between output link utilization and the delay distribution for different intervals in the second network
- FIG. 8 is a graph illustrating a fraction of overloaded intervals across time for different thresholds and intervals in the first network
- FIG. 9 is a graph illustrating the fraction of overloaded intervals across time for different thresholds and intervals in the second network
- FIG. 10 is a graph showing delay statistics computed over five minute intervals in the first network.
- FIG. 11 is a graph illustrating delay statistics computed over five minute intervals across time in the second network.
- the study of packet traces collected inside an IP network can help to identify the time scales over which micro-congestion episodes occur.
- a method is provided that can be easily implemented by a router so that data can be reported at regular intervals such as the standard five minute intervals through SNMP to shed light into the micro-behavior of the carried traffic.
- micro-congestion episodes can be characterized with respect to their amplitude, frequency and duration.
- Two parameters including a time scale ⁇ and a utilization threshold th are useful in defining frequency, amplitude, and duration of micro-congestion episodes.
- the amplitude of a micro-congestion episode may be determined by observing statistics collected for a specific time scale ⁇ across different utilization thresholds th.
- the frequency of a micro-congestion episode may be evaluated by observing the number of intervals of duration ⁇ that report a utilization level greater than a selected utilization threshold th across each 5-minute or other broad collection time interval.
- the duration of a micro-congestion episode may be evaluated by observing statistics collected for specific values of the utilization threshold th across multiple time scales ⁇ .
- a network 10 includes a router 20 .
- Incoming packets 22 and outgoing packets 24 travel through the router 20 .
- a collection system 100 collects data from the router 20 .
- the router 20 may be contained within the network 10 , and in an embodiment of the invention, the network 10 is a SprintTM Tier-1 IP backbone network.
- FIG. 2 illustrates an embodiment of the monitoring system 100 .
- the monitoring system 100 exists as part of the router 20 . Given that this monitoring system may collect 300,000 or more measurements within a 5-minute interval, it would be impractical to export the information. Accordingly, the collected information is processed at the router.
- the monitoring system 100 may be a part of the router 20 or a specialized card inside the router.
- the monitoring system 100 may include a processing unit 102 , a network interface 104 , and an operator interface 106 .
- the monitoring system 100 may further include a memory 110 including application programs for performing methods in accordance with embodiments of the invention.
- a system bus 122 may be provided to couple various system components including the system memory to the processing unit 102 . Each of the aforementioned components is further described below.
- the memory 110 is shown as a generalized memory and may include timers 112 , statistics calculation tools 114 , and stored statistics 116 .
- a data collection module 118 collects data 120 for use by the aforementioned modules.
- the statistics calculation tools may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer.
- program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types.
- the monitoring system 100 implements the timers 112 for different durations of ⁇ .
- the data collection module 118 collects the number of bytes transmitted by each one of the interfaces through the duration ⁇ . Then, given the capacity of the links, the statistics calculation tools 114 compute the link utilization by dividing the collected number of bytes with the capacity of the link. The monitoring system 100 then bins the computed utilization value according to the desired granularity by applying selected thresholds. If the computed utilization value is greater than th, the data collection module 118 increases the counter corresponding to the number of intervals exceeding th. At the end of the 5-minute interval, the monitoring system 100 inside the router transmits the number of intervals exceeding specific threshold values for the different time scales. Alternatively, the monitoring system 100 stores the information in as stored statistics 116 and the network operator could access the statistics through an operator interface.
- the memory 110 is shown as a generalized memory, but may include a variety of computer readable media.
- computer readable media may comprise computer storage media and communication media.
- the system memory 110 may include computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM) and random access memory (RAM).
- ROM read only memory
- RAM random access memory
- a basic input/output system (BIOS) containing the basic routines that help to transfer information between elements within system 100 , such as during start-up, is typically stored in ROM.
- the RAM typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 102 .
- the memory 110 includes an operating system, application programs, other program modules, and program data.
- the monitoring system 100 may also include other removable/nonremovable, volatile/nonvolatile computer storage media.
- a hard disk drive may read from or write to nonremovable, nonvolatile magnetic media
- a magnetic disk drive may read from or write to a removable, nonvolatile magnetic disk
- an optical disk drive may read from or write to a removable, nonvolatile optical disk such as a CD ROM or other optical media.
- Other removable/nonremovable, volatile/nonvolatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like.
- the hard disk drive would typically be connected to the system bus 122 through a non-removable memory interface and the magnetic disk drive and optical disk drive are typically connected to the system bus 122 by a removable memory interface.
- a user may enter commands and information into the monitoring system 100 through the operator interface 106 using input devices such as a keyboard and pointing device, commonly referred to as a mouse, trackball or touch pad.
- input devices such as a keyboard and pointing device, commonly referred to as a mouse, trackball or touch pad.
- a monitor or other type of display device may also be available to the operator.
- An operator may access the monitoring system 100 inside the router 20 through a networked environment using a remote computer. Furthermore, the monitoring system 100 inside the router 20 is able to access network interfaces to collect utilization metrics at different time scales.
- the monitoring system environment 100 is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the invention. Neither should the monitoring system environment 100 be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary monitoring system environment 100 . Moreover, those skilled in the art will appreciate that the invention may be practiced with other computer system configurations, including hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
- FIG. 3 illustrates a monitoring method in accordance with an embodiment of the invention.
- an operator sets one or more monitoring intervals ⁇ .
- the operator sets one or more utilization thresholds th.
- the system monitors traffic through the router 20 .
- the monitoring system 100 collects data from the router 20 and in procedure E the collected data is analyzed.
- evaluation of the amplitude, duration, and frequency of a micro-congestion episode requires determination of one of more appropriate time scales ⁇ and appropriate utilization thresholds th.
- procedure A the choice of the appropriate value of ⁇ is not trivial. If the value of ⁇ is too small, then high throughput values may relate to the simple transmission of a small number of packets back to back.
- procedure B the choice of the appropriate threshold th may be determined in accordance with the characteristics of the network and the terms of customer service level agreements.
- the monitoring interval selected in procedure A is distinct from the collections process of procedure D. In procedure D, the measurements may continue to be reported at five-minute intervals or any other currently used interval.
- the total amount of traffic flowing on a link computed over intervals of different duration can exhibit great variations. Traffic burstiness is likely to demonstrate itself through high throughput values at time scales of milliseconds, while throughput measurements collected over intervals on the order of minutes are likely to be within the provider's acceptable utilization levels. Accordingly, it is necessary to find appropriate values for the timescale ⁇ , over which throughput measurements should be collected in order to reveal performance degradation due to traffic burstiness at small time scales.
- TCP Transmission Control Protocol
- the dominant Internet protocol has a maximum size of window of packets that it can send simultaneously. If TCP sends a full window of packets that arrive at the router's incoming interface back to back, in the absence of cross-traffic, these packets will appear back to back at the output link, if it is of the same capacity as the input link. If the measurement interval contains those TCP packets alone, the measured utilization level is 100%. However, packets have experienced no queuing delay since they arrived at an equal-speed link and faced no cross-traffic. Consequently, high output throughput values in this case do not correspond to increased delays through the router.
- monitoring equipment can monitor every data packet at an input link of a router and at an output link of a router with very accurate time stamps.
- Time stamps may be GPS synchronized and can give accuracy down to a microsecond. It is then possible to determine which of the input packets were transmitted from the monitored output link and how much time each packet took. This information helps to determine the utilization of the output links. If the utilization were too low, the delay would be zero. If the packet doesn't experience any delay, then the time scale may be too small to permit conclusions about what happens to the packets.
- the examination of a single time scale may generally be inadequate for determining the effects of micro-congestion episodes.
- an examination may reveal that congestion occurs but will not reveal the duration of the congestion. It is also necessary to determine the duration of the congestion in order to assess overall effects.
- delays may or may not occur.
- a small time scale such as the 1 ms time scale reports high utilization levels over a link
- delays may or may not occur.
- the delays can be more accurately revealed. If the 10 ms time scale shows much lower utilization levels, the operator can infer that the any existing delays are not having lasting adverse effects. If utilization levels are still high at 10 ms, delays and/or drops may be occurring.
- a customer level agreement might promise a 30 ms delivery. If it takes 28 ms to deliver a packet without delays, any delay exceeding 2 ms will cause the customer level promise to be broken. Drops are an extreme case. Packets will be dropped if the queue is full.
- the results may have no relevance to the time scale at which queues are building up inside the router.
- the maximum time scale of relevance for the definition of micro-congestion episodes should be related to the total amount of buffering available to packets while transiting the router. This latter time scale is useful for analyzing the loss characteristics of the network, but it may be too large for bounding the delay through each node. In reality even a small queue buildup of 1 ms may be considered prohibitive if it occurs frequently in space and/or time.
- the utilization thresholds may be pre-set to 50% . . . 60%, or other threshold.
- the thresholds are required because otherwise the system would be required to process an excessive number of measurements.
- the network operator can set whichever thresholds are interesting when the characteristics of the network are considered.
- each router keeps counters of utilization at multiple time scales.
- procedure D every five minutes (or other selected time period), the router reports how many of the periods from each time scale (for example, how many 1 ms intervals and how many 10 ms intervals) exceed a pre-set utilization threshold.
- Monitored time periods may be overlapping. For instance, in a 10 ms time period, the network operator may choose to monitor the 1 ms time span, a 5 ms time span and the entire 10 ms time span. Furthermore, if data collection occurs every five minutes, the five minutes can be divided into the aforementioned intervals. The router could use multiple timers and whenever a timer expires, the router can compute how many bytes are sent out from a particular link. Using this technique, for each five-minute period, the operator will know how many time periods exceeded the threshold(s). The data will not reveal which monitored time periods exceeded the threshold, but may reveal how many time periods experienced utilizations exceeding the pre-set thresholds.
- An output link utilization analysis at different time scales and for different values of the “acceptable utilization levels” or thresholds will help to characterize the frequency of micro-congestion episodes. If a micro-congestion episode is persistent, then its effect should be significant across multiple time scales. For instance, if a link experiences a micro-congestion episode that lasts for multiple, sequential 1 ms intervals, then its effect will “leak” into the higher analyzed time scale, in this case 10 ms. Thus, one way to identify the persistence of micro-congestion episodes in time is by analyzing link utilization at multiple time scales simultaneously.
- Embodiments of the method of the invention characterize micro-congestion episodes with respect to their amplitude, frequency and duration.
- the approach relies on the analysis of link utilization at multiple time scales simultaneously and can provide insight into the time scales at which congestion occurs, while offering information that cannot be recovered by similar analysis at a single time scale.
- the router(s) 20 perform monitoring tasks.
- Optical splitters may be disposed on selected links inside a selected network.
- the system captures the first 40 bytes seen on each unidirectional IP link.
- the collected data sets or packet traces list every packet seen on the monitored link for a specific duration of time (usually limited by the amount of disk space available at the collection station) along with accurate GPS timestamp for its transmission on the link.
- the participating routers may be characterized as “virtual output queued switches” and delays experienced by packets through them are not due to output queuing alone.
- the single-hop delay can be computed for the matches as the difference between the departure and arrival timestamp of a packet at the router. These delay measurements characterize the delay performance experienced by packets on the router path between the monitored input and output link. Nevertheless, the output link receives traffic from other input interfaces, and the input link sends packets to other non-monitored output links.
- the collected single-hop delay measurements approximate a random sample for the delays experienced by any packet on the monitored output link. Thus, the collected data are representative with respect to the amount of time packets need to transit the monitored router and get transmitted from that specific output link.
- Procedure E determines the amplitude of a micro-congestion episode by observing statistics collected for a specific time scale across different utilization thresholds.
- Procedure E further computes the duration of a micro-congestion episode by observing statistics collected for specific values of the utilization threshold across multiple time scales.
- the analysis procedure E may evaluate the frequency of a micro-congestion episode by observing the number of intervals of a specific duration that report a utilization level greater than a pre-selected threshold during the collection interval.
- a method of the invention collects and analyzes four packet traces collected inside the network from OC-3 and OC-12 capacity links. Results were obtained for optical carriers such as OC-3 and OC-12.
- optical carriers such as OC-3 and OC-12.
- the designations of OC-3 and OC-12 are used to specify the speed of fiber optic networks. Transmission speeds for OC-3 are typically 155.52 Mbps and for OC-12 are typically 622.08 Mbps.
- packet traces are selected because they correspond to pairs of monitored links that attach to the same router inside the network. Each pair of packet traces may be labeled as a data set. Those packets that are common to both incoming and outgoing links are identified and referred to as matches. Table 1 includes information about the collected packet traces.
- the amplitude of a micro-congestion episode can be detected through its impact on the delay experienced by packets through a single node.
- the collected delay measurements for the packets that departed within that interval form a set of measurements ⁇ D(n) ⁇ .
- the total number of packets seen on the output link for the same interval n can be used to compute link throughput B as the total amount of traffic transmitted in that same interval.
- each link throughput level (quantized per 1 Mbps) is associated with a delay distribution.
- results describe the delay behavior experienced by packets for specific levels of output link utilization.
- the results also describe the relationship between the intensity of an output burst and the probability of increased delay due to cross-traffic. If the link utilization is high and the delays measured are high, then measurements reveal buildup of queues inside the router. If the link utilization is high and the delays experienced are small, then measurements simply reveal packets going through the router at line rate.
- the router was used to measure link throughput at intervals of duration ⁇ equal to 1 ms, 10 ms, 100 ms, 1 sec and 5 minutes and to investigate the relationship between the delays experienced by packets while transiting the monitored router and further to investigate the output link utilization.
- Each level of utilization B is accompanied by a delay distribution.
- the results characterize the obtained distribution using the minimum, represented by curve 29 , average, represented by curve 30 , 90th percentile, represented by curve 31 , and 99th percentile represented by curve 32 . All statistics are computed for the levels of link utilization that feature at least 100 delay measurements. The results are summarized in FIG. 4 and FIG. 5 .
- FIG. 4 presents the delay statistics measured for each level of output link utilization for the OC-3 links when ⁇ is equal to 1 ms.
- Link throughput measured over intervals of 1 ms duration approaches the link capacity (155 Mbps) and is accompanied by delays as large as 2 ms through the monitored router.
- the minimum delay reported for each level of output link utilization is approximately the same across the entire range of possible utilizations. This observation indicates that packets may experience minimal delays during a highly utilized interval. Nevertheless, a significant number of packets is likely to spend milliseconds of delay through a single node. Consequently, measurements of output link utilization at 1 ms intervals are able to reveal the amplitude of micro congestion episodes.
- Curves 29 - 32 illustrate minimum, average, 90%, and 99% values respectively.
- FIG. 5 the relationship of the time interval ⁇ to output link utilization is shown at higher values of ⁇ .
- FIG. 5 presents the relationship between the 99th percentile of the delay distribution and the output link utilization for different values of ⁇ .
- Output link utilization measured at 1 ms, shown by curve 33 , and 10 ms, shown by curve 34 may reach the link capacity despite the fact that a 5 minute average hardly reaches 60%.
- the 99th percentile of the delay distribution reaches higher values when ⁇ is equal to 10 ms and significantly drops for values of ⁇ greater than 1 sec.
- Curves 35 - 37 represent 100 ms, 1000 ms, and 5 minute intervals respectively.
- Delay measurements for the first data set occasionally exceed 1 ms with a maximum value of 4 ms.
- ⁇ is equal to 1 ms
- a network operator could select the “acceptable” link utilization levels that relate to acceptable delays through each node. For instance, if one considers 1 ms of delay through a single node to be prohibitive then links should not be allowed to exceed 70% utilization at a 10 ms timescale and 55% utilization at 100 ms timescale. In that case, 99% of the packets going through the router will experience single-hop delays less than 1 ms.
- FIGS. 6 and 7 illustrate the relationship between experienced delays and output link utilization found using the methodology described above.
- curves 38 - 41 represent minimum, average, 90%, and 99% values respectively.
- curve 43 represents 1 ms
- curve 42 represents 10 ms
- curve 44 represents 100 ms intervals.
- the output link utilization for the OC-12 set is much lower than the output utilization for the OC-3 set even when measured over 1 ms time intervals. As a consequence, and due to the higher link rate, delay measurements for the second data set are much lower. The maximum delay in the data set is 500 ⁇ s. In addition, due to the increased line rate the differences between small and large time scale output link utilization measurements are far more significant.
- the output link utilization computed over 5 minute intervals hardly exceeds 300 Mbps (i.e. 40%). Nevertheless, the output link utilization measured over 1 ms intervals reaches 97%.
- the amplitude of the micro-congestion episodes is better revealed when measurements are taken at 1 ms intervals.
- the 99th percentile of the delays experienced by packets during the “overloaded” 1 ms time intervals is 250 ⁇ s as shown by curve 41 of FIG. 6 , while the respective statistic reported over 5 minute intervals is 70 ⁇ s.
- the fact that the 99th delay percentile significantly drops for 10 ms and 100 ms time intervals indicates that the micro-congestion episodes occur infrequently, thus affecting a small number of packets in total.
- micro-congestion episodes can be better revealed in the investigated networks when measurements are taken at intervals of duration equal to or below 10 ms. Given specific delay values that are considered prohibitive, one can identify the link utilization levels that should not be exceeded for bounded delay performance for each set. These thresholds are likely to be defined by the network operator and in accordance to the Service Level Agreements (SLAs) in place. As set forth above, the impact of higher output link utilization on the delay experienced by packets is not a function of the amplitude of the micro-congestion episodes alone, but also the frequency of these episodes and their duration.
- SLAs Service Level Agreements
- the system may analyze the output packet trace “out2” and compute link throughput at the time scales of 1 ms, 10 ms, and 100 ms since these time scales can better reveal the amplitude of an episode as shown above. Then, for different values of th, the method counts the number of intervals within each 5-minute interval in the trace that exceed the specified threshold values.
- FIG. 8 illustrates the results for three time scales and three different values of the as illustrated by curves 50 - 58 .
- Curve 50 represents 100 ms
- curve 51 represents 1 ms
- curve 52 represents 10 ms intervals.
- Curves 53 - 55 represent 10 ms, 100 ms, and 1 ms intervals respectively.
- Curves 56 - 58 represent 10 ms, 100 ms, and lms intervals respectively.
- the threshold value is set to 60%, the highest fraction of “congested” intervals is reported by the 1 ms timescale, but does not equally affect the measurements at higher time scales. Consequently, utilization measurements collected at 1 ms intervals are likely to exceed 60% but do not persist in time and result in lower values at coarser time scales. Once the threshold is set to 70% the difference in the fraction of “congested” intervals across time scales is even greater.
- FIG. 9 shows results for the OC-12 data set for ⁇ equal to 1 ms, 10 ms, and 100 ms and th equal to 50%, 60% and 70% as shown by curves 60 - 63 .
- Curves 60 , 61 , and 64 represent 1 ms, 10 ms, and 100 ms intervals respectively.
- Curves 62 and 63 represent 1 ms intervals.
- Curves 65 and 66 approximate both 10 and 100 ms intervals.
- the system can characterize the amplitude of the micro-congestion episode. If the throughput measured at 10 ms is greater than 60%, then delays may be significant. By observing the results for the same value of the and across time scales, the system is capable of identifying the duration and frequency of the episodes. For instance, if high values of threshold are accompanied by high fraction of overloaded intervals for the smallest time scale alone, it is likely that the micro-congestion episodes do not last for long periods of time.
- FIGS. 10 and 11 show delays experienced by packets for the two data sets during the entire duration of the trace.
- FIGS. 10 and 11 show the delay statistics collected for each 5-minute interval in the trace for data sets 1 and 2 respectively.
- FIG. 10 shows, through curves 70 - 73 , that packets going through the monitored router may experience delays that reach 1 ms.
- Curves 70 - 73 represent average, 90%, 99%, and minimum values respectively. Nevertheless, this behavior is only observed for the first part of the trace until 3 pm. After 3 pm the measured delays show a significant drop. From an operational standpoint it would be beneficial to identify the time in which packets experience larger delays and what the delay amplitudes are.
- FIG. 11 shows the results for data set 2 , which is the OC-12 data set.
- Delays experienced by packets through the monitored router are under 100 ⁇ s and increase toward the end of the trace. Increase in the single-hop delay occurs after 12 pm.
- the last 2 hours are characterized by 60% and 80% of the 10 ms and 100 ms intervals respectively exceeding 50% utilization. Approximately 50% of the 1 ms intervals exceed similar levels of activity.
- the amplitude of the delay can be inferred by observing that 20% of the 1 ms intervals exceed 60% utilization for the same period of time. Nevertheless, given that only the 1 ms timescale reports such high output link utilization values, it is likely that the actual delay values experienced by packets are rather limited.
- Curves 80 - 84 represent minimum, average, 90%, 99%, and maximum values respectively.
- the time scales at which congestion occurs may in fact vary, but appear to be generally below 100 ms.
- the data gathered over multiple time scales can then be collected over 5-minute intervals shedding light into the behavior of Internet traffic at these time scales.
- the collection of such data would provide much greater insight into the dynamic behavior of IP networks than is currently available, as well as what could be obtained by observing performance at a single time scale. Examination of such data, for example, across several links over the same period of time may reveal the manner in which congestion manifests itself globally and/or relates to other events (e.g. link failures, routing table updates. etc).
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Environmental & Geological Engineering (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
Description
TABLE I |
DETAILS OF TRACES |
Set | Link | Speed | #pkts | Avg. Util. | #matches |
1 | in1 | OC-3 | 793-106 | 70 Mbps | 3-106 |
out1 | OC-3 | 567-106 | 60 Mbps | ||
2 | in2 | OC-12 | 1.3-109 | 150 Mbps | 18-106 |
out2 | OC-12 | 1.1-109 | 250 Mbps | ||
Claims (18)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US10/737,922 US8213323B1 (en) | 2003-12-18 | 2003-12-18 | System and method for network performance monitoring |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US10/737,922 US8213323B1 (en) | 2003-12-18 | 2003-12-18 | System and method for network performance monitoring |
Publications (1)
Publication Number | Publication Date |
---|---|
US8213323B1 true US8213323B1 (en) | 2012-07-03 |
Family
ID=46320202
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US10/737,922 Active 2027-08-02 US8213323B1 (en) | 2003-12-18 | 2003-12-18 | System and method for network performance monitoring |
Country Status (1)
Country | Link |
---|---|
US (1) | US8213323B1 (en) |
Cited By (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20110013511A1 (en) * | 2009-07-17 | 2011-01-20 | Dekai Li | End-to-end pattern classification based congestion detection using SVM |
US20140119196A1 (en) * | 2012-10-29 | 2014-05-01 | T-Mobile Usa, Inc. | Network Device Trace Correlation |
US20140341033A1 (en) * | 2013-05-16 | 2014-11-20 | Power-All Networks Limited | Transmission management device, system, and method |
US20140344473A1 (en) * | 2013-05-16 | 2014-11-20 | Power-All Networks Limited | Transmission management device, system, and method |
US20150058474A1 (en) * | 2013-08-26 | 2015-02-26 | Verizon Patent And Licensing Inc. | Quality of service agreement and service level agreement enforcement in a cloud computing environment |
US9065737B1 (en) * | 2012-09-27 | 2015-06-23 | Sprint Communications Company L.P. | Measuring traffic utilization |
US20160216379A1 (en) * | 2012-09-21 | 2016-07-28 | Merlin Technology, Inc. | Centripetal acceleration determination, centripetal acceleration based velocity tracking system and methods |
US9538409B2 (en) | 2012-10-29 | 2017-01-03 | T-Mobile Usa, Inc. | Quality of user experience analysis |
US10237144B2 (en) | 2012-10-29 | 2019-03-19 | T-Mobile Usa, Inc. | Quality of user experience analysis |
US10313905B2 (en) | 2012-10-29 | 2019-06-04 | T-Mobile Usa, Inc. | Contextual quality of user experience analysis using equipment dynamics |
US10412550B2 (en) | 2012-10-29 | 2019-09-10 | T-Mobile Usa, Inc. | Remote driving of mobile device diagnostic applications |
US10938706B1 (en) * | 2019-09-18 | 2021-03-02 | Cisco Technology, Inc. | Systems and methods for providing traffic generation on network devices |
US10952091B2 (en) | 2012-10-29 | 2021-03-16 | T-Mobile Usa, Inc. | Quality of user experience analysis |
US11196648B2 (en) * | 2016-07-28 | 2021-12-07 | Cisco Technology, Inc. | Detecting and measuring microbursts in a networking device |
Citations (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5867483A (en) * | 1996-11-12 | 1999-02-02 | Visual Networks, Inc. | Method and apparatus for measurement of peak throughput in packetized data networks |
US6222850B1 (en) * | 1996-06-27 | 2001-04-24 | International Business Machines Corporation | Dynamic transmit tuning for ethernet device drivers |
US6266531B1 (en) * | 1998-07-01 | 2001-07-24 | Ericsson Inc. | System and method for adaptive thresholds for cell load sharing |
US6278694B1 (en) * | 1999-04-16 | 2001-08-21 | Concord Communications Inc. | Collecting and reporting monitoring data from remote network probes |
US20020105911A1 (en) * | 1998-11-24 | 2002-08-08 | Parag Pruthi | Apparatus and method for collecting and analyzing communications data |
US20030023750A1 (en) * | 2001-07-24 | 2003-01-30 | Erol Basturk | Control method for data path load-balancing on a data packet network |
US6529954B1 (en) * | 1999-06-29 | 2003-03-04 | Wandell & Goltermann Technologies, Inc. | Knowledge based expert analysis system |
US20030128692A1 (en) * | 2002-01-04 | 2003-07-10 | Mitsumori Derek Hisami | Voice over internet protocol (VoIP) network performance monitor |
US20040257999A1 (en) * | 2001-11-16 | 2004-12-23 | Macisaac Gary | Method and system for detecting and disabling sources of network packet flooding |
US20050025114A1 (en) * | 2003-07-31 | 2005-02-03 | Christopher Ware | System and method for adaptive polling in a WLAN |
US7020082B2 (en) * | 2000-05-05 | 2006-03-28 | Nomadix, Inc. | Network usage monitoring device and associated method |
US7020282B1 (en) * | 2000-09-05 | 2006-03-28 | Chung Nan Chang | Simplified secure, swift cryptographic key exchange |
US20060083460A1 (en) * | 2000-10-27 | 2006-04-20 | Tamil Lakshman S | Apparatus and method for optical switching at an optical switch fabric |
US7142507B1 (en) * | 1999-02-25 | 2006-11-28 | Nippon Telegraph And Telephone Corporation | Traffic monitoring equipment and system and method for datagram transfer |
-
2003
- 2003-12-18 US US10/737,922 patent/US8213323B1/en active Active
Patent Citations (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6222850B1 (en) * | 1996-06-27 | 2001-04-24 | International Business Machines Corporation | Dynamic transmit tuning for ethernet device drivers |
US5867483A (en) * | 1996-11-12 | 1999-02-02 | Visual Networks, Inc. | Method and apparatus for measurement of peak throughput in packetized data networks |
US6266531B1 (en) * | 1998-07-01 | 2001-07-24 | Ericsson Inc. | System and method for adaptive thresholds for cell load sharing |
US20020105911A1 (en) * | 1998-11-24 | 2002-08-08 | Parag Pruthi | Apparatus and method for collecting and analyzing communications data |
US7142507B1 (en) * | 1999-02-25 | 2006-11-28 | Nippon Telegraph And Telephone Corporation | Traffic monitoring equipment and system and method for datagram transfer |
US6278694B1 (en) * | 1999-04-16 | 2001-08-21 | Concord Communications Inc. | Collecting and reporting monitoring data from remote network probes |
US6529954B1 (en) * | 1999-06-29 | 2003-03-04 | Wandell & Goltermann Technologies, Inc. | Knowledge based expert analysis system |
US7020082B2 (en) * | 2000-05-05 | 2006-03-28 | Nomadix, Inc. | Network usage monitoring device and associated method |
US7020282B1 (en) * | 2000-09-05 | 2006-03-28 | Chung Nan Chang | Simplified secure, swift cryptographic key exchange |
US20060083460A1 (en) * | 2000-10-27 | 2006-04-20 | Tamil Lakshman S | Apparatus and method for optical switching at an optical switch fabric |
US20030023750A1 (en) * | 2001-07-24 | 2003-01-30 | Erol Basturk | Control method for data path load-balancing on a data packet network |
US20040257999A1 (en) * | 2001-11-16 | 2004-12-23 | Macisaac Gary | Method and system for detecting and disabling sources of network packet flooding |
US20030128692A1 (en) * | 2002-01-04 | 2003-07-10 | Mitsumori Derek Hisami | Voice over internet protocol (VoIP) network performance monitor |
US20050025114A1 (en) * | 2003-07-31 | 2005-02-03 | Christopher Ware | System and method for adaptive polling in a WLAN |
Cited By (21)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20110013511A1 (en) * | 2009-07-17 | 2011-01-20 | Dekai Li | End-to-end pattern classification based congestion detection using SVM |
US20160216379A1 (en) * | 2012-09-21 | 2016-07-28 | Merlin Technology, Inc. | Centripetal acceleration determination, centripetal acceleration based velocity tracking system and methods |
US9065737B1 (en) * | 2012-09-27 | 2015-06-23 | Sprint Communications Company L.P. | Measuring traffic utilization |
US9407525B1 (en) * | 2012-09-27 | 2016-08-02 | Sprint Communications Company L.P. | Measuring traffic utilization |
US9538409B2 (en) | 2012-10-29 | 2017-01-03 | T-Mobile Usa, Inc. | Quality of user experience analysis |
US10952091B2 (en) | 2012-10-29 | 2021-03-16 | T-Mobile Usa, Inc. | Quality of user experience analysis |
US9237474B2 (en) * | 2012-10-29 | 2016-01-12 | T-Mobile Usa, Inc. | Network device trace correlation |
US11438781B2 (en) | 2012-10-29 | 2022-09-06 | T-Mobile Usa, Inc. | Contextual quality of user experience analysis using equipment dynamics |
US10652776B2 (en) | 2012-10-29 | 2020-05-12 | T-Mobile Usa, Inc. | Contextual quality of user experience analysis using equipment dynamics |
US10412550B2 (en) | 2012-10-29 | 2019-09-10 | T-Mobile Usa, Inc. | Remote driving of mobile device diagnostic applications |
US20140119196A1 (en) * | 2012-10-29 | 2014-05-01 | T-Mobile Usa, Inc. | Network Device Trace Correlation |
US10237144B2 (en) | 2012-10-29 | 2019-03-19 | T-Mobile Usa, Inc. | Quality of user experience analysis |
US10313905B2 (en) | 2012-10-29 | 2019-06-04 | T-Mobile Usa, Inc. | Contextual quality of user experience analysis using equipment dynamics |
US10349297B2 (en) | 2012-10-29 | 2019-07-09 | T-Mobile Usa, Inc. | Quality of user experience analysis |
US20140341033A1 (en) * | 2013-05-16 | 2014-11-20 | Power-All Networks Limited | Transmission management device, system, and method |
US20140344473A1 (en) * | 2013-05-16 | 2014-11-20 | Power-All Networks Limited | Transmission management device, system, and method |
US9537780B2 (en) * | 2013-08-26 | 2017-01-03 | Verizon Patent And Licensing Inc. | Quality of service agreement and service level agreement enforcement in a cloud computing environment |
US20150058474A1 (en) * | 2013-08-26 | 2015-02-26 | Verizon Patent And Licensing Inc. | Quality of service agreement and service level agreement enforcement in a cloud computing environment |
US11196648B2 (en) * | 2016-07-28 | 2021-12-07 | Cisco Technology, Inc. | Detecting and measuring microbursts in a networking device |
US10938706B1 (en) * | 2019-09-18 | 2021-03-02 | Cisco Technology, Inc. | Systems and methods for providing traffic generation on network devices |
US11456942B2 (en) | 2019-09-18 | 2022-09-27 | Cisco Technology, Inc. | Systems and methods for providing traffic generation on network devices |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Papagiannaki et al. | Network performance monitoring at small time scales | |
US8213323B1 (en) | System and method for network performance monitoring | |
Papagiannaki et al. | Measurement and analysis of single-hop delay on an IP backbone network | |
CA2270890C (en) | Method and apparatus for measurement of peak throughput in packetized data networks | |
US6108310A (en) | Display of network traffic attributes based on frequency distribution | |
Hohn et al. | Bridging router performance and queuing theory | |
US20100157840A1 (en) | Method and apparatus for one-way passive loss measurements using sampled flow statistics | |
Katti et al. | MultiQ: Automated detection of multiple bottleneck capacities along a path | |
Barford et al. | Comparing probe-and router-based packet-loss measurement | |
EP1548980A1 (en) | A method of monitoring a network | |
US20080137540A1 (en) | Method And Apparatus For Analysing Traffic In A Network | |
Choi et al. | On the accuracy and overhead of cisco sampled netflow | |
US7701852B1 (en) | Method for analyzing performance of a network through measuring and reporting delay in routing devices | |
Erramilli et al. | Monitoring packet traffic levels | |
Bezerra et al. | Deploying per-packet telemetry in a long-haul network: the AmLight use case | |
de O. Schmidt et al. | Towards bandwidth estimation using flow-level measurements | |
Rowshanrad et al. | A queue monitoring system in openflow software defined networks | |
Tlaiss et al. | Troubleshooting enhancement with automated slow-start detection | |
de Oliveira Schmidt et al. | Impact of packet sampling on link dimensioning | |
Papadopoulos et al. | Efficient identification of uncongested Internet links for topology downscaling | |
Ozturk et al. | DIChirp: Direct injection bandwidth estimation | |
Arifler et al. | Network tomography based on flow level measurements | |
CN111355603A (en) | Computer flow analysis method | |
US20230198878A1 (en) | Method and system for network segment performance monitoring | |
Belenki et al. | Analysis of errors in network load measurements |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: SPRINT COMMUNICATIONS COMPANY L.P., KANSAS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:PAPAGIANNAKI, KONSTANTINA;REEL/FRAME:014821/0074 Effective date: 20031215 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
FPAY | Fee payment |
Year of fee payment: 4 |
|
AS | Assignment |
Owner name: DEUTSCHE BANK TRUST COMPANY AMERICAS, NEW YORK Free format text: GRANT OF FIRST PRIORITY AND JUNIOR PRIORITY SECURITY INTEREST IN PATENT RIGHTS;ASSIGNOR:SPRINT COMMUNICATIONS COMPANY L.P.;REEL/FRAME:041895/0210 Effective date: 20170203 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 8 |
|
AS | Assignment |
Owner name: SPRINT COMMUNICATIONS COMPANY L.P., KANSAS Free format text: TERMINATION AND RELEASE OF FIRST PRIORITY AND JUNIOR PRIORITY SECURITY INTEREST IN PATENT RIGHTS;ASSIGNOR:DEUTSCHE BANK TRUST COMPANY AMERICAS;REEL/FRAME:052969/0475 Effective date: 20200401 Owner name: DEUTSCHE BANK TRUST COMPANY AMERICAS, NEW YORK Free format text: SECURITY AGREEMENT;ASSIGNORS:T-MOBILE USA, INC.;ISBV LLC;T-MOBILE CENTRAL LLC;AND OTHERS;REEL/FRAME:053182/0001 Effective date: 20200401 |
|
AS | Assignment |
Owner name: T-MOBILE INNOVATIONS LLC, KANSAS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SPRINT COMMUNICATIONS COMPANY L.P.;REEL/FRAME:055604/0001 Effective date: 20210303 |
|
AS | Assignment |
Owner name: SPRINT SPECTRUM LLC, KANSAS Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:DEUTSCHE BANK TRUST COMPANY AMERICAS;REEL/FRAME:062595/0001 Effective date: 20220822 Owner name: SPRINT INTERNATIONAL INCORPORATED, KANSAS Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:DEUTSCHE BANK TRUST COMPANY AMERICAS;REEL/FRAME:062595/0001 Effective date: 20220822 Owner name: SPRINT COMMUNICATIONS COMPANY L.P., KANSAS Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:DEUTSCHE BANK TRUST COMPANY AMERICAS;REEL/FRAME:062595/0001 Effective date: 20220822 Owner name: SPRINTCOM LLC, KANSAS Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:DEUTSCHE BANK TRUST COMPANY AMERICAS;REEL/FRAME:062595/0001 Effective date: 20220822 Owner name: CLEARWIRE IP HOLDINGS LLC, KANSAS Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:DEUTSCHE BANK TRUST COMPANY AMERICAS;REEL/FRAME:062595/0001 Effective date: 20220822 Owner name: CLEARWIRE COMMUNICATIONS LLC, KANSAS Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:DEUTSCHE BANK TRUST COMPANY AMERICAS;REEL/FRAME:062595/0001 Effective date: 20220822 Owner name: BOOST WORLDWIDE, LLC, KANSAS Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:DEUTSCHE BANK TRUST COMPANY AMERICAS;REEL/FRAME:062595/0001 Effective date: 20220822 Owner name: ASSURANCE WIRELESS USA, L.P., KANSAS Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:DEUTSCHE BANK TRUST COMPANY AMERICAS;REEL/FRAME:062595/0001 Effective date: 20220822 Owner name: T-MOBILE USA, INC., WASHINGTON Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:DEUTSCHE BANK TRUST COMPANY AMERICAS;REEL/FRAME:062595/0001 Effective date: 20220822 Owner name: T-MOBILE CENTRAL LLC, WASHINGTON Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:DEUTSCHE BANK TRUST COMPANY AMERICAS;REEL/FRAME:062595/0001 Effective date: 20220822 Owner name: PUSHSPRING, LLC, WASHINGTON Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:DEUTSCHE BANK TRUST COMPANY AMERICAS;REEL/FRAME:062595/0001 Effective date: 20220822 Owner name: LAYER3 TV, LLC, WASHINGTON Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:DEUTSCHE BANK TRUST COMPANY AMERICAS;REEL/FRAME:062595/0001 Effective date: 20220822 Owner name: IBSV LLC, WASHINGTON Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:DEUTSCHE BANK TRUST COMPANY AMERICAS;REEL/FRAME:062595/0001 Effective date: 20220822 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 12TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1553); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 12 |