US20040001493A1 - Method and apparatus for queuing data flows - Google Patents
Method and apparatus for queuing data flows Download PDFInfo
- Publication number
- US20040001493A1 US20040001493A1 US10/180,736 US18073602A US2004001493A1 US 20040001493 A1 US20040001493 A1 US 20040001493A1 US 18073602 A US18073602 A US 18073602A US 2004001493 A1 US2004001493 A1 US 2004001493A1
- Authority
- US
- United States
- Prior art keywords
- data
- packet
- latency
- queue
- packets
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims description 59
- 230000006870 function Effects 0.000 claims description 57
- 238000004364 calculation method Methods 0.000 claims description 9
- 230000005540 biological transmission Effects 0.000 abstract description 3
- 238000004422 calculation algorithm Methods 0.000 description 28
- 238000012360 testing method Methods 0.000 description 8
- 238000011144 upstream manufacturing Methods 0.000 description 8
- 238000012546 transfer Methods 0.000 description 6
- 230000001934 delay Effects 0.000 description 5
- 238000013459 approach Methods 0.000 description 4
- 239000000872 buffer Substances 0.000 description 4
- 239000000835 fiber Substances 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000003491 array Methods 0.000 description 3
- 238000000605 extraction Methods 0.000 description 3
- 239000004065 semiconductor Substances 0.000 description 3
- 238000010276 construction Methods 0.000 description 2
- 125000004122 cyclic group Chemical group 0.000 description 2
- 230000007423 decrease Effects 0.000 description 2
- 230000006735 deficit Effects 0.000 description 2
- 230000008520 organization Effects 0.000 description 2
- 230000037361 pathway Effects 0.000 description 2
- 238000001228 spectrum Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 241001522296 Erithacus rubecula Species 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000003466 anti-cipated effect Effects 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000004069 differentiation Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 239000012530 fluid Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
- 230000004043 responsiveness Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/24—Traffic characterised by specific attributes, e.g. priority or QoS
- H04L47/2425—Traffic characterised by specific attributes, e.g. priority or QoS for supporting services specification, e.g. SLA
- H04L47/2433—Allocation of priorities to traffic types
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/24—Traffic characterised by specific attributes, e.g. priority or QoS
- H04L47/2458—Modification of priorities while in transit
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/28—Flow control; Congestion control in relation to timing considerations
- H04L47/283—Flow control; Congestion control in relation to timing considerations in response to processing delays, e.g. caused by jitter or round trip time [RTT]
Definitions
- This invention relates to cable data systems.
- this invention relates to a cable modem termination system (CMTS) and a method for intelligently controlling data flow queues in a CMTS
- CMTS cable modem termination system
- a cable television or “CATV” system is a television broadcasting system that uses coaxial cable or fiber optic cable to distribute a broadband signal containing many separate television program channels that are frequency-multiplexed onto the cable. By allocating the frequency spectrum of one or more television program channels for data transmission, CATV systems are also used to carry data to and from CATV subscribers.
- Data on a cable television system is sent as packets that resemble well-known Ethernet packets.
- Each data packet can have up to several hundred data bytes preceded by several critical data bytes that comprise the frame's header.
- the header includes a preamble, a frame type, a source address and a destination address.
- a cyclic redundancy check or CRC is appended to the end of the packet for error detection.
- Subscribers' computers are coupled to the cable via a cable modem.
- Each cable modem is assigned an address by the head end when the modem is turned on.
- the modem's address effectively identifies the computer to which it is coupled and enables each subscriber to have individualized connections to the Internet.
- Downstream data packets are those sent from the head end of the cable system to the subscribers.
- Upstream packets are those sent from the subscribers to the head end. Once a data packet is physically transmitted onto the cable, all of the computers that are coupled to the cable can “see” the packets. By using Ethernet-like data frames however, the cable modems of the system read only the downstream data packets that are addressed to it and ignore all other packets.
- the cable modem provides to the data packets, the address assigned to the cable modem. That address of the sending cable modem effectively identifies the subscriber from which the packets originated and in so doing, also provide an address to which responsive packets can be sent.
- CMTS cable modem termination system
- the CMTS performs a variety of critical functions in a cable data system, including among other things, cable modem address assignments and data packet address translation by which packets received by the CMTS from an Internet connection are addressed (directed) to the proper cable modem.
- a CMTS also functions as a switch by taking upstream packets from the cable modems and routing them toward their proper destinations.
- a CMTS includes one or more ports through which data is received and sent. Different ports are typically coupled to different pathways such that different ports lead to different logical destination addresses. By way of example, data from a subscriber received at one port of the CMTS might need to be routed out a different port in order for the packets to be routed to their destination (as determined by the destination address in the packet's header). In so doing, a CMTS receives upstream data packets, either from subscribers or Internet servers, and routes said packets to the port on which the intended recipient, or server for that recipient, resides as dictated by the packet's destination address.
- CableLabs® which is a non-profit research and development consortium of cable television operators within North and South America. The CableLabs® web site URL is: www.cablelabs.com.
- Quality of service is concept of guaranteeing a throughput or a data transfer level. It is anticipated that next-generation cable data systems will offer customers guaranteed bandwidths (data rates) to customers willing to pay for it. Customers who pay for data rate guarantees will have their data packets routed through the cable data system ahead of lower-paying customers.
- the concept of being able to guarantee (and to be able to truly provide) different service or priority levels allows cable service providers the opportunity to provide better Internet service, as measured by connection bandwidth, to needy customers while charging a premium rate. Tiered service is expected to increase revenue for cable service providers.
- One way of providing service levels is to organize Internet traffic into data flows.
- Data packets sent from the head end of the cable data system to a particular cable data system subscriber can be considered to be as one “data flow.”
- the upstream packets from the same customer, or some other customer can be considered to be another “data flow.”
- the data flows that they represent can be allotted different service (i.e., priority) levels by reading from a class of service list that a particular subscriber is entitled to.
- CMTS Part of the job of a CMTS is to route incoming packets to their destinations, as determined by their packet destination addresses. In a CMTS, this usually requires that packets received at one port need to be switched out of the CMTS on a different port that is connected to a different physical or logical pathway leading to different logical destinations.
- data packets are stored in priority queues, which are known to those of skill in the networking art to be multi-element lists, each element of which is assigned a priority value or level that determines when the element is to be removed from the queue
- the primary task behind any Ethernet frame-based scheduling algorithm is to examine the contents of the multiple queues that are aimed at a particular shared resource, e.g., a channel time-slot on the cable or a port of the CMTS, and then decide which queue should transmit the next Ethernet frame to that shared resource.
- a relative scheduling algorithm must make decisions regarding the relative treatment each of the queues based on the priority level associated with that particular queue. In general, any scheduling algorithm must make trade-offs between fairness, delay, and implementation complexity.
- GPS Generalized Processor Sharing
- WFQ Weighted Fair Queuing
- Another approach that attempts to provide fair service includes the CORR algorithm disclosed by D. Saha, S. Mukherjee, and S. Tripath, as “Carry-Over Round Robin: A simple cell scheduling mechanism for ATM networks” and Deficit Round Robin Queuing (M. Shreedhar and G. Varghese, “Efficient fair queuing using deficit round robin”).
- these algorithms distort fairness when a particular queue is filled with very long packets while other queues are filled with very short packets. In such a situation, the queue with the long frames will steal more than its fair share of bandwidth on the shared resource.
- a method and apparatus that monitors the latency of every packet passing through the system and uses that information to dynamically adjust (in real time) the performance of the scheduling algorithm would provide improved queuing of data flows in a cable modem termination system and quality of service level differentiation. Such a method would be an improvement over the prior art.
- FIG. 1 shows a cable data system including a representation of a cable modem termination system employing the queuing method and apparatus disclosed herein.
- FIG. 2 shows a representation of two queues used to store data packets for each port in a CMTS that recognizes two different priority levels.
- FIG. 3 shows an exemplary data packet.
- FIG. 4 shows an exemplary data structure.
- FIGS. 5A and 5B show the steps of a method to normalize data flows through several different queues.
- FIG. 1 shows a block diagram of a cable data system 10 that provides data flow queuing according to the preferred embodiment.
- the several computers 12 of the subscribers to the cable data system 10 are operatively coupled to the coaxial cable 16 of a distribution network such as a CATV network (fiber optic cable in some embodiments) over which data packets are transmitted to and received from a data system 18 .
- a distribution network such as a CATV network (fiber optic cable in some embodiments) over which data packets are transmitted to and received from a data system 18 .
- the computers 12 in FIG. 1 are sub-divided into two groups: Group 44 and Group 46 .
- the data system 18 is a cable modem termination system or “CMTS.”
- CMTS cable modem termination system
- the coaxial cable 16 and the CMTS 18 form at least part of a data distribution network. Together, the cable 16 and the CMTS enable the computers 12 to communicate with each other as well as with other computers of the Internet.
- the CMTS 18 shown in FIG. 1 has four ports (denominated 0-3) into which packets are sent and received. Ports 0 and 1 are each coupled to respective groups of computers 12 . Computers of Group 46 are coupled to port 0; computers of Group 44 are coupled to port 1. Ports 2 and 3 are coupled to different servers that each provide access to the Internet backbone.
- One function of the CMTS 18 is to route data packets to their destinations. Data packets can be sent into one port and routed out of another port. In so doing, the CMTS 18 acts as a switch, routing data packets to their logical destinations.
- Port 0 is coupled to the computers in Group 46 .
- Port 1 is coupled to the computers in Group 44 .
- Ports 1 and 2 are coupled to Internet backbone gateways.
- data packets from group 46 computers (received at port 0) can be switched to output ports 0, 1, 2 or 3.
- the CMTS 18 therefore not only enables group A computers 46 to communicate with the set of all computers 12 , but also to communicate with any other computer linked to the Internet backbone accessible via the internet backbone 14 .
- the CMTS 18 needs to intelligently treat packets in higher-priority data flows with a greater level of precedence than packets in lower-priority data flows. This can result in higher-priority data flows getting preferential treatment in many different ways, ranging from offering them more bandwidth on a shared resource to offering them more rapid transport (lower delay and lower latency) when passing through the CMTS.
- CMTS system design Some portions of the CMTS system design (such as the congestion control algorithms that run on each of the input ports) are entirely dedicated to the control of bandwidth associated with each of the data flows. In most commercially-available CMTS systems, the scheduling algorithms that determine the order by which packets are transmitted out of the system are also dedicated to the control of bandwidth associated with each of the data flows.
- a novel scheduling algorithm that determines the order by which packets will be transmitted out of the CMTS can be used to control both the bandwidth and the delay associated with each data flow.
- high-priority packets can be written to a high priority queue, while low priority packets can be written to a low priority queue.
- the service level agreement parameters can be used to define the priority of a packet. The association between a service level agreement and a particular data flow is established when a subscriber's data flow is registered within the CMTS 18 . This registering of the subscriber's service level agreement usually occurs when the subscriber's cable modem is first powered up and makes contact with the CMTS 18 .
- CMTS 18 When a packet passes through CMTS 18 , a classification of the packet into a particular data flow must be performed. Since each data flow is associated with a subscriber who registered for a particular service level, each data flow is then associated with a unique priority. A table look-up (usually into a memory) using the unique priority as a key produces the service level agreement parameters that are associated with the priority. By association, these service level agreement parameters are also associated with the data flow and packet. Examples of service level agreement parameters that can be attached to a priority level include the minimum guaranteed throughput (packets per second), the maximum permitted throughput (packets per second), the probability that general service will be available to the customer, and the probability that a packet from a data flow will be dropped during periods of congestion.
- latency is defined as the interval of time required for the packet to pass between two specified points within its overall path.
- the two specified points are defined to be an input port on the CMTS and an output port on the CMTS, although other points could be specified.
- the actual latency value experienced by any packet passing through the CMTS can be easily calculated by having a counter or free-running clock that specifies time relative to an arbitrary reference time. If the counter or clock is continually incremented as time passes, then a sample of the value in the counter or clock when a packet passes the first point can be designated as the packet's arrival time.
- a sample of the value in the counter or clock when the packet passes the second point can be designated as the packet's departure time. Subtracting the arrival time from the departure time yields the actual latency value for the packet.
- the current latency value is the actual latency value that the packet would experience if it were immediately selected by the scheduling algorithm to be sent to its destination (whether it is actually selected or not).
- the current latency value provides an approximate measure describing how long the packet has been stored in the queue of the CMTS (since delays in other parts of the CMTS system are usually negligible).
- a data packet that has been stored in the queue longer than any other data packet in the queue is said to be the oldest data packet in the queue.
- Examples of desired latency characteristics include the minimum desired latency and the maximum desired latency. Specification of a minimum desired latency and a maximum desired latency yields a desired range for the actual latency for a packet.
- the scheduling algorithm in a CMTS should attempt to transmit packets to their destination so that the actual latency experienced by the packet is greater than or equal to the minimum desired latency and less than or equal to the maximum desired latency.
- guaranteeing a minimum desired latency implies that the packets of some flows should be made to wait a minimum time before being routed. This may be desirable if it gives customers the incentive to pay for higher-priority service levels.
- Packets that have waited a minimum time so that their current latency value is, greater than or equal to the minimum desired latency are said to be eligible for scheduling, and they are called eligible packets. Packets that have not yet waited a minimum time (so that their current latency value is less than the minimum desired latency) are said to ineligible for scheduling.
- Priority levels are usually assigned by a service provider according to the service level that a particular customer wants (or pays for), according to the type of service level that the carried-information requires. Packets of information that carry voice (i.e., telephone) over the Internet (VoIP) need to be routed quickly (at a high-priority level) so as to preserve the integrity of a conversation. Data packets representing an e-mail message can usually be sent at a lower-priority level.
- CMTS Compute resource pool
- Different data flows to or from the same address can also require different priority levels.
- An Internet service subscriber might want a very high service level for voice over IP, a slightly lower service level for browsing the web and an even lower service level for e-mail.
- the voice traffic, web browser and e-mail can each be considered a separate data flow, each of which can have a different service level.
- Data packets of flows are stored or queued a finite amount of time in the course of being switched or routed.
- the CMTS 18 uses a scheduling priority and packet latency in order to determine which flow should have its queued data packets routed to the packet's next destination.
- a packet's destination is determined by its destination address.
- a packet is effectively “delivered” to a destination by routing (switching) it to a CMTS 18 port that will carry the packet toward the packet's destination address.
- a “scheduling priority” is a scheme whereby higher priority packets are routed ahead of lower-priority packets as determined by the class of service that a customer is entitled to.
- a packet or a flow “priority” has no meaning per se.
- a “priority” accorded to one packet or stream, is meaningful only with respect to a different priority accorded to a different packet or stream.
- the priority of a packet, data stream or a data queue is only meaningful when it is expressed relative to the priority of other packet packets, data streams or queues with different priorities.
- the term “relative scheduling priority” therefore only means that the priority accorded to a packet of a particular stream is determined relative to the priority accorded to packets of other streams.
- a “relative scheduling priority” therefore only means that one scheduling priority can be higher, lower or the same as another scheduling priority.
- each packet can be assigned a relative scheduling priority such that some packets should normally be routed to their destinations ahead of others.
- the specification of when a packet of a particular relative scheduling priority should be routed to its destination can be established by specifying or assigning one or more of the aforementioned desired latency characteristics to each relative scheduling priority to be accorded the packets of different streams.
- each packet can be assigned a minimum wait time, a maximum wait time, both a minimum and a maximum wait time, depending on the relative priority (i.e., the relative scheduling priority) to be accorded a packet.
- the priority level of a packet can be determined by looking up the packet's source or destination address and linking the packet's source address or the destination address or both, to the service level(s) that the source or destination is (are) entitled to.
- the priority of a packet can also be determined by looking at other fields in the packet header, and using the values in those fields to look up the associated priority level of the packet. This priority level can then be appended as a data value to each frame of a flow.
- the priority level is determined by reading the source field for upstream packets (or destination field, in the case of downstream packets) and from the identity of the source (destination), looking up the service level that the source (destination) is entitled to.
- CMTS 18 receives all such data flows and stores the packets to priority specific queues.
- the process of queuing data packets entails writing data packets into a data storage device, such as a semi-conductor random access memory, and reading the packets from the memory at a later time, albeit in the order in which the packets were stored.
- Priority queuing is different: packets are read from the queue in an order that follows the priority assigned to the queued elements.
- priority queuing of multiple flows is accomplished by storing high-priority packets in a high-priority queue; low priority packets are stored in a low-priority queue. Packets are read from the queues (low priority and high priority) after taking into account both the latency and priority of packets currently stored in each queue.
- Data packets from a subscriber such as computer 12 - 1 are received at the CMTS 18 through its port 1 and are from there, coupled to the port steering element 20 so as to route the data packets to their respective destinations.
- the destinations are determined by the destination address in each packet.
- a packet's destination address can be an Internet protocol (“IP”) address, among others.
- IP Internet protocol
- the port steering element 20 in the preferred embodiment steers Ethernet packets according to one of two established protocols. Both protocols use unique destination identifier fields residing within the header of each Ethernet Packet. Those of skill in the art of data networks know both protocols. The first protocol is used in a “switch” based network and relies on the Destination Media Access Control (DMAC) address for destination determination. The second protocol is used in a “route” based network and relies on the Destination Internet Protocol (DIP) address for destination determination. In both instances, the steering element indexes a lookup table, usually composed of data stored in a random access memory element, with the DMAC or DIP address, or some derivatives thereof. The result of this lookup operation describes where the packet is to be sent.
- DMAC Destination Media Access Control
- DIP Destination Internet Protocol
- the port steering element 20 switches or routes incoming data packet's received at each port 0-3 to at least one set of queues 24 , 26 , 28 , 30 for each port 0-3. For each port, there is a separate queue to store the data packets for each service level.
- the CMTS 18 has four ports and supports two service levels per port, each port 0-3 would have two separate queues. The entire CMTS would therefore support a total of eight queues or queue equivalents.
- the queues for each port store data packets corresponding to a service level or set of service levels that are to be routed to the corresponding port.
- the CMTS 18 in FIG. 1 has four (4) ports, each of which have a queue into which high priority service level packets are written and a queue into which low priority service level packets are written. Accordingly, CMTS 18 provides two different service levels.
- the port steering element 20 examines a destination address in the packet header, determines the service level accorded to the data packet, then routes each data packet through a time-stamp circuit 22 and on to the appropriate queue for each output port.
- the time-stamp circuit 22 appends a scalar to each packet (also known as a frame) that identifies when the packet arrived at the CMTS 18 .
- time-stamp circuit 22 could be placed in several other meaningful locations within the CMTS 18 , including at the actual input port or above the port steering element. If transit delays through the circuitry of the CMTS 18 are minimal, the differences in time-stamps created by the time-stamp circuit 22 when placed at these different locations should be negligible.
- the CMTS 18 can account for the delay associated with said steering element (and other circuitry) by reducing the service level latency characteristics by the value of said steering element latency (and other circuitry latency).
- the time stamp circuit 22 is a timer that assigns a multi-byte integer representing the current time.
- the data packet is stored to its corresponding priority queue of its destination port.
- the scheduling algorithm presented herein determines the packet's current latency by subtracting the scalar representing the time of arrival from the current time (which is obtained from the time-stamp circuit's timer or from another timer which is roughly synchronized to the time-stamp circuit's timer).
- the resulting packet latency (also a scalar) is then reduced by the minimum latency requirement of its associated service level, TMN, and then normalized. This result is used to determine the next packet to send from the CMTS.
- packet delivery scheduling uses a latency characteristic assigned to a packet's scheduling priority.
- the queues for the ports are part of a queue structure 25.
- queues are collections of information, the contents of which are organized according to rules.
- One queue organization rule uses the order of information entry. Two queues organized by such a rule are commonly known as First-In-First-Out (FIFO) or Last-In-First-Out (LIFO) queues.
- Another queue organization rule is based upon priority ordering whereby the extraction of information, data, or other queue elements therefrom is determined according to an importance, value, significance or, in the case of data networks, service level assigned to or associated with each element in the “priority queue”.
- Still other rules use priority ordering in conjunction with order of entry to determine the order of extraction. The factors, configurations, and techniques leading to queue ordering forms the science known as queuing theory.
- the queue structure 25 is implemented using an array of semiconductor memory organized into N queues or data structures, where N is equal to the number of queues for the entire system.
- N is equal to the number of queues for the entire system.
- Each data structure of the N structures is associated with, or stores data packets for, a certain service level (i.e. priority level) and will contain a variable number of elements, the length of each of which is also variable.
- the elements stored in the data structures/queues are Ethernet packets. The length of each element in the queue is variable because Ethernet packets are of variable length.
- the data structures are organized as FIFO queues.
- CMTS 18 which provides two service levels and which has four ports, has the queue structure 25 organized into eight queues (or data structures). For each of the four ports 0-3, the data packets to be routed from each port are stored into a queue for each priority level. As a result, each port 0-3 has a first queue for the high priority level packets and a second queue for the low priority level packets, resulting in a total of eight queues.
- a delivery sequence of a high-priority upstream packet from the subscriber computer 12 - 1 (residing on port 1 ) to the subscriber computer 12 - 2 (residing on port 0) is as follows: a packet from computer 12 - 1 is received on port 1 of CMTS 18 ; the steering element 20 determines service level associated with said packet and forwards the packet to the high priority queue associated with port 0; the data packet then awaits scheduling for port 0 delivery according to a data packet extraction scheduling rule described below; upon receiving a packet delivery permission grant from the port 0 scheduling engine, the packet is transmitted through port 0 and decoded by the destination subscriber computer 12 - 2 . Similarly, an upstream packet from subscriber 12 - 1 that is a low-priority packet and which is addressed to another computer somewhere on port 1 will be routed to the low priority queue for port 1.
- FIG. 2 shows only two queues: the first queue 36 stores low priority data packets; the second queue 38 stores high priority packets. Both queues 36 and 38 store packets for the same output port. Similarly, all other ports are provided with a queue for each service level.
- Other embodiments of the invention would include providing more than two priority service levels or a mixture of priority service levels (and therefore includes a number of queues other than two queues at each port) as well as data systems (i.e., the CMTS) with more than or less than, four ports.
- Such alternate embodiments that employ the scheduling method disclosed herein would have for each port, a data packet queue for each service level that is not necessarily limited to a set value. For example, one port could provide two (2) different service level priorities while a second port could provide four (4) service level priorities while still other ports could be provide even a greater number of service level priorities.
- FIG. 3 depicts a single Ethernet data packet (also known as a “frame”) 40 .
- Ethernet frames in general can have between 64 and 1518 bytes. The first several bytes (usually the first eighteen) comprise the header of an Ethernet packet. The next forty-six to fifteen hundred bytes are data.
- the packet header shown in FIG. 3 includes the preamble 42 , destination address 44 , source address 46 , packet type 48 and time stamp 50 .
- the data field 51 (which is the information to be carried to the packet's destination) of the packet 40 follows the header.
- a trailer field that is a cyclic redundancy check, or “CRC” 53 is used to detect errors in the packet 40 is the last field.
- the time stamp 50 is “appended” to the data packet 40 by being linked to the packet; the time stamp is not part of the header.
- the data packet's time of arrival is considered to be “assigned” to the data packet (or “associated” with a data packet) whether the data representing the time of arrival is actually within (or embedded in) the header of the packet or the time of arrival is linked to the header and stored separately.
- Each packet is accorded a scheduling priority. As described above, a packet's scheduling priority can be determined from a packet's source address, destination address, or other fields in the packet's header. Each packet is routed to its destination address by the CMTS 18 according to the (relative) priority that it is entitled to.
- FIG. 4 depicts a structure of a priority queue 66 of data packets 62 .
- Data packets 62 can be written into the queues at different times or at the same time.
- Each of the multi-byte packets stored in the queue 66 has the same priority or service level, even though the packets might be from different flows and represent different types of information.
- data packets of an e-mail message of one flow can be stored in the same queue (i.e., queued) with data packets of a different flow, which could be packets for a Voice-over-IP stream or packets for a stream to or from a web browser.
- four scheduling engine processors/circuits denominated 32 - 0 , 32 - 1 , 32 - 2 and 32 - 3 , determine the order in which packets are read from the different queues for the different ports.
- Scheduling engine 32 - 0 determines the order for packets read from the set of queues 24 (which queues contain packets destined for output port 0), scheduling engine 32 - 1 determines the order for packets read from the set of queues 26 (which queues contain packets destined for output port 1), scheduling engine 32 - 2 determines the order for packets read from the set of queues 28 (which queues contain packets destined for output port 3 ), and scheduling engine 32 - 3 determines the order for packets read from the set of queues 30 (which queues contain packets destined for output port 2 ).
- the scheduling engines 32 schedule the delivery of data packets. They read data packets out of each of the queues of the queue structure 25 according to a scheduling methodology by which the waiting time (hereafter referred to as “latency”) of each data packet in each data queue is modified (shifted as described below) and then normalized or scaled. The resulting scaled latency of each data packet in a set of queues is compared to the other scaled latencies of other data packets within the same set of queues to identify the packet in that particular set of queues with the greatest scaled latency. The greatest scaled-latency packet of all packets within a set of queues is the next packet that will be transmitted to the output port associated with that set of queues. The result of this selection allows the packet with the greatest scaled latency to make its way through its associated output port toward its desired destination before all of the remaining packets that are still stored within the particular set of queues.
- the process of normalizing or scaling the actual latency of data packets includes in the scaled latency calculation, both the length of time that a packet has been in queue and also the priority level that a packet is entitled to.
- a latency scaling factor is assigned to each queue.
- the assignment of a latency scaling factor to a queue defines the latency performance of that queue. Accordingly, small latency scaling factors typically lead to small scaled latencies while large latency scaling factors typically lead to large scaled latencies (although the actual scaled latency is also a function of the actual latency associated with the data packet). In the scheduling process, large scaled latencies are scheduled prior to small scaled latencies.
- the CMTS 18 needs to be able to decide which of the many packets stored within a set of queues associated with a particular output port should be next routed to its destination.
- high priority data flows will be granted permission to transfer data packets ahead of low priority data flows.
- low priority data flows should be granted permission to transfer data packets ahead of even higher priority packets if a low priority packet has been in a queue longer than allowed by the customer's class of service or by the customer's service level agreement with the service provider.
- True quality of service can be realized if the waiting time of the data packets in each queue is normalized by factoring into the normalization, the priority level that each packet is entitled to receive, with the oldest packet in each data queue:
- FIGS. 5A and 5B depict a flow-chart representation of the method for scheduling the delivery of data packets from a particular set of queues to a particular output port, using at least one latency characteristic and a packet's relative scheduling priority.
- a unique and separate instantiation of this scheduling algorithm could be implemented in each Scheduling Engine 32 for each of the output ports in the CMTS 18 , or a single instantiation could be implemented in a common Scheduling Engine (not shown) and could be re-used with a unique time-slice and a unique set of state information being dedicated to each of the output ports in the CMTS 18 .
- a unique and separate instantiation of this scheduling algorithm is implemented for each of the output ports 0-3.
- the scheduling method has different sets of related steps (denominated herein as “phases”), each of which includes one or more computational steps.
- phases each of which includes one or more computational steps.
- data packets received from the port steering element 20 are written into the appropriate data queues for the particular port.
- Port steering element 20 uses destination information (such as the DMAC address or the DIP address) and the packet's priority level information to direct the packets to their appropriate queues.
- the current latency for each packet is calculated in step 54 - 1 , and then the oldest packet in each queue is identified.
- the oldest packet will reside at the “head” of the queue and the youngest packet will reside at the “tail” of the queue.
- FIFO implementations of a priority queue do not require phase 54 because all head packets are the oldest packets (with the greatest latency) within the buffer.
- the oldest packet in each queue is tested against a minimum desired latency threshold specified for the relative priority level associated with all of the packets in that queue. This test determines whether the oldest packet in a queue is eligible for scheduling. An eligible packet is one that has a current latency greater than or equal to a minimum desired latency threshold value.
- all eligible packet latencies are shifted by their minimum desired latency value defined for the relative priority associated with all of the packets in their respective queue.
- the shifting operation is accomplished by subtracting the current latency of each eligible packet by the minimum desired latency threshold.
- the result of each shifting operation is scaled according to the appropriate scaling factor (hereafter referred to as the “J FACTOR ”), which is associated with the relative priority for the packets in the queue.
- J FACTOR scaling factor
- a test is then performed in phase 59 (shown in FIG. 5B) to determine if the scaled latency for the eligible packet is greater than a desired target scaled latency.
- Another embodiment performs this test by determining if the actual latency for the eligible packet is greater than the desired latency for this queue (the results are equivalent to the scaled version of the test). The results of this test are used in the fifth and final phase 60 .
- a latency violating queue is considered to be any queue which contains at least one eligible packet whose scaled latency is greater than the target scaled latency for that queue.
- the particular one packet that will be selected next for transmission by the Scheduling Engine 32 will be associated with the latency violating queue whose relative priority is higher than all of the other latency violating queues (which is effectively a form of strict priority queuing between all of the latency violating queues).
- the eligible packet associated with the largest scaled latency will be selected and shipped to the output port.
- Phase 54 determines the maximum latency of all the packets in a queue of length “x” elements.
- step 54 - 1 the time that a first data packet “i” spent in a queue is determined.
- a packet's current latency can be readily determined by calculating the current time (as provided by either the global clock or another real-time clock) minus the time stamp value attached to or associated with the packet as it passed through the time stamp 22.
- “i” equals 1 and the comparison step 54 - 2 will store the packet latency associated with the first evaluated packet to L MAX0 in step 54 - 4 .
- step 54 - 5 the value of the loop counter “i” is tested to see if the end of the queue has been reached. If there are more packets to evaluate, the loop counter is incremented at step 54 - 3 so as to point to the next packet.
- the latency of every packet is iteratively compared to the maximum latency of all previously tested packets.
- said value is stored in a register or memory location called “L MAX0 ” in step 54 - 4 .
- L MAX0 the maximum latency value for packets within the queue.
- the maximum latency value L MAX0 is then used in phase 56.
- the scheduling engines 32 of each queue determine if the oldest packet in the queue, which is L MAX0 , is eligible for transfer to its appropriate output port.
- packet eligibility is determined by whether the packet's latency is greater than or equal to a desired minimum latency, which in some embodiments can also be zero.
- a desired minimum latency which in some embodiments can also be zero.
- priority flows can be made to wait a minimum amount of time, even if there is no other data traffic waiting to be routed. This can be accomplished by holding the packets for a service level in a queue until the resulting delay exceeds the minimum desired latency (T MIN0 ) associated with their priority.
- T MIN0 at step 56 - 2 is a minimum amount of time that a packet needs to wait in a particular queue for a particular service level.
- T MIN0 at step 56 - 2 can be zero or greater than zero.
- the shifting operation (L MAX0 ⁇ T MIN0 ) is an example of a “second function of said packet's latency”, where the actual (unshifted) latency of the oldest packet is specified as L MAX0 .
- each of the shifted latency values i.e. L MAX1 ⁇ T MINi
- This J FACTOR is related to at least one queue latency characteristic associated with a particular queue.
- the J FACTOR (i) (which is associated with queue i) is also used to normalize the desired latency characteristics to create a value that is hereafter referred to as the “TARGET_SCALED_LATENCY”, or simply “TSL”). Queues with a larger J FACTOR approach the TSL more quickly than queues with a smaller J FACTOR . Packets that exceed TSL have waited in their queue longer than their time allotted by T MAX , whereas queues that fall below TSL are below their allotted T MAX . The scaled result of different queues can therefore be quantitatively compared to determine the packet that is most deserving to be scheduled based upon the queue's latency characteristics, T MIN (i) and T MAX (i).
- J FACTOR ′(i) is a function of the inverse of the difference between T MAX (i) and T MIN (i).
- the difference operation (T MAX (i) ⁇ T MIN (i)) is an example of a “third function of the at least one desired latency characteristic”, where this particular third function actually includes two desired latency characteristics: T MAX (i) and T MIN (i).
- J FACTOR ′(i) is a fractional quantity.
- the time stamp element 22 is not necessarily continuous and can increment once every time period (hereafter referred to as “T_TICK”).
- MAX_RANGE MAX ⁇ T MAX (i) ⁇ T MIN (i) ⁇ for all i
- a resolution factor R required to provide adequate resolution of the priority level with the largest maximum range.
- a resolution factor R value of 10 is arbitrarily chosen. When implemented using integer arithmetic, this results in a scaled latency value with a range of at least 0 to 10.
- the multiplication operation of ((L MAX0 (i) ⁇ T MIN0 (i))*(J FACTOR (i))) can be shown to be the same as the division operation of ((L MAX0 (i) ⁇ T MIN0 (i))/(T MAX (i) ⁇ T MIN (i))), and this division operation is an example of a “first function of the packet's latency and the at least one desired latency characteristic assigned to the relative scheduling priority for said packet.”
- this first function is comprised of a ratio of the second function of said packet's latency to the third function of the at least one desired latency characteristic.
- each of the latencies of the oldest packet in each queue is normalized such that a comparison of the different scaled latencies identifies which of the oldest scaled latencies should be next routed to its destination based not only upon its time in queue but also its priority or service level.
- phase 60 The last phase of the scheduling algorithm (phase 60 ) is the selection function that actually picks an eligible packet to be transmitted to the output port. Assuming that strict priority scheduling is not necessary (due to the comparisons in phase 59 ), phase 60 sorts the scaled latencies determined at step 58 - 2 to find the maximum-scaled latency for the set of queues associated with an output port. In the first execution of the loop 60 - 4 , “i” equals 0 and the comparison step 60 - 5 will store the scaled latency associated with the first evaluated queue to SL MAX in step 60 - 6 . At step 60 - 7 , the value of the loop counter “i” is tested to see if the last queue has been reached.
- the loop counter is incremented at step 60 - 8 so as to point to the next queue.
- the scaled latency of every queue is iteratively compared to the maximum scaled latency of all previously tested queues. In the event that a new maximum-scaled latency is determined, this value is stored at SLMAx in step 60 - 6 .
- the value stored at SL MAX will be the maximum-scaled latency value for the set of all queues for a particular port.
- the data packet associated with the maximum scaled latency value is then scheduled for delivery to the output port. If the queues depicted in FIG. 5 are for a port 0, for example, a packet associated with the maximum scaled latency as determined in phase 60 will be output to port 0.
- This large amount of data may be due to a single burst source or may be due to multiple flows creating a large amount of aggregated traffic.
- the maximum target packet latency for a queue, T MAX0 to T MAXN may be exceeded, and congestion will be experienced that increases the number of packets in one or more queues.
- the strict priority scheduling algorithm can take over when it senses such a data overload condition (considered to be one form of a “data congestion threshold”). During these times of heavy congestion, the selection function switches from a latency sensitive scheduling method to a more congestion-responsive scheduling method.
- the more congestion-responsive method can schedule the packet without reference to other queues.
- This is known as “strict” scheduling, whereby any packets in violation of said restrictions are deemed of higher importance than any packets not in violation of said restrictions.
- the modified algorithm schedules the highest priority queue from the set of all queues in violation. This is known as “priority” scheduling.
- “strict scheduling” and “priority scheduling” characteristics yield a scheduling method called “strict priority” scheduling.
- having a latency scheduler switch to a more aggressive scheduling methodology provides a method for handling data congestion overload conditions that a latency scheduler might not be able to accommodate. Accordingly, a latency scheduling method in combination with either a strict scheduling method and/or a priority scheduling method will attempt to unconditionally satisfy higher service levels that exceed their T MAX target first.
- T MAX0 and T MIN0 are assigned to each packet as a packet latency characteristic instead of being assigned to a relative priority level to which many packets can point.
- a latency J FACTOR is calculated for each packet using one of the aforementioned equations for J FACTOR .
- a scaled latency is calculated for each packet by multiplying each packet's latency by the packet's J FACTOR to yield a scaled packet latency. For each packet in a queue, the greatest scaled packet latency is identified using any appropriate sorting or selection algorithm. After the greatest scaled packet latency for each queue is identified, the greatest scaled latency packet of all queues is next routed to its destination.
- the greatest scaled packet latency of all packets in all queues is the packet next sent to its destination.
- each queue will have a set of latency characteristics on a per packet basis. This enables greater flexibility in defining a priority's latency characteristics at the cost of increased complexity.
- the strict priority scheduling described in phase 60 can be eliminated, yielding a scheduling algorithm that is solely based on the latency-sensitive scheduling algorithm (even during times of heavy congestion).
- CMTS complementary metal-oxide-semiconductor
- data packet should be construed to include Ethernet and Ethernet-like data packets (also known as “frames”), token ring packets, ATM (asynchronous transfer mode) packets, SONET (synchronous optical network) packets or any other type of packet to which different service levels can be provided.
- Ethernet and Ethernet-like data packets also known as “frames”
- ATM asynchronous transfer mode
- SONET synchronous optical network
- the priority queues are implemented using relatively high-speed (short access time) addressable semiconductor memory, e.g., SRAM, DRAM, well known to those of ordinary skill in the computer art. Implementations of the data system that don't require or demand high-speed data handling could use slower memory such as a disk storage device.
- relatively high-speed (short access time) addressable semiconductor memory e.g., SRAM, DRAM, well known to those of ordinary skill in the computer art. Implementations of the data system that don't require or demand high-speed data handling could use slower memory such as a disk storage device.
- the time stamp circuit 22 can be implemented using field programmable gate arrays (FPGA), application specific integrated circuits (ASIC), combinational and sequential logic devices or suitably capable microprocessors. Different implementations of the time stamp circuit 22 , function as a processor and each implementation is therefore considered to be a processor.
- the function of the time stamp circuit 22 is to determine a current time and append a current time stamp to data packets.
- the port steering element 20 can also be implemented using field programmable gate arrays (FPGA), application specific integrated circuits (ASIC), combinational and sequential logic devices or suitably capable microprocessors. Different implementations of the port steering element 20 also function as a processor and each implementation is therefore considered to be a processor.
- the function of the port steering element 20 is to direct or route data packets to one or more appropriate data queues in the queue structure 25 .
- scheduling engines 32 can also be implemented using field programmable gate arrays (FPGA), application specific integrated circuits (ASIC), combinational and sequential logic devices or suitably capable microprocessors. Different implementations of the scheduling engines 32 function as a processor and each implementation is therefore considered to be a processor. The function of the scheduling engines is to schedule the delivery of data packets from a particular queue to its output port using the foregoing methodology.
- FPGA field programmable gate arrays
- ASIC application specific integrated circuits
- combinational and sequential logic devices or suitably capable microprocessors.
- the function of the scheduling engines is to schedule the delivery of data packets from a particular queue to its output port using the foregoing methodology.
- Packet latency values are derived from the current value of a global clock and from timestamp information.
- the L MAX0 to L MAXN parameters are the maximum latency value in each queue.
- the J FACTOR parameters are calculated using T MAX0 to T MAXN values, T MIN0 to T MINN values and scaling constants that are all assigned by a system operator. All of the data values are preferably stored in relatively high-speed memory, or in registers, both of which are accessible to the scheduling engines.
- the aforementioned dedicated logic circuit(s), FPGA, ASIC or processor performs a least one calculation including but not limited to: packet sorting, packet latency, packet eligibility, packet overload and packet scaling.
- CMTS complementary metal-oxide-semiconductor
- distribution network such as a coaxial cable or fiber optic network
- Queuing theory is often employed in other services, e.g., call service centers, which attempt to timely provide some form of limited-availability service to customers.
- call service centers e.g., call service centers
- the method finds application to any service provider that wishes to provide differentiated levels of service. Accordingly, for purposes of claim construction the applicability of the disclosed and claimed methods are not limited to or by the term “data system” but to also include any sort of service provider that attempts to provide differentiated levels of service.
- the shared resource bandwidth capacity on an Ethernet or coaxial data link In other non-data systems, the shared resource to be allocated would be the service that the provider nominally provides. In a fiber optic data system, the shared resource would be the availability of the cable to carry information.
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Small-Scale Networks (AREA)
Abstract
Description
- This invention relates to cable data systems. In particular, this invention relates to a cable modem termination system (CMTS) and a method for intelligently controlling data flow queues in a CMTS
- A cable television or “CATV” system is a television broadcasting system that uses coaxial cable or fiber optic cable to distribute a broadband signal containing many separate television program channels that are frequency-multiplexed onto the cable. By allocating the frequency spectrum of one or more television program channels for data transmission, CATV systems are also used to carry data to and from CATV subscribers.
- Data on a cable television system is sent as packets that resemble well-known Ethernet packets. Each data packet can have up to several hundred data bytes preceded by several critical data bytes that comprise the frame's header. Among other things, the header includes a preamble, a frame type, a source address and a destination address. A cyclic redundancy check or CRC is appended to the end of the packet for error detection.
- Subscribers' computers are coupled to the cable via a cable modem. Each cable modem is assigned an address by the head end when the modem is turned on. The modem's address effectively identifies the computer to which it is coupled and enables each subscriber to have individualized connections to the Internet.
- Downstream data packets are those sent from the head end of the cable system to the subscribers. Upstream packets are those sent from the subscribers to the head end. Once a data packet is physically transmitted onto the cable, all of the computers that are coupled to the cable can “see” the packets. By using Ethernet-like data frames however, the cable modems of the system read only the downstream data packets that are addressed to it and ignore all other packets. When packets are sent upstream, the cable modem provides to the data packets, the address assigned to the cable modem. That address of the sending cable modem effectively identifies the subscriber from which the packets originated and in so doing, also provide an address to which responsive packets can be sent.
- At the top or “head end” of a cable television system is a cable modem termination system or “CMTS,” a function of which is to provide complementary functionality to the subscriber's cable modems so as to enable data connectivity to a wide-area network like the Internet. The CMTS performs a variety of critical functions in a cable data system, including among other things, cable modem address assignments and data packet address translation by which packets received by the CMTS from an Internet connection are addressed (directed) to the proper cable modem. A CMTS also functions as a switch by taking upstream packets from the cable modems and routing them toward their proper destinations.
- A CMTS includes one or more ports through which data is received and sent. Different ports are typically coupled to different pathways such that different ports lead to different logical destination addresses. By way of example, data from a subscriber received at one port of the CMTS might need to be routed out a different port in order for the packets to be routed to their destination (as determined by the destination address in the packet's header). In so doing, a CMTS receives upstream data packets, either from subscribers or Internet servers, and routes said packets to the port on which the intended recipient, or server for that recipient, resides as dictated by the packet's destination address. (A description of CMTS functionality can be found on the web site of CableLabs®, which is a non-profit research and development consortium of cable television operators within North and South America. The CableLabs® web site URL is: www.cablelabs.com.)
- Most cable data systems now have more data server subscribers than there are channels that are available to be allocated to carrying data and as a result, cable data system subscribers share the data capacity of the frequency spectrum allocated to carry data. As the number of subscribers on a cable data system increases, bandwidth available to each subscriber (i.e., the service level) decreases. Stated alternatively, on a cable data system, responsiveness decreases as the number of users increases.
- Those of ordinary skill in the art of computer networks in general, and cable data systems in particular, are familiar with the concept of “quality of service.” Quality of service is concept of guaranteeing a throughput or a data transfer level. It is anticipated that next-generation cable data systems will offer customers guaranteed bandwidths (data rates) to customers willing to pay for it. Customers who pay for data rate guarantees will have their data packets routed through the cable data system ahead of lower-paying customers. The concept of being able to guarantee (and to be able to truly provide) different service or priority levels allows cable service providers the opportunity to provide better Internet service, as measured by connection bandwidth, to needy customers while charging a premium rate. Tiered service is expected to increase revenue for cable service providers.
- One way of providing service levels is to organize Internet traffic into data flows. Data packets sent from the head end of the cable data system to a particular cable data system subscriber can be considered to be as one “data flow.” Similarly, the upstream packets from the same customer, or some other customer can be considered to be another “data flow.” By knowing the source and destination addresses of data packets on a cable data system, the data flows that they represent can be allotted different service (i.e., priority) levels by reading from a class of service list that a particular subscriber is entitled to.
- Part of the job of a CMTS is to route incoming packets to their destinations, as determined by their packet destination addresses. In a CMTS, this usually requires that packets received at one port need to be switched out of the CMTS on a different port that is connected to a different physical or logical pathway leading to different logical destinations. In order to provide different service levels, data packets are stored in priority queues, which are known to those of skill in the networking art to be multi-element lists, each element of which is assigned a priority value or level that determines when the element is to be removed from the queue
- The primary task behind any Ethernet frame-based scheduling algorithm is to examine the contents of the multiple queues that are aimed at a particular shared resource, e.g., a channel time-slot on the cable or a port of the CMTS, and then decide which queue should transmit the next Ethernet frame to that shared resource. A relative scheduling algorithm must make decisions regarding the relative treatment each of the queues based on the priority level associated with that particular queue. In general, any scheduling algorithm must make trade-offs between fairness, delay, and implementation complexity.
- Many theoretical models for scheduling have been presented in the literature. The Generalized Processor Sharing (“GPS”) algorithm (A. K. Parekh and R. G. Gallager, “A generalized processor sharing approach to flow control—the single node case,” in IEEE INFOCOM '92, Vol. 2, pp. 915-924, May 1992.) is generally considered to be a fair scheduling algorithm, because it attempts to follow a fluid flow model that transmits (and mixes) output traffic from various active queues at rates proportional to their assigned weights. Unfortunately, the ideal GPS algorithm requires that the arriving packets be broken down into infinitely divisible parts, which is not practical in a data system using variable size packets like Ethernet frames and tends to complicate the algorithm's implementation.
- One approach that attempts to approximate the GPS algorithm is the Weighted Fair Queuing (“WFQ”) algorithm disclosed by A. Demers, S. Keshav, and S. Shenker, in “Analysis and simulation of a fair queuing algorithm,” published in Internetworking: Research and Experience, Vol. 1, No. 1, pp. 3-26, 1990). A limitation of this implementation is maintaining desirable latency performance of low-latency traffic, such as Voice-over-Internet Protocol (VoIP) service, during periods of traffic bursts.
- Another approach that attempts to provide fair service includes the CORR algorithm disclosed by D. Saha, S. Mukherjee, and S. Tripath, as “Carry-Over Round Robin: A simple cell scheduling mechanism for ATM networks” and Deficit Round Robin Queuing (M. Shreedhar and G. Varghese, “Efficient fair queuing using deficit round robin”). However, these algorithms distort fairness when a particular queue is filled with very long packets while other queues are filled with very short packets. In such a situation, the queue with the long frames will steal more than its fair share of bandwidth on the shared resource.
- A method and apparatus that monitors the latency of every packet passing through the system and uses that information to dynamically adjust (in real time) the performance of the scheduling algorithm would provide improved queuing of data flows in a cable modem termination system and quality of service level differentiation. Such a method would be an improvement over the prior art.
- FIG. 1 shows a cable data system including a representation of a cable modem termination system employing the queuing method and apparatus disclosed herein.
- FIG. 2 shows a representation of two queues used to store data packets for each port in a CMTS that recognizes two different priority levels.
- FIG. 3 shows an exemplary data packet.
- FIG. 4 shows an exemplary data structure.
- FIGS. 5A and 5B show the steps of a method to normalize data flows through several different queues.
- FIG. 1 shows a block diagram of a
cable data system 10 that provides data flow queuing according to the preferred embodiment. Theseveral computers 12 of the subscribers to thecable data system 10 are operatively coupled to thecoaxial cable 16 of a distribution network such as a CATV network (fiber optic cable in some embodiments) over which data packets are transmitted to and received from adata system 18. For illustration purposes, thecomputers 12 in FIG. 1 are sub-divided into two groups:Group 44 andGroup 46. In a preferred embodiment, thedata system 18 is a cable modem termination system or “CMTS.” Thecoaxial cable 16 and theCMTS 18 form at least part of a data distribution network. Together, thecable 16 and the CMTS enable thecomputers 12 to communicate with each other as well as with other computers of the Internet. - The
CMTS 18 shown in FIG. 1 has four ports (denominated 0-3) into which packets are sent and received.Ports computers 12. Computers ofGroup 46 are coupled toport 0; computers ofGroup 44 are coupled toport 1.Ports - One function of the
CMTS 18 is to route data packets to their destinations. Data packets can be sent into one port and routed out of another port. In so doing, theCMTS 18 acts as a switch, routing data packets to their logical destinations.Port 0 is coupled to the computers inGroup 46.Port 1 is coupled to the computers inGroup 44.Ports group 46 computers (received at port 0) can be switched tooutput ports CMTS 18 therefore not only enablesgroup A computers 46 to communicate with the set of allcomputers 12, but also to communicate with any other computer linked to the Internet backbone accessible via theinternet backbone 14. - In order to provide truly differentiated service levels to different data flows, the
CMTS 18 needs to intelligently treat packets in higher-priority data flows with a greater level of precedence than packets in lower-priority data flows. This can result in higher-priority data flows getting preferential treatment in many different ways, ranging from offering them more bandwidth on a shared resource to offering them more rapid transport (lower delay and lower latency) when passing through the CMTS. - Some portions of the CMTS system design (such as the congestion control algorithms that run on each of the input ports) are entirely dedicated to the control of bandwidth associated with each of the data flows. In most commercially-available CMTS systems, the scheduling algorithms that determine the order by which packets are transmitted out of the system are also dedicated to the control of bandwidth associated with each of the data flows.
- However, a novel scheduling algorithm that determines the order by which packets will be transmitted out of the CMTS can be used to control both the bandwidth and the delay associated with each data flow. In order to do so, high-priority packets can be written to a high priority queue, while low priority packets can be written to a low priority queue. The service level agreement parameters can be used to define the priority of a packet. The association between a service level agreement and a particular data flow is established when a subscriber's data flow is registered within the
CMTS 18. This registering of the subscriber's service level agreement usually occurs when the subscriber's cable modem is first powered up and makes contact with theCMTS 18. When a packet passes throughCMTS 18, a classification of the packet into a particular data flow must be performed. Since each data flow is associated with a subscriber who registered for a particular service level, each data flow is then associated with a unique priority. A table look-up (usually into a memory) using the unique priority as a key produces the service level agreement parameters that are associated with the priority. By association, these service level agreement parameters are also associated with the data flow and packet. Examples of service level agreement parameters that can be attached to a priority level include the minimum guaranteed throughput (packets per second), the maximum permitted throughput (packets per second), the probability that general service will be available to the customer, and the probability that a packet from a data flow will be dropped during periods of congestion. - In advanced CMTSs, one may also want to specify desired latency characteristics within the priority-based service level agreement, where latency is defined as the interval of time required for the packet to pass between two specified points within its overall path. In many cases, the two specified points are defined to be an input port on the CMTS and an output port on the CMTS, although other points could be specified. The actual latency value experienced by any packet passing through the CMTS can be easily calculated by having a counter or free-running clock that specifies time relative to an arbitrary reference time. If the counter or clock is continually incremented as time passes, then a sample of the value in the counter or clock when a packet passes the first point can be designated as the packet's arrival time. A sample of the value in the counter or clock when the packet passes the second point can be designated as the packet's departure time. Subtracting the arrival time from the departure time yields the actual latency value for the packet. In some implementations, there may be a reason to calculate the current latency value of a packet stored in a queue of the CMTS. The current latency value is the actual latency value that the packet would experience if it were immediately selected by the scheduling algorithm to be sent to its destination (whether it is actually selected or not). The current latency value provides an approximate measure describing how long the packet has been stored in the queue of the CMTS (since delays in other parts of the CMTS system are usually negligible). A data packet that has been stored in the queue longer than any other data packet in the queue is said to be the oldest data packet in the queue.
- Examples of desired latency characteristics include the minimum desired latency and the maximum desired latency. Specification of a minimum desired latency and a maximum desired latency yields a desired range for the actual latency for a packet. In general, the scheduling algorithm in a CMTS should attempt to transmit packets to their destination so that the actual latency experienced by the packet is greater than or equal to the minimum desired latency and less than or equal to the maximum desired latency. In truly differentiated service, guaranteeing a minimum desired latency implies that the packets of some flows should be made to wait a minimum time before being routed. This may be desirable if it gives customers the incentive to pay for higher-priority service levels. Packets that have waited a minimum time so that their current latency value is, greater than or equal to the minimum desired latency are said to be eligible for scheduling, and they are called eligible packets. Packets that have not yet waited a minimum time (so that their current latency value is less than the minimum desired latency) are said to ineligible for scheduling.
- In order to provide differentiated service levels, data packets of data flows entitled to a high priority should typically be routed through the
CMTS 18 ahead of data packets of data flows that are entitled to low priority. Priority levels are usually assigned by a service provider according to the service level that a particular customer wants (or pays for), according to the type of service level that the carried-information requires. Packets of information that carry voice (i.e., telephone) over the Internet (VoIP) need to be routed quickly (at a high-priority level) so as to preserve the integrity of a conversation. Data packets representing an e-mail message can usually be sent at a lower-priority level. The type of information being routed through the data system or CMTS will frequently determine the priority level accorded to a data flow. Different data flows to or from the same address can also require different priority levels. An Internet service subscriber might want a very high service level for voice over IP, a slightly lower service level for browsing the web and an even lower service level for e-mail. The voice traffic, web browser and e-mail can each be considered a separate data flow, each of which can have a different service level. - Data packets of flows are stored or queued a finite amount of time in the course of being switched or routed. Inasmuch as all flows are queued to greater or lesser extent based upon their service level entitlements, the
CMTS 18 uses a scheduling priority and packet latency in order to determine which flow should have its queued data packets routed to the packet's next destination. A packet's destination is determined by its destination address. A packet is effectively “delivered” to a destination by routing (switching) it to aCMTS 18 port that will carry the packet toward the packet's destination address. A “scheduling priority” is a scheme whereby higher priority packets are routed ahead of lower-priority packets as determined by the class of service that a customer is entitled to. - As is well-known, a packet or a flow “priority” has no meaning per se. A “priority” accorded to one packet or stream, is meaningful only with respect to a different priority accorded to a different packet or stream. The priority of a packet, data stream or a data queue is only meaningful when it is expressed relative to the priority of other packet packets, data streams or queues with different priorities. The term “relative scheduling priority” therefore only means that the priority accorded to a packet of a particular stream is determined relative to the priority accorded to packets of other streams. A “relative scheduling priority” therefore only means that one scheduling priority can be higher, lower or the same as another scheduling priority.
- With the foregoing concept of “relative scheduling priority” in mind, each packet can be assigned a relative scheduling priority such that some packets should normally be routed to their destinations ahead of others. The specification of when a packet of a particular relative scheduling priority should be routed to its destination can be established by specifying or assigning one or more of the aforementioned desired latency characteristics to each relative scheduling priority to be accorded the packets of different streams. In other words, each packet can be assigned a minimum wait time, a maximum wait time, both a minimum and a maximum wait time, depending on the relative priority (i.e., the relative scheduling priority) to be accorded a packet.
- The priority level of a packet can be determined by looking up the packet's source or destination address and linking the packet's source address or the destination address or both, to the service level(s) that the source or destination is (are) entitled to. The priority of a packet can also be determined by looking at other fields in the packet header, and using the values in those fields to look up the associated priority level of the packet. This priority level can then be appended as a data value to each frame of a flow. In the preferred embodiment, the priority level is determined by reading the source field for upstream packets (or destination field, in the case of downstream packets) and from the identity of the source (destination), looking up the service level that the source (destination) is entitled to.
CMTS 18 receives all such data flows and stores the packets to priority specific queues. - The process of queuing data packets entails writing data packets into a data storage device, such as a semi-conductor random access memory, and reading the packets from the memory at a later time, albeit in the order in which the packets were stored. Priority queuing is different: packets are read from the queue in an order that follows the priority assigned to the queued elements. In the
CMTS 18, priority queuing of multiple flows is accomplished by storing high-priority packets in a high-priority queue; low priority packets are stored in a low-priority queue. Packets are read from the queues (low priority and high priority) after taking into account both the latency and priority of packets currently stored in each queue. - Data packets from a subscriber such as computer12-1 are received at the
CMTS 18 through itsport 1 and are from there, coupled to theport steering element 20 so as to route the data packets to their respective destinations. The destinations are determined by the destination address in each packet. A packet's destination address can be an Internet protocol (“IP”) address, among others. - The
port steering element 20 in the preferred embodiment steers Ethernet packets according to one of two established protocols. Both protocols use unique destination identifier fields residing within the header of each Ethernet Packet. Those of skill in the art of data networks know both protocols. The first protocol is used in a “switch” based network and relies on the Destination Media Access Control (DMAC) address for destination determination. The second protocol is used in a “route” based network and relies on the Destination Internet Protocol (DIP) address for destination determination. In both instances, the steering element indexes a lookup table, usually composed of data stored in a random access memory element, with the DMAC or DIP address, or some derivatives thereof. The result of this lookup operation describes where the packet is to be sent. - The
port steering element 20 switches or routes incoming data packet's received at each port 0-3 to at least one set ofqueues CMTS 18 has four ports and supports two service levels per port, each port 0-3 would have two separate queues. The entire CMTS would therefore support a total of eight queues or queue equivalents. - The queues for each port store data packets corresponding to a service level or set of service levels that are to be routed to the corresponding port. For purposes of illustration, the
CMTS 18 in FIG. 1 has four (4) ports, each of which have a queue into which high priority service level packets are written and a queue into which low priority service level packets are written. Accordingly,CMTS 18 provides two different service levels. - The
port steering element 20 examines a destination address in the packet header, determines the service level accorded to the data packet, then routes each data packet through a time-stamp circuit 22 and on to the appropriate queue for each output port. The time-stamp circuit 22 appends a scalar to each packet (also known as a frame) that identifies when the packet arrived at theCMTS 18. - Those skilled in the art will recognize that a finite amount of time will elapse between the time that a packet arrives at the
CMTS 18 and when a packet header is edited due to latency associated with theport steering element 20. It should be apparent that the time-stamp circuit 22 could be placed in several other meaningful locations within theCMTS 18, including at the actual input port or above the port steering element. If transit delays through the circuitry of theCMTS 18 are minimal, the differences in time-stamps created by the time-stamp circuit 22 when placed at these different locations should be negligible. Even if these transit delays are significant, they will oftentimes be essentially fixed delay values, so theCMTS 18 can account for the delay associated with said steering element (and other circuitry) by reducing the service level latency characteristics by the value of said steering element latency (and other circuitry latency). In the preferred embodiment, thetime stamp circuit 22 is a timer that assigns a multi-byte integer representing the current time. - After the time of arrival value is appended to each data packet, the data packet is stored to its corresponding priority queue of its destination port. The scheduling algorithm presented herein determines the packet's current latency by subtracting the scalar representing the time of arrival from the current time (which is obtained from the time-stamp circuit's timer or from another timer which is roughly synchronized to the time-stamp circuit's timer). The resulting packet latency (also a scalar) is then reduced by the minimum latency requirement of its associated service level, TMN, and then normalized. This result is used to determine the next packet to send from the CMTS. In other words, packet delivery scheduling uses a latency characteristic assigned to a packet's scheduling priority.
- As shown in FIG. 1, the queues for the ports are part of a
queue structure 25. - In general, “queues” are collections of information, the contents of which are organized according to rules. One queue organization rule uses the order of information entry. Two queues organized by such a rule are commonly known as First-In-First-Out (FIFO) or Last-In-First-Out (LIFO) queues. Another queue organization rule is based upon priority ordering whereby the extraction of information, data, or other queue elements therefrom is determined according to an importance, value, significance or, in the case of data networks, service level assigned to or associated with each element in the “priority queue”. Still other rules use priority ordering in conjunction with order of entry to determine the order of extraction. The factors, configurations, and techniques leading to queue ordering forms the science known as queuing theory.
- In the preferred embodiment, the
queue structure 25 is implemented using an array of semiconductor memory organized into N queues or data structures, where N is equal to the number of queues for the entire system. Each data structure of the N structures is associated with, or stores data packets for, a certain service level (i.e. priority level) and will contain a variable number of elements, the length of each of which is also variable. (The elements stored in the data structures/queues are Ethernet packets. The length of each element in the queue is variable because Ethernet packets are of variable length.) In the preferred embodiment, the data structures are organized as FIFO queues. - For example,
CMTS 18, which provides two service levels and which has four ports, has thequeue structure 25 organized into eight queues (or data structures). For each of the four ports 0-3, the data packets to be routed from each port are stored into a queue for each priority level. As a result, each port 0-3 has a first queue for the high priority level packets and a second queue for the low priority level packets, resulting in a total of eight queues. A delivery sequence of a high-priority upstream packet from the subscriber computer 12-1 (residing on port 1) to the subscriber computer 12-2 (residing on port 0) is as follows: a packet from computer 12-1 is received onport 1 ofCMTS 18; thesteering element 20 determines service level associated with said packet and forwards the packet to the high priority queue associated withport 0; the data packet then awaits scheduling forport 0 delivery according to a data packet extraction scheduling rule described below; upon receiving a packet delivery permission grant from theport 0 scheduling engine, the packet is transmitted throughport 0 and decoded by the destination subscriber computer 12-2. Similarly, an upstream packet from subscriber 12-1 that is a low-priority packet and which is addressed to another computer somewhere onport 1 will be routed to the low priority queue forport 1. - The arrangement of the queues for a port is conceptually depicted in FIG. 2. At
input 34, the data packets to be routed to a particular output port can be stored into eitherqueue first queue 36 stores low priority data packets; thesecond queue 38 stores high priority packets. Bothqueues - Other embodiments of the invention would include providing more than two priority service levels or a mixture of priority service levels (and therefore includes a number of queues other than two queues at each port) as well as data systems (i.e., the CMTS) with more than or less than, four ports. Such alternate embodiments that employ the scheduling method disclosed herein would have for each port, a data packet queue for each service level that is not necessarily limited to a set value. For example, one port could provide two (2) different service level priorities while a second port could provide four (4) service level priorities while still other ports could be provide even a greater number of service level priorities.
- FIG. 3 depicts a single Ethernet data packet (also known as a “frame”)40. (There are presently at least four different types of Ethernet frames, each defined by the structure of the frame, i.e., the number of bytes allocated to each field in the header, data filed and trailer fields.) Ethernet frames in general can have between 64 and 1518 bytes. The first several bytes (usually the first eighteen) comprise the header of an Ethernet packet. The next forty-six to fifteen hundred bytes are data.
- For illustrative purposes, the packet header shown in FIG. 3 includes the
preamble 42,destination address 44,source address 46,packet type 48 andtime stamp 50. The data field 51 (which is the information to be carried to the packet's destination) of thepacket 40 follows the header. A trailer field that is a cyclic redundancy check, or “CRC” 53, is used to detect errors in thepacket 40 is the last field. In the preferred embodiment, thetime stamp 50 is “appended” to thedata packet 40 by being linked to the packet; the time stamp is not part of the header. The data packet's time of arrival is considered to be “assigned” to the data packet (or “associated” with a data packet) whether the data representing the time of arrival is actually within (or embedded in) the header of the packet or the time of arrival is linked to the header and stored separately. Each packet is accorded a scheduling priority. As described above, a packet's scheduling priority can be determined from a packet's source address, destination address, or other fields in the packet's header. Each packet is routed to its destination address by theCMTS 18 according to the (relative) priority that it is entitled to. - FIG. 4 depicts a structure of a
priority queue 66 ofdata packets 62.Data packets 62 can be written into the queues at different times or at the same time. Each of the multi-byte packets stored in thequeue 66 has the same priority or service level, even though the packets might be from different flows and represent different types of information. By way of example, data packets of an e-mail message of one flow can be stored in the same queue (i.e., queued) with data packets of a different flow, which could be packets for a Voice-over-IP stream or packets for a stream to or from a web browser. - In FIG. 1, four scheduling engine processors/circuits, denominated32-0, 32-1, 32-2 and 32-3, determine the order in which packets are read from the different queues for the different ports. Scheduling engine 32-0 determines the order for packets read from the set of queues 24 (which queues contain packets destined for output port 0), scheduling engine 32-1 determines the order for packets read from the set of queues 26 (which queues contain packets destined for output port 1), scheduling engine 32-2 determines the order for packets read from the set of queues 28 (which queues contain packets destined for output port 3), and scheduling engine 32-3 determines the order for packets read from the set of queues 30 (which queues contain packets destined for output port 2).
- The
scheduling engines 32 schedule the delivery of data packets. They read data packets out of each of the queues of thequeue structure 25 according to a scheduling methodology by which the waiting time (hereafter referred to as “latency”) of each data packet in each data queue is modified (shifted as described below) and then normalized or scaled. The resulting scaled latency of each data packet in a set of queues is compared to the other scaled latencies of other data packets within the same set of queues to identify the packet in that particular set of queues with the greatest scaled latency. The greatest scaled-latency packet of all packets within a set of queues is the next packet that will be transmitted to the output port associated with that set of queues. The result of this selection allows the packet with the greatest scaled latency to make its way through its associated output port toward its desired destination before all of the remaining packets that are still stored within the particular set of queues. - The process of normalizing or scaling the actual latency of data packets, which is described below, includes in the scaled latency calculation, both the length of time that a packet has been in queue and also the priority level that a packet is entitled to. A latency scaling factor is assigned to each queue. The assignment of a latency scaling factor to a queue defines the latency performance of that queue. Accordingly, small latency scaling factors typically lead to small scaled latencies while large latency scaling factors typically lead to large scaled latencies (although the actual scaled latency is also a function of the actual latency associated with the data packet). In the scheduling process, large scaled latencies are scheduled prior to small scaled latencies. Consequently, large latency scaling factors lead to short queue delays and a higher-priority service, whereas small scaling factors lead to long queue delays and a lower-priority service. As the method selects among scaled latencies, it is possible that low priority packets with a smaller scaling factor can be routed to their destination ahead of higher-priority data packets with a greater scaled latency factor if the actual latency associated with the low priority packet has become large relative to the actual latency associated with the high priority packet (implying that the low priority packet has been held in its queue for much longer period of time than the high priority packet).
- Even though the data packets are preferably written into queues that are structured and operated as FIFO buffers, it should be understood that to deliver true quality of service, the
CMTS 18 needs to be able to decide which of the many packets stored within a set of queues associated with a particular output port should be next routed to its destination. Typically, high priority data flows will be granted permission to transfer data packets ahead of low priority data flows. However, low priority data flows should be granted permission to transfer data packets ahead of even higher priority packets if a low priority packet has been in a queue longer than allowed by the customer's class of service or by the customer's service level agreement with the service provider. True quality of service can be realized if the waiting time of the data packets in each queue is normalized by factoring into the normalization, the priority level that each packet is entitled to receive, with the oldest packet in each data queue: - FIGS. 5A and 5B depict a flow-chart representation of the method for scheduling the delivery of data packets from a particular set of queues to a particular output port, using at least one latency characteristic and a packet's relative scheduling priority. A unique and separate instantiation of this scheduling algorithm could be implemented in each
Scheduling Engine 32 for each of the output ports in theCMTS 18, or a single instantiation could be implemented in a common Scheduling Engine (not shown) and could be re-used with a unique time-slice and a unique set of state information being dedicated to each of the output ports in theCMTS 18. In a preferred embodiment, a unique and separate instantiation of this scheduling algorithm is implemented for each of the output ports 0-3. - Scheduling Method Overview
- The scheduling method has different sets of related steps (denominated herein as “phases”), each of which includes one or more computational steps. In the
first phase 52, shown in FIG. 5A, data packets received from theport steering element 20 are written into the appropriate data queues for the particular port.Port steering element 20 uses destination information (such as the DMAC address or the DIP address) and the packet's priority level information to direct the packets to their appropriate queues. - In the
second phase 54, the current latency for each packet is calculated in step 54-1, and then the oldest packet in each queue is identified. In instances where a FIFO buffer is used for a priority queue, the oldest packet will reside at the “head” of the queue and the youngest packet will reside at the “tail” of the queue. Hence, FIFO implementations of a priority queue do not requirephase 54 because all head packets are the oldest packets (with the greatest latency) within the buffer. - In the
third phase 56, the oldest packet in each queue is tested against a minimum desired latency threshold specified for the relative priority level associated with all of the packets in that queue. This test determines whether the oldest packet in a queue is eligible for scheduling. An eligible packet is one that has a current latency greater than or equal to a minimum desired latency threshold value. - In the
fourth phase 58, all eligible packet latencies are shifted by their minimum desired latency value defined for the relative priority associated with all of the packets in their respective queue. The shifting operation is accomplished by subtracting the current latency of each eligible packet by the minimum desired latency threshold. The result of each shifting operation is scaled according to the appropriate scaling factor (hereafter referred to as the “JFACTOR”), which is associated with the relative priority for the packets in the queue. A test is then performed in phase 59 (shown in FIG. 5B) to determine if the scaled latency for the eligible packet is greater than a desired target scaled latency. Another embodiment performs this test by determining if the actual latency for the eligible packet is greater than the desired latency for this queue (the results are equivalent to the scaled version of the test). The results of this test are used in the fifth andfinal phase 60. - Assuming the test in
phase 59 indicated that the scaled latency for at least one of the eligible packets was greater than its associated target scaled latency, then the eligible packet associated with one of the queues whose scaled latency exceeded the target latency will be selected and sent to the output port. A latency violating queue is considered to be any queue which contains at least one eligible packet whose scaled latency is greater than the target scaled latency for that queue. The particular one packet that will be selected next for transmission by theScheduling Engine 32 will be associated with the latency violating queue whose relative priority is higher than all of the other latency violating queues (which is effectively a form of strict priority queuing between all of the latency violating queues). Assuming the test inphase 59 indicated that the scaled latency for all of the eligible packets was less than or equal to their associated target scaled latencies, then the eligible packet associated with the largest scaled latency will be selected and shipped to the output port. - Scheduling Method in Detail
- Again with respect to FIG. 5, in the first set of steps that comprise
phase 52, data packets are written into buffers, previously called data structures, according to the destination port and priority level assigned to each data packet. In FIG. 5, there are depicted N separate queues, 0 and N corresponding to N different service levels. -
Phase 54 determines the maximum latency of all the packets in a queue of length “x” elements. At step 54-1, the time that a first data packet “i” spent in a queue is determined. A packet's current latency can be readily determined by calculating the current time (as provided by either the global clock or another real-time clock) minus the time stamp value attached to or associated with the packet as it passed through thetime stamp 22. In the first execution of the loop ofphase 54, “i” equals 1 and the comparison step 54-2 will store the packet latency associated with the first evaluated packet to LMAX0 in step 54-4. At step 54-5, the value of the loop counter “i” is tested to see if the end of the queue has been reached. If there are more packets to evaluate, the loop counter is incremented at step 54-3 so as to point to the next packet. - As the loop counter “i” is incremented with each pass through the loop of
phase 54, the latency of every packet is iteratively compared to the maximum latency of all previously tested packets. In the event that a new maximum latency is determined, said value is stored in a register or memory location called “LMAX0” in step 54-4. When the latency value associated with the last packet in the queue has been evaluated, as determined by steps 54-5 and 54-3, the value stored at LMAX0 will be the maximum latency value for packets within the queue. The maximum latency value LMAX0 is then used inphase 56. - In
phase 56, thescheduling engines 32 of each queue determine if the oldest packet in the queue, which is LMAX0, is eligible for transfer to its appropriate output port. As set forth above, packet eligibility is determined by whether the packet's latency is greater than or equal to a desired minimum latency, which in some embodiments can also be zero. In true quality-of-service, priority flows can be made to wait a minimum amount of time, even if there is no other data traffic waiting to be routed. This can be accomplished by holding the packets for a service level in a queue until the resulting delay exceeds the minimum desired latency (TMIN0) associated with their priority. TMIN0 at step 56-2 is a minimum amount of time that a packet needs to wait in a particular queue for a particular service level. TMIN0 at step 56-2 can be zero or greater than zero. - A test is made at step56-2 whether the oldest packet in a queue has been waiting at least as long as TMIN0. If the current latency (i.e., the wait time) of LMAX0 is greater that TMIN0, then the difference between these two values will be positive, and a determination is made at step 56-2 and 56-6 that the LMAX0 data packet is eligible for transfer to its destination. This difference (LMAX0−TMIN0) is called the shifting operation, and the result is called the shifted latency value. The shifted latency value is forwarded to the
scaling phase 58 for each of the eligible packets. The shifting operation (LMAX0−TMIN0) is an example of a “second function of said packet's latency”, where the actual (unshifted) latency of the oldest packet is specified as LMAX0. In the steps ofphase 58, each of the shifted latency values (i.e. LMAX1−TMINi) in each data queue i is multiplied by a latency-scaling factor identified herein as a “JFACTOR”. This JFACTOR is related to at least one queue latency characteristic associated with a particular queue. - The JFACTOR(i) (which is associated with queue i) is also used to normalize the desired latency characteristics to create a value that is hereafter referred to as the “TARGET_SCALED_LATENCY”, or simply “TSL”). Queues with a larger JFACTOR approach the TSL more quickly than queues with a smaller JFACTOR. Packets that exceed TSL have waited in their queue longer than their time allotted by TMAX, whereas queues that fall below TSL are below their allotted TMAX. The scaled result of different queues can therefore be quantitatively compared to determine the packet that is most deserving to be scheduled based upon the queue's latency characteristics, TMIN(i) and TMAX(i).
- The calculation of JFACTOR′(i) is a function of the inverse of the difference between TMAX(i) and TMIN(i). The difference operation (TMAX(i)−TMIN(i)) is an example of a “third function of the at least one desired latency characteristic”, where this particular third function actually includes two desired latency characteristics: TMAX(i) and TMIN(i).
-
- The evaluation or this function involves the computation of complex floating-point arithmetic. On it's face, JFACTOR′(i) is a fractional quantity. Second, the
time stamp element 22 is not necessarily continuous and can increment once every time period (hereafter referred to as “T_TICK”). These considerations suggest that the calculation of JFACTOR′(i) be simplified using an algorithm, such as the following: - (1) Determine the maximum range of the desired latency characteristics:
- MAX_RANGE=MAX{TMAX(i)−TMIN(i)} for all i
- (2) Determine a resolution factor R, required to provide adequate resolution of the priority level with the largest maximum range. For a preferred embodiment, a resolution factor R value of 10 is arbitrarily chosen. When implemented using integer arithmetic, this results in a scaled latency value with a range of at least 0 to 10.
- R=10
- (3) Determine a scaling factor based on the maximum range and resolution factor:
- SMAX=R×MAX_RANGE
- (4) Scale JFACTOR′(i) by SMAX:
- JFACTOR(i)=J′FACTOR(i)×SMAX
- Without loss of generality, we can assume that SMAX is 1 for the remainder of this description. After the JFACTOR for a queue is determined, that queue has been assigned a latency characteristic. Multiplying the shifted latency value of the oldest eligible packet in each queue by its corresponding JFACTOR yields a scaled latency value for the oldest packet in each queue. The multiplication operation of ((LMAX0(i)−TMIN0(i))*(JFACTOR(i))) can be shown to be the same as the division operation of ((LMAX0(i)−TMIN0(i))/(TMAX(i)−TMIN(i))), and this division operation is an example of a “first function of the packet's latency and the at least one desired latency characteristic assigned to the relative scheduling priority for said packet.” In particular, it can be noted that this first function is comprised of a ratio of the second function of said packet's latency to the third function of the at least one desired latency characteristic.
- The value of this first function is stored in a register or memory location or on disk in step58-2 for subsequent use, described below. As a result of the latency scaling of
phase 58, each of the latencies of the oldest packet in each queue is normalized such that a comparison of the different scaled latencies identifies which of the oldest scaled latencies should be next routed to its destination based not only upon its time in queue but also its priority or service level. - The last phase of the scheduling algorithm (phase60) is the selection function that actually picks an eligible packet to be transmitted to the output port. Assuming that strict priority scheduling is not necessary (due to the comparisons in phase 59),
phase 60 sorts the scaled latencies determined at step 58-2 to find the maximum-scaled latency for the set of queues associated with an output port. In the first execution of the loop 60-4, “i” equals 0 and the comparison step 60-5 will store the scaled latency associated with the first evaluated queue to SLMAX in step 60-6. At step 60-7, the value of the loop counter “i” is tested to see if the last queue has been reached. If there are more queues to evaluate, the loop counter is incremented at step 60-8 so as to point to the next queue. As the loop counter “i” is incremented with each pass through the loop 60-4, the scaled latency of every queue is iteratively compared to the maximum scaled latency of all previously tested queues. In the event that a new maximum-scaled latency is determined, this value is stored at SLMAx in step 60-6. When the scaled latency value associated with the last queue has been evaluated, as determined by steps 60-7 and 60-8, the value stored at SLMAX will be the maximum-scaled latency value for the set of all queues for a particular port. The data packet associated with the maximum scaled latency value is then scheduled for delivery to the output port. If the queues depicted in FIG. 5 are for aport 0, for example, a packet associated with the maximum scaled latency as determined inphase 60 will be output toport 0. - If strict priority scheduling is required in
phase 60 due to the comparisons in phase 59 (Strict(i)=1), then apparently one or more of the queues associated with this output port are latency violating queues. (Recall that a latency violating queue is defined to be a queue whose oldest packet has been waiting for a period of time longer than the maximum desired latency associated with the queue). The particular one packet that will be selected in this case will be the oldest packet associated with the latency violating queue whose relative priority is higher than all of the other latency violating queues. This strict priority scheduling method may be required at times. In particular, there may be transient intervals during which abnormally large amounts of data arrive to be switched through thedata system 10. This large amount of data may be due to a single burst source or may be due to multiple flows creating a large amount of aggregated traffic. In either case, the maximum target packet latency for a queue, TMAX0 to TMAXN, may be exceeded, and congestion will be experienced that increases the number of packets in one or more queues. To compensate for the resulting heavy congestion that may occur when TMAX is exceeded in one or more queues, the strict priority scheduling algorithm can take over when it senses such a data overload condition (considered to be one form of a “data congestion threshold”). During these times of heavy congestion, the selection function switches from a latency sensitive scheduling method to a more congestion-responsive scheduling method. For instance, if the latency for a packet inqueue 0 exceeds its TMAX0, the more congestion-responsive method can schedule the packet without reference to other queues. This is known as “strict” scheduling, whereby any packets in violation of said restrictions are deemed of higher importance than any packets not in violation of said restrictions. In the instance of multiple scheduling violations, the modified algorithm schedules the highest priority queue from the set of all queues in violation. This is known as “priority” scheduling. Combining “strict scheduling” and “priority scheduling” characteristics, yield a scheduling method called “strict priority” scheduling. In essence, having a latency scheduler switch to a more aggressive scheduling methodology provides a method for handling data congestion overload conditions that a latency scheduler might not be able to accommodate. Accordingly, a latency scheduling method in combination with either a strict scheduling method and/or a priority scheduling method will attempt to unconditionally satisfy higher service levels that exceed their TMAX target first. - Slightly different latency-sensitive scheduling methods are implemented in alternate embodiments.
- In one alternate embodiment, TMAX0 and TMIN0 are assigned to each packet as a packet latency characteristic instead of being assigned to a relative priority level to which many packets can point. A latency JFACTOR is calculated for each packet using one of the aforementioned equations for JFACTOR. Using the aforementioned processes, a scaled latency is calculated for each packet by multiplying each packet's latency by the packet's JFACTOR to yield a scaled packet latency. For each packet in a queue, the greatest scaled packet latency is identified using any appropriate sorting or selection algorithm. After the greatest scaled packet latency for each queue is identified, the greatest scaled latency packet of all queues is next routed to its destination. As in the preferred embodiment, the greatest scaled packet latency of all packets in all queues is the packet next sent to its destination. Unlike the preferred embodiment, each queue will have a set of latency characteristics on a per packet basis. This enables greater flexibility in defining a priority's latency characteristics at the cost of increased complexity.
- In a second alternate embodiment, the strict priority scheduling described in
phase 60 can be eliminated, yielding a scheduling algorithm that is solely based on the latency-sensitive scheduling algorithm (even during times of heavy congestion). - For purposes of claim construction, the term “CMTS” and “data system” should also be considered to include any kind of computer, router, or switching system that schedules, provides, or otherwise attempts to provide different levels of service to data flows. The term “data packet” should be construed to include Ethernet and Ethernet-like data packets (also known as “frames”), token ring packets, ATM (asynchronous transfer mode) packets, SONET (synchronous optical network) packets or any other type of packet to which different service levels can be provided.
- In a preferred embodiment, the priority queues are implemented using relatively high-speed (short access time) addressable semiconductor memory, e.g., SRAM, DRAM, well known to those of ordinary skill in the computer art. Implementations of the data system that don't require or demand high-speed data handling could use slower memory such as a disk storage device.
- The
time stamp circuit 22 can be implemented using field programmable gate arrays (FPGA), application specific integrated circuits (ASIC), combinational and sequential logic devices or suitably capable microprocessors. Different implementations of thetime stamp circuit 22, function as a processor and each implementation is therefore considered to be a processor. The function of thetime stamp circuit 22 is to determine a current time and append a current time stamp to data packets. - The
port steering element 20 can also be implemented using field programmable gate arrays (FPGA), application specific integrated circuits (ASIC), combinational and sequential logic devices or suitably capable microprocessors. Different implementations of theport steering element 20 also function as a processor and each implementation is therefore considered to be a processor. The function of theport steering element 20 is to direct or route data packets to one or more appropriate data queues in thequeue structure 25. - Similarly,
scheduling engines 32 can also be implemented using field programmable gate arrays (FPGA), application specific integrated circuits (ASIC), combinational and sequential logic devices or suitably capable microprocessors. Different implementations of thescheduling engines 32 function as a processor and each implementation is therefore considered to be a processor. The function of the scheduling engines is to schedule the delivery of data packets from a particular queue to its output port using the foregoing methodology. - Packet latency values are derived from the current value of a global clock and from timestamp information. The LMAX0 to LMAXN parameters are the maximum latency value in each queue. The JFACTOR parameters are calculated using TMAX0 to TMAXN values, TMIN0 to TMINN values and scaling constants that are all assigned by a system operator. All of the data values are preferably stored in relatively high-speed memory, or in registers, both of which are accessible to the scheduling engines. The aforementioned dedicated logic circuit(s), FPGA, ASIC or processor performs a least one calculation including but not limited to: packet sorting, packet latency, packet eligibility, packet overload and packet scaling.
- While the preferred embodiment contemplates usage in the CMTS of a cable data system, those of skill in the art of queuing theory will recognize that the methodology disclosed herein is applicable to any system that uses or relies on queuing theory to determine the allocation of a resource as dictated by time constraints.
- In a cable data system, having a CMTS using the methodology disclosed herein as well as a distribution network, such as a coaxial cable or fiber optic network a cable system operator can provide truly differentiated service levels and render to subscribers, truly different levels of service.
- Queuing theory is often employed in other services, e.g., call service centers, which attempt to timely provide some form of limited-availability service to customers. In addition to using the above-identified method on only cable data systems, the method finds application to any service provider that wishes to provide differentiated levels of service. Accordingly, for purposes of claim construction the applicability of the disclosed and claimed methods are not limited to or by the term “data system” but to also include any sort of service provider that attempts to provide differentiated levels of service.
- In a CMTS, the shared resource bandwidth capacity on an Ethernet or coaxial data link. In other non-data systems, the shared resource to be allocated would be the service that the provider nominally provides. In a fiber optic data system, the shared resource would be the availability of the cable to carry information.
Claims (33)
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US10/180,736 US7272144B2 (en) | 2002-06-26 | 2002-06-26 | Method and apparatus for queuing data flows |
US11/839,115 US7698461B2 (en) | 2002-06-26 | 2007-08-15 | Method and apparatus for queuing data flows |
US12/719,917 US7958260B2 (en) | 2002-06-26 | 2010-03-09 | Method and apparatus for queuing data flows |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US10/180,736 US7272144B2 (en) | 2002-06-26 | 2002-06-26 | Method and apparatus for queuing data flows |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/839,115 Continuation US7698461B2 (en) | 2002-06-26 | 2007-08-15 | Method and apparatus for queuing data flows |
Publications (2)
Publication Number | Publication Date |
---|---|
US20040001493A1 true US20040001493A1 (en) | 2004-01-01 |
US7272144B2 US7272144B2 (en) | 2007-09-18 |
Family
ID=29778988
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US10/180,736 Active 2025-03-17 US7272144B2 (en) | 2002-06-26 | 2002-06-26 | Method and apparatus for queuing data flows |
US11/839,115 Active 2028-06-05 US7698461B2 (en) | 2002-06-26 | 2007-08-15 | Method and apparatus for queuing data flows |
US12/719,917 Expired - Fee Related US7958260B2 (en) | 2002-06-26 | 2010-03-09 | Method and apparatus for queuing data flows |
Family Applications After (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/839,115 Active 2028-06-05 US7698461B2 (en) | 2002-06-26 | 2007-08-15 | Method and apparatus for queuing data flows |
US12/719,917 Expired - Fee Related US7958260B2 (en) | 2002-06-26 | 2010-03-09 | Method and apparatus for queuing data flows |
Country Status (1)
Country | Link |
---|---|
US (3) | US7272144B2 (en) |
Cited By (66)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20020069340A1 (en) * | 2000-12-06 | 2002-06-06 | Glen Tindal | System and method for redirecting data generated by network devices |
US20020069274A1 (en) * | 2000-12-06 | 2002-06-06 | Tindal Glen D. | System and method for configuration, management and monitoring of network resources |
US20020069367A1 (en) * | 2000-12-06 | 2002-06-06 | Glen Tindal | Network operating system data directory |
US20030051008A1 (en) * | 2001-08-29 | 2003-03-13 | Gorthy Scott B. | System and method for generating a configuration schema |
US20040028069A1 (en) * | 2002-08-07 | 2004-02-12 | Tindal Glen D. | Event bus with passive queuing and active routing |
US20040030771A1 (en) * | 2002-08-07 | 2004-02-12 | John Strassner | System and method for enabling directory-enabled networking |
US20040078457A1 (en) * | 2002-10-21 | 2004-04-22 | Tindal Glen D. | System and method for managing network-device configurations |
US20040117791A1 (en) * | 2002-12-17 | 2004-06-17 | Ajith Prasad | Apparatus, system and method for limiting latency |
US20040148369A1 (en) * | 2002-07-11 | 2004-07-29 | John Strassner | Repository-independent system and method for asset management and reconciliation |
US20040201626A1 (en) * | 2003-03-05 | 2004-10-14 | Quazal Technologies Inc. | Coherent data sharing |
US20040230681A1 (en) * | 2002-12-06 | 2004-11-18 | John Strassner | Apparatus and method for implementing network resources to provision a service using an information model |
US20040267704A1 (en) * | 2003-06-17 | 2004-12-30 | Chandramohan Subramanian | System and method to retrieve and analyze data |
US20050216822A1 (en) * | 2002-02-18 | 2005-09-29 | Hiroshi Kyusojin | Data communication system, data communication management device and method, and computer program |
US20050232155A1 (en) * | 2004-03-15 | 2005-10-20 | Canon Kabushiki Kaisha | Communication apparatus for transmitting frames and priority control method for transmitting frames |
US20050232153A1 (en) * | 2004-04-16 | 2005-10-20 | Vieo, Inc. | Method and system for application-aware network quality of service |
US20050243835A1 (en) * | 2004-04-30 | 2005-11-03 | Puneet Sharma | Assigning WAN links to subflows based on WAN link characteristics and application preferences |
US20060031434A1 (en) * | 2000-12-06 | 2006-02-09 | Tindal Glen D | System and method for configuring a network device |
US20060080434A1 (en) * | 2000-12-06 | 2006-04-13 | Intelliden | Dynamic configuration of network devices to enable data transfers |
US20060140115A1 (en) * | 2003-01-14 | 2006-06-29 | Telefonaktiebolaget L M Ericsson | Resource allocation management |
US20060179131A1 (en) * | 2001-11-26 | 2006-08-10 | Mike Courtney | System and method for generating a representation of a configuration schema |
US20060217923A1 (en) * | 2005-03-23 | 2006-09-28 | Xiaoxue Ma | Policy-based approach for managing the export of network flow statistical data |
US20070097852A1 (en) * | 2005-10-28 | 2007-05-03 | Viasat, Inc. | Adaptive coding and modulation flow control and traffic shaping systems and methods |
US20070294393A1 (en) * | 2006-05-18 | 2007-12-20 | Harris Corporation | Method and system for functional redundancy based quality of service |
US20070291657A1 (en) * | 2006-06-16 | 2007-12-20 | Harris Corporation | Method and system for rule-based sequencing for QoS |
US20080040718A1 (en) * | 2002-06-26 | 2008-02-14 | Cloonan Thomas J | Method and apparatus for queuing data flows |
US20080043766A1 (en) * | 2006-08-21 | 2008-02-21 | Daniel Measurement And Control, Inc. | Method and System of Message Prioritization in a Control System |
US20080089364A1 (en) * | 2006-08-22 | 2008-04-17 | Brilliant Telecommunications, Inc. | Apparatus and method of controlled delay packet forwarding |
EP1913744A2 (en) * | 2005-08-12 | 2008-04-23 | Cellco Partnership d/b/a Verizon Wireless | Integrated packet latency aware qos scheduling using proportional fairness and weighted fair queuing for wireless integrated multimedia packet services |
US20080212517A1 (en) * | 2005-10-28 | 2008-09-04 | Viasat, Inc. | Quality of service enhancements for adaptive coding and modulation |
US7472412B2 (en) | 2001-03-21 | 2008-12-30 | Wolf Jonathan S | Network configuration manager |
US20090059931A1 (en) * | 2005-04-29 | 2009-03-05 | Jos Den Hartog | Method, system and use thereof for controlling real time contiguous data in a packet switched data system, real time contiguous data service provided using said method |
US20090113054A1 (en) * | 2006-05-05 | 2009-04-30 | Thomson Licensing | Threshold-Based Normalized Rate Earliest Delivery First (NREDF) for Delayed Down-Loading Services |
US7756134B2 (en) | 2006-05-02 | 2010-07-13 | Harris Corporation | Systems and methods for close queuing to support quality of service |
US7769028B2 (en) | 2006-06-21 | 2010-08-03 | Harris Corporation | Systems and methods for adaptive throughput management for event-driven message-based data |
US7826376B1 (en) * | 2005-01-25 | 2010-11-02 | Symantec Operating Corporation | Detection of network problems in a computing system |
US20100278055A1 (en) * | 2009-04-29 | 2010-11-04 | Barry Charles F | Apparatus and Method of Compensating for Clock Frequency and Phase Variations by Processing Packet Delay Values |
US7856012B2 (en) | 2006-06-16 | 2010-12-21 | Harris Corporation | System and methods for generic data transparent rules to support quality of service |
US20110004909A1 (en) * | 2009-07-03 | 2011-01-06 | Yuya Ogi | Wireless video transmitter |
US20110002236A1 (en) * | 2004-06-28 | 2011-01-06 | Minghua Chen | Optimization of streaming data throughput in unreliable networks |
US20110066563A1 (en) * | 2009-09-11 | 2011-03-17 | Lalita Jagadeesan | Mashup sevices and methods with quality of sevice (QoS) support |
US20110066415A1 (en) * | 2009-09-15 | 2011-03-17 | Stanley Thompson | Method, computer program product, and apparatus for comparing traffic pattern models of a complex communication system |
US7916626B2 (en) | 2006-06-19 | 2011-03-29 | Harris Corporation | Method and system for fault-tolerant quality of service |
US8064464B2 (en) | 2006-06-16 | 2011-11-22 | Harris Corporation | Method and system for inbound content-based QoS |
US20120269150A1 (en) * | 2009-11-19 | 2012-10-25 | Nvidia Corporation | Wireless terminal for transmitting packets of different types |
US8300653B2 (en) | 2006-07-31 | 2012-10-30 | Harris Corporation | Systems and methods for assured communications with quality of service |
US20130042307A1 (en) * | 2011-08-11 | 2013-02-14 | Fujitsu Limited | Communication method and information processing system |
US20130060960A1 (en) * | 2011-09-01 | 2013-03-07 | International Business Machines Corporation | Optimizing software applications in a network |
US8516153B2 (en) | 2006-06-16 | 2013-08-20 | Harris Corporation | Method and system for network-independent QoS |
US20140126364A1 (en) * | 2009-06-12 | 2014-05-08 | Cygnus Broadband, Inc. | Systems and methods for prioritizing and scheduling packets in a communication network |
US8730981B2 (en) | 2006-06-20 | 2014-05-20 | Harris Corporation | Method and system for compression based quality of service |
US20140146682A1 (en) * | 2012-11-23 | 2014-05-29 | Verizon Patent And Licensing Inc. | Time-based qos scheduling of network traffic |
US20140233402A1 (en) * | 2013-02-15 | 2014-08-21 | United States Government, as represented by the Department of the Navy | Wireless Network Message Prioritization Technique |
US20150127769A1 (en) * | 2013-11-06 | 2015-05-07 | Amazon Technologies, Inc. | Strict queue ordering in a distributed system |
US9065779B2 (en) | 2009-06-12 | 2015-06-23 | Wi-Lan Labs, Inc. | Systems and methods for prioritizing and scheduling packets in a communication network |
US9229792B1 (en) * | 2007-11-21 | 2016-01-05 | Marvell International Ltd. | Method and apparatus for weighted message passing |
US20160062670A1 (en) * | 2014-08-29 | 2016-03-03 | HewIett-Packard Development Company, L. P. | Determining throughput based on system busy level |
US20160233980A1 (en) * | 2015-02-11 | 2016-08-11 | Broadcom Corporation | Codeword builder for communication systems |
US20160261517A1 (en) * | 2013-11-11 | 2016-09-08 | Nec Corporation | Device, session processing quality stabilization system, priority processing method, transmission method, relay method, and program |
US20160261377A1 (en) * | 2015-03-05 | 2016-09-08 | Accton Technology Corporation | Packet transmission method |
US10439951B2 (en) * | 2016-03-17 | 2019-10-08 | Dolby Laboratories Licensing Corporation | Jitter buffer apparatus and method |
WO2020113185A1 (en) * | 2018-11-28 | 2020-06-04 | Mira Labs, Inc. | Control system for a three dimensional environment |
US10973770B2 (en) | 2004-10-25 | 2021-04-13 | Varian Medical Systems, Inc. | Color-coded and sized loadable polymeric particles for therapeutic and/or diagnostic applications and methods of preparing and using the same |
US11052050B2 (en) | 2004-10-25 | 2021-07-06 | Varian Medical Systems, Inc. | Loadable polymeric particles for therapeutic and/or diagnostic applications and methods of preparing and using the same |
US11228540B2 (en) * | 2019-03-20 | 2022-01-18 | Fujitsu Limited | Communication device, communication system, and communication method |
US11394648B2 (en) * | 2020-02-18 | 2022-07-19 | Fujitsu Limited | Information processing apparatus and information processing method |
US20220321483A1 (en) * | 2021-03-30 | 2022-10-06 | Cisco Technology, Inc. | Real-time data transaction configuration of network devices |
Families Citing this family (29)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7640355B1 (en) * | 2003-08-07 | 2009-12-29 | Cisco Technology, Inc. | Supplemental queue sampling technique for packet scheduling |
US7724750B2 (en) * | 2004-04-01 | 2010-05-25 | Nokia Corporation | Expedited data transmission in packet based network |
US7626988B2 (en) * | 2004-06-09 | 2009-12-01 | Futurewei Technologies, Inc. | Latency-based scheduling and dropping |
CN101091350B (en) * | 2004-11-03 | 2010-05-05 | 艾利森电话股份有限公司 | Method and device for performance optimisation of a data distribution network |
US7835293B2 (en) * | 2005-09-13 | 2010-11-16 | Cisco Technology, Inc. | Quality of service testing of communications networks |
JP4701152B2 (en) * | 2006-10-20 | 2011-06-15 | 富士通株式会社 | Data relay apparatus, data relay method, and data relay program |
EP2165541B1 (en) | 2007-05-11 | 2013-03-27 | Audinate Pty Ltd | Systems, methods and computer-readable media for configuring receiver latency |
DE102007052673A1 (en) * | 2007-11-05 | 2009-05-07 | Kuka Roboter Gmbh | A computing system and method for managing available resources of a particular provided for a control of an industrial robot computing system |
US8601069B1 (en) | 2007-11-21 | 2013-12-03 | Marvell International Ltd. | Method and apparatus for message multicasting |
US8228923B1 (en) * | 2008-01-09 | 2012-07-24 | Tellabs Operations, Inc. | Method and apparatus for measuring system latency using global time stamp |
US8363641B2 (en) | 2008-03-07 | 2013-01-29 | At&T Intellectual Property I, Lp | Methods and apparatus to control a flash crowd event in a voice over Internet protocol (VoIP) network |
US20090249343A1 (en) * | 2008-03-25 | 2009-10-01 | Pirasenna Thiyagarajan | System, method, and computer program product for receiving timer objects from local lists in a global list for being used to execute events associated therewith |
US8619558B1 (en) * | 2008-07-21 | 2013-12-31 | Qlogic, Corporation | Memory management in a network adapter |
US8549341B2 (en) * | 2008-08-29 | 2013-10-01 | Netlogic Microsystems, Inc. | System and method for reducing latency associated with timestamps in a multi-core, multi-threaded processor |
US7912951B2 (en) * | 2008-10-28 | 2011-03-22 | Vmware, Inc. | Quality of service management |
US8250197B2 (en) * | 2008-10-28 | 2012-08-21 | Vmware, Inc. | Quality of service management |
US8799914B1 (en) * | 2009-09-21 | 2014-08-05 | Tilera Corporation | Managing shared resource in an operating system by distributing reference to object and setting protection levels |
US8811407B1 (en) * | 2009-11-30 | 2014-08-19 | Cox Communications, Inc. | Weighted data packet communication system |
US8804762B2 (en) * | 2009-12-17 | 2014-08-12 | Avaya Inc. | Method and system for timestamp inclusion in virtual local area network tag |
WO2011118199A1 (en) * | 2010-03-24 | 2011-09-29 | パナソニック株式会社 | Display switching device |
US20110282980A1 (en) * | 2010-05-11 | 2011-11-17 | Udaya Kumar | Dynamic protection of a resource during sudden surges in traffic |
US8943028B2 (en) * | 2010-06-29 | 2015-01-27 | Teradata Us, Inc. | Database compression analyzer |
US8370392B1 (en) * | 2010-10-07 | 2013-02-05 | The Boeing Company | Methods and systems for controlling storage and transmission of data |
US8891392B2 (en) * | 2012-06-21 | 2014-11-18 | Breakingpoint Systems, Inc. | Dynamic latency analysis system |
US9647916B2 (en) | 2012-10-27 | 2017-05-09 | Arris Enterprises, Inc. | Computing and reporting latency in priority queues |
US8873421B2 (en) * | 2013-01-23 | 2014-10-28 | Oracle International Corporation | Integrated test packet generation and diagnostics |
US10356002B2 (en) * | 2017-01-24 | 2019-07-16 | Nokia Solutions And Networks Oy | Processing real-time multipoint-to-point traffic |
JP6923807B2 (en) * | 2018-05-24 | 2021-08-25 | 日本電信電話株式会社 | Transfer device, transfer method and transfer program |
US10959131B2 (en) * | 2019-03-11 | 2021-03-23 | Cisco Technology, Inc. | Dynamic prioritization of roam events based on latency |
Citations (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5886993A (en) * | 1997-05-30 | 1999-03-23 | Motorola, Inc. | System, device, and method for sharing contention mini-slots among multiple priority classes |
US6006264A (en) * | 1997-08-01 | 1999-12-21 | Arrowpoint Communications, Inc. | Method and system for directing a flow between a client and a server |
US6134217A (en) * | 1996-04-15 | 2000-10-17 | The Regents Of The University Of California | Traffic scheduling system and method for packet-switched networks with fairness and low latency |
US6243754B1 (en) * | 1999-01-08 | 2001-06-05 | International Business Machines Corporation | Dynamic selection of network providers |
US6377583B1 (en) * | 1996-06-27 | 2002-04-23 | Xerox Corporation | Rate shaping in per-flow output queued routing mechanisms for unspecified bit rate service |
US6389019B1 (en) * | 1998-03-18 | 2002-05-14 | Nec Usa, Inc. | Time-based scheduler architecture and method for ATM networks |
US20020141423A1 (en) * | 2001-03-30 | 2002-10-03 | Daisuke Namihira | Queue control method and relay apparatus using the method |
US6501733B1 (en) * | 1999-10-01 | 2002-12-31 | Lucent Technologies Inc. | Method for controlling data flow associated with a communications node |
US20030152076A1 (en) * | 2001-09-19 | 2003-08-14 | Barry Lee | Vertical instruction and data processing in a network processor architecture |
US6614790B1 (en) * | 1998-06-12 | 2003-09-02 | Telefonaktiebolaget Lm Ericsson (Publ) | Architecture for integrated services packet-switched networks |
US20030206559A1 (en) * | 2000-04-07 | 2003-11-06 | Trachewsky Jason Alexander | Method of determining a start of a transmitted frame in a frame-based communications network |
US6647419B1 (en) * | 1999-09-22 | 2003-11-11 | Hewlett-Packard Development Company, L.P. | System and method for allocating server output bandwidth |
US20030236904A1 (en) * | 2002-06-19 | 2003-12-25 | Jonathan Walpole | Priority progress multicast streaming for quality-adaptive transmission of data |
US6728265B1 (en) * | 1999-07-30 | 2004-04-27 | Intel Corporation | Controlling frame transmission |
US6882655B1 (en) * | 1999-05-13 | 2005-04-19 | Nec Corporation | Switch and input port thereof |
US6937603B1 (en) * | 2000-06-02 | 2005-08-30 | Intel Corporation | Optimizing buffer latency in a streamed packet delivery session |
US7058751B2 (en) * | 2001-01-05 | 2006-06-06 | Fujitsu Limited | Packet switch |
US7061862B2 (en) * | 2000-12-11 | 2006-06-13 | Kabushiki Kaisha Toshiba | Inter-network relay system and method |
US20060203725A1 (en) * | 2001-06-13 | 2006-09-14 | Paul Harry V | Fibre channel switch |
US7139901B2 (en) * | 2000-02-08 | 2006-11-21 | Mips Technologies, Inc. | Extended instruction set for packet processing applications |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6035348A (en) * | 1997-06-30 | 2000-03-07 | Sun Microsystems, Inc. | Method for managing multiple ordered sets by dequeuing selected data packet from single memory structure |
US6542504B1 (en) * | 1999-05-28 | 2003-04-01 | 3Com Corporation | Profile based method for packet header compression in a point to point link |
TW484283B (en) * | 2000-08-11 | 2002-04-21 | Ind Tech Res Inst | Dynamic scheduling scheduler framework and method for mobile communication |
AU2002242067A1 (en) * | 2001-01-30 | 2002-08-12 | Nomadix, Inc. | Methods and systems providing fair queuing and priority scheduling to enhance quality of service in a network |
US7272144B2 (en) * | 2002-06-26 | 2007-09-18 | Arris International, Inc. | Method and apparatus for queuing data flows |
US7154902B1 (en) * | 2002-10-21 | 2006-12-26 | Force10 Networks, Inc. | Epoch-based packet switching |
US7277425B1 (en) * | 2002-10-21 | 2007-10-02 | Force10 Networks, Inc. | High-speed router switching architecture |
-
2002
- 2002-06-26 US US10/180,736 patent/US7272144B2/en active Active
-
2007
- 2007-08-15 US US11/839,115 patent/US7698461B2/en active Active
-
2010
- 2010-03-09 US US12/719,917 patent/US7958260B2/en not_active Expired - Fee Related
Patent Citations (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6134217A (en) * | 1996-04-15 | 2000-10-17 | The Regents Of The University Of California | Traffic scheduling system and method for packet-switched networks with fairness and low latency |
US6377583B1 (en) * | 1996-06-27 | 2002-04-23 | Xerox Corporation | Rate shaping in per-flow output queued routing mechanisms for unspecified bit rate service |
US5886993A (en) * | 1997-05-30 | 1999-03-23 | Motorola, Inc. | System, device, and method for sharing contention mini-slots among multiple priority classes |
US6006264A (en) * | 1997-08-01 | 1999-12-21 | Arrowpoint Communications, Inc. | Method and system for directing a flow between a client and a server |
US6389019B1 (en) * | 1998-03-18 | 2002-05-14 | Nec Usa, Inc. | Time-based scheduler architecture and method for ATM networks |
US6614790B1 (en) * | 1998-06-12 | 2003-09-02 | Telefonaktiebolaget Lm Ericsson (Publ) | Architecture for integrated services packet-switched networks |
US6243754B1 (en) * | 1999-01-08 | 2001-06-05 | International Business Machines Corporation | Dynamic selection of network providers |
US6882655B1 (en) * | 1999-05-13 | 2005-04-19 | Nec Corporation | Switch and input port thereof |
US6728265B1 (en) * | 1999-07-30 | 2004-04-27 | Intel Corporation | Controlling frame transmission |
US6647419B1 (en) * | 1999-09-22 | 2003-11-11 | Hewlett-Packard Development Company, L.P. | System and method for allocating server output bandwidth |
US6501733B1 (en) * | 1999-10-01 | 2002-12-31 | Lucent Technologies Inc. | Method for controlling data flow associated with a communications node |
US7139901B2 (en) * | 2000-02-08 | 2006-11-21 | Mips Technologies, Inc. | Extended instruction set for packet processing applications |
US20030206559A1 (en) * | 2000-04-07 | 2003-11-06 | Trachewsky Jason Alexander | Method of determining a start of a transmitted frame in a frame-based communications network |
US6937603B1 (en) * | 2000-06-02 | 2005-08-30 | Intel Corporation | Optimizing buffer latency in a streamed packet delivery session |
US7061862B2 (en) * | 2000-12-11 | 2006-06-13 | Kabushiki Kaisha Toshiba | Inter-network relay system and method |
US7058751B2 (en) * | 2001-01-05 | 2006-06-06 | Fujitsu Limited | Packet switch |
US20020141423A1 (en) * | 2001-03-30 | 2002-10-03 | Daisuke Namihira | Queue control method and relay apparatus using the method |
US20060203725A1 (en) * | 2001-06-13 | 2006-09-14 | Paul Harry V | Fibre channel switch |
US20030152076A1 (en) * | 2001-09-19 | 2003-08-14 | Barry Lee | Vertical instruction and data processing in a network processor architecture |
US20030236904A1 (en) * | 2002-06-19 | 2003-12-25 | Jonathan Walpole | Priority progress multicast streaming for quality-adaptive transmission of data |
Cited By (121)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7313625B2 (en) | 2000-12-06 | 2007-12-25 | Intelliden, Inc. | Dynamic configuration of network devices to enable data transfers |
US20020069274A1 (en) * | 2000-12-06 | 2002-06-06 | Tindal Glen D. | System and method for configuration, management and monitoring of network resources |
US20020069367A1 (en) * | 2000-12-06 | 2002-06-06 | Glen Tindal | Network operating system data directory |
US7246162B2 (en) | 2000-12-06 | 2007-07-17 | Intelliden | System and method for configuring a network device |
US7246163B2 (en) | 2000-12-06 | 2007-07-17 | Intelliden | System and method for configuring a network device |
US7650396B2 (en) | 2000-12-06 | 2010-01-19 | Intelliden, Inc. | System and method for defining a policy enabled network |
US20020069340A1 (en) * | 2000-12-06 | 2002-06-06 | Glen Tindal | System and method for redirecting data generated by network devices |
US20060080434A1 (en) * | 2000-12-06 | 2006-04-13 | Intelliden | Dynamic configuration of network devices to enable data transfers |
US20060031435A1 (en) * | 2000-12-06 | 2006-02-09 | Tindal Glen D | System and method for configuring a network device |
US20060031434A1 (en) * | 2000-12-06 | 2006-02-09 | Tindal Glen D | System and method for configuring a network device |
US8219662B2 (en) | 2000-12-06 | 2012-07-10 | International Business Machines Corporation | Redirecting data generated by network devices |
US7472412B2 (en) | 2001-03-21 | 2008-12-30 | Wolf Jonathan S | Network configuration manager |
US8296400B2 (en) | 2001-08-29 | 2012-10-23 | International Business Machines Corporation | System and method for generating a configuration schema |
US20030051008A1 (en) * | 2001-08-29 | 2003-03-13 | Gorthy Scott B. | System and method for generating a configuration schema |
US20060179131A1 (en) * | 2001-11-26 | 2006-08-10 | Mike Courtney | System and method for generating a representation of a configuration schema |
US20050216822A1 (en) * | 2002-02-18 | 2005-09-29 | Hiroshi Kyusojin | Data communication system, data communication management device and method, and computer program |
US20080040718A1 (en) * | 2002-06-26 | 2008-02-14 | Cloonan Thomas J | Method and apparatus for queuing data flows |
US7698461B2 (en) * | 2002-06-26 | 2010-04-13 | Arris Group, Inc. | Method and apparatus for queuing data flows |
US20040148369A1 (en) * | 2002-07-11 | 2004-07-29 | John Strassner | Repository-independent system and method for asset management and reconciliation |
US20040030771A1 (en) * | 2002-08-07 | 2004-02-12 | John Strassner | System and method for enabling directory-enabled networking |
US20040028069A1 (en) * | 2002-08-07 | 2004-02-12 | Tindal Glen D. | Event bus with passive queuing and active routing |
US20040078457A1 (en) * | 2002-10-21 | 2004-04-22 | Tindal Glen D. | System and method for managing network-device configurations |
US20040230681A1 (en) * | 2002-12-06 | 2004-11-18 | John Strassner | Apparatus and method for implementing network resources to provision a service using an information model |
US20040117791A1 (en) * | 2002-12-17 | 2004-06-17 | Ajith Prasad | Apparatus, system and method for limiting latency |
US20060140115A1 (en) * | 2003-01-14 | 2006-06-29 | Telefonaktiebolaget L M Ericsson | Resource allocation management |
US7787492B2 (en) * | 2003-01-14 | 2010-08-31 | Telefonaktiebolaget Lm Ericsson (Publ) | Resource allocation management |
US20040201626A1 (en) * | 2003-03-05 | 2004-10-14 | Quazal Technologies Inc. | Coherent data sharing |
US7527558B2 (en) * | 2003-03-05 | 2009-05-05 | Quazal Technologies Inc. | Coherent data sharing |
US8396847B2 (en) * | 2003-06-17 | 2013-03-12 | Bank Of America Corporation | System and method to retrieve and analyze data for decision making |
US20040267704A1 (en) * | 2003-06-17 | 2004-12-30 | Chandramohan Subramanian | System and method to retrieve and analyze data |
US7450506B2 (en) * | 2004-03-15 | 2008-11-11 | Canon Kabushiki Kaisha | Communication apparatus for transmitting frames and priority control method for transmitting frames |
US20050232155A1 (en) * | 2004-03-15 | 2005-10-20 | Canon Kabushiki Kaisha | Communication apparatus for transmitting frames and priority control method for transmitting frames |
US20050232153A1 (en) * | 2004-04-16 | 2005-10-20 | Vieo, Inc. | Method and system for application-aware network quality of service |
US20050243835A1 (en) * | 2004-04-30 | 2005-11-03 | Puneet Sharma | Assigning WAN links to subflows based on WAN link characteristics and application preferences |
US8514865B2 (en) * | 2004-04-30 | 2013-08-20 | Hewlett-Packard Development Company, L.P. | Assigning WAN links to subflows based on WAN link characteristics and application preferences |
US8379535B2 (en) * | 2004-06-28 | 2013-02-19 | Videopression Llc | Optimization of streaming data throughput in unreliable networks |
US20110002236A1 (en) * | 2004-06-28 | 2011-01-06 | Minghua Chen | Optimization of streaming data throughput in unreliable networks |
US10973770B2 (en) | 2004-10-25 | 2021-04-13 | Varian Medical Systems, Inc. | Color-coded and sized loadable polymeric particles for therapeutic and/or diagnostic applications and methods of preparing and using the same |
US11052050B2 (en) | 2004-10-25 | 2021-07-06 | Varian Medical Systems, Inc. | Loadable polymeric particles for therapeutic and/or diagnostic applications and methods of preparing and using the same |
US7826376B1 (en) * | 2005-01-25 | 2010-11-02 | Symantec Operating Corporation | Detection of network problems in a computing system |
US7143006B2 (en) * | 2005-03-23 | 2006-11-28 | Cisco Technology, Inc. | Policy-based approach for managing the export of network flow statistical data |
US20060217923A1 (en) * | 2005-03-23 | 2006-09-28 | Xiaoxue Ma | Policy-based approach for managing the export of network flow statistical data |
US8081636B2 (en) * | 2005-04-29 | 2011-12-20 | Telefonaktiebolaget Lm Ericsson (Publ) | Method, system and use thereof for controlling real time contiguous data in a packet switched data system, real time contiguous data service provided using said method |
US20090059931A1 (en) * | 2005-04-29 | 2009-03-05 | Jos Den Hartog | Method, system and use thereof for controlling real time contiguous data in a packet switched data system, real time contiguous data service provided using said method |
EP1913744A2 (en) * | 2005-08-12 | 2008-04-23 | Cellco Partnership d/b/a Verizon Wireless | Integrated packet latency aware qos scheduling using proportional fairness and weighted fair queuing for wireless integrated multimedia packet services |
EP1913744A4 (en) * | 2005-08-12 | 2011-03-23 | Cellco Partnership Dba Verizon | Integrated packet latency aware qos scheduling using proportional fairness and weighted fair queuing for wireless integrated multimedia packet services |
US7986624B2 (en) | 2005-10-28 | 2011-07-26 | Viasat, Inc. | Quality of service enhancements for adaptive coding and modulation |
US20070116152A1 (en) * | 2005-10-28 | 2007-05-24 | Viasat, Inc. | Adaptive coding and modulation queuing methods and devices |
US7680040B2 (en) | 2005-10-28 | 2010-03-16 | Viasat, Inc. | Adaptive coding and modulation for broadband data transmission |
US7684425B2 (en) * | 2005-10-28 | 2010-03-23 | Viasat, Inc. | Adaptive coding and modulation queuing methods and devices |
US7689162B2 (en) | 2005-10-28 | 2010-03-30 | Viasat, Inc. | Adaptive coding and modulation flow control and traffic shaping systems and methods |
US20070097852A1 (en) * | 2005-10-28 | 2007-05-03 | Viasat, Inc. | Adaptive coding and modulation flow control and traffic shaping systems and methods |
US20100118768A1 (en) * | 2005-10-28 | 2010-05-13 | Viasat, Inc. | Adaptive coding and modulation using linked list data structures |
US20100128678A1 (en) * | 2005-10-28 | 2010-05-27 | Viasat, Inc. | Adaptive coding and modulation for broadband data transmission |
US20100172234A1 (en) * | 2005-10-28 | 2010-07-08 | Viasat, Inc. | Adaptive coding and modulation flow control and traffic shaping systems and methods |
US7675842B2 (en) | 2005-10-28 | 2010-03-09 | Viasat, Inc. | Adaptive coding and modulation using linked list data structures |
US8358657B2 (en) * | 2005-10-28 | 2013-01-22 | Viasat, Inc. | Adaptive coding and modulation for broadband data transmission |
US20070096788A1 (en) * | 2005-10-28 | 2007-05-03 | Viasat, Inc. | Adaptive coding and modulation using linked list data structures |
US20070116151A1 (en) * | 2005-10-28 | 2007-05-24 | Viasat, Inc. | Adaptive coding and modulation for broadband data transmission |
US20080212517A1 (en) * | 2005-10-28 | 2008-09-04 | Viasat, Inc. | Quality of service enhancements for adaptive coding and modulation |
US8072873B2 (en) | 2005-10-28 | 2011-12-06 | Viasat, Inc. | Adaptive coding and modulation using linked list data structures |
US7756134B2 (en) | 2006-05-02 | 2010-07-13 | Harris Corporation | Systems and methods for close queuing to support quality of service |
US20090113054A1 (en) * | 2006-05-05 | 2009-04-30 | Thomson Licensing | Threshold-Based Normalized Rate Earliest Delivery First (NREDF) for Delayed Down-Loading Services |
US8650293B2 (en) * | 2006-05-05 | 2014-02-11 | Thomson Licensing | Threshold-based normalized rate earliest delivery first (NREDF) for delayed down-loading services |
US7894509B2 (en) | 2006-05-18 | 2011-02-22 | Harris Corporation | Method and system for functional redundancy based quality of service |
US20070294393A1 (en) * | 2006-05-18 | 2007-12-20 | Harris Corporation | Method and system for functional redundancy based quality of service |
US20070291657A1 (en) * | 2006-06-16 | 2007-12-20 | Harris Corporation | Method and system for rule-based sequencing for QoS |
US7990860B2 (en) * | 2006-06-16 | 2011-08-02 | Harris Corporation | Method and system for rule-based sequencing for QoS |
US8064464B2 (en) | 2006-06-16 | 2011-11-22 | Harris Corporation | Method and system for inbound content-based QoS |
US7856012B2 (en) | 2006-06-16 | 2010-12-21 | Harris Corporation | System and methods for generic data transparent rules to support quality of service |
US8516153B2 (en) | 2006-06-16 | 2013-08-20 | Harris Corporation | Method and system for network-independent QoS |
US7916626B2 (en) | 2006-06-19 | 2011-03-29 | Harris Corporation | Method and system for fault-tolerant quality of service |
US8730981B2 (en) | 2006-06-20 | 2014-05-20 | Harris Corporation | Method and system for compression based quality of service |
US7769028B2 (en) | 2006-06-21 | 2010-08-03 | Harris Corporation | Systems and methods for adaptive throughput management for event-driven message-based data |
US8300653B2 (en) | 2006-07-31 | 2012-10-30 | Harris Corporation | Systems and methods for assured communications with quality of service |
WO2008024696A3 (en) * | 2006-08-21 | 2008-09-04 | Daniel Measurement & Control | Method and system of message prioritization in a control system |
WO2008024696A2 (en) * | 2006-08-21 | 2008-02-28 | Daniel Measurement And Control, Inc. | Method and system of message prioritization in a control system |
US20080043766A1 (en) * | 2006-08-21 | 2008-02-21 | Daniel Measurement And Control, Inc. | Method and System of Message Prioritization in a Control System |
US7590061B2 (en) * | 2006-08-22 | 2009-09-15 | Brilliant Telecommunications, Inc. | Apparatus and method of controlled delay packet forwarding |
US20080089364A1 (en) * | 2006-08-22 | 2008-04-17 | Brilliant Telecommunications, Inc. | Apparatus and method of controlled delay packet forwarding |
WO2008024818A3 (en) * | 2006-08-22 | 2008-09-04 | Brilliant Telecommunications I | Apparatus and method of controlled delay packet forwarding |
US9229792B1 (en) * | 2007-11-21 | 2016-01-05 | Marvell International Ltd. | Method and apparatus for weighted message passing |
US20100278055A1 (en) * | 2009-04-29 | 2010-11-04 | Barry Charles F | Apparatus and Method of Compensating for Clock Frequency and Phase Variations by Processing Packet Delay Values |
US8270438B2 (en) | 2009-04-29 | 2012-09-18 | Juniper Networks, Inc. | Apparatus and method of compensating for clock frequency and phase variations by processing packet delay values |
US9319164B2 (en) | 2009-04-29 | 2016-04-19 | Juniper Networks, Inc. | Apparatus and method of compensating for clock frequency and phase variations by processing packet delay values |
US8494011B2 (en) | 2009-04-29 | 2013-07-23 | Juniper Networks, Inc. | Apparatus and method of compensating for clock frequency and phase variations by processing packet delay values |
US9621290B2 (en) | 2009-04-29 | 2017-04-11 | Juniper Networks, Inc. | Apparatus and method of compensating for clock frequency and phase variations by processing packet delay values |
US8031747B2 (en) | 2009-04-29 | 2011-10-04 | Juniper Networks, Inc. | Apparatus and method of compensating for clock frequency and phase variations by processing packet delay values |
US9065779B2 (en) | 2009-06-12 | 2015-06-23 | Wi-Lan Labs, Inc. | Systems and methods for prioritizing and scheduling packets in a communication network |
US20140126364A1 (en) * | 2009-06-12 | 2014-05-08 | Cygnus Broadband, Inc. | Systems and methods for prioritizing and scheduling packets in a communication network |
US9237112B2 (en) | 2009-06-12 | 2016-01-12 | Wi-Lan Labs, Inc. | Systems and methods for prioritizing and scheduling packets in a communication network |
US9065777B2 (en) * | 2009-06-12 | 2015-06-23 | Wi-Lan Labs, Inc. | Systems and methods for prioritizing and scheduling packets in a communication network |
US8732769B2 (en) * | 2009-07-03 | 2014-05-20 | Hitachi Consumer Electronics Co., Ltd. | Wireless video transmitter |
US20110004909A1 (en) * | 2009-07-03 | 2011-01-06 | Yuya Ogi | Wireless video transmitter |
US20110066563A1 (en) * | 2009-09-11 | 2011-03-17 | Lalita Jagadeesan | Mashup sevices and methods with quality of sevice (QoS) support |
US8990056B2 (en) | 2009-09-15 | 2015-03-24 | At&T Intellectual Property I, L.P. | Method, computer program product, and apparatus for comparing traffic pattern models of a complex communication system |
US8315844B2 (en) * | 2009-09-15 | 2012-11-20 | At&T Intellectual Property I, Lp | Method, computer program product, and apparatus for comparing traffic pattern models of a complex communication system |
US20110066415A1 (en) * | 2009-09-15 | 2011-03-17 | Stanley Thompson | Method, computer program product, and apparatus for comparing traffic pattern models of a complex communication system |
US8879491B2 (en) * | 2009-11-19 | 2014-11-04 | Nvidia Corporation | Wireless terminal for transmitting packets of different types |
US20120269150A1 (en) * | 2009-11-19 | 2012-10-25 | Nvidia Corporation | Wireless terminal for transmitting packets of different types |
US9319493B2 (en) * | 2011-08-11 | 2016-04-19 | Fujitsu Limited | Communication method and information processing system |
US20130042307A1 (en) * | 2011-08-11 | 2013-02-14 | Fujitsu Limited | Communication method and information processing system |
US20130060960A1 (en) * | 2011-09-01 | 2013-03-07 | International Business Machines Corporation | Optimizing software applications in a network |
US9276873B2 (en) * | 2012-11-23 | 2016-03-01 | Verizon Patent And Licensing Inc. | Time-based QoS scheduling of network traffic |
US20140146682A1 (en) * | 2012-11-23 | 2014-05-29 | Verizon Patent And Licensing Inc. | Time-based qos scheduling of network traffic |
US20140233402A1 (en) * | 2013-02-15 | 2014-08-21 | United States Government, as represented by the Department of the Navy | Wireless Network Message Prioritization Technique |
US20150127769A1 (en) * | 2013-11-06 | 2015-05-07 | Amazon Technologies, Inc. | Strict queue ordering in a distributed system |
US9654408B2 (en) * | 2013-11-06 | 2017-05-16 | Amazon Technologies, Inc. | Strict queue ordering in a distributed system |
US20160261517A1 (en) * | 2013-11-11 | 2016-09-08 | Nec Corporation | Device, session processing quality stabilization system, priority processing method, transmission method, relay method, and program |
US9864551B2 (en) * | 2014-08-29 | 2018-01-09 | Hewlett Packard Enterprise Development Lp | Determining throughput based on system busy level |
US20160062670A1 (en) * | 2014-08-29 | 2016-03-03 | HewIett-Packard Development Company, L. P. | Determining throughput based on system busy level |
US20160233980A1 (en) * | 2015-02-11 | 2016-08-11 | Broadcom Corporation | Codeword builder for communication systems |
US10411830B2 (en) * | 2015-02-11 | 2019-09-10 | Avago Technologies International Sales Pte. Limited | Codeword builder for communication systems |
US9634805B2 (en) * | 2015-03-05 | 2017-04-25 | Accton Technology Corporation | Packet transmission method |
US20160261377A1 (en) * | 2015-03-05 | 2016-09-08 | Accton Technology Corporation | Packet transmission method |
US10439951B2 (en) * | 2016-03-17 | 2019-10-08 | Dolby Laboratories Licensing Corporation | Jitter buffer apparatus and method |
WO2020113185A1 (en) * | 2018-11-28 | 2020-06-04 | Mira Labs, Inc. | Control system for a three dimensional environment |
US11228540B2 (en) * | 2019-03-20 | 2022-01-18 | Fujitsu Limited | Communication device, communication system, and communication method |
US11394648B2 (en) * | 2020-02-18 | 2022-07-19 | Fujitsu Limited | Information processing apparatus and information processing method |
US20220321483A1 (en) * | 2021-03-30 | 2022-10-06 | Cisco Technology, Inc. | Real-time data transaction configuration of network devices |
US11924112B2 (en) * | 2021-03-30 | 2024-03-05 | Cisco Technology, Inc. | Real-time data transaction configuration of network devices |
Also Published As
Publication number | Publication date |
---|---|
US20100183021A1 (en) | 2010-07-22 |
US20080040718A1 (en) | 2008-02-14 |
US7698461B2 (en) | 2010-04-13 |
US7272144B2 (en) | 2007-09-18 |
US7958260B2 (en) | 2011-06-07 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7272144B2 (en) | Method and apparatus for queuing data flows | |
US7142513B2 (en) | Method and multi-queue packet scheduling system for managing network packet traffic with minimum performance guarantees and maximum service rate control | |
US7619969B2 (en) | Hardware self-sorting scheduling queue | |
KR100323258B1 (en) | Rate guarantees through buffer management | |
JP3715098B2 (en) | Packet distribution apparatus and method in communication network | |
US6757249B1 (en) | Method and apparatus for output rate regulation and control associated with a packet pipeline | |
US7688734B2 (en) | Scheduling incoming packet traffic on an output link of a network device associated with a data network | |
US6201793B1 (en) | Packet delay estimation in high speed packet switches | |
US7161902B2 (en) | Reducing network traffic congestion | |
US6795870B1 (en) | Method and system for network processor scheduler | |
US7969881B2 (en) | Providing proportionally fair bandwidth allocation in communication systems | |
US20060062152A1 (en) | Apparatus and method for rate-based polling of input interface queues in networking devices | |
KR100463697B1 (en) | Method and system for network processor scheduling outputs using disconnect/reconnect flow queues | |
US6952424B1 (en) | Method and system for network processor scheduling outputs using queueing | |
US7292578B1 (en) | Flexible, high performance support for QoS on an arbitrary number of queues | |
KR100425061B1 (en) | Bandwidth sharing using emulated weighted fair queuing | |
US6862292B1 (en) | Method and system for network processor scheduling outputs based on multiple calendars | |
WO2001039467A1 (en) | Method and system for controlling transmission of packets in computer networks | |
JP2002118585A (en) | Method and apparatus for scheduling packet | |
US7756037B2 (en) | Oversubscription of guaranteed bandwidth | |
US7315901B1 (en) | Method and system for network processor scheduling outputs using disconnect/reconnect flow queues | |
US6618391B1 (en) | Method and apparatus for guaranteeing data transfer rates and delays in data packet networks using discrete data transfer rates | |
McKillen et al. | High performance service-time-stamp computation for WFQ IP packet scheduling | |
Al-Khasib et al. | Mini round robin: an enhanced frame-based scheduling algorithm for multimedia networks | |
Iida et al. | Delay analysis for CBR traffic in static-priority scheduling: single-node and homogeneous CBR traffic case |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: ARRIS INTERNATIONAL INC., GEORGIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:CLOONAN, THOMAS J.;HOWE, JEFFREY J.;URBAN, KEVIN P.;REEL/FRAME:013058/0160 Effective date: 20020613 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
AS | Assignment |
Owner name: ARRIS GROUP, INC., GEORGIA Free format text: MERGER;ASSIGNOR:ARRIS INTERNATIONAL, INC.;REEL/FRAME:019910/0933 Effective date: 20061031 |
|
FPAY | Fee payment |
Year of fee payment: 4 |
|
AS | Assignment |
Owner name: ARRIS ENTERPRISES, INC., GEORGIA Free format text: MERGER;ASSIGNOR:ARRIS GROUP, INC.;REEL/FRAME:030228/0349 Effective date: 20130416 |
|
AS | Assignment |
Owner name: BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT, IL Free format text: SECURITY AGREEMENT;ASSIGNORS:ARRIS GROUP, INC.;ARRIS ENTERPRISES, INC.;ARRIS SOLUTIONS, INC.;AND OTHERS;REEL/FRAME:030498/0023 Effective date: 20130417 Owner name: BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT, ILLINOIS Free format text: SECURITY AGREEMENT;ASSIGNORS:ARRIS GROUP, INC.;ARRIS ENTERPRISES, INC.;ARRIS SOLUTIONS, INC.;AND OTHERS;REEL/FRAME:030498/0023 Effective date: 20130417 |
|
FPAY | Fee payment |
Year of fee payment: 8 |
|
AS | Assignment |
Owner name: ARRIS ENTERPRISES LLC, PENNSYLVANIA Free format text: CHANGE OF NAME;ASSIGNOR:ARRIS ENTERPRISES INC;REEL/FRAME:041995/0031 Effective date: 20151231 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 12TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1553); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 12 |
|
AS | Assignment |
Owner name: NETOPIA, INC., PENNSYLVANIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: ARRIS ENTERPRISES, INC., PENNSYLVANIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: GENERAL INSTRUMENT INTERNATIONAL HOLDINGS, INC., P Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: ARRIS GROUP, INC., PENNSYLVANIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: IMEDIA CORPORATION, PENNSYLVANIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: LEAPSTONE SYSTEMS, INC., PENNSYLVANIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: BROADBUS TECHNOLOGIES, INC., PENNSYLVANIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: NEXTLEVEL SYSTEMS (PUERTO RICO), INC., PENNSYLVANI Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: ARRIS KOREA, INC., PENNSYLVANIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: ACADIA AIC, INC., PENNSYLVANIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: GIC INTERNATIONAL HOLDCO LLC, PENNSYLVANIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: BIG BAND NETWORKS, INC., PENNSYLVANIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: ARRIS SOLUTIONS, INC., PENNSYLVANIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: QUANTUM BRIDGE COMMUNICATIONS, INC., PENNSYLVANIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: AEROCAST, INC., PENNSYLVANIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: GENERAL INSTRUMENT CORPORATION, PENNSYLVANIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: POWER GUARD, INC., PENNSYLVANIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: GENERAL INSTRUMENT AUTHORIZATION SERVICES, INC., P Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: 4HOME, INC., PENNSYLVANIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: CCE SOFTWARE LLC, PENNSYLVANIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: SETJAM, INC., PENNSYLVANIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: THE GI REALTY TRUST 1996, PENNSYLVANIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: MODULUS VIDEO, INC., PENNSYLVANIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: JERROLD DC RADIO, INC., PENNSYLVANIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: MOTOROLA WIRELINE NETWORKS, INC., PENNSYLVANIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: TEXSCAN CORPORATION, PENNSYLVANIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: SUNUP DESIGN SYSTEMS, INC., PENNSYLVANIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: UCENTRIC SYSTEMS, INC., PENNSYLVANIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: ARRIS HOLDINGS CORP. OF ILLINOIS, INC., PENNSYLVAN Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: GIC INTERNATIONAL CAPITAL LLC, PENNSYLVANIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: ARRIS HOLDINGS CORP. OF ILLINOIS, INC., PENNSYLVANIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: GENERAL INSTRUMENT AUTHORIZATION SERVICES, INC., PENNSYLVANIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: NEXTLEVEL SYSTEMS (PUERTO RICO), INC., PENNSYLVANIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 Owner name: GENERAL INSTRUMENT INTERNATIONAL HOLDINGS, INC., PENNSYLVANIA Free format text: TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS;ASSIGNOR:BANK OF AMERICA, N.A., AS ADMINISTRATIVE AGENT;REEL/FRAME:048825/0294 Effective date: 20190404 |
|
AS | Assignment |
Owner name: ARRIS ENTERPRISES LLC, GEORGIA Free format text: CHANGE OF NAME;ASSIGNOR:ARRIS ENTERPRISES, INC.;REEL/FRAME:049586/0470 Effective date: 20151231 |
|
AS | Assignment |
Owner name: WILMINGTON TRUST, NATIONAL ASSOCIATION, AS COLLATE Free format text: PATENT SECURITY AGREEMENT;ASSIGNOR:ARRIS ENTERPRISES LLC;REEL/FRAME:049820/0495 Effective date: 20190404 Owner name: JPMORGAN CHASE BANK, N.A., NEW YORK Free format text: ABL SECURITY AGREEMENT;ASSIGNORS:COMMSCOPE, INC. OF NORTH CAROLINA;COMMSCOPE TECHNOLOGIES LLC;ARRIS ENTERPRISES LLC;AND OTHERS;REEL/FRAME:049892/0396 Effective date: 20190404 Owner name: JPMORGAN CHASE BANK, N.A., NEW YORK Free format text: TERM LOAN SECURITY AGREEMENT;ASSIGNORS:COMMSCOPE, INC. OF NORTH CAROLINA;COMMSCOPE TECHNOLOGIES LLC;ARRIS ENTERPRISES LLC;AND OTHERS;REEL/FRAME:049905/0504 Effective date: 20190404 Owner name: WILMINGTON TRUST, NATIONAL ASSOCIATION, AS COLLATERAL AGENT, CONNECTICUT Free format text: PATENT SECURITY AGREEMENT;ASSIGNOR:ARRIS ENTERPRISES LLC;REEL/FRAME:049820/0495 Effective date: 20190404 |
|
AS | Assignment |
Owner name: WILMINGTON TRUST, DELAWARE Free format text: SECURITY INTEREST;ASSIGNORS:ARRIS SOLUTIONS, INC.;ARRIS ENTERPRISES LLC;COMMSCOPE TECHNOLOGIES LLC;AND OTHERS;REEL/FRAME:060752/0001 Effective date: 20211115 |