US9467307B2 - Method of tracking arrival order of packets into plural queues - Google Patents
Method of tracking arrival order of packets into plural queues Download PDFInfo
- Publication number
- US9467307B2 US9467307B2 US12/200,062 US20006208A US9467307B2 US 9467307 B2 US9467307 B2 US 9467307B2 US 20006208 A US20006208 A US 20006208A US 9467307 B2 US9467307 B2 US 9467307B2
- Authority
- US
- United States
- Prior art keywords
- packets
- coe
- queues
- data
- packet
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active, expires
Links
- 238000000034 method Methods 0.000 title claims abstract description 29
- 238000004891 communication Methods 0.000 claims abstract description 15
- 230000004044 response Effects 0.000 claims abstract description 5
- 230000005540 biological transmission Effects 0.000 claims description 20
- 239000000872 buffer Substances 0.000 claims description 19
- 238000012546 transfer Methods 0.000 claims description 8
- 101000909637 Homo sapiens Transcription factor COE1 Proteins 0.000 description 11
- 102100024207 Transcription factor COE1 Human genes 0.000 description 11
- 238000003491 array Methods 0.000 description 9
- 239000004744 fabric Substances 0.000 description 8
- 238000012358 sourcing Methods 0.000 description 8
- 230000032683 aging Effects 0.000 description 7
- 230000003139 buffering effect Effects 0.000 description 6
- 238000001514 detection method Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 230000002776 aggregation Effects 0.000 description 5
- 238000004220 aggregation Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 101000909641 Homo sapiens Transcription factor COE2 Proteins 0.000 description 3
- 101000909628 Homo sapiens Transcription factor COE3 Proteins 0.000 description 3
- 102100024204 Transcription factor COE2 Human genes 0.000 description 3
- 102100024200 Transcription factor COE3 Human genes 0.000 description 3
- 238000012937 correction Methods 0.000 description 3
- 230000037361 pathway Effects 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 241000234282 Allium Species 0.000 description 2
- 235000002732 Allium cepa var. cepa Nutrition 0.000 description 2
- 238000013459 approach Methods 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 230000002457 bidirectional effect Effects 0.000 description 2
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000013213 extrapolation Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000013468 resource allocation Methods 0.000 description 2
- 238000001228 spectrum Methods 0.000 description 2
- 238000011144 upstream manufacturing Methods 0.000 description 2
- 239000006096 absorbing agent Substances 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000005315 distribution function Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 230000035939 shock Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- H04L12/5693—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/50—Queue scheduling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
- G06F13/38—Information transfer, e.g. on bus
- G06F13/40—Bus structure
- G06F13/4004—Coupling between buses
- G06F13/4022—Coupling between buses using switching circuits, e.g. switching matrix, connection or expansion network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/50—Queue scheduling
- H04L47/62—Queue scheduling characterised by scheduling criteria
- H04L47/6215—Individual queue per QOS, rate or priority
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/50—Queue scheduling
- H04L47/62—Queue scheduling characterised by scheduling criteria
- H04L47/624—Altering the ordering of packets in an individual queue
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/50—Queue scheduling
- H04L47/62—Queue scheduling characterised by scheduling criteria
- H04L47/625—Queue scheduling characterised by scheduling criteria for service slots or service orders
- H04L47/6275—Queue scheduling characterised by scheduling criteria for service slots or service orders based on priority
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L49/00—Packet switching elements
- H04L49/90—Buffering arrangements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L49/00—Packet switching elements
- H04L49/90—Buffering arrangements
- H04L49/901—Buffering arrangements using storage descriptor, e.g. read or write pointers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L49/00—Packet switching elements
- H04L49/90—Buffering arrangements
- H04L49/9047—Buffering arrangements including multiple buffers, e.g. buffer pools
- H04L49/9052—Buffering arrangements including multiple buffers, e.g. buffer pools with buffers of different sizes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L49/00—Packet switching elements
- H04L49/90—Buffering arrangements
- H04L49/9057—Arrangements for supporting packet reassembly or resequencing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L49/00—Packet switching elements
- H04L49/90—Buffering arrangements
- H04L49/9084—Reactions to storage capacity overflow
- H04L49/9089—Reactions to storage capacity overflow replacing packets in a storage arrangement, e.g. pushout
- H04L49/9094—Arrangements for simultaneous transmit and receive, e.g. simultaneous reading/writing from/to the storage element
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2213/00—Indexing scheme relating to interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
- G06F2213/0026—PCI express
Definitions
- the present disclosure of invention relates generally to multi-queue systems wherein competing packets queue up in different queues to await service by a downstream resource of limited bandwidth. For sake of fairness in resource allocation at the downstream end, it is desirable to know which of the packets arrived first, which second and so on at the entrance points of the plural queues.
- each packet is layered like an onion to have header-type outer shell sections, a payload core section and one or more error correction sections that cover various parts of the core and/or outer shells.
- Packets may be transmitted individually or as parts of relatively continuous streams or bursts depending on quality of service requirements and/or availability of bandwidth in available transmission links.
- PHY physical interface layer
- DL data link layer
- TL transaction layer
- the physical interface layer may include means for serializing and deserializing data (SERDES).
- the data link layer may include means for managing error checking and error correction (e.g., ECC, CRC).
- the transaction layer may include means for parsing (peeling the onion skin layers of) different parts of each kind of packet so as to get to desired portions of the payload data. Payload data from sequentially ingressing packets may sometimes need to be reordered for purposes of satisfying priority transmission needs or for reconstructing an original data sequence different from the ingress sequence, where the original data sequence may, for example, be required for reconstituting a rasterized graphic image.
- Packet signals leaving a source device typically progress in the reverse order, namely, first by moving outgoing payload data through the transaction layer (TL), then through the data link layer (DL) and finally through the sender's physical interface layer (PHY) for output onto a physical transmission media (e.g., a high frequency cable or printed circuit strip).
- a physical transmission media e.g., a high frequency cable or printed circuit strip.
- Packet data that is ingressing into a receiving device typically gets routed to a parallel set of buffers (e.g., First-In, First-Out data storage buffers) before being further processed and then being output via one or more egress channels.
- the buffers act somewhat like shock absorbers in that each absorbs and smoothes out the often-bursty nature of ingressing data streams on each respective data channel and then stores the data until it is ready to be processed and/or egressed along a respective egress channel.
- arbitration is often employed is because generally the circuitry downstream of the queues has limited bandwidth and/or buffer capacity and thus cannot process all the data from all the queues simultaneously. Additionally, some of the data streams may have higher priority than others and may need to get through faster. It is often desirable to show some fairness in the downstream resource allocation process (e.g., arbitration) by taking into account which front-of-line packet among the plural queues arrived first, which arrived second and so on.
- One conventional arbitration paradigm keeps track of the precise arrival times of all packets in their respective queues so as to thereby keep track of which queued-up packet arrived first, second and so on.
- each packet Upon arrival into its respective queue, each packet has a time stamp (TS) of predefined bit length attached to it.
- TS time stamp
- an arbiter examines the time stamps of packets awaiting departure from the respective departure gates (front of lines) of their resource queues. In essence, these front-of-the-line packets in the different queues are contending with each other for next service by a next available queue-servicing resource.
- the arbiter checks the arrival time stamps in order to determine which of the awaiting packets came first, which second, and so on.
- a conventional arbitration paradigm may use the relative arrival times of the service ready packets as a factor in determining what ordering of service will be fair to the awaiting packets.
- the arbitration paradigm may use the aging of packets in their respective queues (current time minus time-stamped arrival time) as a factor in determining what ordering of service will be fair given their individual wait times in their respective queues and possibly further based on other factors (e.g., payload priority, etc.).
- the time stamping method works reasonably well within systems that employ transmission links of relatively uniform speed and that receive data into their various queues and output data from the different queues according to a relatively smooth (e.g., flat) distribution function. In such cases, designers can determine with some degree of confidence what to expect in terms of average waiting times for all packets in their respective queues and what to expect as an extreme longest wait time. (In some protocols, longest wait time is defined by the protocol and packets whose stamps show aging beyond that limit are kicked out of the queue (e.g., invalidated) without receiving service.
- the longest wait time for a packet can be set by a user to be as little as about 50 ms and up to a maximum of about 1 second, at which point the queued up packet is deemed expired.) If congestion occurs in a downstream part of the packet flows, in-queue packets can begin to accumulate and their wait times grow.
- Designers who use the time stamping method often need to set the bit lengths of their time stamp fields (the number of bits in each field) sufficiently large so as to accommodate the largest expected wait time. And herein lays a problem. As time stamp fields become longer and longer, more system resources including memory space are consumed for supporting their increased number of bits.
- one logically-configured communication channel may be programmably or dynamically formed as an aggregation of many, relatively slow sub-channel resources (e.g., PCI-Express lanes) while another logically-configured channel may be variably formed to have one or just a few, such slow or basic sub-channel resources (e.g., lanes) or even a virtual fraction of single lane.
- the data bandwidth of the channel containing the greater number of basic sub-channel resources will generally be larger (have a faster data throughput ability) than the data bandwidth of the channel having just one or few sub-channel resources aggregated together (or a fraction of a single resource).
- a trade off is generally made between number of sub-channel resources consumed per communication channel and the bandwidth of each such channel.
- the aggregated variable bandwidth channel resources are sometimes referred to as logical “ports” or “links” and the lowest common speed, for a non-fractured sub-channel resource at the physical layer level is often referred to as a “lane”.
- Lanes may be selectively aggregated together to define higher speed or lower speed ports in PCI-Express systems as appropriate for given circumstances.
- Physically aggregated ports may be selectively bifurcated by software to define larger numbers of virtual channels per port albeit with lower bandwidths per virtual channel.
- a large number of slow moving queues may be formed where, due to downstream congestion, each queue may be packed with many packets and each of the many queues may thus have a relatively long wait time.
- maximum wait time dictates the number of bits needed per time stamp. Processing burden on the arbitration process tends to be a function of the number of bits per time stamp multiplied by the number of contending channels.
- queue design it had been conventional to configure all the packet receiving buffers (queues) of a communications or network device to be of the same depth and width. More recently, it has been proposed to have queues of variable lengths so as to make more efficient use of scarce data storage resources. This is another area in which the extra-long lengths possible with conventional time stamping can present a problem.
- the large numbers of storage bits consumed by each of the extra-long time stamps can operate to defeat the objectives of variable depth buffer control, namely efficient use of limited storage capacity by squeezing each individual buffer's size down close to limit of what is actually needed by its respective channel.
- variable bandwidth links in the PCI-Express 1.0TM protocol is discussed.
- the associated software determines how many lanes (subchannel resources) to assign to each PCI-ExpressTM “port” or PCIe logical “link” (the terms PCIe port and PCIe link are sometimes used interchangeably) so as to thereby define the maximum data throughput rate supported by that port.
- a first PCIe port may be programmably configured (during network boot-up) to consist of an aggregation of 8 basic hardware lanes with a lowest common bandwidth per lane in the PCI-Express 1.0TM protocol being for example 2.5 Gb/s (Giga-bits per second) thus giving the ⁇ 8 first Port an aggregated bandwidth of 20 Gb/s.
- That first port can support a corresponding single channel of 20 Gb/s bandwidth or multiple virtual channels with lower bandwidths that can add up to as much as 20 Gb/s.
- a second PCIe port can be programmably configured during the same network boot-up to consist of an aggregation of just 4 basic lanes, thus giving that ⁇ 4 second Port an aggregated bandwidth of 10 Gb/s.
- a third PCIe port can be programmably configured during the same network boot-up to consist of just one lane; thus giving that ⁇ 1 Port a bandwidth of just 2.5 Gb/s.
- the first through third ports may be reconfigured differently due to flexible resource negotiations that can take place during each network reconfiguration.
- a multi-ported switching device In a PCIe system, it is possible for a multi-ported switching device to have one of its ports logically configured after bring-up as an aggregation of 2 basic lanes (thus giving the ⁇ 2 Port a 5.0 Gb/s bandwidth) and another of its ports configured as one lane (a 2.5 Gb/s bandwidth for that communications port) due to adaptive link negotiations that take place during network bring-up.
- the conventional switching device will typically employ fixed-length time stamps for managing arbitration among competing queues. This too can create inefficiencies and/or errors as will become clearer in the below detailed description.
- PCI-Express 1.0TM protocol is mentioned above, the present disclosure is not limited to that protocol. Designs of second and third generation, PCI-Express protocols 2.0 and 3.0 are in development and it is expected that the present disclosure will also be applicable to PCI-Express 2.0 and 3.0 as well as to later generations.
- the newer, faster but backwardly compatible version 2.0 of PCI-Express typically operates at 5.0 Gb/s per lane and the yet newer, faster version 3.0 of PCI-Express is in the works with expected speeds of 8 GigaTransfers per second per lane although that is not finalized at the time of this writing.
- Structures and methods may be provided in accordance with the present disclosure of invention for improving over the above-described, conventional time stamp based approaches for tracking packet arrival order among plural queues (real or virtual).
- An arbitration system in accordance with the present disclosure provides a respective counter for each of a plurality of competing queues where the counter keeps track of the current number of packets (or of other data blocks) awaiting service and residing in that queue.
- the Current Count of Awaiting Packets—or of other awaiting data blocks—(the CCAP) that is maintained by that counter for its queue is decremented.
- the CCAP is incremented.
- systems in accordance with the present disclosure may be designed to store other forms of predefined data blocks of varying lengths that are serviced after waiting their turns in plural queues where the serviced blocks may be later compiled for example to form entire packets or packet payloads.
- that written disclosure reference within the specification to a packet stored in a queue may also be understood to cover a predefined data block of optionally varying length that may be stored in one of plural queues.
- packet may be understood, unless otherwise stated, to mean either a whole packet (with header, payload and error correction trailer) or just a packet payload.
- each of plural packets arriving for storage in respective ones of plural queues has attached to it or otherwise associated with it, not the current CCAP (Current Count of Awaiting Packets) of its own given queue, but rather the CCAP's of one or more of the other queues whose outputs compete with one another and with the output of the given queue for grant of service by one or more downstream and limited resources (e.g., passage through a switch fabric or transmission through a transmission link of limited bandwidth).
- the grant of service may be controlled by an automated arbiter and/or service scheduler.
- the attached/associated instance of CCAP values (which instance is referred to herein as the packet's COE data array, or Count Of Earlier-arrivals array) is updated each time a packet departs from (or is invalidated or expired within) one of those others of the competing-for-service queues.
- the counts in the COE data array of each packet are not decremented to a value below zero or another predefined floor value. In other words, they saturate at a predefined floor count.
- COE-tracked given packet arrives at the departure gate (service gate) of its own given queue, its logically or physically attached COE data array value(s) will indicate how many valid packets (if any) in the others of the one or more competing queues arrived earlier than the given packet.
- An arbiter and/or scheduler can look at the attached COE data array values of all packets awaiting in the respective departure gates of their respective queues and the arbiter/scheduler can then determine relative arrival orders or at least it can determine which packet arrived before all others (it will be the one with the lowest COE counts, typically all zeroes—meaning no other packets arrived earlier into the collection of competing queues).
- COE-tracked packets that have relatively small valued counts in their COE's may be understood in certain circumstances to have arrived in the queues collection after a relatively small number of earlier arriving packets arrived in the other queues.
- COE-tracked packets that have relatively larger valued counts in their respective COE's may be understood to have arrived in the queues collection after a relatively larger number of earlier arriving packets arrived in the other queues.
- a departure awaiting packet with all its COE data array values equal to zero is understood to have no packets (zero packets) that arrived earlier than it in the tracked collection of plural queues.
- An arbiter/scheduler can thus automatically determine which packet is the earliest arriver and/or it can thus automatically determine relative order of arrival among competing packets of plural queues by inspecting the attached COE array values. The arbiter/scheduler can then make arbitration and/or scheduling decisions based at least on such relative orders of arrivals.
- the arbiter/scheduler algorithm may use numerous other factors to determine which of the competing packet(s) will ultimately be serviced ahead of others (e.g., allowed to depart from its given queue first) where these other factors may include extent of downstream congestion or extent of empty buffer space in downstream resources and priority of service indicators associated with the awaiting packets.
- COE data arrays Counter Of Earlier-arrivals arrays
- the COE's count whole numbers of packets (or of other predefined data blocks) as opposed to counting almost limitless units of small quantums of time. Since buffers have finite determinable sizes and packets have finite determinable minimum sizes, the number of bits needed to keep track of numbers of packets in each of plural queues can be much smaller than the number of bits needed by conventional time stamping for keeping precise track of times of arrival.
- the number of bits needed to keep track of arrival order is minimized.
- the single attached CCAP provides order of arrival information as between the queue receiving the new packet and the other selected queue.
- the attached singular value (the COE) is decremented each time a packet departs from that selected other queue (or is expired at the departure gate of that other queue) but it is not decremented to a value below zero.
- a service-ready packet of this one particular embodiment with a singular COE count of zero (0) is understood to have arrived ahead of all packets in the other queue.
- each of the plural queues tracks the departures of one unique other of the queues (e.g., the 2nd queue tracks the 1st, the 3rd tracks the 2nd, . . . , the 1st tracks the Nth queue) then a service-ready packet of this one particular sub-embodiment having a singular COE count of zero (0) is understood to have arrived ahead of all packets in all the other queues.
- each attached CCAP from another queue provides 2-way information about the relationship between data in the two queues, namely, how many packets in the other queue are ahead of this newest packet of this queue and also the information that all afterwards arriving packets in this queue are behind (later arriving) the noted number of packets in the other queue.
- all unique permutations of relative arrival order need to be accounted for.
- the number of unique permutations per the basic combination formula n!/k!(n ⁇ k)! is N ! / ⁇ (2!)(N ⁇ 2)! (in other words, Nc 2 which is N factorial divided by 2!
- COE-based tracking is implemented in a network device (e.g., a packet router) that is structured to selectively dispatch to-be-egressed packet data to plural egress channels of potentially different bandwidths.
- An egress scheduler reads the attached COE values of packets awaiting dispatch from their respective queues (real or virtual) to corresponding egress channels (real or virtual) and uses the COE values as indicators of relative arrival orders.
- a service-ready packet that has a COE data array of all zeroes may be understood to have no packets that arrived ahead of it in the tracked other queues.
- a service-ready given packet that has a COE value of one may be understood to have at least one packet in another queue that arrived ahead of the given packet whose COE equals one.
- the number of bits used by the COE-storing registers (e.g., counters) or COE-storing memory fields which performing COE-tracking on behalf of a given packet of a given queue is equal to the smallest whole power of 2 whose corresponding value of 2 raised to that power is equal to or greater than the maximum number of shortest packets that can be stored at one time in any of the other queues whose CCAP's are being compiled to form the COE data array of the given packet.
- a relatively small number of COE counter bits e.g., 8 bits to count as many as 255 earlier arriving other packets in another queue
- each collection of one or more COE-storing registers (counters) that performs tracking for a given packet has associated with it an index-storing register that stores a unique index number identifying the given packet whose COE data array is being kept by an associated one or more COE-storing registers (counters).
- a packets transfer device in accordance with the present disclosure may be additionally configured to have data buffering FIFO's with programmably variable (elastic) FIFO depths for each of programmably re-configurable ports or communication channels so that FIFO depth may be efficiently tailored (e.g., reduced) to match the bandwidth needs of differently configured ingress and/or egress channels.
- the COE-based tracking of packet arrival order helps to reduce buffer size to less than what may be needed with conventional time stamping.
- a machine-implemented method is provided in accordance with the disclosure for more efficiently utilizing packet dispatch resources by testing the COE's (Count Of Earliers-among-pending-packets-of-other-queues) of each of plural packets awaiting dispatch from other queues to thereby determine which packet arrived earliest into a plurality of competing queues.
- a dispatch arbiter uses the COE values as at least one factor in determining which one or more packets to next dispatch and/or otherwise service.
- FIG. 1 is a block diagram of a first packet switching system having channel-servicing FIFO's for storing pre- or post-process packets that are about to be dispatched to egress channels of differing bandwidths, where packet aging is determined by use of attached time stamps;
- FIG. 2A is a block diagram of part of a second packet switching system, similar to that of FIG. 1 except that order of packet arrivals in the plural competing FIFO's is determined by use of attached COE counters;
- FIG. 2B is a block diagram of part of a third packet switching system, similar to that of FIG. 2A except that the COE counters of each queue are initially loaded with the CCAP's of all queues;
- FIG. 3A is a block diagram of part of a fourth packet switching system, similar to that of FIG. 2A except that the COE counters of each queue are initially loaded with the CCAP of just one other competing queue;
- FIG. 3B is a schematic diagram providing a more detailed look at one embodiment in accordance with FIG. 3A .
- first networked communication system 100 that is schematically illustrated in FIG. 1
- column 105 seen under column 105 is a packet carrying network which is understood to be fed by a plurality of data sourcing devices including first through fourth data sourcing devices (not shown) which feed data into respective ingress pipes 111 - 114 of column 110 .
- the first data sourcing device (the one feeding pipe 111 ) is understood to have a relatively large data output bandwidth —at least for the moment—while the second device (feeding pipe 112 ) is understood to have a relatively smaller or moderate bandwidth.
- the burst mode data output bandwidth of the first sourcing device can be as high as 16 Bytes per clock cycle (16 B/cc) while the burst mode data output bandwidth of the second sourcing device (not shown) can be no higher than 4 B/cc.
- Others of the data sourcing devices e.g., the ones feeding pipes 112 , 113 , etc. are understood to have variable burst mode or steady state output bandwidths ranging from a possible minimum channel bandwidth (as defined by system protocols) to a possible maximum or widest channel bandwidth (as defined by system protocols).
- the third ingress pipe 113 is shown as being fed by a data sourcing device of average bandwidth (e.g., 8 B/cc) and the fourth ingress pipe 114 is shown as being fed by a data sourcing device of relatively large bandwidth (e.g., 16 B/cc).
- the system is understood to allow for more channels than just the illustrative four and the narrowness of the thinnest pipes can be programmably varied to be substantially smaller than 1 B/cc by use of virtual port bifurcation.
- various data stream routing paths through the network 105 can have different transmission latencies.
- the actual arrival time may vary according to variable latency attributes of the network 105 .
- FIG. 1 is merely illustrative. Although just one plurality of competing queues (e.g., FIFO's) 131 - 134 is shown in a block 130 that is interposed between switch fabric 120 and egress dispatcher 150 , it is within the contemplation of the disclosure to have differently positioned or even multiple pluralities of such competing queues in a given device or system. For example an additional or alternate plurality of competing queues can be disposed upstream of the illustrated switch fabric 120 .
- FIFO's e.g., FIFO's
- An additional or alternate arbiter/scheduler like 171 may be provided for this additional or alternate plurality of competing queues (not shown) and the COE-based tracking of packet arrival order may be employed for such an upstream plurality of competing queues (which compete for passage of their packets through switch fabric 120 ) in place of or in addition to the below described COE-based tracking of packet arrival order in the egress side set 130 of competing queues.
- COE-based tracking of packet arrival order need not be limited to ingress and egress sides of a single plurality of queues. Packets may travel through a series of queues where arbitration occurs at the end of the series based on arrival order at the beginning of the series.
- any of the ingress channels (pipes) 111 - 113 —etc. in layer 110 can be programmably or fixedly reconfigured to have different throughput rates almost anywhere in the allowable spectrum of pipe bandwidths of the system between what may be considered a very small bandwidth (system minimum throughput rate) and very large bandwidth (system maximum throughput rate). Specific throughput rates may vary from application to application, or time to time.
- packets or other data blocks
- a distributed network 105 e.g., having variable latencies
- plural queues e.g., FIFO's
- keeping track of order of arrival is important because, for example, an automated arbiter and/or scheduler 170 further downstream is expected to fairly allocate servicing and/or egress time slots for the packets that earlier arrived into the plural and competing queues based, among other things, on packet arrival time within the initial holding queues 131 - 134 —etc.
- FIG. 1 also shows the employment of FIFO's 131 - 134 —etc. of potentially different depths (programmably elastic storage capacities) given that the data sources or data pipes ( 111 - 114 ) feeding data into them can be of different burst-mode or average bandwidths and the data sinks or egress pipes ( 161 - 164 ) pulling data out of the FIFO's (under control by arbiter/scheduler 170 ) can also be of different burst-mode or average bandwidths.
- the illustrated first egress pipe 161 is relatively thin (slow, e.g., 1 B/cc maximum) whereas an ingress pipe that could be currently feeding into corresponding FIFO 131 is relatively wide (fast, e.g., 16 B/cc maximum).
- FIFO depths are adjusted to make efficient use of limited high speed storage space (e.g., buffering memory 130 ) it is additionally beneficial to use an arrival order tracking system which uses a minimal number of tracking bits per packet.
- time-stamp based tracking e.g., 131 T
- time-stamp based tracking e.g., 131 T
- the ingress channels column 110 can represent aggregations of physical transmission media and/or logical bifurcations of logical (virtual) transmission pipes 111 - 114 (e.g., as might occur under the PCI-Express protocol for example) which pipes respectively conduct signals from respective data source devices of the network 105 to an ingressing-data side (data receiving side) of a switch fabric 120 and then to an egress side, of a data buffering memory area 130 , where in the illustrated example memory area 130 contains the plural queues whose outputs compete with each other for a limited resource—e.g., dispatch through dispatcher 150 .
- a limited resource e.g., dispatch through dispatcher 150 .
- the physical transmission media and/or logical data transfer pipes 111 - 114 will generally not have differing appearances and each may appear simply as a single coaxial cable or a single optical fiber or a high frequency transmission strip on a printed circuit board coupled to a physical media interface circuit followed by SERDES circuitry (serializing and de-serializing circuitry).
- all the illustrated pipes 111 - 114 can be multiplexed over a single, bidirectional optical transmission line prior to being demultiplexed and de-serialized into parallel electrical signal flows.
- transmission media/pipes 111 - 114 (which can be bidirectional media/pipes) are schematically shown as being separate wide, narrow or medium width data pipes for ease of understanding. Width indicates bandwidth in this schematic representation.
- Transmission pipe 111 is shown to be a relatively “fat” data flow pipe which means that pipe 111 can handle a relatively large bandwidth of incoming data, say at a rate of 16 Bytes per clock cycle.
- transmission pipe 112 is shown as being a comparatively thinner data flow pipe which means that pipe 112 handles no more than the smaller bandwidth of incoming data, say at a rate of 4 Bytes per clock cycle.
- transmission pipe 161 is shown as a narrow data flow pipe which means that egress pipe 161 handles no more than an even smaller bandwidth of data incoming into that pipe 161 , say at a rate of 1 Byte per clock cycle. It is assumed for sake of example that switch fabric 120 is currently configured to route data such that all outflow from ingress pipe 111 goes into FIFO 131 , all outflow from ingress pipe 112 goes into FIFO 132 , and so on although in practice routing through the switch fabric 120 may be dynamically determined by self-routing data provided in each incoming packet.
- arbiter/scheduler 170 uses an arbitration/-scheduling algorithm that calls for order of arrival into memory 130 (in other words, who arrived first, second, etc. through the left side of box 130 ) as one of its input parameters.
- Shown at 115 is an exemplary data packet having a header section 115 a , a payload section 115 b and an error checking and/or correcting section (ECC) 115 c .
- ECC error checking and/or correcting section
- the source device (on the other side of network 105 ) first requests access through a network pathway that includes a corresponding ingress pipe (e.g., 113 ), sufficient vacancy room in a destination storage means (e.g., FIFO 131 ) and passage through the switch fabric 120 to the destination storage means.
- a network pathway that includes a corresponding ingress pipe (e.g., 113 ), sufficient vacancy room in a destination storage means (e.g., FIFO 131 ) and passage through the switch fabric 120 to the destination storage means.
- a domain controller (not shown) grants the request and the source device then streams a continuous sequence of packet data (for example, short packets 131 a , and 131 b carrying the source data) through the granted network pathway; and when finished, the source device relinquishes use of the pathway resources (e.g., 105 , 113 ) so that other source devices (or reply-completion devices) can use the relinquished network resources for other operations.
- the pathway resources e.g., 105 , 113
- the FIFO-absorbed packets sit in queue within the FIFO (e.g., 131 ) awaiting dispatch into (and/or other service prior to dispatch into) the respective egress pipe (e.g., 161 ) under control of the egress flow scheduler 170 . It is generally undesirable to have packets piling up and waiting in a given queue for very long times, in other words, aging excessively without being dispatched.
- respective arrival time stamps (e.g., TSa, TSb) are physically attached to the incoming packets (e.g., 131 a , 131 b ) as each finishes arriving (e.g., with no ECC error) into the ingress side of the FIFO (e.g., 131 ).
- the attached time stamps propagate through the FIFO with their respective packets until they reach a dispatch stage (e.g., 131 X, or dispatch gate portion) of the FIFO and are thus ready for dispatch via a packet dispatcher circuit 150 to the destination egress pipe (e.g., 161 ).
- Passage through the dispatcher 150 and the downstream pipe may be limited due to numerous factors including for example the finite bandwidths of one or both of these downstream circuits (e.g., by the time slots allocating switch 157 and the pipe 161 ) and/or by buffer congestion further downstream.
- the egress flow scheduler 170 reads the time stamp values (TS's via data read line 145 for example) of the packets awaiting in dispatch gates 131 X- 134 X of the respective queues 131 - 134 to determine which of these packets is the oldest and thus arrived ahead of all the other service-ready packets and is thus perhaps more deserving of next dispatch through dispatcher 150 due to its accumulated time waiting in the queue or its relative place in line as being first to arrive into memory 130 .
- the egress flow scheduler 170 may of course use a complex arbitration algorithm for determining which of competing packets at the dispatch gates (e.g., 131 X- 134 X) wins, where packet arrival order and/or packet aging may or may not play a predominant role.
- TSa-TSg time stamps
- buffered packets do not necessarily travel physically through respective FIFO structures 131 - 134 with their attached time stamps moving along with them. Instead, FIFO structures 131 - 134 may be implemented as circular buffers with rotating pointers defining their respective receive and dispatch areas. Data packets (e.g., 131 a - 134 h ) that are received from the respective ingress pipes 111 - 114 appear at a data-input side of memory region 130 as respective write data flows 121 - 124 .
- Routing means may be optionally provided within the ingress data buffering memory 130 for directing respective data flows 121 - 124 to specific kinds of FIFO buffers 131 - 134 within memory region 130 .
- packet types known as posted (P), non-posted (NP) and completion (CT) packets.
- the internal routing means may route each kind of packet to a respective FIFO for that kind of packet as well as generally causing first write data 121 to be stored in general FIFO area 131 , second write data 122 to be stored in general FIFO area 132 and so on.
- each of FIFO's 131 - 134 has a same data width (bits per storage location) and a same depth (total storage capacity).
- each of FIFO's 131 - 134 is a virtual FIFO with a variable memory capacity (e.g., elastic depth) that adaptively conforms at least to the bandwidth of a specific ingress pipe 111 - 114 or egress pipe 161 - 164 serviced by that FIFO.
- a variable memory capacity e.g., elastic depth
- FIFO depth is elastic
- FIFO 131 since FIFO 131 is outputting to a relatively thin pipe 161 , FIFO 131 will automatically be allocated a relatively large depth (Depth # 1 ) and consume much of memory space 130 .
- the illustrated FIFO 134 that is outputting to a relatively wide (fast) pipe 164 will be automatically allocated a relatively short depth (Depth # 4 ) and thus consume less memory.
- This elastic FIFO depth scheme allows for more efficient use of the finite memory resources of buffering memory 130 .
- the conventional time stamping scheme can disadvantageously consume large amounts of memory space because each time stamp (TSa, TSb, etc.) has to consume a large number of bits.
- FIFO 131 where a large number of relatively short packets (each having a small payload) quickly pile up in that FIFO 131 , where that pile-up occurs because the source pipe (e.g., 111 ) is relatively wide and the egress pipe 161 is substantially thinner.
- the shortness of short packets 131 a , 131 b , etc. is generally a function of their data source at the other side of the network 105 .
- length of packet may also be a function of message type. For example, data layer DLLP packets that carry receipt acknowledgement ACL's or NAK's tend to be relatively short. Completion packets tend to be relatively short.
- packets 131 a - 131 b —etc. are all short and arrive quickly one behind the other in this example, the difference between their respective arrival times and thus the difference in value between their respective time stamps (e.g., TSb minus TSa) will be fairly small, thus calling for fine resolution of time differences.
- the arrival time stamper 131 T that generates those time stamps must be able to resolve timing differences to the smallest value possible between the shortest and fastest incoming packets in the case where they arrive into plural queues one immediately after the other.
- FIFO 132 is the case where received packets 132 d and 132 are very long and the source pipe (e.g., 112 ) that supplies them is relatively thin.
- the arrival time stamper 132 T that services FIFO 132 and generates its time stamps may have to resolve timing differences to the largest value possible between the longest and slowest arriving of the packets in the case where they arrive into different queues very far apart, one from the other due to slowness of ingress side pipes (e.g., 112 ).
- the picked bit length is finite, there may be times when the bit length is too small to account for a larger span 132 e - d between packets (e.g., 132 d , 132 e ) that arrive far apart. There may be times when the bit length is too small to account for a consequence of counter rollover (e.g., wrapping around from FFFF back to 0000) when a digital time stamp value is subtracted for example from the current time counter (e.g., 131 T). As a result of unexpected counter rollover, the egress flow scheduler 170 may be fooled into making incorrect scheduling decisions.
- counter rollover e.g., wrapping around from FFFF back to 0000
- FIG. 1 may be somewhat misleading in that dispatch paths 151 - 154 do not have to be separate paths.
- dispatch paths 151 - 154 multiplex through a common dispatch bus having for example, a 16 B/cc maximum bandwidth.
- the dispatcher(s) can only dispatch one data block at a time over this common dispatch bus (not shown).
- the scheduler 170 may pick out the longest running of pending egress streams as a primary dispatch job and then the scheduler 170 may try to fit other pending egress jobs into whatever slack time may be left over by the primary dispatch job.
- FIFO 132 contains a long stream of high priority packets 132 d - 132 e , that does not preclude shorter packets (e.g., 131 a ) from being simultaneously dispatched with slack time slots through from respective others of the egress pipes even as the long job slow dribbles out through its narrow discharge outlet 172 .
- Packets of the primary dispatch job e.g., FIFO 132
- Packets of a threaded-in secondary job do not necessarily have to be comparatively “shorter” packets. These examples are picked simply for amplifying the problem that would evolve if the scheduler 170 limited itself to allowing only one stream to dispatch at a time on an exclusive basis.
- the scheduler 170 can keep the dispatcher(s) 150 and the FIFO memory unit 130 busy for as much of the available time as is possible or practical by detecting slack time slots in the dispatching of the primary dispatch job (e.g., FIFO 132 ) and by determining what additional secondary dispatch jobs can be squeezed in to take advantage of the available slack dispatch bandwidth.
- the scheduler 170 may find a next still ongoing or pending dispatch job (e.g., that of FIFO 133 ) to be designated as the new primary dispatch job. The scheduler 170 then tries to opportunistically fill the holes of slack times of that new, primary dispatch job with secondary dispatch jobs.
- FIG. 2A shown is a portion of a system 200 similar to system 100 of FIG. 1 except that the network 105 , ingress pipes 110 and egress pipes 160 are not shown. It is to be understood that such variable bandwidth transmission means may nonetheless be present in a network communicating device that includes the illustrated portion 200 . It is also to be understood that where practical, reference numbers in the 200 century series are used in FIG. 2A to represent alike elements of FIG. 1 having reference numbers in the 100 century series. As such a description for the general FIFO structures will not be repeated.
- the improved buffering system 200 of FIG. 2A uses counts of pending packets in each FIFO to keep track of arrival order. More specifically, the first FIFO 231 has associated with it a first counter 231 K that keeps count of the number of pending-packets stored in FIFO 231 that have fully arrived (e.g., safely into FIFO 231 , that is, with a good ECC if such a good ECC is needed to get loaded into the FIFO) but have not yet been fully dispatched to a link partner (e.g., safely copied out of FIFO 231 with an acknowledgement, e.g., a DLLP ACK, having been received from the link partner of safe receipt of the packet).
- a link partner e.g., safely copied out of FIFO 231 with an acknowledgement, e.g., a DLLP ACK, having been received from the link partner of safe receipt of the packet.
- a corresponding arrivals detector 231 i (for queue number Q 1 ) that detects completed good arrivals of new packets into the ingress side of FIFO 231 and a corresponding departures detector 231 z (for queue number Q 1 ) that detects completed good departures of dispatched packets from the egress side of FIFO 231 .
- the first pending-packets counter 231 K is incremented (+1) with each detection by the Q 1 arrivals detector 231 i of a new well-arrived packet and it is decremented ( ⁇ 1) with each detection by the Q 1 departures detector 231 z of a packet successfully dispatched from FIFO 231 .
- the pending-packets counter 231 K stops decrementing when it hits a zero count and it resets to zero when FIFO 231 is reset.
- the second FIFO 232 similarly has associated with it a second pending-packets counter 232 K which keeps track of the number of packets stored in FIFO 232 that have been fully received but not yet dispatched out safely to a link partner.
- the third FIFO 233 has a third pending-packets counter 233 K which keeps track of the number of packets stored in FIFO 233 .
- Associated with the second and third pending-packets counters 232 K- 233 K are respective Q 2 and Q 3 arrivals detectors 232 i and 233 i that detect completed good arrivals of new packets into the ingress sides of their respective FIFO's 232 - 233 .
- the respective counts of the first through Nth pending-packets counters 231 K- 23 NK are output as respective signals, CCAP( 1 ) through CCAP(N) where CCAP is short for Count of Current Awaiting Packets in the enumerated queue (1 through N).
- the corresponding first pending-packets counter 231 K is incremented (+1).
- the newly received packet e.g., 231 a
- the so-attached COE data array e.g., COEa(2:N) of packet 231 a
- Array forming unit 261 is understood to generate the COE data array signal (e.g., COEa(2:N) for packet 231 a ) and to directly or indirectly cause the generated COE data array signal to remain bound (physically or logically) with its associated packet as the packet advances physically or logically through the FIFO.
- COE is short of Count Of Earlier-arrivers, where in the case of FIG. 2A it is a count of earlier-arrivers in each of the other queues (e.g., Q 2 to QN inclusive) except that of the respective FIFO (e.g., CCAP( 1 ) of FIFO 231 ).
- the COE data array is stored in an array of counters (e.g., 2:N, not individually shown) that are logically associated with the corresponding packet for which the COE is kept. (See for example the COE 1 . z counter of FIG. 3B .)
- each COE's-storing array is a stage within a large serial shift register (not shown) whose stages can shift stored data (physically or virtually) from one stage to the next as corresponding packets advance (physically or virtually) forward in the corresponding queue (e.g., 231 ) so that the output stage of the shift register holds the COE array of the packet awaiting dispatch from the departure gate (e.g., 231 X) of its respective queue.
- a given queue say 231
- the count (COE) in its departure gate is artificially forced to a large number (e.g., all 1's or FFFF in hex) to indicate that the information in the departure gate (e.g., 231 X) is not an early arrived packet but rather no packet at all.
- all queues are empty, then all their departure gate COE counts are artificially forced to the large number (e.g., all 1's or FFFF in hex) to indicate that the information in the respective departure gates (e.g., 231 X- 233 X) does not represent an early arrived packet.
- the corresponding second pending-packets counter 232 K is incremented.
- the newly received packet e.g., 232 c
- the newly received packet is bound to a respective COE data array where that so-attached COE data array (e.g., COEc(1,3:N) of packet 232 c ) indicates the then current set of CCAP signals of all other queues (1,3:N) except that (CCAP( 2 )) of the second FIFO 232 .
- Array forming unit 262 is understood to generate the COE data array signal (e.g., COEc for packet 232 c ) and to directly or indirectly cause the generated COEc signal to remain bound (physically or logically) with its associated packet 232 c as that packet advances in its queue. Similar array forming structures 263 - 26 N (last not shown for N>3) apply for the third through Nth FIFO's 233 - 23 N (where the Nth one is not shown for case of N>3).
- the corresponding first pending-packets counter 231 K is decremented. Additionally, the Q( 1 ) field in each of the COE data arrays that have such field (COEb of FIG. 2 a does not) are also decremented. This decrementing operation is carried out in FIG. 2A by, for example, COE decrementing updaters 272 - 273 .
- Each COE decrementing updater (e.g., 271 ) has N ⁇ 1 parallel output lines that are reset to ‘0’ by default and are temporarily set to ‘1’ when a packet departure has been detected by the corresponding departure detector (e.g., 232 z ) of a respective queue (2:N).
- the COE decrementing updater (e.g., 271 ) has an addressing control input (e.g., 271 i ) that points its N ⁇ 1 parallel output lines towards first selectively decrementing ( ⁇ 1) the COEa(2:N) registers of a first packet, then the COEb(2:N) registers of a second packet, and then pointing further towards the front end of FIFO 231 so as to selectively decrement the appropriate COE field in each COE data array so as to reflect the latest packet departure(s), if any, out of one or more of other queues 2 through N (2:N).
- an addressing control input e.g., 271 i
- each COE field indicates how many earlier arrivers that are still valid remain in the respective other queue (e.g., 2:N) where that earlier arriver packet arrived before the logically bound packet (e.g., 231 b ) and was still pending when the logically bound packet (e.g., 231 b ) arrived in its queue (e.g., 231 ). While the example of COE decrementing updaters 272 - 273 given in FIG.
- 2A calls for a sequential address scanning form (in other words, selectively updating the COE data array of the oldest pending packet in the FIFO first, and then the next oldest packet, and so on), it is within the contemplation of the disclosure to use other forms of selective updaters including those that update all the COE data arrays of pending packets in the FIFO at a same time—in other words, in parallel).
- the COE data array of each respective packet keeps track of how many earlier arrivers still remain in each of the other queues.
- the COE data array replaces totally the per packet time stamps, this is not true for all embodiments.
- the time-out invalidation does not have to be a precise one that invalidates an overly-aged packet at the very microsecond (or nanosecond) it becomes too old. Invalidation due to excess age can be done crudely, say to a resolution of no better than 10 ms to 25 ms.
- the departure detectors (e.g., 231 z - 233 z ) treat packet invalidation due to aging as if the invalidation were a departure and the appropriate CCAP count is decremented in response to invalidation of an in-queue packet.
- Array forming unit 263 responsively generates a COE data array, COEf(1:2) for the packet 233 f having just two COE fields; one indicating that there were 6 earlier arrivers in queue 232 and another indicating that there were 7 earlier arrivers in queue 231 .
- the COE decrementing updater 273 will automatically cause the fields in array COEf(1:2) to indicate that there are still 3 earlier arrivers pending in queue 232 and 5 earlier arrivers pending in queue 231 .
- packet 233 f advances into FIFO departure gate 233 X, its COEf(1:2) array of this example; or COEf(1:2,4:N) array in the general example where N>3; will show how many packets remain in each of respective FIFO's (1:2,4:N) where those packets were pending ahead of packet 233 f .
- This allows the egress arbiter or scheduler 270 (described below) to determine the relative arrival orders of all packets awaiting dispatch in departure gates 231 X through 23 NX.
- the ‘000’ state represents the case where three packets arrived simultaneously at queues 231 - 233 and the same three packets later show up simultaneously at departure gates 231 X- 233 X.
- Each of the three has a COE array reporting that there are zero (0) earlier arrivers in the other two queues, this condition across all three queues being denoted here as ‘000’.
- each COE field is held by a respective hardware counter of predefined bit length where that bit length (e.g., BL 1 of FIG. 3B ) is at least equal to the smallest whole power of 2 whose corresponding value of 2 raised to that power is equal to or greater than the maximum number of shortest packets that can be stored at one time in any of the queues whose CCAP's are being tracked by the given COE field counter.
- the bit length of the COE field counter can of course be bigger.
- each of FIFO's 231 - 23 N can store no more than 15 of the shortest possible packets storable in those FIFO's.
- each COE field counter stops decrementing when it hits a zero count (bottoms out at zero).
- each COE data array has associated with it an index register that holds a unique index value which identifies the packet for which the COE data array is maintaining a tracking of remaining earlier arrivers in the other queues.
- the bit length (e.g., BL 2 of FIG. 3B ) of the index-holding register need not be greater than the maximum number of bits needed for uniquely identifying the maximum number of packets storable in the given queue of the associated COE data array.
- An index table may be included in the network device 200 for storing a start of frame address (SOF) corresponding to each index value where the SOF address indicates where in memory 230 the corresponding packet resides.
- SOF start of frame address
- the arbiter/scheduler 270 can read the current COE data arrays of the ready-to-depart packets (via read line 245 ) and can determine therefrom which of the packets awaiting in the departure gates arrived first, which second and so on relative to one another.
- the oldest of the awaiting packets will have a COE data array filled with zeroes, meaning that no other queue has any packets that arrived earlier and was still valid and pending when the given packet arrived.
- the next oldest of the awaiting packets (again, assuming no ties) will have a COE data array with just one COE field equal to one, that field being the one corresponding to the queue holding the oldest of the awaiting packets.
- the third oldest of the awaiting packets will have a COE data array with either two COE fields each equal to one or one COE counter field set to two, those set fields being the one(s) corresponding to the queues or queue holding the oldest and next oldest of the awaiting packets. And so forth.
- the arbiter/scheduler 270 may then make arbitration and/or scheduling decisions as a function of at least of data read from the COE data arrays (read via bus 245 ) and it may control (via control line 279 ) the packet dispatcher(s) 250 to dispatch selected ones of the awaiting packets from their departure gates ( 231 X- 23 NX) to their respective egress pipes in accordance with dispatch arbitration and/or scheduling decisions made by the arbiter/scheduler 270 in accordance with a predefined arbitration and/or scheduling algorithm that is programmably stored in the arbiter/scheduler 270 .
- the arbitration and/or scheduling algorithm stored in the arbiter/scheduler 270 can be changed on the fly and may be made responsive to, or nonresponsive to, the available COE data array values (readable via bus 245 ) in accordance with the desires of the system designer.
- the arbiter/scheduler 270 receives vacancy feedback data 278 (e.g., buffer slack counts or backpressure flags) from the downstream link partners of the associated egress pipes (not shown, see instead 160 of FIG. 1 ) and the arbiter/scheduler 270 then makes its arbitration and/or scheduling decisions as a function of the vacancy feedback data 278 as well as a function of the read COE data array values.
- the vacancy feedback data 278 is sent from the link partners within packets traveling from the link partners to the network device 200 of the given arbiter/scheduler 270 .
- FIG. 2B shown is part of a third packet switching device 200 ′ similar to that ( 200 ) of FIG. 2A except that the COE data arrays of each packet in each queue have the same number of COE counter fields, denoted as 1 through N.
- This version is generally wasteful of storage space. It is wasteful because there will be one COE counter field in each COE data array (e.g., COEc(1:N) of packet 232 c ) that will always read 0 when its packet (e.g., 232 c ) reaches the departure gate (e.g., 232 x ′) of its FIFO.
- each of array forming units 261 ′- 263 ′- 26 N′ (last not shown) can be structured the same way and each COE decrementing updater 271 ′- 273 ′- 27 N′ (last one not shown) can be structured the same way (e.g., with N parallel output lines) since all departure detections ( 231 z ′- 23 Nz′) from all queues are being responded to with an update operation.
- a given packet e.g., 232 c ′
- the associated COE data array of that packet will still indicate how many packets in the other queues arrived ahead of the given packet and where pending at its time of arrival.
- the given packet e.g., 232 d ′
- the given packet is further back in its respective queue (e.g., 232 ′) in FIG.
- COEd(1:N) its associated Count Of Earlier arrivals
- COEd(1:N) will also indicate a nonzero value representing the number of earlier arrivers and still pending packets in its own same queue (e.g., 232 ′) that arrived ahead of the given packet (e.g., 232 d ′).
- COE decrementing updaters e.g., 271 - 27 N, 271 ′- 27 N′ of FIGS.
- 2A-2B can be structured to handle the case where only one packet can be dispatched at a time through the dispatcher unit 250 / 250 ′ or they can be structured to handle the case where multiple packets can be dispatched at a same time from different departure gates and through the dispatcher unit 250 / 250 ′. If more than one packet can be dispatched at a same time from a same queue, then a more complex accounting circuit may be needed in the COE updaters 271 - 27 N (rather than a simple ⁇ 1) to keep track of how many packets depart during each associated time slot and to debit all the associated COE counter fields accordingly for each associated time slot.
- FIG. 3A shown is part of a packet switching device 300 similar to that ( 200 ) of FIG. 2A . It is to be understood that where practical, reference numbers in the 300 century series are used in FIG. 3A to represent alike elements of FIG. 2A having reference numbers in the 200 century series. As such a description for the general FIFO and other illustrated structures will not be repeated.
- the COE counters of each queue are initially loaded with the CCAP signal from just one corresponding other of the competing queues.
- the linkage of the COE counters each to the CCAP signal from just one other of the competing queues is done on a circular basis.
- the COE's of the first queue 331 are each initially loaded with the current CCAP signal (CCAP(N)) of the Nth queue (not shown) when those COE's are generated and associated with corresponding packets (e.g., 331 a , 331 b , etc.) entering the first queue 331
- the COE's of the second queue 332 are each initially loaded with the current CCAP signal (CCAP( 1 )) of the 1st queue 331 when those COE's (e.g., COEc, COEd, etc.) are generated and associated with corresponding packets (e.g., 332 c , 332 d , etc.) entering the second queue 332
- the COE's of the third queue 333 are each initially loaded with the current CCAP signal (CCAP( 2 )) of the second queue 332 when those COE's (e.g., COEe, COEf, etc.) are generated and associated with corresponding packets (e.g., COE
- COE updaters 371 , 372 , . . . , 37 N are replaced by wires extending to the decrement command lines of all COE counters in the respective FIFO since there is no need for selective updating. There is just one COE counter field per packet in this embodiment.
- FIGS. 2B and 3A represent ends of a wide spectrum of other possible implementations.
- each COE data array of each pending packet keeps track of how many earlier arrivers (still pending ones) remain in each of the N FIFO's.
- each COE data array (actually just one counter) of each pending packet keeps track of how many earlier arrivers (still pending ones) remain in one corresponding other of the N FIFO's.
- updaters 271 , 272 , . . . , 27 N may have programmably activatable inputs which can be individually selectably forced into a zero state rather than receiving the departure detection signal of a corresponding one of departure detectors 231 z through 23 Nz.
- CCAP signal collecting nodes 261 - 26 N (last one not shown) have programmably activatable inputs which can be individually selectably forced into a zero state rather than receiving the current CCAP signal of a corresponding one of counters 231 K- 23 NK (last one not shown).
- Such a programmable structure allows users to determine which of one or more of the other queues, if any, the COE data arrays of a given queue will track.
- the COE's of third queue 333 track the number of earlier arrivals in only the 2nd queue 332 , and so forth, when a given packet (e.g., 333 e ) of the third queue 333 enters the one-at-a-time departure gate (e.g., 333 x ) of its respective queue, the associated COEe of that packet (e.g., 333 e ) will indicate how many packets only in the 2nd queue 332 arrived ahead of the given packet (e.g., 333 e ).
- the associated COEc of that packet (e.g., 332 c ) will indicate how many packets only in the 1st queue 331 arrived ahead of the given packet (e.g., 332 c ), and so on.
- the arbiter/scheduler 370 can read the current COE data arrays of the ready-to-depart packets (via read line 345 ) and can determine therefrom which of the awaiting packets in the departure gates ( 331 X- 33 NX) arrived first, which second and so on.
- a fully circular daisy chained interconnection scheme In a fully circular daisy chained interconnection scheme, assuming there are no ties allowed for arrivals, only one departing packet can have zero packets ahead of it and thus it is the oldest. If a fully circular daisy chained interconnection scheme is not used for the COE generators ( 361 - 36 N) because, let's say generator 361 is not programmably coupled to receive the CCAP(N) signal; then the oldest of the departure-awaiting packets may be deemed to be the one residing in the highest numbered queue where its departure COE value is equal to zero, this meaning (assuming the connection permutation shown in FIG. 3A is used with CCAP-N to 361 broken open) that no other packet among the queues arrived earlier.
- the latter concept is illustrated as follows.
- the third queue (FIFO) 333 ′ is filled with packets denoted as having respective COE counters COE 3 .A through COE 3 .Z (not necessarily 26 such packets).
- the departure awaiting packet of gate 333 X′ has an associated COE 3 .A value equal to K. That means that in the next above FIFO, 332 ′ there are K in-queue packets (counting backwards from departure gate 332 X′, namely, those with COE's represented as COE 2 .A through COE 2 .K) which arrived earlier.
- the Kth packet in queue 332 ′ has an associated COE 2 .K value equal to J.
- 331 ′ there are J in-queue packets (counting backwards from departure gate 331 X′ , namely, those with COE's represented as COE 1 .A through COE 1 .J) which arrived even earlier. If COE 1 .J equals zero, that means that no packets in queue N arrived earlier. On the other hand, if COE 1 .J equals a nonzero value, L then that means the first most L packets of queue N (where N>3) arrived earlier.
- a scheduler may look up (may read from memory) the COE counts of specific packets in this back tracking manner, and the arbiter/scheduler 370 may then determine the specific arrival orders of the packets if such specific information is needed by its arbitration/scheduling algorithm.
- COE 1 .Z of FIG. 3B shown in the associated dashed box is one possible embodiment where the COE count is maintained in a decrementable, zero bottoming out counter (COE 1 .Z Counter) having a first bit length (BL 1 ). additionally, an index value that uniquely identifies the associated packet in queue 331 ′ is maintained in a register (COE 1 .Z Index Reg) having a second bit length (BL 2 ).
- each COE-storing counter e.g., COE 1 .Z Counter
- COE-storing memory field of a given queue has associated with it an index-storing register (e.g., COE 1 .
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Mathematical Physics (AREA)
- Computer Hardware Design (AREA)
- General Physics & Mathematics (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
Description
Claims (21)
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US11/390,754 US8255599B2 (en) | 2006-03-28 | 2006-03-28 | Packets transfer device having data absorbing buffers with elastic buffer capacities |
Publications (2)
Publication Number | Publication Date |
---|---|
US20100054268A1 US20100054268A1 (en) | 2010-03-04 |
US9467307B2 true US9467307B2 (en) | 2016-10-11 |
Family
ID=38662433
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/390,754 Expired - Fee Related US8255599B2 (en) | 2006-03-28 | 2006-03-28 | Packets transfer device having data absorbing buffers with elastic buffer capacities |
US12/200,062 Active 2031-09-17 US9467307B2 (en) | 2006-03-28 | 2008-08-28 | Method of tracking arrival order of packets into plural queues |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/390,754 Expired - Fee Related US8255599B2 (en) | 2006-03-28 | 2006-03-28 | Packets transfer device having data absorbing buffers with elastic buffer capacities |
Country Status (1)
Country | Link |
---|---|
US (2) | US8255599B2 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9760514B1 (en) * | 2016-09-26 | 2017-09-12 | International Business Machines Corporation | Multi-packet processing with ordering rule enforcement |
Families Citing this family (94)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9003292B2 (en) | 2006-07-06 | 2015-04-07 | LiveAction, Inc. | System and method for network topology and flow visualization |
US20080052431A1 (en) * | 2006-08-22 | 2008-02-28 | Freking Ronald E | Method and Apparatus for Enabling Virtual Channels Within A Peripheral Component Interconnect (PCI) Express Bus |
US20090097401A1 (en) * | 2007-10-12 | 2009-04-16 | Wael William Diab | Method and system for configurable data rate thresholds for energy efficient ethernet |
JP4465394B2 (en) * | 2008-04-08 | 2010-05-19 | 富士通株式会社 | Packet relay device, packet relay method, and packet relay program |
US9047421B2 (en) * | 2008-04-30 | 2015-06-02 | Alcatel Lucent | Serial link buffer fill-level compensation using multi-purpose start of protocol data unit timing characters |
US7562168B1 (en) | 2008-05-29 | 2009-07-14 | International Business Machines Corporation | Method of optimizing buffer usage of virtual channels of a physical communication link and apparatuses for performing the same |
US8713697B2 (en) * | 2008-07-09 | 2014-04-29 | Lennox Manufacturing, Inc. | Apparatus and method for storing event information for an HVAC system |
US8184760B2 (en) * | 2008-09-02 | 2012-05-22 | Taiwan Semiconductor Manufacturing Company, Ltd. | Adaptive elastic buffer for communications |
US8527096B2 (en) | 2008-10-24 | 2013-09-03 | Lennox Industries Inc. | Programmable controller and a user interface for same |
US8874815B2 (en) | 2008-10-27 | 2014-10-28 | Lennox Industries, Inc. | Communication protocol system and method for a distributed architecture heating, ventilation and air conditioning network |
US8295981B2 (en) | 2008-10-27 | 2012-10-23 | Lennox Industries Inc. | Device commissioning in a heating, ventilation and air conditioning network |
US8452456B2 (en) | 2008-10-27 | 2013-05-28 | Lennox Industries Inc. | System and method of use for a user interface dashboard of a heating, ventilation and air conditioning network |
US8655491B2 (en) | 2008-10-27 | 2014-02-18 | Lennox Industries Inc. | Alarm and diagnostics system and method for a distributed architecture heating, ventilation and air conditioning network |
US8977794B2 (en) | 2008-10-27 | 2015-03-10 | Lennox Industries, Inc. | Communication protocol system and method for a distributed-architecture heating, ventilation and air conditioning network |
US8560125B2 (en) | 2008-10-27 | 2013-10-15 | Lennox Industries | Communication protocol system and method for a distributed-architecture heating, ventilation and air conditioning network |
US8788100B2 (en) | 2008-10-27 | 2014-07-22 | Lennox Industries Inc. | System and method for zoning a distributed-architecture heating, ventilation and air conditioning network |
US8437877B2 (en) | 2008-10-27 | 2013-05-07 | Lennox Industries Inc. | System recovery in a heating, ventilation and air conditioning network |
US8994539B2 (en) | 2008-10-27 | 2015-03-31 | Lennox Industries, Inc. | Alarm and diagnostics system and method for a distributed-architecture heating, ventilation and air conditioning network |
US8452906B2 (en) | 2008-10-27 | 2013-05-28 | Lennox Industries, Inc. | Communication protocol system and method for a distributed-architecture heating, ventilation and air conditioning network |
US9651925B2 (en) | 2008-10-27 | 2017-05-16 | Lennox Industries Inc. | System and method for zoning a distributed-architecture heating, ventilation and air conditioning network |
US8762666B2 (en) | 2008-10-27 | 2014-06-24 | Lennox Industries, Inc. | Backup and restoration of operation control data in a heating, ventilation and air conditioning network |
US8600558B2 (en) | 2008-10-27 | 2013-12-03 | Lennox Industries Inc. | System recovery in a heating, ventilation and air conditioning network |
US8600559B2 (en) | 2008-10-27 | 2013-12-03 | Lennox Industries Inc. | Method of controlling equipment in a heating, ventilation and air conditioning network |
US8463442B2 (en) | 2008-10-27 | 2013-06-11 | Lennox Industries, Inc. | Alarm and diagnostics system and method for a distributed architecture heating, ventilation and air conditioning network |
US8892797B2 (en) | 2008-10-27 | 2014-11-18 | Lennox Industries Inc. | Communication protocol system and method for a distributed-architecture heating, ventilation and air conditioning network |
US8855825B2 (en) | 2008-10-27 | 2014-10-07 | Lennox Industries Inc. | Device abstraction system and method for a distributed-architecture heating, ventilation and air conditioning system |
US8661165B2 (en) | 2008-10-27 | 2014-02-25 | Lennox Industries, Inc. | Device abstraction system and method for a distributed architecture heating, ventilation and air conditioning system |
US8655490B2 (en) | 2008-10-27 | 2014-02-18 | Lennox Industries, Inc. | System and method of use for a user interface dashboard of a heating, ventilation and air conditioning network |
US8564400B2 (en) | 2008-10-27 | 2013-10-22 | Lennox Industries, Inc. | Communication protocol system and method for a distributed-architecture heating, ventilation and air conditioning network |
US8798796B2 (en) | 2008-10-27 | 2014-08-05 | Lennox Industries Inc. | General control techniques in a heating, ventilation and air conditioning network |
US8774210B2 (en) | 2008-10-27 | 2014-07-08 | Lennox Industries, Inc. | Communication protocol system and method for a distributed-architecture heating, ventilation and air conditioning network |
US8437878B2 (en) | 2008-10-27 | 2013-05-07 | Lennox Industries Inc. | Alarm and diagnostics system and method for a distributed architecture heating, ventilation and air conditioning network |
US8442693B2 (en) | 2008-10-27 | 2013-05-14 | Lennox Industries, Inc. | System and method of use for a user interface dashboard of a heating, ventilation and air conditioning network |
US8543243B2 (en) | 2008-10-27 | 2013-09-24 | Lennox Industries, Inc. | System and method of use for a user interface dashboard of a heating, ventilation and air conditioning network |
US8615326B2 (en) | 2008-10-27 | 2013-12-24 | Lennox Industries Inc. | System and method of use for a user interface dashboard of a heating, ventilation and air conditioning network |
US9678486B2 (en) | 2008-10-27 | 2017-06-13 | Lennox Industries Inc. | Device abstraction system and method for a distributed-architecture heating, ventilation and air conditioning system |
US9268345B2 (en) | 2008-10-27 | 2016-02-23 | Lennox Industries Inc. | System and method of use for a user interface dashboard of a heating, ventilation and air conditioning network |
US8802981B2 (en) | 2008-10-27 | 2014-08-12 | Lennox Industries Inc. | Flush wall mount thermostat and in-set mounting plate for a heating, ventilation and air conditioning system |
US9632490B2 (en) | 2008-10-27 | 2017-04-25 | Lennox Industries Inc. | System and method for zoning a distributed architecture heating, ventilation and air conditioning network |
US8744629B2 (en) | 2008-10-27 | 2014-06-03 | Lennox Industries Inc. | System and method of use for a user interface dashboard of a heating, ventilation and air conditioning network |
US9325517B2 (en) | 2008-10-27 | 2016-04-26 | Lennox Industries Inc. | Device abstraction system and method for a distributed-architecture heating, ventilation and air conditioning system |
US8463443B2 (en) | 2008-10-27 | 2013-06-11 | Lennox Industries, Inc. | Memory recovery scheme and data structure in a heating, ventilation and air conditioning network |
US8725298B2 (en) | 2008-10-27 | 2014-05-13 | Lennox Industries, Inc. | Alarm and diagnostics system and method for a distributed architecture heating, ventilation and conditioning network |
US8433446B2 (en) | 2008-10-27 | 2013-04-30 | Lennox Industries, Inc. | Alarm and diagnostics system and method for a distributed-architecture heating, ventilation and air conditioning network |
US9432208B2 (en) | 2008-10-27 | 2016-08-30 | Lennox Industries Inc. | Device abstraction system and method for a distributed architecture heating, ventilation and air conditioning system |
US8694164B2 (en) | 2008-10-27 | 2014-04-08 | Lennox Industries, Inc. | Interactive user guidance interface for a heating, ventilation and air conditioning system |
US8548630B2 (en) | 2008-10-27 | 2013-10-01 | Lennox Industries, Inc. | Alarm and diagnostics system and method for a distributed-architecture heating, ventilation and air conditioning network |
US20100254388A1 (en) * | 2009-04-04 | 2010-10-07 | Oracle International Corporation | Method and system for applying expressions on message payloads for a resequencer |
US9124448B2 (en) * | 2009-04-04 | 2015-09-01 | Oracle International Corporation | Method and system for implementing a best efforts resequencer |
US20100312928A1 (en) * | 2009-06-09 | 2010-12-09 | Brownell Paul V | System and method for operating a communication link |
US20110103395A1 (en) * | 2009-11-03 | 2011-05-05 | Qualcomm Incorporated | Computing the burst size for a high speed packet data networks with multiple queues |
US20120281703A1 (en) * | 2009-12-04 | 2012-11-08 | Napatech A/S | Apparatus, an assembly and a method of operating a plurality of analyzing means reading and ordering data packets |
EP2547048A1 (en) * | 2010-03-11 | 2013-01-16 | Fujitsu Limited | Data block read-out control device |
US9215486B2 (en) | 2010-08-13 | 2015-12-15 | Simon Fraser University | System and method for multiplexing of variable bit-rate video streams in mobile video systems |
JP5682391B2 (en) * | 2011-03-22 | 2015-03-11 | 富士通株式会社 | Data transfer apparatus, parallel computer system, and data transfer apparatus control method |
US10838886B2 (en) * | 2011-04-19 | 2020-11-17 | Micron Technology, Inc. | Channel depth adjustment in memory systems |
JP6013711B2 (en) * | 2011-09-01 | 2016-10-25 | ラピスセミコンダクタ株式会社 | Semiconductor integrated circuit and semiconductor integrated circuit debugging method |
US10409445B2 (en) | 2012-01-09 | 2019-09-10 | Activevideo Networks, Inc. | Rendering of an interactive lean-backward user interface on a television |
CN102611621B (en) * | 2012-02-24 | 2015-09-23 | 华为技术有限公司 | The processing method of cell cascading and equipment |
US9800945B2 (en) | 2012-04-03 | 2017-10-24 | Activevideo Networks, Inc. | Class-based intelligent multiplexing over unmanaged networks |
US9602433B2 (en) | 2012-07-26 | 2017-03-21 | Qualcomm Incorporated | Systems and methods for sharing a serial communication port between a plurality of communication channels |
US8855127B2 (en) * | 2012-10-02 | 2014-10-07 | Lsi Corporation | Method and system for intelligent deep packet buffering |
US9189433B2 (en) | 2012-12-18 | 2015-11-17 | International Business Machines Corporation | Tracking a relative arrival order of events being stored in multiple queues using a counter |
US9009370B2 (en) * | 2013-03-04 | 2015-04-14 | Lsi Corporation | Intelligent data buffering between interfaces |
WO2014145921A1 (en) | 2013-03-15 | 2014-09-18 | Activevideo Networks, Inc. | A multiple-mode system and method for providing user selectable video content |
EP3005712A1 (en) | 2013-06-06 | 2016-04-13 | ActiveVideo Networks, Inc. | Overlay rendering of user interface onto source video |
US9626319B2 (en) | 2013-08-23 | 2017-04-18 | Lenovo Enterprise Solutions (Singapore) Pte. Ltd. | Allocating lanes in a peripheral component interconnect express (‘PCIe’) bus |
FR3011704A1 (en) * | 2013-10-07 | 2015-04-10 | Orange | METHOD FOR IMPLEMENTING A COMMUNICATION SESSION BETWEEN A PLURALITY OF TERMINALS |
US9137285B2 (en) * | 2013-10-21 | 2015-09-15 | Broadcom Corporation | Adaptive audio video (AV) stream processing |
US9426215B2 (en) * | 2014-04-08 | 2016-08-23 | Aol Inc. | Determining load state of remote systems using delay and packet loss rate |
US9788029B2 (en) | 2014-04-25 | 2017-10-10 | Activevideo Networks, Inc. | Intelligent multiplexing using class-based, multi-dimensioned decision logic for managed networks |
US10313248B2 (en) | 2014-06-18 | 2019-06-04 | Adobe Inc. | Data flow node validation |
US9548941B2 (en) * | 2014-06-18 | 2017-01-17 | Adobe Systems Incorporated | Data flow node provisioning |
US9258256B2 (en) * | 2014-07-01 | 2016-02-09 | Netronome Systems, Inc. | Inverse PCP flow remapping for PFC pause frame generation |
US9515946B2 (en) * | 2014-07-01 | 2016-12-06 | Netronome Systems, Inc. | High-speed dequeuing of buffer IDS in frame storing system |
US9575822B2 (en) * | 2014-08-01 | 2017-02-21 | Globalfoundries Inc. | Tracking a relative arrival order of events being stored in multiple queues using a counter using most significant bit values |
US9824058B2 (en) * | 2014-11-14 | 2017-11-21 | Cavium, Inc. | Bypass FIFO for multiple virtual channels |
US10264293B2 (en) | 2014-12-24 | 2019-04-16 | Activevideo Networks, Inc. | Systems and methods for interleaving video streams on a client device |
US10523985B2 (en) | 2014-12-24 | 2019-12-31 | Activevideo Networks, Inc. | Managing deep and shallow buffers in a thin-client device of a digital media distribution network |
US10078473B2 (en) * | 2015-03-23 | 2018-09-18 | Netapp, Inc. | Resource allocation in networked storage systems |
US10834672B2 (en) * | 2015-09-23 | 2020-11-10 | International Business Machines Corporation | Power management of network links |
WO2017096377A1 (en) * | 2015-12-04 | 2017-06-08 | Activevideo Networks, Inc. | Managing deep and shallow buffers in a thin-client device of a digital media distribution network |
JP7087419B2 (en) * | 2018-02-02 | 2022-06-21 | 富士通株式会社 | Data receiving device, data transmission / reception system, and control method of data transmission / reception system |
WO2019214801A1 (en) * | 2018-05-07 | 2019-11-14 | Huawei Technologies Co., Ltd. | Memory device for a high bandwidth high capacity switch |
US10409743B1 (en) * | 2018-06-29 | 2019-09-10 | Xilinx, Inc. | Transparent port aggregation in multi-chip transport protocols |
US11055156B2 (en) * | 2019-08-20 | 2021-07-06 | International Business Machines Corporation | Processing of a message stream |
US20210058334A1 (en) * | 2019-08-21 | 2021-02-25 | Intel Corporation | Timestamp-based fairness egress from ingress queues |
US11349771B2 (en) * | 2020-04-30 | 2022-05-31 | Hewlett Packard Enterprise Development Lp | Method and system for enhanced queue management in a switch |
US11431629B2 (en) * | 2020-08-12 | 2022-08-30 | Micron Technology, Inc. | Data packet management |
US11743270B2 (en) * | 2021-04-16 | 2023-08-29 | Visa International Service Association | Method, system, and computer program product for protocol parsing for network security |
CN115687221A (en) * | 2021-07-22 | 2023-02-03 | 智原微电子(苏州)有限公司 | Transaction layer circuit for high-speed peripheral component interconnection and operation method thereof |
CN114185822B (en) * | 2021-11-05 | 2024-05-24 | 北京智芯微电子科技有限公司 | Multi-pointer elastic buffer, method for adding and deleting control characters and storage medium |
US12068971B2 (en) * | 2022-02-25 | 2024-08-20 | Google Llc | Robust age-saturation mechanism for age-based arbitration in packet networks |
US20230289297A1 (en) * | 2022-03-14 | 2023-09-14 | Samsung Electronics Co., Ltd. | Systems and methods for managing memory utilization |
Citations (17)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5892979A (en) * | 1994-07-20 | 1999-04-06 | Fujitsu Limited | Queue control apparatus including memory to save data received when capacity of queue is less than a predetermined threshold |
US5996019A (en) * | 1995-07-19 | 1999-11-30 | Fujitsu Network Communications, Inc. | Network link access scheduling using a plurality of prioritized lists containing queue identifiers |
US20010033581A1 (en) * | 2000-03-22 | 2001-10-25 | Kenichi Kawarai | Packet switch, scheduling device, drop control circuit, multicast control circuit and QoS control device |
US6526495B1 (en) * | 2000-03-22 | 2003-02-25 | Cypress Semiconductor Corp. | Multiport FIFO with programmable width and depth |
US6556572B1 (en) * | 1998-03-26 | 2003-04-29 | Oki Electric Industry Co., Ltd. | Scheduler for adjusting cell forwarding dependent upon traffic and delay |
US6603772B1 (en) * | 1999-03-31 | 2003-08-05 | Cisco Technology, Inc. | Multicast routing with multicast virtual output queues and shortest queue first allocation |
US6738371B1 (en) * | 1999-09-28 | 2004-05-18 | Ericsson Inc. | Ingress data queue management in a packet data router |
US20050053078A1 (en) * | 2003-07-23 | 2005-03-10 | International Business Machines Corporation | Algorithm and system for selecting acknowledgments from an array of collapsed VOQ's |
US20050083920A1 (en) * | 2003-10-21 | 2005-04-21 | Alcatel | Scalable and QoS aware flow control |
US20050182887A1 (en) * | 2004-02-17 | 2005-08-18 | Toshimi Sakurai | PCI-express to PCI/PCI X translator |
US20050259651A1 (en) * | 2004-05-20 | 2005-11-24 | Kabushiki Kaisha Toshiba | Data processing apparatus and flow control method |
US20050281259A1 (en) * | 2004-06-19 | 2005-12-22 | Kevin Mitchell | Method of generating a monitoring datagram |
US6985451B1 (en) * | 1997-10-14 | 2006-01-10 | Alvarion Israel (2003) Ltd. | Method and apparatus for baseband transmission between a top floor unit and an outdoor unit in a terminal for a wireless metropolitan area network |
US7221678B1 (en) * | 2001-10-01 | 2007-05-22 | Advanced Micro Devices, Inc. | Method and apparatus for routing packets |
US20070201360A1 (en) * | 2000-07-25 | 2007-08-30 | Ming Hung | Network switch |
US7400629B2 (en) * | 2002-12-19 | 2008-07-15 | International Business Machines Corporation | CAM based system and method for re-sequencing data packets |
US8098674B2 (en) * | 2005-10-03 | 2012-01-17 | Fujitsu Semiconductor Limited | Queue selection method and scheduling device |
Family Cites Families (33)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7380092B2 (en) * | 2002-06-28 | 2008-05-27 | Rambus Inc. | Memory device and system having a variable depth write buffer and preload method |
US5842224A (en) * | 1989-06-16 | 1998-11-24 | Fenner; Peter R. | Method and apparatus for source filtering data packets between networks of differing media |
EP0706297A1 (en) * | 1994-10-07 | 1996-04-10 | International Business Machines Corporation | Method for operating traffic congestion control in a data communication network and system for implementing said method |
GB9618137D0 (en) * | 1996-08-30 | 1996-10-09 | Sgs Thomson Microelectronics | Improvements in or relating to an ATM switch |
US6285679B1 (en) * | 1997-08-22 | 2001-09-04 | Avici Systems, Inc. | Methods and apparatus for event-driven routing |
US20030017497A1 (en) * | 1998-02-04 | 2003-01-23 | Thomas Kieber-Emmons | Peptide mimotopes of carbohydrate antigens |
US6499079B1 (en) * | 1998-11-23 | 2002-12-24 | Advanced Micro Devices, Inc. | Subordinate bridge structure for a point-to-point computer interconnection bus |
US6570877B1 (en) * | 1999-04-07 | 2003-05-27 | Cisco Technology, Inc. | Search engine for forwarding table content addressable memory |
US6519225B1 (en) * | 1999-05-14 | 2003-02-11 | Nortel Networks Limited | Backpressure mechanism for a network device |
US6661788B2 (en) * | 1999-05-14 | 2003-12-09 | Nortel Networks Limited | Multicast scheduling for a network device |
US6850490B1 (en) * | 1999-10-06 | 2005-02-01 | Enterasys Networks, Inc. | Hierarchical output-queued packet-buffering system and method |
US6954424B2 (en) * | 2000-02-24 | 2005-10-11 | Zarlink Semiconductor V.N., Inc. | Credit-based pacing scheme for heterogeneous speed frame forwarding |
US6898182B1 (en) * | 2000-07-21 | 2005-05-24 | Arris International, Inc | Congestion control in a network device having a buffer circuit |
US6888831B1 (en) * | 2000-09-28 | 2005-05-03 | Western Digital Ventures, Inc. | Distributed resource reservation system for establishing a path through a multi-dimensional computer network to support isochronous data |
JP4691804B2 (en) * | 2001-03-02 | 2011-06-01 | ソニー株式会社 | Wireless transmission apparatus and wireless transmission method |
US6859866B2 (en) * | 2001-10-01 | 2005-02-22 | International Business Machines Corporation | Synchronizing processing of commands invoked against duplexed coupling facility structures |
US7586909B1 (en) * | 2002-03-06 | 2009-09-08 | Agere Systems Inc. | Striping algorithm for switching fabric |
US7319695B1 (en) * | 2002-03-06 | 2008-01-15 | Agere Systems Inc. | Deficit-based striping algorithm |
US7936672B2 (en) * | 2002-10-09 | 2011-05-03 | Juniper Networks, Inc. | System and method for buffer management in a packet-based network |
US7451254B2 (en) * | 2003-07-31 | 2008-11-11 | Hewlett-Packard Development Company, L.P. | System and method for adaptive buffer allocation in a memory device interface |
US7103823B2 (en) * | 2003-08-05 | 2006-09-05 | Newisys, Inc. | Communication between multi-processor clusters of multi-cluster computer systems |
US7269700B2 (en) * | 2004-07-26 | 2007-09-11 | Integrated Device Technology, Inc. | Status bus accessing only available quadrants during loop mode operation in a multi-queue first-in first-out memory system |
US8230174B2 (en) * | 2004-07-26 | 2012-07-24 | Integrated Device Technology, Inc. | Multi-queue address generator for start and end addresses in a multi-queue first-in first-out memory system |
US7257687B2 (en) * | 2004-07-26 | 2007-08-14 | Integrated Device Technology, Inc. | Synchronization of active flag and status bus flags in a multi-queue first-in first-out memory system |
US7099231B2 (en) * | 2004-07-26 | 2006-08-29 | Integrated Device Technology, Inc. | Interleaving memory blocks to relieve timing bottleneck in a multi-queue first-in first-out memory system |
US7805552B2 (en) * | 2004-07-26 | 2010-09-28 | Integrated Device Technology, Inc. | Partial packet write and write data filtering in a multi-queue first-in first-out memory system |
US7154327B2 (en) | 2004-07-26 | 2006-12-26 | Integrated Device Technology, Inc. | Self-timed multiple blanking for noise suppression during flag generation in a multi-queue first-in first-out memory system |
US7870310B2 (en) * | 2004-07-26 | 2011-01-11 | Integrated Device Technology, Inc. | Multiple counters to relieve flag restriction in a multi-queue first-in first-out memory system |
US7523232B2 (en) * | 2004-07-26 | 2009-04-21 | Integrated Device Technology, Inc. | Mark/re-read and mark/re-write operations in a multi-queue first-in first-out memory system |
US7257655B1 (en) * | 2004-10-13 | 2007-08-14 | Altera Corporation | Embedded PCI-Express implementation |
US7424566B2 (en) * | 2005-11-16 | 2008-09-09 | Sun Microsystems, Inc. | Method, system, and apparatus for dynamic buffer space allocation |
US7660270B2 (en) * | 2006-11-08 | 2010-02-09 | Sicortex, Inc. | Computer system and method using efficient module and backplane tiling to interconnect computer nodes via a Kautz-like digraph |
US7533197B2 (en) * | 2006-11-08 | 2009-05-12 | Sicortex, Inc. | System and method for remote direct memory access without page locking by the operating system |
-
2006
- 2006-03-28 US US11/390,754 patent/US8255599B2/en not_active Expired - Fee Related
-
2008
- 2008-08-28 US US12/200,062 patent/US9467307B2/en active Active
Patent Citations (17)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5892979A (en) * | 1994-07-20 | 1999-04-06 | Fujitsu Limited | Queue control apparatus including memory to save data received when capacity of queue is less than a predetermined threshold |
US5996019A (en) * | 1995-07-19 | 1999-11-30 | Fujitsu Network Communications, Inc. | Network link access scheduling using a plurality of prioritized lists containing queue identifiers |
US6985451B1 (en) * | 1997-10-14 | 2006-01-10 | Alvarion Israel (2003) Ltd. | Method and apparatus for baseband transmission between a top floor unit and an outdoor unit in a terminal for a wireless metropolitan area network |
US6556572B1 (en) * | 1998-03-26 | 2003-04-29 | Oki Electric Industry Co., Ltd. | Scheduler for adjusting cell forwarding dependent upon traffic and delay |
US6603772B1 (en) * | 1999-03-31 | 2003-08-05 | Cisco Technology, Inc. | Multicast routing with multicast virtual output queues and shortest queue first allocation |
US6738371B1 (en) * | 1999-09-28 | 2004-05-18 | Ericsson Inc. | Ingress data queue management in a packet data router |
US20010033581A1 (en) * | 2000-03-22 | 2001-10-25 | Kenichi Kawarai | Packet switch, scheduling device, drop control circuit, multicast control circuit and QoS control device |
US6526495B1 (en) * | 2000-03-22 | 2003-02-25 | Cypress Semiconductor Corp. | Multiport FIFO with programmable width and depth |
US20070201360A1 (en) * | 2000-07-25 | 2007-08-30 | Ming Hung | Network switch |
US7221678B1 (en) * | 2001-10-01 | 2007-05-22 | Advanced Micro Devices, Inc. | Method and apparatus for routing packets |
US7400629B2 (en) * | 2002-12-19 | 2008-07-15 | International Business Machines Corporation | CAM based system and method for re-sequencing data packets |
US20050053078A1 (en) * | 2003-07-23 | 2005-03-10 | International Business Machines Corporation | Algorithm and system for selecting acknowledgments from an array of collapsed VOQ's |
US20050083920A1 (en) * | 2003-10-21 | 2005-04-21 | Alcatel | Scalable and QoS aware flow control |
US20050182887A1 (en) * | 2004-02-17 | 2005-08-18 | Toshimi Sakurai | PCI-express to PCI/PCI X translator |
US20050259651A1 (en) * | 2004-05-20 | 2005-11-24 | Kabushiki Kaisha Toshiba | Data processing apparatus and flow control method |
US20050281259A1 (en) * | 2004-06-19 | 2005-12-22 | Kevin Mitchell | Method of generating a monitoring datagram |
US8098674B2 (en) * | 2005-10-03 | 2012-01-17 | Fujitsu Semiconductor Limited | Queue selection method and scheduling device |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9760514B1 (en) * | 2016-09-26 | 2017-09-12 | International Business Machines Corporation | Multi-packet processing with ordering rule enforcement |
Also Published As
Publication number | Publication date |
---|---|
US20100054268A1 (en) | 2010-03-04 |
US8255599B2 (en) | 2012-08-28 |
US20070260782A1 (en) | 2007-11-08 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9467307B2 (en) | Method of tracking arrival order of packets into plural queues | |
US8437369B2 (en) | Packets transfer device that intelligently accounts for variable egress channel widths when scheduling use of dispatch bus by egressing packet streams | |
US20220239587A1 (en) | Algorithms for use of load information from neighboring nodes in adaptive routing | |
US9977745B2 (en) | Flow control through packet router | |
US6526451B2 (en) | Method and network device for creating circular queue structures in shared memory | |
US7609636B1 (en) | System and method for infiniband receive flow control with combined buffering of virtual lanes and queue pairs | |
EP1779607B1 (en) | Network interconnect crosspoint switching architecture and method | |
US8085801B2 (en) | Resource arbitration | |
US8462628B2 (en) | Method of improving over protocol-required scheduling tables while maintaining same | |
US7620693B1 (en) | System and method for tracking infiniband RDMA read responses | |
US7227841B2 (en) | Packet input thresholding for resource distribution in a network switch | |
US7852867B2 (en) | Integrated memory for storing egressing packet data, replay data and to-be egressed data | |
US7337253B2 (en) | Method and system of routing network-based data using frame address notification | |
EP0990990A2 (en) | Flow control in a fifo memory | |
US7327749B1 (en) | Combined buffering of infiniband virtual lanes and queue pairs | |
US20210344616A1 (en) | Use of stashing buffers to improve the efficiency of crossbar switches | |
JP2009516478A (en) | Method and system for reducing interconnect latency | |
EP1014649A2 (en) | Method and system of data transfer control | |
US20020062415A1 (en) | Slotted memory access method | |
US10387355B2 (en) | NoC interconnect with linearly-tunable QoS guarantees for real-time isolation | |
US7486689B1 (en) | System and method for mapping InfiniBand communications to an external port, with combined buffering of virtual lanes and queue pairs | |
US7342934B1 (en) | System and method for interleaving infiniband sends and RDMA read responses in a single receive queue | |
US7773591B2 (en) | Integrated memory for storing egressing packet data, replay data and to-be egressed data | |
US10289598B2 (en) | Non-blocking network | |
US10185606B2 (en) | Scalable autonomic message-transport with synchronization |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: INTEGRATED DEVICE TECHNOLOGY, INC.,CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:DIVIVIER, ROBERT;REEL/FRAME:021696/0632 Effective date: 20081013 Owner name: INTEGRATED DEVICE TECHNOLOGY, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:DIVIVIER, ROBERT;REEL/FRAME:021696/0632 Effective date: 20081013 |
|
AS | Assignment |
Owner name: INTEGRATED DEVICE TECHNOLOGY, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:PMC-SIERRA US, INC.;REEL/FRAME:030832/0604 Effective date: 20130712 |
|
AS | Assignment |
Owner name: PMC-SIERRA US, INC., CALIFORNIA Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE CONVEYING AND RECEIVING PARTIES PREVIOUSLY RECORDED ON REEL 030832 FRAME 0604. ASSIGNOR(S) HEREBY CONFIRMS THE CONVEYING PARTY DATA: INTEGRATED DEVICE TECHNOLOGY, INC. TECEIVING: PMC-SIERRA US, INC;ASSIGNOR:INTEGRATED DEVICE TECHNOLOGY, INC.;REEL/FRAME:030919/0040 Effective date: 20130712 |
|
AS | Assignment |
Owner name: BANK OF AMERICA, N.A., NORTH CAROLINA Free format text: SECURITY INTEREST IN PATENTS;ASSIGNORS:PMC-SIERRA, INC.;PMC-SIERRA US, INC.;WINTEGRA, INC.;REEL/FRAME:030947/0710 Effective date: 20130802 |
|
AS | Assignment |
Owner name: PMC-SIERRA, INC., CALIFORNIA Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:BANK OF AMERICA, N.A.;REEL/FRAME:037675/0129 Effective date: 20160115 Owner name: PMC-SIERRA US, INC., CALIFORNIA Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:BANK OF AMERICA, N.A.;REEL/FRAME:037675/0129 Effective date: 20160115 Owner name: WINTEGRA, INC., CALIFORNIA Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:BANK OF AMERICA, N.A.;REEL/FRAME:037675/0129 Effective date: 20160115 |
|
AS | Assignment |
Owner name: MORGAN STANLEY SENIOR FUNDING, INC., NEW YORK Free format text: PATENT SECURITY AGREEMENT;ASSIGNORS:MICROSEMI STORAGE SOLUTIONS, INC. (F/K/A PMC-SIERRA, INC.);MICROSEMI STORAGE SOLUTIONS (U.S.), INC. (F/K/A PMC-SIERRA US, INC.);REEL/FRAME:037689/0719 Effective date: 20160115 |
|
AS | Assignment |
Owner name: MICROSEMI STORAGE SOLUTIONS (U.S.), INC., DELAWARE Free format text: CHANGE OF NAME;ASSIGNOR:PMC-SIERRA US, INC.;REEL/FRAME:037961/0519 Effective date: 20160115 |
|
AS | Assignment |
Owner name: MICROSEMI STORAGE SOLUTIONS (US), INC., CALIFORNIA Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNEE NAME AND ADDRESS PREVIOUSLY RECORDED AT REEL: 037961 FRAME: 0519. ASSIGNOR(S) HEREBY CONFIRMS THE CHANGE OF NAME;ASSIGNOR:PMC-SIERRA US, INC.;REEL/FRAME:038102/0874 Effective date: 20160115 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
AS | Assignment |
Owner name: MICROSEMI STORAGE SOLUTIONS, INC., CALIFORNIA Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:046251/0271 Effective date: 20180529 Owner name: MICROSEMI STORAGE SOLUTIONS (U.S.), INC., CALIFORN Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:046251/0271 Effective date: 20180529 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |
|
FEPP | Fee payment procedure |
Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |