JP2009516478A - Method and system for reducing interconnect latency - Google Patents

Abstract

  Methods and systems for reducing arbitration latency use speculative transmission (STX) in combination with planned arbitration of the routing structure without prior arbitration. Packets are sent from the origin location to the routing structure via planned arbitration and also via speculative arbitration, and the pre-reserved output of the routing structure is sent to the speculatively transmitted packet. Assign them to avoid collisions.

Description

Priority claim

  This application is a US patent provisional application 60 / 736,779 entitled “METHOD AND SYSTEM TO REDUCE INTERCONNECT LATENCY” filed on November 14, 2005. Claim priority by issue.

  The present invention relates to the field of packet switching, and in particular to the field of packet switching architecture with input queuing, which is particularly applicable to computer interconnect networks.

  This invention was made with support from the US government under contract number W-7405-ENG48 awarded by DOE / NNSA. The US government has certain rights in this invention.

  Advances in transmission technology and parallel processing of communications and computations continue to increase the bandwidth available for transferring information data. For example, advances such as wavelength division multiplexing (WDM), high density WDM (DWDM), etc., greatly increase the available bandwidth by multiplexing multiple channels on a single fiber. Individual channels operate at optical carrier (OC-x) speeds OC-48 (2.5 Gb / sec), OC-192 (10 Gb / sec), or OC-768 (40 Gb / sec). Using the latest DWDM technology, data exceeding 5 terabits / second can be carried by a single fiber.

  At the same time, the gap between the ever-increasing speed offered by such advances and the speed at which available switches can be switched is widening. Optical switches offer theoretical advantages such as routing through free sections, minimal signal attenuation over long distances, and elimination of optical to electrical and electrical to optical domain conversions, etc. However, current all-optical switches are relatively slow or very expensive. Furthermore, optical storage of information is very cumbersome and often impractical. Until this shortcoming of optical switching is overcome, electrical switches will continue to play a major role in packet switching schemes.

  Backplane switches, more generally routing structures, are commonly used for board interconnection. In networking systems, these boards are called line cards, and in computing and storage, they are often called adapters or blades. A wide range of systems including telecommunications switches, multiservice provisioning platforms, add / drop multiplexers, digital cross-connects, storage switches, routers, large enterprise switches, embedded platforms, multiprocessor systems and blade servers Connect the board using.

  When information data is transmitted from a source to a destination via an interconnection system, information is often first divided into a plurality of data packets. In general, each data packet includes a header part, a payload part, and a tail part, and is further divided into smaller units. Data packets are switched through the routing structure simultaneously with other data packets originating from other sources. Many current packet switch systems, including those used for interconnecting parallel computers, Internet routers, S (t) AN networks, asymmetric transfer mode (ATM) networks, and optical networks in particular, use input queuing configurations. Queues sorted for each line card output (such a configuration, often referred to as virtual output queuing (VOQ) eliminates the head-of-line blocking inherent in the use of FIFO queues), crossbar routing structures, and A centralized scheduler (e.g., arbiter or arbitrator) that allocates switching resources and arbitrates between queues.

FIG. 1 shows a conventional switching configuration using a VOQ architecture. In the configuration of FIG. 1, data packets (eg, cells, frames, or datagrams) are received from each of the _N data links 2a ₁ -2a _N of the individual line card 102. Data packets are sent to one of the N buffers of N buffer groups 121 via multiplexer 105a for each output 3 _{1 to} 3 _{N of} routing structure 106 (shown as an N × N crossbar). To be sorted. That is, each input line card 102 maintains a separate queue for each output 3 _{1 to} 3 _N , resulting in N ² VOQs on the input side of the routing structure 106. Arbiter 107 is provided to manage competition between data packets for the same output of router structure 106 and to match the input to the output. Arbiter 107 communicates with each line card along control paths 108, 109 and provides a switching configuration to routing structure 106 along control path 112. The arbiter 107 is typically physically located near the routing structure 106.

The arbiter 107 performs processing including assigning an input port and an output port to a packet waiting in each buffer of the buffer group 121 so that no collision occurs. These processes include allocation and arbitration. Allocation is performed so that at most one packet is selected from each buffer group 121 with respect to the output to one output resource at most, the inputs 2b _{1 to} 2bN and the outputs 3 _{1 to} 3 _N of the routing structure 106 Determine the matching between. Arbitration resolves multiple requests for each single output resource 3 _{1-3 N} and assigns one of these outputs to one of a group of requesters. In the conventional configuration of FIG. 1, the arbiter 107 receives a switch access request from the line card 102 on the control path 108. The arbiter 107 calculates a match based on the received request and an appropriate matching algorithm, and in each of a number of time slots, determines which of the inputs 2b ₁ -2bN is allowed to forward the data packet to which output. decide. Each line card 102 that has gained access (ie, has been granted access) is permitted to transmit a unit of data, such as a packet, along a control path 109 to a specified output in a particular time slot or switching cycle. A control message is transmitted to inform the line card 102 of the completion. During that time slot, the arbiter 107 sends the calculated switching configuration to the routing structure 106, and each winning line card 102 passes a unit of data packet from the queue in its buffer group 121 via the demultiplexer 105b. The data unit is transmitted to the routing structure 106 along the corresponding input of any of 2b ₁ to 2b _N. Each data packet is then transmitted via routing structure 106 to the requested one of outputs 3 _{1 to} 3 _N along the path configured by arbiter 107.

  As can be seen from the drawing, such an input queuing system includes a control path including a flow of control information from the line card to the arbiter (eg, request) and a flow of control information back to the line card (eg, permission), and an input. There are two basic communication paths, a data path that includes the flow of data packets from the line card through the crossbar to the output line card.

Note that the conventional packet switching configuration shown in FIG. 1 shows a routing structure 6 having only one-way communication paths, but this general concept also includes bidirectional data and control paths. For example, each of the data links 2b _{1 to} 2b _N and the individual outputs 3 _{1 to} 3 _N shown in FIG. 1 includes an input buffer (egress buffer) and an output buffer (egress buffer). Can be represented as a bi-directional link. In this case, the line card 102 can be regarded as both a source location and a destination location for transmitting data packets using a routing structure. Similarly, requests, authorization, and links _2a 1 to 2A region _N, can be represented as a two-way link.

  With increasing capacity, the physical size of packet switches has also increased. At the same time, as the line rate increases, the packet size remains approximately constant, so the duration of a single packet or cell (T = L / B, where L is the packet length in bits, B is the link speed in bits / second). These trends directly imply that the round trip (RT) in the switch, measured in packet time, jumps considerably. For input queuing switches with centralized arbitration, the minimum transit latency is (1) the wait time to submit a request to the arbiter and wait until the corresponding permission arrives (flight to and from the arbiter Time (including time-of-flight) and time for arbitration), and (2) serial time / deserialization (SerDes), transmission, and time-of-flight latency to transmit packets through the switch This effect is doubled. Roughly speaking, these latencies are at least 2 · (RT) packet times, which is twice that of similar switches (except those with buffered routing structures).

  Since these wait times have recently become a problem, they have received little attention. A practical preferred solution is to place a board such as a line card with input queues (generally organized in a VOQ style) physically close to the routing structure (eg switch core including crossbar and arbiter). Was to place. However, due to current packaging and power constraints, it is not possible to place a large number of line cards near the switch core. As a result, with conventional configurations, simply increasing the number of line cards placed in the routing structure cannot address the ever-increasing demand for more bandwidth.

  In Patent Literature 1, an attempt is made to increase the total system bandwidth by increasing the number of line cards by physically separating the line cards from the routing structure. Most of the buffering and processing is performed on physically remote line cards. FIG. 2A shows a system according to this approach.

  As shown in FIG. 2, the system includes a switch core 210 and a plurality of line cards 202 provided at positions physically separated from the switch core 210. Each line card 202 includes an input VOQ buffer group (queue) 221 and an output buffer 222. The switch core 210 includes a plurality of port modules 280 (ie, “switch ports”), a parallel slice self-routing crossbar structure module 206, and a centralized arbiter module 207. Data packets are transmitted and received along the data link 231 between the line card 202 and the switch port 280 and the data link 203 between the switch port 280 and the crossbar routing structure 206. Each line card 202 includes a buffer group 221 that stores packets transmitted through the forward path and an output buffer 222 that stores packets of the return path. Control messages are transmitted and received along the control path 232 between the line card 202 and the switch port 280 and the control link 204 between the switch port 280 and the arbiter module 207. Arbiter 207 determines the appropriate configuration for each time slot and supplies the configuration to routing structure 207 along configuration link 212. In order to minimize the RT between the VOQ 221 and the arbiter 207, a small buffer 281 having a VOQ is arranged near the switch core 210 for each input port of the switch core 210. This is accomplished using a line card to switch (LCS) protocol that allows lossless communication between the line card 202 and the switch port 280.

  The main disadvantage of the technique of Patent Document 1 is that the switch port 280 requires only a small amount, that is, buffering a sufficient number of packets to cover one RT, and the line card 202. And switch port 280 both include buffers. These buffered switch ports increase cost, card space, power, and latency (eg, additional SerDes and buffering). They also duplicate the functionality already present in the line card 202.

  Even if the technique described in Patent Document 1 is used, it is practically difficult to make the round trip time between the switch port 280 and the arbiter module 207 shorter than one cell time. Furthermore, in the specific case of a switch structure with an optical link and optical routing structure from the line card to the switch core (to cover the long distance from the line card to the switch core), the optical buffer is currently practical. The switch port 280 further requires electricity-to-light and light-to-electrical conversion for buffering in the electrical / CMOS chip because it is not or economically feasible. The addition of such a conversion circuit significantly increases the cost and complexity of the system.

  Another technique for reducing the latency of the interconnect network shown in Non-Patent Document 1 includes “speculation with lookahead”. As described in Non-Patent Document 1, the router matching arbiter uses a look-ahead speculation to examine the queue of the input VOQ deeper than the first element (beginning of line, that is, HoL), and to some switch resources, Pre-assign an expectation (hope) that these subsequent packets will be granted permission. This approach attempts to minimize the number of pipeline stages by allowing the router to perform some matching and configuration tasks in parallel. A look-ahead speculation is advantageous for packets that are added to the queue and are present in the input VOQ and for packets that have already been requested for transmission and cannot be immediately addressed by the arbiter, but have not yet been received by the arbiter. And / or does not increase the transmission rate of packets that have just arrived at the input VOQ.

  Furthermore, the look-ahead speculation mainly deals with latency in the arbiter algorithm and usually does not deal with latency in transmission time from the larger transmitter to the switch structure. Prior to the present application, the concept of double and triple speculative execution (see page 317 of Non-Patent Document 1, for example) that relies on increased internal switch speed and light switch load to allocate more switch resources speculatively. ) Has failed in most applications. In many conventional strict non-blocking switch structures, the internal structure is internally partitioned into a plurality of successive switching stages. In double and triple speculative execution, these stages are incrementally set (assigned) to speculative loads. These approaches typically perform speculative execution only when the speed increase is extreme or lightly loaded, and are usually successful in allowing transmission through the entire multi-stage structure. As the load increases, this speculative allocation approach impairs performance because it reserves resources that are better allocated to well-arbited requests.

  Furthermore, all of the systems described above still have the initial RT latency problem that the line card must wait for a grant to arrive after submitting a request for output resources.

Thus, there remains a need in the art for a more efficient, less complex and lower cost method for reducing the latency associated with the routing structure of an interconnect system.
US Pat. No. 6,647,019 WJ Dally et al., “Principles and Practices of Interconnection Networks,” Morgan Kaufman, 2004, pp. 316-318

  Accordingly, it is an object of the present invention to provide a method and apparatus that substantially eliminates one or more of the disadvantages and problems resulting from the limitations and drawbacks of the related art.

  The present invention includes systems and methods for transmitting information units that reduce the overall latency caused by planned arbitration in a system that uses a routing structure. The method includes a switching structure in which data packets are transmitted from multiple sources to multiple inputs of a routing structure with or without planned arbitration, and the input packets are switched to the output of a specified routing structure. Especially suitable for the system.

  In one aspect of the present invention, a method transmits at least one data packet according to a result of prior arbitration (ie, planned arbitration), if planned arbitration results exist, If there is no arbitration result, the method includes a step of selecting a data packet to be speculatively transmitted. The method also includes simultaneously issuing a speculative request that includes the output identifier of the selected data packet and sending the selected packet to the routing structure.

  In another aspect of the invention, a system for sending data packets from multiple inputs to multiple outputs of a routing structure with or without planned (pre-) arbitration, where pre-arbitration results exist A mechanism for transmitting at least one data packet according to the result of the planned arbitration, and a data packet to be transmitted speculatively when the result of the planned arbitration does not exist, and an output identifier of the selected data packet And a mechanism for simultaneously issuing a speculative request including, and transmitting the selected packet to the routing structure.

  Another aspect of the present invention includes a system for managing the flow of information units between multiple locations. The system includes a plurality of inputs for receiving a plurality of access requests to a switching structure for transmitting information units from a plurality of source locations to a plurality of destination locations. This access request includes a request for access permission by planned arbitration and a request for speculative access without permission by planned arbitration. Each request also includes an indication of the destination location to which the associated information unit should be transferred. The system includes an arbiter that determines assignments to the group or subgroup of permission requests through planned arbitration that do not cause a conflict of switching structure resources. The system receives a request for speculative access when the group in which the decision is made is to be applied next to the switching structure and grants or denies the received speculative request based on the determined assignment. Including arbiter.

  Yet another aspect of the present invention is to increase the transmission rate of data packets that have just arrived at the input queue without requiring pre-allocation or reservation of switching resources that would be wasted if the grant was ultimately denied. To make it easier.

  Another aspect of the present invention includes reducing the average latency by reducing the amount of round trip transmission between the packet source and the routing structure. For example, a successful speculative transmission of a packet eliminates at least one round trip (ie, from sending a request until receiving a grant) required to send the packet through the routing structure.

  Additional aspects and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned from the practice of the invention. The aspects and advantages of the present invention will be realized and attained by the system and method particularly pointed out in the written description and claims hereof as well as the appended drawings.

  It should be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the invention as claimed.

  The accompanying drawings, which are included to provide a further understanding of the invention, and are incorporated in and constitute a part of this specification, are illustrative of the invention that serve to describe and explain the principles of the invention. An embodiment is shown.

  These and other aspects of the invention will now be described in more detail in connection with the examples illustrated in the accompanying drawings.

  As described above with respect to the related art, centralized arbitration configurations achieve high maximum throughput, but at low to moderate usage rates, there is a penalty for arbitration latency. The present invention addresses such latency using a new type of speculative execution called speculative transmission (STX). The STX concept described herein relates to the basic concept behind ALOHA and Ethernet, which does not require prior arbitration (ie, in anticipation of a given transmission success). Packets are transmitted over shared media. However, in the present invention, this concept is applied to the context of switched media in order to eliminate arbitration latency. When operating a packet switch using only speculative transmissions, frequent packet collisions occur, resulting in catastrophic performance degradation at moderate to high usage rates. In combination to obtain the advantages of both concepts: high maximum throughput and low latency at low to moderate utilization.

  The present invention reduces arbitration latency using STX without prior arbitration, in combination with routing structure arbitration, without implementing additional buffer stages near the routing structure (fabric). Furthermore, the present invention allows a very deep pipeline of speculative processing that can effectively deal with round trips that are (much) greater than the packet duration. The present invention can reduce the overall latency by 50% or less by substantially reducing or eliminating the average latency caused by arbitration in the switching structure.

  In contrast to prior art speculative execution schemes that involve pre-allocation of switch resources that are often wasted, the nature of the efficient non-blocking single stage transmission of the routing structure described herein is such Do not pre-allocate resources. In other words, since the structure (fabric) resources are not actually speculatively set, the present invention avoids waste of switch resources by implementing a single stage selector in the switching structure.

  The present invention addresses issues related to STX selection policy, collisions, retransmissions, out-of-order delivery, resequencing, duplex delivery, resending and resequencing window size determination. However, it should be understood that other problems and particularities related to the switching structure for any particular application of the present invention will be readily apparent to those skilled in the art.

  FIG. 3 illustrates an exemplary system according to the present invention. As shown in FIG. 3, the system includes N bidirectional full-duplex input / output data links 301 connected to N line cards 302. A data packet having payload information and a header including information indicating the destination of the requested packet are transmitted / received via the data link 301. Each line card 302 is connected to the switch core 310 via a bidirectional full-duplex intra-structure data link 303. The switch core 310 has a routing structure 306, illustrated as a crossbar having N input ports and N output ports. Each line card 302 is also connected to a centralized allocation / arbiter 307 with a dedicated bi-directional control link 304 that exchanges control messages (eg, including requests and permissions) between the line card 302 and the arbiter 307. ing. Arbiter 307 is connected to crossbar 306 via configuration link 312. Each line card 2 further includes an input buffer unit 321 and an output buffer unit 322. Hereinafter, the configurations and functions of the arbiter 307, the input buffer unit 321 and the output buffer unit 322 will be described in more detail with reference to FIGS.

  The present invention may be used with other types of routing structures. However, the invention is particularly suitable for switches based on crossbars. Thus, for purposes of illustrating a preferred example, the system shown in FIG. 3 will be described in the context of a switch core that includes a crossbar-based switch, but those skilled in the art will recognize that the present invention has many switching architecture architectures. You will see that it is applicable.

  A characteristic of a crossbar switch is that one input can always be connected to only one output, and one output can be connected to only one input. That is, the input and output of the switch are matched one to one. In order to obtain good performance in terms of latency and throughput, matching is typically calculated by a centralized assignment / arbiter. Furthermore, the allocation / arbiter resolves the competition for one output resource of the switch. In the exemplary system shown in FIG. 3, arbiter 307 performs these allocation and arbitration functions. In FIG. 3, the arbiter 307 is shown as a single unit, but the allocation and arbitration functions are performed by two or more hardware devices and / or one or more hardware devices operating in conjunction with program instructions. It should be understood that it may be performed.

  The arbiter 307 receives the output resource request from the line card 302. The request includes an output port identifier indicating that the line card 302 that originated the request wants to transmit a packet to a particular output port. The arbiter 307 calculates an appropriate one-to-one matching between the input port and the output port based on the received request. This corresponds to the bipartite graph matching problem. The arbiter 307 returns a corresponding permission to the line card 302 based on the calculated matching. The permission includes an output port identifier which means that the line card 302 that has received this permission can transmit a packet to a specific output port. Optionally, if no output resource is available for a particular request, arbiter 307 may return a response corresponding to the rejected request. When the line card 302 receives permission, the line card 302 extracts one packet from the corresponding VOQ of the input buffer unit 321 and transmits the packet to the crossbar 306 on the data link 303. Crossbar 306 routes incoming packets to data link 303 according to the configuration (ie, matching) computed by arbiter 307 and applied via configuration link 312.

  FIG. 4 is a diagram showing the input buffer unit 321 in more detail. The input buffer unit 321 includes an input demultiplexer 410, a plurality of virtual output queues (VOQ) 411 corresponding to each output, a plurality of retransmission (RTX) queues 412 corresponding to each output, A plurality of demultiplexers 413 corresponding to one for each VOQ 411 (these demultiplexers actually have a duplication function in addition to the routing function), and a plurality of multiplexers corresponding to one for each VOQ 411 414, an output multiplexer 415, and a control unit 416.

  In operation, a packet arrives on the input data link 301 connected to the input buffer unit 321, and the demultiplexer 410 routes the received packet to the corresponding VOQ 411 according to each destination (output port). The arrival of the packet is also transmitted to the control unit 416. The control unit 416 transmits a request to the arbiter unit 307 and receives permission from the arbiter unit 307 via the control channel 304. When a sequence number is required for the reliable delivery method, the control unit 416 assigns a subsequent sequence number to subsequent packets destined for the same output. Planned mediation requests and speculative requests, as well as grants and responses, all include an output identifier. The speculative request, grant, and response may further include a sequence number. For example, certain types of reliable delivery (RD) methods (some of which will be described later) have a sequence number for each of a data packet transmitted as a result of planned arbitration and a speculatively transmitted data packet. It may be necessary to have

  The system has two transmission modes: arbitrated transmission (ie, transmission caused by planned arbitration) and speculative transmission (STX). The planned arbitration transmission mode is known in the prior art. For example, C. Minkenberg et al., IP. com, June 3, 2004, article number IPCOM00000028815D (the entire contents of which are incorporated herein by reference). The present invention addresses speculative transmission mode and the interaction between planned arbitration transmission mode and speculative transmission mode. This is accomplished by distinguishing between planned arbitration requests and corresponding grants and speculative requests and corresponding positive and negative responses.

  Returning to FIG. 4, in the preferred embodiment, the line card 302 is eligible to make a speculative transmission only if it has not received any permission for a planned arbitration transmission in a given time slot. When no grant has been received, the controller 416 selects a non-empty VOQ 411 according to a given policy, from which one packet is retrieved for speculative transmission. This policy can be, for example, random, round robin, preferred first cell (OCF), oldest cell first (YCF), youngest cell first (YCF), etc. In the preferred embodiment, the policy is chosen to maximize latency gain. For example, a policy that prioritizes newer arrivals, such as prioritizing the cell that has arrived the latest, tends to maximize the latency gain.

  The selected packet is stored in the retransmission buffer 412 via the multiplexer 413, and transmitted to the crossbar 306 on the data link 303 via the multiplexers 413, 414, and 415. At the same time, the control unit 416 transmits a corresponding speculative transmission request including the output identifier to the arbiter 307 via the control link 304. The purpose of storing the packet in the retransmission buffer 412 is to enable retransmission of the packet when speculative transmission fails. The line card 302 may issue a planned arbitration request and a speculative request for permission at the same time. In the preferred embodiment, packets stored in the retransmission queue 412 are not eligible for speculative transmission because they have already been transmitted speculatively. That is, there is little or no benefit in speculatively retransmitting a failed STX packet, since the potential latency advantage is likely already lost.

  If the line card 302 receives a grant with planned arbitration for a given output in a given time slot, the line card 302 is eligible to send a packet to this output in the current time slot. . The control unit first checks the usage rate of the corresponding retransmission queue 412. If the corresponding retransmission queue 412 is not empty, one packet is extracted from the retransmission queue 412 and retransmitted to the crossbar 306 over the data link 303 via the multiplexers 414 and 415. When the retransmission queue 412 is empty, the control unit checks the usage rate of the corresponding VOQ 411. When the retransmission queue 411 is not empty, one packet is extracted from the VOQ 411 and transmitted to the crossbar 306 over the data link 303 via the multiplexers 413, 414 and 415. It should be understood that if a grant arrives, a packet waiting in the RTX queue 412 is usually prioritized over a packet in the corresponding VOQ 411. This is to ensure that failed STX packets are quickly delivered to their destination. However, the priority may also depend on other factors such as the nature of the data waiting or arriving in the queue 411 (eg, incoming data sensitive to latency such as speech or video). Because of the speculative mode of operation, permission may arrive for an empty VOQ 411. Such permission is considered useless. In the preferred embodiment, the line card 302 is eligible to send speculative packets in any time slot that receives a useless grant.

  In the exemplary embodiment of the invention described below, when the line card 302 receives a response, the line card 302 indicates that the packet having the sequence number included in the response is indicated by the output identifier also included in the response. It is checked whether or not it exists in the retransmission queue 412. If this packet does not exist, the response is ignored. When this packet exists, an action is taken according to a specific application example of the present invention (for example, an application example in which a high-reliability distribution method is integrated (see the section “Reliable distribution”)). For example, in an exemplary embodiment using Selective Retry (SR), the responded packet (if any) is retrieved from any location in the retransmission queue 412. In another exemplary embodiment (e.g., Go-Back-N (GBN), Stop and Wait (S & W)), the replied packet is the head of the retransmission queue (GBN) or VOQ (S & W). Only taken out if

FIG. 5 shows an example of the output buffer unit 322a of the line card 302 that is not re-sequenced. The buffer unit 322a can be used together with an RD method that does not require re-sequencing such as stop-and-wait and Go-Back-N (described later). The output buffer unit 322 a includes an enqueue unit 521, an output queue 526, a dequeue unit 527, and a control unit 528.

  In operation, the packet arrives at the output buffer unit 322 a along the data link 303. The control unit 528 examines the header of the packet and determines whether the enqueue unit 521 should store or drop the packet in the output queue 526. If a stop-and-wait or Go-Back-N RD scheme is used, packets are accepted only in the correct order. However, it is understood that even if stop-and-wait is used, double delivery can occur, while even using GBN, double delivery or out-of-order delivery can occur. I want. Accordingly, the controller 528 maintains the next expected sequence number for each input. The control unit 528 checks whether the sequence number of the received packet is equal to the next expected sequence number for the corresponding input (for example, the sequence number of the latest successfully received packet + 1). If affirmed (ie, if the expected sequence number and the received sequence number are equal), the packet is added to the output queue 526. If not, the packet is dropped due to dual delivery (eg, sequence number too low) or out of order delivery (eg, sequence number too high).

  When there is a packet in the output queue 526, the dequeue unit 527 extracts the packet from the output queue 526 and transmits the packet on the external data link 301.

When Re-Sequencing with Output FIG. 6 a shows another exemplary embodiment of the output buffer 322 b of the line card 302. This embodiment should be used with RD schemes that require resequencing, such as selective retry (SR). The output buffer unit 322b includes a demultiplexer 621, a demultiplexer 622, a plurality of resequencing (RSQ) queues 623 corresponding to each input, a multiplexer 624, a multiplexer 625, and an output queue 626. , A dequeue unit 627 and a control unit 628.

  In operation, the output buffer unit 322 b receives a packet on the data link 303. The control unit 628 checks whether the arriving packet is in order (ie, whether its sequence number is equal to the next expected sequence number for the corresponding input). If so, the packet is routed through demultiplexer 621 and multiplexer 625 and added directly to output queue 626, incrementing the expected next sequence number. It should be understood that the next expected number value depends on the particular type of sequencing scheme selected. For example, in one exemplary embodiment described below, the packet sequence number is assigned using an integer that is incremented by 1 for each next packet in the expected order. Of course, resequencing can be used in many other sequencing schemes, such as a scheme in which the number is decremented in sequence order and / or a scheme in which the sequence number of a packet is incremented or decremented by a value other than 1. This can be done with any one. If a packet arrives out of order (eg, if its sequence number is greater than the next expected sequence number for the corresponding input), the packet is routed via demultiplexer 621 and demultiplexer 622. Routed to the resequencing queue 623 corresponding to that input. Accordingly, the resequencing queue 625 stores all packets that arrive out of order. These packets can go to the output queue 626 only if all previous packets are correctly received. If the packet is dual delivery (eg, if its sequence number is less than the next expected sequence number for the corresponding input), the packet is dropped. Doubly delivered packets in the resequencing queue are not dropped according to this policy, but simply overwrite the same packets already stored.

  Whenever an in-order packet arrives, the control unit 628 checks whether the in-order arrival enables the retrieval of the packet from the corresponding resequencing queue 623. If so, one or more of such packets are removed from the resequencing queue 623, routed through the multiplexer 624 and multiplexer 625, added to the output queue 626, and the next for the corresponding input. The expected sequence number is updated.

  FIG. 6b shows an example of resequencing in the output buffer unit 322b. For the data link 301 and the specific input line card 302, all packets including the sequence number 3 are correctly received. In addition, packets with numbers 5, 6 and 8 have been received correctly, but they are out of order. Thus, packets 5, 6 and 8 are stored in the resequencing queue 623 for the data link 301. Next, packet 4 arrives and is immediately routed to output queue 626 because it corresponds to the next expected sequence number. In addition, packets 5 and 6, except packet 8, can now be retrieved from resequencing buffer 623 and added to output queue 626. When this is complete, the next expected sequence number for data link 301 is seven.

  If there is a packet in the output queue 626, the dequeue unit 627 extracts the packet from the output queue 626 and transmits the packet on the external data link 301.

  FIG. 7 is a diagram schematically illustrating an example arbiter unit 307. The arbiter unit 307 includes a plurality of control message receiving units 761 corresponding to each input control link 304a, a matching unit 762, a plurality of matching delay lines 763 corresponding to each input, A speculative request arbitration unit 764 and a plurality of control message transmission units 765 each corresponding to each output control link 304b are included.

  As shown in FIG. 7, the arbiter unit 307 receives a control message from the control link 304a. The control message receiving unit 761 decodes the incoming control message. The decoded planned arbitration request is sent to the matching unit 762, and the decoded speculative request is sent to the speculative request arbitration unit 764. The matching unit 762 can be PIM, i-SLIP (see N. McKeown, “Scheduling Algorithms for Input-Queued Switches,” PhD. Thesis, University of California at Berkeley, 1995), or dual round robin matching (DRRM). ) Calculate a one-to-one matching between input and output according to a suitable known matching algorithm such as an algorithm. Since such algorithms are well known to those skilled in the art, a detailed description thereof is omitted for the sake of brevity. The newly calculated matching is sent to the control message transmission unit 765 and the matching delay line 763. The delay line 763 delays matching by a predetermined time in order to synchronize the crossbar configuration with the arrival of the corresponding packet. In practice, this delay is equal to the round trip time. Round trip time includes the time it takes for permission to travel to the line card and the time it takes for the packet to travel to the crossbar (including serialization and deserialization delays, transmission delays, and flight times) This is the time after deducting the time taken to send the configuration information to the routing structure and configure the routing structure accordingly.

  The speculative request arbitration unit 764 receives the current matching of the planned arbitration transmission request to be applied to the crossbar and the current speculative request as inputs. The speculative request arbitration unit 764 rejects all speculative requests corresponding to an output for which matching is already determined in the current matching. That is, in order to ensure a high maximum throughput, planned arbitration transmission is always prioritized over speculative transmission. For each output for which at least one speculative request has been made and the matching is undecided, the speculative request arbitration unit 764 selects what grants permission according to some policy (eg, round robin) and all other Reject things. The speculative request arbitration unit 764 transmits a response (ACK) corresponding to each successful speculative transmission request to the control message transmission unit 765 corresponding to the input that transmitted the request. As an option, the speculative request arbitration unit 764 transmits a negative response (NAK) corresponding to each rejected speculative transmission request. A successful speculative transmission request is added to the match and the result is applied to the crossbar via the crossbar configuration link 312.

  Each control message transmission unit 765 assembles a control message for the corresponding line card 302 and transmits the control message to the line card 302 via the control link 304b. The control message may include a response regarding the new grant by arbitration (if any) and a successful speculative transmission (if any).

  In general, when speculative transmission is used, in-flight speculative packets are out of sequence, which may cause out-of-order packets to arrive at the output buffer unit 322. One or more packets can be lost due to a collision. Therefore, depending on the specific RD method, the output buffer unit 322 may need to perform sequence check and / or re-sequencing.

Reliable delivery In the STX mode of operation, packets transmitted by speculative arbitration may collide with other speculative packets or packets transmitted by planned arbitration. If a collision occurs, one or more packets must be dropped, resulting in a lossy switching process even if there are no errors. Further measures can be taken to ensure reliable, correct delivery of each single copy in order. For example, the present invention can be integrated with many reliable delivery (RD) schemes to increase the reliability of the system. In general, such a method is performed by storing a packet on the transmission side until a response (ACK) indicating that the reception side has correctly received a specific packet is received. Retransmissions are typically triggered either implicitly by a timeout or explicitly by a negative response (NAK). Depending on the specific implementation, a sequence number may be required to identify a particular packet, and / or in addition, a resequencing buffer may be required at the output to restore the correct packet order.

  For example, in order to enhance reliable and in-order delivery even in the presence of speculative transmissions, STX arbitration combined with planned arbitration is replaced with known stop-and-wait (S & W), Go-Back-N ( GBN) and selective retry (SR) RD schemes. These exemplary schemes are described below.

Stop and Wait In the case of Stop and Wait (S & W), at any given time, only one unacknowledged packet per output is allowed outstanding at each input. The next packet for this input-output pair can only be sent if permission for the corresponding output arrives at the input or when the corresponding response arrives. In this case, the packet, speculative transmission request, and response have an indicator (eg, a 1-bit sequence number) that allows detection of dual delivery. For example, the output buffer can check the sequence number (eg, the sequence numbers with the correct order must alternate) and if OK (ie, the order is correct) Add to queue, otherwise drop the packet. For example, FIG. 5 shows an output buffer unit 322a implemented without resequencing.

  S & W is easy to implement and has low overhead. For example, a packet can remain at the beginning of a VOQ until it is answered or granted, so there is no need to physically implement a separate RTX queue. However, speculative transmission cannot be pipelined, so if the round trip time is greater than one packet time, the expected latency improvement is low. This can be particularly detrimental when traffic is bursting.

Go-Back-N
In the case of Go-Back-N (GBN), a predetermined maximum number of packets per output is allowed to be unanswered at each input at any given time. This number varies depending on the maximum allowable length of the retransmission queue 412. In GBN, the output buffer unit can be implemented without resequencing, for example, according to the output buffer unit 322a of FIG. However, since the packets are only accepted if they are in the correct order, the above sequence check is necessary. That is, all packets out of order are dropped. In addition, responses are accepted only if they are in order. For example, when ACK arrives, the input buffer unit 321 checks whether or not this ACK corresponds to the first packet in the corresponding retransmission queue 412. If the sequence numbers match and the most recent previous successful transmission for the same VOQ 411 has occurred at least 1 RT before, the packet is successfully delivered and the packet is removed from the queue and dropped. It is. If the sequence numbers do not match, the ACK is ignored. This additional condition for the latest successful transmission (which may be a packet transmission resulting from planned or speculative arbitration) is the result of a previous speculative transmission failure, even for a successful speculative transmission. This is necessary to handle a possible scenario where the arrival is out of order and can be dropped later in the output buffer 322a. In this case, ACK is erroneously detected.

  This is because if the first of the (pipelined) STX request sequence fails, there will be a gap in the corresponding ACK sequence and all subsequent ACKs will be ignored until the failed packet is successfully delivered. It means you have to. As a result, if the ACK is missing, all packets in the RTX queue must be retransmitted and are therefore referred to as Go-Back-N. A new STX request can only be issued if the RTX queue is not full.

  GBN is more complex to implement than S & W. This complexity arises in part from the fact that the RTX queue has a first-in first-out (FIFO) configuration. The extra overhead may include long packet, STX request and response sequence numbers. Input buffers can also be implemented differently (eg, shift register RTX queue and additional NAK queue).

  On the other hand, in a system with a large round trip, use of GBN may be more suitable than S & W. For example, in GBN, it is possible to speculatively transmit many packets in succession, which is not possible with the S & W method.

Selective Retry Selective retry (SR) allows a predetermined maximum number of packets per output to be unacknowledged at each input at any given time. This number depends on the maximum allowable length of the retransmission queue 412 and the maximum allowable length of the resequencing queue 623. The output buffer can be implemented, for example, according to the output buffer 322b shown in FIGS. 6a and 6b, which can accept and resequence out-of-order packets as described above. It is. The input buffer unit 321 also accepts responses in any order. This means that only failed STX packets are retransmitted and is therefore called selective retry, unlike retransmitting the entire RTX queue like GBN.

  SR is much more complex to implement than GBN due to the added complexity of multiple resequencing queues in output buffer unit 322. Furthermore, the RTX queue in the SR requires a random access configuration because packets can be taken not only from the head of the queue but also from any point in the queue.

Resizing and resequencing buffer sizing In general, for maximum benefit in terms of latency reduction, the size of the resending buffer should be speculatively ordered in order of at least one full round-trip packet. It is useful to determine that it can be sent to. In this case, the link can be used at a rate of 100% with speculative transmission.

  In order to achieve optimal benefits, in the preferred embodiment, the size of each resequencing queue (RSQ) is determined to be able to store at least as many packets as the RTX queue corresponding to the RSQ buffer. For example, the size of the RSQ 623 may be determined based on the difference between the sequence number of the first packet in the RTX queue 412 corresponding to the RSQ 623 and the sequence number of the last packet in the RTX queue 412 corresponding to the RSQ 623, for example.

Speculative Implementation in an Asymmetric Routing Structure The above example of the present invention generally illustrates a “square” crossbar-like routing structure (ie, a structure having the same number of inputs and outputs). A board such as a line card is generally connected to one structure input and one structure output, but the present invention is also implemented in a switch structure characterized by a greater number of outputs than inputs. obtain. For example, if the number of outputs is K times the number of inputs (K is an integer greater than 1), exactly K routing structure outputs can be assigned to each output line card. This allows each output line card to receive up to K packets in each time slot.

FIG. 8 illustrates an exemplary system that includes an asymmetric routing structure. As shown in FIG. 8, this system includes a switch core 810 and N input data links 803a ₁ to 803a ₁ each provided between the outputs of one input buffer 821 of N line cards 802. 803a _N and data links 803b _{1 to} 803b _K provided between the routing structure 806 and each output buffer 822. Accordingly, the output from the routing structure 806 of the switch core 810 includes a total of N · K data links 803b _x (x is 1 to k). A control link similar to that described above exists between switch core 810 and line card 802, but the control link is not shown for the sake of brevity.

  The switch core 810 includes a routing structure 806 and an arbiter 807. Arbiter 807 determines the allocation of switching structure resources that do not cause collisions to groups or subgroups of planned and / or speculative arbitration requests in order to provide appropriate matching of input and output ports. Configuration link 812 provides the switching configuration determined by arbiter 807 to routing structure 806. As described above, each output 822 is provided with the number of outputs from the routing structure that is K times the number of inputs from the line card 802. For example, in a system implementing K = 2 asymmetry, the output buffer 822 has one additional data link (output from the switching structure) that can receive packets from any one of the inputs of the switching structure. . In order to handle the increase in data packets received at any time or cycle, the queue size of the output buffer 822 is determined according to load and / or various constraints. The output buffer 822 preferably has K times the write bandwidth.

In the exemplary system of FIG. 8, a header containing data packets with payload information and information indicating the destination of the requested packet is transmitted and received via N bidirectional full-duplex input / output data links 801. . The received data packet is taken from the input buffer 821 and transmitted to the input of the routing structure 806 via the data links 803a _{1 to} 803a _N. Since each line card 802 can receive up to K outputs from the remaining N−1 line cards, the asymmetric example routing structure 806 of the present invention is, for example, by each of the N line cards 802. One packet transmitted by planned and / or speculative arbitration can be matched with one of the 803b _x data lines that are output to the output buffer 822 of any other line card 802. Thus, each line card 820 may have up to K packets that may include packets transmitted from the other N-1 line cards 802 by planned arbitration, speculative arbitration, or a combination thereof. Can be received.

  This feature can be beneficially used to allow multiple planned arbitration transmissions, multiple speculative transmissions, or a combination of both to be accepted at each output in any time slot. In one embodiment of the present invention, for example, this feature can be used to provide a great advantage of the speculative transmission method. For example, if there are no planned arbitration transmissions in a given time slot, each output 822 can accept up to K speculative transmissions. If there is one planned arbitration transmission, each output 822 can accept up to K-1 speculative transmissions. In this example, planned arbitration transmission still constitutes a one-to-one match between input and output, and speculative arbitration transmission can utilize the bandwidth of all remaining structural outputs. In this way, it can be ensured that the speculative transmission has a much higher probability of success, and a much higher latency reduction (depending on the load) is achieved.

  Existing one-to-one matching algorithms require little or no modification to implement the asymmetric example of the present invention. However, it should be understood that many schemes can be used to match the input and output of the routing structure for each of multiple types of transmitted packets and / or many types of transmitted packets. .

  To handle up to K speculative requests / responses per output in a time slot, the speculative request arbitration unit 764 of FIG. 7 must be modified. First, for each output, the maximum number of speculative requests that can be answered must be determined. For all outputs for which matching has already been determined in the current matching, this maximum number is K-1. For all other outputs, this maximum number is K. Next, the speculative request arbitration unit 764 performs a selection from the speculative request received for each output, selects the winners up to the determined maximum number, and rejects all others. This selection is preferably done in a round robin fashion for fairness, but may be done in other ways.

  The present invention has applicability to interconnect networks, such as parallel computing applications including high performance computing systems, clusters and IO networks, where latency directly affects the overall system computing performance and low latency is desirable or necessary. . The present invention may be used with routing structures including electrically controlled electrical switches, electrically controlled optical switches, and optically controlled optical switches.

  In order to facilitate understanding of the present invention, many aspects of the present invention have been described in terms of sequences of actions performed by elements of an interconnect system. In each exemplary embodiment, the various actions are performed by program instructions executed by one or more processors by special circuitry (e.g., discrete logic gates interconnected to perform special functions). It will also be appreciated that this can be done by a combination of both. Further, the present invention provides a solid state memory, magnetic disk, optical disk or carrier wave (eg, radio frequency, audible frequency or optical frequency carrier) that includes a suitable set of computer instructions that cause the processor to perform the techniques described herein. It may be embodied in any form of computer readable carrier, such as. Accordingly, various aspects of the invention can be embodied in many different forms, and all such forms are intended to be included within the scope of the invention.

  In the above example, a board (eg, a line card) is eligible to make a speculative transmission only if it has not received permission for a given time slot. However, in applications where the effective capacity of the link to the crossbar exceeds the effective capacity of the external link, such that multiple packets can be transmitted in one time slot, the available capacity is speculative and It can also be used to make authorized transmissions.

  The invention has been described with reference to specific embodiments. However, it will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the invention. Therefore, to the extent that such modifications of the invention fall within the scope of the appended claims and their equivalents, the present invention is intended to encompass those modifications.

1 shows a high-level overview of a conventional system that includes virtual output buffering. FIG. 1 shows a conventional unoptimized system using a line card to switch (LCS) protocol. FIG. 1 illustrates an exemplary system according to the present invention. The figure which shows the input buffer part by an example of this invention. The figure which shows an example of the output buffer part of the line card by this invention. The figure which shows another example of the output buffer part of the line card by this invention. FIG. 6b shows a more detailed example of the output buffer shown in FIG. 6a. The figure which shows typically the example arbiter part by this invention. FIG. 3 illustrates another exemplary system according to the present invention.

Explanation of symbols

301 Data Link 302 Line Card 303 Data Link 304 Control Channel 306 Crossbar (Routing Structure)
307 Arbiter 310 Switch core 312 Configuration link 321 Input buffer unit 322 Output buffer unit 410 Input demultiplexer 411 Virtual output queue (VOQ)
412 Retransmission (RTX) queue 413 Demultiplexer 414 Multiplexer 415 Output multiplexer 416 Control unit

Claims (10)

A method of transmitting data packets from multiple inputs of a routing structure to multiple outputs with or without planned arbitration comprising:
Transmitting at least one data packet according to the result of the planned arbitration if the result of the planned arbitration exists;
If there is no result of the planned mediation,
Selecting a data packet to be speculatively transmitted;
Issuing a speculative request including an output identifier of the selected data packet and simultaneously transmitting the selected packet to the routing structure.   The method of claim 1, further comprising storing the selected data packet and storing the packet until a response or permission to the stored packet is received.   The method of claim 1, wherein the routing structure is not buffered. For each time slot, matching the input of the routing structure to the output according to the result of the planned arbitration;
If there is an issued speculative request for each unmatched output in the time slot where the matched input and output are applied to the routing structure, one speculative request is subject to a selection policy. A process to select;
The method of claim 1, further comprising: adding an input-to-output mapping for each winning speculative request to the current result of the planned arbitration applied to the routing structure.   5. The method of claim 4, comprising the step of returning a response corresponding to each winning selected speculative request.   When the speculatively issued request is determined while suppressing other requests, the transmission of the transmission permission issued to the routing structure is synchronized with the arrival of the speculatively issued data packet. The method of claim 1, wherein the switch state of the structure is set at a time when it is just right so as not to prevent the data packet from passing through the routing structure. In a system that manages the flow of information units between multiple locations,
A plurality of inputs for receiving a plurality of access requests to a switching structure for transmitting a unit of information from a plurality of source locations to a plurality of destination locations, wherein the request is a request for access permission by planned arbitration; A plurality of inputs, each of which includes an indication of a destination location to which an associated unit of information should be transferred, and a request for speculative access without permission by planned arbitration;
An arbiter for determining an allocation in which no conflict of resources of the switching structure occurs for a group or subgroup of permission requests by the planned arbitration;
A speculative that receives the request for speculative access and grants or denies the received request based on the determined assignment when the group in which the determination is made is to be applied next to the switching structure; A system comprising: a request arbiter.   An information unit associated with an authorized one of the received speculative access requests is passed through the switching structure and sent to an output not reserved for the group or subgroup. The system of claim 7.   8. The system of claim 7, wherein the switching structure does not include a buffer circuit for storing the information unit.   8. The system of claim 7, wherein each of the plurality of inputs of the switching structure is simultaneously connectable to an individual output of the switching structure.

Images