WO2004102989A1

WO2004102989A1 - Time-division multiplexing circuit-switching router

Info

Publication number: WO2004102989A1
Application number: PCT/IB2004/050622
Authority: WO
Inventors: Paul Wielage
Original assignee: Koninklijke Philips Electronics N.V.
Priority date: 2003-05-14
Filing date: 2004-05-10
Publication date: 2004-11-25
Also published as: ATE360329T1; EP1625757A1; JP2007500985A; US20070010205A1; DE602004005980D1; CN1788500A; EP1625757B1

Abstract

A time-division multiplexing circuit-switching router comprises a plurality of input means (i1,..., iN), at least one output means (ol,..., oM), switching means for switching between said input means (i1,...,iN) and said output means (ol,..., oM) and for connecting a selected input means to output means during a predetermined time slot, and a router table means for controlling said switching means, said router table means including instructions which input means be connected to output means for a predetermined time slot. Said router table means is divided into a plurality of tables, each table having a weight which specifies the amount of bandwidth per reservation in one table in relation to a reservation in the other table(s).

Description

Time-division multiplexing circuit-switching router

FIELD OF THE INVENTION

The present invention relates to a time-division multiplexing circuit-switching router, comprising a plurality of input means, at least one output means, switching means for switching between the input means and the output means and for connecting a selected input means to output means during a predetermined time slot, and a router table means for controlling said switching means, said router table means including instructions which input means be connected to output means for a predetermined time slot.

BACKGROUND OF THE INVENTION To realize precision in latency and throughput for communication over shared interconnection, conventional communication architectures rely typically on the arbitration scheme called time-multiplexed multiple access (TDMA). An arbitration scheme does contention resolution and is essential in case of communication over shared interconnect lines. TDMA works like a time wheel (of slots) where each slot can be statically reserved for a unique master. If the time wheel consists of S slots and each slot takes an equal amount of time, then every slot reservation corresponds with 1/Sth of the available bandwidth B of the bus. Multiple slots have to be reserved for connections, which need more bandwidth than B/S. The slot reservations are stored in a table, which is typically implemented by an embedded memory like e.g. a random access memory (RAM) or a first-in-first-out (FIFO) buffer.

A problem arises when the range of bandwidth requirements of the programmed connections is large (e.g. 1 Mb/s to 20 Gb/s). Then either many slots (>20000 for the given example) in the time wheel or something else are needed to realize a large ratio with less than 20000 slots. Managing the complexity of designing chips containing billions of transistors requires decoupling computation from communication. For communication, scalable and compositional interconnects, such as networks on chip (NoC), must be used. So, the future of on-chip communication is an on-chip network of routers. Circuit- switching allows to establish connection over a conceptual physical path from a source to a destination. An on-chip router network consists, among other parts, of interconnected routers.

US 4 466 060 A discloses an adaptive distributed message routing algorithm for controlling the routing of data messages in a packet message switching digital computer network. Network topology information is exchanged only between neighbour nodes in the form of minimum spanning trees, referred to as exclusionary trees.

An exclusionary tree is formed by excluding the neighbour node and its links from the tree. From the set of exclusionary trees received a route table and transmitted exclusionary trees are constructed. WO 01/89158 Al discloses a method for controlling resources in a communication network comprising nodes interconnected by links, each carrying a bitstream which is divided into frames, each frame in turn being divided into time slots which are allocatable to form circuit-switched channels. Resources in the form of write access to time slots are associated with administrative entities. Allocation of resources is then done in such a way the allocation of resources to channels pertaining to a subject administrative entity is guaranteed to the extent by which resources have been associated with the subject administrative entity.

In an on-chip router network using time-division multiplexing (TDM), physical links can be shared to achieve a higher utilization of the interconnect resources. This requires control to set a switch inside the router and this control information is stored in a so-called slot, i. e. a predetermined unit of time, or router table.

SUMMARY OF THE INVENTION

An object of the present invention is to provide a time-division multiplexing circuit-switching router which is able to be used in an on-chip router network under reduced costs.

In order to achieve the above and further objects, there is provided a time- division multiplexing circuit-switching router, comprising a plurality of input means, at least one output means, switching means for switching between said input means and said output means and for connecting a selected input means to a selected output means during a predetermined time slot, and a router table means for controlling said switching means, said router table means including instructions which input means be connected to output means for a predetermined time slot, characterized in that said router table means is divided into a plurality of tables, each table having a weight which specifies the amount of bandwidth per reservation in one table in relation to a reservation in the other table(s).

Due to the invention the size of the router table means is reduced resulting in a reduction of the corresponding silicon area and overhead and, thus, in a saving of costs which is important for the provision of an on-chip router network. Further, the invention allows for a finer bandwidth granularity for the same size of the router table means and, thus, the same costs resulting in more efficient use of the available bandwidth in the network, since high bandwidth data streams can be covered by a higher weighted table such that less time slots need to be allocated. The invention can be used in all digital system-on-chip ICs.

Preferably, the weights of the tables are programmable. Each table can include a number (Si) of rows, and per predetermined time period the tables are cycled a number (wj) of times corresponding to the respective weight (wi > 1), so that preferably the effective slot cycle period (S_e) is

L

S_e = Σ W/ • S; /= 1

The way in which entries of the tables are enumerated depends on the latency requirements through a network the router is connected to.

In a further preferred embodiment comprising a plurality of buffer means, each connected between an input means and the switching means, respectively, each buffer means comprises a plurality of buffer portions corresponding to the plurality of tables, each buffer portion being allocated to a table, respectively, wherein the router table means is provided for controlling the buffer portions in accordance with the tables. Such a buffering concept is more elegant than a shared buffering concept, since the incoming flow control digits are stored in such buffer means per table so that the various levels of the TDMA schedule become logically independent. Preferably, said buffer means is a first-in-first-out (FIFO) buffer means.

The above described objects and other aspects of the present invention will be better understood by the following description and the accompanying Figures.

In the following a preferred embodiment of the present invention is described with reference to the drawings in which

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 shows a schematic basic block diagram of a time-division multiplexed circuit-switching router; Figure 2 schematically shows a combination of two routers connected in series and the flow of four guaranteed throughput data streams;

Figure 3 schematically shows an example of a simple router network with two 2x2 -routers and the flow of three data streams, two being best-effort and one being guaranteed-throughput;

Figure 4 shows a schematic block diagram of a time-division multiplexed circuit-switching router including a multi-layer router table according to a preferred embodiment of the invention;

Figure 5 a schematic diagram of the flow of three data streams, which propagate through a network consisting of two routers according to a preferred embodiment of the invention; and

Figure 6 shows a schematic block diagram of a plurality of buffers which are included in the router of Figure 4 per input.

DESCRIPTION OF A PREFERRED EMODIMENT

The architecture of a simple router for circuit-switching is depicted in Figure 1 for explanation purposes. The router consists of N input ports including buffers, M output ports and a switch to forward data from the inputs to the outputs (concurrently) according to a router table. Circuit-switching allows to establish connections over a physical path from a source to a destination for a certain amount of time (Leijten, J.A.J.; van Meerbergen, J.L.; Timmer, A.H.; Jess, J.A.G.; "Stream communication between real-time tasks in a high- perfonnance multiprocessor", Design, Automation and Test in Europe, 1998, Proceedings, 23-26 Feb 1998, page 125 -131).

In the routers the data is for a certain amount of time stored in queues because of timing implementation reasons. Consequently, circuit-switching over a router network differs from a shared bus TDMA architecture in that the data transport over the network involves multiple hops (one for each router on the path) instead of only one, wherein each hop (router) has a different router table. Furthermore circuit-switching is a special form of TDMA where by master-slave, or in the context of routers input-output port, pairs are scheduled as explained below.

The router table of an individual router contains the information to program a crossbar switch in a contention free manner over time. For this reason, time is divided into fixed units of time called slots. During a slot, a unit of data called a flit (flow control digit) can be forwarded by the crossbar switch from a router input-buffer to an output. The input/output mapping in a specific slot is specified by the router table T, being a matrix of size S x M, where S is the number of slot entries and M is the number of output terminals of the router. The elements of T are in the set {0, 1, . . ., N}. The value n = T(s, m), with 0 < s < S and 0 < m ≤ M , means that in slot s, if n ≠ 0, a flit is forwarded from input /^*„ to output o_m. So, row s of T specifies the mapping in slot s. The slot assignment T is periodically repeated over time according to s = k mod S, with k being a slot iterator.

Accordingly, the router table of every router in the network has S time slots. There is a logical notion of synchronicity: All routers in a network are in the same fixed- duration slot, as already mentioned before. In a slot iteration k, at most one block of data is written per output port. The outputs of the routers in a network are connected to inputs of routers by means of links between input / output pairs. Such a link causes a block that is being written to an output in slot iteration £ to be present in the queue of an input that is connected via a link, at the next slot iteration. During the next slot k+1 or later, the arrived blocks are again written to their appropriate output ports. The blocks thus propagate in a store and forward fashion. The latency a block incurs per router is equal to the duration of a slot multiplied by the difference in the arrival and departure time of the block (which is given by the reservations of two subsequent routers along the path). The bandwidth is guaranteed in multiples of block size per S slots.

The slots reserved for a path from a source to a destination increase at least by one (modulo S) per router. If slot s is reserved in some router on the path and slot (s + q)%S, with q > 0, is reserved in the next router on the path, the incurred latency for this part of path is q slots.

The order in which blocks at an input of a router arrive must be the same as the order in which these blocks are being written through one of the outputs of the router. This allows implementing the queues connected to the inputs by means of FIFOs.

The entries of the router table map outputs to inputs for every slot, i.e. T(s, o) - i. An entry is empty, when there is no reservation for that output in that slot. No contention arises because there is at most one input per output. Sending a single input to multiple outputs (multicast) is possible. In a GT (Guaranteed-Throughput) routing approach, every GT token, which is read in time slot s in some router, is read in time slot (s+q)%S in the next router in the path the token follows. The value of q is at least one and is a result of the chosen schedule. It is preferably as small as possible since the overall latency of connection is equal to the sum of all g's along the path. Guaranteed -throughput (GT) services require resource reservation for worst-case scenarios, which can be expensive.

An example of a simple router network including two 2x2-routers Rl and R2 with a router table size S = 4 is shown in Figure 2. In this Figure four GT connections are represented by the data streams $ι, s₂, S3, and 5₄. The number of time slots allocated for that data stream is shown in parentheses in Figure 2.

The first output port (shown as upper port in Figure 2) of the first router Rl is unused and, consequently, the first column of the routing table is empty. The second column of the routing matrix of the first router Rl indicates that tokens from its inputs are written alternately on the second output port (shown as the lower port in Figure 2). Consequently, both data streams s\ and 5 are routed with the desired bandwidth without contention in the first router Rl . In the second router R2, the first output port (shown as the upper port in Figure 2) receives tokens of the data streams si and S3. Since the tokens from the data stream i are routed in the time slots 0 and 2 in the first router Rl, they are routed at time slots 1 and 3 in the second router R2. This is seen by the two "1" in the first column of the router table of the second router R2. The single time slot required by the data stream S is scheduled in the time slot 2 of the first column. Similarly, as indicated by "1" in the second column of the router table of the second router R2, tokens of the data stream s₂ are scheduled in the time slots 0 and 2. Finally, the tokens of the data stream &t are scheduled in the time slot 1. It is not required that a GT token is available in every reserved time slot.

When no GT packet arrives in a reserved time slot, a BE (best effort) packet can be sent over the claimed but unused time slot of the link. Best-effort (BE) services do not reserve any resource, and hence provide no guarantees, but use resources well because they are typically designed for average-case scenarios instead of worst-case scenarios. The number S of slots in the router table determines the granularity in which the total amount of bandwidth of a link can be divided. If B represents the amount of bandwidth per link, then a single connection can allocate bandwidth in chunks of B/S. Hence, increasing S, which means increasing the number of slot-table entries of all routers, results in a finer granularity. However, a bigger size of the router table results in higher costs of the router in terms of silicon area. Current estimations show that the router table can take as much as 50% of the total router silicon area. A large router table has also an operational disadvantage. Namely, for the high and medium bandwidth connections a large number of slots must be programmed. This is expensive in terms of the connection setup and teardown time. Figure 3 shows as an example a combination of two 2x2-routers Rl and R2 connected in series, wherein the two 2x2 -routers are indicated by Rl and R2, and the network terminals are identified by (i = 1, 2, ..., 6).

Assume that the first router Rl receives BE packets via terminal ti, which are all destined to the terminal t; and that the bandwidth of these packets require 10 % of the capacity of a link. Similarly, packets go from the terminal t₂ to the terminal tβ and require only 1 % of the link capacity. The second router R2 receives a GT data stream via the terminal t which is destined to the terminal tβ. The GT data stream claims and uses 99 % of the bandwidth and thus occupies the output link from output port b of the router R2 to the terminal for 99 % of time. So, the BE stream sharing port b can send a flit only in the remaining 1 % link capacity, and every time GT data arrives for port b the transmission of the BE packet over port b is pre-empted.

This can cause long latencies for the packets of the 1 % BE data stream, wherein latency is defined as the duration a packet is transported over the network. It also causes the link between the routers Rl and R2 to be occupied almost continuously by the 1 % BE stream because flits of different packets are not interleaved. Thus, BE packets of the 10 % data stream obtain less than 10 % of the rate of the link. This means that in the example of Figure 3 the link between the routers Rl and R2 has a utilization that is even below 11 % of its theoretical capacity. In order to overcome this problem there are basically three approaches: (1.) using virtual cut-through routing rather than a so-called wormhole routing, (2.) performing GT communication in relatively large blocks of data and large periods of no data, and (3.) using a GT service for the 1% BE stream.

The first approach guarantees that a complete packet will be accepted in the next router such that the incoming link of the next router does not block. However, this is at the cost of extra memory.

The second approach ensures that flit pre-emption rarely occurs. When the 99 % of GT data is grouped in blocks of 10 time units, then this bandwidth is obtained by alternative sending 99 blocks of data followed by 10 time units nothing. When the packet size ' of the BE data stream is small compared to such 10 time units, a complete packet of the 1 % BE data stream is sent in the 10 time units and the link between the routers Rl and R2 can be used by the 10 % BE data stream immediately after the packet has been sent. While the first approach suffers from additional memory requirements in the router, this second approach suffers from additional latency in the BE data stream. In the third approach, a GT service is used to realize the connection between the terminals t₂ and tβ. Consequently, the relatively low bandwidth stream is scheduled at specific moments in time by means of reserving 1 out of every 100 slots in the routing table. This requires the slot table to have a size of at least 100 entries. Since a GT service results in a circuit-switched connection during the reserved period over time, the connection uses at most 1 % of the link capacity between the routers Rl and R2. The remaining link capacity is available for the 10 % BE stream.

The third approach requires a provision for efficiently storing a set of connections with both low and high bandwidth requirement. This is achieved by means of a layered reservation table. Given the substantial amount of area overhead consumed by the reservation table, it is structured into L layers: T- (T_\, ... , Tι). The table of layer 1 = 1, . . . , L has a size of Si rows and a weight of wι ≥ 1. The weight specifies the amount of bandwidth a slot in the corresponding reservation table represents in proportion to the weight of the other layers. This is realized by constructing a combined schedule of the L tables, in which per period the tables T_\, I = 1 , . . . , L are cycled wι times respectively. Hence the effective slot cycle period S_e becomes

L (1)

S_e = Σ W/ • S/

/= 1 and this at the cost of much less physical reservation table entries

L (2)

S = Σ S, '= 1

From equation (1) it follows that a slot at layer I corresponds with a fraction wj /S_e of the total link bandwidth B.

Such a router architecture including multi-layer router table is schematically shown in Figure 4.

Figure 5 shows the filling of the router tables for the situation as illustrated in Figure 3 according to the multi-layer approach. Here, two layers are required. One stream is a best-effort stream, which is denoted by bei, and two other streams are guaranteed-throughput. These are denoted by gti and gt₂. The router table of each router, which schedules both streams, is divided in two layers, each having a different weight. The first layer 1 has a weight of 1 and supports gt₂. The second layer 2 has a weight of 99 and supports gtj. The matrices Tli and T2ι define two sub-tables associated with the first layer 1 for the routers Rl and R2 respectively. The matrices Tl and T2₂ give the reservations for the second layer 2. Consequently, a reservation of a slot in the second layer 2 requires 99 times more bandwidth allocation than a reservation of a slot in the first layer 1. As a result of the two-layer approach, the total number of slot entries S does not need to be larger than 3 for this case. The way in which the entries of the various tables are enumerated depends on the latency requirements through the network and if it is wanted to spend extra costs in the terms of independent buffering per layer.

The following description deals with two buffer options. In both cases switching from one layer to another is assumed to be done synchronously for all routers in the network.

Since the tables of the various layers are interleaved in time, the layer controller of the router will, sooner or later, interrupt the enumeration of the table of one layer to continue with one of the other layers. If a first-in-first-out (FIFO) buffer policy is employed per input, the FIFOs should not contain data that belongs to the level when the controller switches to another layer, otherwise data get messed up. It is not trivial to find such a point in the tables of all routers for a specific layer, because in general many paths through the network do overlap each other in time. A natural point where a clean switch to a different layer can be performed without intersecting paths could be after the last entry of the table. But in case of a circular schedule such a point does not exit at all. Namely, a circular schedule allows to divide a path through the routers in two pieces; the first part uses slots at the end of the table, the second part uses slots at the beginning of the table. In other words, a path can be wrapped over the boundary of the table. In practice, a schedule with valid interruption points for the "single FIFO per input approach" can result in a deterioration of the link utilization. A more elegant buffer approach stores the incoming flits in a FIFO per level as depicted in Figure 6 in conjunction with Figure 4. As shown in Figure 4, a plurality of buffers Q is provided, wherein each input i_\ to ;^' _N is coupled to such a buffer Q. In Figure 6, the construction of such a buffer Q is schematically shown. In this concept, the various levels of the TDMA schedule use different queues, as such becoming logically independent. Hence, reservation tables are allowed to be circular and switching between the layers is possible at any moment in time.

It is to be noted that the latency through the network is not the same for the two buffering strategies. For reasons of convenience, the ratio between the high and low bandwidth connections and the number of connections are kept small, respectively 1 to 99 and 3. In practice however, the ratio and the number of connections can be much larger.

The advantage of a multi-level slot table is shown as follows. For reasons of simplicity, suppose a network-on-chip consisting of just one router according to Fig. 4. Furthermore, let us focus on the guaranteed throughput connections that flow through one particular output port. Suppose there are 60 GT streams through this output. The bandwidth requirements of these streams is as follows: 50 GT-streams of 1 Mb/s and 10 GT-streams of 1 Gb/s. Hence, the total aggregated bandwidth is at least 10.05 Gb/s. Three examples A, B and C of the slot-table, which differ in the number of layers and the number of slot-table entries, will be discussed as follows.

Example A makes use of one slot-table consisting of 10050 slots. Let the bandwidth of a single link be 10.05 Gb/s such that the bandwidth per slot becomes 1/10050 x 10.05 Gb/s = 1 Mb/s. Now the 50 GT-streams of 1 Mb/s need to reserve 1 slot each and the 10 GT-streams of 1 Gb/s need to reserve 1000 slots each.

Example B again makes use of a single layered slot-table but now consisting of just 250 slot entries. This reduced number of slot entries saves a significant amount of costs. The optimal distribution of the 256 slots over the 60 streams is as follows: the 50 streams of IMb/s use one slot each, the 10 streams of 1 Gb/s use the remaining slots which means 20 each. Now, to fulfil the bandwidth requirement of all streams the bandwidth of the link must be 250/20 x 1 Gb/s = 12.5 Mb/s. Consequently, the bandwidth per slot is 50 Mb/s. One can see that this realization has disadvantages; firstly, it requires links with 25% more bandwidth than Example A and secondly, this extra bandwidth is not available for other connections since the bandwidth granularity of 50 Mb/s does not allow so. Example C makes use of a two layer slot-table. The first layer of the slot-table consists of 50 entries with a bandwidth per slot of 1 Mb/s. The second layer of the slot-table consists of 10 entries, where the bandwidth of each slot is 1 Gb/s. Consequently the weights, W_\, of the subsequent layers is 1 and 1000. This realization requires the bandwidth of the link to be 10.05 GB/s just like in example A, however now we need only 60 slot table entries in total which is just 0.6% of the number in example A.

Although the invention is described above with reference to examples shown in the attached drawings, it is apparent that the invention is not restricted to it, but can vary in many ways within the scope disclosed in the attached claims.

Claims

CLAIMS:

1. A router, comprising a plurality of input means (ii,..., I_N), at least one output means (oi, ..., O ), switching means for switching between said input means (i_\,...,1_N) and said output means (o_1}..., O_M) and for connecting a selected input means to output means during a predetermined time slot, and a router table means for controlling said switching means, said router table means including instructions which input means be connected to output means for a predetermined time slot, characterized in that said router table means is divided into a plurality of tables (T_/) (/ = 1 , ... ,L), each table having a weight (w/ > 1) which specifies the amount of bandwidth per reservation in one table in relation to a reservation in the other table(s).

2. The router according to claim 1 , wherein the router table means is divided into a plurality of hierarchical levels and each table is allocated to a certain hierarchical level.

3. The router according to claim 1 or 2, wherein the weights of said tables are programmable.

4. The router according to at least any one of the preceding claims, wherein each table (T_/) includes a number (S_/) of rows.

5. The router according to at least any one of the preceding claims, wherein per predetermined time period the tables (T/) are cycled a number (w/) of times corresponding to the respective weight (w/ > 1).

6. The router according to claims 4 and 5, wherein the effective slot cycled period (S_e) is L

S_e = Σ W/ • S/

/= 1

7. The router according to at least any one of the preceding claims, wherein the way in which entries of the tables (T/) are enumerated depends on latency requirements through a network of which the router is being a part.

8. The router according to at least any one of the preceding claims, comprising a plurality of buffer means (Q), each connected between an input means (h , ... , I_N) and the switching means, respectively, wherein each buffer means (Q) comprises a plurality of buffer portions (1,..., L) corresponding to the plurality of tables (T;), each buffer portion being allocated to a table, respectively, wherein the router table means is provided for controlling the buffer portions in accordance with said tables.

9. The router according to claim 8, wherein said buffer means (Q) is a first-in- firstrout buffer means.