EP4735970A1 - System and methods for fast-switched optical data center networks - Google Patents

Abstract

The present invention relates to methods and devices for transmitting packets in a data center network (DCN), the data center comprising a multitude of host servers, a multitude of top-of- rack, ToR, switches connected to the host servers and an optical network fabric connected to the multitude of top-of-rack switches, wherein the optical network fabric operates according to a given schedule, wherein the given schedule defines, for each time slice in a sequence of time slices, which pairs of top-of-rack switches are connected by a dedicated optical circuit established by an optical controller of the optical network fabric for said time slice, wherein a top-of- rack switch: synchronizes to another top-of-rack switch; receives a packet at an ingress port; and sends the packet to an egress port. According to the invention, synchronizing comprises sending a synchronization message to said another top-of-rack switch in-band.

Description

SYSTEM AND METHODS FOR FAST-SWITCHED OPTICAL DATA CENTER

NETWORKS

The present invention relates to a system and to methods transmitting data in data center networks (DCN).

TECHNICAL BACKGROUND

Designs for data-center-networks have largely benefited from Moore’s law for networking - the bandwidth of electrical switches doubles every two years at the same cost and power. As this bandwidth scaling slows down, a series of optical DCN architectures have been proposed to leverage the bandwidth, power, and cost advantages of optical interconnects [13-16, 20, 23, 27, 31-33, 36, 37, 42^16], Compared to electrical interconnects in traditional DCNs, optical interconnects use circuit switching to establish dedicated optical circuits between end points and shift the circuits across “time slices” to create time-shared networks.

So-called slow-switched optical DCNs have tens of milliseconds of switching delays [20, 37, 42, 44-46]. Limited by the switching speed, this type of optical network has to work in tandem with an electrical network to avoid network partitioning, e.g., either augmenting the electrical DCN with on-demand circuits to offload heavy traffic [20, 42], or serving as “patch panels” for electrical switches and reconfigure the network topology on a seconds to hours granularity [14, 37, 44-46], For example, Jupiter — Google’s DCN fabric — has achieved 5x capacity increase, 41% power reduction, and 30% cost reduction after deploying slow-switched optical interconnects in the network core [37]. These optical interconnects provide large port counts to interconnect electrical switches and reconfigure the DCN topology when needed, e.g., at device upgrade and failure times, or once a few hours as the DCN traffic evolves.

Figure 1 , on the other hand, shows an example of a typical, general purpose fast-switched optical DCN that has increasingly been recognized as an alternative to slow-switched DCNs in recent years. The fast-switched optical DCN comprises optical switches that interconnect electrical top-of-rack switches (ToRs) and end servers [31-33, 36]. The fabric uses circuit switching to establish dedicated optical circuits that are time-shared amongst the different ToR pairs for high-speed transmission of aggregated traffic. Once established, a circuit is retained for a fixed interval of time, called a time slice, during which the connected ToRs have exclusive use of it, i.e., no contention with other ToRs. An optical controller, e.g., an FPGA board [13, 27, 32 and 33] controls the circuit switches to change the circuits continually, once per time slice, to route traffic in the optical domain on an all-optical network fabric. The sequence of ToR-wise connections associated with their time slices constitutes an optical schedule. Normally, the schedule is pre-defined and repeats every optical cycle. There is at least one circuit between every ToR pair per cycle. The removal of the electrical network further reduces cost compared to slow-switched optical DCNs, but at the same time deviates from the all-to-all connectivity assumed by conventional DCN designs. The switching delays of fast-switched optical DCNs vary between several nanoseconds [13] to tens of microseconds [32, 33, 36].

PRIOR ART

Table 1 summarizes the limitations of fast-switched DCN architectures proposed until now:

Table 1 : Implementation limitations of existing work.

As table 1 shows, existing architectures have not gone beyond proof-of-concept prototypes of the proposed optical network fabric, and their systems with bare minimum testing functionalities do not suggest actual end-to-end system implementations.

A common problem of ToRs emulated with Linux servers [33, 36], and hosts emulated with FPGA boards [13, 27] is time synchronization of ToRs and hosts with the optical fabric. Some architectures have the optical controller notify ToRs of upcoming circuits [27, 36], but it is hard to scale this design to large DCNs. Some others rely on link up/down events on ToR/host ports to detect circuit on/offs [32, 33], but resetting ports incurs millisecond-scale delays — not responsive enough for fast-switched optical DCNs.

Mordia [36] and RotorNet [33] apply multi-hop routing to both long or heavy (“elephant”) flows and short or light (“mice”) flows (latency sensitive traffic), which requires large buffers on ToRs, e.g., ~70 MB per s witch port in Mordia on a 100 Gbps DCN. Mordia and SiP-Ring use dynamic optical schedules computed using real-time traffic estimation [27, 36], which is hard in fast- switched optical DCNs especially for bursty traffic, and neither implemented traffic estimation in their prototypes. Opera imposes a rigid relation between the number of ToRs and uplinks to ensure the existence of multi-hop paths for every ToR pair at any moment [32], making deployment and expansion challenging. Sirius and SiP-Ring require customized hardware [13, 27], e.g., custom optical modules and optical interfaces on GPUs, which are not manufactured yet.

Hence, multiple challenges remain for the implementation and eventual deployment of fast- switched optical DCNs. First, network devices need to be time- synchronized network-wide at sub-microsecond or even nanosecond accuracy to keep traffic in sync with the rapidly reconfigured circuits. Second, Top-of-Rack switch (ToR) and host systems need to maintain good performance as the circuit duration drops to the same scale as the DCN RTT and delays on the host stack. Third, even if implemented, each optical architecture is a closed ecosystem with heavily coupled optical hardware and networked system, which needs to be upgraded from one architecture to another after deployment.

OBJECT OF THE INVENTION

It is therefore an object of the invention to provide methods and devices for transmitting packets in a data center network that address the above discussed aspects.

SUMMARY OF THE INVENTION

This object is achieved by the method and the devices according to the independent claims. Advantageous embodiments are defined in the dependent claims.

According to a first aspect, the invention comprises a method for transmitting packets in a data center network (DCN), the data center comprising a multitude of host servers, a multitude of top- of-rack, ToR, switches connected to the host servers and an optical network fabric connected to the multitude of top-of-rack switches, wherein the optical network fabric operates according to a given schedule, wherein the given schedule defines, for each time slice in a sequence of time slices, which pairs of top-of-rack switches are connected by a dedicated optical circuit established by an optical controller of the optical network fabric for said time slice, wherein a top-of- rack switch: synchronizes to another top-of-rack switch; receives a packet at an ingress port; and sends the packet to an egress port, wherein synchronizing comprises sending a synchronization message to said another top-of-rack switch in-band, that is via a circuit of the optical network fabric. The in-band synchronization allows to eliminate an extra out-of-band network, such as an electrical network, for the purposes of synchronization.

The synchronization message may be sent, based on the given schedule. The given schedule may be an initial synchronization schedule, used for synchronizing at coarse accuracy, before the data center network becomes operational. The given schedule may be an operational schedule, after the initial synchronization has terminated. Synchronizing may comprise re-synchronizing the top- of-rack switch. Re-synchronization may be performed when a current amount by which the synchronized clock of said top-of-rack switch has drifted from a master clock exceeds a predefined threshold T. The amount of drift may correspond to a sum of synchronization and drift errors of the top-of-rack switch. The synchronization error and the drift error may be estimated, based on an empirical profile or statistics of the synchronization and drift errors of the top-of- rack switch. The pre-defined threshold T may be determined based on an accuracy and an overhead of re-synchronization. The pre-defined threshold T may further be determined based on a relative significance of the synchronization error and the drift error. If the synchronization error dominates, T may be made large to reduce a re-synchronization phase into re-synchronizing said top-of-rack switch directly with a lead top-of-rack switch, once per a cycle of the schedule. If the drift error dominates, T maybe made small to re-synchronize each ToR multiple times in a cycle through intermediate reference ToRs before its clock drifts off. A ToR with a minimal amount of drift may be selected as said another ToR for re-synchronization. A time slice and a port connected to said another top-of-rack switch may be stored in a lookup table on said top-of-rack switch, the lookup table being used to direct synchronization messages in different time slices to specific ports connected to top-of-rack switches to synchronize with. The lookup table may be preloaded into a control plane of said top-of-rack switch. Batches of the lookup table may periodically be injected into the data plane of said top-of-rack switch. Synchronization may return an offset for adjusting a local clock of the top-of-rack switch. The synchronization message may be a DPTP message.

According to a second aspect, the invention further comprises a top-of-rack switch, implementing one or more of the above-mentioned the method steps.

According to a third aspect, the invention comprises a method for transmitting packets in a data center network (DCN), the data center comprising a multitude of host servers, a multitude of top- of-rack, ToR, switches connected to the host servers and an optical network fabric connected to the multitude of top-of-rack switches, wherein the optical network fabric operates according to a given schedule, wherein the given schedule defines, for each time slice in a sequence of time slices, which pairs of top-of-rack switches are connected by a dedicated optical circuit established by an optical controller of the optical network fabric for said time slice, wherein a top-of- rack switch: synchronizes to another top-of-rack switch; receives a packet at an ingress port; and sends the packet to an egress port, wherein mice flows are routed over the fastest (multi-hop) path, according to the techniques described below in connection with HOHO routing. This allows to minimize their latencies.

According to a fourth aspect, the invention further comprises a top-of-rack switch, implementing one or more steps of the HOHO routing method.

In the following, methods, devices and systems according to the invention are also referred to by the general term OpenOptics. OpenOptics provides a general framework for easy end-to-end implementation of fast-switched optical DCNs. Its efficient design overcomes system limitations of existing work. It uses commercial devices for readily realizable architectures. But by decoupling the system from optical hardware, OpenOptics can adapt to novel optical hardware should they be commercialized in the future.

OpenOptics does to fast-switched optical DCNs what OpenFlow [9] did to traditional networks. It allows specific optical hardware to be integrated into the general framework in a plug-and-play manner to have a workable end-to-end system, and cloud applications can run without changes as if on traditional DCNs. As optical technologies advance, different optical architectures can be realized straightforwardly on top of OpenOptics, and the system can remain intact when the DCN fabric is upgraded to newer optical hardware. By decoupling the software system from the optical hardware, the niche area of optical DCNs is made more accessible to network researchers.

The enabler of generality in OpenOptics is a unified routing method, subsequently referred to as Hop-On Hop-Off (HOHO) routing, which can be applied to different fast-switched optical DCN architectures. Most fast-switched optical DCN architectures use a pre-defined optical schedule, i.e., a repetitive sequence of circuit connections over time slices, to avoid expensive real-time traffic estimation and circuit planning under the short time slice durations. HOHO routing takes advantage of this fact to abstract each architecture by its optical schedule. It takes the optical schedule as input and computes offline the lo west-latency paths (proven to be optimal) for mice flows. The specific routing algorithm of each architecture is replaced with HOHO routing, which produces better paths for mice flows and preserves the direct paths between source and destination ToRs for elephant flows.

With unified HOHO routing, the ToR and host systems can be unified across architectures as well. The runtime system for the offline HOHO algorithm can be implemented on ToRs and hosts, using P4 on Intel Tofino2 switches and libvma on Mellanox NICs. Bearing the aforementioned challenges in mind, a systematic design is embraced by testing the boundaries of these commercial tools for fast-switched optical DCNs. Specifically, network- wide in-band ToR synchronization is realized based on profiled synchronization errors between Tofino2 switches; HOHO routing on ToRs is implemented with careful measurements of system delays on Tofmo2 switches per the critical steps; and an application-agnostic host network is built with a fair judgement of the overheads of kernel and kernel -bypass options.

A micro-benchmark evaluation of OpenOptics performance shows that in-band ToR synchronization according to the invention can keep the synchronization errors under 15ns, the inventive ToR system achieves zero packet loss with 99.93% achievable network utilization, and the inventive host system sends 99.4% packets inside the scheduled time slices. The generality of OpenOptics is demonstrated by realizing Mordia [36], RotorNet [33], and Opera [32] - three fast-switched optical DCN architectures - on top of it. Case studies running Memcached [6] and Gloo [2] applications on them show that the tail flow completion times for mice flows in OpenOptics is comparable to that of an electrical DCN.

BRIEF DESCRIPTION OF THE FIGURES

Fig. 1 shows an example of a fast-switched optical data center network (DCN).

Fig. 2 shows an OpenOptics system according to an embodiment of the invention.

Fig. 3 shows an illustration of the backtracking algorithm for HOHO routing according to an embodiment of the invention.

Fig. 4 shows a schematic setup for profiling synchronization errors according to an embodiment of the invention.

Fig. 5 shows the 99.9th percentile of (a) multi-hop sync errors and (b) drift errors with different time durations across switch pairs.

Fig. 6 shows an initial sync phase of the ToR sync algorithm according to an embodiment of the invention. Hk is the profiled k -hop sync error (Fig. 5a). D i, is the profiled drift error (Fig. 5b) of ToR j relative to ToR i in one time slice.

Fig. 7 shows an example of the routing procedure according to an embodiment of the invention.

Fig. 8 shows a queue (a) delay estimation error & (b) a rotation offset according to an embodiment of the invention.

Fig. 9 shows kernel vs. libvma delay difference, (a) distance of signal reception, (b) signal-reply turn-around time according to an embodiment of the invention.

Fig. 10 shows ToR synchronization evaluation according to an embodiment of the invention - (a) setup and (b) errors; the subfigures in (b) show errors in synchronizing each ToR with the lead ToR once per cycle (top part) and in frequent re- synchronization with threshold T = 15ns (bottom part).

Fig. 11 (a) shows a total queue occupancy per port & (b) count of consecutive queues in use according to an embodiment of the invention.

Fig. 12 shows a delivery time of the last packet per time slice relative to the end of the time slice according to an embodiment of the invention.

Fig. 13 shows flow completion times of (a) Memcached SETs and (b) Gloo allreduce transfers according to an embodiment of the invention. DETAILED EMBODIMENT

Figure 2 shows an OpenOptics system according to an embodiment of the invention, comprising an offline and an online part. Core to the OpenOptics framework is HOHO routing, taking the unified input of optical schedule, irrespective of the specific optical architecture, as the input for offline path computation. It generates routing tables for each ToR which routes mice flows over the fastest (multi-hop) paths to minimize their latencies. Routing tables give a tentative egress port and queue which are mapped to a routing path. Packet admission control checks if the tentative path is feasible. If the given route is feasible, the packet will be enqueued, otherwise, it will be recirculated to check the next optimal route with an incremented recirculation count. The subsystems for implementing it on ToRs are described in the following. ToR synchronization is also described in the following as a prerequisite of HOHO routing for clocking the time slices correctly. In offline computation, the synchronization schedule planner takes in the profiled synchronization errors and drift errors between switches and the optical schedule to generate the synchronization schedule. The runtime synchronization process enforces the schedule to achieve desired synchronization performance and cost.

The ToR system only stores mice flows owing to memory constraints. Elephant flows, which are throughput-demanding but latency-insensitive, are routed over direct circuits between ToRs, rather than multi -hop paths via intermediate ToRs, to save bandwidth. The host system, as will be explained in the following, sends mice flows to the ToR to be handled by HOHO routing, and pauses elephant flows until signaled by the ToR with the arrival of direct circuits to destination ToRs.

This design is driven by the short time-slice duration of fast-switched optical DCNs and the generality requirement of OpenOptics. Programmable switches facilitate sophistications of the ToR system, and the latencies and system overheads on the switch data plane are orders of magnitude lower than on hosts. A ToR-centric design is also less likely to require modifications to applications, e.g., the host system is completely transparent to TCP applications.

ROUTING

As figure 1 illustrates, existing optical routing algorithms route mice flows over multi-hop optical paths through intermediate ToRs to avoid the long waiting time for direct connections. Such paths are “non-stop” in the sense that after packets hop on a path, they must stick to it until arriving at the destination ToR. HOHO routing seeks the “fastest” paths instead. Aided by new features of programmable switches, HOHO routing allows packets to hop off the original path at intermediate ToRs and hop on to another optical path that later offers an earlier arrival time at the destination ToR.

HOHO routing is made general by decoupling the routing computation from the runtime system. Assuming a packet takes zero time to traverse a ToR (with empty ToR queues), the paths can be computed offline given the optical schedule as input. HOHO produces the fastest path per source-destination ToR pair, for every time slice of possible packet arrival in the optical schedule. The path contains all the traversed ToRs, each associated with the time slice to leave the ToR, e.g., the departure time slices (notated as “t=<n>”) in figure 1. HOHO routing has the property that a full path can be decomposed into per-hop lookups on the traversed ToRs. So, the output paths are converted into static routes for next-hop lookups on each ToR. Before the lookup, the runtime system on each ToR predicts whether the actual queuing delay would cause a packet to miss the scheduled departure time slice, and it reroutes the packet to the next fastest path as needed.

The goal of HOHO routing is to forward a packet from a source to a destination ToR via the fastest path. The fastest path in an optical DCN is defined as the path that requires the minimum number of time slices. Depending on when a packet arrives at a ToR (i.e., its arrival time slice) and the optical schedule, the fastest path may “hop” through intermediate ToRs.

HOHO routing consists of an offline routing algorithm, which computes the fastest paths, and a runtime system, which orchestrates packet forwarding along these paths.

The HOHO routing algorithm is agnostic to optical DCN architectures and is general to a wide range of time slice durations. Given a cyclic optical schedule, the offline routing algorithm first computes the fastest paths for all source-destination ToR pairs, for all corresponding arrival time slices. The full paths are then converted into next-hop lookup tables for ease of implementation in the switch data plane. Logically, for each ToR, there is a next-hop lookup table per arrival time slice. At run time, when a ToR receives a packet within a particular time slice, it looks up in the next-hop lookup table corresponding to that (arrival) time slice to get the egress port and the send time slice within which to transmit the packet. If the send time slice is later than the arrival time slice, the switch temporarily buffers the packet until the send slice, i.e., the time slice when the packet should be transmitted. In designing HOHO routing, we assume that packets always arrive at the beginning of a time slice and that there is no queuing delay at the ToRs. These assumptions effectively decouple the design of the static routing algorithm and the runtime on- switch system. In practice, if breaking these assumptions renders a packet to miss the planned send time slice to reach the next hop, we perform run-time adjustments to match it to the next time slice. This mechanism finds the next optimal path. The runtime system on ToRs detects upcoming slice-miss events based on a fair estimation of the packet egress time from the packet arrival time and the queuing delay. The cost function in the HOHO routing algorithm only takes transmission latency as the cost at the moment. To reduce slice-miss events, we could revise this cost function, for example, to include queueing delays. We then need a reasonable measurement or estimation of queuing delays across different paths network-wide, which can be collected through network telemetry. Since the tradeoff between performance improvement and system complexity is, debatably, unclear, we leave this discussion to future work.

Figure 3 shows an illustration of the backtracking algorithm for HOHO routing. The time slices of optical circuits are denoted as absolute values, with the arrival time slice being t = 0. The destination calls Routing to find the earliest last hops (A and B with t = 5), which then call Subpath to find the shortest feasible path through them from the source (S— >B— >D). Paths violating various constraints (see explanations in red) are pruned out from this backtracking search.

The (optical) circuits in an optical DCN are analogous to “buses”: the circuits (buses) transport packets (people) from a source ToR to a destination ToR. The time slices of circuits are “deadlines” to get on the “buses”. The earliest time to arrive at a destination ToR is (as illustrated in figure 3) completely determined by the earliest time slice of the last-hop ToR, i.e., when the “bus” from the last “stop” departs for the destination (e.g., t=5 in figure 3). To find the fastest path, one must find the earliest “bus” at the last hop to the destination (step 1). An “itinerary” is then planned from the source ToR to the last-hop ToR that satisfies the “deadlines” for making all the “transitions” (step 2): arriving at the next-hop ToR either earlier than the time slice to hop onward (e.g., waiting for the next “bus”) or in the same time slice (e.g., being on time for the next “bus”). In figure 3, a packet arriving from S— »B in the t=3 slice to reach D must, for instance, wait at B for 2 time units, whereas one from G— >B in the t=5 slice is “on time” to use the B— >D circuit. Once the last-hop ToR is chosen in step 1, the arrival time does not change no matter how complicated the “itinerary” is in step 2, albeit a shorter path with fewer “transitions” is preferable.

Following this intuition, a backtracking algorithm (Algorithm 1) for HOHO routing can be designed that comprises two procedures: Routing and Subpath, which correspond to steps 1 and 2, respectively:

Algorithm 1 Hop-On Hop-Off Routing Algorithm

Require:

M <— max hop count src, dst ← source ToR, destination ToR

So «— the packet arrival time slice at src

S(s,d) <— the earliest time slice when ToR s and ToR d are connected, where S(s,d) > So must hold

» Find the fastest path per ToR pair per time slice 1 : procedure RoUTiNG(src, dst , SO)

2: Sort all ToRs by S (tor, dst ) in ascending order

3 : path = min_time = S(ToR\o\,dst ) ,mln_hop = co

4: for each tor in ToRs do

5: if S(tor,dst ) > mln_time and path # 0 then

6: return path

7 : min time = S(tor,dst )

8: path' = SuBPATH(src, tor , S (tor, dst ) , 1 , {dst })

9: if path' #0 and hop(path') < min_hop then

10: path = path', min_hop = hop (path')

11 : return path

» Find the shortest feasible subpath through an intermediate ToR 12: procedure SuBPATH(src, tor , S, level , subpath) 13: if level > M then

14: return θ

15: if S(src_,tor ) S then

16: return src + subpath

17: feasible = { }

18: for each tor ' in ToRs and tor ' not in subpath do

19: if S (to', tor ) < S then

20: p = SuBPATH(src, tor S (tor', tor ) , level +1, tor +subpath)

21 : if p f 0 then

22: p feasible

23: return shortest (feasible) or 0

For a packet that arrives at the source ToR in a particular time slice, the routing procedure finds the fastest optical path to the destination ToR. It finds the last-hop ToR that provides the earliest arrival at the destination ToR, by sorting the time slices of all candidate ToRs connecting to this destination (line 2). For each candidate ToR, it calls the Subpath procedure to find a feasible subpath from the source ToR (line 20). The procedure exits on finding the first valid path (line 6) or when the search ends (line 11). Since each ToR pair is guaranteed a circuit in the optical schedule, Routing will always find a path - the direct path (S— >D in figure 3) in the worst case, if no faster path exists. When multiple fastest paths (via A and B in figure 3) exist, the shortest path is chosen (lines 9-10).

The Subpath procedure finds a feasible sub-path from the source ToR to an intermediate ToR recursively. The procedure can terminate in two ways: (i) when it fails (lines 13-14) to find a path of length at most the maximum hop count, e.g., S— >H— >E— >A— >D in figure 3, or (ii) when it finds a connection from the source ToR and its time slice can make the ’’deadline” for the nexthop transmission (lines 15-16). In figure 3, for instance, S— >B— >D meets this condition, as the time slice t=3 for S— *B is earlier than the time slice t=5 for B— >D, while S— >A— >D violates this condition. No matter how many hops are traversed, the path must start from the source ToR. So, a path is found if and only if the source ToR is directly connected to the current intermediate ToR. Otherwise, Subpath calls itself to search onward to other intermediate ToRs not already in the sub-path and finds feasible sub-paths that constantly meet “deadlines” (lines 18-22). If Subpath find multiple feasible sub-paths, we select the shortest one (line 23). In figure 3, A →F →A→ D is filtered for repetitive “A”s, and S→B→ D is chosen ultimately because it is shorter than the other feasible path S→ G— >B— >D.

The HOHO routing algorithm is optimal: the chosen path is the shortest that leads to the minimal latency. Proof. Let p be the selected path whose last-hop ToR to dst is r and path length is I. The time slice of the optical connection between r and dst is s. If there exists a better path p' from src to dst with the last-hop ToR r' at slice s" and the path length is /', then either s > s', or s = s' and I > . This may be proven by contradiction: Case I: s > s'. In Routing, last-hop ToRs are traversed by their time slices to dst ascendingly. So, p' must be found earlier than p, which is a contradiction. Case II: s = s' and I > C. When Routing breaks the tie on the same time slice, p' would overwrite p and be chosen (lines 9-10), which is a contradiction. □

HOHO routing produces full paths, including every hop along the way, but the routing lookup on each intermediate ToR is based only on the immediate next hop. This implementation preserves the optimal paths. In other words, per-hop lookups yield the optimal path. Proof. Let p be the selected path whose first-hop ToR from src is r, last-hop ToR to dst is r the optical connection between src and r is at time slice s, the connection between r ' and dst is at slice s', and the path length is I. The residual path from r to dst is p' = p - src, the arrival time at r is s, and the path length is I' = I - 1. It may be proven that p' is an optimal path for Routing(r, dst, s). If there exists a better path p'' than p' from r to dst at slice s, whose last-hop ToR to dst is r", the optical connection between r" to dst is s"', and the path length is V, then s' > s"', or s' = s ' and I' > V. For either case, because p'' starts at slice s where src and r are connected, there must be a path p" = src + p'' from src to dst, which arrives at dst at slice s'', and the path length is l = l' + 1. Comparing p' to p, one has s' > s'', or s' = s'' and I > . So, p' is better than p, which contradicts Property 1 that the chosen path p is optimal. Now that p' is optimal, since Routing selects a single path out of the feasible paths, Routing(r, dst, s) may return a different optimal path p*' equivalent to p', that is s' = s'' and I' = V. Then for the full paths p and p' from src, s' = s" and I = l~. So, p' is also optimal. Repeating the above proof hop by hop until dst, we have hopwise lookups produce the optimal path. □ If a packet misses its planned time slice, the switch system adjusts at runtime to reroute the packet by the next available time slice. According to the invention, the runtime adjustment is robust to find the next optimal path starting from the current ToR. In other words, rerouting after missing a planned send time slice gives the next optimal path. Proof. Let p be the optimal path from src to dst, R be the set of intermediate ToRs along p, and 5 be the time slices set for ToR connections of adjacent hops. Assume Si is missed at Ri and the current time slice is Sc (Sc > Si). By per-hop lookup, we get a path p' from Ri to dst at slice Sc. According to Property 2, p' is optimal w.r.t. the current time slice Sc.

For a given optical schedule, the HOHO routing algorithm is run offline per time slice per source destination ToR pair, and the output is the full path containing all the traversed ToRs, each associated with the time slice to leave the ToR. HOHO routing has the property that per-hop lookups reproduce the full path, so the output paths are encoded as static routes for next-hop lookups in a HOHO routing table on each ToR.

ToR synchronization is a prerequisite of HOHO routing for clocking the time slices correctly. The ToR system only stores mice flows owing to memory constraints. Elephant flows, which are throughput-demanding but latency-insensitive, are routed over direct circuits between ToRs, rather than multi -hop paths via intermediate ToRs, to save bandwidth. The host system sends mice flows to the ToR to be handled by HOHO routing and pauses elephant flows until signaled by the ToR with the arrival of direct circuits to destination ToRs.

TOR SYNCHRONIZATION

Current fast-switched optical DCN architectures propose synchronizing both ToRs and hosts with the optical schedule. They perform this time synchronization either by detecting link up/down events [32, 33] or via out-of-band signaling over a management network [27, 36]. Using either of these approaches in OpenOptics violates its feasibility and generality principles. It takes up to milliseconds to set up ports after detecting link up/down events on commercial switches and hosts, and frequent link up/down events in fast-switched optical DCNs can cause port flapping. Although DCNs typically have an out-of-band management network, it is hard to always have a high-speed management network connecting all devices for accurate time synchronization. The invention relies instead on in-band time synchronization that simply works on fast-switched optical DCNs without additional assumptions beyond the optical network fabric. ToRs, instead of hosts, are synchronized against the optical schedule for better scalability. DPTP is customized to serve as the time synchronization protocol, because it supports both switch-to-switch and switch-to-host synchronization in the data plane, with accuracy in the order of tens of nanoseconds [26],

It is challenging, though, to apply DPTP to optical DCNs in band. DPTP assumes a fully- connected network for continuous time synchronization, albeit optical DCNs connect ToR pairs only at discrete time slices. The transient nature of the optical circuits fundamentally limits when and where synchronization messages can be sent. Besides, ToRs are unaware of the time slices before synchronization, making it hard to reliably exchange messages.

Intuitively, optical circuits are reconfigured by an optical controller, e.g, an FPGA board. A ToR can synchronize with such a controller via PTP which is natively supported by many FPGA boards [21] and compatible with DPTP running on the ToR. It can then serve as the lead ToR for synchronizing other ToRs.

Although ToRs can be preloaded with the optical schedule and time slice duration, they do not know the starting point of the optical schedule prior to synchronization; they cannot, hence, send synchronization messages at the right time. To resolve this dilemma, an initial synchronization is performed before the network becomes operational. This process is bootstrapped with a self- defined optical schedule, which is used only for synchronization and quite different from the operational optical schedule. Initially, the lead ToR has the clock. A circuit is then added between the lead ToR and another ToR in the synchronization schedule to enable the lead ToR to “push” the clock to that ToR. The ToR that now received the clock can propagate it to yet another ToR, in the next time slice of the synchronization schedule. This synchronization schedule and time slice duration is preloaded onto each ToR. Once a ToR learns the clock, it knows where to propagate it and when. Once the ToRs are synchronized, the network transitions to the operational optical schedule and begins carrying traffic. The ToRs then use periodic re-synchronizations to re-calibrate their clocks.

Initial time synchronization can be performed naively by pushing the clock from the lead ToR to every other ToR, one by one. Each individual synchronization requires a different optical circuit that lasts for one time slice. This procedure, hence, takes as many time slices as the number of ToRs. In case of high clock drifts or a large number of ToRs, the clocks of early-synchronized ToRs might have drifted off by the time the rest are synchronized. Fast convergence of the global clock, hence, requires ToRs to be synchronized in a chain fashion through reference ToRs that have already been synchronized.

The synchronization accuracy is under the mutual influence of synchronization error, i.e., the artifact of the synchronization protocol, and drift error, i.e., the clock drift between the two times a ToR is synchronized [26]. Here, synchronization error determines the length of the synchronization chain, and drift error determines the frequency of synchronization. These two key parameters are specific to the physical characteristics of switch chips, and, hence, they are profiled to design a generic synchronization algorithm for different types of ToRs.

Synchronization and drift errors both need to be measured on the same physical clock. According to an embodiment of the invention, they are estimated with the profiled values in an offline synchronization algorithm. The algorithm statically calculates how the estimated errors develop and how they should be reduced through ToR-wise synchronizations. The algorithm outputs the optical schedule for the initial synchronization, and the synchronization steps for both the initial synchronization and re-synchronization.

PARAMETER PROFILING

In the present embodiment, synchronization and drift errors are profiled on three EdgeCore DCS810 Intel Tofino2 Switches, denoted as SW1, SW2, SW3. The profiling results are hardware-specific, but general insights can be distilled from them to inform the design of the synchronization algorithm.

Figure 4 shows a schematic setup for profiling synchronization errors according to an embodiment of the invention. ToRs synchronize with each other over direct optical circuits, so the synchronization and drift errors are profiled between switch pairs. The three switches are permuted, and, for every pair (in order), one switch is made the primary and the other the secondary. They are virtualized as logical ToRs, where the numbers denote the logical ToR IDs. Each logical ToR propagates the clock to the next one, downstream. ToRO is the lead ToR, and the ToR IDs in figure 4 also represent the synchronization hop counts. Logical ToRs on the same physical switch share the clock, so the 2-, 4-, 6-, 8-, 10-hop synchronization errors can be measured by calculating the difference between the synchronized clock and the ground-truth. Since in-band time synchronization is interleaved with operational traffic, the experiment is run with and without background traffic, at 100 Gbps line rate with 64 B and 1500 B packets.

Drift errors are independent of synchronization hops and background traffic. Drift errors are measured between every switch pair (also with order) following the methodology in the DPTP paper [26]. Specifically, one switch is synchronized against another directly (1-hop synchronization), and, for consecutive synchronization requests, the drift error is calculated as the elapsed time between the two requests based the local clock minus the elapsed time based on the master clock. The drift errors are measured under various time durations.

OBSERVATIONS.

The 99.9th percentile synchronization errors between every pair of the three switches are summarized in figure 5a, as stable upper-bounds for the synchronization errors that need to be accounted for in the synchronization algorithm. Apparently, the 99.9th percentile of synchronization errors are switch-independent and grow linearly with the synchronization hop count. The synchronization errors across switch pairs have noticeable differences, but the 99.9th percentile numbers cover a small range - within 6ns for 10 hops and 2 ns for 2 hops. The observations are fitted with a linear function to estimate the synchronization error per synchronization hop count, including odd hop counts that cannot be measured. The multi-hop synchronization error derived (for the synchronization algorithm) from this fitting function is unaffected by the specific ToRs.

The 99.9th percentile drift errors between every pair of the three switches are summarized in figure 5b, as stable upper-bounds for the drift errors. The 99.9th percentile of drift errors are specific to switches and grow linearly with the time duration. Clock drifts are intrinsic to physical characteristics of the switch chips, so the drift errors are specific to switches. The drift errors between every switch pair are fit with an individual linear function for predicting drift errors across different time durations. Users of OpenOptics must measure the drift errors between every switch pair before running the synchronization algorithm; they can use our measurement tools to automate this process.

SYNCHRONIZATION ALGORITHM

Algorithm 2 presents a ToR synchronization algorithm according to an embodiment of the invention:

Algorithm 2 Synchronization Protocol

Require:

S: the set of N ToRs {so, S1, ...,SN -1} so: the lead ToR (master clock)

M: the set of ToRs that have been synchronized

He profiled K- hop synchronization error

HΔ: synchronization error increase with one more sync hop

Dij : profiled drift error of ToR sj relative to ToR si after one time slice, where 0 < i < N, 0 < j < N , and i # j

E;: estimated error (profiled synchronization error + profiled drift error) of ToR st relative to the lead ToR s₀ rr. id of reference ToR which ToR Si is last synchronized against

T : threshold of accumulated error

L: optical schedule, as a set {(t, st «-> s; )} where each element is an optical circuit between ToRs Si and sj in time slice t

Ri^t: the set of ToRs that ToR si connects to in time slice t

Ensure:

L': optical schedule for initial sync, also in the form of a set {(t, si sj )} like in the operational optical schedule L

P: sync steps in initial sync, as a set {(t, Si Sj )}, where each element means using ToR

Si to sync ToR sj in time slice t

P': sync steps in re-sync, also as a set {(t, st sy )} like in P

1 : procedure INITIAL SYNC 2: >> Synchronize every ToR once

3: M = {s0}, E0 = 0, rO = 0

4: for t in rangefQ, \log2N]) do

5: for si in M do

6: Find sj in S - M with the minimum Dij

7: P.add ({t, Si —> Sj \), rj = i

8: L’.add ({t, Si <-> sj )), M.add (sf)

9: Ey = E; + HA + Dij

10: Ei = Ei + Dri,i

11 : procedure RE-SYNC

12: >> Re-synchronize ToRs whose errors are larger than T

13 : for every t E L do

14: for every si E M where Ei > T do

15: Find sj G with the minimum E;

16: if Ei > Ej + HΔ then

17: P'.add ({t, Sj → Si)), n =j

18: E; = Ey + HA

19: Ei = Ei + Dri,i

Figure 6 shows the important steps of the synchronization algorithm. In the initial synchronization phase, each ToR is quickly synchronized (once) at coarse accuracy, using a self-defined optical schedule. The quickest way to propagate the master clock from the lead ToR to the rest is to build a binary tree, as shown in figure 6. Level t of the binary tree consists of ToRs that have been synchronized already after t time slices. These ToRs serve as intermediaries to synchronize others in a chain. In the next time slice, each synchronized ToR pushes its clock to a remaining ToR with the least drift error. The idea is for each ToR to synchronize with a similar ToR, which will further pass the clock down to others.

An estimate of how much the synchronized clock has drifted from the master clock is based on the profiling result. The estimated error of a ToR is the sum of its profiled synchronization and drift errors, and the change in the estimated error is tracked at each synchronization step, as shown in figure 6. The estimated error of ToRO is always 0, since it owns the master clock. At the end of time slice t=l, ToRl synchronizes with ToRO, so the estimated error of ToRl is Hi+Do,i: It suffers from the 1-hop synchronization error Hi (derived from figure 5a) and the drift error Do,i relative to ToRO during one time slice (inferred from figure 5b). In the next time slice, the estimated error of ToRl becomes Hi + 2Do,i, since the drift error has increased by Do,i after one time slice. The newly synchronized ToR5 inherits the estimated error from its reference clock ToRl, but we need to change Hi into H2 for the 2-hop synchronization R5 has experienced, and the drift error D1 5 relative to ToRl is added for that time slice. This process continues for already synchronized and newly synchronized ToRs as time proceeds.

The re-synchronization phase uses the operational optical schedule and synchronizes the ToRs with each other to improve the synchronization accuracy. Usually, a ToR is connected to many others per time slice depending on how many optical uplinks it has as well as the optical schedule. Among these, the ToR with the minimum estimated error (if smaller than the ToR’s own) is a candidate for re-synchronization.

Re-synchronization performed only when the current estimated error of the ToR exceeds a predefined threshold T that balances the accuracy and overhead of re-synchronization. It is determined by the time slice duration (and the tolerance to asynchronism) and the relative significance of the synchronization error and the drift error. If the synchronization error dominates, T can be made large to reduce the re-synchronization phase into re-synchronizing each ToR directly with the lead ToR once per cycle of the schedule. If, however, the drift error dominates, T can be made small to re-synchronize each ToR multiple times in a cycle through intermediate reference ToRs before its clock drifts off.

The algorithm generates the synchronization optical schedule for the initial synchronization, and the synchronization steps for both the initial synchronization and re-synchronization. The synchronization schedule, similar to the operational schedule, is enforced by the optical controller as circuit connections on the optical network fabric. The synchronization steps can be realized via lookup tables on the ToRs, to direct synchronization messages in different time slices to specific ports connected to the ToRs to synchronize with. If there are a large number of synchronization steps, they can be preloaded into the control plane of the programmable ToRs and later injected into the data plane periodically in batches. Besides these explicit outputs, every DPTP synchronization returns the offset to adjust the local clock. An implementation of time slices with the clock offsets will be described in the following.

ROUTING RUNTIME SYSTEM

Now, the system design shall be outlined to demonstrate the feasibility of realizing HOHO routing in practice. Each ToR requires three key functionalities. First, for an arriving packet, the next hop (egress port) determined by HOHO routing depends on the packet’s arrival time slice. Therefore, each ToR switch would need to keep track of the current time slice by time synchronizing with the optical network fabric. Second, each ToR needs to implement a routing table, which can match on the packet’s arrival time slice and the destination ToR, and look up the egress port (next hop) and the send slice. Third, a ToR also needs to implement time-scheduled packet transmission such that each packet can be transmitted precisely within the send slice determined by HOHO routing.

As HOHO routing requires packets to be routed precisely based on time slices, several challenges for implementation exist. First, because the optimal path of HOHO routing may cause a packet to be sent out in a later time slice, a mechanism is needed to buffered packets temporarily until the exact time slice in the future. Second, because DPTP in ToR synchronization gives a clock offset, which cannot be used directly for time-scheduled queue management, a method is needed to translate clock offsets into time triggers to dequeue packets. Finally, because HOHO routing computes static paths assuming empty switch queues, albeit in practice queuing delays may cause packets to miss the allocated send time slice, queuing delays need to be measured and admission control needs to be performed before enqueuing packets to prevent slice misses. These challenges are addressed in the following.

For implementing queue rotation, the latest programmable switches, e.g., Intel Tofmo2, support per-packet queue selection and pausing/resuming of target queues triggered by ingress packets in the data plane [28]. These features may be leveraged to enqueue packets meant to be sent out in the same time slice into a designated queue, which are paused until the start of that send time slice. Once resumed, the queue stays active for exactly one time slice. The pausing/resuming of the queue, as well as its active duration, can be controlled by incoming packets from the on-chip packet generator [25]. The packet generator sends one packet into the ingress pipeline reliably per configured time interval, which in our case is the time slice duration. This design can be built atop the calendar queues framework [40]. A calendar queue is a priority queue, where each priority is associated with a “calendar day.” Packets can be enqueued at a priority level, i.e., a “rank”, for a future calendar day. In the present case, a calendar day is a time slice, and each egress port owns a set of calendar queues, as packets are matched to the port on a per time slice basis. Each time slice is assigned a queue in every egress port, sequentially until queue exhaustion to wrap around. Queue rotation happens every time the system advances into a new time slice, i.e., pausing the current queue and resuming the next, triggered by two consecutive packets from the packet generator.

The rank of an incoming packet is the difference between the send time slice and the current time slice, i.e., how far in the future the packet’s transmission is scheduled.

Figure 7 shows an example of the routing procedure. Incoming packet at time slice 2 initially matched to 1st entry of lookup table for optimal path (1) , should be enqueued to queue 0 (for the current time slice t2) of egress port 5 which is full, matched to 2nd entry for secondary path (2) , successfully enqueued to queue 2 (for t4 2 time slices away) of port 1. In figure 7, a packet that comes to the ToR in time slice t=3 will be scheduled to be transmitted in t=4 from egress port=l, by the third entry in the lookup table. The active queue for the current time slice being queue=0 in port=l, the packet is enqueued into queue=l to be let out one time slice later.

For time slice enforcement, the packet generator for queue rotation needs to be started after the initial time synchronization and updated per resynchronization to be globally in sync with the optical schedule. This task may be accomplished with the clock offset from DPTP synchronization. The clock offset is stored in an SRAM-based register on the programmable ToR. A kickoff packet generator is started with the minimal packet interval, which per tests carried out by the inventors is consistently within 10 ns. The absolute time of the first time slice, which is globally agreed upon, is a parameter that can be set by the control plane program. Every packet from the kickoff packet generator reads the offset value and checks if the first time slice has arrived, by adding the offset to the ingress timestamp as the local data plane clock time. On detecting the start of the first time slice, the P4 program signals the control plane to turn off the kickoff packet generator to save packet generation and processing resources on the switch. Intel Tofino2 allows packet generators to be configured by the control plane and later started in the data plane, via triggering by an ingress packet [28]. A pair of identical rotation packet generators is defined for queue rotation, whose packet interval is the time slice duration. One such generator is started by the last packet from the kickoff packet generator, which marks the start of the first time slice, with a maximum delay of 10 ns (based on minimal packet interval). Thus, the queue rotation procedure may be bootstrapped.

Queue rotation is adjusted only after the clock drift has exceeded a pre-defined threshold, which is different from T (above): The decision of re-synchronization is based on the estimated (profiled) errors, while the offset change in DPTP is from real-world measurements. The clock offset used for the current queue rotation is stored in another register. Every DPTP packet reads this register to check the offset change. When the threshold is reached, this DPTP packet restarts the kickoff packet generator, which now monitors the new clock offset to determine the start of the next time slice, derived from the start time of the first time slice, the time slice duration, and the number of passed time slices. The packet from the kickoff packet generator that has detected the start of the next time slice, based on the new clock offset, is used to update queue rotation. It simultaneously disables the current rotation packet generator and enables the other one to follow the updated clock. The P4 program recalculates the current time slice number with the new clock to start the new queue rotation from the correct queue rank. Again, the kickoff packet generator is turned off immediately after use. This process continues, and the two rotation packet generators are continually swapped, one in use and one idle, to enable live clock adjustments on the data plane.

Queuing delays (as mentioned earlier) need to be estimated in order to assess whether a packet can be delivered successfully in the scheduled time slice. It is, however, hard to predict them on commercial switches: Real-time queue information cannot be accessed in the ingress pipeline before a packet enters the queuing system.

In order to work around this limitation, a queuing delay estimation approach according to the present embodiment of the invention tracks the occupancy of each queue with a register array. Packets can be enqueued to any queue, but only active queues for the current time slice in each port can drain traffic. One adds to the queue occupancy if a packet is enqueued, and one decreas- es the occupancy of active queues every update interval (via a packet generator) by the drained traffic volume during this period (the line rate times the update interval). Thus, the queuing delay of an incoming packet is the current queue occupancy plus the packet size (assuming the packet will be enqueued) divided by the line rate of transmission. The choice of update interval is subject to the tradeoff between the estimation accuracy and the consumption of the pipeline processing resources. The inventors find 50ns intervals reasonable in practice: the packet generator sends one packet every 50ns, or at 20Mpps, which creates only 1.3% pipeline forward overhead on a Tofino2 switch with 1.5Bpps processing capacity. The evaluation in figure 8a shows the estimated queuing delay with 50ns update interval is only 58ns.

According to the present embodiment of the invention, recursive lookup is introduced in order to augment static lookup with dynamic packet admission control. For every ingress packet, admission control compares its estimated queuing delay with the allowable transmission time of the assigned queue, which is the full time slice duration for inactive (paused) queues and the remaining time slice duration for active (draining) queues. The packet is enqueued only if admission control decides it can be sent out in the scheduled time slice, otherwise, the packet is recirculated to redo the lookup for the next optimal path. The lookup table is realized as a match-action table on Tofino2 switches. The match fields are the packet’s arrival time slice, the destination ToR, and the recirculation count that the packet has experienced. The lookup (action) data returned consists of the egress port and the send time slice when the packet should be transmitted to the next hop.

Figure 7 illustrates how the lookup table is generated. Suppose a packet arrives in time slice 2, the initial lookup without packet recirculation, e.g., the first entry in the lookup table, corresponds to the optimal path, e.g., path (1). Every recirculation degrades the lookup to a less desirable path, such as the second entry with one recirculation for the second best path (2). A packet after recirculated n times has the same effect as having a new packet arrive n time slices later. That is why the second and third entries in figure 7 both point to path (2).

Although packet recirculation is a heavy operation, it prevents slice misses and simplifies the design and operation of the HOHO routing algorithm. According to an embodiment of the invention, a limit to the recirculation count is set and packets are dropped that have been recirculated too many times.

The evaluation in table 3 (later below) shows less than 0.5% packets of production DCN traffic experiences recirculation, under the critically short 1μ s time slices, and the packets are recirculated at most once. By eliminating packet losses caused by slice misses, recirculation prevents packet reordering from sporadic packet retransmissions. Packet reordering still exists, but it is rare as will be discussed later.

There are fixed pipeline processing, packet serialization and on-wire propagation delays in the system, besides the dynamic queuing delay that can be estimated. These fixed delays are offset, i.e., by sending traffic earlier, to maximize the circuit utilization. The offset amount is the delay from queue rotation to the first packet’s delivery at the destination ToR. The offset is measured with a source and destination ToR virtualized on an Intel Tofmo2 switch, and lOOGbps traffic is sent between them with different packet sizes. As figure 8b shows, the minimum delay is 1287ns. Offsetting by this amount guarantees the least delayed packet can catch the upcoming time slice. The maximum delay is 1324ns, making a narrow delay range of 34ns. It indicates, in the worst case, the most delayed packet arrives 34ns after the circuit is established, wasting a minimal amount of the slice time.

Other unavoidable system overheads, including the errors from ToR synchronization, time slice enforcement, and queuing delay estimation, must be protected by the guard band between consecutive optical time slices, during which data transmission is disallowed. According to figure 10b, the error of ToR synchronization in a 512-ToR DCN is below 20ns, and the worst case of setting a large T to synchronize every ToR with the lead ToR has an error below 40ns. As explained, queue rotation starts within a 10ns delay after the beginning of a time slice. The queuing delay estimation with 50ns update interval shows 58ns tail error in figure 8a. Guardband was introduced for reconfiguring optical circuits [13, 20, 33, 36], which can overlap with slice updates in the system. The overall system overheads of 126ns is easily covered by the mainstream microsecond-scale optical switching technologies nowadays, so OpenOptics does not require extra guard-band. HOST SYSTEM

As already explained, hosts send mice flows as normal to be dealt with by ToRs, but let out elephant flows when direct circuits to the destination ToRs are available. For generality, the host system is designed to be transparent to TCP applications.

A recent work on slow-switched optical DCNs notifies hosts of upcoming optical circuits with signaling messages from ToRs [17], but whether this approach adapts well to fast-switched optical DCNs is unknown, given the signal transmission delay and other overheads from the host stack. The feasibility of circuit notification by signaling was tested with a small measurement study. Circuit signaling was implemented in kernel and libvma kernel bypass versions, where synchronized ToRs signal the start of each time slice and the circuit connections to their connected hosts. To minimize the number of signal messages, the effective time slice duration (with the guard band accounted for) was programmed onto the hosts, letting them infer the end of the time slice with the CPU clock. Signals go through high priority paths: from a high priority queue on the ToR to a dedicated ring in libvma or a dedicated RX queue in the kernel module. The moving average technique is used to filter signal jitters: every host tracks the signal arrival intervals and uses the average value to substitute exceptionally early and late signals that are off the average by a considerable amount. Signaling packets are sent from a ToR to a connected host continuously at 100/is intervals. On every signal arrival, the host replies a 1500B packet to emulate the first data packet.

Figure 9a shows the raw signal distances before the moving average correction. The libvma implementation has consistent signal distances within ± 0.25μ s to the expected 100μ s for 95% of the data points.

Figure 9b plots the time duration from sending the signal to receiving the reply measured on the ToR. This turn-around time mostly varies in a small range of 0.75μ s with the libvma implementation. Here, a moving average was applied, so the variance of turn-around time is dominated by the delay variance on the return path.

These results show the usability of the signaling approach. After offsetting the minimum turnaround delay in figure 8b, the sub-microsecond variance is shorter than the microsecond-scale switching delay of most fast-switching technologies [27, 32, 33, 36], so the inventive host system can work well on time slices more than tens of microseconds long.

In order to implement flow pausing, flow aging [12] is adopted in order to differentiate mice and elephant flows without flow size information, which is essentially transparent to applications. A flow is considered a mice flow, until the accumulated traffic volume exceeds the stepping threshold to be degraded into an elephant flow. The stepping threshold may be set to 10KB, according to flow size distributions from DCN traffic measurements [11, 34, 39].

Kernel module vs. libvma. As a strawman solution, we implement flow pausing in a kernel module on the netfilter framework [8], We assign each destination ToR a dynamic memory buffer. Outgoing packets are stored in corresponding buffers until released by upcoming circuits to the destination ToRs. The inventors also implemented a libvma [5] kernel -bypass alternative and compare their performance. Libvma links TCP sockets to the userspace IwIP TCP stack, thus requiring zero changes to applications. We maintain regular socket behaviors by allowing applications to write to the segment queue normally. New segments can be written as long as they fit, and send attempts are pushed back when the queue is full. The inventors hack into the TCP implementation to not send out segments when the flow is paused. After a flow resume signal is received, the segment queue is flushed until the sending window is exhausted. In the application experiments carried out by the inventors, the throughput of the libvma implementation is proportional to that of vanilla libvma (without pausing) by the percentage of active sending time slices.

From figure 9, it may be observed that libvma has much higher signaling consistency (figure 9a) and much lower and more stable signal reply turn-around delays (figure 9b). Also, libvma has a cleaner design. Directly implementing the pausing mechanism inside the userspace TCP stack assures that the TCP state machine matches the actual network stack state, which is not the case with the kernel module as it operates on a different layer, netfilter pauses packets that TCP considers already sent out and is oblivious to the sending window. The inventors adopted libvma for the final host system implementation but still make the kernel module public for users without libvma-capable NICs. Elephant flows to the same destination ToR share the circuits fairly. Regarding a host offset and margin, signals should be sent ahead of time to offset the delays between ToRs and hosts. A minimum turn-around delay of 3/zs from figure 9b is taken as the offset value, which ensures the earliest arriving packet to catch the beginning of the upcoming time slice. Packets with larger delays maybe sent out of slice, i.e., sent to the ToR after the time slice ends. The ToR system would take them as mice flows and handle them with HOHO routing, so no packet loss is expected. 99.94% delay measurements falls within the range of 1/zs, so setting a margin to stop sending data Ips before the time slice ends can avoid burdening the ToR system for out-of-slice packets.

EVALUATION

In this section, we evaluate the performance of the OpenOptics framework with microbenchmark studies. Regarding an optical topology and schedule in the experimental setup, Opera provides the most efficient topology-routing co-design for fast-switched optical DCNs to date [32], The inventors emulated an Opera network with 108 ToRs, six lOOGbps optical uplinks per ToR, and six hosts per ToR also at lOOGbps. The Opera optical schedule was adopted, but the native Opera routing was replaced with HOHO routing to integrate Opera into the OpenOptics framework. ToRs, hosts, and optical fabric. The entire Opera network cannot fit into a testbed, so a representative sender ToR was implemented, a representative receiver ToR, and an emulated optical network fabric each on the three profiled switches. Three servers each with a Mellanox ConnectX-5 lOOGbps dual-port NIC are connected to the sender ToR to work as six individual hosts.

The inventors ran the Facebook traces collected from Database, Hadoop, and Web services [39], which are used extensively for DCN research nowadays [19, 24, 30, 34, 47]. The traces were scaled to the present topology size, and replayed on the servers to feed the sender ToR with incoming traffic. Indirect traffic that the sender ToR is supposed to forward on behalf of other ToRs is also injected from the servers. Traffic sent to the unimplemented ToRs (other than the sender and receiver ToRs) is dropped by the emulated optical fabric. The sender ToR and servers were made the heaviest talkers in the traces for worst-case analysis of the ToR and host systems. The inventors also devised heavy synthetic traffic, e.g., line-rate traffic and bursty traffic, to further stress test certain aspects of the system. Measurement of the synchronization accuracy requires the observed ToRs to use the same physical clock, so the setup in figure 10a was designed to emulate a large-scale DCN. The three profiled switches were reused and virtualized into ten logical ToRs. They are chained in the direction of clock propagation to represent the rightmost nodes of the synchronization tree (figure 6) in the initial synchronization phase of the synchronization algorithm. These nodes are the worst synchronized on each tree level. In this way, they act as a subset of relatively inaccurately synchronized ToRs in a 512-ToR DCN, if the tree is fully populated. After the initial synchronization in the chain order, the ToRs are re-synchronized with each other following the second phase of the algorithm. A round-robin operational optical schedule among the ToRs was assumed as a reasonable subset of the rich connections in a large DCN. The synchronization errors of ToR3, ToR6, ToR9 located on the same switch as the lead ToR ToRO were measured. As ToRs on the same switch have no drifts, drift errors were added to them from the profiled numbers for the other two switches.

Figure 1 Ob compares the synchronization errors of the inventive algorithm with the strawman solution of synchronizing individual ToRs with the lead ToR once per optical cycle. Time slice 0 shows the result of the initial synchronization. The inventive tree-building algorithm is effective: the error of synchronization is only 10ns after a short initial phase, in comparison with the 44ns error in the strawman solution, which has to wait for the direct connection to the master ToR. The inventive re-synchronization algorithm keeps the error to under 20ns, as opposed to up to 58ns the strawman solution drifts off to every optical cycle. The threshold T = 15ns is set based on the estimated error from the profiled results, i.e., re-synchronizing every time the estimated error is over 15ns. Considering the inevitable one-hop synchronization error up to 15ns in figure 5a, this very close actual error of 20ns validates the accuracy of the inventive error profiling and estimation schemes. When setting a large T, the re-synchronization phase steps back to the strawman solution.

Regarding queuing delay estimation accuracy, figure 8 a plots the queuing delay estimation error under a stress test, where we combine line-rate traffic and bursty traffic to fill and drain the queue periodically. The error is the difference between our estimated queuing delay and the measured ground-truth queuing delay. The result validates the inventive method in that the estimation accuracy is determined by the interval of updating the queue occupancy register: the estimation accuracy improves with more frequent register updates, and the 99.9th percentile estimation error, across curves, falls within the precision margin of the update interval.

The following table 2 shows a relative resource usage of an OpenOptics-enabled ToR.

As can be seen in table 2, the resource usage in the sizable 108-ToR DCN is a small percentage relative to switch.p4, the baseline P4 program that implements core L2/L3 switching functions. The low usage of SRAM, VLIW Actions, and TCAM indicates the efficient implementation of registers and lookup tables in OpenOptics. Stateful ALU and Ternary Xbar have higher usage, due to the arithmetic calculations and branching operations for packet admission control. All the resources are under 15% of an Intel Tofmo2 switch’s capacity, leaving sufficient room for OpenOptics to scale up to even larger DCNs.

Figure I l a presents the total queue occupancy per port, i.e., the sum of individual queue occupancy across the calendar queues. The tail occupancy is as low as 5.704KB for the heavy Hadoop traffic. With a high-end 128-port ToR where half ports are used as optical uplinks, we only need a total buffer size of 365KB. Figure 1 1 b shows the number of consecutive queues in use per port, or the minimum number of queues needed by calendar queues. At any moment, calendar queues occupy at most 6 queues per port. Our queue usage is significantly below the capacity limit of commercial switch ASICs [38], and it proves the effectiveness of the inventive queue rotation implementation in clearing traffic timely. Regarding packet recirculation, it was already explained that packets that will miss the scheduled send time slice are recirculated. Table 3 shows the rarity of packet recirculation under production DCN traffic.

Less than 0.5% packets are recirculated under the extraordinarily low time slice duration of Ips, which is barely attainable given the microsecond-scale switching delays of mainstream optical technologies. The number is further reduced to under 0.05% for 1 Ops time slices, because it is easier to fit packets into longer time slices. Hadoop traffic experiences slightly more recirculation, because it mostly contains large packets for bulk data transfer. Packets are recirculated at most once. These observations suggest packet recirculation adds little overhead to the inventive ToR system.

Regarding packet loss and circuit utilization, OpenOptics prevents packet loss with the joint effort of packet admission control, delay offset, and guard band. No packet loss is observed from the traces and even line-rate traffic, which proves the efficacy of these mechanisms. This result is validated by measuring when the last packet per time slice is received by the destination ToR, because there exists a tradeoff between packet loss and circuit utilization. The inventive queuing delay estimation has errors (figure 8a). If admission control enqueues packets conservatively, packet loss can be avoided, but at the cost of under-utilizing the time slices. As figure 12 shows, at most one packet is rejected at the end of a time slice. In the worst case of rejecting a 1500B packet, the leftmost point in figure 12 leads to an expected waste of 145ns x50%=72.5ns, which translates to the utilization of 99.93% on lOOps time slices and 99.28% on lOps time slices.

Moreover, OpenOptics has packet reordering in theory, because of packet recirculation and the resulting dynamics on different paths (at different time slices) to the same destination ToR. A “forwarding ToR” is virtualized on the receiver ToR just to forward traffic from the sender ToR to the receiver ToR. This new path with one more hop and the original direct path are set to be used in alternative time slices. No packet reordering was observed from numerous runs of the traces, which is consistent with the rarity of packet recirculation.

With flow pausing, the inventive host system only lets out long flows in scheduled time slices, but some packets may still arrive out of the intended time slice because of the host-ToR delay variance. The host offset and margin mitigate this problem, and one may observe zero out-ofslice packets running the traffic traces. As a stress test, iperf was run on the host at full speed. Very few out-of-slice packets may be seen, and the margin almost eliminates them. Out-of-slice packets are not lost, but stored on ToRs and handled as from mice flows.

CASE STUDIES

The inventors also implemented three fast-switched optical DCN architectures on top of OpenOptics as case studies. Real DCN applications were run on these optical architectures and show end-to-end application performance.

For cluster setup, the three profiled Tofino2 switches (SW1-SW3) were taken to realize eight logical ToRs (four on SW1 and four on SW2) and an emulated optical fabric (on SW3). Each logical ToR connects to the emulated optical fabric through four lOOGbps optical uplinks. Four servers each having a Mellanox ConnectX-5 lOOGbps dual-port NIC are connected to the eight ToRs (each through one lOOGbps NIC port) to work as eight individual hosts.

Opera [32], RotorNet [33], and Mordia [36] were implemented on OpenOptics, and “direct through” rules were set on the emulated optical fabric to have a baseline electrical DCN. Opera supports multi-hop routing (through intermediate ToRs) for mice flows and ensures the existence of (usually multi -hop) optical paths between any pair of ToRs at any moment. It caps the queue size on ToRs to 8 maximum packets to estimate a fixed queuing delay, and requires a time slice to be longer than the maximum RTT in the network under the estimated queuing delay. Its optical schedule for the delicate routing design has rigid requirements on the network structure and time slice duration. The present cluster topology is to fit the minimum-scale Opera network, and we set the time slice duration to 50μ s for all architectures, as required by this Opera scale. Opera routes elephant flows through direct circuits from source to destination ToRs to save bandwidth. RotorNet uses round-robin optical schedules and routes traffic over direct circuits. Mordia proposes to generate on-demand optical schedules based on real-time traffic. This is hard to achieve in practice, so we approximate on-demand schedules with efficient schedules for the pre-known traffic patterns in the applications. These architectures run HOHO routing on OpenOptics. To compare with their native routing schemes, queue rotation was disabled and the lookup table of each particular architecture was loaded.

Latency-sensitive and throughput intensive applications were run to validate the performance of mice and elephant flows on OpenOptics. The inventors used the Memcached [6] key-value storage for the latency-sensitive application, where two Memcached servers and six Memslap [7] benchmarking clients are run each on a host, and a client writes 4.2KB data to a server in each SET operation. The Gloo collective communication library [2] was chosen for the throughputintensive application, and ring allreduce was run on the eight hosts with varying data sizes from 800KB to 20MB.

From figure 13 a, it may be found that the multi-hop “always-on” paths in Opera effectively reduces flow completion times (FCTs), while RotorNet suffers long delays waiting for direct circuits to the destination ToRs. As expected, the “on-demand” schedule approximated for Mordia outperforms the round-robin schedule in RotorNet. Because every server talks to six clients, the optimal schedule is to iterate the four optical uplinks on the server ToR through the six client ToRs, which is more efficient than global round robin over seven other client ToRs. HOHO routing improves routing for each architecture by providing the fastest paths that may span multiple hops.

Particularly, the tail FCT of HOHO is 69% lower than that of RotorNet and 85% lower than that of Mordia, because HOHO enables “non-stop” paths to unblock packets that wait for direct paths. It is 16% lower than that of Opera, because HOHO achieves shorter path length on average by allowing packets to pause at intermediate ToRs. Also, the fixed queuing delay in Opera only admits packets to time slices conservatively, and our accurate queuing delay estimation clears the false negatives. As a result, the tail FCT of Opera with HOHO is only 30% longer than that of the electrical DCN baseline. Note that our electrical DCN setup sets a highly idealized loose upper bound where ToRs are connected directly, as opposed to multi-layer Clos networks with longer path lengths, more congestion, and higher FCTs in practice.

In figure 13b, the median allreduce FCTs of Opera and RotorNet are similar, because elephant flows can only go through direct circuits. But the direct circuits in the Opera schedule are more diverse than purely round robin, leading to slightly lower FCTs. Mordia in this case has enough ToR uplinks to form static eight-ToR rings, so the performance is the same as the electrical DCN baseline. HOHO routing only handles mice flows, so it is irrelevant here. From these results, we demonstrate generality and simplicity of OpenOptics for implementing diverse fast-switched optical DCN architectures and having side-by-side performance comparisons. OpenOptics achieves similar performance as electrical DCNs: for mice flows with HOHO, and for elephant flows if given the needed circuits. Its correctness is also validated with the expected behaviors of these architectures.

REFERENCES

[1] [n.d.]. All-reduce collective communication pattern, https://mpitutorial.com/tutorials/mpi- reduce-and-allreduce/. ([n.d.]).

[2] [n. d.]. Gloo. https://github.com/facebookincubator/gloo. ([n. d.]).

[3] [n. d.]. How to achieve low latency with lOGbps Ethernet. https://blog.cloudflare.com/how-to-achieve-low-latency/. ([n. d.]).

[4] [n. d.]. linuxptp. https://linuxptp.sourceforge.net/. ([n. d.]).

[5] [n. d.]. Mellanox Messaging Accelerator. https://github.com/Mellanox/libvma/blob/master/README. ([n. d.]).

[6] [n. d.]. Memchached. https://memcached.org/. ([n. d.]).

[7] [n. d.]. Memslap, http://docs.libmemcached.org/bin/memslap.html. ([n. d.]).

[8] [n. d.]. Netfilter. https://www.netfilter.org/. ([n. d.]).

[9] [n. d.]. OpenFlow - Open Networking Foundation, https://opennetworking.org. ([n. d.]).

[ 10] [n. d.] . Scaling in the Linux Networking Stack. https://www.kemel.org/doc/Documentation/networking/scaling.txt. ([n. d.]).

[11] Berk Atikoglu, Yuehai Xu, Eitan Frachtenberg, Song Jiang, and Mike Paleczny. 2012. Workload analysis of a large-scale key-value store. In Proceedings of the 12th ACM SIGMETRICS/PERFORMANCE joint international conference on Measurement and Modeling of ComputerSystems. 53-64.

[12] Wei Bai, Li Chen, Kai Chen, Dongsu Han, Chen Tian, and Hao Wang. 2017. PIAS: Practical information-agnostic flow scheduling for commodity data centers. IEEE/ ACM Transactions on Networking 25, 4 (2017), 1954—1967.

[13] Hitesh Ballani, Paolo Costa, Raphael Behrendt, Daniel Cletheroe, Istvan Haller, Krzysztof Jozwik, Fotini Karinou, Sophie Lange, Kai Shi, Benn Thomsen, et al. 2020. Sirius: A flat datacenter network with nanosecond optical switching. In Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication. 782-797.

[14] Kai Chen, Ankit Singla, Atul Singh, Kishore Ramachandran, Lei Xu, Yueping Zhang, Xitao Wen, and Yan Chen. 2013. OSA: An optical switching architecture for data center networks with unprecedented flexibility. IEEE/ACM Transactions on Networking 22, 2 (2013), 498-511.

[15] Kai Chen, Xitao Wen, Xingyu Ma, Yan Chen, Yong Xia, Chengchen Hu, and Qunfeng Dong. 2015. WaveCube: A scalable, fault-tolerant, high performance optical data center architecture. In 2015 IEEE Conference on Computer Communications (INFOCOM). IEEE, 1903-1911.

[16] Li Chen, Kai Chen, Zhonghua Zhu, Minlan Yu, George Porter, Chunming Qiao, and Shan Zhong. 2017. Enabling {Wide-Spread} Communications on Optical Fabric with {MegaSwitch}. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17). 577-593.

[17] Shawn Shuoshuo Chen, Weiyang Wang, Christopher Canel, Srinivasan Seshan, Alex C Snoeren, and Peter Steenkiste. 2022. Time-division TCP for reconfigurable data center networks. In Proceedings of the ACM SIGCOMM 2022 Conference. 19-35.

[18] Dah-Ming Chiu and Raj Jain. 1989. Analysis of the increase and decrease algorithms for congestion avoidance in computer networks. Computer Networks and ISDN systems 17, 1 (1989), 1-14.

[19] Inho Cho, Keon Jang, and Dongsu Han. 2017. Credit-scheduled delay bounded congestion control for datacenters. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication.239-252.

[20] Nathan Farrington, George Porter, Sivasankar Radhakrishnan, Hamid Hajabdolali Bazzaz, Vikram Subramanya, Yeshaiahu Fainman, George Papen, and Amin Vahdat. 2010. Helios: a hybrid electrical/optical switch architecture for modular data centers. In Proceedings of the ACM SIGCOMM 2010 Conference. 339-350.

[21] Alex Forencich, Alex C Snoeren, George Porter, and George Papen. 2020. Corundum: An open-source 100-gbps nic. In 2020 IEEE 28th Annual International Symposium on Field- Programmable Custom Computing Machines (FCCM). IEEE, 38-46.

[22] Yilong Geng, Shiyu Liu, Zi Yin, Ashish Naik, Balaji Prabhakar, Mendel Rosenblum, and Amin Vahdat. 2018. Exploiting a natural network effect for scalable, fine-grained clock synchronization. In 15th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 18). 81-94.

[23] Monia Ghobadi, Ratul Mahajan, Amar Phanishayee, Nikhil Devanur, Janardhan Kulkami, Gireeja Ranade, Pierre- Alexandre Blanche, Houman Rastegarfar, Madeleine Glick, and Daniel Kilper. 2016. Projector: Agile reconfigurable data center interconnect. In Proceedings of the 2016 ACM SIGCOMM Conference. 216-229.

[24] Shuihai Hu, Wei Bai, Gaoxiong Zeng, Zilong Wang, Baochen Qiao, Kai Chen, Kun Tan, and Yi Wang. 2020. Aeolus: A building block for proactive transport in datacenters. In Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication. 422- 434.

[25] Raj Joshi, Ben Leong, and Mun Choon Chan. 2019. Timertasks: Towards time-driven execution in programmable dataplanes. In Proceedings of the ACM SIGCOMM 2019 Conference Posters and Demos. 69-71.

[26] Pravein Govindan Kannan, Raj Joshi, and Mun Choon Chan. 2019. Precise timesynchronization in the data-plane using programmable switching asics. In Proceedings of the 2019 ACM Symposium on SDN Research. 8-20.

[27] Mehrdad Khani, Manya Ghobadi, Mohammad Alizadeh, Ziyi Zhu, Madeleine Glick, Keren Bergman, Amin Vahdat, Benjamin Klenk, and Eiman Ebrahimi. 2021. SiP-ML: high- bandwidth optical network interconnects for machine learning training. In Proceedings of the 2021 ACM SIGCOMM 2021 Conference. 657-675.

[28] Jeongkeun Lee. 2020. Advanced congestion & flow control with programmable switches. In P4 Expert Roundtable Series. https://bit.lyZ3 J8x 7 fw

[29] Ki Suh Lee, Han Wang, Vishal Shrivastav, and Hakim Weatherspoon. 2016. Globally synchronized time via datacenter networks. In Proceedings of the 2016 ACM SIGCOMM Conference. 454-467. [30] Yuliang Li, Rui Miao, Hongqiang Harry Liu, Yan Zhuang, Fei Feng, Lingbo Tang, Zheng Cao, Ming Zhang, Frank Kelly, Mohammad Alizadeh, et al. 2019. HPCC: High precision congestion control. In Proceedings of the ACM Special Interest Group on Data Communication. 44-58.

[31] Yunpeng James Liu, Peter Xiang Gao, Bernard Wong, and Srinivasan Keshav. 2014.

Quartz: a new design element for low-latency DCNs. ACM SIGCOMM Computer Communication Review 44, 4 (2014), 283-294.

[32] William M Mellette, Rajdeep Das, Yibo Guo, Rob McGuinness, Alex C Snoeren, and George Porter. 2020. Expanding across time to deliver bandwidth efficiency and low latency. In 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20). 1-18.

[33] William M Mellette, Rob McGuinness, Arjun Roy, Alex Forencich, George Papen, Alex C Snoeren, and George Porter. 2017. Rotomet: A scalable, low-complexity, optical datacenter network. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication. 267-280.

[34] Behnam Montazeri, Yilong Li, Mohammad Alizadeh, and John Ousterhout. 2018. Homa: A receiver-driven low-latency transport protocol using network priorities. In Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication. 221-235.

[35] Matthew K Mukerjee, Christopher Canel, Weiyang Wang, Daehyeok Kim, Srinivasan Seshan, and Alex C Snoeren. 2020. Adapting TCP for Reconfigurable Datacenter Networks.. In NSDI. 651-666.

[36] George Porter, Richard Strong, Nathan Farrington, Alex Forencich, Pang Chen-Sun, Taj ana Rosing, Yeshaiahu Fainman, George Papen, and Amin Vahdat. 2013. Integrating microsecond circuit switching into the data center. ACM SIGCOMM Computer Communication Review 43, 4 (2013), 447-458.

[37] Leon Poutievski, Omid Mashayekhi, Joon Ong, Arjun Singh, Mukarram Tariq, Rui Wang, Jianan Zhang, Virginia Beauregard, Patrick Conner, Steve Gribble, et al. 2022. Jupiter evolving: transforming google’s datacenter network via optical circuit switches and software- defined networking. In Proceedings of the ACM SIGCOMM 2022 Conference.66-85.

[38] Ting Qu, Raj Joshi, Mun Choon Chan, Ben Leong, Deke Guo, and Zhong Liu. 2019. SQR: In-network packet loss recovery from link failures for highly reliable datacenter networks. In Proceedings of ICNP. [39] Arjun Roy, Hongyi Zeng, Jasmeet Bagga, George Porter, and Alex C Snoeren. 2015.

Inside the social network’s (datacenter) network. In Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication. 123-137.

[40] Naveen Kr Sharma, Chenxingyu Zhao, Ming Liu, Pravein G Kannan, Changhoon Kim, Arvind Krishnamurthy, and Anirudh Sivaraman. 2020. Programmable calendar queues for highspeed packet scheduling. In Proceedings of NSDI.

[41] Arjun Singhvi, Aditya Akella, Dan Gibson, Thomas F Wenisch, Monica Wong-Chan, Sean Clark, Milo MK Martin, Moray McLaren, Prashant Chandra, Rob Cauble, et al. 2020. Irma: Re-envisioning remote memory access for multi -tenant datacenters. In Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication. 708-721.

[42] Guohui Wang, David G Andersen, Michael Kaminsky, Konstantina Papagiannaki, TS Eugene Ng, Michael Kozuch, and Michael Ryan. 2010. c- Through: Part-time optics in data centers. In Proceedings of the ACM SIGCOMM 2010 Conference. 327-338.

[43] Weiyang Wang, Moein Khazraee, Zhizhen Zhong, Zhijao Jia, Dheevatsa Mudigere, Ying Zhang, Anthony Kewitsch, and Manya Ghobadi. 2022. TopoOpt: Optimizing the Network Topology for Distributed DNN Training. arXiv preprint arXiv:2202.00433 (2022).

[44] Dingming Wu, Yiting Xia, Xiaoye Steven Sun, Xin Sunny Huang, Simbarashe Dzi- namarira, and TS Eugene Ng. 2018. Masking failures from application performance in data center networks with shareable backup. In Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication. 176-190.

[45] Yiting Xia, Mike Schlansker, TS Eugene Ng, and Jean Tourrilhes. 2015. Enabling Topological Flexibility for Data Centers Using OmniSwitch. In HotCloud.

[46] Yiting Xia, Xiaoye Steven Sun, Simbarashe Dzinamarira, Dingming Wu, Xin Sunny Huang, and TS Eugene Ng. 2017. A tale of two topologies: Exploring convertible data center network architectures with flat-tree. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication. 295-308.

[47] Qiao Zhang, Vincent Liu, Hongyi Zeng, and Arvind Krishnamurthy. 2017. High- resolution measurement of data center microbursts. In Proceedings of the 2017 Internet Measurement Conference. 78-85.

Claims

Claims

1. Method for transmitting packets in a data center network (DCN), the data center comprising a multitude of host servers, a multitude of top-of-rack, ToR, switches connected to the host servers and an optical network fabric connected to the multitude of top-of-rack switches, wherein the optical network fabric operates according to a given schedule, wherein the given schedule defines, for each time slice in a sequence of time slices, which pairs of top-of-rack switches are connected by a dedicated optical circuit established by an optical controller of the optical network fabric for said time slice, wherein a top-of-rack switch: synchronizes to another top-of-rack switch; receives a packet at an ingress port; and sends the packet to an egress port, characterized in that synchronizing comprises sending a synchronization message to said another top-of-rack switch in-band, that is via a circuit of the optical network fabric.

2. The method of claim 1, wherein the synchronization message is sent, based on the given schedule.

3. The method of claim 2, wherein the given schedule is an initial synchronization schedule, used for synchronizing at coarse accuracy, before the data center network becomes operational.

4. The method of claim 3, wherein the given schedule is an operational schedule, after the initial synchronization has terminated.

5. The method of claim 4, wherein synchronizing comprises re-synchronizing the top-of- rack switch.

6. The method of claim 5, wherein re-synchronization is performed when a current amount by which the synchronized clock of said top-of-rack switch has drifted from a master clock exceeds a pre-defined threshold T.

7. The method of claim 6, wherein the amount of drift corresponds to a sum of synchronization and drift errors of the top-of-rack switch.

8. The method of claim 7, wherein the synchronization error and the drift error are estimated, based on an empirical profile or statistics of the synchronization and drift errors of the top-of- rack switch.

9. The method of claim 8, wherein the pre-defined threshold T is determined based on an accuracy and an overhead of re-synchronization.

10. The method of claim 9, wherein the pre-defined threshold T is further determined based on a relative significance of the synchronization error and the drift error.

11. The method of claim 10, wherein, if the synchronization error dominates, T is made large to reduce a re-synchronization phase into re-synchronizing said top-of-rack switch directly with a lead top-of-rack switch, once per a cycle of the schedule.

12. The method of claim 11 , wherein, if the drift error dominates, T is made small to resynchronize each ToR multiple times in a cycle through intermediate reference ToRs before its clock drifts off.

13. The method of claim 12, wherein a ToR with a minimal amount of drift is selected as said another ToR for re-synchronization.

14. The method of claim 13, wherein a time slice and a port connected to said another top- of-rack switch are stored in a lookup table on said top-of-rack switch, the lookup table being used to direct synchronization messages in different time slices to specific ports connected to top-of-rack switches to synchronize with.

15 The method of claim 14, wherein the lookup table is preloaded into a control plane of said top-of-rack switch.

16. The method of claim 15, wherein batches of the lookup table are periodically injected into the data plane of said top-of-rack switch.

17. The method of claim 16, wherein synchronization returns an offset for adjusting a local clock of the top-of-rack switch.

18. The method of claim 17, wherein the synchronization message is a DPTP message.

19. Top-of-rack switch, implementing the method steps of one of claims 1 to 18.