Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
  • H04L41/08—Configuration management of networks or network elements
  • H04L41/0896—Bandwidth or capacity management, i.e. automatically increasing or decreasing capacities
• • H—ELECTRICITY
  • H05—ELECTRIC TECHNIQUES NOT OTHERWISE PROVIDED FOR
  • H05K—PRINTED CIRCUITS; CASINGS OR CONSTRUCTIONAL DETAILS OF ELECTRIC APPARATUS; MANUFACTURE OF ASSEMBLAGES OF ELECTRICAL COMPONENTS
  • H05K7/00—Constructional details common to different types of electric apparatus
  • H05K7/14—Mounting supporting structure in casing or on frame or rack
  • H05K7/1485—Servers; Data center rooms, e.g. 19-inch computer racks
  • H05K7/1498—Resource management, Optimisation arrangements, e.g. configuration, identification, tracking, physical location
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L12/00—Data switching networks
  • H04L12/28—Data switching networks characterised by path configuration, e.g. LAN [Local Area Networks] or WAN [Wide Area Networks]
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L12/00—Data switching networks
  • H04L12/64—Hybrid switching systems
  • H04L12/6418—Hybrid transport
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04B—TRANSMISSION
  • H04B10/00—Transmission systems employing electromagnetic waves other than radio-waves, e.g. infrared, visible or ultraviolet light, or employing corpuscular radiation, e.g. quantum communication
  • H04B10/27—Arrangements for networking

Definitions

• Networking costs are one of the major expenses; as is known, the cost associated with providing line speed communications bandwidth between an arbitrary pair of servers in a server cluster generally grows super-linearly to the size of the server cluster.
• Production data center networks use high-bandwidth links and high-end network switches to provide the needed capacity, but they are still over-subscribed (lacking capacity at times) and thus suffer from sporadic performance problems. Oversubscription is generally the result of a combination of technology limitations, the topology of these networks (e.g., tree-like) that requires expensive "big-iron” switches, and pressure on network managers to keep costs low. Other network topologies have similar issues.
• each flyway comprises an additional network communication path between network machines that may be dynamically set up as needed, and torn down as desired.
• each rack of servers in a network has an associated flyway mechanism that can communicate with other flyway mechanisms.
• the flyway mechanisms may be based on wireless or wired technologies.
• a controller dynamically provisions a flyway upon detecting a need for additional network communications capacity between the two racks by having the flyway mechanisms communicate with one another. Once provisioned, the additional communications link between machines is used to carry at least some of the network traffic between those machines.
• FIGURE 1 is a block diagram showing an example data center incorporating flyway mechanisms by which flyways may be established.
• FIG. 2 is a representation of flyways set up between network machines.
• FIG. 3 is a representation of a partial data center showing server racks and the range of an example 60GHz device mounted atop a rack near the center of a set of racks.
• FIG. 4 is a flow diagram showing example steps for provisioning a flyway between a pair of network machines.
• Various aspects of the technology described herein are generally directed towards a network design that retains the benefits of existing networks, yet substantially improves their performance.
• a hybrid architecture in which a base network (including whether tree-like or mesh-like) is provisioned for the average case (and is thus oversubscribed), but handles hotspots through the inclusion of extra links that are added on an on-demand basis to provide additional capacity where and when it is needed in the network.
• links are referred to as flyways.
• future network designs e.g., Clos networks
• the enhanced design allows networks to maintain their performance as they scale-up in server counts, without significant increase in the quantity and the cost of networking equipment.
• any of the examples described herein are non- limiting examples. As such, the present invention is not limited to any particular embodiments, aspects, concepts, structures, functionalities or examples described herein. Rather, any of the embodiments, aspects, concepts, structures, functionalities or examples described herein are non-limiting, and the present invention may be used in various ways that provide benefits and advantages in computing and computer networks in general.
• FIG. 1 shows a production network based upon a tree-like topology.
• a plurality of racks 102i-102 n each have servers, which communicate through a top of rack switch 104i-104 n .
• a typical network has twenty to forty servers per rack, with increasingly powerful links and switches going up the tree.
• flyways are not limited to treelike topologies, but can be used in any topology, including clos networks and other forms of mesh topologies, and FatTree topologies.
• each top of rack switch 104i-104 n is coupled to one another through one or more aggregation switches 106i-106 k .
• each server may communicate with any other server, including a server in a different rack.
• a higher-level aggregation switch 108 couples the rack-level aggregation switches 106i-106 k , and there may be one or more additional levels of aggregation switch couplings.
• flyways are implemented via flyway mechanisms 1 lOi-l 10 n that are controlled by a flyway controller 112.
• the flyways may be dynamically set up by the flyway controller 112 on an as- needed basis, and taken down when not needed (or needed elsewhere).
• FIG. 2 shows how one flyway 220 may be used to link racks 102i and 102 n and their respective top-of-rack switches 104i and 104 n , while another flyway 222 links racks 102 2 and 102 m and their respective top-of-rack switches 104 2 and 104 n .
• a rack / top-of-rack switch may have more than one flyway at any time, as represented by the flyway 221 between racks 102i and 102 2 . While a single flyway mechanism is shown per rack, it can be appreciated that there may be more than one flyway mechanism per rack (or multiple devices in a single flyway mechanism), possibly using different communications technologies (e.g., wireless and optical).
• top-of-rack switches are "hot," that is, they are sending and/or receiving a large volume of traffic.
• top-of-rack switches typically exchange much of their data with only a few other top-of-rack switches. This translates into skewed bottlenecks, in which just a few of the top-of-rack switches lag behind the rest and hold back the entire network.
• the flyways described herein provide extra capacity to these few top-of-rack switches and thus significantly improve overall performance. Indeed, only a few flyways, with relatively low bandwidth, significantly improve the performance of an
• Flyways may be added to a network at a relatively small additional cost. This may be accomplished by the use of wireless links (e.g., 60GHz, optical links and/or 802.1 In) and/or the use of commodity switches to add capacity in a randomized manner. In general, any flyway mechanism can link to any other flyway mechanism, as long as they meet coupling requirements (e.g., within range for wireless, has line-of-sight for optical and so on).
• wireless links e.g., 60GHz, optical links and/or 802.1 In
• commodity switches to add capacity in a randomized manner.
• any flyway mechanism can link to any other flyway mechanism, as long as they meet coupling requirements (e.g., within range for wireless, has line-of-sight for optical and so on).
• the flyways may be implemented in various ways, including via wireless links that are set up on demand between the flyway mechanisms (e.g., suitable wireless devices), and/or commodity switches that interconnect subsets of the top-of-rack switches.
• the flyway mechanisms e.g., suitable wireless devices
• commodity switches that interconnect subsets of the top-of-rack switches.
• 60 GHz wireless technology is one implementation for creating the flyways, as it supports short range (1-10 meters), high-bandwidth (lGbps) wireless links. Further, the high capacity and limited interference range of 60GHz provides benefits.
• flyway controller 112 gathers demand data, adapts the flyways in a dynamic manner, and switches paths to route traffic.
• Wireless flyways are able to form links on demand, and thus may be used to distribute the available capacity to whichever top-of-rack switch pairs need it as determined by the central flyway controller 112.
• Wired flyways provide equivalent benefits; when inexpensive switches are connected to subsets of the top-of-rack switches, the limited backplane bandwidth at these switches can be divided among whichever of the many top-of-rack switch pairs that are connected need it.
• Wired flyways are somewhat more restrictive in that only if a top-of-rack switch pair happen to be connected via one of the flyway switches, can they directly benefit from a flyway (although multi-hop flyways are one alternative that allows indirect benefits, as described below).
• flyways are more likely to keep up with wired speeds (for example, as NICs go up to lOGbps and links go to 40Gbps). In any event, flyways provide benefits relative to the alternative of spreading the same bandwidth across all racks, as much of it will go unused on links that do not need it. Further, flyways can be deployed today on existing topologies of production data centers. [0023] To add flyways to provide extra capacity between hot top-of-rack switches and some other top-of-rack switches with which they exchange traffic, pairs of flyway mechanisms that provide the most speedup may be selected. Each flyway also needs sufficient capacity.
• flyways With respect to how much capacity each flyway needs, consider an example of adding flyways between the top ten top-of-rack switches and each of the five other top-of- rack switches with which they exchange the most data (i.e., a total of fifty flyways). Most flyways need less than ten percent the top-of-rack switch's uplink bandwidth to be useful; one reason is that while the top-of-rack switch's uplink carries traffic to all of the other top-of-rack switches, a flyway has to only carry traffic to one other top-of-rack switch.
• flyways generally results from application characteristics that cause sparse demand matrices. Though flyways may not provide benefits for all demands, a large set of practical applications benefit from flyways. For example, in data centers that support web services, the request traffic is load balanced across servers, each of which in turn assembles a response page by possibly asking a few other servers to generate parts of the response (e.g., advertisements).
• the reduce part of map-reduce in data mining jobs is possibly a worst case scenario, with each reducer pulling data from all the mappers; the job is bottlenecked until all the reducers complete. Even then, it is rare to have so many mappers and reducers that all top-of-rack switches are congested simultaneously.
• the 60 GHz frequency band is a 7 GHz wide band (57-64 GHz) that was set aside by the FCC in 2001 for use by unlicensed devices.
• the channel bandwidth specified by ISO and IEEE is 2.160 GHz.
• the substantially larger bandwidth in the 60 GHz band facilitates higher capacity links. For example, a simple encoding scheme that achieves as little as 1 bps/Hz makes it possible to construct a link with a nominal bandwidth of 7 Gbps.
• 60 GHz devices will likely become spectrally efficient and seven orthogonal 1 Gbps channels will be common, and 1 Gbps link speeds are adequate for constructing flyways. Note that 4-5 Gbps link speeds are already available using multiple channels in the 60GHz band.
• 60 GHz devices have a relatively short range (5 to 10 meters). In wireless communications path, loss is directly proportional to the square of the frequency and so 60 GHz signals attenuate more quickly with distance when compared with 2.4 and 5 GHz Wi-Fi signals. For another, highly directional 60 GHz communications are easy to achieve. Because signal wavelength is inversely proportional to frequency, 60 GHz signals have very short wavelengths (5 mm) and this allows RF chip designers to incorporate antenna arrays into the transceiver chips. The ability to integrate many antennas and transceivers with proper phasing into a single chip allows beam-forming and hence directional communication.
• phased-array antennas boost signal quality leading to a cleaner signal with low bit error rates at short distances.
• a one-square inch antenna provides a signal boost of 25dBi at 60 GHz.
• a side benefit of the integration of the antennas on the chip is that this avoids the need for wires to carry signals to and from the chip, reducing the cost of packaging by up to two orders of magnitude.
• the low range and high capacity directional links are beneficial in data centers where server racks are typically packed tightly as physical space is a premium resource, and mobility and a power draw of approximately ten watts per server rack is a non-issue.
• Directionality allows the network designers to increase the overall spectrum efficiency through spatial reuse. For example, two sets of communications between four top-of-rack switches can occur simultaneously because of directionality and range.
• racks are about 24 inches wide and are arranged in a dense, regular pattern as shown in FIG. 3.
• each box represents a 24x48 inch rack, for example, arranged in rows often.
• the circle represents the 10m range of a 60 GHz device mounted atop a rack in the center, which contains about 70 other racks.
• small (2-3 cubic inches) 60 GHz network cards mounted on top of the racks can instantiate multiple non-interfering gigabit flyways between several racks on an as needed basis.
• Electronically steerable phased-array antennas with beam widths of about 60 degrees can be steered with millisecond latency.
• NLOS non-line-of-sight
• the 60 GHz band is available world-wide.
• chipsets and devices are usable all over the world and providing economies of scale and pushing the cost down.
• Wired flyways may be constructed by using additional switches, such as of the same make as contemporary top-of-rack switches that interconnect random subsets of the top-of-rack switches, e.g., commodity switches may interconnect twenty top-of-rack switches with lGbps links each. To keep links short, the flyway switches may connect racks that are near to one another in the datacenter. When deploying wired flyways, spectrum allocation or interference is not an issue. However, their random construction constrains wired flyways; e.g., top-of-rack switch pairs that exchange a lot of traffic and can benefit from surplus capacity may end up without a wired flyway.
• either wired or wireless flyway construction is an improvement compared to dividing the same amount of bandwidth uniformly across all pairs of top-of- rack switches. Rather than spread bandwidth uniformly and have much of it wasted, as happens when the demand matrix is sparse, flyways provide a way to use the spare bandwidth to target the parts of the demand matrix that can benefit the most from surplus capacity.
• the central flyway controller 1 12 gathers estimates of demands between pairs of the rack switches. For example, information can be gathered from lightweight instrumentation at the end servers themselves, or by polling SNMP counters at the switches. Using these estimates, the controller 1 12 may periodically or otherwise run a placement algorithm (described below) to place the available flyways.
• the topology of a flyway -based network is thus dynamic, and requires multipath routing. Solutions to similar problems are known.
• the controller 112 determines how much of the traffic between a pair of top-of-rack switches go along the base network, or instead take a flyway from the sending top-of-rack switch to the receiving top-of-rack switch, if one exists.
• the top-of-rack switch splits traffic as per this ratio by assigning different flows onto different MPLS (Multiprotocol Label Switching) label switched paths.
• MPLS Multiprotocol Label Switching
• rij is the rate achieved for the top-of-rack switch pair i, j and r- j is the portion of that pair's traffic on link /.
• step 402 represents gathering the network traffic data for the pairs being considered.
• step 404 represents gathering the network traffic data for the pairs being considered.
• the most optimal flyway that is needed e.g., the switches are hot but a flyway is not already provisioned
• top-of-rack switches top-of-rack switches
• a flyway also may be used to provide connectivity to racks whose
• Such a failure-compensating flyway may be considered the most optimal flyway that is needed at step 404, for example, or may be separately considered, such as before selecting others based upon capacity.
• Step 406 represents evaluating whether that most optimal flyway can be provisioned; for example, if the total number of flyways is a constraint, there may not be i 0 another flyway available at this time. However, if there is not a flyway available, yet the most optimal non-provisioned flyway is more needed that another flyway that is currently provisioned, step 408 may tear down that other flyway (or flyways, if a multi-hop scenario is in use as described below) so that a flyway becomes available. For example, the least needed flyway may be torn down.
• Step 410 provisions the most optimal flyway (if appropriate). Once provisioned, the flyway may be used to handle at least some of the network traffic between the selected pair, as represented by step 412. The process then repeats. Note that FIG. 4 is only one example; as an alternative, up to the top-N most optimal flyways that are not already provisioned may be added between each polling of the pairs at step 402.
• flyways With flyways, the physical topology of a network needs to be considered, as well as constraints due to the technology. Wireless flyways are constrained by range, while wired flyways, which are constructed by inter-connecting random subsets of top-of-rack switches, can only provide surplus capacity between these random subsets. 60 GHz flyways span a distance often meters and use the datacenter layout (e.g., FIG. 3) from a production data-center. For wired flyways, 24 port, lGbps switches may be used, with some of the ports used for the flyway channel. Although such constraints impact the benefit of flyways, the gains are still significant.
• wireless flyways need to be added to obtain the same benefits of wireless flyways. For example, when fifty 24-port switches are added, 1,200 duplex links are added to the network. Wireless flyways provide equivalent performance with only 50 half-duplex links. Thus is because wireless flyways are added in a targeted manner: they help speed up those pairs of top-of-rack switches that need additional capacity. Wired flyways are added at random, and benefit pairs only if they happen to be among the ones chosen.
• Multi-hop flyways are feasible. Consider that a machine in FIG. 3 within the circle wants to communicate with a machine outside of the circle. However, a second hop from a machine inside the circle that is close enough to another machine can be used to accomplish the communication.
• the controller 1 12 can consider whether two machines (e.g., A and Z) are outside of single hop range. If so, a flyway may be constructed between two other machines (e.g., C and M) that are close but communicating via the top-of-rack switches, with their communication switched to the flyway. The bandwidth savings provided by the newly constructed flyway for C and M then may be used to free up the top-of-rack switches to allow the machines A and Z to communicate using the top-of-rack switches.
• two machines e.g., A and Z
• C and M machines that are close but communicating via the top-of-rack switches, with their communication switched to the flyway.
• the bandwidth savings provided by the newly constructed flyway for C and M then may be used to free up the top-of-rack switches to allow the machines A and Z to communicate using the top-of-rack switches.

Landscapes

• Engineering & Computer Science (AREA)
• Computer Networks & Wireless Communication (AREA)
• Signal Processing (AREA)
• Computer Hardware Design (AREA)
• General Engineering & Computer Science (AREA)
• Microelectronics & Electronic Packaging (AREA)
• Mobile Radio Communication Systems (AREA)
• Data Exchanges In Wide-Area Networks (AREA)
• Computer And Data Communications (AREA)
• Small-Scale Networks (AREA)
• Traffic Control Systems (AREA)

Priority Applications (4)

Applications Claiming Priority (4)

Publications (2)

Family

ID=43855715

Family Applications (1)

Country Status (6)

Cited By (2)

Families Citing this family (61)

Citations (1)

Family Cites Families (26)

• 2010
  • 2010-03-15 US US12/723,697 patent/US8972601B2/en active Active
  • 2010-10-06 EP EP10822642.4A patent/EP2486702B1/en active Active
  • 2010-10-06 KR KR1020127009062A patent/KR101786131B1/ko active Active
  • 2010-10-06 WO PCT/US2010/051698 patent/WO2011044288A2/en not_active Ceased
  • 2010-10-06 JP JP2012533293A patent/JP5632923B2/ja active Active
  • 2010-10-06 CN CN201080045182.4A patent/CN102549972B/zh active Active

Patent Citations (1)

Also Published As

Similar Documents

Legal Events