Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L45/00—Routing or path finding of packets in data switching networks
  • H04L45/74—Address processing for routing
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L12/00—Data switching networks
  • H04L12/28—Data switching networks characterised by path configuration, e.g. LAN [Local Area Networks] or WAN [Wide Area Networks]
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
  • G06F1/26—Power supply means, e.g. regulation thereof
  • G06F1/32—Means for saving power
  • G06F1/3203—Power management, i.e. event-based initiation of a power-saving mode
  • G06F1/3234—Power saving characterised by the action undertaken
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
  • G06F12/02—Addressing or allocation; Relocation
  • G06F12/0223—User address space allocation, e.g. contiguous or non contiguous base addressing
  • G06F12/0284—Multiple user address space allocation, e.g. using different base addresses
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
  • G06F13/14—Handling requests for interconnection or transfer
  • G06F13/20—Handling requests for interconnection or transfer for access to input/output bus
  • G06F13/24—Handling requests for interconnection or transfer for access to input/output bus using interrupt
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
  • G06F13/38—Information transfer, e.g. on bus
  • G06F13/40—Bus structure
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
  • G06F3/0601—Interfaces specially adapted for storage systems
  • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
  • G06F3/0604—Improving or facilitating administration, e.g. storage management
  • G06F3/0605—Improving or facilitating administration, e.g. storage management by facilitating the interaction with a user or administrator
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
  • G06F3/0601—Interfaces specially adapted for storage systems
  • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
  • G06F3/0629—Configuration or reconfiguration of storage systems
  • G06F3/0631—Configuration or reconfiguration of storage systems by allocating resources to storage systems
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
  • G06F3/0601—Interfaces specially adapted for storage systems
  • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
  • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F9/00—Arrangements for program control, e.g. control units
  • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
  • G06F9/46—Multiprogramming arrangements
  • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
  • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
  • G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
  • G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L45/00—Routing or path finding of packets in data switching networks
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L45/00—Routing or path finding of packets in data switching networks
  • H04L45/60—Router architectures
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L45/00—Routing or path finding of packets in data switching networks
  • H04L45/74—Address processing for routing
  • H04L45/745—Address table lookup; Address filtering
  • H04L45/74591—Address table lookup; Address filtering using content-addressable memories [CAM]
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L49/00—Packet switching elements
  • H04L49/10—Packet switching elements characterised by the switching fabric construction
  • H04L49/109—Integrated on microchip, e.g. switch-on-chip
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L49/00—Packet switching elements
  • H04L49/15—Interconnection of switching modules
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L49/00—Packet switching elements
  • H04L49/20—Support for services
  • H04L49/201—Multicast operation; Broadcast operation
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L49/00—Packet switching elements
  • H04L49/25—Routing or path finding in a switch fabric
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L49/00—Packet switching elements
  • H04L49/25—Routing or path finding in a switch fabric
  • H04L49/253—Routing or path finding in a switch fabric using establishment or release of connections between ports
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L49/00—Packet switching elements
  • H04L49/30—Peripheral units, e.g. input or output ports
  • H04L49/3009—Header conversion, routing tables or routing tags
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L49/00—Packet switching elements
  • H04L49/35—Switches specially adapted for specific applications
  • H04L49/351—Switches specially adapted for specific applications for local area network [LAN], e.g. Ethernet switches
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L49/00—Packet switching elements
  • H04L49/35—Switches specially adapted for specific applications
  • H04L49/356—Switches specially adapted for specific applications for storage area networks
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L47/00—Traffic control in data switching networks
  • H04L47/10—Flow control; Congestion control
• • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
  • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
  • Y02B—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO BUILDINGS, e.g. HOUSING, HOUSE APPLIANCES OR RELATED END-USER APPLICATIONS
  • Y02B70/00—Technologies for an efficient end-user side electric power management and consumption
  • Y02B70/10—Technologies improving the efficiency by using switched-mode power supplies [SMPS], i.e. efficient power electronics conversion e.g. power factor correction or reduction of losses in power supplies or efficient standby modes
• • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
  • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
  • Y02B—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO BUILDINGS, e.g. HOUSING, HOUSE APPLIANCES OR RELATED END-USER APPLICATIONS
  • Y02B70/00—Technologies for an efficient end-user side electric power management and consumption
  • Y02B70/30—Systems integrating technologies related to power network operation and communication or information technologies for improving the carbon footprint of the management of residential or tertiary loads, i.e. smart grids as climate change mitigation technology in the buildings sector, including also the last stages of power distribution and the control, monitoring or operating management systems at local level
• • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
  • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
  • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
  • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
• • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
  • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
  • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
  • Y02D30/00—Reducing energy consumption in communication networks

Definitions

• the disclosure relates generally to a switching fabric for a computer-based system.
• Figures 1 A and IB show a classic data center network aggregation as is currently well known.
• Figure 1A shows a diagrammatical view of a typical network data center architecture 100 wherein top level switches 101 a-n are at the tops of racks 102a-n filled with blade servers 107a-n interspersed with local routers 103a-f. Additional storage routers and core switches. 105a-b and additional rack units 108a-n contain additional servers 104 e-k and routers 106a-g
• Figure lb shows an exemplary physical view 110 of a system with peripheral servers 11 la-bn arranged around edge router systems 112a-h, which are placed around centrally located core switching systems 113. Typically such an aggregation 110 has 1-Gb Ethernet from the rack servers to their top of rack switches, and often 10 Gb Ethernet ports to the edge and core routers.
• Figures 1 A and IB illustrate a typical data center system
• Figure 2 is an overview of a network aggregation system
• Figure 3 illustrates an overview of an exemplary data center in a rack system
• Figure 4 illustrates a high-level topology of a network aggregating system
• Figure 5 A illustrates a block diagram of an exemplary switch of the network aggregation system
• Figure 5B illustrates the MAC address encoding
• Figure 6 illustrates a first embodiment of a broadcast mechanism of the network aggregation system
• Figure 7 illustrates an example of unicast routing of the network aggregation system
• Figure 8 illustrates an example of fault-resistant unicast routing of the network aggregation system
• Figure 9 illustrates a second embodiment of a broadcast mechanism of the network aggregation system.
• the disclosure is particularly applicable to a network aggregation system and method as illustrated and described below and it is in this context that the disclosure will be described. It will be appreciated, however, that the system and method has greater utility since the system and method can be implemented using other elements and architectures that are within the scope of the disclosure and the disclosure is not limited to the illustrative embodiments described below.
• the system and method also supports a routing using a tree-like or graph topology that supports multiple links per node, where each link is designated as an Up, Down, or Lateral link, or both, within the topology.
• each node in the system maybe be a combination computational/switch node, or just a switch node, and input outpout (I O) can reside on any node as described below in more detail.
• the system may also provide a system with a segmented Ethernet Media Access Control (MAC) architecture which may have a method of re-purposing MAC IP addresses for inside MACs and outside MACs, and leveraging what would norraally be the physical signaling for the MAC to feed into the switch.
• the system may also provide a method of non-spoofing communication, as well as a method of fault-resilient broadcasting,
• WESTO22615071.1 which may have a method of imicast misrouting for fault resilience.
• a spoofing attack is a situation in which one person or program successfully masquerades as another by falsifying data and thereby gaining an illegitimate advantage.
• the system may also provide a rigorous security between the management processors, such that management processors can "trust" one another.
• the management processors there is a management processor within each SoC (the M3 microcontroller, block 906, figure 5A).
• the software running on the management processor is trusted because a) the vendor (in this case Smooth-Stone) has developed and verified the code, b) non- vendor code is not allowed to run on the processor. Maintaining a Trust relationship between the management processors allow them to
• the system may also provide a network proxy that has an integrated microcontroller in an always-on power domain within a system on a chip (SOC) (hat can take over network proxying for the larger onboard processor, and which may apply to a subtree.
• SOC system on a chip
• the system also provide a multi-domaining technique that can dramatically expand the size of a routable fat tree like structure with only trivial changes to the routing header and the routing table.
• FIG. 2 illustrates a network aggregation system 300.
• the network aggregation supports one or more high speed links 301 (thick lines), such as a 10-Gb/sec Ethernet communication, that connect an aggregation router 302 and one or more racks 303, such as three racks 303a-c as shown in Figure 3.
• the network aggregation system provides multiple highspeed 10 Gb paths, represented by thick lines, between one or more Smooth-Stone computing unit 306a-d, such as server computers, on shelves within a rack. Further details of each Smooth- Stone computing unit are described in more detail in U.S. Provisional Patent Application Serial No.
• An embedded switch 306a-d in the Smooth-Stone computing units can replace a top-of-rack switch, thus saving a dramatic amount of power and cost, while still providing a 10 Gb Ethernet port to the aggregation router 302.
• the network aggregation system switching fabric can integrate traditional Ethernet (1 Gb or 10 Gb) into the XAUI fabric, and the
• WEST ⁇ Z22615071.1 Smooth-Stone computing units can act as a top of rack switch for third-party Ethernet connected servers.
• a middle rack 303b illustrates another configuration of a rack in the network aggregation system in which one or more Smooth-Stone computing units 306e, f can integrate into existing data center racks that already contain a top-of-rack switch 308a.
• the IT group can continue to have their other computing units connected via 1 Gb Ethernet up to the existing top- of-rack switch and the internal Smooth-Stone computing units can be connected via 10 Gb XAUI fabric and they can integrate up to the existing top-of-rack switch with either a 1 Gb or 10 Gb Ethernet interconnects as shown in Figure 2.
• a third rack 303c illustrates a current way that data center racks are traditionally deployed. The thin red lines in the third rack 303c represent 1 Gb Ethernet.
• FIG 3 shows an overview of an exemplary "data center in a rack” 400 according to one embodiment of the system.
• the "data center in a rack” 400 may have 10-Gb Ethernet PHY 401a-n and 1-Gb private Ethernet PHY 402.
• Large computers (power servers) 403a-n support search; data mining; indexing; Apache Hadoop, a Java software framework; Map educe, a software framework introduced by Google to support distributed computing on large data sets on clusters of computers; cloud applications; etc.
• Computers (servers) 404a-n with local flash and/or solid-state disk (SSD) support search, MySQL, CDN, software-as-a-serviee (SaaS), cloud applications, etc.
• Data center 400 has an array 406 of hard disks, e.g., in a Just a Bunch of Disks (JBOD) configuration, and, optionally, Smooth-Stone computing units in a disk form factor (for example, the green boxes in arrays 406 and 407), optionally acting as disk controllers.
• Hard disk servers or SS disk servers may be used for web servers, user applications, and cloud applications, etc.
• an array 407 of storage servers and historic servers 408a, b (any size, any vendor) with standard Ethernet interfaces for legacy applications.
• the data center in a rack 400 uses a proprietary system interconnect approach that dramatically reduces power and wires and enables heterogeneous systems, integrating existing
• a complete server or storage server is put in a disk or SSD form factor, with 8-16 SATA interfaces with 4
• ServerNodesTM and 8 PCIe x4 interfaces with 4 ServerNodesTM. It supports disk and/or SSD + ServerNode tM , using a proprietary board paired with a disk(s) and supporting Web server, user applications, cloud applications, disk caching, etc.
• the Smooth-Stone XAUI system interconnect reduces power, wires and the size of the rack. There is no need for high powered, expensive Ethernet switches and high-power Ethernet Phys on the individual servers. It dramatically reduces cables (cable complexity, costs, significant source of failures). It also enables a heterogeneous server mixture inside the rack, supporting any equipment that uses Ethernet Or SATA or PCIe. It can be integrated into the system interconnect.
• the SOC is not a fully functionally equivalent to an industry-standard network switch, such as, for example, a Cisco switch or router. But for certain applications discussed throughout this document, it offers a better price/performance ratio as well as a power/performance ratio. It contains a layer 2 packet switch, with routing based on source / destination MAC addresses. It further supports virtual local area network (VLAN), with configurable VLAN filtering on domain incoming packets to minimize unnecessary traffic in a domain.
• VLAN virtual local area network
• the embedded MACs within the SOC do have complete VLAN support providing VLAN capability to the overall SOC without the embedded switch explicitly having VLAN support. It can also wake up the system by management processor notifying the management processor on link state transitions to reprogram routing configurations to route around faults. Such functionality does not require layer3 (or above) processing (i.e., it is not a router). It also does not offer complete VLAN support, support for QoS / CoS, address learning, filtering, spanning tree protocol (STP), etc.
• FIG 4 shows a high-level topology 800 of the network system that illustrates XAUI connected SoC nodes connected by the switching fabric.
• the 10 Gb Ethernet ports EthO 801a and Ethl 801b come from the top of the tree.
• Ovals 802a-n are Smooth-Stone nodes that comprise both computational processors as well as the embedded switch.
• the nodes have five XAUI links connected to the internal switch.
• the switching layers use all five XAUI links for switching.
• WEST ⁇ 22261 5071.1 can be used as XAUI, 10 Gb Ethernet, PCIe, S ATA, etc., for attachment to I O.
• the vast majority of trees and fat trees have active nodes only as leaf nodes, and the other nodes are pure switching nodes. This approach makes routing much more straightforward.
• Topology 800 has the flexibility to permit every node to be a combination computational and switch node, or just a switch node. Most tree-type implementations have I/O on the leaf nodes, but topology 800 let the I/O be on any node. In general, placing the Ethernet at the top of the tree minimizes the average number of hops to the Ethernet.
• FIG. 6 illustrates one example
• FIG. 6 When looking at a conventional implementation of a topology e.g. in Figure 6, usually computing nodes are found in the lower level leaf nodes (e.g. NO0-NO8), and the upper level nodes don't have computing elements but are just network switching elements (N10-N21). With the node architecture shown in Figure 6A, the A9 Cores (905) may be optionally enabled, or could be just left powered-off. So the upper level switching nodes (N10-N21) in Figure 6 can be used as pure switching elements (like traditional implementations), or we can power on the A9 Cores module and use them as complete nodes within the computing cluster.
• the lower level leaf nodes e.g. NO0-NO8
• N10-N21 just network switching elements
• FIG. 5a shows a block diagram of an exemplary switch 900 according to one aspect of the system and method disclosed herein. It has four areas of interest 10a-d. Area 910a corresponds to Ethernet packets between the CPUs and the inside MACs. Area 910b corresponds to Ethernet frames at the Ethernet physical interface at the inside MACs* that contains the preamble, start of frame, and inter-frame gap fields. Area 10c corresponds to Ethernet frames at the Ethernet physical mterface at the outside MAC, that contains the preamble, start of frame, and inter-frame gap fields..
• Area 910d corresponds to Ethernet packets between the processor of routing header 901 and outside MAC 904.
• This segmented MAC architecture is asymmetric.
• the inside MACs have the Ethernet physical signaling interface into the routing header processor, and the outside MAC has an Ethernet packet interface into the routing header processor.
• the MAC IP is re-purposed for inside MACs and outside MACs, and what would normally be the physical signaling for the MAC to feed into the switch is leveraged.
• MAC configuration is such that the operating system device drivers of A9 cores 905
• WESTM22615071 .1 manage and control inside EthO MAC 902 and inside ETHl MAC 903,
• the device driver of management processor 906 manages and controls Inside Eth2 MAC 907.
• Outside Eth MAC 904 is not controlled by a device driver.
• MAC 904 is configured in Promiscuous mode to pass all frames without any filtering for network monitoring. Initialization of this MAC is coordinated between the hardware instantiation of the MAC and any other necessary management processor initialization.
• Outside Eth MAC 904 registers are visible to both A9 905 and management processor 906 address maps. Interrupts for Outside Eth MAC 904 are routable to either the A9 or management processor.
• the XGMAC supports several interruptible events that the CPUs may want to monitor, including any change in XGMII link fault status, hot-plugging or removal of PHY, alive status or link status change, and any RMON counter reaching a value equal to the threshold register.
• the routing frame header processor may standardize these fields.
• the XAUI interface may need some or all of these fields.
• the routing header processor at area 910d needs to add these going into the switch, and to remove them leaving the switch.
• these three fields may be removed (if the XAUI interface allows it).
• the routing header processor at area 910b will need to strip these going into the switch, and add them back leaving the switch.
• the routing frame header processor receives an Ethernet frame from a MAC, sending a routing frame to the switch. It also standardizes the preamble, start of frame, and inter-frame gap fields, prepends a routing header, and receives a routing frame from the switch, sending the Ethernet frame into a MAC. This processor then strips the routing header and standardizes the preamble, start of frame, and inter-frame gap fields. Note that all frames that are flowing within the fabric are routing frames, not Ethernet frames. The Ethernet frame / routing frame conversion is done only as the packet is entering or leaving the fabric via a MAC. Note also that the routing logic within the switch may change fields within the routing frame. The Ethernet frame is never modified (except the adding / removing of the preamble, start of frame, and inter-frame gap fields).
• the routing frame is composed of the routing frame header plus the core part of the
• Ethernet frame and is structured as shown in Table 1 , below:
• bit sizing is 4096 nodes 12 bit node IDs. These fields may be resized during implementation as needed.
• the routing frame header consists of the fields shown in Table 2, below:
• a switch If a switch receives a packet that fails the checksum, the packet is dropped, a statistic counter is incremented, and the management processor is notified.
• the routing frame processor differentiates between several destination MAC address encodings. As a reminder, MAC addresses are formatted as shown in Figure 5b. The following table describes the usage of the 3 byte OUI and 3 byte NIC specific field within the MAC
• WESTT222615071.1 address One of the novel aspects of the system and method disclosed herein is the use of additional address bits to encode an internal to external MAC mapping, as shown also in the Table 3, below, in the second entry under "Fabric Internal Node local address Hits MAC Lookup CAM".
• SS MAC LINK ENCODED MAGIC are constant identifiers used for uniquely identifying these MAC address types.
• the term "magic number” is a standard industry term for a constant numerical or text value used to identify a file format or protocol. These magic numbers are configured in two registers (magicNodeEneodedMAC and magicLinkEncodedMAC that default to standard values during hardware initialization, but allow the management processor software to change them if necessary.
• the header processor contains a MAC Lookup CAM (Content Addressable Memory), macAddrLookup, that maps from 6 byte MAC addresses to 12-bit Node IDs, as shown in Table 4, below.
• MAC Lookup CAM Content Addressable Memory
• macAddrLookup maps from 6 byte MAC addresses to 12-bit Node IDs, as shown in Table 4, below.
• the number of rows in this CAM is implementation dependent, but would be expected to be on the order of 256-1024 rows.
• the management processor initializes the CAM with Node ID mappings for all the nodes within the SS fabric. There are two types of rows, depending upon the setting of the Node Local bit for the row.
• the Node Local field allows a 4: 1 compression of MAC addresses in the CAM for default MAC addresses, mapping all four MACs into a single row in the CAM table, which is Table 5, below.
• WESTO2261S07 I The arbitrary rows in the CAM allow mapping of the MAC address aliases to the nodes. Linux (and the MACs) allow the MAC addresses to be reassigned on a network interface (e.g., with ifconfig ethO hw ether 00:80:48:BA:dl :30). This is sometime used by virtualization / cloud computing to avoid needing to re-ARP after starting a session.
• the switch architecture provides for a secondary MAC Lookup CAM that only stores the 3 bytes of the NIC Specific part of the MAC address for those addresses that match the Switch OUI.
• the availability of this local OUI CAM is determined by the implementation. See Table 6, below.
• the maximum number of nodes limitation for three types of MAC address encodings may be evaluated as follows;
• Default MAC Addressees management processor sets Node Local mappings for each of the nodes in the fabric. There is one entry in the CAM for each node. Max # of nodes is controlled by maximum # of rows in the MAC Address Lookup CAM.
• Node Encoded Addresses All the MACs are reprogrammed to use Node Encoded Addresses. In this way the Node IDs are directly encoded into the MAC addresses. No entries in the MAC Lookup CAM are used. Max # of nodes is controlled by maximum # of rows in the Unicast lookup table (easier to make big compared to the Lookup CAM). Note that this also gives us some risk mitigation in case the MAC Lookup CAM logic is busted. Provides use case for the node encoded addresses idea.
• Arbitrary MAC Address Aliases Takes a row in the CAM. As an example, a 512-row CAM could hold 256 nodes (Node local addresses) + 1 MAC address alias per node.
• the management processor Since the Lookup CAM is only accessed during Routing Header creation, the management processor actually only needs to populate a row if the MAC address witiiin the fabric is being used as a source or destination MAC address within a packet. In other words, if two nodes never will talk to each other, a mapping row does not need to be created. But usually
• WESTO2261 5071.1 the management processor won't have that knowledge, so it's expected that mappings for all nodes are created in all nodes. Also note that even if an entry is not created in the Lookup CAM, the routing will actually still succeed by routing the packet out the Ethernet gateway, through an external router, back into the Fabric, to the destination node.
• Table 7 defines how to set fields within the Routing Header for all the fields except for destination node and port.
• Table 8 defines how to set destination node and port for addresses within the fabric:
• Table 9 defines how to set destination node and port for addresses outside the fabric:
• management processor software architecture of the system and method disclosed here currently depends on the ability of management processor nodes to "trust" each other. This more rigorous security on management processor to management processor communication is desirable, as well a better security on private management LANs across the fabric.
• This fabric issue may be mitigated by simply defining, for environments that require multiple "hard” security domains, that customers simply don't mix security domains within a fabric. In such cases, it may be possible to connect 14-node boards to the top of rack switch, allowing customers to have VLAN granularity control of each 14-node board.
• the multi-domain fabric architecture addresses the lack of VLAN support by creating secure "tunnels" and domains across the fabric, and it can interoperate with VLAN protected router ports on a 1:1 basis.
• the approach to domain management in the system and method disclosed here is as follows: Support multiple domain IDs within the fabric. Allow each of the MACs within a node (management processor, MACO, MAC1, Gateway) to be assigned to a domain ID individually (and tagged with domain 0 if not set). Allow each of the MACs within a node to have a bit indicating access to the management domain.
• the domain IDs associated with a MAC could only be assigned by the management processor, and could not be altered by the ⁇ .
• the routing frame processor would tag the routing frame with the domain ID and management domain state associated with that MAC. Domains would provide the effect of tunnels or VLANs, in that they keep packets (both unicast and
• WEST ⁇ 222615071. 1 multicast within that domain, allowing MACs outside that domain to be able to neither sniff or spoof those packets. Additionally, this approach would employ a five-bit domain ID. It would add options to control domain processing, such as, for example, a switch with a boolean per MAC that defines whether packets are delivered with non-defined (i.e., zero) domain ID, or a switch that has a boolean per MAC that defines whether packets are delivered with defined (non-zero) but non-matching domain IDs. A further option in the switch could turn off node encoded MAC addresses per MAC (eliminating another style of potential attack vector).
• the management domain bit on all management processor MACs could be marked.
• the management processor should route on domain 1 (by convention).
• domain 1 by convention.
• Such a technique allows all the management processor's to tunnel packets on the management domain so that they cannot be inspected or spoofed by any other devices (inside or outside the fabric), on other VLANs or domains.
• a gateway MAC that has the management domain bit set could be assigned, keeping management packets private to the management processor domain.
• the switch fabric could support "multi-tenant" within itself, by associating each gateway MAC with a separate domain.
• each gateway MAC could connect to an individual port on an outside router, allowing that port to be optionally associated with a VLAN. As the packets come into the gateway, they are tagged with the domain ID, keeping that traffic private to the MACs associated with that domain across the fabric.
• the switch supports a number of registers (aka CSRs, aka MMRs) to allow software or firmware to control the switch.
• CSRs aka MMRs
• MMRs magnetic resonance registers
• the actual layout of these registers will be defined by the implementation.
• the fields listed in Table 10 are software read/write. All these registers need to have a mechanism to secure them from writing from the A9 (could be secure mode or on a management processor private bus).
• nodeRangeEnable 1 bit Enables the expanded Node ED matching of
• nodeRangeHi 12 bits Enabled with nodeRangeEnable Specifies high node ID of node range match.
• nodeRangeLo 12 bits Enabled with nodeRangeEnable Specifies low node ID of node range match.
• Array elements are the Ports enumeration (management processor, MACO, MACl . OUT).
• promiscuousPortVec 4 bits Can be configured for Promiscuous Mode allowing traffic on one or more links to be snooped by the management processor or A9s in order to collect trace data or to implement an Intruder Detection System (IDS).
• IDS Intruder Detection System
• routeForeignMAC sOut 1 bit When enabled, a MAC address that does not contain a myOUI address, will not check the MAC lookup CAM, and will get treated as a MAC lookup CAM miss, thus getting routed to the gateway port. This saves latency in the common case of not populating the CAM with foreign MAC aliases.
• the registers shown in Table 11 are contained within the Switch implementation, but need not be software accessible.
• Figure 6 shows an exemplary broadcast mechanism 1000 according to one aspect of the system and method disclosed herein.
• the link between nodes N10100I and N21 1002 is down, as indicated by the dashed line 1003.
• the source node puts an incremented broadcast ID for that source node in the routing frame
• Multicast II rframe.rfType NeighborMulticast
• N04 1004 initiates a broadcast to all neighbors, i.e., Nl I 1105.
• Nl 1 has not seen the packet, so it broadcasts to all non-incoming neighbors, which, in this example, are N21 1002, N20 1006, N03 1007, and N05 1008, and accepts the packet internally.
• Nodes N03 and N05 haven't seen the packet, so they accept the broadcast internally and are done.
• N21 hasn't seen the packet, so it broadcasts the packet to all active, non-incoming links (e.g., Nl 0, N12 1009), and accepts the packet internally.
• N20 broadcasts the packet to all active* non-incoming links (i.e., N12), and accepts the packet internally.
• N10 broadcasts down to N00 1010, N01 101 1 , and N02 1012.
• N12 rebroadcasts to N06 1013, N07 1014, N0S 1015 and to one of N21 and N20 (the one it didn't get the broadcast packet from).
• N20 and N21, and N12 see the packet twice. They take action only on their first instance, the secondary times it hits the broadcast CAM as a duplicate, and the packet is ignored.
• Unicast routing (as shown in Figure 7) is responsible for routing non-multicast (i.e. unicast) packets to the next node. This is done by utilizing a software computed unicastRoute[] next node routing table that provides a vector of available links to get to the destination node.
• the packet will be routed upward until a common parent of (source, destination) is reached. This upward routing can be deterministic, oblivious, or adaptive. The packet is then routed downward to the destination using deterministic routing.
• Figure 7 illustrates a packet routing from node N00 1010 to N08 1015.
• the packet is routed in the upward phase to the common ancestor (N21 ) through node 10 1001, and then a descent phase to the destination.
• the first candidate link could be chosen deterministicaHy, or an adaptive algorithm could dynamically select either of the links. But, once the node reaches the common ancestor and turns downward, there are no redundant paths (in general) for the node to reach the destination.
• link weights should represent:
• the adaptive register is checked to determine whether to do adaptive or deterministic routing.
• o adaptive— 0 indicates that deterministic routing is to be used, so the first link is chosen from the prioritized candidate list
• the switch implementation will choose an algorithm for adaptively choosing the target link from the prioritized candidate list. This adaptive algorithm could be as simple as round-robin around the list. Alternatively, may choose to factor in other attributes e.g. FIFO free depth, link speed, ...
• An implementation option could be to add a register option to allow the router to adaptively choose from all non-zero weights, or to only adaptively choose from the highest priority candidate lists.
• links can be adaptively chosen from redundant links so it can be straightforward to avoid a link with the normal adaptive link selection.
• Figure 8 illustrates a link failure (Nl 0,N21) and unicast routing selected the (Nl 0, N20) link using the normal adaptive routing algorithm previously described. But note, if the packet is routed up to N20 and link (N20.N12) is down, it has no easy path to get to the destination.
• the fabric architecture includes a technique that we refer to as "misrouting" .
• Misrouting provides for iterative backtracking.
• N10 sees that it has no paths to get to N3, other than the link it came in on. N10 sets the misrouting bit in the routing header, and sends it back to N6.
• misrouteVector in the routing header chooses an alternative link that has not been misrouted, and sends the packet to Nl 1.
• N6 sees that the packet is being misrouted, adds Nl 1 link to the misrouteVector (now contains Nl 0 and Nl 1 link IDs), chooses an alternative link that has not been misrouted, and sends it N7.
• N7 sees that the misrouting bit is set, but does have a valid link to N3 (to N12), and thus clears the misrouting bit in the header, and forwards the packet to N12.
• Normal adaptive routing will not choose the direct link to N3 since it's down, and will route the packet to N8, then finally to N3.
• the implementation may want to have a register bit (enableRecursiveMisrouting) to enable this retry at lower layer option.
• register enableMisrouting that allows software to control whether the switch will initiate the misrouting algorithm.
• Multi-Domaining whose goal is to increase the addressability of nodes to a large number of nodes (e.g., 64K nodes), without having to increase the size of the unicast routing table to 64K nodes.
• the unicast routing table is a single-dimension array indexed by node number (i.e. 0 to MAX NODES-l), where a typical implementation will be between 256 and 4K nodes.
• the node namespace is changed from a node ID from 0 to MAX NODES-l , to a 2-tuple of (domain ID, node ID), where both domain ID and node ID range from 0 to 255. So, there can effectively be 256 domains where each domain can contain up to 256 nodes.
• the unieast routing table is changed from a single dimension table of size
• the unieast routing table When routing to a node within this domain, the unieast routing table is accessed as unicastRoute[0][node ID], and provides a weighted link vector to route to the specified node ID from the current node,
• o Remote domain routing When routing to a node within a remote domain, the unieast routing table is accessed as unicastRoute[l ] [domain ID], and provides a weighted link vector to route to the specified domain ID from the current node. Routing frame: One bit is added to the routing frame, dstRemote, which is set true when routing to a remote domain.
• Table 2 describes the algorithms for creating the routing frame header. This is augmented in the multi-domaining case by:
• Network Proxy The concept of network proxy is the ability of the main processors (Figure 5A, 905) to maintain network presence while in a low-power sleep / hibernation state, and intelligently wake when further processing is required.
• ⁇ There is a CSR (portRemap) to allow the remapping of Port IDs.
• this Port Remapping CSR allows software to remap MACO to the management processor MAC (e.g. Figure 5A, 907) and have the packet delivered to the management processor for Network Proxy processing.
• This remapping CSR could also be used to remap MAC1 traffic to MACO, or MAC 1 traffic to the management processor.
• the switch looks at the destination node ID of the routing frame to decide whether the packet is delivered to an internal port within the node, or gets routed to other XAUI connected nodes. This is done by matching Destination Node ID to "My Node ID”.
• myNodeID Destination_Node. This allows a node to proxy for a subtree of nodes.
• Management processor maintains the IP to MAC address mappings for MACO and
• MAC1 on the node. This can be done via either explicit communication of these mappings from the main processor OS to the management processor, or can be done implicitly by having the management processor snoop local gratuitous ARP broadcasts.
• the main processor coordinates with the management processor to go to a low power dormant state. During this transition, the management processor sets up the Port ID remapping CSR to route MACO and MAC 1 traffic to the management processor.
• the management processor processes any incoming MAC0 MAC1 packets. There are 3 categories of processing:
• the computer to be woken is shut down (sleeping, hibernating, or soft off; i.e., ACPI state Gl or G2), with power reserved for the network card, but not disconnected from its power source.
• the network card listens for a specific packet containing its MAC address, called the magic packet, broadcast on the broadcast address for that particular subnet (or an entire LAN, though this requires special hardware or configuration).
• the magic packet is sent on the data link or layer 2 in the OSI model and broadcast to all NICs within the network of the broadcast address; the IP-address (layer 3 in the OSI model) is not used.
• the network card checks the packet for the correct information. If the magic packet is valid, the network card takes the computer out of hibernation or standby, or starts it up.
• the magic packet is a broadcast frame containing anywhere within its payload: 6 bytes of ones (resulting in hexadecimal FF FF FF FF FF FF FF), followed by sixteen repetitions of the target computer's MAC address. Since the magic packet is only scanned for the string above, and not actually parsed by a full protocol stack, it may be sent as a broadcast packet of any network- and transport-layer protocol. It is typically sent as a UDP datagram to port 0, 7 or 9, or, in former times, as an IPX packet.
• the management processor can support these Wake-On-LAN packets. It will get these broadcast packets, will know the MAC addresses for the other MACs on the node, and be able to wake up the main processor as appropriate. No further functionality is needed in the switch to support these Wake-on-LAN packets.

Landscapes

• Engineering & Computer Science (AREA)
• Signal Processing (AREA)
• Computer Networks & Wireless Communication (AREA)
• Theoretical Computer Science (AREA)
• General Engineering & Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Physics & Mathematics (AREA)
• Human Computer Interaction (AREA)
• Software Systems (AREA)
• Computer Hardware Design (AREA)
• Data Exchanges In Wide-Area Networks (AREA)
• Small-Scale Networks (AREA)
• Multi Processors (AREA)

Priority Applications (5)

Applications Claiming Priority (4)

Publications (1)

Family

ID=43922457

Family Applications (1)

Country Status (6)

Cited By (2)

Families Citing this family (233)

Citations (6)

Family Cites Families (360)