WO2017123253A1

WO2017123253A1 - Generating peer-to-peer network addresses

Info

Publication number: WO2017123253A1
Application number: PCT/US2016/013686
Authority: WO
Inventors: Peter Hansen; Patrick A. Raymond
Original assignee: Hewlett Packard Enterprise Development LP
Current assignee: Hewlett Packard Enterprise Development LP
Priority date: 2016-01-15
Filing date: 2016-01-15
Publication date: 2017-07-20
Anticipated expiration: 2018-07-15

Abstract

An apparatus includes a module to interface to a network bus. A module descriptor specifies functionality of the module. A port on the module connects to a mating connector and receives a slot location where the module is connected. A controller generates a peer-to-peer network address for the module to communicate on the network bus. The peer-to-peer network address incorporates the module descriptor and the slot location to generate a unique network address for the network bus. The peer-to-peer network address enables other peer modules on the network bus to communicate directly with the module.

Description

GENERATING PEER-TO-PEER NETWORK ADDRESSES

BACKGROUND

[0001] Networks and server systems supporting high-end cloud applications include embedded management microprocessors that, when combined with an array of sensors, memories, and I/O devices, form a management subsystem. These devices often use some form of low cost, reliable interconnect or bus technology to function correctly. Some examples of bus technologies include Serial Peripheral Interfaces (SPI) bus, Inter Integrated Circuit (I2C) bus, serial Universal Asynchronous

Receiver/Transmitter (UART) bus (at times employing RS-232 or RS485 signaling), Control Area Network (CAN) bus, or 1 -wire bus, for example. Such connections may be bus or point-to-point configurations, for example. The decision on which management bus to implement in a given subsystem is based on tradeoffs that can include bus speed, cost, pin count, and participating component compatibility with the bus. In some examples, these subsystems are customizable with some form of modular chassis that accepts multiple types of feature options (e.g., computer components, storage components, I/O components, and so forth) that plug in with connectors and may communicate to each other, such as using one of the above mentioned bus

technologies.

BRIEF DESCRIPTION OF THE DRAWINGS

[0002] FIG. 1 is a block diagram illustrating an example of an apparatus to generate a peer-to-peer network address in accordance with the present disclosure.

[0003] FIG. 2 is a block diagram illustrating an example of a system to employ peer- to-peer network addressing that includes module functionality and location in the network address in accordance with the present disclosure.

[0004] FIG. 3 is a block diagram illustrating an example of a network address specifying module functionality and location in the network address in accordance with the present disclosure. [0005] FIG. 4 is a block diagram illustrating an example of a system to employ a bridge module to facilitate communications between an internal network bus and an external public network in accordance with the present disclosure.

[0006] FIG. 5 is a flow chart illustrating an example of a method of power-up operations for modules using a network employing peer-to-peer network addressing in accordance with the present disclosure.

[0007] FIG. 6 is a block diagram illustrating an example of a controller to facilitate peer-to-peer network communications in accordance with the present disclosure.

DETAILED DESCRIPTION

[0008] This disclosure relates to network address identification for peer-to-peer communication among modules in a local network. A module's location and

functionality are encoded as part of the network address for the module. This encoding enables efficient discovery of the functionality of the module along with peer-to-peer communications between modules without intervention from an intervening controller, as in master-slave architectures like I2C (Inter-Integrated Circuit). In one example, an apparatus is provided that includes a module to interface to a network bus. A module descriptor on the module specifies functionality of the module. The functionality specified in the network address can include the type or purpose of the module (e.g., it is for computing, storage, I/O, cooling, and so forth). The functionality specified in the network address can further specify functional features of the module, such as power conversion capabilities, communications capabilities, and/or infrastructure support of the module (e.g., cooling capabilities, such as single or dual fan module). A port on the module is to connect to a mating connector (e.g., a rack connection to a backplane) and receives a slot location in the bus where the module is connected. A controller on the module generates a peer-to-peer network address for the module to enable

communication to and from the module via the network bus. As mentioned, the peer-to- peer network address of a module incorporates or is derived from the module descriptor for the module and the slot location to provide a unique network address for the network bus.

[0009] The peer-to-peer network address enables other peer modules connected to the network bus to communicate directly with the module in a peer-to-peer manner without going through a master base controller or an address resolution cycle as with other systems. Systems implementing such peer-to-peer communications are afforded efficient discovery of module functionality and reduction of complex controller firmware operations (e.g., where the firmware comprises a set of machine-readable instructions) to support master and slave bus operations. The network addressing further enables a discovery process that locates not only the presence of specific modules by function in a chassis ecosystem, but also the respective location and possibly other meta-data associated with the module (e.g., additional functional features of the module). Peer-to- peer network addressing, as disclosed herein, also provides a multi-master capability to the subcomponents communicating on the network bus to facilitate their discovery of each other.

[00010] In a system, a base controller can initiate module discovery to all modules in a rack and can sequence power-up operations depending on detected capabilities and needs of modules in a rack. For example, the base controller discovers a system requirement of each of the plurality modules and sequences power-up operations among the plurality of modules in a particular sequence to enhance synergy among the modules. The power-up sequence can be implemented, for instance, to enable cooling modules to be powered before system computing and storage modules that may need additional cooling from the cooling module. In some examples, the internal network bus utilizing the peer-to-peer network addressing described herein can be connected to external public networks via a bridge module that bridges communications between the internal network bus and the external public network. A module (e.g. each module) in the system can address the bridge module to send messages to the external public network. [00011] FIG. 1 illustrates an example of an apparatus 100 that generates a peer-to- peer network address that includes module functionality and location in the network address. The apparatus 100 includes a module 1 10 to interface to a network bus 120. A module descriptor 124 on the module 1 10 specifies functionality of the module. The module descriptor 124 can be a firmware designation, register setting, and/or switch setting to specify the module's functionality and/or other module features. A port 130 on the module 1 10 connects to a mating connector 140 and receives a slot location where the module is connected. For example, the mating connector 140 can receive pin encodings from a given slot in a rack where the module 1 10 is located on the bus (See e.g., FIG. 2). If an eight slot rack were employed for example, three pins in the mating connector 140 would be utilized to identify the module's slot location to the module 1 10 via the port 130. A controller 150 includes an address generator to generate a peer-to- peer network address for the module to communicate on the network bus 120. The peer-to-peer network address generated by the controller 150 incorporates the module descriptor 124 and the slot location received from the port 130 to generate a unique network address for the network bus 120. The peer-to-peer network address enables other peer modules on the network bus 120 to communicate directly with the module 1 10 and without intervention of another module, such as a base controller operating the network bus 120. Although the port 130 is shown as a separate component in this example, it could be an internal port on the controller 150 in another example.

[00012] In one example, the network bus 120 utilizes an Ethernet protocol to facilitate communications on the network bus. Other network protocols are possible, such as ARPNet or Token Ring, for example. The module descriptor 124 and the slot location received from the port 130 can be encoded as a MAC address within the bus protocol (e.g., Ethernet). The module descriptor 124 can include a field to describe the module's primary role, a field to describe the module's communications capability to another network external to the network bus, or a field to describe the module's infrastructure capability to support other modules. For example, the field to describe the module's primary role can include a compute node identifier (e.g., processing resources), a storage identifier (e.g., memory resources), an I/O identifier (e.g., module input and/or output capability), a cooling identifier (e.g., fan capability), or a power capability identifier (e.g., AC or DC power source). The field to describe the module's communications capability to other networks can include an Ethernet identifier (e.g., 10 Gbyte or 1 GByte capability), a fiber channel identifier, or an infiniband identifier, for example. The field to describe the module's infrastructure capability to support other modules can include a power converter identifier (e.g., DC/DC converter), a rectifier identifier (e.g., AC/DC rectifier or high voltage rectifier), or a fan type identifier (e.g., single rotor fan, dual rotor fan).

[00013] The Ethernet protocol includes communications frames including packets of data that contain a header followed by a payload. The header portion of an Ethernet packet can be 14 bytes and contains a plurality of (e.g., three or more) fields. For example, a six byte destination MAC address in the header portion identifies the node the packet is being sent to. A six byte source MAC address identifies the network adapter that sent the frame. A two byte "ethertype" identifier that in this example includes the values 0x0800 can identify the frame as IPv4 formatted, but is neutral to higher level protocols running over the network bus 120 since such protocols are generally unnecessary in a closed communications environment (e.g., closed to global communications) as described herein. In some existing systems, the six byte MAC addresses in Ethernet is further subdivided into two fields - an organizationally unique identifier (OUI) that is unique among all Ethernet products produced by an organization and a network interface controller (NIC) specific address that is 24 bits long and unique among this controller's organization that manufactured it. In this example, however, it is presumed that the network bus 120 is contained to a local operating environment and such global uniqueness is unnecessary. Thus, the module's MAC address can be repurposed to specify the modules functionality described herein where local uniqueness on the network bus 120 is maintained via the slot location derived from the port 130.

[00014] By providing the peer-to-peer addressing capabilities specifying a module's functionality and location in the network address, various features can be supported. For example, the network bus is many orders of magnitude faster than a conventional I2C bus for example which can be useful in high bandwidth management functions such as firmware update. The network bus can be configured as a differential low voltage network that facilitates noise immunity. The peer-to-peer addressing has a high level of compatibility with embedded microprocessor modules where data that is being collected inside a module can be increased over conventional systems and efficiently

communicated across the network bus 120. Thus, efficient methods can be supported to collect such data and transfer via the network bus 120 and addressing protocol described herein. For instance, Ethernet with its speeds to 10 Gb/s or greater can be supported on the network bus 120.

[00015] FIG. 2 illustrates an example of a system 200 that employs peer-to-peer network addressing that includes module functionality and location in the network address. The system 200 includes a plurality of modules shown as modules 1 though M with M being a positive integer, to interface to a network bus 210. Each of the plurality of modules can be inserted in to a slot shown as slots 1 though N which are housed in a rack 220, where N is at least equal to M. The rack 220 includes slots that computers, servers, routers, storage, I/O modules and/or other devices are mounted in. As used herein, the term slot refers to a location where a module can be located in the rack 220. Another type of slot is a bay that can similarly house a module similar to the slot. In some cases, a rack 220 can include both slots and bay locations which can be alternatively identified in the peer-to-peer addresses described herein. Each of the plurality of modules 1 -M can include a module descriptor on the respective module to specify functionality of the respective module and a port on the respective module to connect to a mating connector on the respective module and to receive a slot location where the respective module is connected (See e.g., of FIG. 1 ).

[00016] A controller on each of the modules 1 -M can generate a peer-to-peer network address for the respective module to communicate on the network bus 210. The peer- to-peer network address is based on the module descriptor of the respective module and the slot location of the respective module to generate a unique network address for communicating via the network bus 210. A base controller 230 (e.g., baseboard management controller [BMC], which may provide out-of-band functionality) mounted on a baseboard circuit 240 can communicate with each of the plurality of modules 1 -M via the network bus 210. The peer-to-peer network address of the respective modules on the network bus can enable discovery among each of the plurality of modules on the network bus without intervention from the base controller 230 based on the specified functionality of the module descriptor of the respective module.

[00017] The network bus 210 can utilize an Ethernet protocol to facilitate

communications on the network bus where the module descriptor and the slot location of the respective module is encoded as a MAC address within the Ethernet protocol. The base controller 230 can provide a predetermined MAC address (e.g.,

0x2020202020) that is unique from each of the plurality of modules 1 -M such that each of the plurality of modules communicates to the base controller via the predetermined MAC address. Each of the modules 1 -M can be provided (e.g., during an initialization process) with the predetermined address of the base controller 230 or can have the predetermined address preprogrammed in a register or dip switch setting, for example. The base controller 230 can perform system level discovery of modules via iterative polling of modules and/or via broadcast discovery requests generated from the base controller, for example.

[00018] For communications between the base controller 230 and the modules 1 -M in the rack 220, the modules 1 -M can implement a higher level request / response type of protocol that can use the hardware MAC addresses for their respective OSI layer 2 level communications medium, for example. The protocol could be an industry standard - such as TCP/IP or UDP/IP - or it could be a proprietary protocol using the Ethernet payload in ways deemed suitable for the application. Discovery of both the function and location of all the devices in the system can be achieved according to different processes. In one process, the base controller 230 can iteratively send out all possible packets with destination MACs set to the target's slot ID, and role. For modules that are present, they can respond and the base controller 230 can learn of their presence. In another example discovery process, the base controller 230 can send out a broadcast request over the network bus 210 (e.g., destination MAC is all 1 s (OxFFFFFFFFFFFF) so that all devices on the bus can detect it and respond. The base controller 230 then collects the responses and stores the responses (e.g., in a table indicating functional capabilities of each module in rack) according to the detected functionality. A discovery process can continually run, run periodically or be run intermittently in response to an event to discover insertions and removals of modules from the rack 220.

[00019] Another benefit of the peer-to-peer capability described herein is that it provides a multi-master capability to the modules communicating on the network bus 210 since each module can determine network address and functions of other modules by analyzing network traffic on the bus. The modules can discover each other in a manner similar to base controller discovery (or during) and they can directly address the other modules by formulating their MAC addresses via module descriptors and slot location as described herein. Such module-based discovery can be of benefit in baseboard computer systems that have product side-bus arrangements such as direct attach storage nodes on separate modules in an adjacent slot in a system or a "sidecar" or storage blade that is located in the slot next to its associated computing node and keyed with another bus (e.g., an expansion bus, such as a PCI-E bus), for example.

[00020] The discovery process described herein can also support other operations such as sequential start-up of modules within the rack 220. For example, after completing discovery of the modules 1 -M, the base controller 230 may power some modules down until other modules have had a chance to enter a normal operating state (e.g., for a predetermined time until a given status is received such as temperature). After the selected modules are powered, other modules can be re-started. Sequential operations can include ensuring there is enough cooling capacity for temperature sensitive modules in the correct locations in a system prior to powering them on, for example. This can include performing desired I/O keying operations such as verifying and/or matching appropriate production bus and technologies in the rack 220 prior to powering on certain modules. This can also include providing a mechanism that powers-up the modules in the system in a desired sequence. For example, a base controller 230 can employ the unique network address of the modules (learned from the discovery process), to identify the types and capabilities of the respective modules. The base controller 230 can then configure and implement a sequential process to provide for efficient and safe operation of the respective modules. As an example, the controller can power on I/O devices, then storage (so it can connect to the I/O), then compute nodes (so they can connect to both I/O and storage) to ensure desired connectivity of these components as they power up. Other orders can be implemented, which may be established by the controller according to the particular functions and locations of the respective module 1 -M on the network bus 210.

[00021] FIG. 3 illustrates an example of a network address 300 specifying module functionality and location in the network address. The network address can be provided as part of an Ethernet MAC address. For example the network address 300 can include a field specifying a module's role (e.g., the module's function) at 310. Such role can be specified via bits in the address 330 and, for example, can include computing, storage, cooling, power, infrastructure support, and so forth. At 320, the network address 300 can include data specifying communications capabilities (specified as bits), such as providing access to another network outside the network bus described herein.

Example networks can include Ethernets operating at different speeds (e.g., 1 G, 10G, 100G), fiber channel networks, and Infiniband networks, for example which can also operate at different speeds. At 330, a field in the network address 300 can be reserved for future functional capabilities of the module that are not presently available. At 340, a field in the network address 300 can specify infrastructure capability of a given module. If the module is a functional module such as a compute module, storage module, or I/O module, this field can be set to zero. If the module is an infrastructure support module, depending on bit pattern, this field 340 can specify DC/DC converter, AC/DC rectifier, HVDC rectifier, single rotor fan, dual rotor fan, and so forth. At 350, the module's slot identifier (ID) is provided reflecting the slot location determined by the controller reading the module port described herein.

[00022] FIG. 4 illustrates an example of a system 400 that employs a bridge module 410 to facilitate communications between modules and devices coupled to an internal network bus 420 and an external network (e.g., a wide area network (WAN), such as the Internet or other public network) 430. As shown, a module 440 can send a peer-to- peer (P2P) message to the bridge module 410, where the bridge module reformats the message according to the protocol of the external network. If the external network 430 implemented Ethernet, for example, the bridge module 410 would provide a globally unique address for the external Ethernet network while utilizing a local Ethernet address derived from function and slot location of the bridge module, as described herein, to communicate with the module 440 over the internal network bus 420. As shown, a base controller 450 could also communicate to the external network 430 via the bridge module 410. In another example, the base controller 450 could route a message over the internal network bus 420 to the module 440 which could then re-route the received base controller message to the bridge module 410.

[00023] The bridge module 410 enables one base controller (e.g., the base controller 450) to control multiple chassis systems. Such a bridge module 410 can include a small microcontroller with two network interfaces. One contains a MAC that identifies it as a bridging device. The other network controller would be an external facing "public" interface that would follow the guidelines of such a module (e.g., operating on a public Ethernet network). A bridging protocol can be constructed in the payload of the

Ethernet frame and the module would then convert and route the internal packets to external networks and vice versa - building a routing table in the process.

[00024] In view of the foregoing structural and functional features described above, an example method will be better appreciated with reference to FIG. 5. While, for purposes of simplicity of explanation, the method is shown and described as executing serially, it is to be understood and appreciated that the method is not limited by the illustrated order, as parts of the method could occur in different orders and/or concurrently from that shown and described herein. Such method can be executed by various

components and executed by an integrated circuit, computer, or a controller, for example.

[00025] FIG. 5 illustrates an example of a method 500 that utilizes peer-to-peer network addressing that includes module functionality and location in the addressing to sequence power-up operations between modules (e.g., modules 1 10, 1 -M, 440). At 510, the method 500 includes generating a discovery request on a network bus (e.g., bus 120, 210, 420) to discover functionality of a plurality of modules communicating on the network bus (e.g., via the base controller 230 of FIG. 2). The discovery request can be encoded as a predetermined MAC address that is unique to base controller for the network bus, for example. As disclosed herein, discovery can include sending a broadcast request to the network bus to discover the functionality of each of the plurality of modules. In another example, discovery can include iteratively polling each module to discover its respective functionality.

[00026] At 520, the method 500 includes receiving (e.g., at the base controller 230 of FIG. 2) a discovery response from each of the plurality of modules on the network bus. The discovery response from each of the modules can include an address (e.g., MAC address) having a module descriptor on the respective module to specify the

functionality of the respective module and a slot location where the respective module is connected. The slot location can be encoded as a unique MAC address for each of the plurality of modules, for example. At 530, the method 500 includes sequencing power- up operations of each of the plurality of modules based on the functionality of the modules determined based on the discovery response received from each of the plurality of modules (e.g., via the base controller 230 of FIG. 2). As noted herein, for example, this can include powering up some modules in a prescribed order to achieve desired level of functionality (e.g., cooling and power) before powering other modules (e.g., computing or storage modules).

[00027] FIG. 6 illustrates an example of a controller 600 to facilitate peer-to-peer network communications. The controller 600 includes a network interface 610 to send and receive addresses and associated messages on the network bus. An address generator 620 configures network addresses utilizing the slot location and module descriptor as previously described. For example, the address generator 620 receives the slot location and the module descriptor from the controller 600 and formats a MAC address packet. The MAC address can contain other network data along with the module descriptor and slot location data. After formatting the MAC address, the address generator 620 submits the formulated network address to the network interface that provides the address to the network bus. The address can be generated in response to a discovery request from the base controller and/or in response to a message request from another module and/or the base controller. An address parser 630 analyzes network packets received by the network interface 610. The address parser 630 can isolate the slot locations and module functionality associated with the respective modules in the rack to build a functional table of module destinations that can then be used by the controller 600 to send peer messages to other modules in the rack.

[00028] What have been described above are examples. It is, of course, not possible to describe every conceivable combination of components or methods, but one of ordinary skill in the art will recognize that many further combinations and permutations are possible. Accordingly, examples described herein are intended to embrace all such alterations, modifications, and variations that fall within the scope of this application, including the appended claims. Additionally, where the disclosure or claims recite "a," "an," "a first," or "another" element, or the equivalent thereof, it should be interpreted to include one or more than one such element, neither requiring nor excluding two or more such elements. As used herein, the term "includes" means includes but not limited to, and the term "including" means including but not limited to. The term "based on" means based at least in part on.

Claims

CLAIMS What is claimed is:

1 . An apparatus, comprising:

a module to interface to a network bus, the module storing a module descriptor to specify a functionality of the module;

a port on the module to connect to a mating connector and to receive a slot location where the module is connected; and

a controller to generate a peer-to-peer network address for the module to communicate on the network bus, the peer-to-peer network address incorporates the module descriptor and the slot location to generate a unique network address for the network bus, the peer-to-peer network address to enable other peer modules on the network bus to communicate directly with the module.

2. The apparatus of claim 1 , wherein the network bus utilizes an Ethernet protocol to communicate on the network bus.

3. The apparatus of claim 2, wherein the module descriptor and the slot location are encoded as a MAC address for the module within the Ethernet protocol for the network bus.

4. The apparatus of claim 1 , wherein the module descriptor includes at least one of a field to describe a primary role of the module, a field to describe a communications capability of the module to one or more other networks external of the network bus, or a field to describe an infrastructure capability of the module to support other modules.

5. The apparatus of claim 4, wherein the field to describe the primary role specifies a compute node identifier, a storage identifier, an I/O identifier, a cooling identifier, or a power capability identifier.

6. The apparatus of claim 4, wherein the field to describe the communications capability to one or more other networks includes an Ethernet identifier, a fiber channel identifier, or an Infiniband identifier.

7. The apparatus of claim 4, wherein the field to describe the module's

infrastructure capability to support other modules includes a power converter identifier, a rectifier identifier, or a fan type identifier.

8. A system, comprising:

a plurality of modules to interface to a network bus, each of the plurality of modules comprising:

a module descriptor on the respective module to specify a functionality of the respective module;

a port on the respective module to connect to a mating connector on the network bus and to receive a slot location from the mating connector where the respective module is connected; and

a controller including an address generator to generate a peer-to-peer network address for the respective module to communicate on the network bus, the peer-to-peer network address being derived from the module descriptor of the respective module and the slot location of the respective module to generate a unique network address for the module on the network bus; and

a base controller to communicate to the each of the plurality of modules via the network bus, the peer-to-peer network address of the respective module on the network bus to enable discovery between each of the plurality of modules on the network bus without intervention from the base controller based on the functionality of the module descriptor of the respective module.

9. The system of claim 8, wherein the network bus utilizes an Ethernet protocol to facilitate communications on the network bus and the module descriptor and the slot location of the respective module is encoded as a MAC address within the Ethernet protocol.

10. The system of claim 9, wherein the base controller provides a predetermined MAC address that is unique from each of the plurality of modules such that each of the plurality of modules communicates to the base controller via the predetermined MAC address.

1 1 . The system of claim 8, wherein the base controller iteratively sends packets set to a target slot identifier of each of the plurality of modules across the network bus to discover the functionality of each of the plurality of modules, or the base controller sends a broadcast request to the network bus to discover the functionality of each of the plurality of modules.

12. The system of claim 1 1 , wherein the base controller receives discovery responses from each of the plurality modules and sequences power-up operations among the plurality of modules based on the functionality of each of the plurality of modules to enable cooling modules to be powered before system computing and storage modules.

13. The system of claim 8, wherein one of the plurality of modules is configured as a bridge module to enable communications between the network bus and to an external network outside of the network bus.

14. A method, comprising:

generating a discovery request on a network bus to discover a functionality of a plurality of modules communicating on the network bus, the discovery request encoded as a predetermined MAC address that is unique to the network bus;

receiving a discovery response from each of the plurality of modules on the network bus, the discovery response from each of the modules includes a module descriptor on the respective module to specify the functionality of the respective module and a slot location where the respective module is connected, the slot location encoded as a unique MAC address for each of the plurality of modules; and

sequencing power-up operations of each of the plurality of modules based on the functionality of the modules determined based on the discovery response received from each of the plurality of modules.

15. The method of claim 14, comprising sending a broadcast request to the network bus to discover the functionality of each of the plurality of modules.