US20200252349A1 - Computing devices using extremely high frequency (ehf) electromagnetic communication - Google Patents
Computing devices using extremely high frequency (ehf) electromagnetic communication Download PDFInfo
- Publication number
- US20200252349A1 US20200252349A1 US16/855,801 US202016855801A US2020252349A1 US 20200252349 A1 US20200252349 A1 US 20200252349A1 US 202016855801 A US202016855801 A US 202016855801A US 2020252349 A1 US2020252349 A1 US 2020252349A1
- Authority
- US
- United States
- Prior art keywords
- ehf
- rack
- server
- rack mountable
- ehf communication
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 238000004891 communication Methods 0.000 title claims abstract description 347
- 238000012546 transfer Methods 0.000 claims abstract description 8
- 230000015654 memory Effects 0.000 claims description 85
- 238000003860 storage Methods 0.000 claims description 44
- 230000005855 radiation Effects 0.000 claims description 12
- 239000000463 material Substances 0.000 claims description 10
- 238000000034 method Methods 0.000 claims description 10
- 238000007726 management method Methods 0.000 description 54
- 238000013500 data storage Methods 0.000 description 16
- 238000012545 processing Methods 0.000 description 13
- 230000002950 deficient Effects 0.000 description 10
- 239000004744 fabric Substances 0.000 description 7
- 238000012544 monitoring process Methods 0.000 description 5
- 230000002093 peripheral effect Effects 0.000 description 4
- 230000002085 persistent effect Effects 0.000 description 4
- 238000001816 cooling Methods 0.000 description 3
- 239000003989 dielectric material Substances 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000012423 maintenance Methods 0.000 description 3
- 230000006855 networking Effects 0.000 description 3
- 239000004033 plastic Substances 0.000 description 3
- 230000008439 repair process Effects 0.000 description 3
- 239000000758 substrate Substances 0.000 description 3
- 230000002776 aggregation Effects 0.000 description 2
- 238000004220 aggregation Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 239000003795 chemical substances by application Substances 0.000 description 2
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 238000002955 isolation Methods 0.000 description 2
- 239000007769 metal material Substances 0.000 description 2
- 229920001296 polysiloxane Polymers 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 229910000679 solder Inorganic materials 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 239000004593 Epoxy Substances 0.000 description 1
- 206010051602 Laziness Diseases 0.000 description 1
- 206010029412 Nightmare Diseases 0.000 description 1
- 206010035148 Plague Diseases 0.000 description 1
- 241000607479 Yersinia pestis Species 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 239000011358 absorbing material Substances 0.000 description 1
- 230000004931 aggregating effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 229910052799 carbon Inorganic materials 0.000 description 1
- 239000000919 ceramic Substances 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 150000001875 compounds Chemical class 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000010292 electrical insulation Methods 0.000 description 1
- 230000005670 electromagnetic radiation Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 230000003862 health status Effects 0.000 description 1
- 239000011810 insulating material Substances 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 239000002184 metal Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000246 remedial effect Effects 0.000 description 1
- 238000005201 scrubbing Methods 0.000 description 1
- 238000004904 shortening Methods 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 229910000859 α-Fe Inorganic materials 0.000 description 1
Images
Classifications
-
- H04B5/72—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L49/00—Packet switching elements
- H04L49/10—Packet switching elements characterised by the switching fabric construction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
- G06F1/16—Constructional details or arrangements
- G06F1/18—Packaging or power distribution
- G06F1/181—Enclosures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F1/00—Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00
- G06F1/16—Constructional details or arrangements
- G06F1/18—Packaging or power distribution
- G06F1/183—Internal mounting support structures, e.g. for printed circuit boards, internal connecting means
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04B—TRANSMISSION
- H04B5/00—Near-field transmission systems, e.g. inductive loop type
- H04B5/0025—Near field system adaptations
- H04B5/0031—Near field system adaptations for data transfer
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L49/00—Packet switching elements
- H04L49/35—Switches specially adapted for specific applications
- H04L49/356—Switches specially adapted for specific applications for storage area networks
- H04L49/358—Infiniband Switches
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W84/00—Network topologies
- H04W84/02—Hierarchically pre-organised networks, e.g. paging networks, cellular networks, WLAN [Wireless Local Area Network] or WLL [Wireless Local Loop]
- H04W84/10—Small scale networks; Flat hierarchical networks
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W88/00—Devices specially adapted for wireless communication networks, e.g. terminals, base stations or access point devices
- H04W88/08—Access point devices
Definitions
- Spaghetti cabling is used to refer to the unkempt cabling mess that is often found springing out from the back panels of server racks at a data center.
- Spaghetti cabling may result from inexperience, lack of foresight, or laziness of the ones responsible for setting up the servers, and generally leads to poor cooling from impeded air flow and to maintenance nightmares.
- spaghetti cabling may be prevented, or at least mitigated, by shortening, bundling, and labeling the cables, the use of cables itself may be a limiting factor in the scalability of a server rack.
- a cable port to which a cable connects generally has dimensions that conform to a widely-used standard and thus cannot be easily changed. Due to the inflexible dimensions of the cable ports and the limited amount of space on a server rack back panel, a limited number of cable ports can be physically implemented on the server rack back panel. This means that, even though the bandwidth of the server rack can be increased by increasing the number of cable ports, a port density issue may arise in which the amount of increase in the bandwidth would still be constrained by the limited number of cable ports that can be physically implemented.
- a system includes components that use extremely high frequency (EHF) communication devices to form EHF electromagnetic communication channels for data transfer.
- the system may include a server rack including one or more rack mountable devices.
- a rack mountable device includes a rack mountable chassis having multiple sides, a circuit board having a surface that extends along a side of the rack mountable chassis, and one or more EHF communication devices attached to the surface of circuit board. Each EHF communication device converts between an EHF electromagnetic signal and an electrical data signal.
- the system may include a rack frame including multiple bays.
- Rack mountable devices are inserted into the bays.
- the EHF communication devices of each rack mountable device may be located along top and bottom surfaces of the rack mountable device to form EHF electromagnetic communication channels with other EHF communication devices of other rack mountable devices in the rack frame.
- the rack mountable devices may include servers, with the EHF communication devices providing server-to-server communications.
- an EHF communication device may be located along a side surface of the rack mountable device to form an EHF electromagnetic communication channel with another EHF communication device located along a side of the rack mountable chassis, such as to provide server-to-rack communications.
- a rack mountable device uses an EHF communication device to communicate management data via an EHF electromagnetic signal.
- the management data may include an identifier of the rack mountable device and state information regarding components of the rack mountable device.
- a rack mountable device uses EHF communication devices to communicate between a main board and a data storage drive in the rack mountable device.
- the main board may include a processor that accesses data from a memory device of the data storage drive using one or more EHF electromagnetic communication channels formed by EHF communication devices on the main board and the data storage drive.
- the data storage drive may include traditional mechanical disks (HDD), non-volatile memory (FLASH/SSD/NVMe) and non-volatile main memory such as persistent DRAMs (NVDIMM-P).
- FIG. 1 shows a server rack, in accordance with some embodiments.
- FIG. 2 shows a server rack system, in accordance with some embodiments.
- FIG. 3 shows a rack mountable device for a server rack, in accordance with some embodiments.
- FIG. 4 shows a cross sectional side view of rack mountable devices in a server rack, in accordance with some embodiments.
- FIG. 5 shows a coupler tile of a rack mountable device, in accordance with some embodiments.
- FIG. 6A shows a cover panel structure, in accordance with some embodiments.
- FIG. 6B shows a cross sectional side view of the cover panel structure of FIG. 6A , in accordance with some embodiments.
- FIG. 7A shows an extremely high frequency (EHF) communication device, in accordance with some embodiments.
- EHF extremely high frequency
- FIG. 7B shows a perspective view of the EHF communication device of FIG. 7A , in accordance with some embodiments.
- FIG. 8 shows communication paths in a rack mountable device, in accordance with some embodiments.
- FIG. 9 shows rack mountable devices in a server rack, in accordance with some embodiments.
- FIG. 10 shows a server rack mountable compute device in wireless communication with a rack mountable memory device, in accordance with some embodiments.
- FIG. 11 shows a storage rack mountable device of a server rack, in accordance with some embodiments.
- FIGS. 12A, 12B, 12C, 12D, 12E, and 12F show wireless communication architectures between processors and memories in a server rack, in accordance with some embodiments.
- FIG. 13 shows a coupler strip, in accordance with some embodiments.
- FIG. 14 shows wireless communication between EHF communication devices of a rack mountable device and EHF communication devices of the server rack, in accordance with some embodiments.
- FIG. 15A shows EHF communication devices providing communication between a processor and memory in a rack mountable device, in accordance with some embodiments.
- FIG. 15B shows a side view of the EHF communication devices providing communication between the processor and the memory in the rack mountable device, in accordance with some embodiments.
- FIG. 16 shows a flow chart of a process for managing rack mountable devices, in accordance with some embodiments.
- FIG. 17 shows a management system for a rack mountable device, in accordance with some embodiments.
- a server rack system may include one or more rack mountable devices, each of which may be a server.
- rack mountable device is herein used interchangeably with the term “server.”
- a rack mountable device is not limited to a server and may include any device.
- a server may include a chassis and one or more EHF communication devices along one or more surfaces of the chassis.
- the server may include a computing device, a storage device, a switching device, a communication device, etc. and may be configured to mount in a server rack.
- Each EHF communication device may form an EHF communication channel with another EHF communication device, which may be disposed in an adjacent server in the server rack or in a bay of the server rack, to provide high bandwidth data transfer for the server.
- FIG. 1 shows a server rack 100 , in accordance with some embodiments.
- the server rack 100 includes a rack frame 102 defining an array of bays 104 .
- a server 106 may be configured to insert into each of the bays 104 .
- the server rack 100 may provide power, data communication, and cooling to each of the servers 106 in the server rack 100 .
- Each server 106 may include a rack mountable chassis with EHF communication devices on the chassis.
- EHF communication devices may be located at the top and bottom surfaces of the servers 106 to provide server-to-server wireless communications
- EHF communication devices may be located on side surfaces of the servers 106 to provide server-to-rack wireless communications.
- Examples of servers 106 include 1U, 2U and 4U servers, networking equipment, and/or storage devices.
- the server rack 100 further includes one or more rack switches 108 , such as a top-of-rack (TOR) switch.
- a rack switch 108 may provide networking capability to the servers 106 , as well as other components.
- a server 106 may communicate with other networked devices outside of the server rack 100 via the rack switch 108 .
- Servers 106 in the server rack 100 may also communicate with each other via the rack switch 108 .
- the servers 106 may communicate with the rack switch 108 using EHF communication devices.
- EHF communication devices of a server 106 may communicate with EHF communication devices of the server rack 100 via electromagnetic signals, while the EHF communication devices in the server rack 100 may be electrically connected to the rack switch 108 , thereby bridging communication between the servers 106 and the rack switch 108 .
- FIG. 2 shows a server rack system 200 , in accordance with some embodiments.
- the system 200 includes multiple server racks 100 connected via a spine router 202 .
- the spine router 202 may include switches that provide network connection points for servers in the servers 106 of the server racks 100 .
- the spine router 202 connects with the rack switches 108 (e.g., TOR switches) in the server racks 100 and provides a spine and leaf layer topology for the system 200 .
- server racks 100 may be connected using other types of network topologies.
- FIG. 3 shows a server 300 for a server rack 100 , in accordance with some embodiments.
- the server 300 is an example of a server 106 configured to insert into the bays 104 of the server rack 100 .
- the server 300 includes a chassis 302 enclosing computing components.
- the chassis 302 is a rack mountable chassis having a rectangular cuboid shape with three pairs of opposing sides.
- the chassis 302 includes a top cover panel 304 , a bottom cover panel 306 , left and right cover panels 308 , a front cover panel, and a rear cover panel.
- the chassis 302 may also be referred to as a case.
- One or more of the cover panels 304 , 306 , and 308 may include EHF communication devices 310 that provide wireless communications with EHF communication devices of other servers 300 in the server rack 100 , or with other EHF communication devices in the server rack 100 .
- Each EHF communication device 310 converts between an EHF electromagnetic signal and an electrical data signal.
- the EHF electromagnetic signal includes electromagnetic radiation in an EHF frequency range.
- the communication devices 310 may transmit or receive electromagnetic signals in other radio frequency ranges.
- the EHF communication devices 310 in the top cover panel 304 may form EHF electromagnetic communication channels with an above-adjacent server 300 using EHF electromagnetic signals
- the EHF communication devices 310 in the bottom cover panel 306 may form EHF electromagnetic communication channels with a below-adjacent server 300 using EHF electromagnetic signals.
- the EHF communication devices 310 may be located in the left and/or right side cover panels 308 to form EHF electromagnetic communication channels with EHF communication devices in the server rack 100 .
- the server 300 may include EHF communication devices 310 in the left and right side cover panels 308 to communicate with EHF communication devices at the sides of the rack frame 102 .
- the back or front side cover panels of the server 300 may include EHF communication devices.
- the EHF communication devices 310 in the top cover panel 304 and/or the bottom cover panel 306 are formed on coupler tiles 312 .
- Each coupler tile 312 includes multiple EHF communication devices 310 on a printed circuit board (PCB). Multiple coupler tiles 312 may be arranged in a cover panel along a side of the chassis 302 to support differently sized chassis 302 .
- the EHF communication devices 310 in the left and right cover panels 308 may be part of coupler strips 314 .
- Each coupler strip 314 includes multiple EHF communication devices 310 on a printed circuit board.
- the coupler strips 314 may have different dimensions (e.g., rectangular) than the coupler tiles 312 (e.g., square) to support mounting at narrower surfaces. Multiple coupler strips 314 may be arranged in the left and right cover panels 308 .
- the coupler tile 312 and coupler strip 314 show example circuit board dimensions that may be used to attach the EHF communication devices 310 .
- the size or shape of coupler tiles 312 or coupler strips 314 may vary, such as based on the dimensions of the chassis 302 .
- the EHF communication devices 310 may be mounted to circuit boards in various patterns or arrangements.
- coupler strip 314 may be included in the top cover panel 304 or bottom cover panel 306 when space remains after placing the coupler tiles 312 , as shown in FIG. 3 .
- FIG. 4 shows a cross sectional side view of servers 106 in a server rack 100 , in accordance with some embodiments.
- the servers 106 a , 106 b , and 106 c are vertically stacked on top of each other and include EHF communication devices 310 on their top and bottom cover panels to enable wirelessly communication with adjacent servers 106 via EHF electromagnetic communication channels 410 (or “EHF channels 410 ”).
- EHF electromagnetic signals are transmitted and received across the EHF electromagnetic communication channels 410 to transfer data between separate servers 106 .
- the server 106 b has EHF communication devices 310 at the top cover panel to communicate with the server 106 a above, and has EHF communication devices 310 at the bottom cover panel to communicate with the server 106 c below.
- multiple servers 106 may be stacked horizontally in a server rack 100 .
- Each EHF communication device 310 of a server 106 is paired with a corresponding EHF communication device 310 of an adjacent server 106 to send and receive EHF electromagnetic signals.
- EHF communication devices 310 on adjacent servers 106 are spaced apart with a common pitch and aligned with each other.
- the EHF electromagnetic signals provide near-field coupling between paired EHF communication devices 310 , allowing the EHF communication devices 310 to be arranged in dense arrays without cross interference with nearby pairs of EHF communication devices 310 .
- each server may include a main board 402 , a connector board 404 , and coupler tiles 312 .
- the main board 402 is a motherboard or other primary board of the server 106 b , and may support other computing components such as processors, memories, etc.
- the connector board 404 is connected to the main board 402 to provide an interface to the EHF communication devices 310 , and may be a riser board.
- the connector board 404 may be a PCI-express compatible board that plugs into a PCI-express connector of the main board 402 .
- Each coupler tile 312 may include a connector to connect with the connector board 404 via a cable 412 that includes conductive wires so that data can be communicated between the coupler tile 312 and the connector board 404 .
- the coupler tile 312 may be connected to the connector board 404 and/or the main board 402 via a mezzanine connector rather than a cable.
- Coupler strips 314 may be similarly connected to the connector board 404 as shown in FIG. 4 .
- a cable or mezzanine connector may be used to connect a coupler strip 314 to the connector board 404 and/or the main board 402 .
- each coupler tile 312 (or coupler strip 314 ) includes a circuit board 406 , the EHF communication devices 310 , and one or more data aggregator devices 408 .
- the EHF communication devices 310 and the data aggregator devices 408 are located on and interconnected by the circuit board 406 .
- Each aggregator device 408 performs routing of data to and from at least a subset of EHF communication devices 310 of the coupler tile 312 .
- the aggregator device 408 may receive electrical data signals from multiple EHF communication devices 310 (e.g., in parallel), combine lower data-rate data from the electrical data signals into higher data-rate data, and provide the higher data-rate data to the main board 402 via the connector board 404 .
- the aggregator device 408 may receive outgoing data from the main board 402 (e.g., via the connector board 404 and the circuit board 406 ), and generate multiple electrical data signals from the outgoing data at lower data rates, and provide the electrical data signals to respective EHF communication device 310 for wireless transmission (e.g., in parallel). That is, the aggregator device may receive high data-rate electrical data signals and split them across multiple lower-speed EHF channels. In some embodiments, the aggregator device 408 may also aggregate EHF signals into a high data-rate set of electrical signals.
- the aggregator device 408 may also perform additional functions, including: monitoring the link, encryption, decryption, forward error correction, cyclic redundancy code (CRC) generation, error checking, and passing through sideband information from low-speed electrical signals or from the on-board processing on the aggregator.
- additional functions including: monitoring the link, encryption, decryption, forward error correction, cyclic redundancy code (CRC) generation, error checking, and passing through sideband information from low-speed electrical signals or from the on-board processing on the aggregator.
- CRC cyclic redundancy code
- the aggregator device 408 may be on an opposite surface of the coupler tile 314 from the EHF communication devices 310 , or may be on the same surface of the coupler tile 314 as the EHF communication devices 310 . In some embodiments, the aggregator devices 408 control switching between transmission and reception states of EHF communication devices 310 . In some embodiments, the connector board 404 includes the aggregator devices 408 of the server 106 , and the aggregator devices may be omitted from the coupler tiles 312 or coupler strips 314 .
- FIG. 5 shows a coupler tile 312 of a server 106 , in accordance with some embodiments.
- the coupler tile 312 includes the circuit board 406 , the EHF communication devices 310 , and the aggregator devices 408 .
- Each aggregator device 408 handles data routing, aggregation, and distribution for a group of four adjacent EHF communication devices 310 .
- the circuit board 406 provides electrical interconnections between the aggregator devices 408 and the EHF communication devices 310 .
- the EHF communication devices 310 and the aggregator devices 408 are formed on the same surface of the circuit board 406 .
- the coupler tile 312 includes sixteen aggregator devices 408 , each facilitating data aggregation for four EHF communication devices 310 .
- the coupler tile 312 thus includes sixty-four EHF communication devices 310 .
- Thirty-two of the EHF communication devices 310 are receiving devices that receive EHF electromagnetic signals, and thirty-two of the EHF communication devices 310 are transmitting devices that transmit EHF electromagnetic signals. If each of the EHF communication devices 310 has a data rate of 6 Gigabits/second (Gbps), then the coupler tile 312 can support a read data rate of greater than 128 Gbps and a write data rate of greater than 128 Gbps.
- the coupler tile 312 can support a read data rate of greater than 256 Gbps and a write data rate of greater than 256 Gbps. In another example, if each of the EHF communication devices 310 has data rate of 28 Gbps, then the coupler tile 312 can support a read data rate of greater than 512 Gbps and a write data rate of greater than 512 Gbps.
- FIGS. 6A and 6B respectively show perspective and cross sectional views of a cover panel 600 , in accordance with some embodiments.
- the cover panel 600 includes the circuit board 406 , the EHF communication devices 310 (as shown in FIG. 6B ), a cover 606 , which may include a metal material, that provides structural protection for the EHF communication devices 310 , radiation absorbing shields 602 that provide EHF electromagnetic signal isolation among nearby EHF communication devices 310 on the same circuit board 406 , and waveguides 604 for guiding the EHF electromagnetic signals to facilitate coupling of paired EHF communication devices 310 .
- each EHF communication device 310 is surrounded by an EHF radiation absorbing shield 602 to shield the EHF communication device 310 from EHF electromagnetic signals of nearby or adjacent EHF communication devices 310 on the circuit board 406 .
- Each EHF radiation absorbing shield 602 defines a cavity with an EHF communication device 310 located in the cavity.
- the EHF radiation absorbing shields 602 are shown as having a hollow cylindrical shape, but other shapes may also be used.
- the radiation absorbing shield 602 includes a metallic material or other absorbing material (such as ferrite loaded silicone (or other dielectrics) or carbon loaded silicone (or other dielectrics)).
- the waveguide 604 is a dielectric lens waveguide for the EHF signals emitted from and received by the EHF communication device 310 .
- the waveguide 604 is positioned over the EHF communication device 310 and is partially surrounded by the EHF radiation absorbing shield 602 .
- EHF electromagnetic signals emitted by the EHF communication device 310 is received by the waveguide 604 , and is directed by the waveguide 604 through an opening 608 in the cover 606 .
- the waveguide 604 provides a medium to propagate the EHF electromagnetic signal between the opening 608 and an EHF communication device 310 .
- the waveguide 604 is plastic, dielectric material having a dielectric constant of at least about 2.
- the waveguides 604 are formed as a unitary lens structure that is shared across multiple EHF communication devices 310 and that includes protrusions extending through the openings 608 in the cover 606 . In some embodiments, waveguides other than the waveguide 604 may be used.
- the cover 606 is formed over the waveguide 604 , and structurally protects the covered components.
- the cover 606 may include the openings 608 through which the waveguide 604 may extend to guide EHF electromagnetic signals from the EHF communication devices 310 through the openings 608 .
- the cover 606 is a metallic, conductive sheet. In other embodiments, a plastic or dielectric casing material is used instead of metal sheeting.
- aggregator devices 408 are also formed on the circuit board 406 .
- the data aggregator devices 408 may be positioned on the surface of the circuit board 406 opposite the EHF communication devices 310 , or on the same surface of the circuit board 406 as the EHF communication devices 310 , such as in spaces defined between radiation absorbing shields 602 that surround the EHF communication devices 310 .
- the data aggregator devices 408 are omitted from the cover panel 600 .
- cover panel 600 that includes a coupler tile 312 may also be applicable to a cover panel that includes coupler strips 314 .
- the cover panel may also include a coupler strip 314 , cover, lens, and radiation absorbing shields to support the EHF communication devices 310 of the coupler strip 314 .
- the coupler tiles and coupler strips may be separate from and not integrated into the cover panel 600 .
- the coupler tiles may still be located close to the cover panel 606 such that the coupler tiles and coupler strips are parallel to the cover panel 600 and the EHF communication devices 310 are aligned with the openings in the cover panel 606 .
- FIG. 7A shows a side view of an extremely high frequency (EHF) communication device 700 , in accordance with some embodiments.
- the EHF communication device 700 is an example of an EHF communication device 310 of a server 300 .
- the EHF communication device 700 may be an integrated circuit (IC) package mounted on the circuit board 406 .
- FIG. 7B shows a perspective view of the EHF communication device 700 , in accordance with some embodiments.
- the EHF communication device 700 may include a die 702 , a lead frame 728 (shown in FIG. 7B ), one or more conductive connectors such as bond wires 704 , a transducer such as antenna 706 , and an encapsulating material 708 .
- the die substrate may be formed using any suitable material, such as, but not limited to, FR4, a glass-reinforced epoxy composite material.
- the die 702 may be mounted in electrical communication with the lead frame 728 .
- the lead frame 728 may be any suitable arrangement of electrically conductive leads that allow one or more other circuits to operatively connect with the die 702 .
- the leads of the lead frame 728 may be embedded or fixed in a lead frame substrate.
- the lead frame substrate may be formed using any suitable insulating material configured to substantially hold the leads in a predetermined arrangement.
- the electrical communication between the die 702 and other components may be accomplished by any suitable method using conductive connectors, such as one or more bond wires.
- the bond wires 704 (shown in FIG. 7B ) may be used to electrically connect points on a circuit of the die 702 with corresponding leads on the lead frame 728 .
- the die 702 may be disposed in a flip chip configuration, in which case the die may be inverted, and the conductive connectors may be bumps or die solder balls, rather than bond wires 704 .
- the antenna 706 may be any suitable structure configured as a transducer to convert between electrical and electromagnetic signals.
- the antenna 706 may be configured to operate in an EHF spectrum (30 GHz to 300 GHz), and may be configured to transmit and/or receive electromagnetic signals.
- the antenna 706 may be constructed as a part of the lead frame 728 .
- the antenna 706 may be separate from but operatively connected to the die 702 by any suitable method, and may be located adjacent to the die 702 .
- the antenna 706 may be connected to the die 702 using the bond wires 704 .
- the antenna 706 may be connected to the die 702 without the use of the bond wires 704 .
- the antenna 706 may be disposed on the die 702 or on the circuit board 406 .
- the die 702 may include a transmitter circuit, a receiver circuit, or a transceiver circuit that is coupled to the antenna 706 .
- the transmitter circuit receives an electrical data signal that includes outbound data and modulates an EHF carrier signal using the electrical data signal to generate an EHF electrical signal provided to the antenna 706 .
- the receiver circuit receives an EHF electrical signal from the antenna 706 and demodulates the EHF electrical signal into an electrical data signal that includes inbound data.
- the transceiver circuit may perform the functions of both the transmitter and the receiver circuits.
- the encapsulating material 708 may hold the various components of the EHF communication device 700 in fixed relative positions.
- the encapsulating material 708 may be any suitable material that provides electrical insulation and physical protection for the electrical and electronic components of the EHF communication device 700 .
- the encapsulating material 708 may be a mold compound, glass, plastic, or ceramic.
- the encapsulating material 708 may be formed in any suitable shape.
- the encapsulating material 708 may be in the form of a rectangular block, encapsulating all components of the EHF communication device 700 except the unconnected leads of the lead frame.
- One or more external connections may be formed with other circuits or components.
- external connections may include ball pads and/or external solder balls for connection to a printed circuit board.
- the EHF communication device 700 may be mounted on the circuit board 406 as discussed above.
- the circuit board 406 may include one or more laminated layers 712 , one of which may be a ground plane 710 .
- the ground plane 710 may be any suitable structure configured to provide an electrical ground to circuits and components on the circuit board 406 .
- FIG. 8 shows a schematic diagram of communication paths in a server 300 , in accordance with some embodiments.
- the communication paths provide for multi-box connectivity.
- the EHF communication devices 310 of a server 300 may be coupled to server-to-server communication components 802 and pass-through communication components 804 in the server 300 that route data between the EHF communication devices 310 .
- server-to-server communication components 802 a portion of the EHF communication devices 310 of the server 300 may be dedicated to server-to-server communications with adjacent servers 300 , and using the pass-through communication components 804 , another portion of the EHF communication devices 310 of the server 300 may be dedicated to pass through communications with more distant servers 300 (e.g., pass communication signals between servers that are not adjacent to each other).
- server-to-server communication components 802 are used for adjacent server-to-server communication in which packets are produced and consumed by the server and adjacent servers, while pass-through communication components 804 are used for pass-through communication between non-adjacent servers in the rack (e.g., packet originating from a server are not consumed by an adjacent server).
- the server 300 includes server-to-server communication components 802 and pass through communication components 804 .
- the server-to-server communication components 802 include server-to-server communication switch devices 808 that perform data routing between the EHF communication devices 310 (e.g., at top and bottom sides of the server 300 ) via link connections 806 .
- Each server-to-server communication switch device 808 may be connected with a plurality of EHF communication devices 310 , which are not shown in FIG. 8 to avoid overcomplicating the disclosure.
- the link connections 806 that interconnect server-to-server communication switch devices 808 may be wired connections, such as serial link connections.
- the pass-through communication components 804 include pass-through communication switch devices 814 that perform data routing between the EHF communication devices via link connections 806 .
- Each pass-through communication switch device 814 may be connected with one or more EHF communication devices 310 , which are not shown in FIG. 8 to avoid overcomplicating the disclosure.
- the server-to-server communication switch devices 808 and the pass-through communication switch devices 814 are also connected, thus connecting the EHF communication devices 310 in the server 300 .
- the link connections 806 connect the switch devices 808 and 814 via the main board 402 , such as through servers on the main board 402 that control the data routing.
- each switch device 808 / 814 is a 4 ⁇ 4 switch IC chip.
- the switch devices 808 / 814 and link connections 806 may provide multiple paths between EHF communication devices such that data can be rerouted in the case of a defective link connection, switch device, EHF communication device, etc.
- the connector board 404 includes the switch devices 808 / 814 .
- Data received by the server 300 may be routed to another (e.g., adjacent) server 106 without passing through the main board 402 .
- the server 106 a may transmit data to the server 106 c using the server 106 b as a pass through.
- the server 106 a sends data to the server 106 b using its pass-through communication components, and the server 106 b may route the data to the server 106 c using its pass-through communication components.
- the data being routed through one or more servers 106 being used as pass through is not routed through the main boards 402 of the devices 106 being used as pass through, or a component on these main boards 402 . Instead, the data may be routed through switch devices 814 on the connector board 404 or some other component separate from the main board 402 . Among other things, the speed of wireless data transfer across stacked servers 300 in the server rack 100 is improved.
- the switch devices 808 / 814 may be in other locations in the server 300 .
- the switch devices 808 or 814 may be on the main board 402 , or on a circuit board with the EHF communication devices.
- a first group of switch devices 808 or 814 route data between EHF communication devices along the top and bottom sides of the server 300
- another group of the switch devices 808 or 814 route data between other EHF communication devices of the server 300 and the main board of the server 300 .
- FIG. 9 shows servers in a server rack 900 , in accordance with some embodiments.
- the server rack 900 is an example of a server rack 100 including a top-of-rack (TOR) switch 902 , compute and memory servers 904 , storage servers 906 , and a power unit 908 .
- the TOR switch 902 is at the top of the server rack 900
- the compute and memory servers 904 are located under the TOR switch 902
- the storage servers 906 are located under the compute and memory servers 904
- the power unit is located under the storage servers 906 at the bottom of the server rack 900 .
- the TOR switch 902 provides networking to the compute and memory servers 904 and the storage servers 906 .
- the compute and memory servers 904 may include processing and memory components of various types.
- the storage servers 906 provide persistent storage resources, which may be functionally decoupled from the computing resources of the compute and memory servers 904 . In some embodiments, the storage servers 906 are used to implement databases or backup storage.
- the storage servers 906 may also be located in a different chassis from the compute and memory servers 904 .
- the power unit 908 provides power to the other components of the server rack 900 , and may be connected to an external power source.
- the compute and memory servers 904 wirelessly communicate with each other via EHF communication devices disposed at the top and/or bottom sides.
- the compute and memory servers 904 further include EHF communication devices at one or more sides that communicate with EHF communication devices in the rack frame 102 of the server rack 100 .
- the rack frame includes wired communication channels 910 that carry data between the compute and memory servers 904 and the storage servers 906 and the TOR switch 902 .
- a wired communication channel 910 may be formed from one or more conductive wires.
- the storage servers 906 or TOR switch 902 includes EHF communication devices to wirelessly connect with the wired communication channels 910 via EHF communication devices in the rack frame 102 .
- the storage servers 906 or TOR switch 902 connect with the wired communication channels 910 using wired connections.
- the wired communication channels 910 include Ethernet connections, such as 40 gigabits/second (G), 100 G, or higher speed connections.
- the compute and memory servers 904 have a rack size of 3 rack units (U) and the storage servers 906 have a rack size of 8 U, and the power unit 908 has a rack size of 5 U.
- the rack frame 102 includes bays 104 that support these rack sizes.
- the size of the bays 104 may be adjustable to allow different stacked configurations of the TOR switch 902 , compute and memory servers 904 , storage servers 906 , and the power unit 908 .
- FIG. 10 shows a compute server 1002 in wireless communication with a memory server 1004 , in accordance with some embodiments.
- the memory and computing resources of the compute and memory servers 904 shown in FIG. 9 may be functionally decoupled and located in separate chassis that are wirelessly connected using EHF communication devices.
- the compute server 1002 includes processors such as central processing units (CPUs), graphical processing units (GPUs), application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs).
- the CPUs implements server applications and or other applications, while the GPUs, FPGAs, or ASICs operate as computing accelerators.
- the memory server 1004 includes memory that may be used by the processing resources of the compute server 1002 , or some other processing resource in the server rack 100 or server rack system 200 .
- EHF communication channels may be used to form connections between components in adjacent rack mountable chassis.
- the connected components may include CPU core to CPU core, CPU core to accelerator (e.g., GPU/FGPA/ASIC), accelerator to accelerator, CPU core to memory, or memory to memory.
- the server rack 900 may include the compute server 1002 in wireless communication with the memory server 1004 using EHF communication devices at interfacing top/bottom sides.
- the EHF communication devices provide a fast, high bandwidth top/bottom pipe between the processing resources in the compute server 1002 and the memory resources in the memory server 1004 .
- the compute server 1002 and memory server 1004 may further include EHF communication devices along one or more sides to wirelessly communicate with the wired communication channels 910 via EHF communication devices in the rack frame 102 .
- Each of the compute server 1002 and memory server 1004 may be connected to and communicate data with the TOR switch 902 .
- the EHF communication devices at the sides provide a fast, high bandwidth side pipe between the processing resources in compute servers and decoupled memory resources in the memory servers.
- the compute server 1002 has a rack size of 2 U
- the memory server 1004 has a rack size of 1 U.
- the compute server 1002 and the memory server 1004 have a combined rack size of 3 U, which is equivalent to the 3 U rack size of the compute and memory servers 904 .
- Groups of a rack mountable compute device 1002 wirelessly coupled to a memory server 1004 as shown in FIG. 10 may be stacked on top of each other, and with adjacent groups communicating with each other at top and bottom sides using EHF communication devices, such as shown for the compute and memory servers 904 in FIG. 9 .
- FIG. 11 shows a storage server 906 , in accordance with some embodiments.
- the storage server 906 includes a hot storage device 1102 , a standby storage device 1104 , and a cold storage device 1106 .
- the devices 1102 , 1104 , and 1106 may provide for redundancy and backup of data of primary systems.
- the hot storage device 1102 may store a backup system that receives frequent updates from a primary system, and may even include real-time updates to keep the primary and backup system synchronized.
- the standby storage device 1104 receives less frequent updates from the primary system than does the hot storage device 1102 , providing a warm standby.
- the cold storage device 1106 receives even less frequent updates from the primary system, and executes when the primary system fails.
- the hot storage device 1102 has a rack size of 2 U
- the standby storage device 1104 has a rack size of 2 U
- the cold storage device 1106 has a rack size of 4 U.
- the hot storage device 1102 , standby storage device 1104 , cold storage device 1106 have a combined rack size of 8 U, which is equivalent to the 8 U rack size of the storage device 905 .
- FIGS. 12A, 12B, 12C, 12D, 12E, and 12F show wireless communication architectures between processors and memories in a server rack 100 , in accordance with some embodiments.
- the processors and memories of the server rack 100 may be connected using EHF communication devices 310 .
- the processors are in a first server 106 and the memory is in a second server 106 adjacent to the first server 106 in the server rack 100 .
- the processors and memory are in a single server 106 .
- Computing and memory resources in a server 106 may communicate wirelessly using EHF communication devices, as shown in greater detail in FIGS. 15A and 15B .
- the wireless communication architecture 1200 includes separate central processing units (CPUs) 1202 wirelessly connected a shared memory 1204 via EHF electromagnetic communication connections formed using EHF communication devices 310 .
- the shared memory 1204 may include uniform memory access (UMA) or symmetric multiprocessing (SMP) architectures.
- the wireless communication architecture 1210 includes CPUs 1212 each connected to a separate memory 1214 using EHF electromagnetic communication connections formed using EHF communication devices 310 .
- Each CPU 1212 is further connected to a network 1218 using EHF communication channels formed with EHF communication devices 310 , or using wired connections.
- the wireless communication architecture 1220 includes CPUs 122 connected to a memory interface switch fabric 1228 using EHF electromagnetic communication connections formed using EHF communication devices 310 .
- the memory interface switch fabric 1228 is connected with a public memory 1224 , such as using wired connections or using EHF electromagnetic communication connections formed with EHF communication devices 310 .
- the public memory is connected with private memories 1226 , and provides an interface between the private memories 1226 and the CPUs 1222 .
- the wireless communication architecture 1230 includes separate central processing units (CPUs) 1232 .
- CPUs central processing units
- a portion of the CPUs 1232 are wirelessly connected a shared memory 1234 via EHF electromagnetic communication connections formed with EHF communication devices 310
- another portion of the CPUs 1232 are wirelessly connected to a shared memory 1236 via EHF electromagnetic communication connections formed using EHF communication devices 310 .
- Each CPU 1232 is further connected to a quick path interconnect (QPI)/HyperTransport switch 1238 using EHF electromagnetic communication channels formed with EHF communication devices 310 , or using wired connections.
- QPI quick path interconnect
- HyperTransport switch 1238 using EHF electromagnetic communication channels formed with EHF communication devices 310 , or using wired connections.
- the wireless communication architecture 1240 includes CPUs 1242 each connected to an associated memory 1244 via EHF electromagnetic communication connections formed using EHF communication devices 310 .
- Each memory 1244 has a direct memory access (DMA) unit 1246 connected to a remote direct memory access (RDMA)/Infiniband (IB) network 1248 .
- the DMA units 1246 communicate with the network 1248 via EHF electromagnetic communication connections formed using EHF communication devices 310 .
- the CPUs 1242 may also be connected to the RDMA/IB network 1248 via the EHF communication channels allow remote access to each of memories 1244 independent of their associated CPUs 1242 (e.g., without consuming CPU processing or cache resources).
- the wireless communication architecture 1250 includes a CPU 1252 connected with accelerators 1254 (e.g., GPUs) via an accelerator (Accel)/GPU/Coprocessor switch fabric 1256 .
- the accelerators 1254 may be used to perform data processing tasks to accelerate workloads of the CPU 1252 .
- the accelerators 1254 may be connected with the CPU 1252 via EHF electromagnetic communication connections formed using EHF communication devices 310 , or via wired connections.
- the CPU 1252 and accelerators 1254 are each connected to a memory interface switch fabric 1262 via EHF electromagnetic communication connections formed using EHF communication devices 310 .
- the public memory 1258 is connected to the memory interface switch fabric 1262 , such via EHF electromagnetic communication connections or wired connections.
- the public memory 1258 is connected with private memories 1260 , and provides an interface between the private memories 1260 and the CPUs 1252 .
- the CPU 1252 , accelerators 1254 , and switch fabric 1256 are in a first server and the switch fabric 1262 , public memory 1258 , and private memory 1260 are in a second server.
- FIG. 13 shows a coupler strip 314 of a server 300 , in accordance with some embodiments.
- the coupler strip 314 may provide server-to-rack EHF electromagnetic wireless communications.
- the coupler strip 314 may be similar to the coupler tile 312 but dimensioned to fit along sides 308 of the chassis 302 .
- the EHF communication devices 310 of the coupler strip 314 may be dedicated for communications with the rack switch 108 via EHF communication devices 310 in the rack frame 102 of the server rack 100 .
- the coupler strip 314 includes a circuit board 1302 , EHF communication devices 310 , and aggregator devices 408 .
- the circuit board 1302 provides electrical interconnections between the aggregator devices 408 and the EHF communication devices 310 , as discussed above for the coupler tile 312 .
- the discussion herein for the coupler tile 312 may be applicable to the coupler strip 314 .
- the coupler strip 314 may also include components that provide structural protection, waveguides, and wireless signal isolation such as the radiation absorbing shields 602 , waveguides 604 , and cover 606 shown in FIGS. 6A and 6B .
- the coupler strip 314 includes four aggregator devices 408 , each facilitating data transfer for four EHF communication devices 310 .
- the coupler strip 314 thus includes sixteen EHF communication devices 310 . If each of the EHF communication devices 310 has a data rate of 6 Gigabits/second (Gbps), then the coupler strip 314 can have a read data rate of greater than 32 Gbps and a write data rate of greater than 32 Gbps. In another example, if each of the EHF communication devices 310 has data rate of 12 Gbps, then the coupler strip 314 can have a read data rate of greater than 64 Gbps and a write data rate of greater than 64 Gbps. In another example, if each of the EHF communication devices 310 has data rate of 28 Gbps, then the coupler strip 314 can have a read data rate of 1 greater than 28 Gbps and a write data rate of greater than 128 Gbps.
- FIG. 14 shows wireless communication between EHF communication devices 310 of a server 1406 and EHF communication devices 1402 of the server rack 100 , in accordance with some embodiments.
- the server rack 100 includes a rack frame 102 defining bays 104 .
- Each of the servers 1406 a , 1406 b , and 1406 c is inserted into a bay 104 .
- Each of the EHF communication devices 310 at the side of the servers 1406 wirelessly communicates with an EHF communication device 1402 located at the side of the bay 104 using EHF electromagnetic signals.
- the EHF communication devices 1402 of the rack frame 102 are connected via the wired communication channel 910 to the rack switch 108 at the top of the server rack 100 .
- the rack switch 108 may include wired connections to the wired communication channel 910 .
- the rack switch 108 also may include EHF communication devices, and the rack switch 108 communicates with the wired communication channel 910 using EHF electromagnetic signals.
- the EHF communication devices 310 at the side of the servers 1406 and/or the EHF communication device 1402 located at the side of the bay 104 are formed in coupler strips 314 .
- the server rack 100 does not include a back panel or associated cabling, thus increasing air flow in the server rack 100 and reducing component costs.
- FIGS. 15A and 15B respectively, show top and side views of EHF communication devices 1510 providing communication between processors 1504 and data storage drives 1514 in a server 1500 , in accordance with some embodiments.
- the server 1500 is an example of a server 106 , and may also use EHF communication devices for external data communication.
- the server 1500 includes a main board 1502 that communicates with one or more data storage drives 1514 using EHF electromagnetic signals.
- the main board 1502 includes, among other things, one or more processors such as CPUs 1504 , and EHF communication devices 1510 .
- the main board 1502 connects the CPUs 1504 to the EHF communication devices 1510 via buses 1582 .
- the CPU is an example of an electronic IC device.
- Each data storage drive 1514 includes one or more non-volatile storage and memory elements 1516 and EHF communication devices 1512 .
- Each data storage drive 1514 also includes a peripheral board 1590 that connects the memory element 1516 to the EHF communication devices 1512 .
- Each EHF communication device 1512 forms an EHF electromagnetic communication channel with an EHF communication device 1510 using EHF electromagnetic signals.
- the EHF communication devices 1510 and 1512 connect the memory element 1516 and the CPU 1504 and allow the CPU to access data from the memory element 1516 .
- the CPU 1504 may read data from and/or write data to the memory element 1516 via the buses 1582 and EHF electromagnetic communication channels.
- the EHF communication devices 1510 and/or 1512 may be formed on coupler strips 314 or coupler tiles 312 , and/or may incorporate additional components such as the radiation absorbing shields 602 , lenses 604 , and the cover 606 as shown in FIGS. 6A and 6B .
- the EHF communication devices 1510 are mounted directly on the main board 1502 , or the EHF communication devices 1512 are mounted directly on the peripheral board 1590 .
- One or more fans 1580 move air 1570 from the front side 1518 of the chassis to the back side 1520 of the chassis, or may otherwise move air through the interior of the chassis.
- the air 1570 is drawn across the data storage drives 1514 and across the main board 1502 and its components.
- the airflow cools the devices within the computing device and can provide better cooling when the restrictions in the airflow path are reduced.
- the fans 1580 may be located close to either the front side 1518 or back side 1520 of the chassis.
- the front side 1518 and the rear side 1520 of the chassis may be formed from cover panels with openings that allow the air to flow through the cover panels.
- the memory element 1516 includes non-volatile memory circuits and non-volatile main memory (e.g., persistent DRAMs (NVDIMM-P)) that store data in a persistent manner.
- the memory element 1516 may use the NVM Express (NVMe) interface specification for peripheral component interconnect PCI Express (PCIe).
- NVMe NVM Express
- PCIe peripheral component interconnect PCI Express
- the memory element 1516 may be connected to the main board 1502 via the EHF electromagnetic communication channels formed by the EHF communication devices 1510 and 1512 .
- the main board 1502 may include one or more buses 1582 that electrically connect the CPU 1504 to the EHF communication devices 1510 .
- the bus 1582 is a peripheral component interconnect (PCIe) express bus.
- the EHF electromagnetic channels provide wireless NVMe communication.
- use of PCIe cables can be eliminated to simplify cable management in the server, which reduces blocking of airflow in the server.
- each data storage drive 1514 includes four memory devices 1516 , and six data storage drives 1514 are connected to the main board 1502 .
- each server 1500 includes twenty-four memory devices 1516 .
- the main board 1502 includes two processors 1504 , with the EHF electromagnetic communication channels connecting each of the processors 1504 to the data storage drives 1514 .
- the data storage drives 1514 are at the front area of the server 1500 that is closer to the front side 1518 than the back side 1520 .
- the main board 1502 is at the back area of the server 1500 between the data storage drives 1514 and the back side 1520 .
- the data storage drives 1514 may be mounted by sliding into the chassis 1598 . This allows efficient access to the data storage drives 1514 from the front side of the server rack 100 for maintenance tasks.
- FIG. 16 shows a flow chart of a process 1600 for managing servers, in accordance with some embodiments.
- a portion of the EHF communication devices in the server may be used to facilitate management of the server.
- EHF communication devices at the side of the server may be used to communicate management data to a central manager via the rack switch 108 .
- the EHF communication devices provide a dedicated management channel for communication of the management data that remains operational even when other components become defective.
- a device management sub-system in the server monitors components of the server (at 1605 ).
- the management sub-system may include a management processor in the server that executes an agent application.
- the monitored components may include hardware components such as processors, memory, switches, aggregators, EHF communication devices, boards, etc.
- monitoring the components may include monitoring states of software executing on the components such as servers, applications, services, containers, virtual machines, operating systems, etc.
- the agent application may monitor the functionality of the components, such as the performance of the components.
- the management sub-system may include sensors that monitor hardware components for components that are unpowered, unresponsive to network requests, providing slow processing or data transfer rates, operating at abnormal temperatures, or otherwise functioning improperly.
- the server includes sensors to monitor the components of the server.
- the server may include a temperature sensor, an air flow sensor, etc.
- the management sub-system generates state data for the components based on the monitoring of the components of the server (at 1610 ).
- the state data indicates the states of the components as determined from the monitoring.
- the management sub-system determines whether the state data indicates a defective component (at 1615 ). For example, the states of the components may be compared with thresholds, and a defective component may be determined upon satisfaction of a threshold. In another example, the state data may indicate that a particular component is not operating, and thus may need repair or replacement.
- the management sub-system In response to determining that the state data indicates a defective component in the server, the management sub-system generates management data including the state data and one or more identifiers of the server (at 1620 ).
- the management data may include a server rack value that identifies the server rack 100 of the server rack system 200 in which the server is located, and a device value that identifies the location of the server in the server rack 100 .
- the management sub-system may store the server rack value and the device value, such as in a memory of the server, and may retrieve these values when generating the management data.
- the management data further includes the state data used to identify the defective component.
- the state data may provide an indication of the nature of the defect to facilitate appropriate remedial action.
- a light emitting diode (LED) service indicator on the chassis of the server is activated in response to determining a defective component to facilitate visual identification.
- LED light emitting diode
- the management sub-system sends the management data to a rack switch using a portion of the EHF communication devices of the server (at 1625 ). With reference to FIG. 14 , the management sub-system sends the management data using one or more of the EHF communication devices 310 at a side surface of the server 1406 .
- the management data is wirelessly transmitted to EHF communication devices 1402 in the rack frame 102 , and then transmitted to the rack switch via the wired communication channel 910 .
- the rack switch routes the management data to a central manager (at 1630 ).
- the central manager may be an application executing on a processor of another server in the server rack 100 .
- EHF communication devices may also be used to transmit the management data from the rack switch to the server executing the central manager.
- the central manager executes in a server of another server rack 100 of the server rack system 200 .
- the rack switch may transmit the management data to the server in the other server rack 100 via the spine router 202 , the rack switch of the other server rack, and then from the rack switch of the other server rack to the other server using EHF communication devices.
- central manager is on a device remote from the server rack system 200 , and the management data is routed through the server rack system 200 accordingly by the rack switch and spine router 202 .
- the rack switch includes a processor that executes the central manager.
- EHF communication devices provide a dedicated management channel for communication between the management sub-system and the central manager.
- This dedicated management channel may be separate from other communication paths.
- Such other communication paths may include communication paths involving other EHF communication devices that are not dedicated to the management channel (e.g., shown in FIG. 8 ), and may further include other communication paths such as those without EHF communication devices (e.g., wired data connections).
- the dedicated management channel may use hardware components separate from the other communication paths such that the dedicated management channel may remain operational even when other components become defective.
- the central manager reports the management data (at 1635 ) and may be a software program.
- the central manager may be used by an administrator of the server rack system 200 to manage the servers and their components in each of the server racks 100 .
- the management data can be reported in various ways.
- the central manager may generate a user interface that indicates the server rack value and the device value of the server including the defective component.
- the central manager sends a notification (e.g., application message, text message, email, etc.) including the management data to another device of the administrator.
- the server rack value and the device value may be associated with known locations in the server rack system 200 .
- the reporting allows the server to be located by the administrator for maintenance, repair or replacement.
- the central manager provides for communication with the server including the defective component using the dedicated management channel through EHF communication devices.
- the central manager may query the management sub-system or other component (e.g., server) in the server via the dedicated management channel, or perform remote testing or software repair via the dedicated management channel.
- FIG. 17 shows a management system 1700 for a server, in accordance with some embodiments.
- the system 1700 includes a server 1702 that is being monitored by a central manager 1704 .
- the server 702 includes a management sub-system 1706 including a management processor 1708 and sensors 1710 .
- the management processor 1708 uses the sensors 1710 to generate the management data describing one or more states of a monitored server 1702 , and sends the management data to the central manager 1704 .
- the sensors 1708 may include environmental sensors or software sensors that monitor the states of servers or applications.
- the sensors 1708 may include sensors for temperature, parity check for memories, memory scrubbing results, SMART hard drive data, memory usage (volatile and non-volatile), processor usage, I/O bandwidth utilization, users on a server, I/O activity, component health status, NVMe wear data, NVMe error data, power consumption, air flow, humidity, etc.
- the management processor 1708 is a processor on the main board 402 of the server 1702 . In other embodiments, the management processor 1708 is separate from processors of the main board. For example, the management processor 1708 may be on the connector board 404 of the server 1702 .
- the central manager 1704 may be implemented by one or more processors separate from the server 1702 .
- the central manager 1704 may be in another server in the same server rack as the management sub-system 1706 , or may be in another server rack of the same server rack system, or may be in a different computing device.
- the management sub-system 1706 and the central manager 1704 communicate via one or more sets of EHF electromagnetic communication channels 1712 formed from paired sets of EHF communication devices as discussed herein.
Abstract
A system, such as a server rack, includes rack mountable devices that communicate using one or more extremely high frequency (EHF) communication devices. A rack mountable device includes a rack mountable chassis including multiple sides, and a circuit board along a side of the rack mountable chassis, the circuit board having a surface. The EHF communication devices are positioned along the side of the rack mountable chassis and attached to the surface of the circuit board. Each of the EHF communication devices is configured to convert between an EHF electromagnetic signal and an electrical data signal. The EHF communication devices form EHF communication channels with other EHF communication devices in other rack mountable devices, or in the server rack to provide high bandwidth data transfer utilizing space on the sides of the rack mountable device.
Description
- This application is a continuation of U.S. application Ser. No. 15/838,238, filed Dec. 11, 2017, which is hereby incorporated by reference in its entirety.
- An issue that is known to plague data centers is cabling. For example, the term “spaghetti cabling” is used to refer to the unkempt cabling mess that is often found springing out from the back panels of server racks at a data center. Spaghetti cabling may result from inexperience, lack of foresight, or laziness of the ones responsible for setting up the servers, and generally leads to poor cooling from impeded air flow and to maintenance nightmares. Although spaghetti cabling may be prevented, or at least mitigated, by shortening, bundling, and labeling the cables, the use of cables itself may be a limiting factor in the scalability of a server rack.
- For example, a cable port to which a cable connects generally has dimensions that conform to a widely-used standard and thus cannot be easily changed. Due to the inflexible dimensions of the cable ports and the limited amount of space on a server rack back panel, a limited number of cable ports can be physically implemented on the server rack back panel. This means that, even though the bandwidth of the server rack can be increased by increasing the number of cable ports, a port density issue may arise in which the amount of increase in the bandwidth would still be constrained by the limited number of cable ports that can be physically implemented.
- A system includes components that use extremely high frequency (EHF) communication devices to form EHF electromagnetic communication channels for data transfer. The system may include a server rack including one or more rack mountable devices. A rack mountable device includes a rack mountable chassis having multiple sides, a circuit board having a surface that extends along a side of the rack mountable chassis, and one or more EHF communication devices attached to the surface of circuit board. Each EHF communication device converts between an EHF electromagnetic signal and an electrical data signal.
- The system may include a rack frame including multiple bays. Rack mountable devices are inserted into the bays. The EHF communication devices of each rack mountable device may be located along top and bottom surfaces of the rack mountable device to form EHF electromagnetic communication channels with other EHF communication devices of other rack mountable devices in the rack frame. The rack mountable devices may include servers, with the EHF communication devices providing server-to-server communications. In another example, an EHF communication device may be located along a side surface of the rack mountable device to form an EHF electromagnetic communication channel with another EHF communication device located along a side of the rack mountable chassis, such as to provide server-to-rack communications.
- In some embodiments, a rack mountable device uses an EHF communication device to communicate management data via an EHF electromagnetic signal. The management data may include an identifier of the rack mountable device and state information regarding components of the rack mountable device.
- In some embodiments, a rack mountable device uses EHF communication devices to communicate between a main board and a data storage drive in the rack mountable device. For example, the main board may include a processor that accesses data from a memory device of the data storage drive using one or more EHF electromagnetic communication channels formed by EHF communication devices on the main board and the data storage drive. The data storage drive may include traditional mechanical disks (HDD), non-volatile memory (FLASH/SSD/NVMe) and non-volatile main memory such as persistent DRAMs (NVDIMM-P).
-
FIG. 1 shows a server rack, in accordance with some embodiments. -
FIG. 2 shows a server rack system, in accordance with some embodiments. -
FIG. 3 shows a rack mountable device for a server rack, in accordance with some embodiments. -
FIG. 4 shows a cross sectional side view of rack mountable devices in a server rack, in accordance with some embodiments. -
FIG. 5 shows a coupler tile of a rack mountable device, in accordance with some embodiments. -
FIG. 6A shows a cover panel structure, in accordance with some embodiments. -
FIG. 6B shows a cross sectional side view of the cover panel structure ofFIG. 6A , in accordance with some embodiments. -
FIG. 7A shows an extremely high frequency (EHF) communication device, in accordance with some embodiments. -
FIG. 7B shows a perspective view of the EHF communication device ofFIG. 7A , in accordance with some embodiments. -
FIG. 8 shows communication paths in a rack mountable device, in accordance with some embodiments. -
FIG. 9 shows rack mountable devices in a server rack, in accordance with some embodiments. -
FIG. 10 shows a server rack mountable compute device in wireless communication with a rack mountable memory device, in accordance with some embodiments. -
FIG. 11 shows a storage rack mountable device of a server rack, in accordance with some embodiments. -
FIGS. 12A, 12B, 12C, 12D, 12E, and 12F show wireless communication architectures between processors and memories in a server rack, in accordance with some embodiments. -
FIG. 13 shows a coupler strip, in accordance with some embodiments. -
FIG. 14 shows wireless communication between EHF communication devices of a rack mountable device and EHF communication devices of the server rack, in accordance with some embodiments. -
FIG. 15A shows EHF communication devices providing communication between a processor and memory in a rack mountable device, in accordance with some embodiments. -
FIG. 15B shows a side view of the EHF communication devices providing communication between the processor and the memory in the rack mountable device, in accordance with some embodiments. -
FIG. 16 shows a flow chart of a process for managing rack mountable devices, in accordance with some embodiments. -
FIG. 17 shows a management system for a rack mountable device, in accordance with some embodiments. - The figures depict embodiments of the present disclosure for purposes of illustration only. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles, or benefits touted, of the disclosure described herein.
- In accordance with some embodiments disclosed herein, extremely high frequency (EHF) communication devices that convert between EHF electromagnetic signals and electrical data signals may replace or reduce the use of cables for establishing communication links in or between components of server rack systems, thereby overcoming or reducing cabling issues such as those discussed above. A server rack system may include one or more rack mountable devices, each of which may be a server. For the purpose of describing particular embodiments and convenience, the term “rack mountable device” is herein used interchangeably with the term “server.” However, a rack mountable device is not limited to a server and may include any device. As an example, a server may include a chassis and one or more EHF communication devices along one or more surfaces of the chassis. The server may include a computing device, a storage device, a switching device, a communication device, etc. and may be configured to mount in a server rack. Each EHF communication device may form an EHF communication channel with another EHF communication device, which may be disposed in an adjacent server in the server rack or in a bay of the server rack, to provide high bandwidth data transfer for the server.
-
FIG. 1 shows aserver rack 100, in accordance with some embodiments. Theserver rack 100 includes arack frame 102 defining an array ofbays 104. Aserver 106 may be configured to insert into each of thebays 104. Theserver rack 100 may provide power, data communication, and cooling to each of theservers 106 in theserver rack 100. Eachserver 106 may include a rack mountable chassis with EHF communication devices on the chassis. For example, EHF communication devices may be located at the top and bottom surfaces of theservers 106 to provide server-to-server wireless communications, and EHF communication devices may be located on side surfaces of theservers 106 to provide server-to-rack wireless communications. Examples ofservers 106 include 1U, 2U and 4U servers, networking equipment, and/or storage devices. - The
server rack 100 further includes one or more rack switches 108, such as a top-of-rack (TOR) switch. Arack switch 108 may provide networking capability to theservers 106, as well as other components. In such case, aserver 106 may communicate with other networked devices outside of theserver rack 100 via therack switch 108.Servers 106 in theserver rack 100 may also communicate with each other via therack switch 108. Theservers 106 may communicate with therack switch 108 using EHF communication devices. For example, EHF communication devices of aserver 106 may communicate with EHF communication devices of theserver rack 100 via electromagnetic signals, while the EHF communication devices in theserver rack 100 may be electrically connected to therack switch 108, thereby bridging communication between theservers 106 and therack switch 108. -
FIG. 2 shows aserver rack system 200, in accordance with some embodiments. Thesystem 200 includesmultiple server racks 100 connected via aspine router 202. Thespine router 202 may include switches that provide network connection points for servers in theservers 106 of the server racks 100. Thespine router 202 connects with the rack switches 108 (e.g., TOR switches) in the server racks 100 and provides a spine and leaf layer topology for thesystem 200. In other embodiments, server racks 100 may be connected using other types of network topologies. -
FIG. 3 shows aserver 300 for aserver rack 100, in accordance with some embodiments. Theserver 300 is an example of aserver 106 configured to insert into thebays 104 of theserver rack 100. Theserver 300 includes achassis 302 enclosing computing components. Thechassis 302 is a rack mountable chassis having a rectangular cuboid shape with three pairs of opposing sides. Thechassis 302 includes atop cover panel 304, abottom cover panel 306, left andright cover panels 308, a front cover panel, and a rear cover panel. Thechassis 302 may also be referred to as a case. - One or more of the
cover panels EHF communication devices 310 that provide wireless communications with EHF communication devices ofother servers 300 in theserver rack 100, or with other EHF communication devices in theserver rack 100. EachEHF communication device 310 converts between an EHF electromagnetic signal and an electrical data signal. The EHF electromagnetic signal includes electromagnetic radiation in an EHF frequency range. In other embodiments, thecommunication devices 310 may transmit or receive electromagnetic signals in other radio frequency ranges. - When the
server 300 is mounted in abay 104 adjacent toother servers 300, theEHF communication devices 310 in thetop cover panel 304 may form EHF electromagnetic communication channels with an above-adjacent server 300 using EHF electromagnetic signals, and theEHF communication devices 310 in thebottom cover panel 306 may form EHF electromagnetic communication channels with a below-adjacent server 300 using EHF electromagnetic signals. In another example discussed in greater detail in connection withFIG. 14 , theEHF communication devices 310 may be located in the left and/or rightside cover panels 308 to form EHF electromagnetic communication channels with EHF communication devices in theserver rack 100. More particularly, theserver 300 may includeEHF communication devices 310 in the left and rightside cover panels 308 to communicate with EHF communication devices at the sides of therack frame 102. In other embodiments, the back or front side cover panels of theserver 300 may include EHF communication devices. - In some embodiments, the
EHF communication devices 310 in thetop cover panel 304 and/or thebottom cover panel 306 are formed oncoupler tiles 312. Eachcoupler tile 312 includes multipleEHF communication devices 310 on a printed circuit board (PCB).Multiple coupler tiles 312 may be arranged in a cover panel along a side of thechassis 302 to support differentlysized chassis 302. TheEHF communication devices 310 in the left andright cover panels 308, defined between the oppositetop cover panel 304 andbottom cover panel 306, may be part of coupler strips 314. Eachcoupler strip 314 includes multipleEHF communication devices 310 on a printed circuit board. The coupler strips 314 may have different dimensions (e.g., rectangular) than the coupler tiles 312 (e.g., square) to support mounting at narrower surfaces. Multiple coupler strips 314 may be arranged in the left andright cover panels 308. - The
coupler tile 312 andcoupler strip 314 show example circuit board dimensions that may be used to attach theEHF communication devices 310. Generally, the size or shape ofcoupler tiles 312 or coupler strips 314 may vary, such as based on the dimensions of thechassis 302. TheEHF communication devices 310 may be mounted to circuit boards in various patterns or arrangements. In some embodiments,coupler strip 314 may be included in thetop cover panel 304 orbottom cover panel 306 when space remains after placing thecoupler tiles 312, as shown inFIG. 3 . -
FIG. 4 shows a cross sectional side view ofservers 106 in aserver rack 100, in accordance with some embodiments. Theservers EHF communication devices 310 on their top and bottom cover panels to enable wirelessly communication withadjacent servers 106 via EHF electromagnetic communication channels 410 (or “EHF channels 410”). EHF electromagnetic signals are transmitted and received across the EHFelectromagnetic communication channels 410 to transfer data betweenseparate servers 106. For example, theserver 106 b hasEHF communication devices 310 at the top cover panel to communicate with theserver 106 a above, and hasEHF communication devices 310 at the bottom cover panel to communicate with theserver 106 c below. In other embodiments,multiple servers 106 may be stacked horizontally in aserver rack 100. - Each
EHF communication device 310 of aserver 106 is paired with a correspondingEHF communication device 310 of anadjacent server 106 to send and receive EHF electromagnetic signals. Thus,EHF communication devices 310 onadjacent servers 106 are spaced apart with a common pitch and aligned with each other. The EHF electromagnetic signals provide near-field coupling between pairedEHF communication devices 310, allowing theEHF communication devices 310 to be arranged in dense arrays without cross interference with nearby pairs ofEHF communication devices 310. - With reference to the
server 106 b, each server may include amain board 402, aconnector board 404, andcoupler tiles 312. Themain board 402 is a motherboard or other primary board of theserver 106 b, and may support other computing components such as processors, memories, etc. Theconnector board 404 is connected to themain board 402 to provide an interface to theEHF communication devices 310, and may be a riser board. Theconnector board 404 may be a PCI-express compatible board that plugs into a PCI-express connector of themain board 402. Eachcoupler tile 312 may include a connector to connect with theconnector board 404 via acable 412 that includes conductive wires so that data can be communicated between thecoupler tile 312 and theconnector board 404. In other embodiments, thecoupler tile 312 may be connected to theconnector board 404 and/or themain board 402 via a mezzanine connector rather than a cable. Coupler strips 314 may be similarly connected to theconnector board 404 as shown inFIG. 4 . For example, a cable or mezzanine connector may be used to connect acoupler strip 314 to theconnector board 404 and/or themain board 402. - In some embodiments, each coupler tile 312 (or coupler strip 314) includes a
circuit board 406, theEHF communication devices 310, and one or moredata aggregator devices 408. TheEHF communication devices 310 and thedata aggregator devices 408 are located on and interconnected by thecircuit board 406. Eachaggregator device 408 performs routing of data to and from at least a subset ofEHF communication devices 310 of thecoupler tile 312. - For receiving data, the
aggregator device 408 may receive electrical data signals from multiple EHF communication devices 310 (e.g., in parallel), combine lower data-rate data from the electrical data signals into higher data-rate data, and provide the higher data-rate data to themain board 402 via theconnector board 404. - For transmitting data, the
aggregator device 408 may receive outgoing data from the main board 402 (e.g., via theconnector board 404 and the circuit board 406), and generate multiple electrical data signals from the outgoing data at lower data rates, and provide the electrical data signals to respectiveEHF communication device 310 for wireless transmission (e.g., in parallel). That is, the aggregator device may receive high data-rate electrical data signals and split them across multiple lower-speed EHF channels. In some embodiments, theaggregator device 408 may also aggregate EHF signals into a high data-rate set of electrical signals. - In addition to performing the aggregating and disaggregating/splitting functions, the
aggregator device 408 may also perform additional functions, including: monitoring the link, encryption, decryption, forward error correction, cyclic redundancy code (CRC) generation, error checking, and passing through sideband information from low-speed electrical signals or from the on-board processing on the aggregator. - The
aggregator device 408 may be on an opposite surface of thecoupler tile 314 from theEHF communication devices 310, or may be on the same surface of thecoupler tile 314 as theEHF communication devices 310. In some embodiments, theaggregator devices 408 control switching between transmission and reception states ofEHF communication devices 310. In some embodiments, theconnector board 404 includes theaggregator devices 408 of theserver 106, and the aggregator devices may be omitted from thecoupler tiles 312 or coupler strips 314. -
FIG. 5 shows acoupler tile 312 of aserver 106, in accordance with some embodiments. Thecoupler tile 312 includes thecircuit board 406, theEHF communication devices 310, and theaggregator devices 408. Eachaggregator device 408 handles data routing, aggregation, and distribution for a group of four adjacentEHF communication devices 310. Thecircuit board 406 provides electrical interconnections between theaggregator devices 408 and theEHF communication devices 310. In this example, theEHF communication devices 310 and theaggregator devices 408 are formed on the same surface of thecircuit board 406. - In some embodiments, the
coupler tile 312 includes sixteenaggregator devices 408, each facilitating data aggregation for fourEHF communication devices 310. Thecoupler tile 312 thus includes sixty-fourEHF communication devices 310. Thirty-two of theEHF communication devices 310 are receiving devices that receive EHF electromagnetic signals, and thirty-two of theEHF communication devices 310 are transmitting devices that transmit EHF electromagnetic signals. If each of theEHF communication devices 310 has a data rate of 6 Gigabits/second (Gbps), then thecoupler tile 312 can support a read data rate of greater than 128 Gbps and a write data rate of greater than 128 Gbps. In another example, if each of theEHF communication devices 310 has data rate of 12 Gbps, then thecoupler tile 312 can support a read data rate of greater than 256 Gbps and a write data rate of greater than 256 Gbps. In another example, if each of theEHF communication devices 310 has data rate of 28 Gbps, then thecoupler tile 312 can support a read data rate of greater than 512 Gbps and a write data rate of greater than 512 Gbps. -
FIGS. 6A and 6B respectively show perspective and cross sectional views of acover panel 600, in accordance with some embodiments. Thecover panel 600 includes thecircuit board 406, the EHF communication devices 310 (as shown inFIG. 6B ), acover 606, which may include a metal material, that provides structural protection for theEHF communication devices 310,radiation absorbing shields 602 that provide EHF electromagnetic signal isolation among nearbyEHF communication devices 310 on thesame circuit board 406, andwaveguides 604 for guiding the EHF electromagnetic signals to facilitate coupling of pairedEHF communication devices 310. - As shown in
FIG. 6B , eachEHF communication device 310 is surrounded by an EHFradiation absorbing shield 602 to shield theEHF communication device 310 from EHF electromagnetic signals of nearby or adjacentEHF communication devices 310 on thecircuit board 406. Each EHFradiation absorbing shield 602 defines a cavity with anEHF communication device 310 located in the cavity. The EHFradiation absorbing shields 602 are shown as having a hollow cylindrical shape, but other shapes may also be used. In some embodiments, theradiation absorbing shield 602 includes a metallic material or other absorbing material (such as ferrite loaded silicone (or other dielectrics) or carbon loaded silicone (or other dielectrics)). - The
waveguide 604 is a dielectric lens waveguide for the EHF signals emitted from and received by theEHF communication device 310. Thewaveguide 604 is positioned over theEHF communication device 310 and is partially surrounded by the EHFradiation absorbing shield 602. EHF electromagnetic signals emitted by theEHF communication device 310 is received by thewaveguide 604, and is directed by thewaveguide 604 through anopening 608 in thecover 606. Thewaveguide 604 provides a medium to propagate the EHF electromagnetic signal between theopening 608 and anEHF communication device 310. In some embodiments, thewaveguide 604 is plastic, dielectric material having a dielectric constant of at least about 2. In some embodiments, thewaveguides 604 are formed as a unitary lens structure that is shared across multipleEHF communication devices 310 and that includes protrusions extending through theopenings 608 in thecover 606. In some embodiments, waveguides other than thewaveguide 604 may be used. - The
cover 606 is formed over thewaveguide 604, and structurally protects the covered components. Thecover 606 may include theopenings 608 through which thewaveguide 604 may extend to guide EHF electromagnetic signals from theEHF communication devices 310 through theopenings 608. In some embodiments, thecover 606 is a metallic, conductive sheet. In other embodiments, a plastic or dielectric casing material is used instead of metal sheeting. - In some embodiments,
aggregator devices 408 are also formed on thecircuit board 406. Thedata aggregator devices 408 may be positioned on the surface of thecircuit board 406 opposite theEHF communication devices 310, or on the same surface of thecircuit board 406 as theEHF communication devices 310, such as in spaces defined betweenradiation absorbing shields 602 that surround theEHF communication devices 310. In some embodiments, thedata aggregator devices 408 are omitted from thecover panel 600. - The discussion herein regarding the
cover panel 600 that includes acoupler tile 312 may also be applicable to a cover panel that includes coupler strips 314. For example, the cover panel may also include acoupler strip 314, cover, lens, and radiation absorbing shields to support theEHF communication devices 310 of thecoupler strip 314. - In one embodiment, the coupler tiles and coupler strips may be separate from and not integrated into the
cover panel 600. When the coupler tiles and coupler strips are separate from thecover panel 600, the coupler tiles may still be located close to thecover panel 606 such that the coupler tiles and coupler strips are parallel to thecover panel 600 and theEHF communication devices 310 are aligned with the openings in thecover panel 606. -
FIG. 7A shows a side view of an extremely high frequency (EHF)communication device 700, in accordance with some embodiments. TheEHF communication device 700 is an example of anEHF communication device 310 of aserver 300. TheEHF communication device 700 may be an integrated circuit (IC) package mounted on thecircuit board 406.FIG. 7B shows a perspective view of theEHF communication device 700, in accordance with some embodiments. TheEHF communication device 700 may include adie 702, a lead frame 728 (shown inFIG. 7B ), one or more conductive connectors such asbond wires 704, a transducer such asantenna 706, and an encapsulatingmaterial 708. The die substrate may be formed using any suitable material, such as, but not limited to, FR4, a glass-reinforced epoxy composite material. Thedie 702 may be mounted in electrical communication with thelead frame 728. Thelead frame 728 may be any suitable arrangement of electrically conductive leads that allow one or more other circuits to operatively connect with thedie 702. The leads of thelead frame 728 may be embedded or fixed in a lead frame substrate. The lead frame substrate may be formed using any suitable insulating material configured to substantially hold the leads in a predetermined arrangement. - The electrical communication between the die 702 and other components may be accomplished by any suitable method using conductive connectors, such as one or more bond wires. The bond wires 704 (shown in
FIG. 7B ) may be used to electrically connect points on a circuit of the die 702 with corresponding leads on thelead frame 728. In another embodiment, thedie 702 may be disposed in a flip chip configuration, in which case the die may be inverted, and the conductive connectors may be bumps or die solder balls, rather thanbond wires 704. - The
antenna 706 may be any suitable structure configured as a transducer to convert between electrical and electromagnetic signals. Theantenna 706 may be configured to operate in an EHF spectrum (30 GHz to 300 GHz), and may be configured to transmit and/or receive electromagnetic signals. In some embodiments, theantenna 706 may be constructed as a part of thelead frame 728. In other embodiments, theantenna 706 may be separate from but operatively connected to the die 702 by any suitable method, and may be located adjacent to thedie 702. For example, theantenna 706 may be connected to the die 702 using thebond wires 704. Alternatively, in a flip chip configuration, theantenna 706 may be connected to the die 702 without the use of thebond wires 704. In other embodiments, theantenna 706 may be disposed on thedie 702 or on thecircuit board 406. - The
die 702 may include a transmitter circuit, a receiver circuit, or a transceiver circuit that is coupled to theantenna 706. The transmitter circuit receives an electrical data signal that includes outbound data and modulates an EHF carrier signal using the electrical data signal to generate an EHF electrical signal provided to theantenna 706. The receiver circuit receives an EHF electrical signal from theantenna 706 and demodulates the EHF electrical signal into an electrical data signal that includes inbound data. The transceiver circuit may perform the functions of both the transmitter and the receiver circuits. - The encapsulating
material 708 may hold the various components of theEHF communication device 700 in fixed relative positions. The encapsulatingmaterial 708 may be any suitable material that provides electrical insulation and physical protection for the electrical and electronic components of theEHF communication device 700. For example, the encapsulatingmaterial 708 may be a mold compound, glass, plastic, or ceramic. The encapsulatingmaterial 708 may be formed in any suitable shape. For example, the encapsulatingmaterial 708 may be in the form of a rectangular block, encapsulating all components of theEHF communication device 700 except the unconnected leads of the lead frame. One or more external connections may be formed with other circuits or components. For example, external connections may include ball pads and/or external solder balls for connection to a printed circuit board. - The
EHF communication device 700 may be mounted on thecircuit board 406 as discussed above. Thecircuit board 406 may include one or morelaminated layers 712, one of which may be aground plane 710. Theground plane 710 may be any suitable structure configured to provide an electrical ground to circuits and components on thecircuit board 406. -
FIG. 8 shows a schematic diagram of communication paths in aserver 300, in accordance with some embodiments. The communication paths provide for multi-box connectivity. TheEHF communication devices 310 of aserver 300 may be coupled to server-to-server communication components 802 and pass-throughcommunication components 804 in theserver 300 that route data between theEHF communication devices 310. Using the server-to-server communication components 802, a portion of theEHF communication devices 310 of theserver 300 may be dedicated to server-to-server communications withadjacent servers 300, and using the pass-throughcommunication components 804, another portion of theEHF communication devices 310 of theserver 300 may be dedicated to pass through communications with more distant servers 300 (e.g., pass communication signals between servers that are not adjacent to each other). In other words, server-to-server communication components 802 are used for adjacent server-to-server communication in which packets are produced and consumed by the server and adjacent servers, while pass-throughcommunication components 804 are used for pass-through communication between non-adjacent servers in the rack (e.g., packet originating from a server are not consumed by an adjacent server). - The
server 300 includes server-to-server communication components 802 and pass throughcommunication components 804. The server-to-server communication components 802 include server-to-servercommunication switch devices 808 that perform data routing between the EHF communication devices 310 (e.g., at top and bottom sides of the server 300) vialink connections 806. Each server-to-servercommunication switch device 808 may be connected with a plurality ofEHF communication devices 310, which are not shown inFIG. 8 to avoid overcomplicating the disclosure. Thelink connections 806 that interconnect server-to-servercommunication switch devices 808 may be wired connections, such as serial link connections. - The pass-through
communication components 804 include pass-throughcommunication switch devices 814 that perform data routing between the EHF communication devices vialink connections 806. Each pass-throughcommunication switch device 814 may be connected with one or moreEHF communication devices 310, which are not shown inFIG. 8 to avoid overcomplicating the disclosure. The server-to-servercommunication switch devices 808 and the pass-throughcommunication switch devices 814 are also connected, thus connecting theEHF communication devices 310 in theserver 300. In some embodiments, thelink connections 806 connect theswitch devices main board 402, such as through servers on themain board 402 that control the data routing. In some embodiments, eachswitch device 808/814 is a 4×4 switch IC chip. Theswitch devices 808/814 andlink connections 806 may provide multiple paths between EHF communication devices such that data can be rerouted in the case of a defective link connection, switch device, EHF communication device, etc. - In some embodiments, the
connector board 404 includes theswitch devices 808/814. Data received by theserver 300 may be routed to another (e.g., adjacent)server 106 without passing through themain board 402. With reference toFIG. 4 , theserver 106 a may transmit data to theserver 106 c using theserver 106 b as a pass through. Theserver 106 a sends data to theserver 106 b using its pass-through communication components, and theserver 106 b may route the data to theserver 106 c using its pass-through communication components. In some embodiments, the data being routed through one ormore servers 106 being used as pass through is not routed through themain boards 402 of thedevices 106 being used as pass through, or a component on thesemain boards 402. Instead, the data may be routed throughswitch devices 814 on theconnector board 404 or some other component separate from themain board 402. Among other things, the speed of wireless data transfer across stackedservers 300 in theserver rack 100 is improved. - In some embodiments, the
switch devices 808/814 may be in other locations in theserver 300. For example, theswitch devices main board 402, or on a circuit board with the EHF communication devices. - In some embodiment, a first group of
switch devices server 300, another group of theswitch devices server 300 and the main board of theserver 300. -
FIG. 9 shows servers in aserver rack 900, in accordance with some embodiments. Theserver rack 900 is an example of aserver rack 100 including a top-of-rack (TOR)switch 902, compute andmemory servers 904,storage servers 906, and apower unit 908. TheTOR switch 902 is at the top of theserver rack 900, the compute andmemory servers 904 are located under theTOR switch 902, thestorage servers 906 are located under the compute andmemory servers 904, and the power unit is located under thestorage servers 906 at the bottom of theserver rack 900. - The
TOR switch 902 provides networking to the compute andmemory servers 904 and thestorage servers 906. The compute andmemory servers 904 may include processing and memory components of various types. Thestorage servers 906 provide persistent storage resources, which may be functionally decoupled from the computing resources of the compute andmemory servers 904. In some embodiments, thestorage servers 906 are used to implement databases or backup storage. Thestorage servers 906 may also be located in a different chassis from the compute andmemory servers 904. Thepower unit 908 provides power to the other components of theserver rack 900, and may be connected to an external power source. - The compute and
memory servers 904 wirelessly communicate with each other via EHF communication devices disposed at the top and/or bottom sides. The compute andmemory servers 904 further include EHF communication devices at one or more sides that communicate with EHF communication devices in therack frame 102 of theserver rack 100. The rack frame includeswired communication channels 910 that carry data between the compute andmemory servers 904 and thestorage servers 906 and theTOR switch 902. Awired communication channel 910 may be formed from one or more conductive wires. In some embodiments, thestorage servers 906 orTOR switch 902 includes EHF communication devices to wirelessly connect with the wiredcommunication channels 910 via EHF communication devices in therack frame 102. In other embodiments, thestorage servers 906 orTOR switch 902 connect with the wiredcommunication channels 910 using wired connections. In some embodiments, the wiredcommunication channels 910 include Ethernet connections, such as 40 gigabits/second (G), 100G, or higher speed connections. - In some embodiments, the compute and
memory servers 904 have a rack size of 3 rack units (U) and thestorage servers 906 have a rack size of 8 U, and thepower unit 908 has a rack size of 5 U. Therack frame 102 includesbays 104 that support these rack sizes. In some embodiments, the size of thebays 104 may be adjustable to allow different stacked configurations of theTOR switch 902, compute andmemory servers 904,storage servers 906, and thepower unit 908. -
FIG. 10 shows acompute server 1002 in wireless communication with amemory server 1004, in accordance with some embodiments. The memory and computing resources of the compute andmemory servers 904 shown inFIG. 9 may be functionally decoupled and located in separate chassis that are wirelessly connected using EHF communication devices. Thecompute server 1002 includes processors such as central processing units (CPUs), graphical processing units (GPUs), application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). In some embodiments, the CPUs implements server applications and or other applications, while the GPUs, FPGAs, or ASICs operate as computing accelerators. Thememory server 1004 includes memory that may be used by the processing resources of thecompute server 1002, or some other processing resource in theserver rack 100 orserver rack system 200. - In various embodiments, EHF communication channels may be used to form connections between components in adjacent rack mountable chassis. The connected components may include CPU core to CPU core, CPU core to accelerator (e.g., GPU/FGPA/ASIC), accelerator to accelerator, CPU core to memory, or memory to memory.
- With reference to
FIG. 10 , theserver rack 900 may include thecompute server 1002 in wireless communication with thememory server 1004 using EHF communication devices at interfacing top/bottom sides. The EHF communication devices provide a fast, high bandwidth top/bottom pipe between the processing resources in thecompute server 1002 and the memory resources in thememory server 1004. Thecompute server 1002 andmemory server 1004 may further include EHF communication devices along one or more sides to wirelessly communicate with the wiredcommunication channels 910 via EHF communication devices in therack frame 102. Each of thecompute server 1002 andmemory server 1004 may be connected to and communicate data with theTOR switch 902. The EHF communication devices at the sides provide a fast, high bandwidth side pipe between the processing resources in compute servers and decoupled memory resources in the memory servers. - In some embodiments, the
compute server 1002 has a rack size of 2 U, and thememory server 1004 has a rack size of 1 U. As such, thecompute server 1002 and thememory server 1004 have a combined rack size of 3 U, which is equivalent to the 3 U rack size of the compute andmemory servers 904. Groups of a rackmountable compute device 1002 wirelessly coupled to amemory server 1004 as shown inFIG. 10 may be stacked on top of each other, and with adjacent groups communicating with each other at top and bottom sides using EHF communication devices, such as shown for the compute andmemory servers 904 inFIG. 9 . -
FIG. 11 shows astorage server 906, in accordance with some embodiments. Thestorage server 906 includes ahot storage device 1102, astandby storage device 1104, and acold storage device 1106. Thedevices hot storage device 1102 may store a backup system that receives frequent updates from a primary system, and may even include real-time updates to keep the primary and backup system synchronized. Thestandby storage device 1104 receives less frequent updates from the primary system than does thehot storage device 1102, providing a warm standby. Thecold storage device 1106 receives even less frequent updates from the primary system, and executes when the primary system fails. - In some embodiments, the
hot storage device 1102 has a rack size of 2 U, thestandby storage device 1104 has a rack size of 2 U, and thecold storage device 1106 has a rack size of 4 U. As such, thehot storage device 1102,standby storage device 1104,cold storage device 1106 have a combined rack size of 8 U, which is equivalent to the 8 U rack size of the storage device 905. Groups of ahot storage device 1102, astandby storage device 1104, and acold storage device 1106 as shown inFIG. 10 may be stacked on top of each other, with each of thehot storage device 1102,standby storage device 1104,cold storage device 1106 including EHF couplers at one or more side surfaces to wirelessly connect with theTOR switch 902 via the wiredcommunication channels 910. -
FIGS. 12A, 12B, 12C, 12D, 12E, and 12F show wireless communication architectures between processors and memories in aserver rack 100, in accordance with some embodiments. The processors and memories of theserver rack 100 may be connected usingEHF communication devices 310. In some embodiments, the processors are in afirst server 106 and the memory is in asecond server 106 adjacent to thefirst server 106 in theserver rack 100. In other embodiments the processors and memory are in asingle server 106. Computing and memory resources in aserver 106 may communicate wirelessly using EHF communication devices, as shown in greater detail inFIGS. 15A and 15B . - With reference to
FIG. 12A , thewireless communication architecture 1200 includes separate central processing units (CPUs) 1202 wirelessly connected a shared memory 1204 via EHF electromagnetic communication connections formed usingEHF communication devices 310. The shared memory 1204 may include uniform memory access (UMA) or symmetric multiprocessing (SMP) architectures. - With reference to
FIG. 12 B, thewireless communication architecture 1210 includesCPUs 1212 each connected to aseparate memory 1214 using EHF electromagnetic communication connections formed usingEHF communication devices 310. EachCPU 1212 is further connected to anetwork 1218 using EHF communication channels formed withEHF communication devices 310, or using wired connections. - With reference to
FIG. 12C , thewireless communication architecture 1220 includes CPUs 122 connected to a memoryinterface switch fabric 1228 using EHF electromagnetic communication connections formed usingEHF communication devices 310. The memoryinterface switch fabric 1228 is connected with apublic memory 1224, such as using wired connections or using EHF electromagnetic communication connections formed withEHF communication devices 310. The public memory is connected withprivate memories 1226, and provides an interface between theprivate memories 1226 and theCPUs 1222. - With reference to
FIG. 12D , thewireless communication architecture 1230 includes separate central processing units (CPUs) 1232. A portion of theCPUs 1232 are wirelessly connected a sharedmemory 1234 via EHF electromagnetic communication connections formed withEHF communication devices 310, and another portion of theCPUs 1232 are wirelessly connected to a sharedmemory 1236 via EHF electromagnetic communication connections formed usingEHF communication devices 310. EachCPU 1232 is further connected to a quick path interconnect (QPI)/HyperTransport switch 1238 using EHF electromagnetic communication channels formed withEHF communication devices 310, or using wired connections. - With reference to
FIG. 12E , thewireless communication architecture 1240 includesCPUs 1242 each connected to an associatedmemory 1244 via EHF electromagnetic communication connections formed usingEHF communication devices 310. Eachmemory 1244 has a direct memory access (DMA)unit 1246 connected to a remote direct memory access (RDMA)/Infiniband (IB)network 1248. In some embodiments, theDMA units 1246 communicate with thenetwork 1248 via EHF electromagnetic communication connections formed usingEHF communication devices 310. TheCPUs 1242 may also be connected to the RDMA/IB network 1248 via the EHF communication channels allow remote access to each ofmemories 1244 independent of their associated CPUs 1242 (e.g., without consuming CPU processing or cache resources). - With reference to
FIG. 12F , thewireless communication architecture 1250 includes aCPU 1252 connected with accelerators 1254 (e.g., GPUs) via an accelerator (Accel)/GPU/Coprocessor switch fabric 1256. Theaccelerators 1254 may be used to perform data processing tasks to accelerate workloads of theCPU 1252. Theaccelerators 1254 may be connected with theCPU 1252 via EHF electromagnetic communication connections formed usingEHF communication devices 310, or via wired connections. TheCPU 1252 andaccelerators 1254 are each connected to a memoryinterface switch fabric 1262 via EHF electromagnetic communication connections formed usingEHF communication devices 310. Thepublic memory 1258 is connected to the memoryinterface switch fabric 1262, such via EHF electromagnetic communication connections or wired connections. Thepublic memory 1258 is connected withprivate memories 1260, and provides an interface between theprivate memories 1260 and theCPUs 1252. In some embodiments, theCPU 1252,accelerators 1254, andswitch fabric 1256 are in a first server and theswitch fabric 1262,public memory 1258, andprivate memory 1260 are in a second server. -
FIG. 13 shows acoupler strip 314 of aserver 300, in accordance with some embodiments. Thecoupler strip 314 may provide server-to-rack EHF electromagnetic wireless communications. Thecoupler strip 314 may be similar to thecoupler tile 312 but dimensioned to fit alongsides 308 of thechassis 302. TheEHF communication devices 310 of thecoupler strip 314 may be dedicated for communications with therack switch 108 viaEHF communication devices 310 in therack frame 102 of theserver rack 100. - The
coupler strip 314 includes acircuit board 1302,EHF communication devices 310, andaggregator devices 408. Thecircuit board 1302 provides electrical interconnections between theaggregator devices 408 and theEHF communication devices 310, as discussed above for thecoupler tile 312. The discussion herein for thecoupler tile 312 may be applicable to thecoupler strip 314. For example, thecoupler strip 314 may also include components that provide structural protection, waveguides, and wireless signal isolation such as theradiation absorbing shields 602,waveguides 604, and cover 606 shown inFIGS. 6A and 6B . - In some embodiments, the
coupler strip 314 includes fouraggregator devices 408, each facilitating data transfer for fourEHF communication devices 310. Thecoupler strip 314 thus includes sixteenEHF communication devices 310. If each of theEHF communication devices 310 has a data rate of 6 Gigabits/second (Gbps), then thecoupler strip 314 can have a read data rate of greater than 32 Gbps and a write data rate of greater than 32 Gbps. In another example, if each of theEHF communication devices 310 has data rate of 12 Gbps, then thecoupler strip 314 can have a read data rate of greater than 64 Gbps and a write data rate of greater than 64 Gbps. In another example, if each of theEHF communication devices 310 has data rate of 28 Gbps, then thecoupler strip 314 can have a read data rate of 1 greater than 28 Gbps and a write data rate of greater than 128 Gbps. -
FIG. 14 shows wireless communication betweenEHF communication devices 310 of a server 1406 andEHF communication devices 1402 of theserver rack 100, in accordance with some embodiments. As discussed above, theserver rack 100 includes arack frame 102 definingbays 104. Each of theservers bay 104. Each of theEHF communication devices 310 at the side of the servers 1406 wirelessly communicates with anEHF communication device 1402 located at the side of thebay 104 using EHF electromagnetic signals. TheEHF communication devices 1402 of therack frame 102 are connected via the wiredcommunication channel 910 to therack switch 108 at the top of theserver rack 100. Therack switch 108 may include wired connections to the wiredcommunication channel 910. In some embodiments, therack switch 108 also may include EHF communication devices, and therack switch 108 communicates with the wiredcommunication channel 910 using EHF electromagnetic signals. In some embodiments, theEHF communication devices 310 at the side of the servers 1406 and/or theEHF communication device 1402 located at the side of thebay 104 are formed in coupler strips 314. - In some embodiments, the
server rack 100 does not include a back panel or associated cabling, thus increasing air flow in theserver rack 100 and reducing component costs. - EHF Communication within a Server
-
FIGS. 15A and 15B , respectively, show top and side views ofEHF communication devices 1510 providing communication betweenprocessors 1504 and data storage drives 1514 in aserver 1500, in accordance with some embodiments. Theserver 1500 is an example of aserver 106, and may also use EHF communication devices for external data communication. - The
server 1500 includes amain board 1502 that communicates with one or more data storage drives 1514 using EHF electromagnetic signals. Themain board 1502 includes, among other things, one or more processors such asCPUs 1504, andEHF communication devices 1510. Themain board 1502 connects theCPUs 1504 to theEHF communication devices 1510 viabuses 1582. The CPU is an example of an electronic IC device. - Each
data storage drive 1514 includes one or more non-volatile storage andmemory elements 1516 andEHF communication devices 1512. Eachdata storage drive 1514 also includes aperipheral board 1590 that connects thememory element 1516 to theEHF communication devices 1512. EachEHF communication device 1512 forms an EHF electromagnetic communication channel with anEHF communication device 1510 using EHF electromagnetic signals. TheEHF communication devices memory element 1516 and theCPU 1504 and allow the CPU to access data from thememory element 1516. TheCPU 1504 may read data from and/or write data to thememory element 1516 via thebuses 1582 and EHF electromagnetic communication channels. In some embodiments, theEHF communication devices 1510 and/or 1512 may be formed oncoupler strips 314 orcoupler tiles 312, and/or may incorporate additional components such as theradiation absorbing shields 602,lenses 604, and thecover 606 as shown inFIGS. 6A and 6B . In some embodiments, theEHF communication devices 1510 are mounted directly on themain board 1502, or theEHF communication devices 1512 are mounted directly on theperipheral board 1590. - One or
more fans 1580move air 1570 from thefront side 1518 of the chassis to theback side 1520 of the chassis, or may otherwise move air through the interior of the chassis. Theair 1570 is drawn across the data storage drives 1514 and across themain board 1502 and its components. The airflow cools the devices within the computing device and can provide better cooling when the restrictions in the airflow path are reduced. Thefans 1580 may be located close to either thefront side 1518 or backside 1520 of the chassis. Thefront side 1518 and therear side 1520 of the chassis may be formed from cover panels with openings that allow the air to flow through the cover panels. - In some embodiments, the
memory element 1516 includes non-volatile memory circuits and non-volatile main memory (e.g., persistent DRAMs (NVDIMM-P)) that store data in a persistent manner. Thememory element 1516 may use the NVM Express (NVMe) interface specification for peripheral component interconnect PCI Express (PCIe). However, rather than being connected by PCIe slots on themain board 1502, or by a cable connected to a riser card (e.g., riser cards may block airflow and thick cables for PCI express can be difficult to manage) inserted into the PCIe slots on themain board 1502, thememory element 1516 may be connected to themain board 1502 via the EHF electromagnetic communication channels formed by theEHF communication devices main board 1502 may include one ormore buses 1582 that electrically connect theCPU 1504 to theEHF communication devices 1510. In some embodiments, thebus 1582 is a peripheral component interconnect (PCIe) express bus. The EHF electromagnetic channels provide wireless NVMe communication. Among other things, use of PCIe cables can be eliminated to simplify cable management in the server, which reduces blocking of airflow in the server. - In some embodiments, each
data storage drive 1514 includes fourmemory devices 1516, and six data storage drives 1514 are connected to themain board 1502. Thus, eachserver 1500 includes twenty-fourmemory devices 1516. Themain board 1502 includes twoprocessors 1504, with the EHF electromagnetic communication channels connecting each of theprocessors 1504 to the data storage drives 1514. - In some embodiments, the data storage drives 1514 are at the front area of the
server 1500 that is closer to thefront side 1518 than theback side 1520. Themain board 1502 is at the back area of theserver 1500 between the data storage drives 1514 and theback side 1520. The data storage drives 1514 may be mounted by sliding into thechassis 1598. This allows efficient access to the data storage drives 1514 from the front side of theserver rack 100 for maintenance tasks. -
FIG. 16 shows a flow chart of aprocess 1600 for managing servers, in accordance with some embodiments. A portion of the EHF communication devices in the server may be used to facilitate management of the server. For example, EHF communication devices at the side of the server may be used to communicate management data to a central manager via therack switch 108. In some embodiments, the EHF communication devices provide a dedicated management channel for communication of the management data that remains operational even when other components become defective. - A device management sub-system in the server monitors components of the server (at 1605). The management sub-system may include a management processor in the server that executes an agent application. The monitored components may include hardware components such as processors, memory, switches, aggregators, EHF communication devices, boards, etc. In some embodiments, monitoring the components may include monitoring states of software executing on the components such as servers, applications, services, containers, virtual machines, operating systems, etc. The agent application may monitor the functionality of the components, such as the performance of the components. For example, the management sub-system may include sensors that monitor hardware components for components that are unpowered, unresponsive to network requests, providing slow processing or data transfer rates, operating at abnormal temperatures, or otherwise functioning improperly.
- In some embodiments, the server includes sensors to monitor the components of the server. For example, the server may include a temperature sensor, an air flow sensor, etc.
- The management sub-system generates state data for the components based on the monitoring of the components of the server (at 1610). The state data indicates the states of the components as determined from the monitoring.
- The management sub-system determines whether the state data indicates a defective component (at 1615). For example, the states of the components may be compared with thresholds, and a defective component may be determined upon satisfaction of a threshold. In another example, the state data may indicate that a particular component is not operating, and thus may need repair or replacement.
- In response to determining that the state data indicates a defective component in the server, the management sub-system generates management data including the state data and one or more identifiers of the server (at 1620). For example, the management data may include a server rack value that identifies the
server rack 100 of theserver rack system 200 in which the server is located, and a device value that identifies the location of the server in theserver rack 100. The management sub-system may store the server rack value and the device value, such as in a memory of the server, and may retrieve these values when generating the management data. In some embodiments, the management data further includes the state data used to identify the defective component. The state data may provide an indication of the nature of the defect to facilitate appropriate remedial action. In some embodiments, a light emitting diode (LED) service indicator on the chassis of the server is activated in response to determining a defective component to facilitate visual identification. - The management sub-system sends the management data to a rack switch using a portion of the EHF communication devices of the server (at 1625). With reference to
FIG. 14 , the management sub-system sends the management data using one or more of theEHF communication devices 310 at a side surface of the server 1406. The management data is wirelessly transmitted toEHF communication devices 1402 in therack frame 102, and then transmitted to the rack switch via the wiredcommunication channel 910. - The rack switch routes the management data to a central manager (at 1630). The central manager may be an application executing on a processor of another server in the
server rack 100. Here, EHF communication devices may also be used to transmit the management data from the rack switch to the server executing the central manager. In other embodiments, the central manager executes in a server of anotherserver rack 100 of theserver rack system 200. The rack switch may transmit the management data to the server in theother server rack 100 via thespine router 202, the rack switch of the other server rack, and then from the rack switch of the other server rack to the other server using EHF communication devices. In other embodiments, central manager is on a device remote from theserver rack system 200, and the management data is routed through theserver rack system 200 accordingly by the rack switch andspine router 202. In some embodiments, the rack switch includes a processor that executes the central manager. - In some embodiments, EHF communication devices provide a dedicated management channel for communication between the management sub-system and the central manager. This dedicated management channel may be separate from other communication paths. Such other communication paths may include communication paths involving other EHF communication devices that are not dedicated to the management channel (e.g., shown in
FIG. 8 ), and may further include other communication paths such as those without EHF communication devices (e.g., wired data connections). The dedicated management channel may use hardware components separate from the other communication paths such that the dedicated management channel may remain operational even when other components become defective. - The central manager reports the management data (at 1635) and may be a software program. For example, the central manager may be used by an administrator of the
server rack system 200 to manage the servers and their components in each of the server racks 100. The management data can be reported in various ways. The central manager may generate a user interface that indicates the server rack value and the device value of the server including the defective component. In another example, the central manager sends a notification (e.g., application message, text message, email, etc.) including the management data to another device of the administrator. The server rack value and the device value may be associated with known locations in theserver rack system 200. The reporting allows the server to be located by the administrator for maintenance, repair or replacement. - In some embodiments, the central manager provides for communication with the server including the defective component using the dedicated management channel through EHF communication devices. For example, the central manager may query the management sub-system or other component (e.g., server) in the server via the dedicated management channel, or perform remote testing or software repair via the dedicated management channel.
-
FIG. 17 shows amanagement system 1700 for a server, in accordance with some embodiments. Thesystem 1700 includes aserver 1702 that is being monitored by acentral manager 1704. Theserver 702 includes amanagement sub-system 1706 including amanagement processor 1708 andsensors 1710. As discussed in greater detail above in connection withFIG. 16 , themanagement processor 1708 uses thesensors 1710 to generate the management data describing one or more states of a monitoredserver 1702, and sends the management data to thecentral manager 1704. Thesensors 1708 may include environmental sensors or software sensors that monitor the states of servers or applications. For example, thesensors 1708 may include sensors for temperature, parity check for memories, memory scrubbing results, SMART hard drive data, memory usage (volatile and non-volatile), processor usage, I/O bandwidth utilization, users on a server, I/O activity, component health status, NVMe wear data, NVMe error data, power consumption, air flow, humidity, etc. In some embodiments, themanagement processor 1708 is a processor on themain board 402 of theserver 1702. In other embodiments, themanagement processor 1708 is separate from processors of the main board. For example, themanagement processor 1708 may be on theconnector board 404 of theserver 1702. - The
central manager 1704 may be implemented by one or more processors separate from theserver 1702. For example, thecentral manager 1704 may be in another server in the same server rack as themanagement sub-system 1706, or may be in another server rack of the same server rack system, or may be in a different computing device. Themanagement sub-system 1706 and thecentral manager 1704 communicate via one or more sets of EHFelectromagnetic communication channels 1712 formed from paired sets of EHF communication devices as discussed herein. - The foregoing description of the embodiments has been presented for the purpose of illustration; it is not intended to be exhaustive or to limit the patent rights to the precise forms disclosed. Persons skilled in the relevant art would appreciate that many modifications and variations are possible in light of the above disclosure.
- The scope of the patent rights is not limited by this detailed description, but rather by any claims that issue on an application based hereon. Accordingly, the disclosure of the embodiments is intended to be illustrative, but not limiting, of the scope of the patent rights, which is set forth in the following claims.
Claims (20)
1. A system, comprising:
a rack frame including bays;
a wired communication channel extending across the bays of the rack frame;
a first extremely high frequency (EHF) communication device located along a side of a first bay of the rack frame and coupled to the wired communication channel; and
a rack mountable device in the first bay, comprising:
a rack mountable chassis including a plurality of sides; and
a second EHF communication device disposed along a side of the rack mountable chassis facing the side of the first bay, the first EHF communication device configured to form an EHF electromagnetic communication channel with the second EHF communication device to transfer data between the rack mountable device and the wired communication channel.
2. The system of claim 1 , wherein the rack mountable device includes a circuit board having a surface extending along the side of the rack mountable chassis, the second EHF communication device being attached to the surface of the circuit board.
3. The system of claim 1 , wherein the rack mountable device includes a cover panel at the side of the rack mountable chassis, the cover panel including an opening through which an EHF electromagnetic signal is transmitted via the EHF electromagnetic communication channel.
4. The system of claim 3 , wherein the rack mountable device includes a waveguide extending from the second EHF communication device into the opening of the cover panel.
5. The system of claim 4 , wherein the rack mountable device includes a radiation absorbing shield surrounding the second EHF communication device.
6. The system of claim 1 , wherein the rack mountable device further includes:
a main board in the rack mountable chassis and having a connector;
a connector board inserted into the connector; and
a cable connecting the connector board to the circuit board,
wherein data from the main board is provided to the second EHF communication device via the connector board and the cable.
7. The system of claim 1 , wherein each of the first and second EHF communication devices includes:
an integrated circuit;
an antenna connected to the integrated circuit; and
an encapsulating material over the integrated circuit and the antenna.
8. The system of claim 1 , wherein the rack mountable device further includes:
a plurality of second EHF communication devices including the second EHF communication device;
a circuit board having a surface extending along the side of the rack mountable chassis, the second EHF communication devices being attached to the surface;
a main board in the rack mountable chassis, the main board connected with the second EHF communication devices via the circuit board; and
a data aggregator device attached to the circuit board, the aggregator device configured to:
receive electrical data signals from the plurality of second EHF communication devices, each electrical data signal including first data at a first data rate,
generate second data at a second data rate by combining the first data outputted from the second EHF communication devices; and
output the second data.
9. The system of claim 1 , wherein the second data rate is higher than the first data rate.
10. The system of claim 1 , further comprising a rack switch in a second bay of the rack frame, the rack switch configured to communicate data with the rack mountable device via the wired communication channel and the EHF electromagnetic communication channel.
11. The system of claim 10 , further comprising a third EHF communication device located along a side of the second bay and coupled to the wired communication channel, and wherein the rack switch includes a fourth EHF communication device configured to form a second EHF electromagnetic communication channel with the third EHF communication device.
12. The system of claim 10 , further comprising a second rack mountable device in a third bay connected to the rack mountable device via the rack switch.
13. The system of claim 12 , wherein the rack mountable device includes compute servers and the second rack mountable device includes storage servers.
14. The system of claim 12 , wherein the rack mountable device includes compute servers and the second rack mountable device includes memory servers.
15. The system of claim 12 , wherein the rack mountable device includes a hot storage device, the second rack mountable device includes a standby storage device, and the system further includes a third rack mountable device in a third bay, the third rack mountable device connected to the rack mountable device and the second rack mountable device via the rack switch, the third rack mountable device includes a cold storage device.
16. A method, comprising:
converting a first electrical data signal into an extremely high frequency (EHF) electromagnetic signal;
transmitting the EHF electromagnetic signal from a first EHF communication device of a first rack mountable device in a first bay of a rack frame to a second EHF communication device located along a side of the first bay, the second EHF communication device coupled to a wired communication channel extending across bays of the rack frame; and
converting the EHF electromagnetic signal transmitted to the second EHF communication device into a second electrical data signal; and
sending the second electrical data signal to a second rack mountable device in a second bay of the rack frame via the wired communication channel.
17. The method of claim 16 , further comprising:
receiving, by a plurality of first EHF communication devices of the first rack mountable device, second EHF electromagnetic signals from a plurality of second EHF communication devices located along the side of the first bay;
converting, by the first rack mountable device, the second EHF electromagnetic signals into third electrical data signals, each third electrical data signal including first data at a first data rate; and
generating second data at a second data rate by combining the first data of the third electrical data signals.
18. The method of claim 16 , wherein the second rack mountable device includes a rack switch.
19. The method of claim 16 , wherein sending the second electrical data signal to the second rack mountable device in the second bay of the rack frame via the wired communication channel includes:
converting the second electrical data signal into a second EHF electromagnetic signal;
transmitting the second EHF electromagnetic signal from a third EHF communication device located along a side of the second bay to a fourth EHF communication device of the second rack mountable device.
20. The method of claim 16 , wherein the first rack mountable device includes:
a cover panel including an opening through which the EHF electromagnetic signal is transmitted;
a waveguide extending from the first EHF communication device into the opening of the cover panel; and
a radiation absorbing shield surrounding the first EHF communication device.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/855,801 US20200252349A1 (en) | 2017-12-11 | 2020-04-22 | Computing devices using extremely high frequency (ehf) electromagnetic communication |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/838,238 US10673780B2 (en) | 2017-12-11 | 2017-12-11 | Computing devices using extremely high frequency (EHF) electromagnetic communication |
US16/855,801 US20200252349A1 (en) | 2017-12-11 | 2020-04-22 | Computing devices using extremely high frequency (ehf) electromagnetic communication |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/838,238 Continuation US10673780B2 (en) | 2017-12-11 | 2017-12-11 | Computing devices using extremely high frequency (EHF) electromagnetic communication |
Publications (1)
Publication Number | Publication Date |
---|---|
US20200252349A1 true US20200252349A1 (en) | 2020-08-06 |
Family
ID=66629328
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/838,238 Active US10673780B2 (en) | 2017-12-11 | 2017-12-11 | Computing devices using extremely high frequency (EHF) electromagnetic communication |
US16/855,801 Abandoned US20200252349A1 (en) | 2017-12-11 | 2020-04-22 | Computing devices using extremely high frequency (ehf) electromagnetic communication |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/838,238 Active US10673780B2 (en) | 2017-12-11 | 2017-12-11 | Computing devices using extremely high frequency (EHF) electromagnetic communication |
Country Status (3)
Country | Link |
---|---|
US (2) | US10673780B2 (en) |
CN (1) | CN110058656B (en) |
DE (1) | DE102018131769B4 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20190327780A1 (en) * | 2018-04-18 | 2019-10-24 | Rochester Institute Of Technology | Direct Server-to-Server Wireless Data Center Network and Method thereof |
US10798011B2 (en) * | 2017-08-31 | 2020-10-06 | Abb Schweiz Ag | Method and system for data stream processing |
Families Citing this family (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110098156B (en) * | 2018-01-29 | 2023-04-18 | 光宝新加坡有限公司 | Capacitive coupling packaging structure for capacitive coupling isolator |
US10785786B2 (en) * | 2018-09-26 | 2020-09-22 | Micron Technology, Inc. | Remotely executable instructions |
US10779145B2 (en) * | 2018-09-26 | 2020-09-15 | Micron Technology, Inc. | Wirelessly utilizable memory |
US11800676B2 (en) * | 2020-01-31 | 2023-10-24 | Hewlett Packard Enterprise Development Lp | System and method for secure management of a rack |
Family Cites Families (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8554136B2 (en) | 2008-12-23 | 2013-10-08 | Waveconnex, Inc. | Tightly-coupled near-field communication-link connector-replacement chips |
CN106059634B (en) * | 2011-05-12 | 2019-04-02 | 基萨公司 | Expansible high-bandwidth connectivity |
US8714459B2 (en) | 2011-05-12 | 2014-05-06 | Waveconnex, Inc. | Scalable high-bandwidth connectivity |
CN103947126B (en) * | 2011-07-05 | 2016-07-06 | 基萨公司 | There is the EHF communication of electric isolution and electrolyte transmission medium |
WO2013040396A1 (en) * | 2011-09-15 | 2013-03-21 | Waveconnex, Inc. | Wireless communication with dielectric medium |
US9219546B2 (en) * | 2011-12-12 | 2015-12-22 | Corning Optical Communications LLC | Extremely high frequency (EHF) distributed antenna systems, and related components and methods |
US9531425B2 (en) * | 2012-12-17 | 2016-12-27 | Keyssa, Inc. | Modular electronics |
US9496592B2 (en) * | 2014-03-27 | 2016-11-15 | Intel Corporation | Rack level pre-installed interconnect for enabling cableless server/storage/networking deployment |
US9450635B2 (en) * | 2014-04-03 | 2016-09-20 | Intel Corporation | Cableless connection apparatus and method for communication between chassis |
CA2875578A1 (en) * | 2014-12-24 | 2016-06-24 | Stephan HEATH | Systems, computer media, and methods for using electromagnetic frequency (emf) identification (id) devices for monitoring, collection, analysis, use and tracking of personal, medical, transaction, and location data for one or more individuals |
JP2019054315A (en) * | 2016-04-28 | 2019-04-04 | 日本電産エレシス株式会社 | Mounting board, waveguide module, integrated circuit mounting board, microwave module, radar device and radar system |
-
2017
- 2017-12-11 US US15/838,238 patent/US10673780B2/en active Active
-
2018
- 2018-12-11 DE DE102018131769.7A patent/DE102018131769B4/en active Active
- 2018-12-11 CN CN201811511402.2A patent/CN110058656B/en active Active
-
2020
- 2020-04-22 US US16/855,801 patent/US20200252349A1/en not_active Abandoned
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10798011B2 (en) * | 2017-08-31 | 2020-10-06 | Abb Schweiz Ag | Method and system for data stream processing |
US20190327780A1 (en) * | 2018-04-18 | 2019-10-24 | Rochester Institute Of Technology | Direct Server-to-Server Wireless Data Center Network and Method thereof |
US11950304B2 (en) * | 2018-04-18 | 2024-04-02 | Rochester Institute Of Technology | Direct server-to-server wireless data center network and method thereof |
Also Published As
Publication number | Publication date |
---|---|
DE102018131769A1 (en) | 2019-06-13 |
CN110058656B (en) | 2023-07-14 |
DE102018131769B4 (en) | 2021-02-25 |
US20190182179A1 (en) | 2019-06-13 |
CN110058656A (en) | 2019-07-26 |
US10673780B2 (en) | 2020-06-02 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20200252349A1 (en) | Computing devices using extremely high frequency (ehf) electromagnetic communication | |
US10474460B2 (en) | Technologies for optical communication in rack clusters | |
US10768841B2 (en) | Technologies for managing network statistic counters | |
JP2020009433A (en) | Routing assembly | |
GB2524140A (en) | Computer system with groups of processor boards | |
US9791644B2 (en) | Data bus-in-a-box (BiB) system design and implementation | |
US11869544B2 (en) | Electronic device with parallel backplanes and storage device with parallel backplanes | |
US11805592B2 (en) | Circuit board assembly and electronic device | |
US20130077223A1 (en) | Server | |
US20140254640A1 (en) | De-correlating training pattern sequences between lanes in high-speed multi-lane links and interconnects | |
US20200125529A1 (en) | Modular remote direct memory access interfaces | |
US20190243796A1 (en) | Data storage module and modular storage system including one or more data storage modules | |
Saljoghei et al. | MCF-SMF hybrid low-latency circuit-switched optical network for disaggregated data centers | |
Li et al. | Transferring high-speed data over long distances with combined FPGA and multichannel optical modules | |
CN109857218A (en) | The domestic highly reliable blade server of Godson 3A3000 based on VPX framework | |
CN113364606A (en) | Method and system for peer-to-peer data channel of storage device | |
CN109757052A (en) | A kind of communications device data signal transmitting apparatus and communication equipment | |
JP6880242B2 (en) | Breakout module system | |
TWI488573B (en) | Server rack system and server | |
US11178751B2 (en) | Printed circuit board having vias arranged for high speed serial differential pair data links | |
CN102541714B (en) | The implementation method of chip monitoring and device | |
US20230345620A1 (en) | Hatching ground under a pad in a printed circuit board | |
US20230319978A1 (en) | Micro-ground vias for improved signal integrity for high-speed serial links | |
US9491886B1 (en) | Compute and networking function consolidation | |
CN216391067U (en) | Switch management board and switch |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: KEYSSA SYSTEMS, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:PAK, EDWARD T.;ISAAC, ROGER DWAIN;SIGNING DATES FROM 20171214 TO 20171218;REEL/FRAME:052507/0120 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |