WO2019212461A1

WO2019212461A1 - Co-packaged multiplane networks

Info

Publication number: WO2019212461A1
Application number: PCT/US2018/030111
Authority: WO
Inventors: Nicolas MCDONALD; Gary Gostin; Alan Davis
Original assignee: Hewlett Packard Enterprise Development LP
Current assignee: Hewlett Packard Enterprise Development LP
Priority date: 2018-04-30
Filing date: 2018-04-30
Publication date: 2019-11-07
Anticipated expiration: 2020-10-30
Also published as: DE112018007537B4; US11637719B2; DE112018007537T5; CN111869173B; US20210167992A1; CN111869173A

Abstract

A co-packaged, multiplane network includes: an enclosure; a portion of a first network plane disposed within the enclosure and comprising a first plurality of interconnected switches; a portion of a second network plane disposed within the enclosure and comprising a second plurality of interconnected switches, the second network plane being independent of the first network plane and having the same topology as the first network plane; and a plurality of connectors, each connector being communicatively coupled to a respective port of each of the first plurality of interconnected switches and the second plurality of interconnected switches.

Description

CO-PACKAGED MULTIPLANE NETWORKS

BACKGROUND

[0001] Networks frequently include many network components such as network interfaces, switches, routers, etc. The network components receive and generally include a number of interconnected device components that interact with each other and with other network components. The electrical signals of the device components in these components are commonly routed to the edge of an enclosure, which is frequently a chassis, and presented in a“cage”. A cage is an empty connector where different types of connectors can be placed in them. Cables are then run between the network components to establish their interconnectivity.

BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Examples described herein may be understood by reference to the following description taken in conjunction with the accompanying drawings, in which like reference numerals identify like elements.

[0003] Figure 1 conceptually illustrates a particular embodiment of the subject matter claimed below.

[0004] Figure 2 conceptually illustrates a particular embodiment in which the number of network planes is scale upwardly.

[0005] Figure 3 conceptually illustrates a particular embodiment in which the number of switches per network plane is scaled upwardly.

[0006] Figure 4 illustrated one particular embodiment in which the subject matter claimed below is deployed in a mesh network. [0007] Figure 5 illustrated one particular embodiment in which the subject matter claimed below is deployed in a ring network.

[0008] While examples described herein are susceptible to various modifications and alternative forms, the drawings illustrate specific embodiments herein described in detail by way of example. It should be understood, however, that the description herein of specific embodiments is not intended to be limiting to the particular forms disclosed, but on the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the examples described herein and the appended claims.

DETAILED DESCRIPTION

[0009] Illustrative embodiments of the subject matter claimed below will now be disclosed. In the interest of clarity, not all features of an actual implementation are described in this specification. It will be appreciated that in the development of any such actual embodiment, numerous implementation-specific decisions will be made to achieve the developers’ specific goals, such as compliance with system -related and business-related constraints, which will vary from one implementation to another. Moreover, it will be appreciated that such a development effort, even if complex and time-consuming, would be a routine undertaking for those of ordinary skill in the art having the benefit of this disclosure.

[0010] As computing systems become more complex and the demand on their resources intensifies, so has the size, complexity, and cost of the networks being used. Some of these cables that establish interconnectivity between network components are strictly passive and connect to the electrical signals directly. Others have optical transceivers built into the connector. In today’s world , the passive electrical cables are about 3 meters long and cables lengths longer than that use active optical cables (“AOCs”) which are an order of magnitude more expensive. In today’s large-scale interconnection networks, the cost of the cables throughout the system can dominate the total cost.

[0011] For systems that demand large amounts of bandwidth, people resort to connecting a single“node” to multiple network connection points. There are two ways this can be accomplished. First, a network can be built twice as large to handle the nodes connecting twice to the network (/.e. , 16k network endpoints for 8k bi-connected nodes). The other way is to create two distinct networks and connect each node to each network (two 8k networks for 8k dual connected nodes). The cost of network topologies grows in a super-linear fashion which means that a 1 6k network is more than twice the cost of an 8k network. Thus, the second option (two distinct networks) for doubling bandwidth is more cost efficient.

[0012] The present disclosure presents an approach to co-packaging multiplane networks in a manner that reduces cabling demands and costs, one embodiment of which is conceptually shown in Figure 1 . The co-packaged, multiplane network 100 includes first and second network planes 105, 1 10 disposed within an enclosure 1 15. The network planes 105, 1 10 are“disposed within the enclosure” 1 15 in the sense that they are located and positioned within the enclosure 1 15. The manner in which the networks planes 105, 1 10 are located and positioned within the enclosure 115 is not material.

[0013] A“network plane” for present purposes is a set of interconnected switches with no interconnection to any other switch. So, if a switch is interconnected with another switch but not any third switch, then those two interconnected switches define a “network plane”. If the two switches are interconnected to a third switch as well as each other, then the three switches define a network plane. For example, in FIG. 1 , each network plane 105, 1 10 includes a single switch 120, 125, respectively, and the switches 120, 125 are not interconnected. Each of the switches therefore defines a separate network plane relative to one another. This also means that, in this particular embodiment, the entire network planes 120, 125 are located within the enclosure 1 15. [0014] However, the subject matter claimed below is not so limited. Many network planes will have many switches. By definition, within the network plane each of these switches will be interconnected. Also by definition, none of the switches in one network plane will be interconnected with any switch of another network plane. In these embodiments, it is not necessary for the entirety of the network planes to be disposed within the same enclosure. In these embodiments, at least a portion of each of the network planes is disposed within the same enclosure. Note that, in these embodiments, the network planes will have the same topology (e.g., the same number of switches interconnected in the same configuration). Furthermore, the portions of the network planes disposed within any given enclosure will be the corresponding portions of that same topology. At least one such embodiment is disclosed below.

[0015] Returning to Figure 1 , the enclosure 1 15 may be any type of enclosure known to be suitable for this purpose. For example, a chassis may be used. The network planes 105, 1 10 may also be disposed within the enclosure 1 15 in any suitable manner known to the art. For example, they may be mounted on a board (not shown) having flanges on either side that then mate with a groove defined by the enclosure 1 15 as the board is inserted into the enclosure 1 15. Again, however, the manner in which the network planes 105, 1 10 are mounted and positioned within the enclosure 1 15 is not material so long as the network planes 105, 1 10 are disposed within the enclosure 1 15.

[0016] In this particular embodiment, each network plane 105, 1 10 comprises a single switch 120, 125. Note, however, alternative embodiments may employ network planes comprised of any number of switches. The switches 120, 125 may comprise application specific integrated circuits (“ASICs"). The switches 120, 125 include a first plurality of ports and a second plurality of ports, the number of ports in each plurality being equal. In this embodiment, each of the switches 120, 125 has four ports 0-3.

[0017] The network planes 105, 1 10 are“independent” of one another, meaning that there are no electrical or optical interconnects between the two of them. Thus, there are no plane to plane connections. The network planes 105, 1 10 also share the same topology. In the present context,“same topology” means the network planes 105, 1 10 have equal numbers of switches interconnected in the same way, the switches having equal numbers of ports. Since each network plane 105, 1 10 comprises only a single switch, the network planes 105, 1 10 will have the same topology. However, in embodiments wherein network planes comprise multiple switches, this characteristic will also hold true.

[0018] The co-packaged, multiplane network 100 also includes a plurality of connectors 130-133 mounted to the enclosure 1 15. The connectors 130-133 may be mounted to the enclosure 1 15 in any suitable manner known to the art. For example, they may be mounted to the enclosure 1 15 as a part of a cage (not shown) as described above. Each connector 130-133 is communicatively coupled to a respective port 0-3 of each of the switches 120, 125. Since each of the switches 120, 125 has four ports 0-3, there are four connectors 130-133. The connector 130 is communicatively coupled to the port 0 of each of the switches 120, 125. The connector 131 is communicatively coupled to the port 1 of the switches 120, 120, 125. Similarly, the connectors 132, 133 are communicatively coupled to their respective ports 2, 3 of the switches 120, 125 as shown.

[0019] The connectors 130-133 are communicatively coupled to their respective ports 0-3 over lines 140, only one of which is indicated by a reference numeral in Figure 1 . In this particular embodiment, the lines 140 are optical fibers or cables. However, alternative embodiments may use, for example, electrical cables or traces on a printed circuit board (“PCB”). The lines 140 may be active or passive, depending on the implementation, and may employ both electrical and optical technologies. For example, there are electrical cables known to the art that include optical transceivers in one or both of the connectors.

[0020] This approach greatly reduces the number of cables used for interconnection with other network components relative to conventional practice. There are only four connectors 130-133 because there are four ports 0-3 in each switch 120, 125 and because there is only one switch 120, 125 per network plane 105, 1 10. Because there are four connectors 130-133, only four cables are used 150-153— one for each port 0-3 of the switches 120, 125 for interconnection with other network components. In conventional practice, on the other hand, in co-packaged network planes each network plane is connected to a node using a separate set of cables, one cable for each port on each switch for each network plane.

[0021] Each cable 150-153 includes two lines 155, 156— one for the switches 120, 1 25 in each of the network planes 105, 1 10. The lines 155, 156 in the illustrated embodiment are optical fibers such that the cables 150-153 are optical cables. The cable end connectors 160, 165, for each of the cables 150-153 are, accordingly, any suitable connector known in the art for optical connections. Note that only one of each cable end connectors 160, 165 and lines 155, 156 are referenced by number in Figure 1 . However, alternative embodiments may use, for example, electrical lines or some other suitable technology.

[0022] Thus, the number of connectors 130-133 and the number of external cables 150-153 are equal to the number of ports 0-3 in the switches 120, 125. This means that the number of connectors 130-133 is independent of the number of network planes 105, 1 10. Furthermore, the number of lines 155, 156 in each cable 150-1 53 is equal to the number of network planes 105, 1 10.

[0023] Figure 2 depicts an embodiment in which more than two network planes are present in the enclosure to illustrate how this approach scales with the number of network planes. More particularly, Figure 2 conceptually illustrates a co-packaged, multiplane network 200 including three independent network planes 205, 210, 215, each having the same topology. Each is disposed within an enclosure 220. Each network plane 205, 210, 215 comprises a respective switch 225, 230, 235. Each switch 225, 230, 235 includes a plurality of ports 0-3. [0024] The co-packaged, multiplane network 200 also includes a plurality of connectors 240-243. Each connector 240-243 is communicatively coupled to a respective port 0-3 of each of the switches 205, 210, 215 as shown. Note that there are four ports 0-3 for each switch 205, 210, 215 such that the number of connectors 240- 243 is independent of the number of network planes 225, 230, 235 in the enclosure 220. Each external cable 250-253 includes three cabled lines 255-257 one for each of the network planes 225, 230, 235. The lines 255-257 are, again, optical but may be electrical in alternative embodiments.

[0025] The approach also scales with the number of switches per network plane as illustrated in Figure 3. The co-packaged, multiplane network 300 includes two independent network planes 305, 310, which have the same topology, disposed within the enclosure 315. The network plane 305 includes a first switch 320 and a second switch 325. The network plane 310 includes a third switch 330 and.a fourth switch 335. Each of the switches 320, 325, 330, and 335 includes four ports 0-3.

[0026] The co-packaged, multiplane network 300 also contains two pluralities of connectors 340-343 and 350-353, one set for each of the switches in each of the network planes 305, 310. Each of the ports 0-3 for the first switch 320 and the third switch 330 are communicatively coupled to the connectors 340-343 as described above. Each of the ports 0-3 of the second switch 325 and the fourth switch 335 are communicatively coupled to the connectors 350-353 as described above.

[0027] Thus, while the number of connectors 340-343 and 350-353 are independent of the number of network planes, they are dependent on the number switches per network plane. In embodiments where there are three switches per network plane, then there will be three pluralities of connectors as a general rule. However, this relationship is not strictly maintained in all embodiments as will be discussed further below.

[0028] Furthermore, there are two pluralities of cables now, the cables 360-363 and 370-373, one plurality of cables 360-363 and 370-373 for each of the pluralities of connectors 340-343 and 350-353. As discussed above, the number of lines per cable in the cables 360-363 and 370-373 is a function of the number of network planes 305, 310 represented in the enclosure 315. Since there are two network planes 305, 310, each of the cables 360-363 and 370-373 includes two lines 375, 376. Again, the lines 375, 375 are optical but alternative embodiments may use electrical cables instead.

[0029] Figure 4 shows a mesh network 400. In the mesh network 400, there are four network planes designated network plane A, network plane B, network plane C, and network plane D. These network planes A-D are, as described above, independent and have the same topology. They are distributed through the nodes 405— each defined by an enclosure 410 (only one indicated) such that each node 405 contains a respective switch A-D from each of the network planes A-D. That is, at least a portion of each of the network planes A-D is disposed within each of the enclosures 410.

[0030] Each of the switches A-D includes four ports N (“North”), S (“South"), E (“East”), and W (“West”). As there is a single switch from each of the network planes A- D, there is a single plurality of connectors ports N (“North”), S (“South”), E (“East”), and W (‘West”). Each of the N ports of the switches A-D is communicatively coupled to the respective N connectors, and the same holds true for the S ports and S connectors, the E ports and the E connectors, and the W ports and the W connectors. And, since there are four network planes A-D represented within the enclosure 410 of the node, the external cables 415 (only one indicated) will each contain four lines (not shown) as described above.

[0031] Accordingly, the mesh network 400 is a two-dimensional (“2D”) 3x3 array of nodes 405 with four network planes A-D, with 9 switches per network plane implemented using 36 ASICs. While FIG. 4 is shown with 3x3 arrays, the application of co-packaging as described herein is not limited to 3x3 mesh arrays, and that it could be used in a 2x2 array (not shown) or 4x4 arrays (also not shown), for example. It also is not limited to a 2D network. It could be implemented in a one-dimensional mesh network (not shown) implemented using, for example, a string of routers. [0032] The approach disclosed herein may also be deployed in other types of network topologies. For example, the network 500 in Figure 5, has a ring topology. There are four network planes A-D, again, with one switch A-D from each network plane A-D being disposed within the enclosures 505. Each of the switches A-D has three ports L, X, and R and, accordingly, each enclosure 505 includes three connectors L, X, and R. The L ports of the switches A-D are communicatively coupled to the L connectors, the X ports are communicatively coupled to the X connectors, and the R ports are communicatively coupled to the R connectors as shown. Since there is one switch per plane, there is only one plurality of connectors L, X, and R. Finally, because there are four network planes A-D in each enclosure 505, each external cable 515, both between the enclosures 505 and between the enclosures 505 and the endpoints 520, will each include four lines as discussed above.

[0033] Mesh and ring topologies are typically used in small-scale networks. However, the claimed subject matter may also be used in large scale networks as well. Figure 6 illustrates a two-dimensional, 4x4 Hyper X network 600. There are two network planes 605, 610, with the network plane 610 shown on“top” of the network plane 605. FIG. 6 illustrates a conceptual depiction. Each point 0,0-3, 3 in the network plane 605 is interconnected with every other point 0,0-3, 3. Similarly, each point 0,0-3, 3 in the network plane 610 is interconnected with every other point 0,0-3, 3. But no point 0,0-3, 3 in the network plane 605 is interconnected with any point 0,0-3, 3 in the network plane 610 and vice versa. Hence, the network planes 605, 610 definitionally are separate network planes as discussed above.

[0034] Each point 0,0-3, 3 in each network plane 605, 610 is co-packaged with the corresponding point 0,0-3, 3 in the other network plane 605, 610 since both network planes 605, 610 have the same topology. So, for example, the point 0,3 in the network plane 605 is co-packaged with the point 0,3 in the network plane 610. This is true throughout both the network planes 605, 610. Thus, note that only a portion of each network plane 605, 610 is disposed within any given enclosure 620 (only one indicated). This holds true as well for the networks 400, 500 in Figure 4 and Figure 5, respectively. [0035] In each of the networking contexts of Figure 4 and Figure 6, all of the network planes are completely co-packaged throughout the network. That is, each switch of each network plane is co-packaged with at least one switch of each other network plane. This may not be the case in all embodiments. In some embodiments, one or more of the network planes in the network may be omitted from the co-packaging. In other embodiments, one or more switches within the network plane may be co- packaged while others are not. And in still other embodiments, these two variations may be practiced together.

[0036] As noted above, there may be exceptions wherein the number of connectors is not directly related to the number of switches. Because some embodiments may employ multiple switches from a single network plane within the same enclosure, the interconnection between these switches need not necessarily leave the chassis. These interconnections may instead be made entirely within the enclosure. Thus, in these contexts, some connectors may be omitted from the enclosure.

[0037] Thus, in general, this disclosure presents a co-packaged, multiplane network comprising an enclosure, a first network plane, a second network plane, and a plurality of connectors. The first network plane is disposed at least in part within the enclosure and comprises a first plurality of interconnected switches. The second network is also disposed at least in part within the enclosure and comprises a second plurality of interconnected switches. The second network plane is independent of the first network plane and has the same topology as the first network plane. Each connector is communicatively coupled to a respective port of each of the first and second switches.

[0038] In a larger networking context, a co-packaged, multiplane network comprises a plurality of enclosures, a plurality of network planes, and a plurality of cables. Each network plane is independent of the other network planes, has the same topology as the other network planes, and is at least partially disposed within each of the enclosures. The cables interconnect each of the enclosures. The number of cables in each interconnection is a function of the number of ports per switch and the number of switches per plane disposed in each of the enclosures.

[0039] This concludes the detailed description. The particular embodiments disclosed above are illustrative only, as examples described herein may be modified and practiced in different but equivalent manners apparent to those skilled in the art having the benefit of the teachings herein. Furthermore, no limitations are intended to the details of construction or design herein shown, other than as described in the claims below. It is therefore evident that the particular embodiments disclosed above may be altered or modified and all such variations are considered within the scope and spirit of the appended claims. Accordingly, the protection sought herein is as set forth in the claims below.

Claims

CLAIMS WHAT IS CLAIMED IS:

1 . A co-packaged, multiplane network comprising:

an enclosure;

a portion of a first network plane disposed within the enclosure and comprising a first plurality of interconnected switches;

a portion of a second network plane disposed within the enclosure and comprising a second plurality of interconnected switches, the second network plane being independent of the first network plane and having the same topology as the first network plane; and

a plurality of connectors, each connector being communicatively coupled to a respective port of each of the first plurality of interconnected switches and the second plurality of interconnected switches.

2. The co-packaged, multiplane network of claim 1 , wherein the portion of the first network plane disposed within the enclosure and the portion of the second network planed disposed with the enclosure each include two switches.

3. The co-packaged, multiplane network of claim 1 , further comprising a portion of a third network plane disposed within the enclosure, the third network plane being independent of the first network plane and the second network plane and having the same topology as the first network plane and the second network plane, the third network plane comprising a third plurality of interconnected switches, wherein each connector is communicatively coupled to a respective port of the third plurality of interconnected switches.

4. The co-packaged, multiplane network of claim 1 , wherein the each connector is communicatively coupled to the respective ports of the first switch and the second switch using optical cables, electrical cables, or printed circuit board traces.

5. The co-packaged, multiplane network of claim 1 , wherein the enclosure is a chassis.

6. The co-packaged, multiplane network of claim 1 , wherein the network is a part of a mesh network or a ring network.

7. A co-packaged, multiplane network comprising:

an enclosure;

a portion of a first network plane disposed within the enclosure and comprising a first switch, the first switch including a first plurality of ports; a portion of a second network plane disposed within the enclosure and comprising a second switch, the second network plane being independent of the first network plane and having the same topology as the first network plane, the second switch including a second plurality of ports, the number of ports in the first plurality of ports and the second plurality of ports being equal; and

a plurality of connectors mounted to the enclosure, each connector being communicatively coupled to a respective port of each of the first switch and the second switch.

8. The co-packaged, multiplane network of claim 7, further comprising a portion of a third network plane disposed within enclosure, the third network plane being independent of the first network plane and the second network plane and having the same topology as the first network plane and the second network plane, the third network plane comprising a third switch, the third switch including a third plurality of ports equal in number to the first plurality of ports and the second plurality of ports, and wherein each of the connectors is communicatively coupled to a respective port of the third switch.

9. The co-packaged, multiplane network of claim 7, wherein the first network plane further comprises a third switch including a third plurality of ports and the second network plane further comprises a fourth switch including a fourth plurality of ports, the number of ports in the third and fourth pluralities of ports begin equal in number to the first plurality of ports and the second plurality of ports; and

wherein each of the connectors is communicatively coupled to a respective port of the third and fourth switches.

10. The co-packaged, multiplane network of claim 7, wherein the connectors are communicatively coupled to the respective ports of the first switch and the second switch using optical cables, electrical cables, or printed circuit board traces.

1 1 . The co-packaged, multiplane network of claim 7, wherein the enclosure is a chassis.

12. The co-packaged, multiplane network of claim 7, wherein the network is a part of a mesh network or a ring network.

13. A co-packaged, multiplane network comprising:

a plurality of enclosures;

a plurality of network planes, each network plane being independent of the other network planes, having the same topology as the other network planes, and being at least partially disposed within each of the enclosures; a plurality of cables interconnecting each of the enclosures, the number of cables in each interconnection being a function of the number of ports per switch and the number of switches per plane disposed in each of the enclosures.

14. The co-packaged, multiplane network of claim 13, further comprising further comprising a network plane that is not co-packaged with the plurality of network planes.

15. The co-packaged, multiplane network of claim 13, wherein the portion of each network plane disposed within the enclosures comprises multiple interconnected switches.

16. The co-packaged, multiplane network of claim 13, wherein the connectors are communicatively couple using optical cables, electrical cables, or printed circuit board traces.

17. The co-packaged, multiplane network of claim 13, wherein the enclosures are chassis.

18. The co-packaged, multiplane network of claim 13, wherein the network is at least a part of a mesh network or a ring network.