JP2012118974A - Method, apparatus, and computer program for failover operation - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a failover operation for a connection between a PCIE bridge and an input/output (IO) device.SOLUTION: A first set of bussed bits is exchanged between a first PCIE bridge and a first IO device over a first link using a first set of lanes of the first PCIE bridge. In response to detecting a failure in the first link, the first set of lanes is swapped with a second set of lanes of the first PCIE bridge over a second link using the second set of lanes. The second link connects a second PCIE bridge with a second IO device. In response to detecting the failure in the first link, at an IO device end, the first set of lanes is switched with the second set of lanes for exchanging the second set of bussed bits between the first PCIE bridge and the first IO device over the second link using the second set of lanes.

Description

  Embodiments of the present invention generally relate to a Peripheral Component Interconnect Express (PCIE) bus, and more specifically to a method and apparatus for providing cable redundancy and failover to a multi-lane PCIE IO interconnect.

  The PCI (Peripheral Component Interconnect) standard was first introduced in the early 1990s. By using a PCI bridge chip connected to the frontside bus and processor, PCI provides direct access to system memory in the computer system to any peripheral device connected to the PCI bus. The PCI bridge chip can adjust the speed of the PCI bus independently of the speed of the processor to achieve high reliability.

  The PCI Express (PCIE) standard is a successor to the PCI standard, and its related content is incorporated herein by reference. PCI Express can achieve higher transmission rates with fewer physical pins than PCI. Unlike previous generation PCI buses, PCI Express uses a point-to-point bus architecture. Thus, a dedicated bus is used for data transactions between two devices using the PCIE bus system. Dedicated buses are facilitated by switches that establish point-to-point connections between communicating devices. For this reason, the switch is used as an intermediate device and is physically and logically located between two devices attached to the computer system.

  The PCIE switch includes multiple ports to facilitate device attachment to a computer system. The physical connection between a device and a switch port is commonly referred to as a link. Each link includes one or more lanes, and each lane can transmit data in both directions. Each lane is therefore a full-duplex connection.

  A link including a single lane is called an x1 link. Similarly, a link including two lanes or four lanes is referred to as an x2 link or an x4 link, respectively. PCI Express satisfies different bandwidth requirements of various peripheral devices by pre-considering different width interfaces such as x1, x2, x4, x8, x12, x16, and x32. Thus, the dedicated bus can be 1 lane, 2 lanes, 4 lanes, 8 lanes, 12 lanes, 16 lanes, or 32 lanes wide.

  In many cases, modern server class computers use PCI IO adapters as the primary IO adapter technology. CPU enclosures often include a limited number of PCIE adapter slots to customize the IO options for a particular server. However, CPU chassis packaging typically limits this to a very small number of such slots, while computer power per chassis is significantly increased by multi-core processor chips. In general, the server provides a mechanism for connecting a CPU to a PCI adapter slot in one or more additional “IO expansion” chassis. For example, in a PCIE system, a PCI root port (PRP), also referred to as a PCI host bridge (PHB), is a component of the CPU electronics circuit that generates the PCI bus, which directly creates a single PCI I / O. Connect to either adapter slot or PCIE switch in IO expansion chassis. The IO expansion chassis expands the PHB bus to multiple PCIE adapter slots below the switch. IO expansion requires placing one or more PCIE adapter slots in the expansion chassis and connecting those slots to PHBs in the CPU chassis.

  Typically, these IO expansion chassis are physically different electronics circuit chassis or enclosures, so the electronic connection between the PHB in the CPU chassis and the PCIE adapter slot in the IO expansion chassis is Physical cabling between multiple enclosures is required. These cables require an interconnect distance of more than a few inches between the PHB and the PCIE adapter slot and may even extend between different physical racks including CPU and IO expansion enclosures.

  Some aspects of the present disclosure provide a method for providing a failover operation for a connection between a first PCIE bridge and a first input / output (IO) device. The method generally includes a first bus transmission bit between a first PCIE bridge and a first IO device over a first link using a first lane set of the first PCIE bridge. In response to detecting a failure in the first link and exchanging the set, the second lane set of the first PCIE bridge is set between the first PCIE bridge and the first IO device. Swapping from use of the first lane set to use of the second lane set at the PCIE bridge end to use to exchange a second bus transmission bit set over the second link The second link connects the second PCIE bridge to the second IO device, and in response to detecting a failure in the first link, the first PCIE bridge and the first In order to exchange the second bus transmission bit set with the IO device via the second link using the second lane set, at the IO device end, the first lane set Switching from use to use of the second lane set.

  Some aspects of the present invention provide an apparatus for providing a failover operation for a connection between a first PCIE bridge and a first input / output (IO) device. The apparatus is a first link that connects a first PCIE bridge to a first IO device, the first PCIE bridge first between the first PCIE bridge and the first IO device. Connecting the first link and the second PCIE bridge to the second IO device, used to exchange the first bus transmission bit set over the first link using a set of lanes At least one second link, and in response to detecting a failure in the first link, between the first PCIE bridge and the first IO device, the second of the first PCIE bridge In order to exchange the second bus transmission bit set over the second link using the lane set, at the PCIE bridge end, the first PCIE bridge is connected to the first lane set. To the use of the second lane set, and between the first PCIE bridge and the first IO device via the second link using the second lane set. At least one switch for switching from the use of the first lane set to the use of the second lane set at the IO device end to exchange the bus transmission bit set.

  Some aspects of the present invention provide a computer program for providing a failover operation for a connection between a first PCIE bridge and a first input / output (IO) device. The computer program generally includes a computer readable medium that includes code. This code generally includes a first bus transmission bit between the first PCIE bridge and the first IO device via the first link using the first lane set of the first PCIE bridge. Using the second lane set of the first PCIE bridge between the first PCIE bridge and the first IO device in response to detecting a failure in the first link To swap the second bus transmission bit set over the second link, at the PCIE bridge end, swap from the use of the first lane set to the use of the second lane set, and the second Link connects the second PCIE bridge to the second IO device, and in response to detecting a failure in the first link, between the first PCIE bridge and the first IO device, the second Les The use of the first lane set to the use of the second lane set at the IO device end to exchange a second bus transmission bit set over the second link using the second set. Includes code to switch to.

  In order that the foregoing aspects may be achieved and understood in detail, a more particular description of the embodiments of the invention briefly summarized above may be had by reference to the accompanying drawings.

  However, the attached drawings only illustrate exemplary embodiments of the present invention and therefore should not be considered as limiting its scope. This is because the present invention can recognize other equally effective embodiments.

FIG. 3 illustrates an exemplary computer system incorporating a PCI Express (PCIe) structure topology, in accordance with an embodiment of the present disclosure. FIG. 6 illustrates an exemplary base operation of a PCIE system for providing a cable failover mechanism using a multiplexer or crosspoint switch at both the CPU end and the switch end, in accordance with an embodiment of the present disclosure. is there. FIG. 3 illustrates an example of a failover mechanism used during cable failure in the PCIE system of FIG. 2 according to one embodiment of the present disclosure. FIG. 6 illustrates an exemplary process for providing a cable failover mechanism using the PCIE system of FIGS. 2 and 3 using multiplexers at both the CPU end and the switch end, in accordance with an embodiment of the present disclosure. FIG. FIG. 6 is a flow diagram illustrating an exemplary operation for providing a cable failover mechanism, in accordance with an embodiment of the present disclosure. FIG. 4 is an exemplary diagram illustrating the base operation of a PCIe system for providing a failover mechanism using a multiplexer at the switch end, in accordance with an embodiment of the present disclosure. FIG. 7 is an exemplary diagram illustrating a failover mechanism during a cable failure in the PCIe system of FIG. 6, in accordance with an embodiment of the present disclosure. FIG. 8 is a flow diagram illustrating an exemplary process for providing a cable failover mechanism using the PCIE system of FIGS. 6 and 7 using a multiplexer at the switch end, in accordance with an embodiment of the present disclosure. FIG. 6 is a flow diagram illustrating an exemplary operation for providing a cable failover mechanism, in accordance with an embodiment of the present disclosure.

  FIG. 1 is an exemplary diagram illustrating a computer system 100 incorporating a PCI Express (PCIe) structure topology in accordance with an embodiment of the present disclosure. The computer system includes a CPU 102 connected to the root complex 104. The root complex 104 typically generates transaction requests on behalf of the CPU 102. The root complex can be functionally implemented as a discrete device or integrated into a processor (eg, CPU 102). The root complex 104 can include two or more PCI express ports, and multiple PCIE switch devices can be connected to these ports or cascaded from one or more ports.

  The root complex 104 can include a number of PCIE host bridges (PHBs), such as PHBs 106 and 108, for example. According to some aspects, the PHBs 106, 108 can be implemented as discrete devices or integrated within the root complex 104. Each PHB 106, 108 can be connected to a corresponding PCIe switch 142, 144 via an input / output (I / O) bus 130, 132. For example, the PHB 106 is connected to the PCIe switch 142 via the bus 130, and the PHB 108 is connected to the PCIe switch 144 via the bus 132. Each of the switches 142, 144 can be further connected to a PCIe endpoint (EP) 150 via a link 152. Switches 142 and 144 typically provide fanout to their respective I / O buses 130 and 132. By doing so, the ratio of PHB to PCIE adapter can be increased and the number of PHBs required in the CPU chassis can be minimized.

  According to some aspects, the switches 142, 144 have one or more ports to which connectors are attached via links 152, and each connector is further attached to the endpoint 150. Endpoints typically use link 152 to process data with any other device on the computer system (including another endpoint). Each of the switches 142, 144 establish a number of point-to-point connections between the upstream root port and the endpoint device 150 connected to these switches in the computer system.

  A link is a full-duplex communication path between two components in computer system 100. Logically, a port is an interface between a component and a PCI Express link. Physically, a port is a group of transmitters and receivers located on the same chip that defines a link. One link must support at least one lane. Each lane represents a differential signal pair set (one transmission pair and one reception pair). For bandwidth scaling, the link can aggregate multiple lanes denoted xN. Here N is one of the supported link widths. For example, x1 indicates a link having one physical lane, and x8 indicates a link having eight physical lanes. PCI Express provides multiple physical lanes such as single lane, 4 lanes, 8 lanes, 16 lanes, and 32 lanes to accommodate different bandwidth requirements of peripheral devices compliant with PCI Express. In some aspects, each of the IO buses 130, 132 can also have multiple lanes, and this number of lanes typically corresponds to the number of lanes of links that connect to the endpoint 150. For example, each of buses 130 and 132 is an x8 bus. According to some aspects, the PCIE link is a cable, embedded board wiring, board-to-board connection, and any other connection that allows communication between the PCIE bridge and the PCIE switch or device. Including.

  According to some aspects, link / bus lanes may be physically divided into multiple lane sets. For example, the I / O bus 130 connecting the PHB 106 and the PCIe switch 142 is divided into two lane sets 110 and 112 each having 4 lanes. Similarly, the I / O bus 132 connecting the PHB 108 to the PCIe switch 144 is divided into two lane sets 114 and 116 each having 4 lanes. Dividing each of the links / buses into two lane sets is merely for illustrative purposes, and the bus / link is divided into any number of lane sets with each lane set having a minimum of one lane. It will be appreciated that this is possible. For example, a x16 bus / link can be divided into two x8 buses, four x4 buses, eight x2 buses, or sixteen x1 buses. According to some embodiments, this physical partitioning of the bus is not subject to software control and is permanent based on the hardware design.

  At startup, the PCI Express device typically negotiates with the switch to determine the maximum number of lanes that can configure the link. This link width negotiation depends on the maximum width of the link itself (ie, the actual number of physical signal pairs that make up the link), the width of the connector to which the device is attached, and the width of the device itself.

  In some aspects, because the PCIe switches 142, 144 are physically different electronic chassis or enclosures, each of the I / O buses connecting the PHB to each switch is a separate cable such as a physical cable 134, 136, etc. Through the link. For example, the bus 130 passes through the cable 134 and the bus 132 passes through the cable 136. A cable connector (CC) 160 provided at the end of each cable 134, 136 provides an electrical connection between the lane set and the cable. According to some aspects, one problem that arises from connecting the CPU and PCIe switch to an external cable is that the connection to the IO slot that is communicating on the cable due to a cable or cable connector failure There is a service action where the cable connection in one or the other chassis is accidentally disconnected or the cable needs to be removed to repair the cable.

  According to some aspects, redundant cables are provided by connecting each PHB 106, 108 to switches 142, 144, respectively, using separate physical cables, and fail over if one or the other cable 134, 136 fails. Help provide a mechanism. In some aspects, if the cable connecting the first PHB and the first PCIe switch fails due to the failover mechanism, the first PHB and the first At least a part of the data can be exchanged with the PCIe switch. The second active cable is the one connecting the second PHB and the second PCIe switch. For example, if the computer system 100 detects that the cable 136 has failed, data can be exchanged between the PHB 108 and the switch 144 using some of the lanes in the cable 134 that may still be active. However, PHB 106 and switch 142 continue to exchange data on other lanes in cable 134.

  The PCIe architecture allows lane downshifting and upshifting. As a result, the CPU firmware can reduce (downshift) the number of active lanes between the PHB and the corresponding switch, and can restore (upshift) the original number of active lanes. In some aspects, this PCIe system feature is used to provide a failover mechanism during cable failure. For example, if a failure is detected in cable 136, both buses 130 and 132 are downshifted to use only one lane set. Thus, each bus is downshifted from the x8 bus to the x4 bus. Once the lane downshift is complete, the unused lane set on bus 130 is used to exchange data on the active lane set on bus 132 between PHB 108 and switch 144. In some aspects, only bus 130 is downshifted from x8 to x4 and the inactive lane set of bus 130 is used to exchange data between PHB 108 and switch 144.

  FIG. 2 is an exemplary diagram illustrating the base operation of the PCIE system 200 for providing a cable failover mechanism using multiplexers at both the CPU end and the switch end, in accordance with an embodiment of the present disclosure. is there.

  As shown in FIG. 2, the PCIe root complex 104 includes PHBs 106 and 108. The PHB 106 exchanges data with the PCIe switch 142 using the I / O bus 130, and the PHB 108 exchanges data with the PCIe switch 144 using the I / O bus 132. Each of buses 130 and 132 is an x8 bus. The I / O bus 130 is divided into two lane sets 110 and 112 each having 4 lanes. Similarly, the I / O bus 132 is divided into two lane sets 114 and 116 each having 4 lanes. In some aspects, for bus 130, lane set 110 represents higher order lanes 0-3 and lane set 112 represents lower order lanes 4-7. Similarly for bus 132, lane set 114 represents higher order lanes 0-3 and lane set 116 represents lower order lanes 4-7. Cable connectors 160a-160d provided at the end of each cable 134, 136 provide an electrical connection between the lane set and the cable.

  Lane multiplexers (MUXs) (or cross-point electrical switches) 202a-202d are connected between the lane sets of each bus 130 and 132 on both ends (CPU end and switch end) of cables 134 and 136. Enable switching. In some aspects, during the base operation of the PCIe system, cables 134 and 136 are both active. The following is a typical configuration of the MUX during base operation.

At the CPU end, MUX 202a connects lane set 110 to CC 160a and lane set 116 to CC 160c.
At the CPU end, MUX 202c connects lane set 114 to CC 160c and lane set 112 to CC 160a.
At the switch end, MUX 202b connects lane set 110 to PCIe switch 142 and lane set 116 to PCIe switch 144.
At the switch end, MUX 202d connects lane set 114 to PCIe switch 144 and lane set 112 to PCIe switch 142.

  Accordingly, during base operation, lane sets 110 and 112 of bus 130 connect PHB 106 to switch 142, and lane sets 114 and 116 of bus 132 connect PHB 108 to switch 144. The MUX in the switch can be integrated into an IO chassis that includes a PCIe switch. In some aspects, at the CPU end, all lanes can be routed from each PHB to both cables using a multiplexer. For example, the MUX 202a can connect the lane set 110 to the CC 160a or switch the lane set to connect the lane set 110 to the CC 160c. Similarly, a multiplexer at the switch end can reroute a subset of the lanes in each cable to any switch. Of course, the number and configuration of MUXs shown in FIG. 2 are for illustrative purposes, and any number or configuration of MUXs may be used to implement various aspects of the disclosure. Those skilled in the art will recognize.

  According to some aspects, the CPU firmware controls the operation of the PHB and MUXs 202a and 202c. In some aspects, MUXs 202b and 202d are controlled by one or more devices (or EPs) connected to any one of switches 142 and 144. For example, MUX control devices 204a and 204b connected to switches 142 and 144, respectively, can be programmed to control MUXs 202b and 202d. In some aspects, the CPU firmware configures or communicates with the MUX control devices 204a and 204b to perform a lane switch at the switch end using the MUXs 202b and 202d. In some aspects, each of the MUX control devices 204a and 204b can control both the MUXs 202b and 202d. In some embodiments, the MUX control device is a special type of IO chassis control element that is also a PCIE device connected to a switch.

  FIG. 3 is an exemplary diagram illustrating a failover mechanism during a cable failure in the PCIE system of FIG. 2 according to one embodiment of the present disclosure.

  As described above with respect to FIG. 2, during the base operation, all lanes of the PCIE are connected between the PHB and the switch in the IO chassis. Thus, each of the two x8 PHBs 106, 108 connects all eight lanes of the buses 130, 132 to each switch in the IO chassis. If one cable is lost, the CPU firmware first reduces the number of active lanes between the other PHBs and their respective switches using PCIe “lane downshift”. For example, reduce it from x8 bus to x4 bus. This frees up 4 lanes for use by PHB and I / O buses that have lost cable connectivity to the switch. In some aspects, the freed lanes are typically x8 PCIe bus lanes 4-7. According to some aspects, the PHB or root port signals a link state change event to the CPU firmware, such as a link down event related to a PCIE link connection (or cable) failure or loss, and the cable or link CPU firmware can be prompted to check the operating state of In other embodiments, other instructions or mechanisms may be utilized to prompt the CPU firmware to check for cable failures as well.

  In some aspects, the CPU firmware sets the lost PHB on the x4 bus. As already mentioned, all lanes can be routed from each PHB to both cables using "multiplexers" or crosspoint electrical switches. Thus, if one of the cables 134, 136 fails, a failover mechanism is initiated and the CPU firmware sets a multiplexer at the CPU end to connect the lane set (typically lanes 0-3) to the cable. From the lost PHB to an unused lane (typically lanes 4-7) of the other active PHB cable. Also as described above, the multiplexer at the switch end allows a subset of the lanes in each cable to be rerouted to any switch. Again, as part of the failover mechanism, the CPU firmware communicates to the MUX controller device on the PCIE switch via the active PHB cable, sets the multiplexer, and activates the active cable for the PHB. Are routed to lanes 0 to 3 of other switches. This uses PHB lanes 4-7 in the actively connected cable to connect PHB lanes 0-3, which have lost cable connection, to the originally connected switch lanes 0-3, Maintain active PHB lanes 0-3 connectivity to switch lanes 0-3 on uninterrupted cables separate from the PCIE downshift protocol.

  For example, FIG. 3 shows a cable failover mechanism when cable 132 fails and only cable 130 is active. The CPU detects a failure of the cable 132 and uses lane downshifts accordingly to reduce the number of active lanes on the buses 130 and 132 from the x8 bus to the x4 bus. In this example, the lane downshift instructs the PHB 106 to use only the lane set 110 (lanes 0 to 3), and instructs the PHB 108 to use only the lane set 114 (lanes 0 to 3). The CPU firmware instructs the MUX 202c to establish a connection between the lane set 114 and the lanes 4-7 of the active bus 130. At the same time, the CPU firmware communicates with the MUX control device 204a using the I / O bus 130 of the active cable 134. The MUX control device 204a instructs the MUX 202d to establish a connection between the lanes 4-7 of the bus 130 and the lanes 0-3 of the PCIe switch 144. Therefore, this cable failover mechanism uses the lanes 4 to 7 of the PHB 106 in the cable 134 that is actively connected, and the lanes of the PCIe switch 144 that originally connected the lanes 0 to 3 of the PHB 108 that lost the cable connection. Connect to 0-3.

  According to some aspects, upon completing a service action to restore lost cable connectivity, the CPU firmware is informed about the restored cable connectivity (or detected by an electronic cable presence signal). It is also possible to save the configuration of the MUX executed during the failover mechanism, thereby changing the lane multiplexer settings and connecting to each PHB via the corresponding cable Reroute all lanes to the upstream port of the switch.

  FIG. 4 is an illustrative example for providing a cable failover mechanism using the PCIE system of FIGS. 2 and 3 using multiplexers at both the CPU end and the switch end, according to one embodiment of the present disclosure. FIG. At 402, the CPU firmware checks for failures in cables 134 and 136 that connect PHBs 106 and 108 to switches 142 and 144, respectively. If the CPU firmware detects a failure in a cable, such as cable 136, at 404, then at 406, the CPU firmware downshifts both buses 130 and 134 from x8 to x4. At 408, the firmware instructs the MUX 202c at the CPU end to switch the lane set 114 of the PHB with cable failure to lanes 4-7 of the active bus 130. At 410, the firmware uses the switch-end MUX control device 204a to instruct the MUX 202d to switch lanes 4-7 of the active bus 130 to lanes 0-3 of the switch 144. If the cable 136 has not been restored to full connection at 412, the process 400 continues to operate in failover mode and continues to check the cable 136 for connection restoration. If the CPU firmware detects that the cable 136 is restored at 412, the PCIE system is restored to the base operation of FIG. 2 and the bus is restored to the x8 bus.

  According to some aspects, the lane multiplexing electronics circuit (or MUX) can be removed at the CPU end of the cable to gain benefits for electronics circuit cost savings and packaging requirements. However, this has the disadvantage that it limits the cable length and can limit some IO chassis cabling configurations. This is because the electronics circuit timing requirements require tight cable length tolerances between the two cables used for these redundant configurations. Each redundant configuration has a different firmware sequence for the failover and restore mechanism. One skilled in the art may select any of the redundant configurations to suit the needs of a particular PCIE implementation.

  FIG. 5 is a flow diagram illustrating an exemplary operation 500 for providing a cable failover mechanism according to one embodiment of the present disclosure.

  Operation 500 begins at 502 with a first bus between a first PCIe bridge and a first IO device via a first cable using a first lane set of the first PCIe bridge. Exchange transmission bit set. Bus transmission bits are typically data bits transmitted on the bus. At 504, a failure is detected in the first cable. At 506, in response to detecting a failure in the first cable, the unused portion of the second cable connecting the second PCIe bridge and the second IO device is used to Exchange the first bus transmission bit set with the first IO device.

  FIG. 6 is an exemplary diagram illustrating the base operation of a PCIe system 600 for providing a failover mechanism using a multiplexer only at the switch end, according to one embodiment of the present disclosure.

  As shown in FIG. 6, lane multiplexers (MUX) (or crosspoint electrical switches) 202b and 202d provide lane switching between the lane sets of each bus 130 and 132 on the switch ends of cables 134 and 136. It becomes possible. However, there is no MUX at the CPU end, and the lane set is directly connected to the cable. In some aspects, during the base operation of the PCIe system, cables 134 and 136 are both active. The following is a typical configuration of the PCIe system 600 during base operation.

At the CPU end
For PHB 106, lane set 110 connects directly to CC 160a and lane set 112 connects directly to CC 160c.
For PHB 108, lane set 114 connects directly to CC 160c and lane set 116 connects directly to CC 160a.
・ At the switch end,
MUX 202b connects lane sets 110 and 112 to PCIe switch 142 and can be used to switch between lane sets 110 and 112.
MUX 202d connects lane sets 114 and 116 to PCIe switch 144 and can be used to switch between lane sets 114 and 112.

  Accordingly, during base operation, lane sets 110 and 112 of bus 130 connect PHB 116 to switch 142, and lane sets 114 and 116 of bus 132 connect PHB 108 to switch 144. In some aspects, at the CPU end, all lanes can be routed from each PHB to both cables using lane swapping. For example, lane set 114 can be swapped with lane set 116 so that lane set 114 can be routed via cable 160a instead of 160c. Similarly, lane sets 110 and 112 can be swapped and each lane set can be routed via either cable 160a and 160b. As shown in the above paragraph, MUXs 202b and 202d at the switch end can reroute a subset of the lanes in each cable to any switch. Of course, the number and configuration of MUXs are for illustrative purposes, and it will be appreciated by those skilled in the art that any number or configuration of MUXs can be used to implement various aspects of this disclosure. Let's be recognized.

  According to some aspects, the CPU firmware controls lane swapping at the CPU end. As already mentioned, the CPU firmware controls the operation of the PHB, and the MUXs 202b and 202d are connected to one or more devices (or to one of the switches 142 and 144, such as MUX control devices 204a and 204b, for example). EP). Also, each of the MUX control devices 204a and 204b can control both the MUXs 202b and 202d.

  FIG. 7 is an exemplary diagram illustrating a failover mechanism used in response to detection of a cable failure in the PCIe system of FIG. 6, in accordance with one embodiment of the present disclosure. As described above with respect to FIG. 6, during the base operation, all lanes of each PCIE bus 130 and 132 are connected between the PHB and each switch in the IO chassis. Thus, each of the two x8 PHBs 106, 108 connects all 8 lanes in the bus 130, 132 to each switch in the IO chassis. If one of the cables 134, 136 loses connectivity, the CPU firmware uses PCIe “lane downshift” to reduce the number of active lanes between the other PHBs and their respective switches. For example, reduce it from x8 bus to x4 bus. This frees up 4 lanes for use by PHB and I / O buses that have lost cable connectivity to the switch. In some aspects, the freed lanes are typically x8 PCIe bus lanes 4-7.

  According to some embodiments, the CPU firmware sets the PHB with the cable lost to the x4 bus. As already mentioned with respect to the base operation of the PCIE system of FIG. 6, lane swapping can be used to route all lanes from each PHB to both cables. Thus, in a cable failure situation, the failover mechanism begins and the CPU firmware moves the lane set (typically lanes 0-3) from the PHB where the cable was lost to the unused lanes of other active PHB cables ( To reroute to lanes 4-7) typically, swap the lane set of PHBs that have lost cables. Also, as described above, a multiplexer at the switch end allows a subset of the lanes in each cable to be rerouted to any switch. Therefore, as part of the failover mechanism, the CPU firmware communicates to the MUX controller device on the PCIE switch via the active PHB cable, sets the multiplexer, and sets lanes 4-7 of the active cable. Then, the PHB with the lost cable is routed to lanes 0 to 3 of other switches to which the cable was originally connected. This uses PHB lanes 4-7 in the actively connected cable to connect PHB lanes 0-3, which have lost cable connection, to the originally connected switch lanes 0-3, Maintains connectivity of active PHB lanes 0-3 to switch lanes 0-3 on uninterrupted cables separate from the PCIE downshift protocol.

  For example, as shown in FIG. 7, cable 132 fails and only cable 130 is active. The CPU detects a failure of the cable 132 and in response the CPU firmware uses lane downshift to reduce the number of active lanes on the buses 130 and 132 from the x8 bus to the x4 bus. In this example, the lane downshift instructs the PHB 106 to use only the lane set 110 (lanes 0 to 3), and instructs the PHB 108 to use only the lane set 114 (lanes 0 to 3). The CPU firmware supports the PHB 108 to swap between lane sets 114 and 116, and lane set 114 (lanes 3-0) is not the failed cable 136 but lane 4 of bus 130 of the active cable 134. Be routed through ~ 7. At the same time, the CPU firmware communicates with the MUX control device 204a using the I / O bus 130 of the active cable 134 and communicates to the MUX 202d between the lanes 4-7 of the bus 130 and the lanes 0-3 of the PCIe switch 144. Instruct to establish a connection. As a result, the cable failover mechanism uses the lanes 4 to 7 of the PHB 106 in the actively connected cable 134 to use the lanes 0 to 3 of the PHB 108 that have lost the cable connection to the PCIe switch to which the PHB 108 was originally connected. Connect to 144 lanes 3-0.

  According to some aspects, upon completing a service action to restore lost cable connectivity, the CPU firmware is informed about the restored cable connectivity (or may be detected by an electronic cable presence signal). ), Lane sets 116 and 114 can be swapped back to the MUX 202b and 202d configuration performed during the failover mechanism.

  FIG. 8 illustrates an exemplary process 800 for providing a cable failover mechanism using the PCIE system of FIGS. 6 and 7 using a multiplexer at the switch end only, according to one embodiment of the present disclosure. FIG. At 802, the CPU firmware checks for failures in cables 134 and 136 that connect PHBs 106 and 108 to switches 142 and 144, respectively. If the CPU firmware detects a failure in a cable, such as cable 136, at 804, process 800 proceeds to 806, where the CPU firmware downshifts both buses 130 and 132 from x8 to x4. At 808, the firmware instructs the PHB 108 at the CPU end to swap between the lane sets 114 and 116 to route the lane set 114 using the lanes 4-7 of the active bus 130. At 810, the firmware uses the switch end MUX control device 204a to instruct the MUX 202d to establish a connection between lanes 4-7 of the bus 130 and lanes 3-0 of the PCIe switch 144. If the cable 136 has not been restored to full connection at 812, the process 800 continues to operate in failover mode and continues to check the cable 136 for connection restoration. If the CPU firmware detects that the cable 136 is restored at 812, the PCIE system is restored to the base operation of FIG. 6 and the bus is restored to the x8 bus.

  FIG. 9 is a flow diagram illustrating an example operation 900 for providing a cable failover mechanism in accordance with an embodiment of the present disclosure. As shown, operation 900 begins at 902 and is between a first PCIe bridge and a first IO device via a first cable using a first lane set of the first PCIe bridge. To exchange the first bus transmission bit set. At 904, a failure is detected in the first cable. At 906, in response to detecting a failure in the first cable, the first PCIe bridge and the first IO on the second cable using the second lane set of the first PCIE bridge. Exchange the first bus bit set with the device.

  In the following, reference is made to embodiments of the invention. It will be understood, however, that the invention is not limited to the specific embodiments described. The following features and combinations of elements are considered to implement and carry out the present invention, whether related to different embodiments or not. For example, those skilled in the art will appreciate that the present invention is equally applicable to PCIE links that do not utilize cables or do not involve multiple physical enclosures. Such other embodiments utilize wires embedded in computer circuit boards or such as PCIE links between computer circuit boards, such as through midplane connectors, and in the same or adjacent physical enclosures. Equally addressed by the invention, it provides failover for loss of PCIE links. It will also be appreciated that the failover mechanism discussed above is applicable to PCIE configurations where IO devices or adapters are connected directly to the PHB rather than being connected via a switch. A similar failover procedure can also be used to manage failed connections between IO devices connected to switches using switches and MUX control devices.

  Furthermore, while embodiments of the present invention can achieve advantages over other possible solutions and / or prior art, whether a particular advantage is achieved by a given embodiment is not limited to the present invention. Is not limited. For this reason, the following aspects, features, embodiments and advantages are merely exemplary, except as expressly stated in the claims (claims). It is not considered a claim element or limitation. Similarly, references to “the present invention” are not to be construed as generalizations of the subject matter of the present invention incorporated herein, but are explicitly stated in the claims (claims). It is not considered an element or limitation of the appended claims except.

  As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method, or computer program. Accordingly, aspects of the present invention are generally hardware embodiments, generally software embodiments (including firmware, resident software, microcode, etc.), or embodiments that combine software and hardware aspects. Which can all be generally referred to herein as "circuitry", "module", or "system". Further, aspects of the invention may take the form of a computer program embodied in one or more computer readable media having computer readable program code embodied therein.

  Any combination of one or more computer readable media may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or any suitable combination of the foregoing. More specific examples (non-exhaustive listing) of computer readable storage media include: Electrical connections including one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory) Memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. it can.

  A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such propagated signals can take any of a variety of forms, including but not limited to electromagnetic, light, or any suitable combination thereof. A computer readable signal medium is not a computer readable storage medium, but any computer readable medium capable of transmitting, propagating, or transferring a program for use by or in connection with an instruction execution system, apparatus, or device. Possible medium.

  Program code embodied on a computer readable medium may be any suitable medium including, but not limited to, wireless, wired, fiber optic cable, RF, etc., or any suitable combination of the foregoing. Can be transmitted.

  Computer program code for performing the operations of aspects of the present invention includes object oriented programming languages such as Java®, Smalltalk®, C ++, and “C” programming language or similar programming languages, etc. It can be written in any combination of one or more programming languages, including conventional procedural programming languages. The program code may be entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on the remote computer, or entirely It can be run on a remote computer or server. In the latter case, the remote computer can connect to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be , To an external computer (eg, via the Internet using an Internet service provider).

  Aspects of the invention are described below with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing device to generate a machine, thereby enabling the computer or other programmable data processing device to Instructions executed by the processor are adapted to generate means for performing the functions / acts defined in the flowcharts and / or block diagrams, or in both blocks or blocks.

  These computer program instructions may also be stored on a computer readable medium, thereby instructing a computer, other programmable data processing apparatus, or other device to function in a particular manner. The instructions stored on the computer readable medium can generate an article of manufacture that includes instructions to perform the functions / acts defined in the flowcharts and / or block diagrams, or in both blocks or blocks. ing.

  A computer program instruction is loaded into a computer, other programmable data processing apparatus, or other device, and a series of operation steps are executed on the computer, other programmable apparatus, or other device. An implementation process can be generated whereby instructions executing on a computer or other programmable device can perform functions / acts defined in a flowchart and / or block diagram or blocks. Provide a process.

  The flowcharts and block diagrams in the above-described drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code that includes one or more executable instructions for performing a defined logical function (s). it can. It should also be noted that in some alternative implementations, the functions specified in the blocks may occur out of the order specified in the drawings. For example, depending on the functionality involved, two blocks shown in succession may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Also, each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, are special purpose hardware-based systems or special purpose hardware that perform the specified functions or acts. It should be noted that this can be implemented by a combination of computer instructions.

  While the foregoing is directed to embodiments of the invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof, the scope of which is In accordance with the following claims.

100 Computer system 102 CPU
104 PCIE root complex 106, 108 PHB
110, 112, 114, 116 Lane / set 130, 132 Bus 134, 136 Cable 142, 144 PCIE switch 150 EP
160 Cable connector

Claims (20)

A method for providing a failover operation for a connection between a first PCIE bridge and a first input / output (IO) device comprising:
A first bus transmission bit set is set between the first PCIE bridge and the first IO device via a first link using a first lane set of the first PCIE bridge. To exchange,
In response to detecting a failure in the first link, a second lane set of the first PCIE bridge is used between the first PCIE bridge and the first IO device. Swapping the use of the first lane set to the use of the second lane set at the PCIE bridge end to exchange the second bus transmission bit set over the other link. The second link connects a second PCIE bridge to a second IO device;
Responsive to detecting the failure in the first link, between the first PCIE bridge and the first IO device via the second link using the second lane set. Switching from use of the first lane set to use of the second lane set at an IO device end to exchange the second bus transmission bit set;
Including a method.   The method of claim 1, further comprising configuring the first PCIE bridge to stop using the second lane set in response to detecting the failure on the first link. Method.   In response to detecting the failure in the first link, the second IO device and the third bus using the third lane set of the second PCIE bridge using the first link. The method of claim 1, further comprising configuring the second PCIE bridge exchanging transmission bit sets to stop using the third lane set.   The method of claim 1, wherein each of the first and second PCIE bridges includes a PCIE host bridge (PHB).   The method of claim 1, wherein each of the first and second IO devices includes a PCIE switch.   The method of claim 1, wherein the first and second PCIE bridges are housed in a PCIE root complex. An apparatus for providing a failover operation for a connection between a first PCIE bridge and a first input / output (IO) device, comprising:
A first link connecting the first PCIE bridge to the first IO device, wherein the first PCIE bridge is connected between the first PCIE bridge and the first IO device; A first link used to exchange a first bus transmission bit set over a first link using a lane set;
At least one second link connecting the second PCIE bridge to the second IO device;
And using a second lane set of the first PCIE bridge between the first PCIE bridge and the first IO device in response to detecting a failure in the first link. In order to exchange a second bus transmission bit set over the second link, at the PCI bridge end, the first PCIE bridge removes the second lane set from use of the first lane set. Swap to use lane set,
Further, the second bus transmission bit set is exchanged between the first PCIE bridge and the first IO device via the second link using the second lane set. Therefore, an apparatus comprising at least one switch for switching from use of the first lane set to use of the second lane set at an IO device end.   8. The apparatus of claim 7, wherein the first PCIE bridge is configured to stop using the second lane set in response to detecting the failure on the first link.   In response to detecting the failure in the first link, the second IO device and the third bus using the third lane set of the second PCIE bridge using the first link. 8. The apparatus of claim 7, wherein the second PCIE bridge exchanging transmission bit sets is configured to stop using the third lane set.   8. The apparatus of claim 7, wherein the first and second PCIE bridges are controlled by central processing unit (CPU) firmware.   8. The apparatus of claim 7, further comprising a control device connected to the second IO device to control the switch and controlled by the CPU firmware.   The apparatus of claim 7, wherein each of the first and second PCIE bridges includes a PCIE host bridge (PHB).   The apparatus of claim 7, wherein each of the first and second IO devices includes a PCIE switch.   8. The apparatus of claim 7, wherein the first and second PCIE bridges are housed in a PCIE root complex. A computer program for providing a failover operation for a connection between a first PCIE bridge and a first input / output (IO) device,
A computer-readable medium,
A first bus transmission bit set is set between the first PCIE bridge and the first IO device via a first link using a first lane set of the first PCIE bridge. Replace
In response to detecting a failure in the first link, a second lane set of the first PCIE bridge is used between the first PCIE bridge and the first IO device. Swapping the use of the first lane set to the use of the second lane set at the PCI bridge end to exchange the second bus transmission bit set over the link Two links connect the second PCIE bridge to the second IO device;
Responsive to detecting the failure in the first link, between the first PCIE bridge and the first IO device via the second link using the second lane set. To switch from using the first lane set to using the second lane set at an IO device end to exchange the second bus transmission bit set.
A computer program comprising a computer-readable medium containing code for the use.   16. The method of claim 15, further comprising configuring the first PCIE bridge to cease use of the second lane set in response to detecting the failure on the first link. Computer program.   In response to detecting the failure in the first link, the second IO device and the third bus using the third lane set of the second PCIE bridge using the first link. 16. The computer program product of claim 15, further comprising configuring the second PCIE bridge to exchange a transmission bit set to stop using the third lane set.   The computer program product of claim 15, wherein each of the first and second PCIE bridges includes a PCIE host bridge (PHB).   The computer program product of claim 15, wherein each of the first and second IO devices includes a PCIE switch.   The computer program product of claim 15, wherein the first and second PCIE bridges are housed in a PCIE root complex.

Images