JP5842491B2 - Relay device and communication system - Google Patents

Description

  The present invention relates to a relay device and a communication system.

  PCI (Peripheral Component Interconnect) Express (hereinafter abbreviated as “PCIe”), which is a standard for local buses, is widely known as a standard for high-speed data transfer buses used in communications within PCs, servers, and embedded devices. ing. In the PCIe standard, a communication band can be arbitrarily set depending on the number of connected lanes. For example, when the root complex side is compatible with 16 lanes and the end point side is compatible with 8 lanes, these route complexes and endpoints are linked up with 8 lanes. When the route complex side is compatible with 16 lanes and the end point side is compatible with 4 lanes, these route complexes and endpoints are linked up with 4 lanes.

  PCIe has been developed for the purpose of relatively short-distance host-device communications, such as computer systems such as PCs. However, PCIe has also been applied to communications over distances. (PCI Express External Cabling Specification: hereinafter referred to as “PCIe cable standard”) has been established. When connecting PCs according to the PCIe cable standard, a PCIe cable interface board is generally mounted in each PCIe card slot provided in each PC. Then, the PCIe cable interface boards mounted on each PC are connected by a cable to perform communication between the PCs.

  The PCIe cable standard is based on the premise that cables corresponding to the number of lanes (1 lane, 2 lanes, 4 lanes, 8 lanes, 16 lanes, and 32 lanes) specified in the PCIe standard are used. For this reason, even when communicating in a communication band that can be handled with a smaller number of lanes than the specified number of lanes, cables corresponding to the specified number of lanes must be used and matched to the connection conditions such as the communication band. There is a problem that the number of cables used in a flexible manner cannot be reduced. For example, if the specified number of lanes is 8 lanes, even if connection is possible with a communication band of 5 lanes, connect with a cable for 8 lanes, which is the minimum number of lanes when exceeding 5 lanes. It is necessary to mount an extra cable for 3 lanes.

  As a technique for switching the number of lanes used for data transfer conforming to the PCIe standard, as described in Patent Document 1, switching between an 8-lane connection and a 4-lane connection can be performed by switching a bus switch arranged on the board. This is known.

  However, the technique described in Patent Document 1 is a technique related to switching between a predetermined number of lanes. Therefore, even if the technology described in Patent Document 1 is applied to a communication system that performs communication compliant with the PCIe cable standard, communication is still performed using cables corresponding to the specified number of lanes. The problem of having to mount an extra cable when communicating in a communication band that can be handled with a smaller number of lanes than the specified number of lanes cannot be solved.

  The present invention has been made in view of the above, and is capable of flexibly responding to a communication band necessary for data transfer, and problems such as an increase in cost and an increase in power consumption caused by mounting a cable more than necessary. It is an object of the present invention to provide a relay device and a communication system that can effectively avoid the above.

In order to solve the above-described problems and achieve the object, a relay apparatus according to the present invention includes an upstream port connected to a first device and a set of a plurality of non-transparent ports each having a communication band of one lane. A downstream port connected to the second device, and a packet received from the upstream port is routed to one of the plurality of non-transparent ports and transferred to the second device. Transfer means for routing a packet received from one of the non-transparent ports to the upstream port and transferring it to the first device, wherein each of the non-transparent ports has a unique address. The transfer means includes an address used by the first device and a plurality of the non-transparent ports. In accordance with an address conversion table that associates the address of the first port with the upstream port, by performing address conversion between the address used by the first device and the address of the non-transparent port used for packet transfer, The packet is routed to and from the non-transparent port .

The communication system according to the present invention is a communication system in which a first device and a second device are communicably connected via a first relay device and a second relay device, the first relay device. Is a set of a first upstream port connected to the first device and a plurality of non-transparent ports each having a communication band of one lane, and is connected to the second device via the second relay device. The first downstream port to be connected and the packet received from the first upstream port are routed to one of the plurality of non-transparent ports and transferred to the second device, and the plurality of non-transparent ports a packet received from one of the ports, and routed to the first upstream port and a first transfer means for transferring to the first device, the first downstream The plurality of non-transparent ports in the port each have a unique address, and the first transfer means uses an address used by the first device and addresses of the plurality of non-transparent ports in the first downstream port. Is converted between the address used by the first device and the address of the non-transparent port of the first downstream port used for packet transfer according to the first address conversion table in which Routing the packet between the first upstream port and the non-transparent port of the first downstream port, wherein the second relay device is a second upstream port connected to the second device; A set of a plurality of non-transparent ports each having a communication band of one lane; The second downstream port connected to the first device via the first relay device and the packet received from the second upstream port are routed to one of the plurality of non-transparent ports, and A second transfer means for transferring the packet received from one of a plurality of non-transparent ports to the second upstream port and transferring the packet to the second device while transferring the packet to the second device; Each of the plurality of non-transparent ports of the second downstream port has a unique address, and the second transfer means includes an address used by the second device and the plurality of the second downstream ports. According to the second address conversion table in which the address of the non-transparent port is associated, the address used by the second device The packet between the second upstream port and the non-transparent port of the second downstream port by performing address conversion between the address of the non-transparent port of the second downstream port used for packet transfer It is characterized by performing routing .

  According to the present invention, it is possible to flexibly respond to a communication band necessary for data transfer, and to effectively avoid problems such as an increase in cost and an increase in power consumption caused by mounting a cable more than necessary. There is an effect.

FIG. 1 is a conceptual diagram of a conventional communication system that performs information communication based on the PCIe cable standard between two hosts. FIG. 2 is a conceptual diagram of an address map in a conventional communication system. FIG. 3 is a diagram illustrating a specific configuration example of a cable connection unit of a conventional communication system. FIG. 4 is an example of a data flow when the first device executes data transfer that requires a communication band corresponding to 5 lanes of PCIe to the second device in the configuration of the cable connection unit of the conventional communication system. FIG. FIG. 5 is a graph showing the relationship between the communication bandwidth required for data transfer, the number of lanes required for the conventional communication system, and the number of redundant lanes. FIG. 6 is a diagram illustrating an example in which a lane failure has occurred in the seventh lane in the configuration of the cable connection unit of the conventional communication system. FIG. 7 is a diagram illustrating an example in which a lane failure has occurred in the fourth lane and the seventh lane in the configuration of the cable connection unit of the conventional communication system. FIG. 8 is a diagram illustrating an example in which a lane failure occurs in the first lane and the eighth lane in the configuration of the cable connection unit of the conventional communication system. FIG. 9 is a diagram illustrating a specific configuration example of the cable connection unit of the communication system according to the embodiment. FIG. 10 is a conceptual diagram of an address map in the cable connection unit of the communication system according to the embodiment. FIG. 11 shows a data flow when the first device executes data transfer that requires a communication band corresponding to 5 lanes of PCIe to the second device in the configuration of the cable connection unit of the communication system of the embodiment. It is a figure which shows an example. FIG. 12 is a diagram illustrating a state in which three repeaters and three metal cables are removed in the configuration of the cable connection unit of the communication system according to the embodiment. FIG. 13 is a graph showing the relationship between the communication bandwidth required for data transfer, the number of lanes required for the communication system of the embodiment, and the number of redundant lanes. FIG. 14 is a diagram illustrating an example in which a lane failure has occurred in the seventh lane in the configuration of the cable connection unit of the communication system according to the embodiment. FIG. 15 is a diagram illustrating an example in which a lane failure occurs in the first lane and the eighth lane in the configuration of the cable connection unit of the communication system according to the embodiment. FIG. 16 is a diagram illustrating a configuration example of a cable connection unit in a communication system in which a second device and a third device are communicably connected to a first device.

  Exemplary embodiments of a relay device and a communication system according to the present invention will be explained below in detail with reference to the accompanying drawings.

(Prior art issues)
First, the outline of the prior art and its problems will be described with reference to FIGS.

  FIG. 1 is a conceptual diagram of a conventional communication system that performs information communication based on the PCIe cable standard between two hosts. The communication system shown in FIG. 1 is a communication system in which a first device A and a second device B are communicably connected. The first device A is a host including a CPU 101a that is a control subject. The second device B is a host including a CPU 101b that is a control subject. Each of the first device A and the second device B is provided with a PCIe card slot. PCIe cable interface boards are mounted in the PCIe card slots of the first device A and the second device B. Then, the PCIe cable interface board of the first device A and the PCIe cable interface board of the second device B are connected by, for example, a multi-lane cable including a plurality of cables.

  By the way, PCIe presupposes a device tree structure having one root complex as a vertex, and does not assume that a plurality of root complexes exist in the device tree. For this reason, in the standard application of PCIe, it is not possible to perform communication between a plurality of hosts individually having a device that is a root complex.

  As a solution to such a problem, a non-transparent port (hereinafter referred to as “NT port”) from a vendor that provides a bridge or switch compliant with PCIe (hereinafter referred to as “PCIe bridge” or “PCIe switch”). There are PCIe bridges and PCIe switches. The NT port is a port that makes the other party of communication non-transparent. If two hosts are connected using the NT port of the PCIe switch, it becomes possible to perform initialization and the like without interfering with each other, and the control body (CPU) of each host is different. It becomes possible to access each other's resources while operating.

  Returning to FIG. 1, the first device A includes a PCIe bridge 100. The upstream port 110 of the PCIe bridge 100 is connected to the root complex 102a. The PCIe bridge 100 has the NT port 120 as described above as a downstream port. The NT port 120 is connected to the root complex 102b of the second device B. The PCIe bridge 100 relays communication between the root complex 102a of the first device A connected to the upstream port 110 and the root complex 102b of the second device B connected to the NT port 120 that is the downstream port. To do.

  In the conventional communication system configured as described above, since the root complex 102b of the second device B is connected to the NT port 120 of the PCIe bridge 100 provided in the first device A, the first device From the root complex 102a of A, the root complex 102b of the second device B is in a non-transparent state (shielded state). In addition, the route complex 102b of the first device A is also non-transparent from the route complex 102b of the second device B. Therefore, even if two route complexes 102a and 102b exist in the communication system, information communication conforming to the PCIe cable standard is possible, and the host between the first device A and the second device B is possible. Communication can be realized.

  FIG. 2 is a conceptual diagram of an address map in the conventional communication system shown in FIG. The first device A uses the address in the address conversion area set in the usable area of the address space of the first device A. On the other hand, the second device B uses the address in the address conversion area set in the available area of the address space of the second device B. When the NT port 120 of the PCIe bridge 100 performs communication between the first device A and the second device B, the address used by the first device A according to a pre-stored address conversion table. And address used by the second device B are performed. For this reason, it is necessary to store not only the address conversion area of the first device A but also the address conversion area of the second device B in the address conversion table of the NT port 120 of the PCIe bridge 100 in advance.

  FIG. 3 is a diagram illustrating a specific configuration example of a portion related to data transfer via a cable (hereinafter referred to as a “cable connection unit”) of the conventional communication system illustrated in FIG. 1. In the example shown in FIG. 3, the PCIe cable interface board 200a attached to the first device A and the PCIe cable interface board 200b attached to the second device B correspond to eight lanes of PCIe (hereinafter, referred to as “the PCIe cable interface board 200a”). It is written as “x8”.) A cable connecting portion connected by a metal cable 210 is shown.

  A PCIe bridge 100 is mounted on the PCIe cable interface board 200a attached to the first device A. The PCIe bridge 100 includes an x8 upstream port 110. The x8 upstream port 110 is connected to the x8 port 220a of the root complex 102a via an x8 card edge connector (not shown). The PCIe bridge 100 includes an x8NT port 120. Each x8NT port 120 is connected to eight metal cables 210 corresponding to one lane of PCIe (hereinafter referred to as “x1”). A repeater 230 a is provided between each lane of the NT port 120 of the PCIe bridge 100 and the metal cable 210. The repeater 230a performs electric signal amplification and shaping processing. In addition, since one lane of PCIe transmits information by differential signals using two signal lines for reception and transmission, four signal lines are required per lane. In this embodiment, one cable is used. A description will be given assuming that these four signal lines are included.

  The PCIe cable interface board 200b attached to the second device B is connected to the x8 port 220b of the root complex 102b and the eight metal cables 210. A repeater 230b is provided between each lane of the x8 port 220b of the route complex 102b and the metal cable 210. Similar to the repeater 230a, the repeater 230b performs electric signal amplification and shaping processing.

  In the example illustrated in FIG. 3, the first device A and the second device B are connected to the metal cable 210, but an optical cable may be used instead of the metal cable 210. When an optical cable is used instead of the metal cable 210, the repeaters 230a and 230b are replaced with optical transceivers. The optical transceiver performs conversion between an electrical signal and an optical signal.

  FIG. 4 shows the configuration of the cable connection unit of the conventional communication system shown in FIG. 3, in which the first device A performs data transfer that requires a communication band equivalent to 5 lanes of PCIe to the second device B. It is a figure which shows an example of the data flow at the time of doing. In the example illustrated in FIG. 4, the first device A performs data transfer using five DMACs (Direct Memory Access Controllers) 240. In FIG. 4, only the first device A side of the cable connection portion is illustrated, and an arrow indicated by a broken line in the drawing indicates a data flow in the first device A.

  In the first device A, five DMACs 240 are mounted. Each DMAC 240 has a data transfer performance of x1. Each DMAC 240 divides the read data into packets while reading data from the memory 250 included in the first device A, and transfers the packets to the x8 port 220a of the route complex 102a via the internal bus. The x8 port 220a of the route complex 102a divides the packet received from the DMAC 240 into 8 bits each and distributes them evenly over all 8 lanes, and the x8 upstream port 110 of the PCIe bridge 100 mounted on the PCIe cable interface board 200a. Forward to. The PCIe bridge 100 converts the data divided into 8 bits into each lane of the x8 upstream port 110 and converts it again into packet data, and transfers it to the x8NT port 120. The x8NT port 120 again divides each received packet into 8 bits, distributes it evenly over all 8 lanes, and transfers it to the second device B side via 8 metal cables 210.

  On the second device B side (not shown), the data sent after being divided by 8 bits into each lane of x8 at the x8 port 220b of the route complex 102b is converted back to packet data again, and the second Transfer to the memory of device B.

  As described above, in the conventional communication system, even when data transfer that requires a communication band corresponding to 5 lanes of PCIe is executed, if the configuration is x8, data transfer is executed using all 8 lanes. I have to. That is, in the conventional communication system, even when communication is performed in a communication band that can be handled with a smaller number of lanes than the number of lanes defined in the PCIe standard, cables corresponding to the specified number of lanes must be used. There is a problem that it is not possible to reduce the number of cables used in a flexible manner in accordance with connection conditions such as a communication band.

  FIG. 5 is a graph showing the relationship between the communication bandwidth required for data transfer, the number of lanes required for the conventional communication system, and the number of redundant lanes. The horizontal axis in the figure indicates a value obtained by dividing the size of the communication band required for data transfer by the band corresponding to one lane (that is, the used band that is actually the number of lanes required for data transfer). The middle vertical axis indicates the number of lanes required in the conventional communication system and the number of redundant lanes. In addition, the solid line graph in the figure shows the number of lanes required in the conventional communication system with respect to the size of the communication bandwidth required for data transfer, and the broken line graph in the figure shows the number of extra lanes. Show.

  In the conventional communication system, since it is assumed that communication is performed with the number of lanes defined by the PCIe standard, the number of lanes actually required for data transfer is as shown in the area surrounded by the one-dot chain line in FIG. There is a region where the number of extra lanes is generated. For this reason, there has been a problem that the mounting cost (the number of cables used, the number of repeaters, the number of optical transceivers, etc.) is wasted. For example, as in the example shown in FIG. 4, even if the communication bandwidth required for data transfer is a bandwidth equivalent to 5 lanes of PCIe, the metal cable 210 (or optical cable) and repeaters 230a and 230b (eight lanes) Or an optical transceiver) must be mounted, and the mounting cost for three lanes is wasted. In particular, the repeaters 230a and 230b (or optical transceivers) are very expensive compared to the cost of the PCIe bridge 100 and the like, and the metal cable 210 (or optical cable) is also expensive compared to the wiring on the PCIe cable interface board 200a. is there. In addition, performing communication with the number of lanes exceeding the communication band necessary for data transfer has a problem that power is consumed more than necessary and power consumption is increased.

  Further, the conventional communication system has a problem that when a failure occurs in some lanes, the influence on the entire communication system is large. Hereinafter, this problem will be described.

  FIG. 6 is a diagram illustrating an example in which a lane failure has occurred in the seventh lane (the seventh lane from the top in the figure) in the configuration of the cable connection unit of the conventional communication system illustrated in FIG. 3. The lane failure is, for example, a failure of the repeater 230a (or optical transceiver), disconnection of the metal cable 210 (or optical cable), or disconnection. In FIG. 6, only the first device A side of the cable connection portion is shown, and the area enclosed by the one-dot chain line in the figure indicates the usable lane, and the area enclosed by the two-dot chain line frame Indicates an unusable lane.

  As described above, in the conventional communication system, it is assumed that communication is performed with the number of lanes defined by the PCIe standard. Therefore, as shown in FIG. 6, a lane failure occurs in one of the eight lanes. If link up is not possible in 8 lanes, link up is possible only in 4 lanes, which is half of 8 lanes, and only 4 lanes can be used. That is, in the conventional communication system, the available communication band is extremely reduced only by the occurrence of a lane failure of one lane. For example, as in the example shown in FIG. 4, if the communication bandwidth required for data transfer is a bandwidth equivalent to 5 lanes of PCIe, there is a communication bandwidth of 3 lanes when no lane failure has occurred. Nevertheless, a communication band necessary for data transfer cannot be secured if a lane failure occurs only in one lane.

  In the example of FIG. 6, an example in which a lane failure has occurred in the seventh lane (the seventh lane from the top in the figure) is shown, but one of the areas surrounded by a dashed-dotted frame in the figure If a lane failure occurs in a lane, link up with 4 lanes in the area enclosed by the chain double-dashed line in the figure by the PCIe standard lane reversal function (function to invert the lane arrangement on the receiving side) become. As a result, the region enclosed by the one-dot chain line in the figure becomes an unusable lane, and the region enclosed by the two-dot chain line becomes a usable lane.

  FIG. 7 shows the configuration of the cable connection part of the conventional communication system shown in FIG. 3, the fourth lane (fourth lane from the top in the figure) and the seventh lane (the seventh lane from the top in the figure). It is a figure which shows the example which lane failure generate | occur | produced. FIG. 7 shows only the first device A side of the cable connection portion, and the area enclosed by the one-dot chain line in the figure indicates the usable lane, and the area enclosed by the two-dot chain line frame. Indicates an unusable lane.

  In the PCIe standard, link training is performed in order from the first lane, and the link is increased with the number of lanes of 2 n (n is an integer of 0 to 5) at which a link can be established. If the PCIe standard lane reversal function is used, link training can be performed in the reverse order from the last lane. Therefore, for example, in the configuration of the cable connection unit of the conventional communication system shown in FIG. 3, even if a lane failure occurs in two of the eight lanes, the two lanes where the failure has occurred are both in the first lane. To the 4th lane (area surrounded by the dashed-dotted line frame in FIG. 6) and the 5th lane to the 8th lane (area surrounded by the dashed-dotted line frame in FIG. 6). Link up in 4 lanes is possible.

  However, as shown in the example of FIG. 7, when a lane failure occurs in the 4th and 7th lanes, the link can be linked up only in 2 lanes using the 1st and 2nd lanes, which can be used. There are only 2 lanes. In other words, in the example of FIG. 7, only a quarter communication band of 8 lanes can be secured, and the performance is greatly reduced.

  FIG. 8 shows the first lane (the first lane from the top in the figure) and the eighth lane (the eighth lane from the top in the figure) in the configuration of the cable connection portion of the conventional communication system shown in FIG. It is a figure which shows the example which lane failure generate | occur | produced. In FIG. 8, only the first device A side of the cable connection portion is illustrated. In the example of FIG. 8, the lane failure has occurred in two of the eight lanes as in the example of FIG. 7, but the lanes in which the lane failure has occurred are lanes at both ends. (A region surrounded by a two-dot chain line in the figure) shows the worst case in which it cannot be used.

  In the PCIe standard, as described above, link training is performed in order from the first lane or from the last lane when the lane reversal function is used, and the link up is performed with the number of 2 n lanes at which a link can be established. Therefore, as shown in the example of FIG. 8, when a lane failure occurs in both the first lane and the last lane, the 8th lane, all 8 lanes become unusable even if the remaining 6 lanes are normal. Can not do.

  As described above, in the conventional communication system, even if the cable connection unit has a configuration corresponding to 8 lanes, when a lane failure occurs in some lanes, 4 lanes, 1 / Link-up is possible only in 2 lanes of 4 and 1 lane of 1/8, and the usable communication bandwidth is greatly reduced. Also, in the worst case where a lane failure occurs in both lanes, there is a problem that link-up itself becomes impossible and communication becomes impossible.

(Communication system to which the present invention is applied)
Next, a communication system according to an embodiment of the present invention will be described with reference to FIGS. The communication system illustrated below has a configuration in which the first device A and the second device B each functioning as a host are communicably connected in the same manner as the communication system described as the prior art. Specifically, a PCIe cable interface board is installed in each of the PCIe card slots of the first device A and the second device B, and the PCIe cable interface boards are connected to each other with a multi-lane cable composed of a plurality of cables. ing.

  FIG. 9 is a diagram illustrating a specific configuration example of the cable connection unit of the communication system according to the present embodiment. In the communication system according to the present embodiment, the PCIe cable interface board 20a attached to the first device A and the PCIe cable interface board 20b attached to the second device B are connected by the x8 metal cable 21. It is a configuration. In the present embodiment, in order to clarify the difference from the prior art, a communication system having an x8 configuration similar to that of the prior art is illustrated and described. However, the communication system according to the present embodiment has an x8 configuration. Not limited to this, it is possible to adopt a configuration corresponding to various numbers of lanes.

  The PCIe cable interface board 20a attached to the first device A and the PCIe cable interface board 20b attached to the second device B have the same configuration. In the present specification, the suffix “a” is added to the reference symbol of the component of the first device A, and the suffix “b” is added to the reference symbol of the component of the second device B to distinguish them.

  PCIe switches 10a and 10b are mounted on the PCIe cable interface boards 20a and 20b, respectively. The PCIe switches 10a and 10b include x8 upstream ports 11a and 11b, respectively. The x8 upstream port 11a of the PCIe switch 10a is connected to the x8 port 22a of the root complex of the first device A via an x8 card edge connector (not shown). Further, the x8 upstream port 11b of the PCIe switch 10b is connected to the x8 port 22b of the root complex of the second device B via an x8 card edge connector (not shown).

  Each of the PCIe switches 10a and 10b includes eight x1NT ports 12a and 12b as downstream ports. The eight x1NT ports 12a of the PCIe switch 10a and the eight x1NT ports 12b of the PCIe switch 10b are connected via x1 metal cables 21, respectively. That is, the communication system according to the present embodiment is the same as the communication system of the prior art in that the first device A and the second device B are connected by the eight x1 metal cables 21. However, in the communication system according to the present embodiment, the first device A and the second device B include PCIe switches 10a and 10b each having a downstream port that is a set of eight x1NT ports 12a and 12b. This is different from the conventional communication system in that the x1NT ports 12a and 12b of the switches 10a and 10b are connected via the x1 metal cable 21.

  A repeater 23 a is provided between the x1NT port 12 a of the PCIe switch 10 a mounted on the PCIe cable interface board 20 a and the metal cable 21. Also, repeaters 23b are provided between the x1NT port 12b of the PCIe switch 10b mounted on the PCIe cable interface board 20b and the metal cable 21, respectively. These repeaters 23a and 23b perform electric signal amplification and shaping processing.

  In the example shown in FIG. 9, the x1NT port 12a of the PCIe switch 10a mounted on the PCIe cable interface board 20a of the first device A and the PCIe switch mounted on the PCIe cable interface board 20b of the second device B. The x1NT port 12b of 10b is connected by the metal cable 21, but may be connected by using an optical cable instead of the metal cable 21. When an optical cable is used instead of the metal cable 21, the repeaters 23a and 23b are replaced with optical transceivers. The optical transceiver performs conversion between an electrical signal and an optical signal.

  The PCIe switches 10a and 10b include transfer units 13a and 13b, respectively.

  The forwarding means 13a of the PCIe switch 10a routes the packet received from the x8 upstream port 11a to one of the eight x1NT ports 12a and forwards it to the second device B, and among the eight x1NT ports 12a The packet received from one of these is routed to the x8 upstream port 11a and transferred to the first device A. Specifically, each of the eight x1NT ports 12a of the PCIe switch 10a has a unique address. The transfer means 13a of the PCIe switch 10a has an address used by the first device A and an x1NT port 12a used for packet transfer when data is transferred between the first device A and the second device B. By performing address conversion between the x1 upstream port 11a and the x1NT port 12a, routing of packets between the x1NT port 12a and the x1NT port 12a is performed.

  Note that the transfer unit 13a of the PCIe switch 10a has a functional configuration realized inside the PCIe switch 10a. Therefore, the transfer means 13a is illustrated as a component different from the x8 upstream port 11a and the x1NT port 12a in FIG. 9, but as a specific form, the transfer means 13a is an x8 upstream having a physical layer and a logical layer. It may be realized as a function of the logical layer of the port 11a, or may be realized as a function of the logical layer of the x1NT port 12a having a physical layer and a logical layer.

  The transfer means 13b of the PCIe switch 10b routes the packet received from the x8 upstream port 11b to one of the eight x1NT ports 12b and transfers the packet to the first device A, and among the eight x1NT ports 12b The packet received from one of these is routed to the x8 upstream port 11b and transferred to the second device B. Specifically, each of the eight x1NT ports 12b of the PCIe switch 10b has a unique address. The transfer means 13b of the PCIe switch 10b, when performing data transfer between the second device B and the first device A, and the address used by the second device B and the x1NT port 12b used for packet transfer By performing address conversion between the x1 upstream port 11b and the x1NT port 12b, the packet is routed, and communication using the x1NT port 12b is relayed.

  Note that the transfer means 13b of the PCIe switch 10b has a functional configuration realized inside the PCIe switch 10b. Therefore, the transfer means 13b is illustrated as a separate component from the x8 upstream port 11b and the x1NT port 12b in FIG. 9, but as a specific form, the transfer means 13b is an x8 upstream having a physical layer and a logical layer. It may be realized as a function of the logical layer of the port 11b, or may be realized as a function of the logical layer of the x1NT port 12b having a physical layer and a logical layer.

  In the communication system according to the present embodiment configured as described above, the x1NT port 12a of the PCIe switch 10a mounted on the PCIe cable interface board 20a of the first device A and the PCIe cable interface board of the second device B. Since the x1NT port 12b of the PCIe switch 10b mounted on 20b is connected, the second device B is in a non-transparent state from the first device A, and the first device also from the second device B A becomes non-transparent. Also, the first device A does not recognize the x1NT port 12b on the second device B side as a device, and the second device B also recognizes the x1NT port 12a on the first device A side as a device. Not. For this reason, even if the metal cable 21 connecting the x1NT port 12a on the first device A side and the x1NT port 12b on the second device B side is disconnected, the port cannot be used. Neither the second device B hangs up. In addition, when starting up the communication system, the first device A is started up first, and the second device B is started up first, so that the link is made normally, and there is a restriction in the order of starting up the devices. Nor.

  FIG. 10 is a conceptual diagram of an address map in the cable connection unit of the communication system according to the present embodiment shown in FIG. In the case of this embodiment, since the NT ports of the x1NT port 12a and the x1NT port 12b are connected to each other, as shown in FIG. 10, the NT port address space (hereinafter referred to as “NT space”) is the first. The address space of the device A and the address space of the second device B can be provided as a single address space. The address conversion area in the NT space can be determined in advance as a fixed area.

  In the address space of the first device A and the address space of the second device B, address conversion areas (addresses) corresponding to the number of metal cables 21 used in the communication system (that is, the number of connected lanes), respectively. There are conversion areas 1 to 8). In the NT space, there are address conversion areas (NTP1 address area to NTP8 address area) for each NT port 12a corresponding to the address conversion areas 1 to 8 of the first device A and the second device B. is there. When communication is performed between the first device A and the second device B, the first device A uses the address in the address conversion area in the address space of the first device A, and the second device B Uses the address in the address translation area in the address space of the second device B. Then, the transfer means 13a of the PCIe switch 10a on the first device A side uses the address used by the first device A and the address designated for the x1NT port 12a in the corresponding NT space (NTP1 address area to NTP8). In accordance with an address conversion table that associates any address in the address area), address conversion between the address used by the first device A and the address used by the x1NT port 12a is performed. Similarly, the transfer means 13b of the PCIe switch 10b on the second device B side uses the address used by the second device B and the address (NTP1 address) specified for each x1NT port 12b in the corresponding NT space. The address conversion between the addresses used by the second device B and the address used by the x1NT port 12b is performed in accordance with an address conversion table in which the addresses of any one of the area to the NTP8 address area) are associated with each other.

  Here, the address area (address area of NTP1 to NTP8) for each x1NT port 12a, 12b in the NT space is determined as a fixed area in advance, and the PCIe switch 10a on the first device A side and the second device B side PCIe switch 10b shares the address in this address translation area. Therefore, the transfer unit 13a of the PCIe switch 10a on the first device A side only needs to grasp the address conversion area of the first device A, and the transfer unit 13b of the PCIe switch 10b on the second device B side. Need only grasp the address translation area of the second device B. That is, the transfer unit 13a of the PCIe switch 10a on the first device A side can appropriately perform address conversion without grasping the address conversion area of the second device B, and the PCIe on the second device B side. The transfer means 13b of the switch 10b can appropriately perform address conversion without grasping the address conversion area of the first device A. That is, in the communication system according to the present embodiment, it is possible to eliminate the dependency between the address spaces of the mutual devices for address conversion when communication is performed between the first device A and the second device B. Address translation can be performed very easily.

  As described above, the communication system according to the present embodiment has a configuration in which the first device A and the second device B are connected to each other through the NT ports, so that the communication system hangs even if the metal cable 21 is disconnected. There is no problem that restrictions are imposed on the startup order of devices, and there is an advantage that address conversion at the time of communication is extremely easy.

  Further, in the communication system according to the present embodiment, the first device A and the second device B are not only connected to each other by the NT ports but also the downstream port (the first port) of the PCIe switch 10a on the first device A side. 2) and a downstream port (on the side connected to the first device A) of the PCIe switch 10b on the second device B side. Port) is configured by a set of eight x1NT ports 12b, and the x1NT ports 12a and 12b are connected by a metal cable 21 independently for each lane. Then, data transfer between the first device A and the second device B is performed selectively using the x1NT ports 12a and 12b by packet routing by the transfer means 13a and 13b of the PCIe switches 10a and 10b. Like to do. As a result, the communication system according to the present embodiment can flexibly cope with the communication band necessary for data transfer, and mounts the metal cable 12 and its accompanying parts (repeaters 23a, 23b, etc.) more than necessary. Problems such as an increase in cost and an increase in power consumption can be effectively avoided. Hereinafter, this point will be described in more detail.

  FIG. 11 shows data in which the first device A requires a communication band equivalent to five lanes of PCIe with respect to the second device B in the configuration of the cable connection unit of the communication system according to the present embodiment shown in FIG. It is a figure which shows an example of the data flow at the time of performing transfer. In FIG. 11, in order to clarify the difference from the prior art, the case where the first device A executes data transfer using five DMACs 24 is illustrated as in the specific example of the prior art shown in FIG. 4. doing. In FIG. 11, only the first device A side of the cable connection portion is illustrated, and an arrow indicated by a broken line in the drawing indicates a data flow in the first device A.

  The first device A is equipped with five DMACs 24. Each DMAC 24 has a data transfer performance of x1. Each DMAC 24 reads data from the memory 25 provided in the first device A, divides the read data into packets, and transfers the packets to the x8 port 22a of the root complex via the internal bus. The x8 port 22a of the root complex divides the packet received from the DMAC 24 into 8 bits each and distributes it evenly over all 8 lanes to the x8 upstream port 11a of the PCIe switch 10a mounted on the PCIe cable interface board 20a. Forward. The transfer means 13a of the PCIe bridge 10a again converts the data divided into 8 bits into each lane of the x8 upstream port 11a and converts it into packet data again, and checks the transfer destination address included in the packet, Address conversion is performed according to the address conversion table as shown in FIG. 10, and the data is routed and transferred to one of the eight x1NT ports 12a. The x1NT port 12a sends the received packet to the second device B side via the metal cable 21 using an address (any address in the NTP1 address area to NTP8 address area) designated for transfer of the packet. Send to.

  On the other hand, on the second device B side (not shown), the x1NT port 12b of the PCIe switch 10b receives a packet transmitted from the first device A side via the metal cable 21. When the packet is received by the x1NT port 12b, the transfer means 13b of the PCIe switch 10b checks the transfer destination address included in the packet and performs address conversion according to the address conversion table as shown in FIG. And route and forward to x8 upstream port 11b. The x8 upstream port 11b divides the received packet into 8 bits, distributes it evenly over all 8 lanes, and forwards it to the x8 port 22b of the root complex. The x8 port 22b of the root complex converts the data divided and sent from each lane of the x8 upstream port 11b into packet data again, and transfers the packet data to the memory provided in the second device B.

  As described above, in the communication system according to the present embodiment, when data transfer that requires a communication band corresponding to 5 lanes of PCIe is executed, all 8 lanes are not used even in the x8 configuration. Use only to perform data transfer. Therefore, for example, as shown in FIG. 12, communication can be appropriately performed even when three repeaters 23a and three metal cables 21 are removed, and communication is efficiently performed according to the actually required communication band. It can be performed. That is, in the communication system according to the present embodiment, when communication is performed in a communication band that can be handled with a smaller number of lanes than the number of lanes defined in the PCIe standard, data transfer may be performed with the number of lanes corresponding to the necessary communication band. The number of repeaters 23a and the number of metal cables 21 to be used can be reduced in a flexible manner in accordance with the communication band required for data transfer.

  FIG. 13 is a graph showing the relationship between the communication bandwidth necessary for data transfer, the number of lanes required for the communication system according to this embodiment, and the number of redundant lanes. The horizontal axis in the figure indicates a value obtained by dividing the size of the communication band required for data transfer by the band corresponding to one lane (that is, the used band that is actually the number of lanes required for data transfer). The vertical axis in the figure indicates the number of lanes required for the communication system according to the present embodiment and the number of redundant lanes. In addition, the solid line graph in the figure indicates the number of lanes required in the communication system of the present embodiment with respect to the size of the communication band required for data transfer, and the broken line graph in the figure indicates the number of lanes that are redundant. Respectively.

  As shown in FIG. 13, in the communication system according to the present embodiment, the number of lanes required matches the use band that is the number of lanes actually required for data transfer. Therefore, as in the region surrounded by the one-dot chain line in the figure, the number of remaining lanes is always 0 regardless of the size of the use band. As described above, in the communication system according to the present embodiment, there is no restriction that the number of lanes defined in the PCIe standard has to be used, and data can be efficiently transferred with the number of lanes according to the required performance. Therefore, problems such as an increase in cost and an increase in power consumption caused by mounting the repeater 23a and the metal cable 21 more than necessary can be effectively avoided.

  Furthermore, in the communication system according to the present embodiment, the influence on the entire communication system is small when a failure occurs in some lanes as compared with the conventional communication system. Hereinafter, assuming the case where a lane failure similar to that of the conventional communication system has occurred (example described with reference to FIGS. 6 and 8), the influence of the lane failure of the communication system according to the present embodiment will be described in detail.

  FIG. 14 shows the configuration of the cable connection unit of the communication system according to the present embodiment shown in FIG. 9, in the same way as in the example shown in FIG. 6, the lane is in the seventh lane (the seventh lane from the top in the figure). It is a figure which shows the example which a failure generate | occur | produced. In FIG. 14, only the first device A side of the cable connection portion is shown, and the area enclosed by the one-dot chain line in the figure indicates the usable lane, and the area enclosed by the two-dot chain line frame Indicates an unusable lane.

  In such a case, in the conventional communication system, only a lane failure of one lane occurs, and the link up can be performed only in four lanes, which is half of the eight lanes, and the available communication band is extremely reduced. There was a problem. On the other hand, in the communication system according to the present embodiment, when a lane failure occurs in one of the eight lanes (the seventh lane in the example of FIG. 14), x1NT corresponding to the lane in which the lane failure has occurred. By changing the address of the ports 12a and 12b to the address of any of the other x1NT ports 12a and 12b, the lane where the lane failure has occurred can be replaced with another lane. All the remaining 7 lanes can be used like the enclosed area. In other words, in the communication system according to the present embodiment, even if a lane failure occurs in one of the eight lanes, the first device A and the second device B Can be continued.

  In the communication system according to the present embodiment, even when a lane failure occurs in two of the eight lanes (the case shown in FIG. 7), x1NT ports corresponding to the two lanes in which the lane failure has occurred. By changing the address of 12a, 12b to the address of any of the other x1NT ports 12a, 12b, two lanes in which a lane failure has occurred can be replaced with another lane, and the communication bandwidth of 5/8 In the state in which is secured, the communication between the first device A and the second device B can be continued. Similarly, in the communication system according to the present embodiment, when a lane failure occurs in 3 lanes of 8 lanes, the communication bandwidth for 5 lanes is obtained, and when a lane failure occurs in 4 lanes of the 8 lanes, 4 is assumed. The communication between the first device A and the second device B is continued in a state where a communication band corresponding to the number of lanes in the lane in which no lane failure has occurred, such as the communication bandwidth for the lane. Can do.

  15 shows the configuration of the cable connection unit of the communication system according to the present embodiment shown in FIG. 9, as in the example shown in FIG. 8, the first lane (first lane from the top in the figure) and 8 It is a figure which shows the example which the lane failure generate | occur | produced in the lane (8th lane from the top in the figure). In FIG. 15, only the first device A side of the cable connection portion is shown, and the area enclosed by the one-dot chain line in the figure indicates the usable lane, and the area enclosed by the two-dot chain line frame Indicates an unusable lane.

  In the conventional communication system, link training is performed in order from the first lane or from the last lane when the lane reversal function is used, and the link up is performed with the number of 2 n lanes at which a link can be established. In such a case, all the 8 lanes are unusable and there is a problem that communication cannot be performed. On the other hand, in the communication system according to the present embodiment, the first device A and the second device B are connected by independent x1NT ports 12a and 12b for each lane. Will be linked training and link up. Therefore, even if a link cannot be raised in a lane in which a lane failure has occurred, the effect does not reach other lanes, and a lane in which no lane failure has occurred can be normally linked up and used.

  As described above in detail with specific examples, the communication system according to the present embodiment includes the downstream port of the PCIe switch 10a on the first device A side as a set of a plurality of x1NT ports 12a. The downstream port of the PCIe switch 10b on the second device B side is configured by a set of a plurality of x1NT ports 12b, and the x1NT ports 12a and 12b are independently connected to each other by a metal cable 21 (or an optical cable). It is said. And, by selectively using any of the x1NT ports 12a and 12b by packet routing by the transfer means 13a and 13b of the PCIe switches 10a and 10b, between the first device A and the second device B Data transfer is performed. Therefore, the communication system according to the present embodiment can flexibly adapt to the communication band required for data transfer, and mounts the metal cable 12 (or optical cable) and repeaters 23a and 23b (or optical transceiver) more than necessary. It is possible to effectively avoid problems such as an increase in cost and an increase in power consumption caused by doing so.

  Furthermore, the communication system according to the present embodiment can continue communication in a state where the communication band is reduced using the remaining normal lanes even if a lane failure occurs in some lanes. Fault tolerance can be achieved.

  In particular, in the communication system according to the present embodiment, the transfer units 13a and 13b of the PCIe switches 10a and 10b use addresses used by the first device A or the second device B and a plurality of x1NT ports 12a and 12b. In accordance with an address conversion table in which addresses to be associated are associated, address conversion between them is performed, and packet routing is performed between the upstream ports 11a and 11b and the x1NT ports 12a and 12b. Therefore, when a lane failure occurs in some lanes, the address of the x1NT port 12a, 12b corresponding to the lane where the lane failure has occurred is changed to the address of any other x1NT port 12a, 12b. By this method, a lane in which a lane failure has occurred can be replaced with another lane, and communication using the remaining normal lane can be continued.

  In addition, since the communication system according to the present embodiment is configured to connect the first device A and the second device B with NT ports, the communication system hangs even if the metal cable 21 (or optical cable) is disconnected. There is no problem that restrictions are imposed on the startup order of devices, and there is an advantage that address conversion at the time of communication is extremely easy.

  The specific embodiments of the present invention have been described above. However, the present invention is not limited to the above-described embodiments as they are, and may be embodied by modifying constituent elements without departing from the scope of the invention in the implementation stage. can do. In other words, the configuration and operation of the communication system according to the above-described embodiment are merely examples, and various modifications can be made according to applications and purposes.

  For example, the above-described embodiment is an example in which the present invention is applied to a communication system that performs information communication based on the PCIe cable standard between two hosts (first device A and second device B). The present invention can also be effectively applied to a communication system that performs information communication conforming to the PCIe cable standard between three or more hosts.

  FIG. 16 is a diagram illustrating a configuration of a cable connection unit when the present invention is applied to a communication system in which the second device B and the third device C are communicably connected to the first device A. is there. In the example shown in FIG. 16, the configurations of the first device A and the second device B are the same as those in the above-described embodiment, and are connected in five lanes as in the example shown in FIG. Similarly to the first device A and the second device B, the third device C includes a PCIe cable interface board 20c on which a PCIe switch 10c is mounted. The PCIe switch 10c has an x4 upstream port 11c, three x1NT ports 12c, and a transfer means 13c. The x4 upstream port 11c is connected to the x4 port 22c of the root complex of the third device C, and three x1NT ports 12c is connected to the x1NT port 12a of the PCIe switch 10a of the first device A via the metal cable 21 (or optical cable). The transfer unit 13c routes the packet received from the x4 upstream port 11c to one of the three x1NT ports 12c and transfers the packet to the first device A, and 1 of the three x1NT ports 12c. The packet received from the first device is routed to the x4 upstream port 11c and transferred to the third device C. In FIG. 16, the repeaters (or optical transceivers) on the PCIe cable interface boards 20a, 20b, and 20c are not shown.

  When the present invention is applied to a communication system that performs information communication conforming to the PCIe cable standard between three or more hosts (first device A, second device B, and third device C in the example of FIG. 16), As shown in the example of FIG. 16, three hosts can be connected with an arbitrary number of lanes (5 lanes and 3 lanes in the example of FIG. 16) other than the number of lanes defined by the PCIe standard. A form of communication can be realized. In this case as well, the configuration of the cable connection section can be configured so as not to waste the communication band necessary for data transfer between the respective hosts. It is possible to effectively avoid problems such as increased cost and increased power consumption, and to realize a communication system with high fault tolerance.

  The above-described embodiment is an example applied to a communication system that performs information communication in conformity with the PCIe cable standard. However, a function equivalent to the NT port of PCIe, that is, a port that makes the communication partner type non-transparent is used. The present invention can be effectively applied to any communication system as long as the communication system performs information communication.

10a, 10b PCIe switch 11a, 11b x8 upstream port 12a, 12b x1NT port 13a, 13b Transfer means 20a, 20b PCIe cable interface board 21 Metal cable 23a, 23b Repeater A First device B Second device

JP 2008-171291 A

Claims (5)

An upstream port connected to the first device;
A set of non-transparent ports each having a communication band of one lane, and a downstream port connected to the second device;
A packet received from the upstream port is routed to one of the non-transparent ports and transferred to the second device, and a packet received from one of the non-transparent ports Transfer means for routing to the upstream port and transferring it to the first device ,
The plurality of non-transparent ports each have a unique address;
The transfer means uses the address used by the first device and the packet used for packet transfer according to an address conversion table in which addresses used by the first device are associated with addresses of the plurality of non-transparent ports. A relay apparatus that performs routing of the packet between the upstream port and the non-transparent port by performing address conversion between the address of the non-transparent port and the non-transparent port . The non-transparent port is connected to the second device via a metal cable;
The repeater according to claim 1, wherein a repeater that performs amplification and shaping processing of an electric signal is provided between the non-transparent port and the metal cable. The non-transparent port is connected to the second device via an optical cable;
The relay apparatus according to claim 1, wherein an optical transceiver that converts an electric signal and an optical signal is provided between the non-transparent port and the optical cable. A communication system in which a first device and a second device are communicably connected via a first relay device and a second relay device,
The first relay device
A first upstream port connected to the first device;
A first downstream port that is a set of a plurality of non-transparent ports each having a communication band of one lane, and is connected to the second device via the second relay device;
A packet received from the first upstream port is routed to one of the plurality of non-transparent ports and transferred to the second device, and received from one of the plurality of non-transparent ports. First transfer means for routing the packet to the first upstream port and transferring the packet to the first device,
The plurality of non-transparent ports of the first downstream port each have a unique address;
The first transfer means follows the first address conversion table in which the address used by the first device and the addresses of the plurality of non-transparent ports of the first downstream port are associated with each other. By performing address conversion between the address to be used and the address of the non-transparent port of the first downstream port used for packet transfer, the first upstream port and the non-transparent port of the first downstream port Routing the packets between them,
The second relay device is
A second upstream port connected to the second device;
A plurality of non-transparent ports each having a communication band of one lane, a second downstream port connected to the first device via the first relay device;
A packet received from the second upstream port is routed to one of the non-transparent ports and transferred to the second device, and received from one of the non-transparent ports A second transfer means for routing the packet to the second upstream port and transferring the packet to the second device ;
The plurality of non-transparent ports of the second downstream port each have a unique address;
The second transfer means follows the second address conversion table in which the address used by the second device and the addresses of the plurality of non-transparent ports of the second downstream port are associated with each other. By performing address conversion between the address used and the address of the non-transparent port of the second downstream port used for packet transfer, the second upstream port and the non-transparent port of the second downstream port A communication system characterized in that the packet is routed between them. The communication system according to claim 4 , comprising a plurality of the second devices.

Images