JP5084197B2 - Processor node system and processor node cluster system - Google Patents

Description

  The present invention relates to a processor node system and a processor node cluster system in which a plurality of processors are interconnected.

  Various peripheral devices are connected to a personal computer or server via a PCI (Peripheral Component Interconnect) bus to constitute an information processing system. Since the input / output bus of the processor and the PCI bus that is the input / output bus of the peripheral device have different standards, the processor and the peripheral device are usually connected via a bridge.

  In order to expand the functions and enhance the performance of information processing systems, graphic processors and high-speed memory devices are sometimes connected as PCI devices, and more peripheral devices are required to be connected via the PCI bus. Yes. Therefore, a plurality of devices are connected to one processor using a PCI Express (trademark or registered trademark) switch. In addition, in order to interconnect a plurality of processor nodes or to interconnect a processor node and a device, an ultra-high-speed interface technology called Infiniband may be used.

  In a cluster system in which a plurality of processor nodes are interconnected using 10 Gigabit Ethernet (trademark or registered trademark) or Infiniband technology, there is an advantage that high-speed communication between processors can be realized, but a switch is still expensive. Therefore, it is difficult to provide a cluster system at a low price, and there is a limit to increasing the number of processor nodes in the cluster. Further, Ethernet (trademark or registered trademark) and Infiniband have disadvantages such as large software overhead such as packet generation and protocol processing.

  The present invention has been made in view of these problems, and an object of the present invention is to realize a technique for realizing high-speed communication between processors by combining a plurality of processors by inexpensive means, and a processor node system and a processor using the technique. To provide a node cluster system.

  In order to solve the above problems, a processor node system according to an aspect of the present invention includes a processor and a bridge that relays data between an input / output bus of the processor and a PCI express to which a peripheral device is connected. A plurality of processor boards. The port of the bridge is configured to be set to a root complex mode in which the processor is a host or an endpoint mode in which the processor is a peripheral device. By connecting to a port set to the end point mode of the bridge of the processor boards, the plurality of processor boards are mutually coupled.

  A PCI express connector provided in a port set in the root complex mode of the bridge of the one processor board and a PCI express connector provided in a port set in the end point mode of the bridge of the other processor board May be connected by wiring.

  The PCI express connector provided in the port set in the root complex mode of the bridge of the one processor board and the PCI express connector provided in the port set in the endpoint mode of the bridge of the other processor board are interconnected. A single backplane substrate may be further provided.

  Another aspect of the present invention is also a processor node system. This processor node system includes four processor boards on which two sets of bridges for relaying data between a processor and a PCI express to which the input / output bus of the processor and peripheral devices are connected are mounted. Each bridge has a port set in a root complex mode where the processor is a host and a port set in an endpoint mode where the processor is a peripheral device. A port set in the root complex mode of one processor board is connected to a port set in the endpoint mode of the bridge of another processor board. Of the four processor boards, two of the four processor boards are mutually coupled using three ports.

  The four processor board surfaces are installed parallel to each other in the housing, and the PCI express connector provided at the bridge port of each processor board is arranged on the rear surface of the housing. The PCI express connectors of each processor board may be connected by a flexible board.

  The four processor board surfaces are installed in a case in parallel to each other, and a PCI express connector provided at a bridge port of each processor board is arranged on the back side of the case. One back for interconnecting a PCI express connector provided in a port set in the root complex mode and a PCI express connector provided in a port set in the endpoint mode of the bridge of the other processor board A plain substrate may be further provided.

  Yet another embodiment of the present invention is a processor node cluster system. This processor node cluster system includes a plurality of processor node systems. By connecting unused ports that are not used for connection between processor boards in the processor node system between two adjacent processor node systems, the plurality of processor node systems are mutually coupled.

  It should be noted that any combination of the above-described constituent elements and the expression of the present invention converted between a method, an apparatus, a system, a computer program, a data structure, a recording medium, and the like are also effective as an aspect of the present invention.

  According to the present invention, an inexpensive and high-performance system can be configured by interconnecting a plurality of processors.

  The cluster system according to the embodiment is configured by tightly coupling a board (board) on which a processor is mounted with a flexible board. The configuration of each processor board will be described with reference to FIG. 1, and the configuration of a node in which four processor boards are tightly coupled by a flexible board will be described with reference to FIG. With reference to FIG. 3, a cluster system configured by connecting a plurality of nodes with a flexible substrate will be described. A form in which the processor boards are connected by a flexible board will be described with reference to FIGS.

  FIG. 1 is a configuration diagram of the processor board 50. On the processor board 50, two multicore processors (hereinafter referred to as “MCP”) 20 and 21 are mounted. Each of the MCPs 20 and 21 is obtained by integrating a plurality of processor cores in one package, and includes one processing element (PE) and a plurality of sub-processing elements (SPE) as the processor core. The PE has a cache memory, and performs information processing while caching data read from the DRAM 10. The PE controls the entire MCPs 20 and 21 in an integrated manner. Each SPE has a local memory, and performs information processing while reading / writing data from / to the local memory. Multiple SPEs operate asynchronously.

  The two MCPs 20 and 21 are connected to each other via an input / output interface (hereinafter referred to as “IOIF”) 64, and high-speed data communication is possible. Further, each MCP 20, 21 is connected to the upstream (upstream) port of the bridges 30, 31 via the IOIFs 62, 63. Various peripheral (peripheral) devices and other processor boards are connected to the downstream (downstream) ports of the bridges 30 and 31 via PCI express 66 and 67.

  Here, PCI Express 66 and 67 comply with the specification of PCI Express (trademark or registered trademark), but are not limited to the current PCI Express standard, and conform to the current PCI Express standard. It may be based on a standard that has been expanded or developed from the current PCI Express standard. Peripheral devices and other processor boards connected by the PCI express 66 and 67 are hereinafter referred to as “PCI devices”.

  The IOIFs 62, 63, and 64 have two channels, upstream and downstream, and realize a high bandwidth comparable to the memory bus, for example, several tens of gigabytes / second. A predetermined memory area of each MCP 20, 21 is memory-mapped into an I / O address space that can be referred to via the IOIFs 62, 63, 64. Each MCP 20, 21 can access the memory area of another MCP mapped to the I / O address space via the IOIFs 62, 63, 64, thereby realizing high-speed interprocessor communication.

  Each bridge 30, 31 “bridges” the IOIFs 62, 63 and the PCI express 66, 67, thereby interconnecting the MCPs 20, 21 and the PCI device. Since the IOIFs 62 and 63 and the PCI express 66 and 67 have different bus standards, the bridges 30 and 31 convert the protocol between the two buses, and exchange data between the MCPs 20 and 21 and the PCI device. Match the format to the specifications of each bus.

  When a PCI device is further connected to the end of the PCI device connected to the PCI express 66 or 67 via the PCI express, the MCP 20 or 21 is set as the root, and the PCI device is connected to the leaf. A PCI device tree structure is formed. Hereinafter, the tree structure of the PCI device is referred to as “PCI tree”.

  Two downstream ports of each bridge 30 and 31 are provided. One can be configured and used as a root complex (RC) and the other as an endpoint (EP). One port may be configured to be able to switch between the root complex mode and the endpoint mode. When the downstream ports of the bridges 30 and 31 are used as a root complex, the MCPs 20 and 21 function as hosts for connecting PCI devices as roots of the PCI tree. When the downstream ports of the bridges 30 and 31 are used as endpoints, the MCPs 20 and 21 function as PCI devices connected to the host.

  Downstream ports of the bridges 30 and 31 are provided with route complex connectors 40R and 41R and endpoint connectors 40E and 41E. In the present embodiment, three connectors out of a total of four connectors 40R, 40E, 41R, 41E provided on the processor board 50 are used for connection to the other three processor boards in the same node, and the remaining One connector is used for connection to the processor board of another node.

  FIG. 2 is a configuration diagram of the node 100 in which four processor boards are tightly coupled. In the node 100, a first processor board 50, a second processor board 51, a third processor board 52, and a fourth processor board 53 are connected to each other by a flexible board. The configuration of each of the processor boards 50 to 53 is as described in FIG.

  Hereinafter, the first processor board 50, the second processor board 51, the third processor board 52, and the fourth processor board 53 are respectively referred to as “processor board 0”, “processor board 1”, “processor board 2”, and “processor board 3”. Call it.

  The two MCPs 20 and 21 mounted on the processor board 0 are called “MCP0” and “MCP1”, respectively, and the bridges 30 and 31 connected to the MCP0 and MCP1 are called “bridge 0” and “bridge 1”, respectively. Similarly, the two MCPs 22 and 23 mounted on the processor board 1 are referred to as “MCP2” and “MCP3”, respectively, and the bridges 32 and 33 connected to the MCP2 and MCP3 are referred to as “bridge 2” and “bridge 3”, respectively. Call. The two MCPs 24 and 25 mounted on the processor board 2 are called “MCP4” and “MCP5”, respectively, and the bridges 34 and 35 connected to the MCP4 and MCP5 are called “bridge 4” and “bridge 5”, respectively. The two MCPs 26 and 27 mounted on the processor board 3 are called “MCP6” and “MCP7”, respectively, and the bridges 36 and 37 connected to the MCP6 and MCP7 are called “bridge 6” and “bridge 7”, respectively.

  An IOIF interconnecting two MCPs in each processor board is called “IOIF0”, and an IOIF between the MCP and the upstream port of the bridge is called “IOIF1”.

  The RC connector and the EP connector of the bridge 0 are referred to as “connector RC0” and “connector EP0”, respectively. Similarly, the RC connectors of the bridges 1 to 7 are called “connectors RC1” to “connectors RC7”, respectively, and the EP connectors of the bridges 1 to 7 are called “connectors EP1” to “connectors EP7”, respectively.

  The connector RC0 of the bridge 0 on the MCP0 side of the processor board 0 is connected to the connector EP3 of the bridge 3 on the MCP3 side of the processor board 1. With this connection, when viewed from the MCP0 of the processor board 0, the MCP0 functions as a root complex, and the MCP3 of the processor board 1 functions as an end point. That is, the MCP0 of the processor board 0 is a host, and the processor board 1 is connected as a PCI device, and a PCI tree is formed by connecting the MCP3 with the MCP0 as a root.

  The connector RC1 of the bridge 1 on the MCP1 side of the processor board 0 is connected to the connector EP4 of the bridge 4 on the MCP4 side of the processor board 2. When viewed from the MCP1 of the processor board 0 that is the root complex, a PCI tree is formed in which the MCP1 of the processor board 0 is the host and the processor board 2 is the device.

  The connector EP1 of the bridge 1 on the MCP1 side of the processor board 0 is connected to the connector RC6 of the bridge 6 on the MCP6 side of the processor board 3. When viewed from the MCP 6 of the processor board 3 that is the root complex, a PCI tree is formed with the MCP 6 of the processor board 3 as a host and the processor board 0 as a device.

  Similarly, the connector RC2 of the bridge 2 on the MCP2 side of the processor board 1 is connected to the connector EP5 of the bridge 5 on the MCP5 side of the processor board 2, and the connector EP2 of the bridge 2 is connected to the bridge 7 on the MCP7 side of the processor board 3. Connected to the connector RC7. The connector RC4 of the bridge 4 on the MCP4 side of the processor board 2 is connected to the connector EP7 of the bridge 7 on the MCP7 side of the processor board 3.

  Bridge connectors not used for connection between processor boards in node 100, ie, connector EP0 of bridge 0 of processor board 0, connector RC3 of bridge 3 of processor board 1, connector RC5 of bridge 5 of processor board 2, and The connector EP6 of the bridge 6 of the processor board 3 is used as a free slot for connection to the processor board of another node.

  FIG. 3 is a configuration diagram of a cluster system 200 in which a plurality of nodes are connected. In the cluster system 200, the nodes 100 to 102, 110 to 112, and 120 to 120 having the configuration described in FIG. For example, the node 101 is connected to the right of the node 100, and the node 102 is connected to the right of the node 101. A node 110 is connected below the node 100, and a node 120 is connected further below the node 110.

  As shown in the figure, the two nodes arranged in the left and right direction are coupled by connecting the connector EP6 of the processor board 3 of the left node and the connector RC3 of the processor board 1 of the right node. The two nodes arranged vertically are coupled by connecting the connector RC5 of the processor board 2 of the upper node and the connector EP0 of the processor board 0 of the lower node.

  In the cluster system 200, the connector on the side where the adjacent node of the node located at the end does not exist becomes an empty slot. By connecting various peripheral devices to this empty slot and further connecting nodes, the system Can be extended.

  As described above, the cluster system 200 has an advantage that the number of nodes can be freely increased by the arrangement on the plane connecting the nodes vertically and horizontally.

  In the cluster system 200, a flexible substrate is used for connection between four processor boards in each node and for connection between nodes. Hereinafter, a connection form using a flexible substrate will be described with reference to FIGS.

  FIG. 4 is a schematic diagram of the wiring on the back surface of the processor board 50. In the figure, wiring between MCP0 and a plurality of DRAMs 10, wiring between MCP0 and bridge 0, and wiring between bridge 0 and connectors RC0 and EP0 are shown. Also, wiring between MCP1 and a plurality of DRAMs 11, wiring between MCP1 and bridge 1, and wiring between bridge 1 and connectors RC1 and EP1 are shown. Each connector RC0, EP0, RC1, EP1 is a PCI-Express × 16 connector and can be connected to a flexible substrate.

  FIG. 5 is a diagram showing a configuration in which the four processor boards 50 to 53 in the node 100 are connected by a flexible board. A flexible substrate is a kind of printed wiring board, also called FPC (Flexible Printed Circuit), and is thin and flexible.

  The processor boards 50 to 53 (processor boards 0 to 3) described in FIG. 2 are easily connected to each other by a flexible board, the processor board 1 (reference numeral 51), the processor board 0 (reference numeral 50), and the processor board 2 (reference numeral). 52) and the processor boards 3 (reference numeral 53) in this order, the board surfaces are arranged parallel to each other.

  The connector RC2 of the processor board 1 is connected to the connector EP5 of the processor board 2 by the flexible board 201. The connector EP2 of the processor board 1 is connected to the connector RC7 of the processor board 3 by the flexible board 202. The connector EP3 of the processor board 1 is connected to the connector RC0 of the processor board 0 by the flexible board 203.

  The connector RC1 of the processor board 0 is connected to the connector EP4 of the processor board 2 by the flexible board 204. The connector EP1 of the processor board 0 is connected to the connector RC6 of the processor board 3 and the flexible board 205. The connector RC4 of the processor board 2 is connected to the connector EP7 of the processor board 3 by the flexible board 206.

  The flexible board 202 that connects the connector EP2 of the processor board 1 and the connector RC7 of the processor board 3 straddles the upper side of the flexible board 204 that connects the connector RC1 of the processor board 0 and the connector EP4 of the processor board 2. If a flexible board is used in this way, a connection configuration in which another wiring passes over the wiring is possible, and four processor boards can be arranged in parallel and tightly coupled to each other to save space. .

  In addition, since the configuration is such that the processor boards are connected using a general-purpose PCI Express connector and a flexible board, it is far cheaper than the configuration in which the processor boards are interconnected by a PCI Express switch or the like, and the manufacturing cost is low. Can be reduced.

  Furthermore, by increasing the component mounting density of the processor board and reducing the size of the processor board, the processor boards can be interconnected with a shorter flexible board, and high-speed signals can be handled. PCI-Express is premised on high-speed communication, and signal propagation is slow in cable connection, and it is difficult to realize tight coupling between processor boards by cable connection. In this embodiment mode, the processor boards are wired with flexible boards, so that high-speed signal propagation is possible.

  FIG. 6 is a diagram showing a configuration in which a plurality of nodes in the cluster system 200 are connected by a flexible substrate. In the figure, the connection form of the four adjacent nodes 100, 101, 110, 111 of FIG. 3 is shown. The four processor boards in each of the nodes 100, 101, 110, and 111 are connected by a flexible board as described with reference to FIG. However, for the node 110, in order to make it easy to grasp the connection form between the nodes, a flexible substrate for connecting the processor boards in the node is not shown.

  The connector EP6 of the processor board 3 of the node 100 is connected to the connector RC3 of the processor board 1 of the node 101 by the flexible board 211. As a result, the two nodes 100 and 101 are coupled in the left-right direction. Similarly, the connector EP6 of the processor board 3 of the node 110 is connected to the connector RC3 of the processor board 1 of the node 111 and the flexible board 221, and the two nodes 110 and 111 are coupled in the left-right direction.

  The connector RC5 of the processor board 2 of the node 100 is connected to the connector EP0 of the flexible board 0 of the node 110 by the flexible board 214. As a result, the two nodes 100 and 110 are coupled in the vertical direction. Similarly, the connector RC5 of the processor board 2 of the node 101 is connected by the connector EP0 of the flexible board 0 of the node 111 and the flexible board 224, and the two nodes 101 and 111 are coupled in the vertical direction.

  In the cluster system 200, a flexible substrate is also used for connection between nodes, and space saving and cost reduction can be achieved. Since the cluster system 200 has a configuration in which a plurality of nodes are arranged in a plane on the top, bottom, left, and right, the distance between adjacent nodes can be shortened, and the length of the flexible substrate used for inter-node connection is sufficient. And a high-speed PCI-Express signal can be handled.

  FIG. 7 is a diagram for explaining the housing of the node 100. The housing of the node 100 accommodates four processor boards 50 to 53, and the connectors on the back are connected by the six flexible boards 201 to 206 as described with reference to FIG. Further, as described with reference to FIG. 6, the processor board 0 is provided with the flexible board 213 for connecting to the processor board 2 of the node adjacent in the upward direction. A flexible substrate 214 for connection to the processor substrate 0 is provided. On the other hand, the processor board 1 is provided with a flexible board 212 for connection to the processor board 3 of the node adjacent in the left direction, and the processor board 3 is connected to the processor board 1 of the node adjacent in the right direction. Flexible substrate 211 is provided.

  FIG. 8 is a diagram for explaining the chassis of the cluster system 200. The chassis of the node 100 of FIG. 7 is arranged vertically and horizontally, and the nodes are connected in the horizontal direction by the flexible boards 211 and 212 described in FIG. 7, and the nodes are connected in the vertical direction by the flexible boards 213 and 214. In this way, the cluster system 200 can be easily configured by arranging the node housings on a plane and connecting them with a flexible substrate. In addition, it is easy to add a node, there is scalability, and a node cluster in which a large number of nodes are combined can be provided in a small space and at a low cost.

  4-8, the processor board was provided with the connector for flexible boards, and the form which connects between the connectors for flexible boards with a flexible board was demonstrated. Thus, the form which connects a general purpose PCI express connector with a flexible substrate is only an example of a connection form, and other connection forms are also conceivable. As another connection form, a single backplane board equipped with a general-purpose PCI Express connector for inserting the card edge of the processor board is prepared, and four processor boards are inserted into the backplane board, which will be described with reference to FIG. The connection between the PCI express connectors may be realized on the backplane board. As yet another connection form, a ZD connector that is a differential signal connector pair may be provided on the processor board, and the ZD connector may be connected on the backplane board. A connection form using such a backplane substrate can realize inexpensive high-speed communication and can save space, similarly to the connection form using a flexible substrate.

  With reference to FIGS. 9A to 9D, a PCI tree formed by a full-mesh coupling of four processor boards in the node 100 described in FIG. 2 will be described. The PCI tree is obtained by searching for a PCI device to which each MCP in the node is connected by PCI Express.

  FIG. 9A is a diagram for explaining a PCI tree when MCP0 or MCP1 is placed at the center. In the figure, MCPs in the same hierarchy in the PCI tree structure are arranged horizontally, with the one closer to the root up and the one closer to the leaf down.

  MCP3 is connected as an end point immediately below the root complex MCP0. MCP2 is on the same level as MCP3 and is connected to MCP3. As a result, a first PCI tree having MCP0 as a root is formed. MCP1 is on the same level as MCP0 and is connected to MCP0. The MCP 4 is connected as an end point to the hierarchy immediately below the MCP 1 that is the root complex. The MCP 5 is on the same level as the MCP 4 and is connected to the MCP 4. As a result, a second PCI tree having MCP1 as a root is formed. The MCP 6 that is the root complex is in the hierarchy immediately above the MCP 1 and connects the MCP 1 as an end point. The MCP 7 is on the same level as the MCP 6 and is connected to the MCP 6. As a result, a third PCI tree having MCP 6 as a root is formed.

  FIG. 9B is a diagram for explaining a PCI tree when MCP2 or MCP3 is placed at the center. The MCP 5 is connected as an end point to the hierarchy immediately below the MCP 2 that is the root complex. The MCP 4 is on the same level as the MCP 5 and is connected to the MCP 5. As a result, a first PCI tree having MCP2 as a root is formed. The MCP 7 that is the root complex is in the hierarchy immediately above the MCP 2 and connects the MCP 2 as an end point. The MCP 6 is on the same level as the MCP 7 and is connected to the MCP 7. As a result, a second PCI tree having MCP 7 as a root is formed. MCP3 is on the same level as MCP2 and is connected to MCP2. The endpoints of other nodes are connected to the hierarchy immediately below the MCP 3 that is the root complex. MCP0, which is the root complex, is in the hierarchy immediately above MCP3 and connects MCP3 as an end point. MCP1 is on the same level as MCP0 and is connected to MCP0. As a result, a third PCI tree rooted at MCP0 is formed.

  FIG. 9C is a diagram illustrating a PCI tree when MCP4 or MCP5 is placed at the center. The MCP 7 is connected as an end point immediately below the root complex MCP 4. The MCP 6 is on the same level as the MCP 7 and is connected to the MCP 7. As a result, a first PCI tree having MCP 4 as a root is formed. The MCP1 that is the root complex is in a hierarchy immediately above the MCP4 and connects the MCP4 as an end point. MCP0 is on the same level as MCP1 and is connected to MCP1. As a result, a second PCI tree having MCP1 as a root is formed. The MCP 5 is on the same level as the MCP 4 and is connected to the MCP 4. The endpoints of other nodes are connected to the hierarchy immediately below the MCP 5 that is the root complex. The MCP2 that is the root complex is in a hierarchy immediately above the MCP5 and connects the MCP5 as an end point. MCP3 is on the same level as MCP2 and is connected to MCP2. As a result, a third PCI tree rooted at MCP2 is formed.

  FIG. 9D is a diagram illustrating a PCI tree when MCP6 or MCP7 is placed at the center. MCP1 is connected as an end point immediately below the root complex MCP6. MCP0 is on the same level as MCP1 and is connected to MCP1. As a result, a first PCI tree having MCP 6 as a root is formed. The MCP 7 is on the same level as the MCP 6 and is connected to the MCP 6. The MCP 2 is connected as an end point to the layer immediately below the MCP 7 that is the root complex. MCP3 is on the same level as MCP2 and is connected to MCP2. As a result, a second PCI tree having MCP 7 as a root is formed. The MCP4 that is the root complex is in a layer immediately above the MCP7, and connects the MCP7 as an end point. The MCP 5 is on the same level as the MCP 4 and is connected to the MCP 4. As a result, a third PCI tree having MCP 4 as a root is formed.

  As described above, by connecting the RC connector and the EP connector between the four processor boards in the node 100 as described in FIG. 2, a plurality of PCI trees having a certain MCP as a root are formed. Each of the MCP0 to MCP7 in the node 100 can perform data communication with another MCP within a PCI tree rooted in itself or across different PCI trees. Since the MCP on the processor board of the adjacent node connected to the connector of the empty slot of the node 100 is in the same PCI tree, data communication is possible across the nodes. However, when data communication is performed with an MCP of another node that is not connected to the connector of the empty slot of the node 100, routing is necessary because it is not in the same PCI tree. For this reason, each MCP in the node 100 performs routing by software and enables communication with MCPs in other PCI trees.

  The memory space of each MCP has another MCP so that each MCP in the node 100 can access a predetermined shared area of the other MCP within its own PCI tree or across different PCI trees. The predetermined shared area is memory-mapped. This memory mapping will be described with reference to FIGS.

  FIG. 10 is a diagram for explaining the memory space 300 of each MCP in the node 100. The memory space 300 includes a coherent local memory area 351 and a non-coherent shared memory area 352. The coherent local memory area 351 is an area in which atomicity of memory access is ensured and synchronization control is performed, and cannot be accessed from other MCPs. The non-coherent shared memory area 352 is mapped to the memory space of another MCP and accessed from the other MCP. The memory space 300 further includes a non-coherent area 353 in which the SPE and PE registers of each MCP and the local store of the SPE are mapped. At least a part of the non-coherent area 353 is mapped to the memory space of another MCP and accessed from the other MCP.

  In the memory space 300, an I / O address space accessible via the IOIF0 is memory-mapped as an IOIF0 area 360. In addition, an I / O address space accessible via IOIF1 is memory-mapped as an IOIF1 area 370.

  Each MCP uses the SPE / PE registers included in the non-coherent area 353 in its own memory space 300, the local store of the SPE, and the non-coherent shared memory area 352 as a shared area via the IOIF0. Open to other MCPs to allow access. Each MCP stores information on a shared area that permits access to other MCPs in an IOIF0 I / O page table (hereinafter referred to as “IOPT”) 310. Other MCPs refer to this IOPT 310 and map the shared area to their memory space to make it accessible.

  FIG. 11 is a diagram for explaining how the shared area of MCP0 is mapped to the memory space 301 of MCP1 and the shared area of MCP1 is mapped to the memory space 300 of MCP0. When an IOPT 310 (“IOPT0”) for IOIF0 of MCP0 is presented to MCP1 via IOIF0, the shared area 321 of MCP0 specified by IOPT0 is mapped to the IOIF0 area 361 of the memory space 301 of MCP1. The shared area 321 of the MCP0 includes the SPE / PE register of the MCP0, the local store of the SPE of the MCP0, and the shared memory of the MCP0.

  On the other hand, when IOPT 311 (“IOPT1”) for IOIF0 of MCP1 is presented to MCP0 via IOIF0, shared area 320 of MCP1 specified by IOPT1 is mapped to IOIF0 area 360 of memory space 300 of MCP0. The shared area 320 of the MCP1 includes the SPE / PE register of the MCP1, the local store of the SPE of the MCP1, and the shared memory of the MCP1.

  As described above, MCP0 and MCP1 connected via IOIF0 can access each other's shared area because the other party's shared area is memory-mapped in their own memory spaces 300 and 301.

  FIG. 12 is a diagram for explaining how the shared areas of MCP0 and MCP1 interconnected by IOIF0 are mapped to the memory spaces of other MCPs connected via IOIF1.

  As described with reference to FIG. 11, the shared area 321 of MCP0 is mapped to the IOIF0 area 361 of the memory space of MCP1. The MCP 1 opens the shared area 321 of the MCP 0 together with its own shared area to other MCPs connected via the IOIF 1 and permits access. The MCP 1 stores information on its own shared area, that is, the SPE / PE register of the MCP 1, the local store of the MCP 1, and the shared memory of the MCP 1 in the IOPT 331 for the IOIF 1. Further, the MCP 1 stores information of the shared area 321 of the MCP 0, that is, the SPE / PE register of the MCP 0, the local store of the MCP 0, and the shared memory of the MCP 0 in the IOPT 331 of the IOIF 1.

  As described with reference to FIG. 9A, the connection relationship between MCP1 and MCP6 is that MCP6 is a root complex and MCP1 is an end point relationship. Therefore, MCP1 that is an end point transmits information of its own shared area to MCP 6 that is the root complex. Present. The MCP 1 presents the IOPT 331 for the IOIF 1 to the MCP 6 connected via the IOIF 1. The MCP 6 maps the shared area 342 of both MCP 0 and MCP 1 specified by the IOPT 331 for IOIF 1 to the IOIF 1 area 376 of its own memory space 306.

  In FIGS. 13A and 13B, when the MCP0 receives an IOPT for the IOIF1 from another MCP connected via the IOIF1, the shared area of the other MCP is mapped to the memory space 300 of the MCP0. FIG.

  As shown in FIG. 13A, the MCP0 that is the root complex is connected to the MCP3 that is the end point via the IOIF1. Since MCP3 is interconnected with MCP2 by IOIF0, the shared area of MCP2 is mapped to the IOIF0 area of the memory space of MCP3. Similar to the presentation of IOPT for IOIF1 from MCP1 to MCP6 described with reference to FIG. 12, MCP3 as an end point stores information on its own shared area and shared area of MCP2 in IOPT for IOIF1 in the root complex. Present to a certain MCP0.

  MCP0 receives the IOPT for IOIF1 from MCP3, and maps MCP2 and MCP3 shared area 340 to IOIF1 area 370 of memory space 300, as shown in FIG.

  In FIGS. 14A and 14B, when MCP1 receives presentation of IOPT for IOIF1 from another MCP connected via IOIF1, the shared area of other MCP is mapped to memory space 301 of MCP1. FIG.

  As shown in FIG. 14A, the MCP1 that is the root complex is connected to the MCP4 that is the end point via the IOIF1. Since MCP4 is interconnected with MCP5 by IOIF0, the shared area of MCP5 is mapped to the IOIF0 area of the memory space of MCP4. The MCP 4 that is the end point stores the information of its own shared area and the shared area of the MCP 5 in the IOPT for the IOIF 1 and presents it to the MCP 1 that is the root complex. MCP1 receives the presentation of IOPT for IOIF1 from MCP4, and maps MCP4 and MCP5 shared area 346 to IOIF1 area 371 of memory space 301 as shown in FIG. 14B.

  Similarly, the MCP 6 stores the information of its own shared area and the shared area of the MCP 7 in the IOPT for the IOIF 1 and presents the information to the MCP 1. The MCP 1 receives the IOPT for the IOIF 1 from the MCP 6, and FIG. ), The shared area 348 of MCP6 and MCP7 is mapped to the IOIF1 area 371 of the memory space 301.

  Next, MCP0 and MCP1 are information on shared areas of other MCPs connected via IOIF1 that are mapped to IOIF1 areas 370 and 371 of memory spaces 300 and 301 in FIGS. 13B and 14B. Exchange each other.

  FIG. 15 is a diagram for explaining how the information of the shared area that is memory-mapped is exchanged between MCP0 and MCP1. The MCP0 gives the information of the MCP2 and the shared area 340 of the MCP3 mapped to the IOIF1 area 370 of the memory space 300 to the MCP1 via the IOIF0. MCP1 maps the shared area of MCP2 and MCP3 to IOIF0 area 361 of its own memory space 301 based on the information given from MCP0.

  On the other hand, the MCP 1 gives the information of the shared area 346 of the MCP 4 and MCP 5 mapped to the IOIF 1 area 371 of the memory space 301 and the information of the shared area 348 of the MCP 6 and MCP 7 to the MCP 0 via the IOIF 0. MCP0 maps the shared areas of MCP4 and MCP5 and the shared areas of MCP6 and MCP7 to IOIF0 area 360 of its own memory space 300 based on the information given from MCP1.

  As a result of the memory mapping of the shared areas of other MCPs in the memory space according to the procedure described with reference to FIGS. 11, 13B, 14B, and 15, the MCP 0 is the first described with reference to FIG. 9A. It becomes possible to access the shared areas of MCP1 to MCP7 in the third PCI tree. This is because one address map is constructed across the first to third PCI trees. Similarly, MCP1 can access the shared areas of MCP0 and MCP2 to MCP7 in the first to third PCI trees described in FIG. 9A.

  In this way, each MCP in the node 100 performs memory mapping of the shared area of other MCPs in the first to third PCI trees to its own memory space, and other MCPs in the first to third PCI trees. It is possible to access the shared area and execute data communication and synchronization control via the shared area with other MCPs in the first to third PCI trees. A hardware configuration is adopted in which the processor boards in the node 100 are connected by a flexible board, and high-speed PCI-Express communication is possible. Accordingly, each MCP in the node 100 can access the memory mapped shared area at high speed, and can efficiently exchange data with other MCPs.

  FIG. 16 is a diagram for explaining a PCI tree including connections with MCPs of connected nodes. The connector EP0 of the bridge 0 of the MCP0 is connected to the connector RC5 of the bridge 5 'of the adjacent node, and the MCP 5' becomes a root complex with respect to the MCP0. The MCP 4 'is on the same level as the MCP 5' and is connected to the MCP 5 '. The MCP 0 receives the IOPT 1 IOPT from the adjacent node MCP 5 ′, and maps the shared area 349 of the MCP 5 ′ and MCP 4 ′ to the IOIF 1 area 370 of the memory space 300.

  FIG. 17 is a diagram for explaining the memory space 300 of MCP0 in the case of the PCI tree of FIG. As shown in FIG. 17, in the IOIF1 area 370, in addition to the shared area 340 of MCP2 and MCP3, the shared area 340 of MCP4 'and MCP5' is memory-mapped. In the IOIF0 area 360, the MCP4 and MCP5 shared area 326, the MCP6 and MCP7 shared area 328, and the MCP1 shared area 320 are memory-mapped.

  In summary, the PCI memory map is configured by setting the base addresses of devices and switches by the host processor at the root of the PCI tree. The device as the end point notifies the host processor of the size of the address area requested by itself, and the host processor constructs a memory map according to the size requested by the device. Specifically, the size of the requested address range is specified by the number of bits implemented in the BAR field of the configuration register.

  When the bridge device according to the present embodiment operates as an endpoint, a configuration register accessible from the outside and a configuration register accessible from the inside are mounted as separate registers, and addresses required for the respective registers. It is possible to set the size of the range, that is, the number of bits installed in the BAR. Thus, a PCI address map is constructed by each host processor according to the address range of the size set at the time of system initialization. Here, each host processor refers to a processor that is a root complex and a processor that operates as an endpoint.

  On the other hand, the memory map of IOIF is set by storing shared area information in IOPT and presenting it to other MCPs. In this operation, when there is a memory access from the outside, an area to be mapped ahead is set. This setting work is required when a transaction is received from the PCI and allowed as a memory access, regardless of whether the user operates as a root complex or as an endpoint.

  In actual operation, the PCI memory map is constructed by setting the address size constructed by PCI with a margin, and the range in which the memory is actually mapped is set by IOPT. As for the address range of the PCI memory map, according to the PCI Express standard, the end point notifies and the root complex constructs the address. Among them, the range is mapped to the memory or local store. As described with reference to FIG. 15, it is necessary to exchange information regarding whether or not there is an original protocol via a shared memory.

  As described above, according to the present embodiment, a general-purpose PCI Express connector on a processor board is directly connected by an inexpensive flexible board or a backplane board, so that a PCI Express switch is not required and is inexpensive. A high-performance cluster system can be constructed.

  The present invention has been described based on the embodiments. The embodiments are exemplifications, and it will be understood by those skilled in the art that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are within the scope of the present invention. .

  In the above embodiment, the case where the multi-core processor is mounted on the processor board has been described, but this may be a single processor. Further, in the embodiment, an example in which two multi-core processors are mounted on a processor board and one node is configured by four processor boards has been described. However, the number of processors mounted on the processor board and one node The number of processor boards, the number of bridge connectors, etc. have design freedom. As long as a plurality of processor boards in a node can be tightly coupled by a flexible board and a node cluster can be configured by further connecting the nodes with a flexible board, the number and arrangement of processor boards in the node, the nodes in the node cluster There can be various patterns of arrangement. In any case, a connection configuration that satisfies the requirements of low cost, space saving, and high-speed communication is preferable.

  In the above embodiment, the processor board bridge port is connected to the processor board bridge port, but a peripheral device is connected to the processor board bridge port, and the system is configured by mutually coupling the processor and various peripheral devices. May be. In addition, although the bridge connects the input / output bus of the processor to PCI Express, an interface other than PCI Express may be used as an external interface to which other processor boards and peripheral devices are connected.

It is a block diagram of a processor board. It is a block diagram of a node in which four processor boards are tightly coupled. It is a block diagram of the cluster system which connected the some node. It is a schematic diagram of the wiring of the back surface of a processor board | substrate. It is a figure which shows the structure which connected between four processor boards in a node by the flexible substrate. It is a figure which shows the structure which connected between the some nodes in a cluster system by the flexible substrate. It is a figure explaining the housing | casing of a node. It is a figure explaining the housing | casing of a cluster system. FIG. 3 is a diagram illustrating a PCI tree formed in the node of FIG. 2. FIG. 3 is a diagram illustrating a PCI tree formed in the node of FIG. 2. FIG. 3 is a diagram illustrating a PCI tree formed in the node of FIG. 2. FIG. 3 is a diagram illustrating a PCI tree formed in the node of FIG. 2. It is a figure explaining the memory space of each MCP in a node. It is a figure explaining a mode that the common area | region of another MCP is mapped by the memory space of a certain MCP. It is a figure explaining a mode that the common area | region of another MCP is mapped by the memory space of a certain MCP. It is a figure explaining a mode that the common area | region of another MCP is mapped by the memory space of a certain MCP. It is a figure explaining a mode that the common area | region of another MCP is mapped by the memory space of a certain MCP. It is a figure explaining a mode that the information of the shared area memory-mapped between two MCPs is exchanged. It is a figure explaining the PCI tree including the connection with MCP of a connection node. It is a figure explaining the memory space of MCP in the case of the PCI tree of FIG.

Explanation of symbols

  10 DRAM, 20 multi-core processor, 30 bridge, 50 processor board, 100 node, 200 cluster system, 201-206, 211-214 flexible board, 300 memory space, 310 IOPT.

Claims (12)

A plurality of processor boards each including a processor and a bridge that relays data between an input / output bus of the processor and a PCI express to which peripheral devices are connected;
The port of the bridge is configured to be set to a root complex mode where the processor is a host or an endpoint mode where the processor is a peripheral device,
By connecting a port set to the root complex mode of the bridge of one processor board to a port set to the endpoint mode of the bridge of another processor board, the plurality of processor boards are mutually coupled. A processor node system characterized by the above.   A PCI express connector provided in a port set in the root complex mode of the bridge of the one processor board and a PCI express connector provided in a port set in the end point mode of the bridge of the other processor board The processor node system according to claim 1, wherein the processor node system is connected by wiring.   The PCI express connector provided in the port set in the root complex mode of the bridge of the one processor board and the PCI express connector provided in the port set in the endpoint mode of the bridge of the other processor board are interconnected. The processor node system according to claim 1, further comprising a single backplane board for performing the processing. Including four processor boards on which two sets of a processor and a bridge for relaying data between an input / output bus of the processor and a PCI express to which peripheral devices are connected;
Each bridge has a port set in a root complex mode where the processor is a host and a port set in an endpoint mode where the processor is a peripheral device,
A port set in the root complex mode of one processor board is connected to a port set in the endpoint mode of the bridge of another processor board. 3. A processor node system comprising three ports, wherein two arbitrary processor boards among the four processor boards are mutually coupled.   The three ports of each processor board are defined as a first port, a second port, and a third port, and three processor boards other than the processor board are designated as a first processor board, a second processor board, and a third processor board. In this case, the first port is connected to the first processor board, the second port is connected to the second processor board, and the third port is connected to the third processor board. The processor node system according to claim 4. Two processors in the processor board are connected via an input / output bus, and by connecting ports between the processor boards, any two of a total of eight processors on the four processor boards can be connected. 6. The processor node system of claim 5 , wherein the processors are communicatively interconnected with each other. A PCI express connector provided in a port set in the root complex mode of the bridge of the one processor board and a PCI express connector provided in a port set in the end point mode of the bridge of the other processor board processor node system according to any one of claims 4 to 6, characterized by comprising hardwired by. The four processor board surfaces are installed parallel to each other in the housing, and the PCI express connector provided at the bridge port of each processor board is arranged on the rear surface of the housing. 8. The processor node system according to claim 7 , wherein PCI express connectors of each processor board are connected by a flexible board. The four processor board surfaces are installed in a case in parallel to each other, and a PCI express connector provided at a bridge port of each processor board is arranged on the back side of the case. One back for interconnecting a PCI express connector provided in a port set in the root complex mode and a PCI express connector provided in a port set in the endpoint mode of the bridge of the other processor board processor node system according to any one of claims 4 to 6, characterized in that further provided a plain substrate. It is configured that each processor is configured to be accessible to the shared area of the other processor by mapping the shared area of the interconnected other processor as an I / O address space in the memory space of each processor. The processor node system according to any one of claims 4 to 9 , wherein A processor node cluster system including a plurality of processor node systems according to any one of claims 4 to 10 and interconnecting each processor node system as a cluster and having a cluster connection form,
A plurality of processor node systems are mutually coupled by connecting empty ports that are not used for connection between processor boards in the processor node system between two adjacent processor node systems. A processor node cluster system. 12. The processor node cluster system according to claim 11 , wherein a PCI express connector is provided in the empty port, and the PCI express connectors of the empty port are connected by wiring with a flexible board.

Images