JP5930046B2 - Information processing apparatus and control method of information processing apparatus - Google Patents

Description

  The present invention relates to an information processing apparatus and a control method for the information processing apparatus.

  An information processing apparatus such as a server has a plurality of CPUs (Central Processing Units), has a physical partitioning function by connecting the CPUs to each other by QPI (Quick Path Interconnect), HT (Hyper Transport), etc. May be taken.

  For example, FIG. 14 is a block diagram illustrating a basic configuration of an information processing apparatus (server) 100 having a physical partitioning function. The server 100 shown in FIG. 14 includes four CPUs 101A to 101D. Note that, when one of the four CPUs is specified, reference numerals 101A to 101D are used, and when an arbitrary CPU is indicated, reference numeral 101 is used. Each CPU 101 has two CPU interconnect ports 102, and the four CPUs 101A to 101D are connected in a ring shape via the CPU interconnect port 102 and the CPU interconnect line 103.

In the example shown in FIG. 14, is divided into two physical servers by the server 100 is a physical path over tee partitioning function, two CPU 101A, 101B and two CPU101C, and the 101D, different partitions # 0, respectively, is assigned to # 1 . At this time, the CPU interconnect line 103 between the two CPUs 101A and 101B belonging to the partition # 0 and the CPU interconnect line 103 between the two CPUs 101C and 101D belonging to the partition # 1 are used as usual. On the other hand, the CPU interconnect lines 103 extending over different partitions # 0 and # 1, that is, the lines 103 between the CPUs 101A and 101D and between the CPUs 101B and 101C are not used while being physically connected, or are connected to the CPU interconnect line 103. The output is kept saved or powered off.

  As shown in FIG. 14, the data communication between the two CPUs 101A and 101D divided by the partition is performed, for example, in the form shown in FIG. 15 or FIG. 16 without using the CPU interconnect line 103 between the CPUs 101A and 101D. It is. FIG. 15 and FIG. 16 are block diagrams showing an example of a communication form between partitions and another example, respectively.

  In the example of the communication form shown in FIG. 15, a PCI Express (registered trademark; PCIe (registered trademark); Peripheral Components Interconnect express) switch 104 is provided under each CPU 101, and a NIC (Network Interface) is provided in the card slot 104 a of each PCIe switch 104. Card) 105 is loaded. Then, NICs 105 and 105 belonging to different partitions # 0 and # 1 are connected via Ethernet (registered trademark) using a cable 106. Thereby, data communication is performed between the CPUs 101A and 101D belonging to different partitions # 0 and # 1, respectively.

  In the other example of the communication form shown in FIG. 16, a PCIe switch 104 ′ that is shared by the CPUs 101A and 101D belonging to the two partitions # 0 and # 1 and has a partitioning function and an NTB (Non-Transparent Bridge) port 104b is provided. It is done. Then, data communication is performed between the CPUs 101A and 101D belonging to different partitions # 0 and # 1 using the NTB port 104b of the PCIe switch 104 ′. The NTB port fulfills the function for sharing data between two different partitions or between hosts in the PCI specification, and has a memory-mapped space in the port and uses that space. It is a port that can exchange data between partitions and between hosts.

JP 2006-40285 A JP 2011-221945 A

  In the communication form shown in FIG. 15, it is necessary to prepare two NICs 105 and a cable 106 separately in order to realize data communication between two different partitions # 0 and # 1. In the communication mode shown in FIG. 16, a switch 104 ′ shared between two different partitions # 0 and # 1 is required. In either communication mode, when communicating between the CPUs 101A and 101D in different partitions, the communication performance is inferior to using the CPU interconnect because the switches 104 and 104 'and the network are used. In this way, when the CPUs 101 belonging to different partitions in the same server 100 communicate with each other, the NIC 105 and the PCIe switch are not utilized even though the low-latency and high-bandwidth interface (CPU interconnect line 103) exists. Communication is performed via 104, 104 ', etc., which is extremely inefficient.

  In one aspect, an object of the present invention is to efficiently realize communication between processing units belonging to different partitions.

  In one proposal, the information processing apparatus includes a plurality of processing units and a setting unit configured to perform settings according to a partition configuration for the plurality of processing units, and each of the plurality of processing units includes other processing units. And a port for communicating with the other processing unit. The port includes a first communication unit, a second communication unit, and a selection unit. The first communication unit communicates with a processing unit belonging to the same partition as the processing unit to which the port belongs. The second communication unit communicates with a processing unit belonging to a partition different from the processing unit to which the port belongs. The selection unit selects one of the first communication unit and the second communication unit according to the setting by the setting unit, and causes the selected communication unit to perform communication with the other processing unit. .

  According to one embodiment, communication between processing units belonging to different partitions can be efficiently realized.

It is a block diagram which shows the principal part structure of the information processing apparatus as 1st Embodiment. It is a block diagram which shows roughly the whole structure of the information processing apparatus of 1st Embodiment to which the principal part structure shown in FIG. 1 is applied. It is a flowchart explaining the function of the setting part shown in FIG. 2, and the selection part shown in FIG. FIG. 3 is a block diagram illustrating a one-partition state of the information processing apparatus illustrated in FIG. 2. FIG. 3 is a block diagram illustrating a two-partition state of the information processing apparatus illustrated in FIG. 2. It is a block diagram which shows the detailed structure of the NTB port upper layer (2nd communication part) shown in FIG. It is a block diagram explaining the address conversion function by the address conversion part of the NTB transaction layer shown in FIG. It is a figure which shows the content of transaction ID. FIG. 7 is a block diagram for explaining a tag conversion / inverse conversion function and a transaction ID conversion / inverse conversion function by an address conversion unit of the NTB transaction layer shown in FIG. 6. It is a block diagram explaining the doorbell function and scratch pad function by the NTB transaction layer shown in FIG. FIG. 7 is a diagram for explaining packet division and encapsulation by a CPU interconnect data link interface (second interface) shown in FIG. 6. It is a block diagram which shows the principal part structure of the information processing apparatus as 2nd Embodiment. It is a block diagram which shows the detailed structure of the NTB port upper layer (2nd communication part) shown in FIG. It is a block diagram which shows the basic composition of the information processing apparatus (server) which has a physical partitioning function. It is a block diagram which shows an example of the communication form between partitions. It is a block diagram which shows the other example of the communication form between partitions.

Hereinafter, embodiments will be described with reference to the drawings.
[1] Information processing apparatus (server) of the first embodiment
[1-1] Basic Configuration of the First Embodiment First, the configuration and function of the information processing apparatus (server) 1 as the first embodiment will be described with reference to FIGS. FIG. 1 is a block diagram illustrating the main configuration of the server 1 according to the first embodiment. FIG. 2 illustrates the overall configuration of the server 1 according to the first embodiment to which the main configuration illustrated in FIG. 1 is applied. FIG. 3 is a flowchart (steps S1 to S7) for explaining the functions of the system firmware (setting unit) 40 shown in FIG. 2 and the multiplexer (selection unit) 25 shown in FIG.

  The server 1 shown in FIGS. 1 and 2 includes four CPUs (processing units) 10A to 10D. Here, when one of the four CPUs is specified, reference numerals 10A to 10D are used, and when an arbitrary CPU is indicated, reference numeral 10 is used. The CPUs 10 are point-to-point connected by QPI, HT or the like. Each CPU 10 has two CPU interconnect ports 20, and the four CPUs 10 </ b> A to 10 </ b> D are connected in a ring shape via the CPU interconnect port 20 and the CPU interconnect line 11.

  In each CPU 10, a CPU interconnect port (hereinafter sometimes referred to simply as a port) 20 is connected to a processor core 60 (see FIGS. 7, 9, and 10), a memory or an I / O via a CPU internal bus 50. An (Input / Output) device 70 (see FIGS. 7 and 9) is connected.

  The server 1 also has a physical partitioning function, and has system firmware (F / W; setting unit) 40 that performs settings in accordance with the partition configuration for the four CPUs 10A to 10D (see FIG. 2, see FIG. 4 and FIG. The system F / W 40 is configured to be operable by a user via the network 90. Further, on each chip of the CPU 10, a port mode selection register (hereinafter sometimes simply referred to as a register) 30 is attached to each of the two ports 20. The register 30 is for setting the operation mode of the port 20. The register 30 stores the physical partition configuration and FIGS. 4 and 5 by the system F / W 40 via the CPU internal bus 50 according to instructions from the user. First information or second information to be described later is set with reference to the reference.

  A port 20 in each CPU 10 is provided on the chip of the CPU 10, is connected to another CPU 10 via the CPU interconnect line 11, and communicates with another processing unit 10. In the port 20, a physical layer 21, a data link layer 22, a CPU interconnect upper layer 23, a PCIe-NTB port upper layer 24, and a multiplexer 25 are provided between the CPU interconnect line 11 and the CPU internal bus 50. .

  In the port 20, the physical layer 21, the data link layer 22, and the CPU interconnect upper layer 23 are existing configurations provided between the CPU interconnect line 11 and the CPU internal bus 50. In the port 20 of the first embodiment, a PCIe-NTB port upper layer 24 and a multiplexer 25 are added to the existing configuration.

  Here, the CPU interconnect upper layer 23 higher than the data link layer 22 functions as a first communication unit that performs communication with the CPU 10 belonging to the same partition as the CPU 10 to which the port 20 belongs. At this time, the CPU interconnect upper layer 23 causes the CPU interconnect port 20 to function as a port for executing communication for sharing the memory space for all memory accesses with other CPUs 10 belonging to the same partition.

  On the other hand, the PCIe-NTB port upper layer 24 is provided in parallel with the CPU interconnect upper layer 23 and functions as a second communication unit that communicates with the CPU 10 belonging to a different partition from the CPU 10 to which the port 20 belongs. At this time, the PCIe-NTB port upper layer 24 causes the CPU interconnect port 20 to function as an NTB (Non-Transparent Bridge) port of the PCI device. The NTB port executes communication explicitly instructed by the processor core 60 constituting the CPU 10 to which the port 20 belongs with another CPU 10 belonging to a different partition.

  A multiplexer 25 is provided between the upper layers 23 and 24 provided in parallel and the data link layer 22. The multiplexer 25 selects one of the CPU interconnect upper layer 23 and the PCIe-NTB port upper layer 24 according to the setting by the system F / W 40, and sends the other upper layer (communication unit) 23 or 24 to the other It functions as a selection unit that executes communication with the CPU 10.

  When the system F / W 40 receives an instruction about physical partitioning from the user via the network 90 (see YES route in step S1 in FIG. 3), the system F / W 40 conforms to the partition configuration according to the instruction to the four CPUs 10A to 10D. Settings are made (see step S2 in FIG. 3). At that time, the system F / W 40 sets the first information or the second information in the register 30 provided for each port 20 as shown in FIG. 4 or FIG. 5 (steps S3 and S4 in FIG. 3). ). Here, the first information is a value indicating that another CPU 10 connected to the port 20 belongs to the same partition, and is, for example, “0”. The second information is a value indicating that another CPU 10 connected to the port 20 belongs to a different partition, and is “1”, for example.

  When the first information “0” is set in the register 30 (see the “0” route in step S5 in FIG. 3), the multiplexer 25 switches the data path so as to select the CPU interconnect upper layer 23, and the port 20 Function as a CPU interconnect port (see step S6 in FIG. 3). On the other hand, when the second information “1” is set in the register 30 (see the “1” route in step S5 in FIG. 3), the multiplexer 25 sets the data path so as to select the PCIe-NTB port upper layer 24. The port 20 is switched to function as an NTB port (see step S7 in FIG. 3).

[1-2] Basic Operation and Effect of First Embodiment Next, the basic operation of the server 1 having the above-described configuration will be described with reference to FIGS. 4 and 5.
Here, FIG. 4 is a block diagram showing a one-partition state of the server 1. In the state shown in FIG. 4, since the four CPUs 10A to 10D belong to the same partition, the system F / W 40 sets the first information “0” in all the registers 30 in the four CPUs 10A to 10D. Accordingly, the multiplexer 25 switches the data path so that the CPU interconnect upper layer 23 is selected, and all the ports 20 function as normal CPU interconnect ports. Accordingly, communication for sharing the memory space for all memory accesses is executed with other CPUs 10 belonging to the same partition.

  FIG. 5 is a block diagram showing a two-partition state of the server 1. In the state shown in FIG. 5, the server 1 is divided into two physical servers by the physical versioning function, and the two CPUs 10A and 10B and the two CPUs 10C and 10D are assigned to different partitions # 0 and # 1, respectively. .

  In this case, the first information “0” is set by the system F / W 40 in the register 30 corresponding to the port 20 that mutually connects the CPUs 10A and 10B belonging to the same partition # 0. Along with this, the multiplexer 25 switches the data path so that the CPU interconnect upper layer 23 is selected at the port 20, and the port 20 functions as a normal CPU interconnect port. Therefore, communication for sharing the memory space for all memory accesses is executed between the CPUs 10A and 10B belonging to the same partition. The CPU 10C and 10D belonging to the same partition # 1 are set and selected in the same manner as described above, and communication for sharing the memory space for all memory accesses is executed between the CPUs 10C and 10D.

  On the other hand, the second information “1” is set by the system F / W 40 in the register 30 corresponding to the port 20 that interconnects the CPUs 10A and 10D belonging to different partitions # 0 and # 1. Accordingly, the multiplexer 25 switches the data path so that the PCIe-NTB port upper layer 24 is selected in the port 20, and the port 20 functions as an NTB port. In other words, the CPU interconnect port 20 that is normally unused can be used as the NTB port. Therefore, communication explicitly instructed by the processor core 60 constituting the CPU 10A or 10D is executed between the CPUs 10A and 10D belonging to different partitions # 0 and # 1 via the NTB port. The CPU 10B and 10C belonging to the different partitions # 0 and # 1 are set and selected in the same manner as described above, and the communication explicitly instructed by the processor core 60 constituting the CPU 10B or 10C is executed between the CPUs 10B and 10C. .

  As described above, a communication path is established between the port 20 and another CPU 10 by selecting the PCIe-NTB port upper layer 24 and causing the unused port 20 to function as an NTB port to execute communication. . The communication path thus established can be used as at least one of a memory mirroring path and a failover path.

As described above with reference to FIG. 14, when the servers 100 are used by being divided into a plurality of physical servers partitioning capabilities, two CPU101 separate partition # 0 in the server 100, # Assigned to 1. In this case, the point-to-point CPU interconnect (port 102 and line 103) that connects the two CPUs 101 is usually unused.

  On the other hand, according to the server 1 of the first embodiment, the CPU interconnect port 20 that has conventionally been unused when connecting CPUs belonging to different partitions functions as an NTB port of the PCI device. . Thereby, a communication path is established between the CPUs 10A and 10D belonging to different partitions # 0 and # 1, or between the CPUs 10B and 10C, and is used as a memory mirroring path or a failover path between different partitions. Therefore, it is possible to efficiently realize communication between the CPUs 10 and 10 belonging to different partitions by using a CPU interconnect (port 20 and line 11) that has not been used conventionally.

  As described above, the NTB port described above fulfills a function for sharing data between two different partitions or between hosts in the PCI specification, and has a memory-mapped space in the port. It is a port that can exchange data between partitions and between hosts using.

[1-3] Detailed Configuration and Operation of PCIe-NTB Port Upper Layer (Second Communication Unit) Next, detailed configuration of PCIe-NTB port upper layer (second communication unit) 24 for realizing such an NTB port (internal) Structure) and operation will be described with reference to FIGS.

  FIG. 6 is a block diagram showing a detailed configuration of the PCIe-NTB port upper layer 24. As shown in FIG. 6, the PCIe-NTB port upper layer 24 includes a CPU internal bus interface (I / F) 241, a PCIe-NTB transaction layer 242, and a CPU interconnect data link interface (I / F) 243. .

  The CPU internal bus I / F (first interface) 241 is connected to the processor core 60 via the CPU internal bus 50. The CPU internal bus I / F 241 performs cache protocol control and converts between data (DATA and CMD (command)) on the processor core 60 side and data conforming to PCIe use on the transaction layer 242 side. In other words, the CPU internal bus I / F 241 functions to divide / combine the cache line size data on the processor core 60 side into the PCIe payload size data on the transaction layer 242 side. The CPU internal bus I / F 241 functions to divide / combine PCIe payload size data on the transaction layer 242 side into cache line size data on the processor core 60 side.

  The CPU internal bus I / F 241 includes buffers 241a to 241d. The buffer 241a receives the command CMD (RX) from the CPU internal bus 50 (processor core 60), and the converted command is output from the buffer 241a to the transaction layer 242. Similarly, the buffer 241b receives the data DATA (RX) from the CPU internal bus 50 (processor core 60), and the converted data is output from the buffer 241b to the transaction layer 242. The buffer 241 c receives a command from the transaction layer 242, and the converted command CMD (TX) is transmitted from the buffer 241 c to the processor core 60 or the memory 70 via the CPU internal bus 50. Similarly, the buffer 241 d receives data from the transaction layer 242, and the converted data DATA (TX) is transmitted from the buffer 241 d to the processor core 60 or the memory 70 via the CPU internal bus 50.

  The CPU interconnect data link I / F (second interface) 243 is connected to another CPU 10 via the multiplexer 25, the data link layer 22, the physical layer 21, and the CPU interconnect line 11. The CPU interconnect data link I / F 243 performs division / reconfiguration of TLP (Transaction Layer Packet) generated in the transaction layer 242. Since the packet length of the TLP and the maximum packet length defined by the CPU interconnect do not always match, the transaction layer 242 performs TLP division or combination as necessary. The packet division and encapsulation of TLP will be described later with reference to FIG. 11 in item [1-6].

  The CPU interconnect data link I / F 243 includes buffers 243a and 243b. The buffer 243a receives the TLP from the transaction layer 242, and the divided or combined packet is transmitted from the buffer 243a to the other CPU 10 via the multiplexer 25, the data link layer 22, the physical layer 21, and the CPU interconnect line 11. Is done. The buffer 243b receives a packet from another CPU 10, and the TLP obtained by the division or combination is sent from the buffer 243b to the transaction layer 242.

  The PCIe-NTB transaction layer 242 is disposed between the CPU internal bus I / F 241 and the CPU interconnect data link I / F 243, and performs TLP generation / disassembly and address conversion. Therefore, the transaction layer 242 includes a PCI configuration register 242a, a memory mapped I / O (MMIO) register 242b, an address conversion unit 242c, a TLP generation unit 242d, a TLP disassembly unit 242e, and an address conversion unit 242f. .

  The PCI configuration register 242a is a register set conforming to the use of PCIe.

The MMIO register 242b has the following six types of registers 242b-1 to 242b-6, which will be described later with reference to FIGS. 7, 9, and 10 in the items [1-4] and [1-5]. .
Base address register (BAR; address conversion register) 242b-1
Limit address register (LAR; address conversion register) 242b-2
Transmission doorbell register (transmission interrupt register) 242b-3
Reception doorbell register (reception interrupt register) 242b-4
Transmission scratch pad register (transmission exchange information register) 242b-5
Reception scratch pad register (reception exchange information register) 242b-6

The address conversion unit 242c determines the data to be transmitted to the other CPU 10 based on the command from the processor core 60 and the value of the MMIO register 242b as will be described later with reference to FIGS. Perform address translation.
The TLP generation unit 242d generates a TLP including data to be transmitted to another CPU 10 based on the information obtained by the address conversion unit 242c, and outputs the TLP to the CPU interconnect data link I / F 243.

The TLP disassembling unit 242e obtains address information and received data by disassembling (disassembling) the TLP received from another CPU 10, and outputs the address information to the address converting unit 242f, while receiving the received data in the CPU internal bus I / F241 (buffer 241d) for output.
The address converting unit 242f performs address conversion on the address information received from the TLP dismantling unit 242e based on the value of the MMIO register 242b as described later in the item [1-4] with reference to FIGS. The result is output as a command to the CPU internal bus I / F 241 (buffer 241c).

Here, the operation of the PCIe-NTB port upper layer 24 configured as described above will be briefly described.
When the PCIe-NTB port upper layer 24 transmits data to another CPU 10 using the port 20 as an NTB port, first, the CPU internal bus I / F 241 converts the data to the other CPU 10 into data conforming to the PCI specification. Convert. Then, the transaction layer 242 generates a TLP based on data from the CPU internal bus I / F 241. Thereafter, the CPU interconnect data link I / F 243 converts the TLP from the transaction layer 242 into a packet conforming to the communication specification (CPU interconnect) between the port 20 and the other CPU 10.

  On the other hand, when the PCIe-NTB port upper layer 24 receives data from another CPU 10 using the port 20 as an NTB port, the CPU interconnect data link I / F 243 first converts a packet from the other CPU 10 into a TLP. . Then, the transaction layer 242 generates data conforming to the PCI specification based on the TLP from the CPU interconnect data link I / F 243. Thereafter, the CPU internal bus I / F 241 converts data from the transaction layer 242 into data for the processor core 60 or the memory (I / O device) 70.

[1-4] Address Conversion Next, the address conversion function by the address conversion units 242c and 242f of the PCIe-NTB transaction layer 242 will be described with reference to FIGS. Since each physical partition # 0, # 1 has a separate memory space, the NTB port generally requires an address translation function. The address conversion function is realized by the address conversion units 242c and 242f in the PCIe-NTB transaction layer 242 based on the set value of the MMIO register 242b (registers 242b-1 and 242b-2).

  First, referring to FIG. 7, using the above address conversion function, write data is transferred from the processor core 60 of partition # 0 to the memory (I / O device) 70 of partition # 1 (Write transfer). The procedure (A1) to (A4) will be described. FIG. 7 is a block diagram illustrating an address conversion function by the address conversion units 242c and 242f of the transaction layer 242 shown in FIG. FIG. 7 shows transfer of write data from the processor core 60 of the CPU 10A belonging to the partition # 0 to the memory 70 of the CPU 10D belonging to the partition # 1.

  (A1) The processor core 60 of the partition # 0 on the transmission side writes the write data to be sent to the memory 70 of the partition # 1 with the BAR 242b-1 of the NTB of the partition # 0 as the destination. The address at this time is A. This address A is an address belonging to the memory space of the partition # 0.

  (A2) The address conversion unit 242c in the PCIe-NTB port upper layer 24 of the partition # 0 confirms that the write address is within the range between the lower limit value and the upper limit value, and performs address conversion. At this time, assuming that the set value (lower limit value) of BAR 242b-1 is B, the address conversion unit 242c confirms that the address A is not less than the set value B and not more than the upper limit value set in the LAR 242b-2. Then, the value indicated by AB is output as the converted address. The translated address AB is an address belonging to the CPU interconnect address space.

  (A3) The address converter 242f in the PCIe-NTB port upper layer 24 of the partition # 1 that has received the data converts the address A-B of the received data into an address in the memory space of the partition # 1. At this time, if the setting value (lower limit; target address (TA)) of BAR 242b-1 of partition # 1 is C, the converted address is A−B + C, and this address A−B + C is the memory of partition # 1. An address belonging to a space.

  (A4) The PCIe-NTB port upper layer 24 of the partition # 1 performs DMA (Direct Memory Address) with respect to the memory (I / O device) 70 using the A-B + C as an address via the CPU internal bus 50. .

  Next, referring to FIG. 8 and FIG. 9, a read access (Read transfer) from the processor core 60 of the partition # 0 to the memory (I / O device) 70 of the partition # 1 is performed using the address conversion function. The procedures (B1) to (B6) will be described. 8 is a diagram showing the contents of the transaction ID (identification), and FIG. 9 is a diagram illustrating the tag conversion / inverse conversion function and the transaction ID conversion / inverse conversion function by the address conversion unit 242f of the transaction layer 242 shown in FIG. FIG. FIG. 9 shows a read access from the processor core 60 of the CPU 10A belonging to the partition # 0 to the memory 70 of the CPU 10D belonging to the partition # 1.

  Also in the case of read access (Read transfer), address conversion for the read target address is performed in the same manner as the procedures (A1) to (A4) described above with reference to FIG. However, in the case of read access, read data is transferred from partition # 1 to partition # 0. The transaction ID of the read data has a one-to-one correspondence with the transaction ID of the read request so that the processor core 60 of the requesting partition # 0 can identify which request is the response from the partition # 1. It is associated. As shown in FIG. 8, the transaction ID includes a requester ID and a tag. The requester ID (RID) is read request source information (ID of the processor core 60) including a bus number, a device number, and a function number. The tag is a sequential number generated for each request by the processor core 60, for example. The requester ID (RID) and tag conversion procedure will be described below with reference to FIG. 9 and procedures (B1) to (B6).

  (B1) The processor core 60 of the partition # 0 on the transmission side issues a read request to the PCIe-NTB port upper layer 24 of the partition # 0 with its own requester ID.

  (B2) The PCIe-NTB port upper layer 24 of the partition # 0 that has received the read request uses the address conversion unit 242c to set a tag of the received transaction ID so that the read request received by the partition # 0 can be uniquely identified. Perform tag conversion to be replaced. Thereby, the PCIe-NTB port upper layer 24 can uniquely recognize the correspondence between the read request and the read data by using only the tag. At this time, the original transaction ID before tag conversion and the tag value after tag conversion are associated and stored in the first table (not shown). Thereafter, the PCIe-NTB port upper layer 24 of the partition # 0 transfers the read request including the transaction ID after the tag conversion to the PCIe-NTB port upper layer 24 of the partition # 1.

  (B3) The address conversion unit 242f in the PCIe-NTB port upper layer 24 of the partition # 1 that has received the read request partitions the requester ID (the ID of the processor core 60 of the partition # 0) among the transaction IDs of the received read request. RID conversion to be replaced with the requester ID on the # 1 side is performed. The requester ID on the partition # 1 side is used when a response (read data / completion notification) from the memory 70 to the read request returns to the PCIe-NTB port upper layer 24 of the partition # 1. The At this time, the transaction ID before RID conversion is stored in the second table (not shown) in the PCIe-NTB port upper layer 24 of the partition # 1.

  (B4) Read data (completion notice) is returned from the memory (I / O device) 70 of the partition # 1 to the PCIe-NTB port upper layer 24.

  (B5) The address conversion unit 242c in the PCIe-NTB port upper layer 24 of the partition # 1 searches for the transaction ID before RID conversion in the second table using the tag as a key, and based on the requester ID of the received read data. RID reverse conversion is performed to replace the requester ID. Thereafter, the PCIe-NTB port upper layer 24 transfers the read data including the transaction ID after RID reverse conversion to the PCIe-NTB port upper layer 24 of the partition # 0.

  (B6) The address conversion unit 242f in the PCIe-NTB port upper layer 24 of the partition # 0 searches the first table using the tag included in the transaction ID of the received read data as a key, and based on the tag of the transaction ID Perform tag reverse conversion to replace the tag. Thereafter, the PCIe-NTB port upper layer 24 transfers the read data including the transaction ID after the tag reverse conversion to the processor core 60 via the CPU internal bus 50.

[1-5] Doorbell Function and Scratch Pad Function Next, the doorbell function and the scratch pad function by the PCIe-NTB transaction layer 242 will be described with reference to FIG. FIG. 10 is a block diagram for explaining a doorbell function and a scratch pad function by the transaction layer 242 shown in FIG.

  In general, an NTB port (PCIe-NTB transaction layer 242) has a doorbell function and a scratchpad function for communication between CPUs 10 (for example, CPUs 10A and 10D) belonging to different partitions # 0 and # 1. It has.

[1-5-1] Doorbell Function The doorbell function is a function for raising an interrupt (INT) from the processor core 60 of one partition # 0 or # 1 to the processor core 60 of the other partition # 1 or # 0. It is. The processor core 60 on the transmission side executes writing to the transmission doorbell register 242b-3 of the PCIe-NTB transaction layer 242 on the transmission side by PIO (Programmed I / O). Upon receiving this write, the PCIe-NTB transaction layer 242 on the transmission side generates a message packet (Msg; TLP) and transmits it to the PCIe-NTB transaction layer 242 on the reception side. The message packet at this time is uniquely defined using a mechanism of Vendor Defined Message of the PCIe specification. When receiving the message packet related to the doorbell function, the PCIe-NTB transaction layer 242 on the receiving side that has received the message packet determines that the content received by the message packet is a doorbell. Then, the PCIe-NTB transaction layer 242 on the reception side executes writing to the reception doorbell register 242b-4, and notifies the reception-side processor core 60 of an interrupt (INT).

  That is, the PCIe-NTB transaction layer 242 (NTB port) generates a TLP indicating an interrupt and transfers it to another CPU when an interrupt instruction for the other CPU 10 is written from the processor core 60 to the transmission doorbell register 242b-3. To do. On the other hand, the PCIe-NTB transaction layer 242 (NTB port) issues an interrupt instruction to the processor core 60 when an interrupt instruction for the processor core 60 from another CPU 10 is written to the reception doorbell register 242b-4.

[1-5-2] Scratch Pad Function The scratch pad function is a function used by the processor cores 60 of the partitions # 0 and # 1 to exchange information of about several bytes. The sending processor core 60 writes the exchange information in the sending scratch pad register 242b-5 of the sending PCIe-NTB transaction layer 242 by PIO. The transmitting side PCIe-NTB transaction layer 242 in which the exchange information is written generates a message packet (Msg; TLP) and transmits it to the receiving side PCIe-NTB transaction layer 242. The message packet at this time is uniquely defined using a mechanism of Vendor Defined Message of the PCIe specification. Upon receiving the message packet related to the scratch pad function, the receiving PCIe-NTB transaction layer 242 that has received the message packet determines that the content received by the message packet is a scratch pad. Then, the PCIe-NTB transaction layer 242 on the receiving side reflects the contents (exchange information) of the message packet in the receiving scratch pad register 242b-6. The processor core 60 on the receiving side can know the exchange information from the processor core 60 on the transmitting side by reading the contents of the receiving scratch pad register 242b-6 by PIO.

  That is, the PCIe-NTB transaction layer 242 (NTB port) includes the exchange information when the exchange information to be exchanged with the other CPU 10 is written from the processor core 60 to the transmission scratchpad register 242b-5 of the transaction layer 242. A TLP is generated and transferred to another CPU 10. On the other hand, the PCIe-NTB transaction layer 242 (NTB port), when exchange information to be exchanged with the processor core 60 from another CPU 10 is written in the receiving scratchpad register 242b-6, an inquiry (PIO) from the processor core 60 In response to this, the processor core 60 is notified of the exchange information of the scratch pad register 242b-6 for reception.

[1-6] TLP Division and Encapsulation Next, packet division and encapsulation by the CPU interconnect data link I / F (second interface) 243 will be described with reference to FIG. FIG. 11 is a diagram for explaining packet division and encapsulation by the CPU interconnect data link I / F 243 shown in FIG.

  The CPU interconnect data link I / F 243 of the transmitting NTB port performs the following processing. First, the CPU interconnect data link I / F 243 receives the PCIe TLP from the upstream PCIe-NTB transaction layer 242. The format of TLP is as shown in the upper part of FIG. The CPU interconnect data link I / F 243 divides the received TLP according to the packet size (packet length) of the CPU interconnect, or performs padding, and performs one or more matching to the packet size of the CPU interconnect. Generate a packet.

  FIG. 11 shows packet division and encapsulation when the received TLP is larger than the packet size of the CPU interconnect. That is, first, the CPU interconnect data link I / F 243 divides the TLP shown in the upper part of FIG. 11 into two packets as shown in the middle part of FIG. The size of one (left side) packet is the same as the packet size of the CPU interconnect. However, since the size of the other (right side) packet is smaller than the packet size of the CPU interconnect, the size of the other packet is the same as the packet size of the CPU interconnect as shown on the right side in the middle of FIG. Padding is performed so that

  Further, in the example shown in FIG. 11, the CPU interconnect data link I / F 243 has a header (Haeder) of the data link layer of the CPU interconnect before and after each of the divided packets as shown in the lower part of FIG. And CRC (Cyclic Redundancy Check) is added. Thereby, each packet is encapsulated. On the other hand, the CPU interconnect data link I / F 243 of the receiving NTB port reconstructs the PCIe TLP from the data link layer packet in a process reverse to the packet splitting and encapsulation process of the transmitting NTB port described above. To do.

[2] Information processing apparatus (server) of the second embodiment
Next, the configuration of the information processing apparatus (server) 1 ′ according to the second embodiment will be described with reference to FIGS. FIG. 12 is a block diagram showing the main configuration of the server 1 ′ of the second embodiment, and FIG. 13 shows the detailed configuration of the PCIe-NTB port upper layer (second communication unit) 24 ′ shown in FIG. It is a block diagram. In the figure, the same reference numerals as those already described indicate the same or substantially the same parts, and the description thereof will be omitted.

  The CPU 10 in the server 1 ′ of the second embodiment has a PCIe port 81 as shown in FIG. When the CPU 10 has the PCIe port 81 as described above, the CPU 10 includes a PCIe root complex 80 between the CPU internal bus 50 and the PCIe port 81. In this case, the CPU internal bus I / F 241 according to the first embodiment may be mounted in the PCIe route complex 80 and shared with other PCIe ports 81.

That is, in the second embodiment, as shown in FIG. 12, the connection destination on the upstream side of the PCIe-NTB port upper layer 24 ′ is not the CPU internal bus 50 but the PCIe root complex 80. It is different from the form (the PCIe-NTB port upper layer 24).
In the second embodiment, the PCIe-NTB port upper layer 24 'is connected to the PCIe route complex 80, as shown in FIG. 12, and therefore, as shown in FIG. The block of the bus I / F 241 is not necessary. Other blocks (the PCIe-NTB transaction layer 242 and the CPU interconnect data link I / F 243) are the same as the PCIe-NTB port upper layer 24 of the first embodiment in the PCIe-NTB port upper layer 24 'of the second embodiment. Similarly provided.

  Thus, in the server 1 ′ of the second embodiment, when the CPU 10 has the PCIe port 81 and the PCIe route complex 80, the CPU internal bus I / F (first interface) in the first embodiment is set by the PCIe route complex 80. The function 241 is realized. Therefore, according to server 1 'of 2nd Embodiment, the effect similar to 1st Embodiment can be acquired with a simpler structure.

[3] Others While preferred embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and changes can be made within the scope of the present invention. It can be changed and implemented.
In addition, although embodiment mentioned above demonstrated the case where four CPU10A-10D was provided in the one server 1, this invention is not limited to this.

  Server including the above-described CPU interconnect upper layer (first communication unit) 23, PCIe-NTB port upper layer (second communication unit) 24, 24 ', multiplexer (selection unit) 25, and system F / W (setting unit) 40 All or some of the various functions 1 and 1 'are realized by a computer (including a CPU, an information processing apparatus, and various terminals) executing a predetermined program.

  The program is, for example, a flexible disk, CD (CD-ROM, CD-R, CD-RW, etc.), DVD (DVD-ROM, DVD-RAM, DVD-R, DVD-RW, DVD + R, DVD + RW, etc.), Blu-ray Disc And the like recorded in a computer-readable recording medium. In this case, the computer reads the program from the recording medium, transfers it to the internal storage device or the external storage device, and uses it.

Here, the computer is a concept including hardware and an OS (operating system), and means hardware operating under the control of the OS. Further, when the OS is unnecessary and the hardware is operated by the application program alone, the hardware itself corresponds to the computer. The hardware includes at least a microprocessor such as a CPU and means for reading a computer program recorded on a recording medium. The program includes a program code for causing the computer as described above to realize the various functions of the servers 1 and 1 'described above. Also, some of the functions may be realized by the OS instead of the application program.
[4] Notes
Regarding the embodiments including the above embodiments, the following additional notes are disclosed.
(Appendix 1)
A plurality of processing units;
A setting unit configured to perform settings in accordance with a partition configuration for the plurality of processing units,
Each of the plurality of processing units has a port connected to another processing unit and communicating with the other processing unit,
The port is
A first communication unit for communicating with a processing unit belonging to the same partition as the processing unit to which the port belongs;
A second communication unit for communicating with a processing unit belonging to a different partition from the processing unit to which the port belongs;
A selection unit that selects one of the first communication unit and the second communication unit according to the setting by the setting unit, and causes the selected communication unit to execute communication with the other processing unit; An information processing apparatus comprising:
(Appendix 2)
Each of the plurality of processing units is
The setting unit further includes a register configured to set first information indicating that the other processing unit belongs to the same partition or second information indicating that the other processing unit belongs to the different partition. And
The selection unit selects the first communication unit when the first information is set in the register, and selects the second communication unit when the second information is set in the register. The information processing apparatus according to appendix 1, wherein:
(Appendix 3)
The first communication unit performs communication for sharing the memory space for all memory accesses with the other processing unit belonging to the same partition. The information processing apparatus described.
(Appendix 4)
The second communication unit performs communication explicitly instructed by the processor core forming the processing unit to which the port belongs with the other processing unit belonging to the different partition. The information processing apparatus according to any one of 1 to appendix 3.
(Appendix 5)
The information processing apparatus according to appendix 4, wherein the port functions as a non-transparent bridge (NTB) port of a peripheral component interconnect (PCI) device when the second communication unit performs communication. .
(Appendix 6)
The second communication unit is
A first interface connected to the processor core;
A second interface connected to the other processing unit;
An NTB transaction layer disposed between the first interface unit and the second interface;
When the second communication unit uses the port as the NTB port to transmit data to the other processing unit, the first interface converts the data to the other processing unit into data conforming to the PCI specification. The NTB transaction layer generates a TLP (Transaction Layer Packet) based on the data from the first interface, and the second interface converts the TLP from the NTB transaction layer into the port and the other processing unit. While converting the packet to comply with the communication specifications between
When the second communication unit uses the port as the NTB port to receive data from the other processing unit, the second interface converts a packet from the other processing unit into a TLP, and the NTB transaction layer Generates data conforming to the PCI specification based on the TLP from the second interface, and the first interface converts the data from the NTB transaction layer into data to the processor core or memory. The information processing apparatus according to appendix 5.
(Appendix 7)
Each of the plurality of processing units has a PCI port and a route complex,
The information processing apparatus according to appendix 6, wherein the first interface of the second communication unit is configured by the route complex.
(Appendix 8)
The second communication unit is
When an interrupt instruction from the processor core to the other processing unit is written in the transmission interrupt register of the NTB transaction layer, a TLP instructing an interrupt is generated and transferred to the other processing unit.
The information according to appendix 6 or appendix 7, wherein an interrupt instruction is issued to the processor core when an interrupt instruction for the processor core is written from the other processing unit to the reception interrupt register of the NTB transaction layer. Processing equipment.
(Appendix 9)
The second communication unit is
When the exchange information to be exchanged with the other processing unit is written from the processor core to the exchange information register for transmission of the NTB transaction layer, a TLP including the exchange information is generated and transferred to the other processing unit.
When the exchange information to be exchanged with the processor core is written from the other processing unit to the reception exchange information register of the NTB transaction layer, the exchange information of the reception exchange information register is received in response to an inquiry from the processor core. The information processing apparatus according to any one of appendix 6 to appendix 8, wherein the processor core is notified.
(Appendix 10)
The path established between the port and the other processing unit when the second communication unit performs communication functions as at least one of a memory mirroring path and a failover path. The information processing apparatus according to any one of 1 to 9.
(Appendix 11)
In an information processing apparatus including a plurality of processing units connected to another processing unit and having a port for communicating with the other processing unit, a control method for setting the plurality of processing units according to a partition configuration There,
In the port, a first communication unit that communicates with a processing unit that belongs to the same partition as the processing unit to which the port belongs, and a second communication unit that communicates with a processing unit that belongs to a partition different from the processing unit to which the port belongs And comprising
An information processing apparatus comprising: selecting one of the first communication unit and the second communication unit according to the setting, and causing the selected communication unit to communicate with the other processing unit. Control method.
(Appendix 12)
Each of the plurality of processing units is set with first information indicating that the other processing unit belongs to the same partition or second information indicating that the other processing unit belongs to the different partition. With a register
When the first information is set in the register, the first communication unit is selected, and when the second information is set in the register, the second communication unit is selected. The control method of the information processing apparatus according to appendix 11.
(Appendix 13)
The supplementary note 11 or the supplementary note 12 is characterized in that the first communication unit performs communication for sharing the memory space for all memory accesses with the other processing unit belonging to the same partition. A control method of the information processing apparatus described.
(Appendix 14)
The second communication unit is caused to execute communication explicitly instructed by a processor core constituting the processing unit to which the port belongs to the other processing unit belonging to the different partition. The control method of the information processing apparatus according to any one of 11 to appendix 13.
(Appendix 15)
15. The information processing apparatus according to appendix 14, wherein the second communication unit causes the port to function as an NTB (Non-Transparent Bridge) port of a PCI (Peripheral Components Interconnect) device by performing communication. Control method.
(Appendix 16)
Note that the second communication unit causes communication to cause a path established between the port and the other processing unit to function as at least one of a memory mirroring path and a failover path. The control method of the information processing apparatus according to any one of 11 to appendix 15.

1,1 'Information processing device (server)
10, 10A to 10D CPU (processing unit)
11 CPU interconnect line 20 CPU interconnect port (port, NTB port)
21 Physical layer 22 Data link layer 23 CPU interconnect upper layer (first communication unit)
24, 24 'PCIe-NTB port upper layer (second communication unit)
241 CPU internal bus interface (first interface)
241a to 241d buffer 242 PCIe-NTB transaction layer (NTB transaction layer)
242a PCI configuration register 242b Memory mapped I / O (MMIO) register 242b-1 Base address register (BAR; register for address conversion)
242b-2 Limit address register (LAR; register for address conversion)
242b-3 Transmission doorbell register (transmission interrupt register)
242b-4 Reception doorbell register (Reception interrupt register)
242b-5 Scratch pad register for transmission (exchange information register for transmission)
242b-6 Receive Scratch Pad Register (Receive Exchange Information Register)
242c Address conversion unit 242d TLP generation unit 242e TLP disassembly unit 242f Address conversion unit 243 CPU interconnect data link interface (second interface)
243a, 243b buffer 25 multiplexer (selection unit)
30 Port mode selection register (register)
40 System firmware (setting part)
50 CPU internal bus 60 Processor core 70 Memory or I / O device 80 PCIe route complex (first interface)
81 PCIe port 90 network

Claims (8)

A plurality of processing units;
A setting unit configured to perform settings in accordance with a partition configuration for the plurality of processing units,
Each of the plurality of processing units has a port connected to another processing unit and communicating with the other processing unit,
The port is
A first communication unit for communicating with a processing unit belonging to the same partition as the processing unit to which the port belongs;
A second communication unit for communicating with a processing unit belonging to a different partition from the processing unit to which the port belongs;
A selection unit that selects one of the first communication unit and the second communication unit according to the setting by the setting unit, and causes the selected communication unit to execute communication with the other processing unit; An information processing apparatus comprising: Each of the plurality of processing units is
The setting unit further includes a register configured to set first information indicating that the other processing unit belongs to the same partition or second information indicating that the other processing unit belongs to the different partition. And
The selection unit selects the first communication unit when the first information is set in the register, and selects the second communication unit when the second information is set in the register. The information processing apparatus according to claim 1, wherein:   The said 1st communication part performs communication for sharing a memory space with respect to all the memory accesses between the said other process parts which belong to the said same partition, The Claim 1 or Claim characterized by the above-mentioned. 2. The information processing apparatus according to 2.   The second communication unit performs communication explicitly instructed by a processor core forming a processing unit to which the port belongs to the other processing unit belonging to the different partition. The information processing apparatus according to any one of claims 1 to 3.   5. The information processing according to claim 4, wherein when the second communication unit performs communication, the port functions as a non-transparent bridge (NTB) port of a peripheral component interconnect (PCI) device. apparatus. The second communication unit is
A first interface connected to the processor core;
A second interface connected to the other processing unit;
An NTB transaction layer disposed between the first interface unit and the second interface;
When the second communication unit uses the port as the NTB port to transmit data to the other processing unit, the first interface converts the data to the other processing unit into data conforming to the PCI specification. The NTB transaction layer generates a TLP (Transaction Layer Packet) based on the data from the first interface, and the second interface converts the TLP from the NTB transaction layer into the port and the other processing unit. While converting the packet to comply with the communication specifications between
When the second communication unit uses the port as the NTB port to receive data from the other processing unit, the second interface converts a packet from the other processing unit into a TLP, and the NTB transaction layer Generates data conforming to the PCI specification based on the TLP from the second interface, and the first interface converts the data from the NTB transaction layer into data to the processor core or memory. The information processing apparatus according to claim 5. Each of the plurality of processing units has a PCI port and a route complex,
Wherein the first interface of the second communication unit may be constituted by the root complex, information processing equipment according to claim 6. In an information processing apparatus including a plurality of processing units connected to another processing unit and having a port for communicating with the other processing unit, a control method for setting the plurality of processing units according to a partition configuration There,
In the port, a first communication unit that communicates with a processing unit that belongs to the same partition as the processing unit to which the port belongs, and a second communication unit that communicates with a processing unit that belongs to a partition different from the processing unit to which the port belongs And comprising
An information processing apparatus comprising: selecting one of the first communication unit and the second communication unit according to the setting, and causing the selected communication unit to communicate with the other processing unit. control how.

Images