JP2017511532A - Method for configuring a software defined PCI Express (PCI-E) switch - Google Patents

Abstract

The present invention relates generally to the technical fields of networking and computer architecture, and more particularly to techniques for configuring a system that includes a software defined network (SDN) box, at least one upstream node, and a plurality of downstream nodes. is there. The nodes can be divided into several groups. Each group has a single upstream node and at least one downstream node. All packets between upstream and downstream nodes are forwarded by the SDN box. The SDN box acts as a “PCI-E switch”, interconnects the upstream node with each downstream node, and forwards the encapsulated packet (internally a PCI-E packet). Throughout the entire communication process, the upstream node treats the downstream node as a normal local PCI-E device.

Description

  The present invention relates generally to the technical fields of networking and computer architecture, and more particularly to techniques for constructing a system consisting of a software defined network (SDN) box, an upstream node, and a downstream node.

  PCI-E (Peripheral Device Interconnect Express) is a third generation high performance I / O bus used to interconnect peripheral devices in a computer system. PCI-E uses the same usage model as the previous generation I / O bus called PCI. PCI-E supports well-known transactions such as memory read / write, I / O read / write, and configuration read / write transactions. Existing OS and device drivers can work without modification on PCI-E systems (for more information on PCI-E, see Ravi Budruk, Don Anderson, Tom Shanley, Addison-Wesley Professional, 2004 PCI Express see system architecture).

  Because there is not much space in the server, the number of PCI-E slots is usually limited. ExpEther (Express Ethernet, see http://www.expether.org/) has been proposed to address the above issues. ExpEther extends PCI-E over Ethernet. The PCI-E packet is encapsulated in a PCI-E-over-Ethernet packet at the sender side of the PCI-E packet, then forwarded by the Ethernet switch and into the PCI-E packet when forwarded to the destination. Decapsulated (open capsule) (see Japanese Publication Patent Application No. 2007-219873A).

  Certain types of packets are broadcast to act as keep alive messages in ExpEther. There is a timeout value associated with it. If the timeout expires and there is still no such received keepalive packet, it is assumed that the upstream node has already been shut down. Once a failure is detected, the connected downstream node should be transferred to another available upstream node as soon as possible.

JP 2007-219873

"PCI Express system architecture", by Ravi Budruk, Don Anderson, Tom Shanley, Addison- Wesley Professional, 2004 Express Ethernet, http://www.expether.org/ OpenFlow, https://www.opennetworking.org/sdn-resources/onf-specifications/openflow

  With the current ExpEther, it is difficult to achieve high-speed failure detection. The faster the failure detection, the shorter the keep-alive packet time-out value, and thus the more broadcast traffic. The more broadcast traffic, the greater the processing capacity of the network device.

  The present invention is proposed to solve the above-described problems of high-speed failure detection and ExpEther handover. The present invention provides a method for configuring a system based on PCI-E, Ethernet and SDN. The proposed system consists of an upstream node, a downstream node and an SDN box.

  In the proposed system of the present invention, there is an encapsulation / decapsulation module at both the upstream and downstream nodes. The upstream node (which can consist of a CPU, memory, hard disk and various I / O devices) acts as a computer system, and the downstream node acts as a PCI-E device, which are connected by an SDN box. The upstream node accesses the downstream node by a PCI-E packet. For both upstream and downstream nodes, the communication process is as follows. When transmitting a PCI-E packet, the PCI-E packet is encapsulated with a specific packet header and transmitted over a network (eg, Ethernet, but not limited to it). When receiving an encapsulated packet, it removes the specific packet header and decapsulates it into a PCI-E packet. The SDN box acts as a “PCI-E switch” and interconnects the upstream and downstream nodes via a specific network to transfer the encapsulated packets.

  In the proposed system of the present invention, when an upstream node is shut down, a notification is delivered to the SDN box (the SDN box is openFlow (https://www.opennetworking.org/sdn-resources/ (See onf-specifications / openflow), but that is not the case. When OpenFlow is used, when an upstream node goes down, an OFPPS_LINK_DOWN message will be sent from the OpenFlow switch to the OpenFlow controller) . Moreover, a PCI-E routing table is maintained on the SDN box, and handover can be achieved by changing the group ID of the connected downstream node.

  To illustrate the foregoing and other illustrative purposes, aspects, and advantages, the following detailed description of exemplary embodiments of the invention will be used with reference to the drawings.

FIG. 2 is a block diagram illustrating one embodiment of a system architecture of a computer system having an SDN box, at least one upstream node and at least one downstream node. A block diagram illustrating one embodiment of one of the possible system architectures based on an SDN box consisting of at least one software defined switch (SD switch) and at least one software defined switch controller (SDSW controller) It is. A block diagram illustrating one embodiment of one of the possible system architectures based on an SDN box consisting of at least one software defined switch (SD switch) and at least one software defined switch controller (SDSW controller) It is. FIG. 2 is a block diagram illustrating one embodiment of an upstream node system architecture. FIG. 2 is a block diagram illustrating one embodiment of a downstream node system architecture. FIG. 2 is a block diagram illustrating one embodiment of a system architecture for software defined (SD) switches. It is a block diagram which shows one Embodiment of the system architecture of the controller (SDSW controller) of a software definition switch. 6 is a table illustrating one embodiment of possible packet formats for DEVINFO packets. FIG. 11 is a sequence diagram of memory read / write between an upstream node, an SDN box, and a downstream node. 6 is a flow diagram illustrating one embodiment of how an upstream node or downstream node sends a PCI-E packet. 3 is a flow diagram illustrating one embodiment of how an upstream node or downstream node receives a PCI-E packet. 2 is a flow diagram illustrating one embodiment of how an SD switch processes an Ethernet packet. 2 is a flow diagram illustrating one embodiment of how the SDSW controller sets up a master transaction layer packet (TLP) routing table during system startup and communicates with the SD switch during the packet transfer process. FIG. 6 is a table illustrating one embodiment of possible data structures for a TLP routing table used as a slave TLP routing table for an SD switch and a master TLP routing table for an SDSW controller when the underlying network is Ethernet. It is a sequence diagram of failure detection and handover between an upstream node, an SDN box, and a downstream node.

  In the following description, a preferred embodiment of the present invention will be described with respect to preferred processing steps and data structures.

<System components>
FIG. 1 is a block diagram illustrating one embodiment of a system architecture of a computer system having a software defined network (SDN) box 106, upstream nodes 101 and 104, and downstream nodes 102, 103 and 105. As shown in FIG. Each component is connected by a network (not just Ethernet, for example). In general, there is at least one group in the system. There are two groups in Figure 1. One group includes an upstream node 101, a downstream node 102, and a downstream node 103. The other group consists of upstream node 104 and downstream node 105. Each group has a single upstream node and at least one downstream node. All packets between upstream and downstream nodes are forwarded by the SDN box.

<System architecture of possible implementations>
2A and 2B illustrate one embodiment of a system architecture of one possible implementation where the SDN box consists of at least one software defined switch (SD switch) and at least one software defined switch controller (SDSW controller). FIG.

  In FIG. 2A, the SDN box 206 includes an SD switch 208 and an SDSW controller 207. There is a communication channel between the SD switch 208 and the SDSW controller 207. The upstream node and the downstream node can be divided into two groups, each group consisting of a single upstream node and at least one downstream node. The upstream node 201 and the two downstream nodes 202 and 203 are connected to the first port (# 1), the second port (# 2), and the third port (# 3) of the SD switch 208, respectively. The upstream node 204 and the downstream node 205 are connected to the fourth port (# 4) and the fifth port (# 5) of the SD switch 208, respectively.

  In FIG. 2B, the SDN box 206 comprises SD switches 208, 209, 211 and 212, and SDSW controllers 207 and 210. There are communication channels between the SDSW controller 207 and the SD switch 208, and between the SDSW controller 207 and the SD switch 209, respectively. Further, there are communication channels between the SDSW controller 210 and the SD switch 211 and between the SDSW controller 210 and the SD switch 212, respectively. The upstream node and the downstream node can be divided into two groups, each group consisting of a single upstream node and at least one downstream node. The upstream node 201 and the two downstream nodes 202 and 203 belong to one group, and the upstream node 204 and the downstream node 205 belong to another group.

  In FIGS. 2A and 2B, all SD switches are connected by a specific network, such as, but not limited to, Ethernet. The communication between the SDSW controller and the SD switch is (but not only) in a specific protocol such as OpenFlow.

  FIG. 3 is a block diagram illustrating one embodiment of an upstream node system architecture. In general, a computer system consists of a CPU, memory, hard disk and various I / O devices. Other components are omitted here for a clearer description of the system architecture. The upstream node 301 includes at least a central processing unit (CPU) 302, an encapsulation / decapsulation (ENCAP / DECAP) module 303, and a network interface card (NIC) 304. The CPU 302 is a hardware component that executes instructions of a computer program by performing basic operations, arithmetic operations, logical operations, and system input / output operations. The NIC 304 is a hardware component that connects the upstream node 301 to a network in which the SDN box is arranged. The encapsulation / decapsulation module 303 is responsible for the encapsulation of the PCI-E packet and the decapsulation of the received encapsulated packet during the communication process. The CPU 302 and the encapsulation / decapsulation module 303 are logically connected, and the encapsulation / decapsulation module 303 and the NIC 304 are respectively connected. In other words, the manner of connection is not limited and may be (but not limited to) either a hardware manner such as the PCI-E bus protocol or any software manner.

  FIG. 4 is a block diagram illustrating one embodiment of a downstream node system architecture. Downstream nodes can consist of CPU, memory, hard disk and various devices. Other components are omitted here for a clearer description of the system architecture. The downstream node 401 includes at least a memory 402, an encapsulation / decapsulation (ENCAP / DECAP) module 403, and a network interface card (NIC) 404. The NIC 404 is a hardware component that connects the downstream node 401 to a network in which the SDN box is arranged. The encapsulation / decapsulation module 403 is responsible for the encapsulation of the PCI-E packet and the decapsulation of the received encapsulated packet during the communication process. The memory 402 and the encapsulation / decapsulation module 403 are connected, and the encapsulation / decapsulation module 403 and the NIC 404 are logically connected. In other words, the manner of connection is not limited and may be (but not limited to) either a hardware manner such as the PCI-E bus protocol or any software manner.

<Packet processing in SDN box>
FIG. 5 is a block diagram illustrating one embodiment of a software-defined (SD) switch system architecture. The SD switch 501 includes at least a RecvD module 502, a SendD module 503, a SendC (send to the control plane) module 504, a RecvC (receive from the control plane) module 509, a decapsulation (DECAP) module 505, and an encapsulation (ENCAP) module. 506, a packet buffer (PKT BUFFER) module 507, and a slave transaction layer packet (TLP) routing table manipulation module 508. The RecvD module 502 is in charge of reception from the upstream node and the downstream node. The SendD module 503 is in charge of transmission to the upstream node and the downstream node. RecvC module 509 is in charge of reception from the SDSW controller. The SendC module 504 is in charge of transmission to the SDSW controller. The decapsulation module 505 decapsulates the encapsulated packet into a PCI-E packet. The encapsulation module 506 encapsulates the PCI-E packet. The packet buffer module 507 is in charge of packet buffering. The TLP routing table manipulation module 508 can generate and insert new slave TLP routing table entries, and supports a slave TLP routing table search function.

  FIG. 6 is a block diagram illustrating an embodiment of a system architecture of a controller (SDSW controller) for a software defined switch. The SDSW controller includes at least a RecvSW module 602, a SendSW module 603, a master TLP routing table manipulation module 604, and a msg parser module 605. RecvSW module 602 is in charge of reception from the SD switch. The SendSW module 603 is in charge of transmission to the SD switch. The master TLP routing table manipulation module 604 can generate and insert new master TLP routing table items, and supports a master TLP routing table search function. The information extracted by the msg parser module 605 includes 1) node ID (node unique ID), 2) node type (upstream node or downstream node), and 3) destination (upstream or downstream) of each node. Address, 4) port number of the SD switch connected to the node (upstream or downstream), 5) VLAN tag used for the group (required only if the underlying network is Ethernet), 6 ) And a TLP routing ID (a unique ID of the TLP routing) from the received packet shown in FIG.

The SDN box consists of at least one SDSW controller and at least one SD switch. The overall processing process is as follows. When an encapsulated packet (its internal packet is a PCI-E packet) is received by the RcevD module 502 of the SD switch 501, it is decapsulated into a PCI-E packet by the decapsulation module 505 and then TLP routing ID is extracted. Then, the slave TLP routing table manipulation module 508 will recover the slave TLP routing table based on the extracted TLP routing ID.
> If not found, the packet is buffered in the packet buffer module 507 and then contains the query packet (TLP routing ID, node type, node ID, and other relevant information as shown in FIG. 13 ) Is transmitted to the SDSW controller 601 by the query (SendC) module 504. On the SDSW controller 601 side, the query is received by a receive (RecvSW) module 602. The master TLP routing table of the SDSW controller 601 will be further recovered by the master TLP routing table manipulation module 604 based on the TLP routing ID.
◇ If not found, the master TLP routing table manipulation module 604 of the SDSW controller 601 determines 1) the node ID (node unique ID) and 2) the node's ID based on the information extracted by the msg parser module 605. Type (upstream node or downstream node), 3) destination address (upstream or downstream) of each node, 4) port number (upstream or downstream) of the SD switch connected to the node, and 5) group Create a new table entry that includes the VLAN tag used for the (required only if the underlying network is Ethernet) and 6) the TLP routing ID shown in Figure 13 (a unique ID for TLP routing) It will be. It is then inserted into the master TLP routing table. The transmission (SendSW) module 603 of the SDSW controller 601 notifies the SD switch 501 to broadcast the PCI-E-over-Ethernet packet.
If found, the send (SendSW) module 603 of the SDSW controller 601 notifies the SD switch 501 to transfer the encapsulated PCI-E packet according to the recovered destination address. This notification is processed in the RecvC module 509, and the same new table entry is inserted into the slave TLP routing table by the slave TLP routing table manipulation module 508 of the SD switch 501.
> If found, the PCI-E packet will be encapsulated by the encapsulation module 506 along with the encapsulated packet header, and the encapsulated packet header will be sent to the destination address returned by the slave TLP routing table manipulation module 508. As filled with recovered addresses.

<DEVINFO packet format>
Two types of packets are used during the entire communication process between the upstream and downstream nodes. One is a PCI-E packet, which is used during memory read / write, I / O read / write, and config read / write operations. The other is a DEVINFO packet newly defined in the present invention. DEVINFO packets are used for 1) initiating communication between upstream and downstream nodes, and 2) periodically making upstream and downstream nodes aware of each other's existence at regular intervals. . The packet format of DEVINFO is (but not only) at least 1) node ID (node unique ID), 2) node type (upstream node or downstream node), and 3) node (upstream or downstream) Data field of 4) TLP routing ID (unique ID used in TLP routing). FIG. 7 is a table illustrating one embodiment of possible definitions of the packet format of the DEVINFO packet.

<Communication between upstream and downstream nodes>
FIG. 8 is a sequence diagram of memory read / write between the upstream node 801, the SDN box 802, and the downstream node 803 (I / O read / write and config read / write processes follow the same sequence. So it is omitted). The communication process is as follows.
(1) The upstream node 801 transmits a PCI-E packet when accessing the memory of the downstream node 803 (for example, read / write operation of the memory). In step 804, the PCI-E packet is encapsulated with a packet header by the encapsulation / decapsulation module 303 and sent out from the NIC 304.
(2) The encapsulated packet reaches a specific port of the SD switch in the SDN box 802. In step 805, the encapsulated packets are decapsulated, and then in step 806 their internal information is extracted. Next, in step 807, the decapsulated packet is encapsulated and transferred to the downstream node 803.
(3) When the encapsulated packet is received by the downstream node 803, it is decapsulated into a PCI-E packet at step 808.

  The communication process from the downstream node to the upstream node (for example, a response to the result of the memory read) is the same as the above steps. The figure is omitted.

FIG. 9 is a flow diagram illustrating one embodiment of how upstream or downstream nodes send PCI-E packets.
(1) When a node (either upstream node or downstream node) tries to send a packet in step 901, if it is a DEVINFO packet in step 902, in step 903, the destination (in the encapsulated packet header) The address must be a broadcast address.
(2) If the destination address is not a broadcast address, it must be a PCI-E packet, and the destination address (in the encapsulated packet header) must be in a predetermined format and is recognized by the SD switch in step 904. obtain.
(3) Finally, the packet is encapsulated with the outer packet header and the destination address comes from step 903 and step 904. The outer packet header type is dependent on the underlying network. For example, if the underlying network is Ethernet, an Ethernet packet header is added as an outer packet header and sent in step 905.

FIG. 10 is a flow diagram illustrating one embodiment of how upstream or downstream nodes receive encapsulated packets.
(1) When an encapsulated packet is received in step 1001, the packet is decapsulated into a PCI-E packet in step 1002.

FIG. 11 is a flow diagram illustrating one embodiment of how the SD switch processes encapsulated packets. The SD switch process is as follows.
(1) If there is no packet from the SDSW controller in step 1101, it is checked in step 1104 whether the packet belongs to a particular encapsulated packet. If the packet does not belong to a particular encapsulated packet, then in step 1105, the packet will be further processed in another packet routine.
(2) If the packet is an encapsulated packet from an upstream node or a downstream node, the TLP routing ID is extracted in step 1106, and the slave TLP routing table is recovered based on the extracted TLP routing ID. If not found, the packet is buffered in step 1108 and a query request is sent to the SDSW controller. If found, in step 1107, the packet is encapsulated and sent out with an encapsulated header filled with the recovered address as the destination address.
(3) If there is a packet received from the SDSW controller in step 1101, the transported TLP routing information is extracted in step 1102, and a new table item is added to the slave routing table in step 1103. Then, in step 1107, the previous buffered packet will be further processed (encapsulate the packet, fill the recovered destination address and send it).

Figure 12 shows that the SDSW controller processes the query request from the SD switch, extracts the TLP information, updates the master TLP routing table, and finally notifies the SD switch to update the slave TLP routing table. 6 is a flow diagram illustrating one embodiment of a manner. The process is as follows.
(1) If a query request packet from the SD switch is received in step 1201, TLP routing information is extracted in step 1202.
(2) In step 1203, the master TLP routing table is recovered based on the TLP routing information. If nothing is found, a new entry is added to the master TLP routing table at step 1204.
(3) Finally, in step 1205, the result of recovery is sent to notify the SD switch.

  FIG. 13 is a possible data structure for the TLP routing table, which is used for both the SD switch and the SDSW controller when the underlying network is Ethernet. The table shows 1) the node ID (node unique ID) column, 2) the node type (upstream or downstream node) column, and 3) the destination address column (upstream or downstream) of each node. And 4) a column of port numbers (upstream or downstream) of the SD switch connected to the node, and 5) a column of VLAN tags (needed only when the underlying network is Ethernet) used for the group And 6) a column of TLP routing ID (a unique ID of TLP routing). FIG. 13 shows two groups of nodes with VLAN ID 1 or 2. The first three items belong to one group because they share the same VLAN ID 1. The node whose node ID is 1 is an upstream node, and its MAC address is MAC_00. The node whose node ID is 1 is connected to the first port of the SD switch, and the TLP ID is busO / devO / funcO. The node whose node ID is 2 is a downstream node, and its MAC address is MAC_01. The node whose node ID is 2 is connected to the second port of the SD switch, and the TLP ID is busl / devl / funcl.

<Fault detection and handover>
FIG. 14 shows an upstream node 1401, an SD switch (for example, but not only an OpenFlow switch) 1402, an SDSW controller (for example, but not only an OpenFlow controller) 1403 and a downstream node (omitted for clarity) FIG. 6 is a sequence diagram of failure detection and handover during The failure detection and handover process is as follows.
(1) Once there is a failure in the upstream node 1401 in step 1404, a link down network signal is transmitted to the SD switch 1402.
(2) When receiving the link down network signal, the SD switch 1402 notifies the SDSW controller 1403 in step 1405. For example, in OpenFlow, the OpenFlow switch will send an OFPPS_LINK_DOWN message to the OpenFlow controller.
(3) Upon receiving the notification from the SD switch 1402, the SDSW controller 1403 finds another valid upstream node for handover in step 1406, and in step 1407 displays the master TLP routing table of the connected downstream node. Change and then notify the SD switch in step 1408.
(4) When receiving the notification from the SDSW controller 1403, the SD switch 1402 changes the slave TLP routing table in step 1409. As a result, the connected downstream node is handed over to the upstream node together with the new group ID.

  While preferred embodiments of the present invention have been described and illustrated above, it should be understood that these are illustrative of the invention and are not to be considered as limiting. Additions, omissions, substitutions, and other changes can be made without departing from the scope of the invention. Accordingly, the invention is not to be seen as limited by the foregoing description, but is only limited by the scope of the appended claims.

101 upstream node
102 Downstream node
103 Downstream node
104 upstream node
105 Downstream node
106 Software Defined Network (SDN) box
201 upstream node
202 Downstream node
203 Downstream node
204 Upstream node
205 Downstream node
206 SDN box
207 SDSW controller
208 SD switch
209 SD switch
210 SDSW controller
211 SD switch
212 SD switch
301 upstream node
302 Central processing unit (CPU)
303 Encapsulation / Decapsulation (ENCAP / DECAP) module
304 Network interface card (NIC)
401 Downstream node
402 memory
403 Encapsulation / Decapsulation (ENCAP / DECAP) module
404 Network interface card (NIC)
501 SD switch
502 RecvD module
503 SendD module
504 SendC (send to control plane) module
505 DECAP module
506 Encapsulation (ENCAP) module
507 packet buffer (PKT BUFFER) module
508 Slave Transaction Layer Packet (TLP) Routing Table Manipulation Module
509 RecvC (receive from control plane) module
601 Software-defined switch controller (SDSW controller)
602 RecvSW module
603 SendSW module
604 Master TLP routing table manipulation module
801 upstream node
802 SDN box
803 Downstream node
1401 upstream node
1402 SD switch
1403 SDSW controller

Claims (9)

A method of using a software defined network (SDN) box as a peripheral component interconnect express (PCI-E) switch across a network including at least one upstream node, at least one downstream node and an SDN box, comprising:
Connecting the upstream node and the downstream node to each other via the network by the SDN box;
Transmitting a PCI-E packet on one side of the upstream node and the downstream node;
Encapsulating the PCI-E packet with a specific packet header;
Sending the encapsulated packet to the network based on a transaction layer packet (TLP) routing identifier (ID) carried by the SDN box along with an internal PCI-E packet;
Receiving the encapsulated packet on the other side of the upstream node and the downstream node;
Removing the specific packet header from the received packet;
Decapsulating the received packet into the PCI-E packet. The upstream node includes a processing unit (CPU), a network interface card (NIC) and an encapsulation / decapsulation module,
The method of claim 1, wherein the downstream node comprises an I / O device, a network interface card (NIC) and an encapsulation / decapsulation module.   The method of claim 1, wherein the upstream node and the downstream node belong to at least one group including a single upstream node and at least one downstream node.   2. The method according to claim 1, wherein the TLP routing ID is the identifier of a TLP routing method in the PCI-E, including address routing, ID routing, and implicit routing.   The SDN box is implemented as a software defined (SD) switch and a software defined switch (SDSW) controller format corresponding to the SD switch, the SD switch includes a slave TLP routing table, and the SDSW controller is a master TLP routing table. The method of claim 1 comprising: Decapsulating the received packet from one of the upstream node and the downstream node using an SD switch, extracting the TLP routing ID, and recovering the slave TLP routing table based on the TLP routing ID There,
When a destination address is found in the TLP routing table, the packet is encapsulated with a packet header, and the destination address is satisfied and sent.
If the destination address is not found in the TLP routing table, buffering the packet and sending a query to the SDSW controller;
Using the SDSW controller, the query request from the SD switch is analyzed, TLP routing information is extracted, the TLP routing information is added to the master TLP routing table as a new table item, and the SD switch is added to the SD switch. And notifying the slave TLP routing table to be updated.   The SD switch and the SDSW controller corresponding to the SD switch are connected using a communication channel, whereby query and notification messages are transferred, and the communication channel is a remote communication channel including Ethernet or UNIX 6. The method of claim 5, wherein the method is any of a local communication channel including a domain socket.   The method as defined in claim 1, wherein in the step of encapsulating a PCI-E packet, the destination address is maintained in a predetermined format that can be recognized by the SDN box.   6. The method of claim 5, wherein when a failure occurs, the built-in notification of the SDN box triggers a TLP routing table change in the SD switch and the SDSW controller, which accomplishes handover of the downstream node.

Images