JP2010039729A - I/o management system, server system and i/o switch management method thereof - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To flexibly allocate an I/O device to a Host, in a server system compatible with MR-IOV (Multi-Root I/O Virtualization). <P>SOLUTION: Information on a configuration of the server system is collected to generate I/O topology information 11, 12. When receiving an allocation request 4 from a user of the server system compatible with MR-IOV, or an external system, I/O path configuration information 13 in which I/O-SWs 23a-d, 32a-d to be used for communication to Blades 22a-22d are defined is generated, and I/O-SW setting information 16 on communication channels of each of the I/O-SWs 23a-d, 32a-d is generated from the I/O path configuration information 13 and set. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a technology for controlling a PCI-Express Switch in a server system.

  PCI-Express, which is a high-speed connection technology for peripheral devices inside computers, has become widespread. In recent years, I / O virtualization technology has been adopted to support server virtualization technology for I / O devices (PCI devices) connected by PCI-Express. An I / O Virtualization (IOV) has appeared. IOV divides PCI devices into multiple virtual devices and assigns single-root IOV (SR-IOV) to virtual machines running on a single host (physical partition), and single PCI devices to multiple hosts There are two types of Multi-Root IOV (MR-IOV) to be assigned. In particular, MR-IOV changes the allocation relationship between the Host and PCI devices from the conventional one-to-many to many-to-many, so it provides great flexibility for host operation, but its management is complicated.

  One example of complicated MR-IOV management is Hot-Plug support. Unlike conventional PCI-Express and SR-IOV, MR-IOV can flexibly change the host that uses a specific PCI device, so it is difficult to anticipate changes in PCI configuration with Hot-Plug in advance. Bus number assignment to some PCI devices by Hot-Plug may be easily changed. If the bus number assignment to a PCI device changes, the data transfer path will change in ID routing using the bus number among the data transfer methods executed by PCI-Express, causing data loss, OS stop, OS restart It may cause troubles such as impossibility.

  As a method for coping with a change in Bus number assignment to a PCI device, there is a method disclosed in Patent Document 1 in the prior art. This is a response packet even if a path ID / device ID change occurs in the PCI-Express switch after a request is issued from a node in the PCI-Express communication system until a response is returned to the request source. This is a PCI-Express control method that can be sent to the request source.

  According to this conventional technology, it is possible to cope with the case where the response destination is changed to a known node during PCI-Express communication. However, as with the Hot-Plug in MR-IOV, an unknown node is If it is added, it will not be enough. In addition, when rebooting the host after Hot-Plug with MR-IOV, the bus number assignment of the boot device may change, and the conventional technology cannot cope with such a case.

JP 2007-316755 A

  Since the conventional technology does not support I / O virtualization and multiple hosts, it is suitable for server systems using PCI-Express Switch (MRS) that supports MR-IOV (hereinafter referred to as MR-IOV-compatible server systems). I / O device allocation cannot be set, and the flexibility of I / O device allocation provided by MR-IOV cannot be fully utilized.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to enable flexible allocation of an I / O device to a Host in an MR-IOV compatible server system.

  In order to solve the above problems, the present invention collects server system configuration information to generate I / O topology information, and receives a device allocation request from a user of an MR-IOV compliant server system or an external system. The I / O path configuration information that defines the I / O-SW through which communication to the device passes, and the I / O-SW related to the communication path of each I / O-SW from the I / O path configuration information Generate and set configuration information. In addition, Host setting information related to the communication path of each Host is generated and set from the I / O path configuration information.

  According to the present invention, a flexible I / O device can be allocated to a host in an MR-IOV compatible server system.

  Hereinafter, embodiments of the present invention will be described.

  FIG. 1 is one example of an embodiment of the present invention. It consists of a managed system that is an MR-IOV compliant server system and an I / O management system that manages the MRS (I / O-SW in the figure) in that server system.

  The managed system includes a server device 21 and an external I / O device 31. The server device 21 includes blade devices 22a to d (Blade 0 to 3) on which a CPU and a memory are mounted, and MRSs 23a to d (I / O-SWs 0 to 3), which are interconnected. The external I / O device 31 includes MRS 32a to d (I / O-SW 4 to 7), NIC (Network Interface Card) for communication with the network, and HBA (Host Bus for communication with peripheral devices). PCI devices 33a to h (Device 0 to 7) such as Adapter) are mounted and connected to each other. The server device 21 and the external I / O device 31 are also interconnected between the MRSs mounted on them, and any blade device and any PCI device can be configured according to the settings from the I / O management system 1. Can communicate with each other. The managed systems 21 and 31 receive settings from the I / O management system 1 so that an application operating on the host configured by one or a plurality of blade devices 22a to 22d performs communication processing with other applications. It becomes possible to execute.

  The I / O management system 1 (hereinafter, this system) receives the host configuration 2, the route search strategy 3, and the device allocation request 4, and executes the host setting and MRS setting of the managed system. The I / O management system 1 includes an I / O topology information generation unit 5, an I / O path configuration information generation unit 6, a Bus # configuration information generation unit 7, a Host setting information generation unit 8, and an I / O-SW setting information generation unit It consists of nine. The system 1 is composed of a processor (CPU, etc.) and a storage device (HDD, memory, etc.) as hardware, and the storage device is connected to an external storage medium (not shown) or a network (not shown). I / O topology information generation unit 5, I / O path configuration information generation unit 6, Bus # configuration information generation unit 7, Host setting information generation unit 8, I / O-SW setting information that are the components of this system 1 The generation unit 9 is stored, and each component is developed on the processor when the system 1 is executed. Similarly, Host configuration information 10, internal I / O-SW configuration information 11, external I / O-SW configuration information 12, I / O path configuration information 13, Bus # configuration, which are data repositories handled by this system 1. Information 14, Host setting information 15, and I / O-SW setting information 16 are also stored on the storage device, and the components of each system are accessed from the processor.

  The I / O topology information generation unit 5 includes the interconnection information between the MRS and PCI devices (collectively referred to as I / O components) detected from the managed systems 21, 31 and the blade devices in the server system of the managed system. Internal I / O-SW configuration information 11 and external I / O-SW configuration information 12 are generated based on interconnection information between MRSs (built-in I / O-SW).

  The I / O path configuration information generation unit 6 generates a route search strategy 3 and a device allocation request 4 based on the Host configuration information 9, the built-in I / O-SW configuration information 11, and the external I / O-SW configuration information 12. In response, I / O path configuration information 13 is generated. This process is executed every time the route search strategy 3 and the device allocation request 4 are received. Also, if there is existing I / O path configuration information 13, it will be executed when there is a change in the internal I / O-SW configuration information 11 or the external I / O-SW configuration information 12, and the existing I / O path configuration information 13 The O path configuration information 13 is updated.

  The Bus # configuration information generation unit 7 generates Bus # configuration information 14 based on the built-in I / O-SW configuration information 11 and the external I / O-SW configuration information 12. This process is executed when the internal I / O-SW configuration information 11 or the external I / O-SW configuration information 12 is changed.

  The host setting information generation unit 8 generates host setting information based on the I / O path configuration information 13. This process is executed each time the I / O path configuration information generation unit 6 generates the I / O path configuration information 13.

  The I / O-SW setting information generation unit 16 generates I / O-SW setting information 16 based on the I / O path configuration information 13. This process is executed each time the I / O path configuration information generation unit 6 generates the I / O path configuration information 13.

  Next, an example of processing of this system will be described with reference to FIG.

  This system starts processing of the I / O path configuration information generation unit 6 in response to a route search strategy 3 and a device allocation request 4 from a user of this system or an external system. Route search strategy 3 is a selection criterion when selecting an MRS through which a communication path (hereinafter referred to as I / O path) connecting the Host and PCI devices passes. For example, “Performance priority” or “Power saving priority” There is. Device allocation request 4 is an allocation request for PCI devices 33a to h to Host, and lists the allocation of one or more PCI devices 33a to 33h for one Host for a single Host or multiple Hosts. is there.

  First, the I / O path configuration information generation unit 6 is based on the external I / O-SW configuration information 12, and is an external I / O device equipped with the PCI devices 33a to h specified by the Device allocation request 4. MRSs 32a to 32d (hereinafter referred to as external I / O-SW) in 31 are specified (step 101). An example of the external I / O-SW configuration information 12 is shown in FIG. As shown in FIG. 5, the external I / O-SW configuration information 12 includes the external I / O-SWs 32a to 32d generated based on the configuration information detected from the managed systems 21 and 31, and the external I / O-SW configuration information 12. A connection relationship in which PCI devices 33a to 33h mounted on the I / O device 31 are associated with each other is stored. The external I / O-SW ID is system unique identification information of the external I / O-SWs 32a to d, and the mounted PCI device ID is mounted on the same external I / O device 31, and its external I / O This is system unique identification information of PCI devices 33a to 33h that can be interconnected from O-SWs 32a to 32d. Using this information, by specifying the PCI devices 33a to 33h, the external I / O-SWs 32a to 32d constituting the I / O path terminating at the PCI devices 33a to 33h are uniquely determined.

  Subsequently, the I / O path configuration information generation unit 6 determines the MRSs 23a to 23d (hereinafter referred to as the built-in I / O-SW) in the server device 21 via the built-in I / O-SW configuration information 11 ( Step 102). An example of the built-in I / O-SW configuration information 11 is shown in FIG. As shown in FIG. 4, the built-in I / O-SW configuration information 11 is mutually connected via the built-in I / O-SWs 23a to 23d generated based on the configuration information detected from the managed systems 21 and 31. Configuration information in which connectable blade devices 22a to 22d and external I / O-SWs 32a to 32d are associated is stored. The Blade ID is system unique identification information of the blade devices 22a to 22d that can be interconnected from the built-in I / O-SWs 23a to 23d identified by the built-in I / O-SW ID. The built-in I / O-SW ID is system unique identification information of the built-in I / O-SWs 23a to 23d. The external I / O-SW ID is the system unique identification information of the external I / O-SWs 32a-d that can be interconnected from the internal I / O-SWs 23a-d specified by the built-in I / O-SW ID. is there. In FIG. 4, only the built-in I / O-SW 23a is shown, but similar configuration information is stored for the other built-in I / O-SWs 23b to d. When the blade devices 22a-d and the built-in I / O-SWs 23a-d, and the built-in I / O-SWs 23a-d and the external I / O-SWs 32a-d are connected by multiple communication paths The built-in I / O-SWs 23a to 23d constituting the I / O path are not uniquely determined. Therefore, the built-in I / O-SWs 23a to 23d constituting the I / O path are determined based on the input route determination strategy 3. For example, if the routing strategy is “performance priority”, the existing I / O path among the built-in I / O-SWs 23a-d interconnecting the host and the PCI device that received the allocation request Select the built-in I / O-SW23a-d that uses the least amount of data. If the routing strategy is “power saving priority”, the performance of the built-in I / O-SW used in the existing I / O path (hereinafter referred to as “built-in I / O-SW in use”) is allowed. As far as possible, select the built-in I / O-SW in use as much as possible. By reducing the number of built-in I / O-SWs in use as much as possible, it is possible to power off other built-in I / O-SWs and enable power-saving operation of the server system. In addition, even without the route determination strategy 4, the built-in I / O-SWs 23a to 23d constituting the I / O path can be selected by a generally well-known load distribution algorithm.

  Next, the I / O path configuration information generation unit 6 uses the external I / O-SWs 32a to 32d and the built-in I / O-SWs 23a to d that constitute the specified I / O path to generate the Host configuration information 9 Based on this, I / O path configuration information 13 is generated (step 103). An example of the Host configuration information 9 is shown in FIG. As shown in FIG. 3, the Host configuration information 9 stores the Host configuration 2 input from the user of this system or an external system. The Host ID is system unique identification information of the Host in the server device 21, and the Blade ID is system unique identification information of one or a plurality of blade devices 22a to 22d constituting each Host. In this embodiment, there are some in which a plurality of blade devices are assigned to a host, but it is also possible to assign a plurality of hosts to a single blade device. Based on the device assignment request received, the start point is one of the hosts that have received any designation, and the end point is one of the PCI devices 33a to 33h that have received the assignment request to that host, and they are connected. The I / O path configuration information 13 includes the built-in I / O-SWs 23a to 23d and the external I / O-SWs 32a to 32d. An example of the I / O path configuration information 13 generated by the I / O path configuration information generation unit 6 is shown in FIG. The Host ID is system unique identification information of the Host that is the starting point of the I / O path, and the I / O path ID is system unique identification information of the I / O path that starts from the Host. The assigned Device ID is the system unique identification information of the PCI device that is the end point of the I / O path, and the internal I / O-SW ID via is the built-in I / O-SW 23a-d that constitutes the I / O path Similarly, the system unique identification information and the relay external I / O-SW ID are also system unique identification information of the external I / O-SWs 32a to 32d constituting the I / O path. Using this I / O path configuration information, the system subsequently generates Host setting information 15 and I / O-SW setting information 16.

  In this system, the Host setting information generation unit generates Host setting information, and the I / O-SW setting information generation unit 9 generates I / O-SW setting information 16.

  In addition to the I / O path configuration information 13, the Bus # configuration information generation unit 7 generates Bus # configuration information 14 in advance (step 104), and the Host setting information generation unit 8 determines that the I / O path configuration information 13 and the Bus Using #configuration information 14, setting information for each host is generated (step 105).

  FIG. 6 shows an example of the Bus # configuration information 14 generated by the Bus # configuration information generation unit 7. The Bus # configuration information 14 stores the Bus number assigned to the PCI bus between the blade devices 22a to d to the PCI devices 33a to h in the managed systems 21 and 31, and the built-in I / O-SW configuration information 11 And generated based on the external I / O-SW configuration information 12. In Bus # configuration information 14, for each PCI bus, the identification information of the I / O component located upstream of the PCI bus is the upstream component, and similarly, the identification information of the I / O component located downstream of the PCI bus Is the downstream component. The bus number assigned to the PCI bus is Bus #, and it is assumed that the server system has the maximum configuration that can be realized according to the server system specifications, regardless of the MRS mounting status and PCI device mounting status of the managed systems 21 and 31. And generate. An example of the Host setting information 15 generated by the Host setting information generation unit 8 based on the Bus # configuration information 14 and the I / O path configuration information 13 is shown in FIG. The Host ID is Host identification information for setting the Host setting information 15. The Bus # range is based on the Bus # configuration information 14 and the generated I / O path configuration information 13, and for the I / O path used by the host, the bus of the PCI bus through which the I / O path passes. A range of numbers is listed. Each Host receives this Host setting information 15 and assigns the assigned bus number to each PCI bus during the PCI device enumeration in the Bus # range of the Host setting information 15 so that it is not used during that enumeration. When a PCI bus is used in the future, a bus number assigned to the PCI bus can be secured. By doing so, it is possible to operate the Host without changing the Bus number of the PCI device even if the PCI configuration of the server system is changed by Hot-Plug or the like.

  The I / O-SW setting information generation unit 9 uses the I / O path configuration information 13 to generate setting information for each I / O-SW (MRS) (step 106).

  FIG. 9 shows an example of the I / O-SW setting information 16 generated by the I / O-SW setting information generating unit 9. The I / O-SW setting information 16 is generated for each I / O-SW. The example in the figure is an example of setting information for the I / O-SW specified by the identification information I / O-SW_2. Host ID is identification information of a Host that uses an I / O path that passes through the I / O-SW, and Device ID is identification information of a PCI device that is an end point of the I / O path. The transfer destination I / O component ID is the I / O to which the I / O-SW transfers the data when there is data input that uses the I / O path with both the Host ID and Device ID as both ends. This is identification information that identifies the component. The transfer destination I / O component is determined as an external I / O-SW if it is a built-in I / O-SW, and a PCI device if it is an external I / O-SW. Also, which I / O-SW is determined uniquely for each combination of Host ID and Device ID according to the I / O path configuration information 6. Each MRS uses this I / O-SW setting information 16 to perform data transfer setting, so that PCI-Express packet routing (ID routing) can be performed appropriately.

  As described above, this system generates Host setting information 15 and I / O-SW setting information 16, and the settings on the storage area of the Host via the setting I / F provided by the Host and I / O-SW, respectively. Set the Bus # range of the Host setting information 15 to the part that stores the Bus # range of the Host ID and Device ID on the storage area of the internal I / O-SW23a to d and external I / O-SW32a to d MR-IOV support by setting the Host ID, Device ID, and transfer destination I / O component ID of the I / O-SW setting information 16 to the part that stores the transfer destination I / O component ID for each combination of Appropriate ID routing is possible in the server system, and even if the PCI configuration of the server system is changed by Hot-Plug, the processing shown in Fig. 2 is re-executed to reconfigure the Host settings and I / O-SW settings. By executing, you can continue to operate the Host without changing the Bus number to the existing PCI device.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in embodiment mentioned above. It will be apparent to those skilled in the art that various modifications or improvements can be made to the above-described embodiment. Furthermore, the form which added such a change or improvement can also be contained in the technical scope of this invention.

1 is a system configuration diagram illustrating the configuration of an I / O management system according to a first embodiment of the present invention. It is explanatory drawing which illustrates the processing flow of an I / O management system. It is explanatory drawing which illustrates the data structure of Host structure information. It is explanatory drawing which illustrates the data structure of built-in I / O-SW structure information. It is explanatory drawing which illustrates the data structure of external I / O-SW structure information. It is explanatory drawing which illustrates the data structure of Bus # structure information. It is explanatory drawing which illustrates the data structure of I / O path structure information. It is explanatory drawing which illustrates the data structure of Host setting information. It is explanatory drawing which illustrates the data structure of I / O-SW setting information.

Explanation of symbols

  1 ... I / O management system, 2 ... Host configuration, 3 ... Route search strategy, 4 ... Device allocation request, 5 ... I / O topology information generator, 6 ... I / O path configuration information generator, 7 ... Bus # Configuration information generation unit, 8 ... Host setting information generation unit, 9 ... I / O-SW setting information generation unit, 10 ... Host configuration information, 11 ... Built-in I / O-SW configuration information, 12 ... External I / O- SW configuration information, 13 ... I / O path configuration information, 14 ... Bus # configuration information, 15 ... Host setting information, 16 ... I / O-SW setting information, 21 ... Server device, 22 ... Blade device, 23 ... MRS (Multi-Root Switch), 31 ... External I / O device, 32 ... MRS (Multi-Root Switch), 33 ... PCI device.

Claims (14)

A blade device having a processor;
A host composed of one or more blade devices;
With multiple I / O devices for communication,
A server system comprising a plurality of I / O switches that relay data transfer between the host and the I / O device,
An I / O management system configured to set the I / O switch based on I / O switch setting information;
The I / O management system
Collects information on the configuration of the server system and generates I / O topology information that associates each I / O switch with the blade device and the I / O device that can be connected to the I / O switch. An I / O topology information generator to
The I / O device allocation request is received, and the host, the I / O that performs data transfer with the host based on the host configuration information that associates the host with the blade device, the I / O topology information, and the allocation request An I / O path configuration information generation unit that generates I / O path configuration information that associates an O device and an I / O switch through which the data transfer passes;
A server system comprising: an I / O switch setting information generation unit that generates the I / O switch setting information that defines a data transfer destination of the I / O switch from the I / O path configuration information . In claim 1,
The I / O management system sets the host based on host setting information,
The I / O management system further includes:
Based on the I / O topology configuration information, a bus number that allocates a bus number to each of the host that performs the data transfer, the I / O switch, and the data transfer between the I / O devices and generates bus number configuration information A configuration information generation unit;
Based on the I / O path configuration information and the bus number configuration information, a host setting information generation unit that generates host setting information that associates the host with the bus number with which the host performs data communication is provided. A server system characterized by that. In claim 1,
Built-in I / O switch and external I / O switch
The I / O topology information includes built-in I / O switch configuration information and external I / O switch configuration information,
Based on the host configuration information, the built-in I / O switch configuration information, the external I / O switch configuration information, and the allocation request, the I / O path configuration information generation unit generates the I / O path configuration information. A server system characterized by generating. In claim 1,
The I / O path configuration information unit accepts a route search strategy that defines criteria for selecting an I / O switch, and is based on the route search strategy, the host configuration information, the I / O topology information, and the allocation request. And generating the I / O path configuration information. In claim 2,
The server system is characterized in that the bus number is based on a maximum configuration in specifications that can be provided in the server system. I / O management system for setting the I / O switch of a server system comprising a host, an I / O device, and a plurality of I / O switches that relay data transfer between the host and the I / O device Because
An I / O topology that collects information on the configuration of the server system and associates each I / O switch with a blade device and the I / O device of the server system that can be connected to the I / O switch. An I / O topology information generator for generating information;
The I / O device allocation request is received, and the host, the I / O that performs data transfer with the host based on the host configuration information that associates the host with the blade device, the I / O topology information, and the allocation request An I / O path configuration information unit that generates I / O path configuration information in which an O device and an I / O switch through which the data transfer passes are associated;
An I / O switch setting information generating unit that generates I / O switch setting information that defines a data transfer destination of the I / O switch from the I / O path configuration information. Management system. In claim 6,
The I / O management system sets the host based on host setting information,
The I / O management system further includes:
Based on the I / O topology configuration information, a bus number that allocates a bus number to each of the host that performs the data transfer, the I / O switch, and the data transfer between the I / O devices and generates bus number configuration information A configuration information generation unit;
Based on the I / O path configuration information and the bus number configuration information, a host setting information generation unit that generates host setting information that associates the host with the bus number with which the host performs data communication is provided. I / O management system characterized by that. In claim 6,
The I / O topology information includes built-in I / O switch configuration information and external I / O switch configuration information,
Based on the host configuration information, the built-in I / O switch configuration information, the external I / O switch configuration information, and the allocation request, the I / O path configuration information generation unit generates the I / O path configuration information. I / O management system characterized by generating. In claim 6,
The I / O path configuration information unit accepts a route search strategy that defines criteria for selecting an I / O switch, and is based on the route search strategy, the host configuration information, the I / O topology information, and the allocation request. The I / O management system for generating the I / O path configuration information. A blade device having a processor;
A host composed of one or more blade devices;
With multiple I / O devices for communication,
In an I / O switch management method for a server system comprising a plurality of I / O switches that relay data transfer between the host and the I / O device,
The I / O topology information generation unit collects information on the configuration of the server system, and associates each I / O switch with the blade device and the I / O device that can be connected to the I / O switch. I / O topology information generation step for generating attached I / O topology information,
The I / O path configuration information generation unit receives an I / O device allocation request, and based on the host configuration information in which a host and the blade device are associated, the I / O topology information, and the allocation request, the host, An I / O path configuration information generating step for generating I / O path configuration information in which the host and the I / O device that performs data transfer and the I / O switch through which the data transfer passes are associated;
An I / O switch setting information generation unit generates the I / O switch setting information that defines the data transfer destination of the I / O switch from the I / O path configuration information and sets the I / O switch setting information in the I / O switch An I / O switch setting information generation step including an I / O switch setting information generation step. In claim 10,
A bus number configuration information generation unit allocates a bus number to each of the host that performs the data transfer, the I / O switch, and the data transfer between the I / O devices based on the I / O topology configuration information. Bus number configuration information generation step for generating number configuration information;
Based on the I / O path configuration information and the bus number configuration information, a host setting information generation unit generates host setting information in which the host and the bus number with which the host performs data communication are associated with each other, and A host system I / O switch management method comprising: a host setting information generation step for setting a host. In claim 10,
Built-in I / O switch and external I / O switch
The I / O topology information includes built-in I / O switch configuration information and external I / O switch configuration information,
In the I / O path configuration information generation step, the I / O path configuration information generation unit includes the host configuration information, the built-in I / O switch configuration information, the external I / O switch configuration information, and the allocation request. A server system I / O switch management method, comprising: generating the I / O path configuration information based on the I / O path configuration information. In claim 10,
In the I / O path configuration information generation step, the I / O path configuration information generation unit accepts a route search strategy that defines a criterion for selecting an I / O switch, the route search strategy, the host configuration information, the An I / O switch management method for a server system, characterized in that the I / O path configuration information is generated based on I / O topology information and the allocation request. In claim 11,
The I / O switch management method for a server system, wherein the bus number is based on a maximum configuration according to specifications that the server system can have.

Images