CN102722458A - I/ O (input/output) remapping method and device for plurality of shared systems - Google Patents

Abstract

The invention discloses a direct I/O remapping method for a multi-root I/O virtualization sharing system. The method includes: step 1, extracting the definition of the base address of the I/O device function, recording the serial number of the base address register and its defined The mapping relationship between the base address serial numbers corresponding to the address window; Step 2, write and store the functional base address in the main control root node PCIe domain, and establish the mapping from the functional base address serial number to the base address of the main control root node PCIe domain; Step 3, The device function ID mapping relationship is written and stored, and the mapping between the device function between the ID numbers in the master control root node PCIe domain and the slave root node PCIe domain is established; step 4, the function base address in the slave root node PCIe domain is written and stored, and established Mapping from the base address of the PCIe domain of the slave root node to its corresponding functional base address serial number; step 5, direct I/O remapping, remapping the transaction packet between the slave root node and the I/O device.

Description

I/O remapping method and device for multi-root sharing system

technical field

The invention relates to I/O virtualization technology, in particular to an I/O remapping method and device of a multi-root I/O virtualization sharing system.

Background technique

The identification of PCIe (Peripheral Component Interface Express) device functions in the system is usually indicated by an ID number (B/D/F) composed of bus number (BUS)/device number (Device)/function number (Function). After the system is powered on, the system initialization software first enumerates, discovers and initializes all the I/O devices in the system according to the ID route, and at the same time assigns its ID number in the PCIe domain and its mapped address to the device function window etc. After the device is initialized, upper-layer software such as system software and drivers can be directed to the device through ID or address routing. In a multi-root I/O virtualization sharing system, in order to avoid system competition and conflict, I/O devices can be used by multiple root nodes, but can only be controlled by one root node. When the system is powered on, the I/O device function is first initialized by the main control root node to configure its ID number and address window in the PCIe domain of the main control root node. When the device function is assigned to the slave root node, use a virtual device function as the device placeholder of the PCIe domain of the slave root node, and accept the ID number and address window in the local PCIe domain assigned by the slave root node. Therefore, for the slave root node to use the ID number and address of the local PCIe domain to access the device function, remapping is required to realize the slave root node directly accessing the I/O device.

Contents of the invention

In order to solve the above problems, the present invention provides a direct I/O remapping method for a multi-root I/O virtualization sharing system, comprising:

Step 1, the I/O device function base address definition is extracted, and the mapping relationship between each base address register serial number and the base address serial number corresponding to the defined address window is recorded;

Step 2, write and store the function base address in the main control root node PCIe domain, and establish the mapping of the function base address serial number to its base address in the main control root node PCIe domain;

Step 3, the device function ID mapping relationship is written and stored, and the mapping between the device function between the ID numbers in the master control root node PCIe domain and the slave root node PCIe domain is established;

Step 4, writing and storing the function base address in the PCIe domain of the slave root node, and establishing a mapping from the base address of the PCIe domain of the slave root node to its corresponding function base address serial number;

Step 5, direct I/O remapping, remapping the transaction packets between the slave root node and the I/O device.

In the direct I/O remapping method for a multi-root I/O virtualization sharing system, step 1 also includes:

The base address definition extraction is to obtain the address window type defined by each base address register (Base Address Register, BAR) in the I/O device function configuration space in the main control root node system discovery stage, and extract the address window type corresponding to each address window. base address mask;

The base address mask is used to specify which bits of the address in the address window of the system configuration are fixed bits, and the base address corresponding to the memory access address can be extracted through the base address mask;

The base address serial number is used to represent the serial number of the number of address windows that the function of the device applies to the system according to the order of the BAR register number. The type and number of address windows that each function of the same type in an I/O device applies to the system are the same .

In the direct I/O remapping method for a multi-root I/O virtualization sharing system, the step 2 also includes:

The function base address write storage in the master control root node PCIe domain is in the master control root node system initialization configuration stage, by parsing the configuration transaction package, extracting the function base address serial number indicated in the transaction package as an address index, and writing and storing the The base address corresponding to the sequence number of the function base address.

The function base address number is composed of the base address number k and the device function number p in the PCIe domain of the main control root node. The base address indicated by the function base address number (k, p) is the function number of the device in the system The nth base address allocated by p.

In the direct I/O remapping method for a multi-root I/O virtualization sharing system, step 3 also includes:

The writing and storage of the device function ID mapping relationship is performed at the stage of device I/O resource allocation by the master control root node, by parsing the device allocation transaction package, and extracting the function number of the device function indicated in the transaction package in the PCIe domain of the master control root node as Address index, write and store a data item consisting of the root node identification ID to which the function belongs and the function number of the function in the PCIe domain of the root node to which the function belongs, and establish the ID number and its ID number of the device function in the PCIe domain of the main control root node Mapping between ID numbers in slave root node PCIe domains.

In the direct I/O remapping method for a multi-root I/O virtualization sharing system, the step 4 also includes:

The function base address write storage in the PCIe domain of the subordinate root node is in the PCIe rescanning stage of the subordinate root node system, by parsing and configuring the transaction package, extracting the function base address sequence number indicated in the transaction package as an address index, and writing and storing the function The base address corresponding to the base address serial number.

In the direct I/O remapping method for a multi-root I/O virtualization sharing system, step 5 also includes:

The direct I/O remapping is to perform downlink ID remapping on the downlink transaction packet based on ID routing initiated by the subordinate root node during the communication process between the subordinate root node and the I/O device, and to perform downlink ID remapping on the downlink transaction packet initiated by the subordinate root node based on Address remapping is performed on downlink transaction packets of address routing, and uplink ID remapping is performed on transaction packets initiated by device functions.

In the direct I/O remapping method for a multi-root I/O virtualization sharing system, step 1 also includes:

Step 71, power on the system, initialize the start address of the BAR register offset pointer, initialize the base address serial number, initialize the 64-bit address application identifier, and turn to step 72;

Step 72, receiving the transaction packet sent from the master control root node; if receiving the transaction packet sent by the master control root node, turn to step 73;

Step 73, determine the transaction packet; parse the received transaction packet, if the transaction packet is a configuration write request packet that writes all 1s to the register specified by the BAR register offset pointer, turn to step 74; otherwise turn to step 72;

Step 74, receiving the completion packet with data sent by the I/O device; if receiving the completion packet with data sent from the I/O device, turn to step 75;

Step 75, judging the address application type; analyzing the data segment of the completed package, judging the address application type defined by the BAR; if the BAR is not used for address application, then turn to step 78; if the BAR defines a 64-bit address, Then turn to step 76; otherwise, turn to step 77;

Step 76, record the content of the data segment to the low L bit of the base address mask corresponding to the base address serial number, enable the 64-bit address application identifier, and record the current BAR register offset pointer and the address window bit defined by the BAR indicated by it For the mapping relationship between the wide type and the base address serial number, turn to step 78;

Step 77, record the data segment, update the base address sequence number and 64-bit address application identifier; if the current 64-bit application identifier is not enabled, indicating that the content of the data segment is the mask of the 32-bit base address, then the content of the data segment The high L bits are recorded to the low L bits of the base address mask corresponding to the base address serial number; otherwise, the data segment content is the upper 32 bits of the 64-bit base address mask corresponding to the base address serial number, then the data segment is recorded to the base address The upper 32 bits of the base address mask corresponding to the serial number; record the mapping relationship between the current BAR register offset pointer and the address window bit width type defined by the BAR indicated by the BAR and the base address serial number, then update the base address serial number and add 1, and reset 64 Bit address application identifier, turn to step 78;

Step 78, update the BAR register offset pointer to point to the next BAR register, and turn to step 79;

Step 79, BAR register offset pointer validity determination; determine whether the register pointed to by the current BAR register offset pointer is still a BAR in the configuration space, if yes, then turn to step 72; otherwise turn to step 710;

Step 710, end, indicating that the process of extracting the base address definition has been completed.

In the direct I/O remapping method for a multi-root I/O virtualization sharing system, step 2 also includes:

Step 81, receiving the transaction packet sent from the root node, analyzing and judging the ID of the root node initiating the transaction packet, if the originating root node indicated by the transaction packet is the main control root node, turn to step 82;

Step 82, determine the downlink transaction packet; analyze the received transaction packet sent by the main control root node, if the transaction packet is a configuration transaction packet, turn to step 83;

Step 83, obtain and analyze the configuration writing packet that writes non-all 1s to the BAR register in the configuration transaction packet, extract the target device function number in the transaction packet, the BAR register offset to be configured, and the content of the data segment described in the data segment are system Configured base address, turn to step 84;

Step 84, according to the mapping relationship obtained in the base address definition extraction process, obtain the base address sequence number and address window bit width type corresponding to the BAR register offset, and turn to step 85;

Step 85, the address window bit width type judgment; the address window bit width type obtained in judging step 84, if indicating that the BAR is defined as the lower 32 bits of the 64-bit address, then register the base address extracted, and then turn to step 81; if indicated 32 is the address or the upper 32 bits of the 64-bit address, then turn to step 86;

Step 86, write and store the base address; if it is a 32-bit address, store the extracted data segment in the storage unit specified by the function number and the base address sequence number; if it is a 64-bit address, then store the base address and The combination of high and low bits of the data segment extracted in step 83 is stored in the storage unit specified by the function number and the base address sequence number; turn to step 81.

In the direct I/O remapping method for a multi-root I/O virtualization sharing system, step 4 also includes:

Step 91, receiving the transaction packet sent from the root node, analyzing and determining the ID of the root node initiating the transaction packet, if the originating root node indicated by the transaction packet is a subordinate root node, turn to step 92;

Step 92, determine the downlink transaction packet; analyze the received transaction packet sent by the slave root node, if the transaction packet is based on the ID transaction packet, turn to step 93;

Step 93, downlink ID remapping; convert the ID number in the transaction packet based on ID routing to the ID number corresponding to the PCIe domain of the main control root node, and turn to step 94;

Step 94, obtain and analyze the configuration write packet that writes non-all 1s to the BAR register in the configuration transaction packet, extract the target device function number in the transaction packet, the offset of the BAR register to be configured, and the base address indicated by the data segment content, and turn to the step 95;

Step 95, according to the mapping relationship obtained in the base address definition extraction process, obtain the base address sequence number and address window bit width type corresponding to the BAR register offset, and turn to step 96;

Step 96, the address window bit width type judgment; the address window bit width type obtained in judging step 95, if indicating that the BAR is defined as the lower 32 bits of the 64-bit address, then register the base address extracted, and then turn to step 91; if indicated 32 is the address or the upper 32 bits of the 64-bit address, then turn to step 97;

Step 97, write storage base address; If it is a 32-bit address, then store the data segment extracted in the storage unit specified by the function number and the base address sequence number; The combination of high and low bits of the data segment extracted in step 94 is stored in the storage unit specified by the function number and the base address sequence number; turn to step 91.

In the direct I/O remapping method for a multi-root I/O virtualization sharing system, the step 5 includes:

The downlink ID remapping refers to the matching conversion of the I/O device function from the ID number of the slave root node PCIe domain to the ID number of the main control root node PCIe domain;

The address remapping refers to the matching conversion from the target access address sent by the slave root node to the address corresponding to the PCIe domain of the main control root node;

The uplink ID remapping refers to the matching transformation from the ID number of the I/O device function in the PCIe domain of the master control root node to its ID number in the PCIe domain of the slave root node.

In the direct I/O remapping method for a multi-root I/O virtualization sharing system, the downlink ID remapping includes:

Step 111, receiving the transaction packet sent from the root node, analyzing and determining the ID of the root node initiating the transaction packet, if the originating root node indicated by the transaction packet is a subordinate root node, turn to step 112;

Step 112, determine the type of transaction packet, if it is a transaction packet based on ID routing, turn to step 113;

Step 113, extract the initiating root node ID indicated in the transaction package and the device function number to be accessed, and combine them as the data content to be compared, accurately match and search, and obtain the function number of the device function in the main control root node PCIe domain , if there is a matching item, turn to step 114; otherwise, turn to step 115;

Step 114, rewrite the device function ID number in the transaction package, use the CAM to match the output address, that is, the function number in the indicated main control root node PCIe, rewrite the device function ID number in the transaction package, and turn to step 111 after the operation;

Step 115, return an unsupported transaction package to the root node indicated by the transaction package, and go to step 111 after the operation.

In the direct I/O remapping method for a multi-root I/O virtualization sharing system, the address remapping includes:

Step 121, receiving the transaction packet sent from the root node, analyzing and determining the ID of the root node initiating the transaction packet, if the originating root node indicated by the transaction packet is a subordinate root node, turn to step 122;

Step 122, determine the downlink transaction packet; analyze the received transaction packet sent by the slave root node, if the transaction packet is based on the address transaction packet, turn to step 123;

Step 123, analyzing the received transaction packet based on address routing, extracting the target access address indicated in the transaction packet, and turning to step 124;

Step 124, obtain the base address; perform an AND operation on the target access address and all K base address masks recorded in the base address definition extraction process to obtain K base addresses, and turn to step 125;

Step 125, function base address sequence number matching search; use the K BAR base addresses obtained in step 124 as comparison data, perform parallel exact matching search, and obtain the function base address sequence number matched by the target access address; if there is a match, turn to step 126 , otherwise turn to step 129;

Step 126, base address and base address mask matching search; use the function base address sequence number output by matching to obtain the matching base address corresponding to the function base address sequence number in the main control root node PCIe domain and the base address of the base address sequence number match mask, turn to step 127;

Step 127, the matching target access address output decoding; according to the obtained matching base address, matching base address mask, together with the target access address extracted in step 123, obtain the target access address in the PCIe domain of the slave root node and map it to the main control root The matching target access address in the node PCIe domain; turn to step 128 after the operation;

Step 129, use the matching target access address obtained in step 127 to rewrite the target access address in the transaction package; turn to step 121 after the operation;

Step 129, return an unsupported transaction package to the root node indicated by the transaction package, and turn to step 121 after the operation is completed.

In the direct I/O remapping method for a multi-root I/O virtualization sharing system, the uplink ID remapping includes:

Step 131, receiving the transaction packet sent from the I/O device function, if the transaction packet is received, turn to step 132;

Step 132, extracting the I/O device function ID number indicated by the transaction package; if the transaction package is a request package, extract the request setter ID number indicated in the request transaction package; if it is if the transaction package is a complete package, extract the complete transaction package Indicated completer ID number, turn to step 133;

Step 133, using the function number in the extracted I/O device function ID number as an address index to obtain the corresponding root node identification ID and function number; if the obtained root node identification ID is not the main control root node ID, turn to step 134; Otherwise, it means that the root node to which the transaction package belongs is the main control root node, and no ID conversion is required, and the process turns to step 135;

Step 134, use the obtained root node identification ID to record in the transaction package; if the transaction package is a request package, then use the obtained function number to modify the ID number of the requester in the transaction package; if the transaction package is a complete package, use the obtained The function number modifies the completer ID number in the transaction package, and turns to step 131 after the operation is completed;

Step 135, rewrite the transaction package using the ID of the main control root node; record the ID of the main control root node into the transaction package, and turn to step 131 after the operation.

The invention discloses a direct I/O remapping device for a multi-root I/O virtualization sharing system, comprising:

ID remapping module, the ID remapping module includes content addressable storage CAM and simple dual-port RAM, realizes the ID number of I/O device function in slave root node PCIe domain to its ID number in master control through the precise matching search of CAM The mapping of the ID number in the PCIe domain of the root node realizes the mapping from the ID number of the I/O device function in the PCIe domain of the master control root node to its ID number in the PCIe domain of the slave root node by writing and reading the RAM storage element;

Address remapping module, described address remapping module is used for realizing I/O device function conversion from the I/O mapping address window of root node PCIe domain to its main control root node PCIe domain I/O mapping address window;

Wherein, the ID number includes a bus number, a device number, and a function number.

In the direct I/O remapping device for a multi-root I/O virtualization sharing system, the CAM and RAM in the ID remapping module include:

Using the function number of the I/O device function as the address index, the data item stored in the storage unit includes two parts: one part is the ID of the root node to which the device function belongs, and the other part is the function of the device function in the PCIe domain of the root node to which it belongs Number.

The I/O remapping device for a multi-root I/O virtualization sharing system, the address remapping module includes:

The base address definition extraction module, the base address mask obtains the address window type defined by each BAR for the I/O device function by parsing the transaction packet of the base address register (BAR) that configures the device function, and extracts each address The base address mask corresponding to the window, and record the mapping relationship between each base address register number and the base address number corresponding to the defined address window;

Functional base address serial number matching module, the functional base address serial number matching module includes a CAM storage array composed of N CAMs with a depth of P and a matching output selection module, each CAM stores the same base address in all functions in the same I/O device The base address BAR corresponding to the serial number realizes the mapping between the base address in the PCIe domain of the slave root node and its corresponding functional base address serial number;

Base address search module, said base address search module includes a depth of Q simple dual-port RAM, by writing and reading the RAM storage element, realizes the mapping of the function base address sequence number to its base address in the main control root node PCIe domain;

Output decoding module, said output decoding module translates the target access address in the slave root node PCIe domain into the master control root Target access address in the node PCIe domain;

Among them, N (N<=6) represents the number of address windows applied for by the device function, P represents the number of functions in the device, and Q represents the number of base addresses of the address windows allocated by the system to all functions in the device.

In the direct I/O remapping device for a multi-root I/O virtualization sharing system, the function base address serial number matching module includes:

The CAM storage array is based on the function base address serial number as the address index, and the stored data items include two parts: one part is the subordinate root node ID to which the I/O device function belongs, and one part is the base address allocated by the subordinate root node, wherein, The base address sequence number in the function base address sequence number selects the CAM to be stored, and the function number in the function base address sequence number selects the CAM storage unit to be stored. The CAM storage array has N comparison data input ports, and each CAM corresponds to one. In the data matching stage, the CAM array inputs N different comparison data items obtained by extracting the target access address of the subordinate root node through N different base address masks, performs matching search in parallel, and outputs N corresponding matching data items. result;

The matching output selection module, if there is (only one) matching item in the N matching results output by the CAM storage array, then select and output the output matching address (that is, the function number p and its corresponding base address) of the CAM that has the matching item The sequence number of the function base address composed of sequence number n, and set the matching flag to 1; otherwise, set the matching flag to 0.

In the direct I/O remapping device for a multi-root I/O virtualization sharing system, the RAM in the base address search module includes:

The function base address serial number is used as the address index, and the stored data item is the base address assigned by the main control root node.

The beneficial effects of the present invention are a direct I/O remapping method and device for a multi-root I/O virtualization sharing system. This direct I/O remapping method and device realizes the device function from the ID number and address window in the slave root node PCIe domain to its corresponding ID number and address window in the master control root node PCIe domain through hardware remapping. The mapping of the address window enables the root node to directly access and operate the device function resources allocated to it, and at the same time provides isolation and protection for each root node when accessing and operating shared physical I/O devices, and the root node cannot access Operate other I/O resources than the allocated I/O resources.

Description of drawings

FIG. 1 is a schematic structural diagram of a multi-root I/O virtualization sharing system;

Fig. 2 is a schematic diagram of the communication between the root node and the I/O device;

Fig. 3 is a schematic structural diagram of an ID remapping module;

FIG. 4 is a flow chart of the downlink ID remapping process;

FIG. 5 is a flowchart of an uplink ID remapping process;

Fig. 6 is a schematic structural diagram of an address remapping module;

Fig. 7 is a flow chart of the base address definition extraction process;

Fig. 8 is a flow chart of the write storage process of the function base address of the main control root node PCIe domain;

Fig. 9 is a flow chart of the write storage process of the functional base address from the root node PCIe domain;

Fig. 10 is a flowchart of the address remapping process;

FIG. 11 is a schematic structural diagram of an address remapping module for an SR-IOV device;

FIG. 12 is a flowchart of a direct I/O remapping method.

Detailed ways

Specific embodiments of the present invention are given below, and the present invention is described in detail in conjunction with the accompanying drawings.

In order to realize that each subordinate root node in the multi-root I/O virtualization sharing system directly accesses and operates the function of the I/O device assigned to it, so as to obtain the I/O performance close to the local machine, the present invention proposes a direct I/O remapping method and device. The direct I/O remapping method and device realizes the mapping of the corresponding ID number and the address window of the mapped I/O of the device function in the PCIe domain of the slave root node and the master control root node through hardware remapping. The direct I/O remapping method, as shown in Figure 12, comprises the following steps:

a) Base address definition extraction of I/O device functions refers to obtaining the address window type defined by each base address register (BAR) for I/O device functions during the discovery stage of the main control root node system, and extracting each address The base address mask corresponding to the window, and record the mapping relationship between each base address register serial number and the base address serial number corresponding to the address window defined by it; wherein, the base address mask is used to indicate the address in the address window configured by the system Which bits are fixed bits, the address used by the root node and the I/O device function communication is a specific access address composed of the base address BAR plus an offset, and the base address BAR corresponding to the access address can be extracted through the recorded base address mask. The base address serial number is used to represent the serial number of the number of address windows applied by the device function to the system according to the sequence of the BAR register number, and the type and number of address windows applied to the system by each function of the same type in an I/O device are the same;

b) Write and store the functional base address in the PCIe domain of the main control root node. The storage of the functional base address in the PCIe domain of the main control root node refers to extracting the transaction package by parsing the configuration transaction package during the system initialization configuration stage of the main control root node The function base address serial number indicated in is used as an address index, and the base address corresponding to the functional base address serial number is written and stored, wherein the functional base address serial number is composed of the base address serial number k and the device function number in the PCIe domain of the main control root node It is composed of p, and the base address indicated by the function base address number (k, p) is the nth base address allocated by the system for the function number p in the device. ;

c) Write and store the device function ID mapping relationship. The device function ID mapping relationship write and store refers to the stage of device I/O resource allocation at the main control root node. By parsing the device allocation transaction package, the device function indicated in the transaction package is extracted. The function number of the main control root node PCIe domain is used as an address index, write and store the root node identification ID to which the function belongs and the function number of the function in the root node PCIe domain to which the function belongs, and establish the device function in the main control root node PCIe domain The mapping between the ID number and its ID number in the slave root node PCIe domain;

d) Write storage of the functional base address in the PCIe domain of the subordinate root node. The write storage of the functional base address in the PCIe domain of the subordinate root node refers to extracting the transaction package by parsing the configuration transaction package during the PCIe rescanning phase of the subordinate root node system. The indicated function base address number is used as an address index, and the base address corresponding to the function base address number is written and stored;

e) Direct I/O remapping. The direct I/O remapping means that when the subordinate root node discovers and uses the I/O device, the downlink ID of the downlink transaction packet based on ID routing initiated by the subordinate root node is Remapping, address remapping is performed on downlink transaction packets based on address routing initiated by the slave root node, and uplink ID remapping is performed on transaction packets initiated by device functions.

FIG. 1 depicts a schematic structural diagram of a multi-root I/O virtualization sharing system. The multi-root I/O virtualization sharing system mainly includes three parts: the root node subsystem, the I/O device subsystem and the multi-root I/O virtualization sharing controller. The multi-root I/O virtualization sharing controller couples the root node subsystem and the I/O device subsystem through the PCIe interface protocol, so that the I/O device resources are directly shared by multiple root nodes.

The root node subsystem in the multi-root I/O virtualization sharing system includes multiple root nodes, and each root node is a single-root PCIe environment, which consists of a root complex (Root Complex, RC) and its connection CPU group (CPU set) and memory (Mem). Among them, the root node running PCIe management-related software is called the master control root node (Master Root Node, mRN), and there is only one master control root node (identified by mRN0 below), which manages and distributes all I/O devices in the system resources; other root nodes are called slave root nodes (Slave Root Node, sRN), and there can be multiple slave root nodes (identified by sRN1, sRN2, ..., sRNn below), and according to the distribution of the main control root node, have a certain I /O Independent usage rights for resources. Multiple homogeneous or heterogeneous virtual machines (Virtual Machines, VMs) can run on the root node, and the virtual machine management program (Virtual Machine Manager, VMM) is responsible for scheduling the allocated I/O resources for use by each VM.

The I/O device subsystem in the multi-root I/O virtualization sharing system includes multiple I/O devices, and each I/O device has the ability to provide services for multiple virtual machines at the same time, and may contain a physical function (Physical Function, PF) and its corresponding multiple virtual functions (Virtual Function, VF) SR-IOV devices or SR-IOV devices containing multiple PFs and their corresponding multiple VFs, or multi-function I/O devices wait. The I/O device subsystem is a single root PCIe environment belonging to the main control root node.

The multi-root I/O virtualization sharing controller in the multi-root I/O virtualization sharing system consists of three parts: several PCIe upstream ports (PCIe Upstream Port), PCIe multi-root switch and several PCIe downstream ports (PCIe Downstream Port) composition. Among them, the multi-root I/O virtualization shared controller is connected to multiple root nodes through the PCIe upstream port, and it is in a multi-root PCIe environment; the PCIe multi-root switch is essentially an N+M network established by multiple PCI bridges. A switch with multiple ports, by establishing a virtual PCIe switch with 1+M ports for each root node, realizes the logical connection between each root node and M I/O devices; The PCIe controller and the direct I/O virtualization interface device defined by the PCIe port type are two functional parts, which are responsible for the interconnection with the I/O device subsystem and realize the direct access of each root node to the physical I/O device function.

Figure 2 depicts a schematic diagram of the communication between the root node and the I/O device. There are three ways to communicate between the root node and the I/O device: I/O register read and write, DMA (Direct Memory Access) data movement, and interrupt event notification. The I/O registers are mainly divided into two categories, one is the configuration space registers used only by system initialization software and error handling software for device configuration, initialization and faulty error handling, and the other is other operating software such as driver programs The device control status register used to enable operation of the device.

The read and write access of the root node to the configuration space register of the I/O device is often directed by the position of the I/O device in the PCIe tree topology, such as access by the ID composed of the bus number, device number and function number. The root node can access the control status register of the I/O device through MMIO (MemoryMapped I/O, memory mapped I/O). When the system is initialized, the system allocates the memory address window according to the requirements defined by the BAR register (BAR, Base Address Register) in the configuration space header of the I/O device function, and sets the base address of the memory address window (hereinafter referred to as the memory base address) Write back to the BAR register. The root node can be directed to the corresponding control status register of the device function through the base address plus the offset of the register to be accessed.

The DMA data movement is divided into two situations: one is that the root node writes data into the memory, and then notifies the I/O device to fetch data at a specified memory address; the other is that the I/O device stores the data in the root node In the specified memory, after the completion, the device initiates an interrupt event to notify the root node that data has arrived, and the root node performs corresponding processing after receiving the interrupt notification. DMA data movement is essentially a memory read and write operation, and an interrupt event is essentially a memory write operation, and the operated memory address is written by the root node to the corresponding control status register of the I/O device through register read and write.

In a multi-root I/O virtualization sharing system, when the system is powered on, all root nodes attempt to enumerate and discover all functions in the I/O device and initialize and configure their ID numbers in their respective PCIe domains and their application mapping address windows The base address register BAR. This will cause system competition, causing the I/O device to not work normally, let alone sharing the I/O device between multiple root nodes. To solve this problem, a method of using a direct I/O virtualization interface device to assist in realizing multi-root sharing of I/O devices is proposed. Among them, the physical I/O device is initialized, configured, assigned and managed by the main control root node; when the I/O device function is allocated to each slave root node, the logical virtual function image LVF in the corresponding direct I/O virtualization interface device is used as the slave The device placeholder of the root node, which receives and stores the initial configuration of the slave root node. The description of the method is described in detail in another invention application (multi-root I/O virtualization sharing system and method) related to the present invention.

However, when the slave root node accesses the device, the ID number and address in the local PCIe domain are used, and the direct I/O virtualization interface device needs to perform I/O device functions in the slave root node PCIe domain and the master control root node PCIe domain. The conversion of the ID number and the mapping address between them can transfer the access of the slave root node to the corresponding physical I/O device function, and realize the purpose of the slave root node directly accessing the I/O device. Described ID and address conversion include three kinds of situations: the ID number conversion of subordinate root node PCIe field to the ID number of master control root node PCIe domain in downlink transaction, is called downlink ID remapping; Master control root node PCIe in uplink transaction The ID number of the domain is converted to the ID number of the PCIe domain of the slave root node; it is called uplink ID remapping; in the downlink transaction, the I/O mapping address window of the PCIe domain of the slave root node is mapped to the I/O mapping of the master control root node PCIe domain Address window translation is called address remapping. For upstream transactions that use address routing, such as DMA and interrupt event notification, since the addresses they operate are allocated by their subordinate root nodes, they are originally local memory addresses that can be recognized by the subordinate root nodes, so address translation is not required.

Fig. 3 depicts a schematic structural diagram of the ID remapping module. The ID remapping module includes a content addressable memory array (Content Addressable Memory, CAM) and a simple dual-port RAM. Among them, the ID remapping module realizes the mapping from the ID number of the I/O device function in the PCIe domain of the slave root node to its ID number in the PCIe domain of the main control root node through the exact matching search of the CAM, and writes and reads the RAM storage element Realize the mapping from the ID number of the I/O device function in the primary control root node PCIe domain to its ID number in the slave root node PCIe domain.

The CAM in the ID remapping module uses the continuity of the function number (FunNo_m) of the I/O device function in the PCIe domain of the main control node, and uses it as the write storage address index (CAM_WrAddr) to store the slave root node to which the function belongs The data item (CAM_Din) composed of the ID (RNID_s) and its function number (FunNo_s) in the PCIe domain of the slave root node, thereby realizing the mapping from (RNID_s, FunNo_s) to FunNo_m. In the stage of device I/O resource allocation by the master control root node, by parsing the device allocation transaction packet, the function number (FunNo_m) of the device function indicated in the transaction packet in the PCIe domain of the master control root node is extracted as the address index (CAM_WrAddr), and written Store the root node identification ID (RNID_s) to which the function belongs and the function number (FunNo_s) of the function in the PCIe domain of the root node to which the function belongs, and establish the ID number of the device function in the PCIe domain of the master control root node to its slave root The mapping of the ID number in the node PCIe domain; when receiving the downlink transaction packet using the ID route sent from the subordinate root node, by performing transaction packet analysis to it, extracting the subordinate root node ID of the transaction indicated in the transaction packet ( RNID_s) and the function number (FunNo_s) of the target device function to be accessed are input as the data item to be compared (CMP_Din), and the storage content of the CAM is precisely searched and matched. If there is a match, the CAM outputs the matching identifier (Match_flag) and the address (CAM_Dout) where the match is stored, that is, the function number (FunNo_m) of the target device function in the main control root node PCIe domain, and then realizes downlink ID remapping.

The uplink ID remapping table RAM uses the continuity of the function number (FunNo_m) of the device function in the PCIe domain of the master control node, and uses it as the write address (RAM_WrAddr) of the RAM to store the slave root node ID (RNID_s) to which the function number belongs. ) and the data item (RAM_Din) composed of its function number (FunNo_s) in the slave root node PCIe domain, and then realize the mapping from FunNo_m to (RNID_s, FunNo_s). In the stage of device I/O resource allocation by the main control root node, by analyzing the device allocation transaction package, the function number (FunNo_m) of the device function indicated in the transaction package in the PCIe domain of the main control root node is extracted as the address index (RAM_WrAddr), and written Store the root node identification ID (RNID_s) to which the function belongs and the function number (FunNo_s) of the function in the PCIe domain of the root node to which the function belongs, and establish the ID number of the device function in the PCIe domain of the slave root node to its master control root The mapping of the ID number in the node PCIe domain; when receiving the uplink transaction packet sent from the I/O device, by performing transaction packet analysis to it, extracting the function number (FunNo_m) indicating the device function that initiates the transaction wherein, and Use this as the read address of the RAM (RAM_RdAddr) to read the RAM storage table, and according to the RAM output (RAM_Dout), that is, the slave root node ID (RNID_s) to which the device function belongs and its function number (FunNo_s) in the PCIe domain of the slave root node , so as to realize uplink ID remapping.

Figure 4 describes the downlink ID remapping process, the specific steps are as follows:

a) Receive the transaction packet sent from the root node, analyze and judge the ID of the root node that initiated the transaction packet, if the originating root node indicated by the transaction packet is a subordinate root node, go to step b);

b) Determine the type of transaction packet, if it is a transaction packet based on ID routing, turn to step c);

c) Query the downlink ID remapping table CAM, extract the originating root node ID indicated in the transaction packet and the function number of the device to be accessed, and combine them as the data content to be compared, input the downlink ID remapping table CAM for exact matching Find, if there is a match, go to step d); otherwise go to step e);

d) Rewrite the device function ID number in the transaction package, use the CAM to match the output address, that is, the function number in the indicated main control root node PCIe, rewrite the device function ID number in the transaction package, and turn to step a) after the operation;

e) Return an unsupported transaction package to the root node indicated by the transaction package, and turn to step a) after the operation is completed.

Figure 6 describes the uplink ID remapping process, the specific steps are as follows:

a) Receive the transaction packet sent from the I/O device function, if the transaction packet is received, go to step b);

b) Extract the function ID number of the I/O device indicated by the transaction packet. If the transaction package is a request package, extract the request setter ID number indicated in the request transaction package; if it is if the transaction package is a completion package, extract the completer ID number indicated in the transaction package, and turn to step c);

c) Query the upstream ID remapping table RAM, use the function number in the extracted I/O device function ID number as the read address index, and read the data item composed of the root node identification ID and the function number of the RAM output. If the ID of the root node output by the RAM is not the ID of the main control root node, go to step d); otherwise, it means that the root node to which the transaction package belongs is the main control root node, no ID conversion is required, and go to step e);

d) Rewrite the transaction packet with the contents of the RAM output. Record the ID of the root node output by the RAM into the transaction package; if the transaction package is a request package, use the function number output by the RAM to modify the ID number of the requester in the transaction package; if the transaction package is a completion package, use the function number output by the RAM Function number, modify the ID number of the completer in the transaction package, and turn to step a) after the operation;

e) Use the main control root node ID to rewrite the transaction package. Record the identification ID of the main control root node into the transaction package, and turn to step a) after the operation.

As mentioned above, the BAR register in the device function configuration space defines and records the base address of the applied memory address window, and the root node accesses the control status register of the device function through the memory address assigned to the device function, thereby realizing communication with the device function . The configuration space of each device function contains 6 32-bit BAR registers, and can apply for up to 6 32-bit address spaces or 3 64-bit address spaces, or less than 6 32-bit BAR addresses and 64-bit address spaces The combination. For all functions in the same I/O device, its BAR requirement configuration is the same, that is, the number of address windows that need to be applied for, and the size of each address window is the same; only in the same PCIe domain, the BAR address of each device function The base addresses of the allocated address windows recorded are different. If FunNo is used to indicate the number of functions of a device, and the number of address windows that each function needs to apply for is indicated by BAR_id, then the entire device will be configured by the main control root node to obtain BAR_id*FuncNo different BAR addresses. Similarly, when the functions in the device are assigned to the slave root nodes, the slave root nodes allocate BAR_id BAR addresses, and different slave root nodes may assign the same BAR addresses to different functions in the same device function. In order to realize that the slave root node can directly access the device function through the address, address remapping is required, that is, the conversion of each function in the device from the address window of the slave root node PCIe domain to its address window in the master control root node PCIe domain.

Fig. 8 has described the structural diagram of realizing the address remapping module by using CAM and RAM. The address remapping structure mainly includes four parts: one is the base address definition extraction module, which obtains the I/O device function defined by each BAR by parsing the transaction package of the base address register (BAR) that configures the device function. address window type, extract the base address mask (BAR_mask) corresponding to each address window, and record the mapping relationship between each base address register number and the base address number (BAR_id) corresponding to the defined address window; the second is the function base address The serial number matching module includes a CAM storage array of BAR_id CAMs with a depth of FuncNo_m and a matching output selection submodule. Each CAM stores the base address BAR corresponding to the serial number BAR_id of the same base address of all functions FuncNo_m in the same I/O device, and realizes the slave The mapping between the base address (BAR_s) in the PCIe domain of the root node and the base address serial number (BAR_id, FuncNo_m) to which it belongs; the third is the base address search module, which includes a RAM storage unit with a depth of BAR_id*FuncNo_m. By writing and reading the RAM storage element, Realize the search of the base address (BAR_m) whose function number is FuncNo_m and whose base address serial number is BAR_id in the PCIe domain of the main control root node; the fourth is to output the decoding module, and find out the base address BAR_m according to the match and its corresponding matching base address mask The code Match_BAR_mask translates the target access address (DstAddr_s) in the PCIe domain of the slave root node into the target access address (DstAddr_m) in the PCIe domain of the master control root node. Through the relay cooperation of these four modules, the conversion of the target access address in the PCIe domain of the slave root node to its mapping in the PCIe domain of the master control root node is completed, and finally the slave root node directly accesses device functions through address routing.

The base address definition extraction module, in the initial configuration stage of the main control root node system, checks whether each BAR defines an address window by analyzing the configuration transaction package, and if an address window is defined, parses the bit width and size of the address window defined by it Type, extract the number of address windows defined by all BARs (BAR_id) and the base address mask of each address window (BAR0_mask, ..., BARn_mask), and record the base address serial number (BAR_id) corresponding to the serial number of each base address register and the address window defined mapping relationship between them. Since a 64-bit wide address window is defined by two BARs and records the base address allocated by the system, through the corresponding relationship between the base address register number and the defined address window bit width type and the base address serial number (BAR_id), you can The address sequence number records the complete base address of the corresponding address window. Figure 9 describes the base address definition extraction process, including the following steps:

a) Power on the system, initialize the start address of the BAR register offset pointer, initialize the base address serial number, initialize the 64-bit address application identifier, and turn to step b).

b) Receive the transaction package sent from the main control root node. If the transaction packet sent by the main control root node is received, go to step c).

c) Transaction package determination. Parse the received transaction packet, if the transaction packet is a configuration write request packet that writes all 1s to the register specified by the offset pointer of the BAR register, turn to step d); otherwise, turn to step b).

d) Receive the completion packet with data from the I/O device. If a completion packet with data is received from the I/O device, go to step e).

e) Address application type judgment. The data segment of the completed packet is parsed, and the address application type defined by the BAR is judged. If the BAR is not used for address application, go to step h); if the BAR defines a 64-bit address, go to step f); otherwise, go to step g).

f) Record the content of the data segment to the low L bit of the base address mask corresponding to the base address number, enable the 64-bit address application identifier, and record the current BAR register offset pointer and the address window bit width defined by the BAR indicated by it The mapping relationship between type and base address number. Go to step h).

g) Record the data segment, update the base address application serial number and the 64-bit address application identification. If the current 64-bit application identifier is not enabled, indicating that the content of the data segment is a mask of a 32-bit base address, the high L bits of the content of the data segment are recorded in the low L bits of the base address mask corresponding to the base address sequence number; Otherwise, the content of the data segment is the upper 32 bits of the encoding of the 64-bit base address corresponding to the base address serial number, and then the data segment is recorded in the upper 32 bits of the base address mask corresponding to the base address serial number. Record the mapping relationship between the current BAR register offset pointer and the address window bit width type defined by the BAR indicated by the BAR and the base address number, then update the base address number and add 1, and reset the 64-bit address application identifier, and turn to step h).

h) Update the BAR register offset pointer to point to the next BAR register, turn to step i);

i) Validity determination of BAR register offset pointer. Determine whether the register pointed to by the current BAR register offset pointer is still the BAR in the configuration space, if so, turn to b); otherwise, turn to j).

j) end. Indicates that the base address definition extraction process has been completed.

The CAM array in the function base address serial number matching module shown in Figure 8 uses the functional base address serial number (BAR_id, FunNo_m) as the address index (CAM_WrAddr), and the stored data item (CAM_Din) includes two parts: one part is the I/O device The subordinate root node ID (RNID_s) to which the function belongs, and a part is the base address (BAR_s) assigned by the subordinate root node. Wherein, the base address number (BAR_id) in the function base address number selects the CAM to be stored, and the function number (FunNo_m) in the function base address number selects the CAM storage unit to be stored. The CAM storage array has N comparison data input ports (CMP_Din0, CMP_Din1, . . . , CMP_Dinn), one for each CAM. When receiving the transaction packet based on address routing sent from the slave root node, the target access address to be accessed by the transaction packet is extracted by analyzing the data packet. Then use it to obtain BAR_id BAR base addresses (BAR0_s, BAR1_s,..., BARn_s) through the base address definition extraction module, and input them as the data input (CMP_Din) for comparison into the CAM for precise matching search, and obtain N matches Output results, including matching flags (Match_flag0, ..., Match_flagn) and matching addresses (Match_Addr0, ..., Match_Addrn). If there is (only one) matching item in the N matching output results of the CAM storage array, then the output address of the CAM is selected by the matching output selection submodule in the function base address serial number matching module as the matching output (that is, the target access address The base address corresponds to the function base address sequence number composed of the function number MatchedFunNo_m and its corresponding base address sequence number MatchedBAR_id, and the matching flag Match_flag is set to 1; otherwise, the matching flag Match_flag is set to 0.

Fig. 10 has described the process that the functional base address serial number matching module writes and stores the functional base address in the primary control root node PCIe domain, including the following steps:

a Receive the transaction packet sent from the root node, analyze and judge the root node ID of the initiating transaction packet, if the initiating root node indicated by the transaction packet is a subordinate root node, turn to step b);

b Downlink transaction packet determination. Parse the received transaction packet sent by the slave root node, if the transaction packet is based on the ID transaction packet, turn to step c);

c Downlink ID remapping. Convert the ID number in the ID-based routing transaction packet to the corresponding ID number in the PCIe domain of the main control root node, and turn to step d).

d Obtain and analyze the configuration write packet that writes non-all 1s to the BAR register in the configuration transaction packet, extract the target device function number, the offset of the BAR register to be configured, and the content of the data segment (that is, the base address of the system configuration) in the transaction packet, Go to step e);

e Define the extraction module according to the base address, obtain the base address serial number and address window bit width type corresponding to the BAR register offset, and turn to step f);

f Address window bit width type determination. Determine the address window bit width type obtained in step e), if it indicates that the lower 32 bits of the 64-bit address defined by the BAR, register the extracted base address, and then turn to step a); if the indicated is 32-bit address or 64-bit The upper 32 bits of the address, then turn to step g);

g write storage base address. If it is a 32-bit address, store the extracted data segment in the CAM unit specified by the function number and base address number; if it is a 64-bit address, store the base address registered in step f) and the extracted data segment in step d) The combination of high and low bits is stored in the CAM unit specified by the function number and base address number; turn to step a).

The RAM in the base address search module shown in Figure 8 uses the function base address serial number (BAR_id, FunNo_m) as the address index (RAM_WrAddr), and the stored data item (RAM_Din) is the base address (BAR_m) allocated by the main control root node. When the base address of the target access address obtains the matching function base address number (BAR_id, FuncNo_m) through the base address number matching module, input the read address of the RAM (RAM_RdAddr), and the corresponding RAM output (RAM_Dout) is the base address of the target access address Corresponds to the base address (BAR_m) of the main control root node PCIe domain.

Fig. 11 has described the process that the base address lookup module writes and stores the functional base address in the slave root node PCIe domain, including the following steps:

a Receive the transaction packet sent from the root node, analyze and judge the ID of the root node that initiated the transaction packet, if the originating root node indicated by the transaction packet is the main control root node, turn to step b);

b Downlink transaction packet determination. Parse the received transaction packet sent by the main control root node, if the transaction packet is a configuration transaction packet, go to step c);

c Obtain and parse the configuration write packet that writes non-all 1s to the BAR register in the configuration transaction packet, extract the target device function number, the offset of the BAR register to be configured, and the content of the data segment (that is, the base address of the system configuration) in the transaction packet, Go to step d);

d Define the extraction module according to the base address, obtain the base address serial number and address window bit width type corresponding to the BAR register offset, and turn to step e);

e Address window bit width type determination. Determine the type of address window bit width obtained in step d), if it indicates that the lower 32 bits of the 64-bit address defined by the BAR, register the extracted base address, and then turn to step 81; if it indicates a 32-bit address or a 64-bit address The upper 32 bits, then turn to step 86;

f Write storage base address. If it is a 32-bit address, store the extracted data segment in the RAM unit specified by the function number and the base address sequence number; if it is a 64-bit address, then store the base address in step 85 and the high and low bits of the data segment extracted in step 83 Combined, stored in the RAM unit specified by the function number and the base address number; turn to step a).

Figure 12 describes the address remapping process from the target access address in the slave root node PCIe domain to the master control root node PCIe domain, including the following steps:

a Receive the transaction packet sent from the root node, analyze and judge the root node ID of the initiating transaction packet, if the initiating root node indicated by the transaction packet is a subordinate root node, turn to step b);

b Downlink transaction packet determination. Parse the received transaction packet sent by the slave root node, if the transaction packet is an address-based transaction packet, go to step c);

c parses the received transaction packet based on address routing, extracts the target access address indicated in the transaction packet, and turns to step d);

d Get the base address. Perform an AND operation on the target access address and all N base address masks recorded in the base address definition extraction module to obtain N base addresses, and turn to step e);

e function base address serial number matching lookup. Use the N BAR base addresses obtained in step d) as comparison data, and input them to each CAM in the CAM array to perform parallel exact matching search, and obtain the matching function base address serial number. If there is a match, go to step f), otherwise go to step i);

f base address and base address mask matching lookup. Use the function base address serial number of matching output, read RAM, obtain described function base address serial number corresponding to the matching base address in main control root node PCIe domain; Simultaneously according to the information of base address definition extraction module record, obtain described base address The base address mask that the sequence number matches, go to step g);

g matches the target access address output decoding. By outputting the obtained matching base address, matching base address mask, and the target access address extracted in step c), the target access address in the slave root node PCIe domain can be mapped to the master control root node PCIe domain through output decoding The matching target access address. After the operation, turn to step h);

h Use the matching target access address obtained in step g) to rewrite the target access address in the transaction packet. After the operation, turn to step a);

i returns an unsupported transaction package to the root node indicated by the transaction package, and turns to step a) after the operation is completed.

In particular, for SR-IOV devices, the structure of the address remapping module described in FIG. 8 can be simplified accordingly. The VF of the SR-IOV device itself does not have the ability to apply for memory addresses. The 6 BARs in the VF configuration space header have no effect, and the addresses it needs are defined and applied for by the SR-IOV extension function in the corresponding PF configuration space. Similar to ordinary devices, there are also 6 32-bit BAR registers VF BARb (b=0,1...,5) that define and record the base address of the address window allocated by the system in the SR-IOV extended function register. Different from ordinary devices, for the size of the address window defined by VF BARb, the system will allocate a large continuous memory address for all VFs, and write its base address to VF BARb in the SR-IOV extended function register middle. In other words, for SR-IOV with VFunNo virtual functions and the number of address windows required by each VF is VFBAR_id, the system only allocates VFBAR_id base addresses to write into the VF_BARb register of the SR-IOV extended function register in the configuration space. Instead of assigning BAR_id*VFunNo base addresses, write them to the BAR register corresponding to each VF configuration space header. VF BARb records the base address of the bth BAR address of VF0, and the starting address BARb of other VFv (v=1,2,...,VFunNo-1) is passed through the formula VF BARb+(v-1)*(VF BARb Range size) is calculated.

FIG. 9 depicts a structural schematic diagram of realizing functional address remapping of SR-IOV devices by using CAM and RAM. Here, only one PF and multiple VFs of SR-IOV devices are considered. The address remapping structure mainly includes four parts: one is the base address definition extraction module, which obtains the address window defined by the VF in the device through each BAR by parsing the transaction package of the base address register (BAR) that configures the device function Type, extract the base address mask (VFBAR_mask) corresponding to each address window, and record the mapping relationship between each base address register number and the base address number (VFBAR_id) corresponding to the defined address window; the second is the function base address number matching module , including the CAM storage array of BAR_id CAMs with a depth of FunNo_m and the matching output selection submodule, each CAM stores the base address BAR corresponding to the same base address serial number BAR_id of all functions FuncNo_m in the same I/O device, and implements the slave root node PCIe The mapping between the base address (BAR_s) in the domain and its functional base address serial number (BAR_id, FuncNo_m); the third is the base address search module, which includes a RAM storage unit with a depth of VFBAR_id. By writing and reading the RAM storage element, the base address serial number is Search for the base address (BAR_m) of VFBAR_id in the PCIe domain of the main control root node; the fourth is to output the decoding module, according to the function number (MatchedFunNo_m) found by matching, the base address BAR_m and its corresponding mask MatchedVFBAR_mask, the slave root node The target access address (DstAddr_s) in the PCIe domain is translated into the target access address (DstAddr_m) in the main control root node PCIe domain.

The invention discloses a direct I/O remapping device for a multi-root I/O virtualization sharing system, comprising:

ID remapping module, the ID remapping module includes content addressable storage CAM and simple dual-port RAM, realizes the ID number of I/O device function in slave root node PCIe domain to its ID number in master control through the precise matching search of CAM The mapping of the ID number in the PCIe domain of the root node realizes the mapping from the ID number of the I/O device function in the PCIe domain of the master control root node to its ID number in the PCIe domain of the slave root node by writing and reading the RAM storage element;

Address remapping module, the address remapping module is used to realize the conversion of the I/O device function in the I/O mapping address window of the subordinate root node PCIe domain to its I/O mapping address window in the main control root node PCIe domain ;

Wherein, the ID number includes a bus number, a device number, and a function number.

In the direct I/O remapping device for a multi-root I/O virtualization sharing system, the CAM and RAM in the ID remapping module include:

Using the function number of the I/O device function as the address index, the data item stored in the storage unit includes two parts: one part is the ID of the root node to which the device function belongs, and the other part is the function of the device function in the PCIe domain of the root node to which it belongs Number.

In the I/O remapping device for a multi-root I/O virtualization sharing system, the address remapping module includes:

The base address definition extraction module, the base address mask obtains the address window type defined by each BAR for the I/O device function by parsing the transaction packet of the base address register (BAR) that configures the device function, and extracts each address The base address mask corresponding to the window, and record the mapping relationship between each base address register number and the base address number corresponding to the defined address window;

Functional base address serial number matching module, the functional base address serial number matching module includes a CAM storage array composed of N CAMs with a depth of P and a matching output selection module, each CAM stores the same base address in all functions in the same I/O device The base address BAR corresponding to the serial number realizes the mapping between the base address in the PCIe domain of the slave root node and its corresponding functional base address serial number;

Base address search module, said base address search module includes a depth of Q simple dual-port RAM, by writing and reading the RAM storage element, realizes the mapping of the function base address sequence number to its base address in the main control root node PCIe domain;

Output decoding module, said output decoding module translates the target access address in the slave root node PCIe domain into the master control root Target access address in the node PCIe domain;

Among them, N (N<=6) represents the number of address windows applied for by the device function, P represents the number of functions in the device, and Q represents the number of base addresses of the address windows allocated by the system to all functions in the device.

In the direct I/O remapping device for a multi-root I/O virtualization sharing system, the function base address serial number matching module includes:

The CAM storage array is based on the function base address serial number as the address index, and the stored data items include two parts: one part is the subordinate root node ID to which the I/O device function belongs, and one part is the base address allocated by the subordinate root node, wherein, The base address sequence number in the function base address sequence number selects the CAM to be stored, and the function number in the function base address sequence number selects the CAM storage unit to be stored. The CAM storage array has N comparison data input ports, and each CAM corresponds to one. In the data matching stage, the CAM array inputs N different comparison data items obtained by extracting the target access address of the subordinate root node through N different base address masks, performs matching search in parallel, and outputs N corresponding matching data items. result;

The matching output selection module, if there is (only one) matching item in the N matching results output by the CAM storage array, then select and output the output matching address (that is, the function number p and its corresponding base address) of the CAM that has the matching item The sequence number of the function base address composed of sequence number n, and set the matching flag to 1; otherwise, set the matching flag to 0.

In the direct I/O remapping device for a multi-root I/O virtualization sharing system, the RAM in the base address search module includes:

The function base address serial number is used as the address index, and the stored data item is the base address assigned by the main control root node.

Various modifications can be made to the above contents by those skilled in the art without departing from the spirit and scope of the present invention defined by the claims. Therefore, the scope of the present invention is not limited to the above description, but is determined by the scope of the claims.

Claims (18)

1. a direct I/O who is used for the many virtual shared systems of I/O remaps method, it is characterized in that, comprising:

Step 1, the definition of I/O functions of the equipments base address is extracted, and writes down the mapping relations between each base address register sequence number base address sequence number corresponding with the address window of its definition;

Step 2, storage is write in the functional group address in the main control root node PCIe territory, sets up functional group address sequence number to its mapping in the base address in main control root node PCIe territory;

Step 3, functions of the equipments ID mapping relations are write storage, the mapping between ID number in main control root node PCIe territory and subordinate root node PCIe territory of apparatus for establishing function;

Step 4, storage is write in the functional group address in the subordinate root node PCIe territory, sets up the mapping of the base address in subordinate root node PCIe territory to its corresponding functional group address sequence number;

Step 5, directly I/O remaps, and the transaction packet of subordinate root node and I/O equipment room is remapped operation.

2. the direct I/O that is used for the many virtual shared systems of I/O as claimed in claim 1 remaps method, it is characterized in that said step 1 also comprises:

It is in the main control root node system discovery stage that the definition of said base address is extracted; Obtain in the I/O functions of the equipments configuration space through each base address register (Base Address Register; BAR) the address window type of definition, and extract the corresponding base address mask of each address window;

Said base address mask, which position that is used in reference to address in the fixed system configured address window is a fixed bit, can extract corresponding base address, memory access address through said base address mask;

According to the tactic sequence number of BAR register number, each function of the same type in I/O equipment is identical with number to the address window type of system's application to the address window number of system's application for said base address sequence number, the function that is used for indication equipment.

3. the direct I/O that is used for the many virtual shared systems of I/O as claimed in claim 1 remaps method, it is characterized in that said step 2 also comprises:

It is at main control root node system initialization configuration phase that storage is write in functional group address in the said main control root node PCIe territory; Through resolving the configuration transaction bag; The functional group address sequence number of indicating in the extraction transaction packet is write the corresponding base address of sequence number, the said functional group of storage address as allocation index.

Said functional group address sequence number is combined by the functions of the equipments p in base address sequence number k and the main control root node PCIe territory, and (k, p) base address of indication is that system is that function number is n base address of p distribution in the equipment to functional group address sequence number.

4. the direct I/O that is used for the many virtual shared systems of I/O as claimed in claim 1 remaps method, it is characterized in that said step 3 also comprises:

It is to carry out equipment I/O resources allocation stage at the main control root node that said functions of the equipments ID mapping relations are write storage; Distribute transaction packet through analyzing device; Extract the functions of the equipments of indicating in the transaction packet in the function in main control root node PCIe territory number as allocation index; Write the data item that storage number is made up of the root node under said function sign ID and the function of said function in affiliated root node PCIe territory, the mapping between in subordinate root node PCIe territory ID number of apparatus for establishing function in main control root node PCIe territory ID number and its.

5. the direct I/O that is used for the many virtual shared systems of I/O as claimed in claim 1 remaps method, it is characterized in that said step 4 also comprises:

It is in the subordinate root node PCIe of the system rescanning stage that storage is write in functional group address in the said subordinate root node PCIe territory; Through resolving the configuration transaction bag; The functional group address sequence number of indicating in the extraction transaction packet is write the corresponding base address of sequence number, the said functional group of storage address as allocation index.

6. the direct I/O that is used for the many virtual shared systems of I/O as claimed in claim 1 remaps method, it is characterized in that said step 5 also comprises:

It is in the process that subordinate root node and I/O equipment communicate that said direct I/O remaps; The descending transaction packet based on the ID route of subordinate root node initiation is carried out descending ID to remap; The descending transaction packet based on the address route of subordinate root node initiation is carried out the address remap, the transaction packet of functions of the equipments initiation is carried out up ID remap.

7. the direct I/O that is used for the many virtual shared systems of I/O as claimed in claim 1 remaps method, it is characterized in that said step 1 also comprises:

Step 71, system powers on, the start address of initialization BAR register offset pointer, initialization base address sequence number, initialization 64 bit address application identifiers turn to step 72;

Step 72 receives the transaction packet of sending from the main control root node; If receive the transaction packet that the main control root node is sent, turn to step 73;

Step 73, transaction packet is judged; The transaction packet that parsing receives if this transaction packet is to the configurable write request package of the register write complete 1 of BAR register offset pointer appointment, turns to step 74; Otherwise turn to step 72;

Step 74, the completion bag of the band data that reception I/O equipment is sent; If receive the completion bag of the band data of sending from I/O equipment, turn to step 75;

Step 75, the application IP addresses type is judged; Resolve the data segment of accomplishing bag, the application IP addresses type of this BAR definition is judged; If this BAR is not used for application IP addresses, then turn to step 78; If this BAR definition is 64 bit address, then turn to step 76; Otherwise, turn to step 77;

Step 76; Low L position with data segment content record base address mask of sequence number correspondence to the base address; Enable 64 bit address application identifiers, and write down address window bit wide type and the mapping relations of base address sequence number of the BAR definition of current BAR register offset pointer and indication thereof, turn to step 78;

Step 77, the record data section is upgraded base address sequence number and 64 bit address application identifiers; If 64 current application identifiers do not enable, designation data section content is the mask of 32 base address, then the high L position of data segment content is recorded the low L position of the corresponding base address mask of base address sequence number; Otherwise the data segment content is mask high 32 of 64 corresponding base address of base address sequence number, then data segment is recorded the high 32 of the corresponding base address mask of base address sequence number; Write down address window bit wide type and the mapping relations of base address sequence number of the BAR definition of current BAR register offset pointer and indication thereof, upgrade the base address sequence number then and add 1, and the 64 bit address application identifiers that reset, turn to step 78;

Step 78 is upgraded BAR register offset pointer and is pointed to next BAR register, turns to step 79;

Step 79, BAR register offset pointer availability deciding; Judge whether the register that current BAR register offset pointer is pointed to still is the BAR in the configuration space, if then turn to step 72; Otherwise turn to step 710;

Step 710 finishes, and sign has been accomplished base address definition leaching process.

8. the direct I/O that is used for the many virtual shared systems of I/O as claimed in claim 1 remaps method, it is characterized in that said step 2 also comprises:

Step 81 receives the transaction packet of sending from root node, and the root node ID that initiates transaction packet is resolved judgement, if the initiation root node of transaction packet indication is the main control root node, turns to step 82;

Step 82, descending transaction packet is judged; The transaction packet of being sent by the main control root node that parsing receives if this transaction packet is the configuration transaction bag, turns to step 83;

Step 83; Obtain and resolve in the configuration transaction bag configurable write bag to BAR register write non-complete 1; The base address that to extract the target device function number in the transaction packet, the BAR register skew that will dispose and the said data segment content of data segment content be system configuration turns to step 84;

Step 84 according to the mapping relations that base address definition leaching process obtains, is obtained corresponding base address sequence number and the address window bit wide type of BAR register skew, turns to step 85;

Step 85, address window bit wide type decision; The address window bit wide type that determination step 84 obtains if that indicate this BAR definition is low 32 of 64 bit address, is then deposited the base address of extraction, turns to step 81 then; If indication is 32 high 32 for address or 64 bit address, then turn to step 86;

Step 86 is write the storage base address; If 32 bit address, then the data segment that extracts is stored into by in function number and the base address sequence number designated memory locations; If 64 bit address, the data segment height bit pattern that base address of then step 85 being deposited and step 83 are extracted is stored into by in function number and the base address sequence number designated memory locations; Turn to step 81.

9. the direct I/O that is used for the many virtual shared systems of I/O as claimed in claim 1 remaps method, it is characterized in that said step 4 also comprises:

Step 91 receives the transaction packet of sending from root node, and the root node ID that initiates transaction packet is resolved judgement, if the initiation root node of transaction packet indication is the subordinate root node, turns to step 92;

Step 92, descending transaction packet is judged; The transaction packet of being sent by the subordinate root node that parsing receives if this transaction packet is based on the ID transaction packet, turns to step 93;

Step 93, descending ID remaps; To turn to step 94 based on converting ID number in main control root node PCIe territory of its correspondence for the ID in the transaction packet of ID route number into;

Step 94 is obtained and resolves in the configuration transaction bag configurable write bag to BAR register write non-complete 1, extracts the target device function number in the transaction packet, the BAR register skew that will dispose and the base address of data segment content indication, turns to step 95;

Step 95 according to the mapping relations that base address definition leaching process obtains, is obtained corresponding base address sequence number and the address window bit wide type of BAR register skew, turns to step 96;

Step 96, address window bit wide type decision; The address window bit wide type that determination step 95 obtains if that indicate this BAR definition is low 32 of 64 bit address, is then deposited the base address of extraction, turns to step 91 then; If indication is 32 high 32 for address or 64 bit address, then turn to step 97;

Step 97 is write the storage base address; If 32 bit address, then the data segment that extracts is stored into by in function number and the base address sequence number designated memory locations; If 64 bit address, the data segment height bit pattern that base address of then step 96 being deposited and step 94 are extracted is stored into by in function number and the base address sequence number designated memory locations; Turn to step 91.

10. the direct I/O that is used for the many virtual shared systems of I/O as claimed in claim 1 remaps method, it is characterized in that said step 5 comprises:

Said descending ID remaps, refer to the I/O functions of the equipments at ID number of subordinate root node PCIe territory to its coupling conversion of ID number in main control root node PCIe territory;

Said address remaps, and refers to the coupling conversion to the address of its correspondence in main control root node PCIe territory of target access address that the subordinate root node sends;

Said up ID remaps, refer to the I/O functions of the equipments at ID number of main control root node PCIe territory to its coupling conversion of ID number in subordinate root node PCIe territory.

11. the direct I/O that is used for the many virtual shared systems of I/O as claimed in claim 10 remaps method, it is characterized in that, said descending ID remaps and comprises:

Step 111 receives the transaction packet of sending from root node, and the root node ID that initiates transaction packet is resolved judgement, if the initiation root node of transaction packet indication is the subordinate root node, turns to step 112;

Step 112, the transaction packet type decision is if based on the transaction packet of ID route, turn to step 113;

Step 113; Extract the initiation root node ID that indicates in the transaction packet with and the functions of the equipments that will visit number; And with its combination as the data content that will compare, exact-match lookup obtains the function of these functions of the equipments in main control root node PCIe territory number; If there is occurrence, turn to step 114; Otherwise turn to step 115;

Step 114 is rewritten in the transaction packet functions of the equipments ID number, the address of using the output of CAM coupling, i.e. and function among the main control root node PCIe of indication number is rewritten in the transaction packet functions of the equipments ID number, turns to step 111 after having operated;

Step 115, the root node of indicating to transaction packet returns a unsupported transaction packet, and operation turns to step 111 after accomplishing.

12. the direct I/O that is used for the many virtual shared systems of I/O as claimed in claim 10 remaps method, it is characterized in that said address remaps and comprises:

Step 121 receives the transaction packet of sending from root node, and the root node ID that initiates transaction packet is resolved judgement, if the initiation root node of transaction packet indication is the subordinate root node, turns to step 122;

Step 122, descending transaction packet is judged; The transaction packet of being sent by the subordinate root node that parsing receives if this transaction packet is based on the address transaction packet, turns to step 123;

Step 123 is resolved the transaction packet based on the address route that receives, and extracts the target access address of indicating in the transaction packet, turns to step 124;

Step 124 is obtained the base address; All K base address masks that write down in target access address and the base address definition leaching process are carried out and operation, obtain K base address, turn to step 125;

Step 125, functional group address sequence number matched and searched; K the BAR base address that step 124 is obtained be as comparing data, and the exact-match lookup that walks abreast obtains the functional group address sequence number of target access matching addresses; If there is occurrence, turns to step 126, otherwise turn to step, 129;

Step 126, base address and base address mask matches are searched; Use the functional group address sequence number of coupling output, obtain the coupling base address of said functional group address sequence number correspondence in main control root node PCIe territory and the base address mask of base address sequence number coupling, turn to step 127;

Step 127, the target access address output decoding of coupling; According to the coupling base address that obtains, coupling base address mask,, obtain the coupling target access address of target access map addresses in the main control root node PCIe territory in the subordinate root node PCIe territory together with the target access address that step 123 is extracted; Turn to step 128 after having operated;

Step 129, use that step 127 obtains coupling target access address, rewrite target access address in the transaction packet; Turn to step 121 after having operated;

Step 129, the root node of indicating to transaction packet returns a unsupported transaction packet, and operation turns to step 121 after accomplishing.

13. the direct I/O that is used for the many virtual shared systems of I/O as claimed in claim 10 remaps method, it is characterized in that, said up ID remaps and comprises:

Step 131 receives the transaction packet of sending from the I/O functions of the equipments, if receive transaction packet, turns to step 132;

Step 132, the I/O functions of the equipments of extraction transaction packet indication ID number; If transaction packet is a request package, the request person of establishing who indicates in extraction request transaction bag ID number; If if transaction packet is to accomplish bag, extracts and accomplish completion person ID number that indicates in the transaction packet, turn to step 133;

Function in the step 133, I/O functions of the equipments of use extracting ID number is number as allocation index, obtains corresponding root node sign ID and function number; If the non-main control root node of the root node that obtains sign ID ID turns to step 134; Otherwise, explain that the root node under the transaction packet is the main control root node, need not carry out the ID conversion, turn to step 135;

Step 134 uses the root node sign ID that obtains to record in the transaction packet; If transaction packet is a request package, the function that then use to obtain number is revised in the transaction packet requestor ID number; If transaction packet for accomplishing bag, then uses the function that obtains number to revise in the transaction packet completion person ID number, turn to step 131 after having operated;

Step 135 uses main control root node sign ID to rewrite transaction packet; The sign ID of main control root node is recorded in the transaction packet, turn to step 131 after having operated.

14. a direct I/O who is used for the many virtual shared systems of I/O remaps device, it is characterized in that, comprising:

ID remaps module; Said ID remaps module content addressable storage CAM and simple two-port RAM; Exact-match lookup through CAM is realized ID number in subordinate root node PCIe territory of I/O functions of the equipments to its mapping of ID number in main control root node PCIe territory, realizes that through write-read RAM memory element ID number in main control root node PCIe territory of I/O functions of the equipments is to its mapping of ID number in subordinate root node PCIe territory;

The address remaps module, and said address remaps module and is used to realize that the I/O functions of the equipments arrive its conversion at the I/O mapping address window in main control root node PCIe territory at the I/O mapping address window in subordinate root node PCIe territory;

Wherein, comprise Bus number, device number, function number for ID number.

15. the direct I/O that is used for the many virtual shared systems of I/O as claimed in claim 14 remaps device, it is characterized in that, CAM and RAM that said ID remaps in the module comprise:

Number as allocation index, the data item of cell stores comprises two parts: a part is the root node sign ID under these functions of the equipments with the function of I/O functions of the equipments, and a part is the function number in the root node PCIe territory of these functions of the equipments under it.

16. the I/O that is used for the many virtual shared systems of I/O as claimed in claim 14 remaps device, it is characterized in that, said address remaps module and comprises:

Base address definition extraction module; Said base address mask is through resolving base address register (the Base Address Register of configuration device function; BAR) transaction packet; Obtain the address window type of I/O functions of the equipments, extract the corresponding base address mask of each address window, and write down the mapping relations between each base address register sequence number base address sequence number corresponding with the address window of its definition through each BAR definition;

Functional group address sequence number matching module; CAM storage array that the CAM that it is P that said functional group address sequence number matching module comprises N the degree of depth forms and coupling output selection module; Each CAM stores in the same I/O equipment base address BAR of same base address sequence correspondence in all functions, realizes the mapping of functional group pairing with it address, the base address sequence number in the subordinate root node PCIe territory;

Module is searched in the base address, and said base address is searched module and comprised the simple two-port RAM that the degree of depth is Q, through write-read RAM memory element, realizes the mapping of functional group address sequence number to its base address in main control root node PCIe territory;

The output decoding module; Base address and corresponding base address mask thereof become the target access address in the main control root node PCIe territory with the target access address translation in the subordinate root node PCIe territory in the main control root node PCIe territory that said output decoding module goes out according to matched and searched;

Wherein, the address window number of N (N ≤6) indication equipment function application, the number of function in the P indication equipment, Q represent that system assignment gives the number of the base address of the address window of all functions in the equipment.

17. the direct I/O that is used for the many virtual shared systems of I/O as claimed in claim 16 remaps device, it is characterized in that, said functional group address sequence number matching module comprises:

Said CAM storage array is to be allocation index with functional group address sequence number, and stored data items comprises two parts: a part is the subordinate root node ID under the I/O functions of the equipments; A part is the base address that the subordinate root node distributes; Wherein, the base address sequence number in the sequence number of functional group address is selected the CAM that will store, and the function in the sequence number of functional group address number is selected the CAM storage unit that will store; Said CAM storage array has N comparing data input port; Corresponding one of each CAM, said cam array are in the Data Matching stage, and input is extracted N the different comparing data item that obtains by the target access address of subordinate root node through N different base address mask; Carry out matched and searched concurrently, export N corresponding matching result;

Module is selected in said coupling output; If there is (only having one) occurrence in N matching result of CAM storage array output; Then select output by the output match address of the CAM that has occurrence (be function p with and the functional group address sequence number formed of corresponding base address sequence number n, simultaneously matching identification is put 1; Otherwise, matching identification is put 0.

18. the direct I/O that is used for the many virtual shared systems of I/O as claimed in claim 16 remaps device, it is characterized in that the RAM that search in the module said base address comprises:

With functional group address sequence number is allocation index, and stored data items is the base address that the main control root node distributes.

Images