WO2022012309A1 - 扩展PCIe系统的方法、PCIe交换设备及PCIe系统 - Google Patents

扩展PCIe系统的方法、PCIe交换设备及PCIe系统 Download PDF

Info

Publication number
WO2022012309A1
WO2022012309A1 PCT/CN2021/102622 CN2021102622W WO2022012309A1 WO 2022012309 A1 WO2022012309 A1 WO 2022012309A1 CN 2021102622 W CN2021102622 W CN 2021102622W WO 2022012309 A1 WO2022012309 A1 WO 2022012309A1
Authority
WO
WIPO (PCT)
Prior art keywords
port
pcie
downstream
proxy
processor
Prior art date
Application number
PCT/CN2021/102622
Other languages
English (en)
French (fr)
Inventor
刘兴强
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from CN202011620807.7A external-priority patent/CN114006875A/zh
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to EP21842488.5A priority Critical patent/EP4167100A4/en
Publication of WO2022012309A1 publication Critical patent/WO2022012309A1/zh
Priority to US18/150,991 priority patent/US20230161728A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
    • G06F13/38—Information transfer, e.g. on bus
    • G06F13/40—Bus structure
    • G06F13/4004—Coupling between buses
    • G06F13/4022—Coupling between buses using switching circuits, e.g. switching matrix, connection or expansion network
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
    • G06F13/38—Information transfer, e.g. on bus
    • G06F13/42—Bus transfer protocol, e.g. handshake; Synchronisation
    • G06F13/4204—Bus transfer protocol, e.g. handshake; Synchronisation on a parallel bus
    • G06F13/4221—Bus transfer protocol, e.g. handshake; Synchronisation on a parallel bus being an input/output bus, e.g. ISA bus, EISA bus, PCI bus, SCSI bus
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2213/00—Indexing scheme relating to interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
    • G06F2213/0026—PCI express

Definitions

  • the present application relates to the field of computer technologies, and in particular, to a method for extending a PCIe system, a PCIe switching device, and a PCIe system.
  • Peripheral component interconnect bus (peripheral component interconnect express, PCIe) is a high-speed serial computer expansion bus standard used to realize the connection between processors and external devices, which are called PCIe devices.
  • the PCIe protocol uses a 32-bit PCIe ID to identify PCIe devices in the system.
  • the 32 bits include: domain(16bits):bus(8bits):device(5bits).function(3bits)).
  • Bus ID provides 256 bus numbers for each PCIe domain.
  • a computer system typically supports one PCIe domain, that is, the entire computer system only provides 256 bus numbers. Since the PCIe protocol is a point-to-point connection, the scale of PCIe devices that can be supported by a PCIe domain is limited by the number of 256 bus numbers, and more PCIe devices cannot be supported in a PCIe domain.
  • the present application provides a method for extending a PCIe system, a PCIe switching device and a PCIe system, so as to solve the problem in the prior art that the scale of the PCIe device is restricted by the number of 256 bus numbers.
  • the present application provides a PCIe switch device, the PCIe switch device includes an upstream port, a port proxy device, and a plurality of first downstream ports, wherein the upstream port is connected to the port proxy device, and the The port proxy device is connected to the plurality of first downstream ports, and the first downstream ports are used for connecting PCIe endpoint devices; the connections between the plurality of first downstream ports and the PCIe endpoint devices below share the same bus number.
  • the switching device of the first aspect includes a port proxy device, which is connected to a plurality of first downstream ports, and provides a proxy for the plurality of first downstream ports, by connecting the plurality of first downstream ports to the PCIe endpoint devices thereunder. Sharing the same bus number reduces the number of bus numbers occupied by PCIe endpoint devices and increases the capacity of the PCIe system.
  • the downstream bus number of the port proxy device is equal to the home bus number, where the home bus number is used to represent the largest bus number in the PCIe subtree under the port proxy device.
  • the PCIe switching device further includes one or more second downstream ports, wherein the one or more second downstream ports are connected to the upstream ports.
  • the second downstream port is connected to the upstream port and does not require proxy services provided by the port proxy device.
  • system enumeration software eg, Basic Input Output System/Operating System, BIOS/OS
  • BIOS/OS Basic Input Output System/Operating System
  • the bus number of the management topology of the bridge device includes the upstream bus number (Primary Bus Number), the downstream bus (Secondary Bus Number) and the subordinate bus number (Subordinate Bus Number).
  • the downstream bus number of the port proxy device is equal to the home bus number.
  • the downstream bus number of the port agent device may be smaller than the home bus number, and at the same time, the value of the home bus number minus the downstream bus number is less than the number of the aforementioned first downstream ports. At this time, some ports in the plurality of first downstream ports share the same downstream bus number.
  • the port proxy device and the plurality of first downstream ports have the same home bus number. Specifically, the port proxy device sets the management topology bus number of the plurality of first downstream ports under it to be the same as the management topology bus number of its own.
  • the serial number of the first downstream port is used to represent the device identifier of the PCIe endpoint device connected to the first downstream port.
  • the processor may use the identifier of the first downstream port to which the PCIe endpoint device is connected to distinguish each endpoint device under the port proxy apparatus.
  • the processor identifies the PCIe endpoint device based on the number of the first downstream port in combination with the bus number D and the function number F.
  • the port proxy apparatus is configured to receive a data packet sent by a PCIe endpoint device from the first downstream port, and replace the device identifier in the data packet with the first downstream port 's number.
  • the port proxy device is specifically configured to replace the device identifier in the requester BDF carried in the request TLP packet with the first device connected to the PCIe endpoint device. the number of the downstream port, or,
  • the port proxy device is specifically configured to replace the device identifier in the completer BDF carried in the completion TLP packet with the first downstream device connected to the PCIe endpoint device The number of the port.
  • the port proxy device is configured to receive the data packet sent by the processor, and use the value of the device identifier in the data packet as the number of the destination port for receiving the data packet, and The device identifier in the data packet is changed to 0, and the data packet is sent to the destination port.
  • the device identifier in the requester BDF carried in the completion TLP packet is replaced with 0.
  • the configuration request TLP package includes a configuration write request TLP package and a configuration read request TLP package.
  • the device identifier of the PCIe endpoint device connected to the first downstream port is replaced with the serial number of the first downstream port.
  • the device identifier assigned by the PCIe system to the PCIe endpoint device is generally 0.
  • the port proxy device changes the device identifier in the data packet sent to the PCIe endpoint device to 0, so that the PCIe endpoint device does not need to change the software configuration for the solution of the present application .
  • the configuration space registers of all the first downstream ports belonging to the port proxy device are mapped in the configuration space registers of the port proxy device.
  • the port proxy device can access the configuration space register of the first downstream port.
  • the value of the enable register of the port proxy device is used to indicate whether the proxy function of the port proxy device is enabled.
  • the PCIe switching device can switch the proxy function on and off, and the solution of the present application is implemented when the proxy function is turned on, and when the proxy function is turned off, the traditional PCIe switching device uses the prior art to provide services.
  • the register of the port agent device is set by firmware or by reading a non-volatile memory (eg, a power-erasable programmable read-only memory EEPROM). .
  • a non-volatile memory eg, a power-erasable programmable read-only memory EEPROM.
  • the port proxy device is configured to receive a first configuration request forwarded by the upstream port, where the first configuration request carries each bus number corresponding to the management topology of the port proxy device ;
  • the port proxy device is further configured to set each bus number corresponding to its own management topology according to the first configuration request, and set the management topology of all the first downstream ports connected under it to the same bus number.
  • the port proxy device By setting all the first downstream ports under the port proxy device to have the same management topology bus number as the port proxy device, the port proxy device acts as a proxy for all the first downstream ports under it, and sends all the first downstream ports under it to the upper-layer system.
  • the downstream port is simulated as a proxy port.
  • the port proxy device is configured to receive a second configuration request sent by the processor, where the bus number carried in the second configuration request is the lower-level bus number of the port proxy device, and the bus number carried in the second configuration request
  • the device identifier is m, where m is an integer greater than or equal to 0; the port proxy device is further configured to route the second configuration request to the corresponding destination according to the value m of the device identifier carried in the second configuration request port, the number of the destination port is equal to m.
  • the processor can enumerate all the first downstream ports under the port proxy device to the system by adjusting the value of m.
  • the PCIe switching device further includes: one or more second downstream ports, and the port proxy device is configured to determine, according to the port enablement record, that the plurality of first downstream ports are proxy-enabled functionality, and determining that the one or more second downstream ports do not have proxy functionality enabled.
  • the port enable record may be recorded in a bitmap register, and the bitmap value recorded in the bitmap register corresponding to each downstream port of the PCIe switch device indicates whether the endpoint device connected to each downstream port has enabled the agent Features.
  • the port enable record may also be a mapping table.
  • the address window of the port proxy device is a collection of address windows of all first downstream ports connected thereunder.
  • the port proxy device receives the data packet in the address routing manner, it can determine whether the data packet belongs to the address range of the first downstream port under it through the address window.
  • the port proxy device performs an addressing operation for the corresponding address.
  • the port proxy device is further configured to receive a message message reported by the first downstream port under it, and replace the BDF of the first downstream port carried in the message message For its own BDF, route the message packet to the upstream port.
  • the message message includes an error message.
  • the error message carries error information of the first downstream port.
  • the port proxy device is further configured to record the status information of the first downstream port that sends the error message.
  • the message message also includes an interrupt message.
  • the port proxy apparatus modifies the BDF in the message packet sent by the first downstream port, so that the first downstream port is invisible to the upper-layer system.
  • the sum of the number of bits of the device identification of the PCIe endpoint device and the number of bits of the function identification is 8.
  • the PCIe switching device is an application specific integrated circuit (Application Specific Integrated Circuit, ASIC) chip.
  • ASIC Application Specific Integrated Circuit
  • the PCIe switching device includes a plurality of parallel port proxy devices, and each port proxy device is connected to a group of the first downstream ports.
  • each port proxy device provides proxy services for a group of first downstream ports under it, and the downstream bus numbers of the first downstream ports in each group are different from the downstream bus numbers of the first downstream ports in other groups.
  • the present application provides another PCIe switching device, including: a processor, a memory, an upstream port, and a plurality of first downstream ports, wherein the memory stores instructions, and when the processor executes the instructions, the foregoing is implemented
  • the port proxy device described in the first aspect.
  • the present application provides a PCIe system, including a processor and the PCIe switching device according to any one of claims 1-19.
  • the PCIe switching device includes a first PCIe switching device and a second PCIe switching device, wherein an upstream port of the second PCIe device is connected to a second downstream port of the first switching device port, the proxy function is not enabled on the second downstream port.
  • the present application provides another PCIe system, the PCIe system includes a first PCIe switching device and a second PCIe switching device, wherein the first PCIe switching device is the PCIe switching device of the first aspect , the second PCIe switching device is a traditional PCIe switching device, that is, the second PCIe device does not include a port proxy device.
  • the second PCIe switching device is an upstream or downstream device of the first PCIe switching device.
  • the present application provides a processor chip, the processor chip comprising a root complex RC and the PCIe switch device according to the preceding aspects, an upstream port of the PCIe switch device is connected to the root complex The root port of the body RC, the PCIe switch device is used to implement the functions of the PCIe switch device in the foregoing aspects.
  • the present application provides a method for extending a PCIe system
  • the PCIe system includes a processor and a PCIe switching device
  • the PCIe switching device includes: an upstream port, a port proxy device, and a plurality of first downstream ports, wherein ,
  • the port proxy device is connected to the processor through the upstream port, the port proxy device is connected to the plurality of first downstream ports, and the first downstream ports are used for connecting PCIe endpoint devices,
  • the method includes:
  • the processor sets the connections between the plurality of first downstream ports and the PCIe endpoint devices below them to the same bus number through the port proxy device.
  • the system enumeration software enumerates the PCIe endpoint devices in the PCIe system, and sets the bus numbers of the connections between the plurality of first downstream ports and the PCIe endpoint devices thereunder during the enumeration process.
  • the present application provides a computer program product, the computer program product includes a computer program stored in a computer-readable storage medium, and the computer program is loaded by a processor to implement the first aspect or the first aspect above.
  • the computer program product includes a computer program stored in a computer-readable storage medium, and the computer program is loaded by a processor to implement the first aspect or the first aspect above.
  • the present application provides a computer-readable storage medium for storing a computer program, where the computing program is loaded by a processor to implement the functions of the above aspect or any possible implementation manner of each aspect.
  • Fig. 1 is a schematic diagram of bus number allocation in a PCIe system
  • FIG. 2 is a schematic diagram of a PCIe device management topology corresponding to FIG. 1;
  • FIG. 3 is a schematic diagram of bus number allocation in a PCIe system provided by an embodiment of the present application.
  • FIG. 4 is a schematic diagram of a PCIe device management topology corresponding to FIG. 3;
  • FIG. 5 is a schematic diagram of a DPA configuration space provided by an embodiment of the present application.
  • FIG. 6 is a schematic diagram of bus number allocation in another PCIe system provided by an embodiment of the present application.
  • FIG. 7 is a schematic diagram of a PCIe device management topology corresponding to FIG. 6;
  • FIG. 8 is a schematic diagram of bus number allocation in another PCIe system provided by an embodiment of the present application.
  • 9A and 9B are schematic flowcharts of enumerating PCIe devices according to an embodiment of the present application.
  • FIG. 10 is a schematic diagram of address window allocation provided by an embodiment of the present application.
  • FIG. 11 is a schematic flowchart of a method for ID routing provided by an embodiment of the present application.
  • FIG. 12 is a schematic diagram of an apparatus structure of a PCIe switching device 1200 provided by an embodiment of the present application.
  • the PCIe bus As a local bus of the processor system, the PCIe bus is similar in function to the PCI bus, and its main purpose is to connect external devices in the processor system.
  • the PCIe bus uses an end-to-end connection method. Only one device can be connected to each end of a PCIe link. These two devices are each other's data transmitter and data receiver.
  • a typical PCIe system includes a root complex (Root Complex, RC), a PCIe switch device (ie, a PCIe switch), and a PCIe endpoint device.
  • the RC can be located on the processor, and the RC itself occupies a bus number.
  • a PCIe switch includes an upstream port and a downstream port. The upstream port is used to connect the root port (root port, RP) of the RC or the downstream port of other PCIe switches.
  • the root port of the RC, the upstream port and the downstream port of the PCIe switch can also become bridge devices, which are used to connect other bridge devices or endpoint devices.
  • a PCIe switch can be a separate physical device, such as a physical switch or a switch chip; it can also be a hardware function module implemented in the chip, for example, it can be a hardware function module integrated in a processor chip to implement PCIe switch function.
  • FIG. 1 it is an exemplary schematic diagram of bus number allocation in a PCIe system, wherein, there are multiple root ports (Root Port, RP) on the processor CPU.
  • the RP may be located on one or more RCs (not shown in the figure).
  • RP0 can be connected to an endpoint device or a bridge device (not shown in the figure)
  • RP1 is connected to an upstream port (Uplink Port, UP) of a PCIe switch (switch), and the PCIe switch includes m+1 downstream ports (Downlink Port, UP).
  • Port, DP marked as DP_0, DP_1, .
  • the root port RP1 connected to the upstream port UP occupies bus X, and the connection (link) between RP1 and UP occupies bus N.
  • the connection between the internal upstream port of the PCIe switch and the m+1 downstream ports occupies bus N+1, and the connection between each downstream port and the corresponding endpoint device occupies bus N+2, bus N+3, ..., bus respectively.
  • N+2+m is an integer greater than or equal to 0
  • N is an integer greater than X.
  • the value of X depends on the bus number occupied by the PCIe structure under the left RC, for example , when the PCIe structure under the left RC occupies 5 bus numbers (that is, bus 0-4 is occupied by the left RC and the PCIe structure below it), the value of X is 5, and the value of N is 6; when When the two RPs shown in the figure are located in the same RC and the RC is the first RC enumerated by the processor, the value of X is 0. If the PCIe structure under the RC on the left occupies 5 bus numbers (that is, bus 0-4 are When the PCIe structure under RC and RP0 is occupied), the value of N is 5 at this time.
  • PCIe devices can be divided into two types, one is a bridge device, such as a root port RP, an upstream port, and a downstream port, and the other is an endpoint device (Endpoint, EP). At both ends of the PCI bridge, two buses are connected, namely the upstream bus (Primary Bus) and the downstream bus (Secondary Bus).
  • a PCIe bridge can manage the PCIe subtree under it.
  • the bridge device has Subordinate Bus Number, Secondary Bus Number and Primary Bus Number registers.
  • the Primary Bus Number register stores the PCIe bus number upstream of the bridge device
  • the Secondary Bus Number register stores the PCIe bus number downstream of the bridge device
  • the Subordinate Bus Number register stores the The PCIe bus number with the highest number in the PCIe subtree downstream of the bridge device, the Subordinate Bus Number is called the slave bus number.
  • FIG. 2 it is a schematic diagram of a PCIe device management topology corresponding to FIG. 1 .
  • the Primary Bus Number of RP1 is X
  • the BDF is X:0.0
  • the Secondary Bus Number is N
  • the Subordinate Bus Number is N+2+m.
  • the Primary Bus Number of the PCIe switch upstream port UP connected to the RP1 is N
  • the BDF is N: 0.0
  • the Secondary Bus Number is N+1
  • the Subordinate Bus Number is N+2+m.
  • the Primary Bus Number of the downstream port DP_0 of the PCIe switch is N+1
  • the BDF is N+1:0.0
  • the Secondary Bus Number and Subordinate Bus Number are N+2.
  • the Primary Bus Number of the downstream port DP_1 of the PCIe switch is N+1, the BDF is N+1:1.0, and the Secondary Bus Number and Subordinate Bus Number are N+3.
  • the Primary Bus Number of the downstream port DP_m of the PCIe switch is N+1, the BDF is N+1:m.0, and the Secondary Bus Number and Subordinate Bus Number are N+2+m.
  • the BDF of EP_0 connected to downstream port DP_0 is N+2:0.0
  • the BDF of EP_1 connected to downstream port DP_1 is N+3:0.0
  • the BDF of EP_m connected to downstream port DP_m is N+2+m:0.0.
  • the aforementioned Primary Bus Number, Secondary Bus Number, and Subordinate Bus Number can follow the definition of the PCIe protocol, where the Primary Bus Number is used to represent the upstream bus number, the Secondary Bus Number is used to represent the downstream bus number, and the Secondary Bus Number is used to represent the The largest bus number in the PCIe subtree under the bus of the current level. Secondary Bus Number and Secondary Bus Number indicate the bus number range of the PCIe subtree under the bus of this level.
  • FIG. 1 and FIG. 2 only give an example of a possible bus assignment and management topology under a PCIe system, which are used to represent the situation of bus number assignment and management topology. It can be seen that, since PCIe is an end-to-end connection method, a PCIe endpoint device needs to occupy a bus number, so that the number of PCIe endpoint devices is limited by the number of bus numbers.
  • an embodiment of the present application provides a schematic diagram of bus number allocation in a PCIe system.
  • the PCIe system shown in FIG. 3 includes a PCIe switch (ie, a PCIe switching device).
  • a downstream port agent Downlink Port Agent, DPA
  • the DPA provides proxy services for its downstream ports, and each endpoint device connected to the downstream port of the DPA shares the same bus, thereby saving the number of buses , which provides the number of endpoint devices that the system can support.
  • the DPA is also called a port proxy device.
  • the DPA may be an ASIC chip or a software module. When the DPA is a software module, the processor loads instructions to implement the DPA function.
  • the PCIe switch may be a separate hardware device, such as a switch or a switch chip; it may also be a hardware function module integrated in other chips, where the hardware function module includes an upstream port, a port proxy device and multiple A downstream port, exemplarily, a PCIe switch can be integrated in the processor chip of the host.
  • the schematic diagram of bus number assignment in the PCIe system shown in FIG. 3 implements a bus number assignment logic different from that in the foregoing FIGS. 1 and 2 .
  • the PCIe system includes a processor CPU, a PCIe switch device (such as the PCIe switch shown in the figure), and a PCIe endpoint device. Subsequent embodiments of the present application take PCIe switch as an example to describe the solution.
  • RP0 can be connected to an endpoint device or a bridge device (not shown in the figure)
  • RP1 is connected to the upstream port UP of the PCIe switch
  • the PCIe switch includes m+1 downstream ports, marked as DP_0, DP_1, . . . , DP_m -1, DP_m
  • the endpoint devices shown in Figure 3 are marked as EP_0, EP_1, ..., EP_m-1, EP_m.
  • the difference from Figure 1 is that a DPA is added to the PCIe switch shown in Figure 3.
  • the upstream port UP is connected to the DPA, and the DPA is connected to some or all of the downstream ports of the PCIe switch.
  • the downstream port connected to the DPA can be called DPA. , a downstream port subordinate to the DPA, or a downstream port with a proxy function enabled. Exemplarily, in the figure, DP_0, DP_1, .
  • the root port RP1 connected to the upstream port UP occupies bus X
  • the connection between RP1 and UP occupies bus N
  • the connection between the upstream port UP and DPA inside the PCIe switch occupies bus N+1
  • the m-1 connections between the downstream ports DP_0, DP_1, ..., DP_m-1 and EP_0, EP_1, ..., EP_m-1 share the bus N+2
  • the connection from UP to DP_m occupies bus N+3.
  • X is an integer greater than or equal to 0
  • N is an integer greater than X. Examples of values for X and N are the same as in Figure 1.
  • the value of X depends on the bus number occupied by the PCIe structure under the left RC, for example , when the PCIe structure under the left RRC occupies 5 bus numbers (that is, bus 0-4 is occupied by the left RC and the PCIe structure below it), the value of X is 5, and the value of N is 6; when When the 2 RPs shown in the figure are located in the same RC and the RC that the RC bit processor enumerates first, the value of X is 0. If the PCIe structure under the left RC occupies 5 bus numbers (that is, bus 0-4 are occupied), the value of N is 5 at this time.
  • an embodiment of the present application further provides a schematic diagram of a PCIe device management topology corresponding to FIG. 3 .
  • the Primary Bus Number of RP1 is X
  • the BDF is X:0.0
  • the Secondary Bus Number is N
  • the Subordinate Bus Number is N+3.
  • the Primary Bus Number of the PCIe switch upstream port UP connected to the RP1 is N
  • the BDF is N: 0.0
  • the Secondary Bus Number is N+1
  • the Subordinate Bus Number is N+3.
  • the Primary Bus Number of the DPA in the PCIe switch is N+1
  • the BDF is N+1:0.0
  • the Secondary Bus Number is N+2
  • the Subordinate Bus Number is N+2.
  • the DPA provides the proxy for the m DPs. Therefore, the Primary Bus Number and Secondary Bus Number of the m DPs of DP_0, DP_1, ..., DP_m-1, And the Subordinate Bus Number is the same as DPA.
  • the BDF of EP_0 is N+2:0.0
  • the BDF of EP_1 is N+2:1.0
  • the BDF of EP_m-1 is N+2:m-1.0.
  • the proxy function is not enabled for DP_m
  • the BDF of DP_m is N+1:1.0
  • the Primary Bus Number of DP_m is N+1
  • the Secondary Bus Number is N+3
  • the Subordinate Bus Number is N+3.
  • FIG. 3 and FIG. 4 show the bus allocation and management topology after the DPA function is enabled in the PCIe switch provided in the embodiment of the present application.
  • the DPA can proxy the functions of multiple downstream ports of the PCIe switch, and present the multiple downstream ports of the proxy to the system as one downstream port.
  • one DPA can proxy the downstream ports of 32 PCIe switches.
  • DPA When DPA routes the transaction layer protocol (Transaction Layer Protocol, TLP) packet, it replaces the device ID (5bits device ID) in the BDF carried by the TLP packet with the downstream port number connected to the corresponding endpoint device, and then routes the TLP packet. to the processor.
  • TLP Transaction Layer Protocol
  • the port proxy device is specifically configured to replace the device identifier in the requester (Requester) BDF carried in the request TLP packet with the same as the device identifier in the requester (Requester) BDF.
  • the device identifier in is replaced with the serial number of the first downstream port connected to the PCIe endpoint device.
  • the DPA maps the PCIe endpoint devices connected to its downstream ports to the lower-level bus of the DPA.
  • multiple PCIe endpoint devices connected to each downstream port under DPA share the same bus number, which solves the problem in the prior art that an endpoint device connected to each downstream port of the PCIe switch must be assigned a lower-level bus.
  • the embodiment of the present application also avoids the waste problem caused in the prior art that the PCIe device connected to the downstream port of the PCIe switch does not use all the 256 PCIe IDs represented by device(5bits).function(3bits).
  • the PCIe switch needs to occupy m+2 bus numbers to connect m PCIe endpoint devices, and in the embodiment of the present application, when the m downstream ports of the PCIe switch are all enabled with the proxy function, the PCIe switch Connecting m PCIe endpoint devices only occupies 3 buses, thereby increasing the number of port devices supported by the PCIe system and improving bus utilization.
  • a computer system can support 7648 PCIe endpoint devices (such as solid-state drives) with only function 0 to expand. The way is as follows:
  • each root port is cascaded with a PCIe switch
  • each pcie switch has 32 DPAs
  • an exemplary embodiment of the present application provides a schematic diagram of a DPA configuration space in a PCIe switch, which is different from the configuration space of a DP defined by the PCIe protocol in that:
  • the configuration space of the configuration space registers of all DPs belonging to the DPA is mapped in the DPA configuration space register BAR1.
  • the DPA driver can indirectly access the configuration spaces of all DPs under the DPA through the BAR1 register, and handle PCIe switch DP links, errors, and hot-plugging.
  • a new DPA function structure is added to the DPA configuration space, and the DPA function structure includes an Enable register and a bitmap register.
  • bit0 of the DPA Enable register is used to enable/disable the DPA function.
  • bit0 of the DPA Enable register is used to enable/disable the DPA function.
  • the value is 1, it indicates that the DPA function is enabled.
  • the bitmap register DPA Port Bit Map LSB/MSB implements a 32-bit bitmap (it can continue to expand when the number of PCIe switch DPs exceeds 32), which is used to configure each DP that DPA needs to proxy. Whether the proxy function is enabled on the DP.
  • LSB Least Significant Bit, which refers to the least significant bit in binary numbers
  • MSB Most Significant Bit, which refers to the most significant bit in binary numbers.
  • the MSB is on the far left of the du of binary numbers and on the far right of the binary number.
  • downstream ports DP_0, DP_1, . . . , DP_m, DP_0, DP_1, . m+1 bit wherein the bit value used to represent DP_0, DP_1, .
  • the downstream port corresponding to the bit has been enabled with the proxy function, and a bit with a value of 0 indicates that the downstream port corresponding to the bit has not been enabled with the proxy function.
  • the PCIe switch including DPA may be applicable to various PCIe architectures.
  • multiple PCIe switches containing DPA can form a multi-layer architecture; PCIe switches containing DPA can also be mixed with traditional PCIe switches in a PCIe system; multiple DPAs can also be implemented in one PCIe switch; thus, under a single DPA
  • the number of downstream ports that can be mounted is limited, the number of endpoint devices that the PCIe system can support can be increased by using multiple DPAs in the PCIe switch.
  • FIG. 6 is another schematic structural diagram of a PCIe system provided by an embodiment of the present application
  • two PCIe switches including DPA are serially connected, and the UP of PCIe switch1 is connected to PCIe DP_m port of switch 0.
  • Figure 6 please refer to Figure 6 for the assignment of bus numbers.
  • FIG. 7 a schematic diagram of a PCIe device management topology corresponding to FIG. 6 is provided in an embodiment of the present application.
  • the Primary Bus Number of RP1 is X
  • the BDF is X:0.0
  • the Secondary Bus Number is N
  • the Subordinate Bus Number is N+6.
  • the Primary Bus Number of the upstream port UP of the PCIe switch 0 connected to the RP1 is N
  • the BDF is N: 0.0
  • the Secondary Bus Number is N+1
  • the Subordinate Bus Number is N+6.
  • the Primary Bus Number of the DPA in PCIe switch 0 is N+1
  • the BDF is N+1:0.0
  • the Secondary Bus Number is N+2
  • the Subordinate Bus Number is N+2.
  • the DPA provides the proxy for the m DPs. Therefore, the Primary Bus Number and Secondary Bus Number of the m DPs of DP_0, DP_1, ..., DP_m-1, And the Subordinate Bus Number is the same as DPA.
  • the BDFs of each endpoint device connected to DP_0, DP_1, ..., DP_m-1 of PCIe switch 0 are N+2:0.0, N+2:1.0, ..., N+2:m-1.0, respectively.
  • the proxy function is not enabled for DP_m of PCIe switch 0.
  • the BDF of DP_m is N+1:1.0, the Primary Bus Number of DP_m is N+1, the Secondary Bus Number is N+3, and the Subordinate Bus Number is N+6.
  • UP of PCIe switch 1 is connected to DP_m of PCIe switch 0.
  • the Primary Bus Number of the UP of PCIe switch 1 is N+3, the BDF is N+3:0.0, the Secondary Bus Number is N+4, and the Subordinate Bus Number is N+6.
  • the Primary Bus Number of the DPA in PCIe switch 1 is N+4, the BDF is N+4:0.0, the Secondary Bus Number is N+5, and the Subordinate Bus Number is N+5.
  • DPA provides the proxy for these m DPs. Therefore, DP_0, DP_1, ..., DP_m-1 in PCIe switch 1, this m
  • the Primary Bus Number, Secondary Bus Number, and Subordinate Bus Number of a DP are the same as the DPA of PCIe switch 1.
  • the BDFs of each endpoint device connected to DP_0, DP_1, ..., DP_m-1 of PCIe switch 1 are N+5:0.0, N+5:1.0, ..., N+5:m-1.0, respectively.
  • the proxy function is not enabled for DP_m of PCIe switch 1.
  • the BDF of DP_m is N+4:1.0, the Primary Bus Number of DP_m is N+4, the Secondary Bus Number is N+6, and the Subordinate Bus Number is N+6.
  • the BDF of the endpoint device connected to DP_m of PCIe switch 1 is N+6:0.0.
  • FIG. 8 is a schematic structural diagram of another PCIe system provided by an embodiment of the present application
  • the PCIe switch includes n parallel DPAs, and each DPA has m downstream ports.
  • the proxy function is enabled, but the proxy function is not enabled for DP_i of the PCIe switch.
  • the number of downstream ports under each DPA can be different, and whether the downstream port is enabled with the proxy function can be flexibly configured.
  • downstream port with the proxy function enabled may be referred to as the first downstream port, and the downstream port without the proxy function enabled may be referred to as the second downstream port.
  • PCIe switch provided by the embodiment of the present application can flexibly appear in various PCIe system structures, and the newly added DAP is used to provide the downstream port of the PCIe switch. proxy function.
  • the embodiments of the present application further provide a workflow of DPA in a PCIe system.
  • the workflow includes at least one of the following: PCIe switch power-on or reset initialization; PCIe device enumeration; PCIe memory address resource allocation; TLP packet ID routing; TLP packet address routing; TLP packet implicit routing; unplugging; error detection and handling, etc.
  • the PCIe switch When the PCIe switch is powered on/reset and initialized, the PCIe switch completes the DPA Enable register bit0 and DPA bitmap register in the DPA function structure through firmware FirmWare or Electrically Erasable Programmable read only memory (EEPROM).
  • Port Bit Map LSB/MSB settings Exemplarily, the bit0 of the DPA Enable register is set to 1, and the bitmap bit0 to bitm-1 (exemplarily, m is less than 31) corresponding to the two registers of the DPA Port Bit Map LSB/MSB corresponds to the position of the port that needs to enable the DPA agent Set to 1 to enable the DPA function and configure the downstream port DP that requires DPA proxy.
  • FIGS. 9A and 9B in conjunction with the PCIe system structure shown in FIG. 3 , a schematic flowchart of enumerating PCIe devices provided by an embodiment of the present application includes:
  • the processor enumerates the PCIe devices connected by bus 0, bus 1, ..., bus N-1 in order.
  • the embodiment of the present application assumes that all traditional PCIe devices are connected under bus 0, bus 1, .
  • the enumeration process may be performed by the processor running system enumeration software (BIOS and/or OS).
  • BIOS system enumeration software
  • the trigger of the enumeration is not limited in the embodiment of the present application, and in a possible implementation manner, the definition of the PCIe protocol may be followed.
  • the processor enumerates the devices connected on bus N, writes the Primary Bus Number register of the upstream port of the PCIe switch as N, the Secondary Bus Number is N+1, and the Subordinate Bus Number is N+1.
  • step 902 includes the following process:
  • the processor sends a configuration read request to the upstream port UP of the PCIe switch, and the configuration read request is used to enumerate the device whose bus is N, device is 0, and function is 0, and obtains the Device ID and Vendor ID of the device.
  • the Device ID and Vendor ID recorded in the PCIe device configuration space are used to record the device ID and manufacturer ID of the PCIe device.
  • the device ID in the configuration space is written by the manufacturer, which is different from the device ID in the BDF.
  • the upstream port UP of the PCIe switch detects that the Primary Bus Number of the port is 0, the Secondary Bus Number is 0, and the Subordinate Bus Number is 0, and returns the Device ID and Vendor ID register values recorded in the upstream port configuration space to the processor. It should be noted that, before the enumeration operation, the initial values of the Primary Bus Number, Secondary Bus Number, and Subordinate Bus Number of each port are all 0.
  • the processor determines that the values of the Device ID and Vendor ID returned by the upstream port of the PCIe switch are not 0xFFFF, it is determined that there is a PCIe device on the bus N.
  • the processor sends a read request to the PCI Header Type register configuration of the device whose bus is N, device is 0, and function is 0. It should be noted that the values of Device ID and Vendor ID refer to the combined value of Device ID and Vendor ID.
  • the uplink port UP of the PCIe switch receives the configuration read request sent by the processor to the PCI Header Type register, detects that the Primary Bus Number of the port is 0, the Secondary Bus Number is 0, and the Subordinate Bus Number is 0, and returns the PCIe switch to the processor.
  • the value of the upstream port configuration space PCI Header Type register is the value of the upstream port configuration space PCI Header Type register.
  • the PCI Header Type register value of the PCI-PCI bridge is 10000001b.
  • the processor recognizes that the bus is N, the device is 0, and the device whose function is 0 is a PCI-PCI bridge.
  • the processor sends a configuration read request to the PCI Express Capability register of the device configuration space where bus is N, device is 0, and function is 0.
  • the upstream port of the PCIe switch receives the configuration read request to the PCI Express Capability register, detects that the Primary Bus Number of the port is 0, the Secondary Bus Number is 0, and the Subordinate Bus Number is 0, and returns the upstream port configuration space PCI of the PCIe switch to the processor The value of the Express Capability register.
  • the processor When the processor reads that the PCI Express Capability register value returned by the upstream port of the PCIe switch is not 0xFFFF, and the Device/Port Type field is 0101b, the processor recognizes the device with bus N, device 0, and function 0 according to the PCIe protocol. It is the upstream port of the PCIe switch.
  • the processor sends a configuration write request to the device whose bus is N, device is 0, and function is 0. It is used to write the Primary Bus Number register of the upstream port of the PCIe switch as N, the Secondary Bus Number register as N+1, and the Subordinate Bus Number register as N+1. The register is written as N+1.
  • the processor After completing the enumeration of the devices connected to the bus N, the processor continues to enumerate the devices connected to the bus N+1. In this embodiment of the present application, that is, the processor enumerates the DPA in the PCIe switch.
  • Step 903 The processor sends a configuration read request to the uplink port UP of the PCIe switch, and the configuration read request is used to enumerate a device whose bus is N+1, device is 0, and function is 0, and is used to obtain the Device ID of the device and Vendor ID.
  • Step 904 Since the bus carried in the configuration read request in step 903 is N+1, the upstream port of the PCIe switch transfers the configuration read request after determining that the Secondary Bus Number of the port is N+1 and the Subordinate Bus Number is N+1. Enter the internal route and send it to the DPA.
  • Step 905 DPA detects that the Primary Bus Number of the port is 0, the Secondary Bus Number is 0, and the Subordinate Bus Number is 0. The DPA further determines that the PCIe switch DPA has been enabled according to the bit0 of the enable register of the configuration space, and determines that it exists according to the port enable record. The downstream port for which the proxy function has been enabled. DPA returns the Device ID and Vendor ID register values recorded in the configuration space of this port to the processor.
  • the vendor ID recorded in the PCIe device configuration space is the ID of the manufacturer, the ID of the PCIe device provided by the deviceID manufacturer, and the Device ID and Vendor ID are used to indicate the validity of the PCIe device.
  • the difference is that the Device ID in the BDF is allocated when the processor enumerates PCIe devices and is used to manage PCIe devices, so the device ID in the BDF is different from the device ID recorded in the PCIe device configuration space.
  • the port enable record may be recorded in a bitmap register, and the bitmap value recorded in the bitmap register corresponding to each downstream port of the PCIe switch device indicates whether the endpoint device connected to each downstream port has enabled the agent Features.
  • the port enable record may also be a mapping table.
  • the bitmap register may be a DPA Port Bit Map LSB/MSB register.
  • Step 906 The processor receives the Device ID and Vendor ID returned by the DPA, and when it is determined that the values of the Device ID and Vendor ID are not 0xFFFF, it is determined that there is a PCIe device on the bus N+1.
  • the processor sends a read request to the PCI Header Type register configuration of the device whose bus is N+1, device is 0, and function is 0.
  • Step 907 DPA receives the configuration read request for the PCI Header Type register sent by the processor, detects that the Primary Bus Number of the port is 0, the Secondary Bus Number is 0, and the Subordinate Bus Number is 0, and returns the DPA port configuration space to the processor The value of the PCI Header Type register.
  • the PCI Header Type register value of the PCI-PCI bridge is 10000001b.
  • the processor recognizes that the bus is N+1, the device is 0, and the device whose function is 0 is a PCI-PCI bridge.
  • Step 908 The processor sends a configuration read request to the PCI Express Capability register of the device configuration space where the bus is N+1, the device is 0, and the function is 0.
  • Step 909 DPA receives the configuration read request for the PCI Express Capability register, detects that the Primary Bus Number of the port is 0, the Secondary Bus Number is 0, and the Subordinate Bus Number is 0, and returns the DPA port configuration space to the processor PCI Express Capability register value of .
  • the processor When the processor reads that the value of the PCI Express Capability register returned by DPA is not 0xFFFF, and the Device/Port Type field is 0110b, the processor recognizes that the bus is N+1, the device is 0, and the function is 0 according to the PCIe protocol.
  • PCIe switch downstream port that is, the processor identifies the DPA as a PCIe switch downstream port.
  • Step 910 The processor sends a configuration write request to the device whose bus is N, device is 0, and function is 0, which is used to write the Primary Bus Number of the PCIe upstream port as N, the Secondary Bus Number as N+1, and the Subordinate Bus Number as N+2.
  • Step 911 The processor sends a configuration write request to the device whose bus is N+1, device is 0, and function is 0, for writing the Primary Bus Number register of DPA as N+1, Secondary Bus Number as N+2, Subordinate Bus Number is N+2.
  • Step 912 The DPA receiving processor sends a configuration write request to the device whose bus is N+1, device is 0, and function is 0, and writes the Primary Bus Number as N+1, the Secondary Bus Number as N+2, and the Subordinate Bus Number as N+2. Further, according to the value of the bitmap register, DPA writes the Primary Bus Number of the downstream port with the DPA proxy function enabled as N+1, the Secondary Bus Number as N+2, and the Subordinate Bus Number as N+2. Exemplarily, corresponding to FIG. 3, the Primary Bus Number of the downstream ports DP_0, DP_1, ..., DP_m-1 is written as N+1, the Secondary Bus Number is N+2, and the Subordinate Bus Number is N+2.
  • the processor continues to enumerate the devices connected to bus N+2, that is, the processor enumerates the endpoint devices connected to the downstream ports 0, 1, ..., m-1 under the DPA.
  • Step 913 The processor sends a configuration read request, and the configuration read request is used to enumerate a device whose bus is N+2, device is 0, and function is 0, and is used to obtain the Device ID and Vendor ID of the device.
  • Step 914 Since the bus carried in the configuration read request in step 913 is N+1, the upstream port of the PCIe switch transfers the configuration read request after determining that the Secondary Bus Number of the port is N+1 and the Subordinate Bus Number is N+2. Enter the internal route and send it to the DPA.
  • Step 915 The DPA detects that the Primary Bus Number of the port is N+1, the Secondary Bus Number is N+2, and the Subordinate Bus Number is N+2.
  • the DPA further determines that the PCIe switch DPA has been enabled according to the configuration space enable register bit0.
  • Bitmap register DPA Port Bit Map LSB/MSB The value of the two registers determines that there is a downstream port that has the proxy function enabled. DPA routes configuration read requests to downstream port DP_0.
  • the downstream port DP_0 detects that the Primary Bus Number of this port is N+1, the Secondary Bus Number is N+2, and the Subordinate Bus Number is N+2, and routes the configuration read request to the endpoint device EP_0 connected to it.
  • Step 916 The endpoint device EP_0 returns a completion TLP (Completion TLP) packet to the downstream port DP_0, and the completion TLP packet carries the Device ID and Vendor ID register values of the endpoint device EP_0. Downstream port DP_0 routes the completion TLP packet to the DPA.
  • completion TLP Completion TLP
  • Step 917 DPA receives the completion TLP packet, DPA determines that the DPA function has been enabled according to the enable register bit0, determines that the downstream port DP_0 has enabled the proxy function according to the bitmap register, and converts the device identifier in the completion TLP packet to 0,
  • the completion TLP packet is then routed to the processor through the PCIe switch upstream port.
  • the device identifier of the completion TLP message is carried in the completer BDF, and the completer BDF is also called the completer ID.
  • Step 918 When the processor receives the completion TLP packet and determines that the values of Device ID and Vendor ID in the completion TLP packet are non-0xFFFF, it is determined that there is a PCIe device on bus N+2.
  • the processor sends a read request to the PCI Header Type register configuration of the device whose bus is N+2, device is 0, and function is 0.
  • Step 919 Similar to the previous solution, the PCIe endpoint device EP_0 returns the PCI Header Type register value of the device.
  • the processor reads the PCI Header Type register value returned by the PCIe endpoint device EP_0 to 00000000b.
  • the system enumeration software recognizes that the bus is N+2, the device is 0, and the device whose function is 0 is an endpoint device according to the PCIe protocol, and only contains one function.
  • Step 920 Similarly, the processor continues to read the device configuration space PCI Express Capability register where the bus is N+2, the device is 0, and the function is 0.
  • Step 921 The PCIe endpoint device EP_0 returns the value of the PCI Express Capability register.
  • the system enumeration software reads that the PCI Express Capability register value returned by the PCIe endpoint device EP_0 is not 0xFFFF, and the Device/Port Type field is 0000b.
  • the system enumeration software recognizes that the bus is N+2, the device is 0, and the device whose function is 0 is the endpoint device according to the PCIe protocol.
  • the system enumeration software reads the PCI Header Type register value returned by the PCIe endpoint device EP_0 to 10000000b. According to the PCIe protocol, the system enumeration software recognizes that the bus is N+2, the device is 0, and the device whose function is 0 is the endpoint device, and only contains multiple functions. Then the system enumeration software repeats the above steps to enumerate the endpoint devices whose bus is N+2, device is 0, and function is 1-7 into the system.
  • a boot parameter for mandatory scanning is added to the kernel of the computer operating system.
  • the processor When the processor receives the PCI Header Type register value returned by EP_0 as 00000000b, it prevents the processor from mistakenly thinking that the bus scan of bus N+2 has been completed because the processor finds that EP_0 is a device with only one function, thus omitting the response to bus N +2 Scan for other device functions on the bus.
  • the processor may also participate in other parameter settings so that the processor can perform a complete scan of the devices connected to the lower-level bus of the DPA, which will not be repeated in this embodiment of the present application.
  • step 922 the processor continues to enumerate bus as N+2, device as 1 to m-1, and the function is The device functions from 1 to 7 are enumerated, and m is an integer greater than or equal to 1.
  • the specific enumeration process is the same as the enumeration process of the processor for the device whose bus is N+2, device is 0, and function is 0. This embodiment of the present application will not be repeated here.
  • the processor skips the port during the enumeration process, skips the device ID corresponding to the port, and continues to enumerate other downstream ports .
  • Step 923 The processor sends a configuration read request for a device whose bus is N+1, device is m, and function is 0, and the configuration read request is used to obtain the Device ID and Vendor ID of the device.
  • the processor after completing the enumeration of the endpoint devices connected to the downstream ports DP_0 to DP_m-1 under the DPA, the processor continues to enumerate the devices connected to the downstream port DP_m.
  • the device ID carried in the configuration read request is 1 (that is, when applying the architecture of Figure 3, in the enumeration process of DP_m, the value of m in the following steps is 1). It should be noted that when there are multiple downstream ports without the proxy function enabled, the device ID of the configuration read request here is incremented by 1.
  • Step 924 Since the bus carried in the configuration read request in step 923 is N+1, the upstream port of the PCIe switch transfers the configuration read request after determining that the Secondary Bus Number of the port is N+1 and the Subordinate Bus Number is N+2. Enter the internal route and send it to DP_m.
  • Step 925 The PCIe switch downstream port DP_m detects that the Primary Bus Number of the port is 0, the Secondary Bus Number is 0, and the Subordinate Bus Number is 0, and returns the Device ID and Vendor ID register values of the downstream port DP_m to the processor.
  • the processor determines that the values of the Device ID and Vendor ID returned by the downstream port DP_m of the PCIe switch are not 0xFFFF, and determines that the bus is N+1, the device is m, and the device whose function is 0 exists.
  • Step 926 Similarly, the processor continues to read the PCI Header Type register of the device configuration space where the bus is N+1, the device is m, and the function is 0. The processor obtains the PCI Header Type register value returned by the downstream port DP_m as 10000001b. The system enumeration software recognizes that the bus is N+1, the device is m, and the device whose function is 0 is a PCI-PCI bridge according to the PCIe protocol.
  • Step 927 Similarly, the processor continues to read the device configuration space PCI Express Capability register whose bus is N+1, device is m, and function is 0. The processor reads that the PCI Express Capability register value returned by the downstream port DP_m of the PCIe switch is not 0xFFFF, and the Device/Port Type field is 0110b. According to the PCIe protocol, the system enumeration software recognizes that the bus is N+1, the device is m, and the device whose function is 0 is the downstream port of the PCIe switch.
  • Step 928 The processor writes the Primary Bus Number register as N, the Secondary Bus Number as N+1, and the Subordinate Bus Number as N+3 to the device configuration of which bus is N, device is 0, and function is 0. That is, the Primary Bus Number register of the upstream port of the PCIe switch is N, the Secondary Bus Number is N+1, and the Subordinate Bus Number is N+3.
  • Step 929 The processor configures and writes the Primary Bus Number register as N+1, the Secondary Bus Number as N+3, and the Subordinate Bus Number as N+3 to the device whose bus is N+1, device is m, and function is 0. That is, the Primary Bus Number register of the downstream port DP_m of the PCIe switch is N+1, the Secondary Bus Number is N+3, and the Subordinate Bus Number is N+3.
  • the processor completes the enumeration of DP_m, and further, the processor continues to enumerate bus N+3, and performs the enumeration of EP_m.
  • the processor When there are other downstream ports without the proxy function enabled, the processor continues to enumerate the other downstream ports and the PCIe devices connected under the other downstream ports in the same manner.
  • the processor completes the enumeration of the PCIe system shown in FIG. 3 .
  • the foregoing steps can be repeated to complete device enumeration in various PCIe systems.
  • the processor (running the system enumeration software) allocates memory addresses to each PCIe device in the PCIe system.
  • the processor may perform memory address allocation according to a traditional PCIe device memory address resource allocation algorithm.
  • the processor first traverses the memory address resources declared by each PCIe endpoint device, then applies for the corresponding address space from the system memory address resources, then writes the starting address to the PCIe endpoint device BAR register, and adjusts the endpoint device to the RP path address window of each bridge device.
  • the 32-bit address window represented by the Memory Limit and Memory Base registers of the PCI-PCI bridge
  • the 64-bit addresses represented by the Prefetchable Memory Limit, Prefetchable Memory Base, Prefetchable Memory Upper Base Address, and Prefetchable Memory Upper Limit Address registers window.
  • the PCIe switch DPA records when the system enumeration software writes a TLP packet to the downstream port of the PCIe switch for the configuration of the Memory Base register of the PCIe endpoint device.
  • the PCIe switch can sense the 32-bit/64-bit start addresses of all endpoint devices connected to the downstream port.
  • the address window of the bridge device (based on the 32-bit and 64-bit address windows of the PCI-PCI bridge) must include the address range used by all downstream PCIe devices, then the PCIe switch firmware FW can determine the starting address of the PCIe endpoint device connected to each downstream port according to the Calculate the end address and set it to the downstream port Memory Limit register and Memory Base register Prefetchable Memory Limit register, Prefetchable Memory Base register, Prefetchable Memory Upper Base Address register and Prefetchable Memory Upper Limit Address register.
  • FIG. 10 a memory address resource allocation diagram is provided in this embodiment of the present application.
  • the address window of the upstream device includes the address window of the downstream device of each layer. In the example of FIG.
  • each address window is continuous.
  • the address window of the upstream port of the PCIe switch is [A, Z]
  • the address window of the DPA in the PCIe switch is [A, Y]
  • the downstream ports DP_0, DP_1, ..., DP_m-1 in the PCIe switch are enabled Proxy function
  • the address window of DP_0 of PCIe switch DPA agent is [A, B]
  • the address window of DP_1 is [B, C]
  • ..., the address window of DP_m-1 is [X, Y]
  • the address window of DP_m capable of proxy function is [Y, Z].
  • the address windows of each device may also use discontinuous address spaces.
  • the PCIe protocol defines workflows such as ID Based Routing, Address Based Routing, and Implicit Routing. The workflow also needs to be adjusted accordingly.
  • ID routing is generally used in the transmission process of configuration requests (configuration read requests/write requests). The method includes:
  • Step 1101 The PCIe switch upstream port receives the configuration request from the processor, and determines whether the bus number in the BDF carried by the configuration request falls within the scope of the lower-level bus and the subordinate bus of the upstream port, and if so, transfers the configuration request to The PCIe switch is internally routed and sent to the DPA.
  • the configuration request may be a configuration read/write request TLP packet sent by the processor. It should be noted that the embodiment of this application is described by taking the configuration request sent to the DPA as an example.
  • the upstream port needs to route the configuration request to the downstream port without the proxy function according to the BDF carried in the configuration request, it can follow the PCIe protocol. The prescribed process is carried out.
  • Step 1102 DPA receives the configuration request, and determines whether the bus number in the target BDF carried by the configuration request falls within the scope of the subordinate bus and the subordinate bus of the DPA. If so, the DPA is in the target BDF carried by the configuration request according to the The D (that is, the device ID, which can be represented by device (5bits)) routes the configuration request to the corresponding downstream port, where the configuration request carries the value of the device ID as the number of the downstream port, and DPA will configure the request carried in the target BDF in the BDF The device ID is changed to 0.
  • the D that is, the device ID, which can be represented by device (5bits)
  • Step 1103 The downstream port routes the configuration request to the endpoint device connected to the downstream port.
  • Step 1104 If the bus number in the BDF carried in the DPA configuration request does not belong to the range defined by the subordinate bus and the subordinate bus of the DPA, it means that a routing error has occurred.
  • the DPA may refer to the process of error handling by the bridge device defined in the protocol to operate, which is not limited in this embodiment of the present application.
  • Step 1105 The PCIe switch upstream port receives the completion TLP packet sent by the processor, and determines whether the completed TLP packet carries the bus number in the requester BDF and falls within the scope of the lower-level bus and the subordinate bus of the upstream port, and if so, then The completed TLP packet is transferred to the internal routing of the PCIe switch and sent to the DPA.
  • Step 1106 DPA receives the completion TLP packet, and determines whether the bus number in the requester BDF carried by the completion TLP packet falls within the scope of the lower-level bus and the subordinate bus of the DPA.
  • the D in the BDF of the requester carried by the package is changed to 0 (that is, the device identifier, which can be represented by device (5bits)), and then the completion TLP packet is routed to the corresponding downstream port, wherein the requester carried by the completion TLP packet
  • the value of the raw device identification in the BDF serves as the number of the downstream port.
  • Step 1107 The downstream port routes the completion TLP packet to the endpoint device connected to the downstream port.
  • Step 1108 When the DPA determines that the bus number in the requester BDF carried in the completion TLP packet does not fall within the range of the subordinate bus and the subordinate bus of the DPA, it indicates that a routing error has occurred, and an error handling operation is performed.
  • Step 1109 The downstream port of the PCIe switch receives the completion TLP packet from the endpoint device, and sends the completion TLP packet to the DPA.
  • Step 1110 DPA receives the completion TLP packet, determines the number of the downstream port that sends the completion TLP packet, determines that the DPA function has been enabled according to the enable register bit0, and determines the number of the downstream port that sends the completion TLP packet according to the value of the bitmap register. After the downstream port has been enabled with the proxy function, the DPA changes the device identifier in the completer BDF carried in the completion TLP packet to the number of the downstream port, and the DPA routes the completion TLP packet to the processor through the upstream port.
  • Step 1111 After receiving the completion TLP packet, if the DPA determines that the DPA function is not enabled according to the enable register bit0, or, according to the value of the bitmap register, it is determined that the downstream port that sends the completion TLP packet is not enabled with the proxy function, Then the DPA does not modify the device identification of the completion TLP packet, but directly sends the completion TLP packet to the processor through the upstream port.
  • the method logic of the PCIe switch including DPA to perform address routing includes:
  • the upstream port of the PCIe switch When the upstream port of the PCIe switch receives a memory operation request (for example, a memory read/write request TLP packet) from the processor, it determines whether the target address falls within the address window of the upstream port of the PCIe switch. If the target address falls within the address window of the upstream port of the PCIe switch, it enters the internal routing of the PCIe switch. The PCIe switch internal routing determines whether the target address falls within the address window of the PCIe switch DPA. If the target address falls within the address window range of the PCIe switch DPA, the PCIe switch upstream port routes the received memory operation request to the PCIe switch DPA. DPA determines whether the target address falls into the downstream port address window of the DPA agent.
  • a memory operation request for example, a memory read/write request TLP packet
  • the memory operation request is routed to the downstream port of the PCIe switch DPA agent if the target address falls within the address window range of the downstream port of the PCIe switch DPA agent.
  • a PCIe switch downstream port routes memory operation requests to its connected PCIe endpoint devices. If DPA determines that the destination address falls within the address window of the downstream port of the PCIe switch that is not proxied by DPA, then a routing error has occurred.
  • a PCIe downstream port When a PCIe downstream port receives a memory operation request (for example, a memory read/write request TLP packet) from an endpoint device, it sends the memory operation request to the DPA.
  • DPA detects whether the proxy function is enabled on the downstream port that sends the memory operation request. If it is not enabled, DPA routes the memory operation request to the processor through the upstream port. If it is enabled, DPA sends the memory operation request to the device The identification is changed to the number of the downstream port that sends the memory operation request, and then the memory operation request is routed to the processor through the upstream port. Specifically, the device identifier of the memory operation request is carried in the BDF in the request.
  • the method logic of PCIe switc including DPA to perform implicit routing includes:
  • the upstream port of the PCIe switch When the upstream port of the PCIe switch receives an implicit TLP packet (that is, a message packet) from the processor, it routes according to the message routing subfield in the implicit TLP packet.
  • an implicit TLP packet that is, a message packet
  • the DPA When the DPA receives the implicit TLP packet sent by the upstream port, it determines whether the proxy function is enabled on the downstream port that sent the implicit TLP packet, and if so, changes the device identifier in the requester BDF in the implicit TLP packet is 0, then, if not, route directly according to the message routing subfield in the TLP packet.
  • the DPA needs to change the requester ID/BDF in the TLP packet to the BDF of the PCIe switch DPA.
  • Implicit TLP packets that are not actively sent by the downstream port of the PCIe switch DPA agent are routed directly according to the message routing subfield in the TLP packet.
  • the DPA When the endpoint device is hot-plugged on the downstream port of the DPA agent, when the interrupt message reported by the downstream port passes through the DPA, the DPA changes the BDF according to the aforementioned implicit routing rules, reports the interrupt to the processor, and triggers the DPA hot-plug. interrupt.
  • the DPA performs port scanning, which is similar to the mandatory device function scan described in the foregoing embodiment. At this time, the DPA needs to forcibly scan 256 device functions represented by device(5bits).function(3bits).
  • the DPA changes the BDF of the error message to the BDF of the DPA, and records the port status information in the configuration space registers of the DPA and the downstream port.
  • the configuration space registers of each downstream port under the DPA are traversed, and the number of the downstream port where the error is detected is determined according to the state of the port recorded above, so as to perform error processing for the downstream port.
  • the specific configuration space register that records the state of the port may be an AER (advanced error reporting) function register.
  • one or more layers of bridge devices may exist between the processor and the PCIe switch, and the processor communicates with the bridge device between the PCIe switch.
  • the foregoing embodiments of the present application are described by taking a PCIe switch directly connected to the processor RP as an example. It can be understood that when there is a traditional PCIe bridge device between the processor and the PCIe switch, the intermediate PCIe bridge device can process the communication between the processor and the PCIe switch provided by the embodiment of the present application according to the definition of the PCIe protocol.
  • the embodiment of this application only allows the DPA to map the PCIe endpoint device connected to the downstream port of its proxy to the device identifier in the BDF, and the device identifier is 5 bits. Therefore, a DPA can only Map PCIe endpoint devices connected by 32 downstream ports. If the Fuction of PCIe endpoint devices is less than 8, there is still the problem of wasting PCIe IDs.
  • the device ID of the PCIe endpoint device connected to the downstream port of the DPA mapping can be set to expand from 5bits to 6bits or 7bits, and correspondingly, the bits occupied by the function are reduced from 3bits to 2bits or 1bit. At this point, the number of PCIe endpoint devices connected by DPA mapping downstream ports and the number of PCIe endpoint device functions are shown in the following table:
  • the embodiment of the present application provides a PCIe switch including DPA, which provides an agent for multiple downstream ports through DPA, and connects multiple downstream ports to the upper-layer system without changing the existing BIOS, OS PCIe subsystem and PCIe device driver. Presented as a port, thereby reducing the occupancy of the bus and increasing the number of endpoint devices supported by the PCIe system.
  • a DPA driver can be added without changing the software of other parts of the PCIe system, and the impact on the existing system is small.
  • the embodiment of the present application does not need to add circuit units such as NTB to isolate two PCIe domains, so the cost is low.
  • the DPA is implemented with an ASIC chip, the performance of performing communication forwarding between the upstream port and the downstream port is high.
  • the PCIe switch device 1200 includes a processor 1201 , an internal connection 1202 , an upstream port 1204 , a plurality of downstream ports 1205 , and a memory 1203 .
  • the processor 1201 may be multiple processor cores, and the processor may include multiple registers for storing configuration information of the PCIe switching device.
  • the endpoint proxy apparatus (DPA in the foregoing embodiment) may use a software module, that is, the processor 1201 executes the instructions in the memory 1203 to implement the foregoing DPA function.
  • the upstream port 1202 may be one or more.
  • processor 1201 may be a general-purpose central processing unit (central processing unit, CPU), a network processor (network processor, NP), a microprocessor, or one or more for controlling the execution of the program of the present application. of integrated circuits.
  • CPU central processing unit
  • NP network processor
  • microprocessor or one or more for controlling the execution of the program of the present application. of integrated circuits.
  • the internal connections 1202 described above may include a path to transfer information between the aforementioned components.
  • the internal connection 1202 is a bus.
  • the above-mentioned memory 1203 can be a read-only memory (read-only memory, ROM) or other types of static storage devices that can store static information and instructions, a random access memory (random access memory, RAM) or other types of storage devices that can store information and instructions.
  • ROM read-only memory
  • RAM random access memory
  • Types of dynamic storage devices which can also be electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM), or other optical storage, CD-ROM storage (including compact discs, laser discs, compact discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by Any other medium accessed by the computer, but not limited to this.
  • the memory can exist independently and be connected to the processor through a bus.
  • the memory can also be integrated with the processor.
  • the memory 1203 is used for storing the application program code for executing the solution of the present application, and the execution is controlled by the processor 1201 .
  • the processor 1201 is configured to execute the application program code stored in the memory 1203, and cooperate with the upstream port 1204 and the downstream port 1205, so that the apparatus 1200 can realize the functions described in the foregoing embodiments of the present application.
  • the processor 1201 may include one or more CPUs, such as CPU0 and CPU1 in FIG. 12 .
  • the PCIe switching device 1200 may include multiple processors. Each of these processors can be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor.
  • a processor herein may refer to one or more devices, circuits, and/or processing cores for processing data (eg, computer program instructions).

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Computer Hardware Design (AREA)
  • Bus Control (AREA)
  • Information Transfer Systems (AREA)

Abstract

本申请公开了一种扩展PCIe系统的方法、PCIe交换设备及PCIe系统，其中，所述PCIe交换设备包括上游端口、端口代理装置和多个第一下游端口，其中，所述上游端口与所述端口代理装置相连，所述端口代理装置与所述多个第一下游端口相连，所述第一下游端口用于连接PCIe端点设备；所述多个第一下游端口与其下的PCIe端点设备的连接共享相同的总线号。通过将多个第一下游端口与其下的PCIe端点设备的连接共享相同的总线号，减少了PCIe端点设备占用总线号的数量，提高了PCIe系统的容量。

Description

扩展PCIe系统的方法、PCIe交换设备及PCIe系统
本申请要求于2020年7月13日提交的申请号为202010677632.7、发明名称为“一种提高系统PCIe设备扩展能力方法”的中国专利申请的优先权，以及于2020年12月30日提交中国专利局、申请号为202011620807.7、申请名称为“扩展PCIe系统的方法、PCIe交换设备及PCIe系统”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及计算机技术领域，特别涉及一种扩展PCIe系统的方法、PCIe交换设备及PCIe系统。
背景技术
外围器件互联总线(peripheral component interconnect express，PCIe)是一种高速串行计算机扩展总线标准，用于实现处理器和外部设备的连接,这些外部设备被称为PCIe设备。
PCIe协议使用32位PCIe ID来标识系统中的PCIe设备，32位包括：domain(16bits):bus(8bits):device(5bits).function(3bits))。8位总线标识bus ID、5位设备标识device ID和3位功能标识function ID，简称为BDF。Bus ID为每个PCIe域提供了256个总线号。通常情况下，计算机系统只支持一个PCIe域，即整个计算机系统只提供256个总线号。由于PCIe协议为点对点连接，因此，一个PCIe域可以支持的PCIe设备的规模受到256个总线号数量的制约，无法在一个PCIe域支持更多的PCIe设备。
发明内容
本申请提供了一种扩展PCIe系统的方法、PCIe交换设备及PCIe系统，以解决现有技术中PCIe设备的规模受到256个总线号数量的制约的问题。
第一方面，本申请提供了一种PCIe交换设备，所述PCIe交换设备包括上游端口、端口代理装置和多个第一下游端口，其中，所述上游端口与所述端口代理装置相连，所述端口代理装置与所述多个第一下游端口相连，所述第一下游端口用于连接PCIe端点设备；所述多个第一下游端口与其下的PCIe端点设备的连接共享相同的总线号。
第一方面的交换设备中包含端口代理装置，且与多个第一下游端口相连，为所述多个第一下游端口提供代理，通过将多个第一下游端口与其下的PCIe端点设备的连接共享相同的总线号，减少了PCIe端点设备占用总线号的数量，提高了PCIe系统的容量。
在一种可能的实施方式中，所述端口代理装置的下游总线号等于归属总线号，其中，所述归属总线号用于表示所述端口代理装置下的PCIe子树中最大的总线号。
在另一种可能的实施方式中，PCIe交换设备还包括一个或多个第二下游端口，其中，所述一个或多个第二下游端口与所述上游端口相连。第二下游端口连接到上游端口，不需要由端口代理装置提供代理服务。
枚举过程中，系统枚举软件(例如，基本输入输出系统/操作系统，Basic Input Output System/Operating System，BIOS/OS)为PCIe系统的PCIe设备(包括桥设备和端点设备) 分配BDF。桥设备的管理拓扑的总线号包括上游总线号(Primary Bus Number)、下游总线(Secondary Bus Number)和从属总线号(Subordinate Bus Number)。枚举完成后，所述端口代理装置的下游总线号等于归属总线号。
在另一种可能的实施方式中，所述端口代理装置的下游总线号可以小于归属总线号，同时，归属总线号减去下游总线号的值小于前述第一下游端口的数量。此时，多个第一下游端口中的部分端口共享相同的下游总线号。
在另一种可能的实施方式中，所述端口代理装置与所述多个第一下游端口具备相同的归属总线号。具体的，所述端口代理装置将其下的所述多个第一下游端口的管理拓扑总线号设置为与自身管理拓扑总线号相同。
在另一种可能的实施方式中，所述第一下游端口的编号用于表示与所述第一下游端口连接的PCIe端点设备的设备标识。处理器可以使用PCIe端点设备连接的第一下游端口的标识来区分所述端口代理装置下的各个端点设备。处理器基于所述第一下游端口的编号、结合总线号D和功能号F识别PCIe端点设备。
在另一种可能的实施方式中，所述端口代理装置用于从所述第一下游端口接收PCIe端点设备发送的数据包，将所述数据包中的设备标识替换为所述第一下游端口的编号。
当所述PCIe端点设备发送的数据包为请求TLP包时，所述端口代理装置具体用于将所述请求TLP包携带的请求者BDF中的设备标识替换为与所述PCIe端点设备相连的第一下游端口的编号，或者，
当PCIe端点设备发送的数据包为完成TLP包时，所述端口代理装置具体用于将所述完成TLP包携带的完成者BDF中的设备标识替换为与所述PCIe端点设备相连的第一下游端口的编号。
在另一种可能的实施方式中，所述端口代理装置用于接收所述处理器发送的数据包，将所述数据包中设备标识的值作为接收所述数据包的目的端口的编号，并将所述数据包中的设备标识更改为0，将所述数据包发送到所述目的端口。
当所述处理器发送的数据包为配置请求TLP包时，将所述配置请求TLP包携带的目的BDF中的设备标识替换为0，或者，
当所述处理器发送的数据包为完成TLP包时，将所述完成TLP包携带的请求者BDF中的设备标识替换为0。
所述配置请求TLP包包括配置写请求TLP包和配置读请求TLP包。
在本实施方式中，处理器在枚举过程中，所述第一下游端口下连接的PCIe端点设备的设备标识被替换为所述第一下游端口的编号。现有技术中，PCIe系统为PCIe端点设备分配的设备标识一般为0。本申请实施例为了避免对PCIe端点设备造成影响，因此由端口代理装置将发给PCIe端点设备的数据包中的设备标识改为0，从而使得PCIe端点设备不需要为本申请的方案更改软件配置。
在另一种可能的实施方式中，所述端口代理装置的配置空间寄存器中映射有归属于所述端口代理装置的所有第一下游端口的配置空间寄存器。通过上述配置空间寄存器在第一下游端口到端口代理装置的映射，实现了端口代理装置对第一下游端口的配置空间寄存器的访问。
在另一种可能的实施方式中，所述端口代理装置的使能寄存器的值用于表示所述端口代 理装置的代理功能是否使能。通过使能寄存器，使得PCIe交换设备可以开关代理功能，在代理功能开启时实施本申请方案，在代理功能关闭时，作为传统的PCIe交换设备采用现有技术的方式提供服务。
具体的，所述PCIe交换设备在上电或者复位初始化时，通过固件或者读取非易失性存储器(例如，带电可擦可编程只读存储器EEPROM)的方式，设置所述端口代理装置的寄存器。
在另一种可能的实施方式中，所述端口代理装置，用于接收所述上游端口转发的第一配置请求，所述第一配置请求携带所述端口代理装置的管理拓扑对应的各总线号；
所述端口代理装置，还用于根据所述第一配置请求，设置自身的管理拓扑对应的各总线号，并将其下连接的所有第一下游端口的管理拓扑设置为相同的总线号。
通过将端口代理装置下的所有第一下游端口设置为与端口代理装置具备相同的管理拓扑总线号，使得端口代理装置代理其下的所有第一下游端口，并对上层系统将其下的所有第一下游端口模拟为一个代理端口。
在另一种可能的实施方式中，所述端口代理装置，用于接收处理器发送的第二配置请求，所述第二配置请求携带的总线号为所述端口代理装置的下级总线号、携带的设备标识为m，m为大于等于0的整数；所述端口代理装置，还用于根据所述第二配置请求携带的设备标识的值m，将所述第二配置请求路由到对应的目的端口，所述目的端口的编号等于m。
在本实施方式中，处理器通过调整m的值，即可把端口代理装置下的所有第一下游端口枚举到系统。
在另一种可能的实施方式中，PCIe交换设备还包括：一个或多个第二下游端口，所述端口代理装置用于根据端口使能记录确定所述多个第一下游端口使能了代理功能，以及确定所述一个或多个第二下游端口未使能代理功能。其中，所述端口使能记录可以记录在位图寄存器，通过位图寄存器中记录的所述PCIe交换设备的各下游端口对应的位图值来表示各下游端口连接的端点设备是否已经使能代理功能。所述端口使能记录还可以为映射表。
在另一种可能的实施方式中，所述端口代理装置的地址窗口为其下连接的所有第一下游端口的地址窗口的合集。当端口代理装置接收到采用地址路由方式的数据包时，可以通过地址窗口确定数据包是否属于其下的第一下游端口的地址范围。当处理器寻址的地址属于端口代理装置的地址窗口范围内时，则端口代理装置执行对应地址的寻址操作。
在另一种可能的实施方式中，所述端口代理装置还用于接收其下的第一下游端口上报的消息报文，将所述消息报文中携带的所述第一下游端口的BDF替换为自身的BDF，将所述消息报文路由到上游端口。
进一步的，所述消息报文包括错误报文。所述错误报文中携带有所述第一下游端口的错误信息。
当消息报文为错误报文时，所述端口代理装置还用于进一步将记录发送所述错误报文的第一下游端口的状态信息。
所述消息报文还包括中断报文。
在本实施方式中，端口代理装置通过修改第一下游端口发送的消息报文中的BDF，使得第一下游端口对上层系统不可见。
在另一种可能的实施方式中，所述PCIe端点设备的设备标识的位数与功能标识的位数之和为8。
在另一种可能的实施方式中，所述PCIe交换设备为专用集成电路(Application  Specific Integrated Circuit，ASIC)芯片。
在另一种可能的实施方式中，所述PCIe交换设备中包含多个并列的端口代理装置，每个端口代理装置与一组所述第一下游端口相连。其中，各端口代理装置分别为其下的一组第一下游端口提供代理服务，各组中的第一下游端口的下游总线号其他组中的第一下游端口的下游总线号不同。
第二方面，本申请提供了另一种PCIe交换设备，包括：处理器、存储器、上游端口和多个第一下游端口，所述存储器中存储有指令，当处理器执行所述指令时实现前述第一方面所述的端口代理装置的功能。
第三方面，本申请提供了一种PCIe系统，包括处理器和如权利要求1-19任一所述的PCIe交换设备。
在一种可能的实施方式中，所述PCIe交换设备包括第一PCIe交换设备和第二PCIe交换设备，其中，所述第二PCIe设备的上游端口连接到所述第一交换设备的第二下游端口，所述第二下游端口未使能代理功能。
第四方面，本申请提供了另一种PCIe系统，所述PCIe系统包括第一PCIe交换设备和第二PCIe交换设备，其中，所述第一PCIe交换设备为所述第一方面的PCIe交换设备，所述第二PCIe交换设备为传统的PCIe交换设备，即第二PCIe设备不包括端口代理装置。所述第二PCIe交换设备为所述第一PCIe交换设备的上游或者下游设备。
第五方面，本申请提供了一种处理器芯片，所述处理器芯片包括根复合体RC和如前述各方面所述的PCIe交换设备，所述PCIe交换设备的上游端口连接到所述根复合体RC的根端口，所述PCIe交换设备用于实现前述各方面中的PCIe交换设备的功能。
第六方面，本申请提供了一种扩展PCIe系统的方法，所述PCIe系统包括处理器和PCIe交换设备，所述PCIe交换设备包括：上游端口、端口代理装置和多个第一下游端口，其中，
所述端口代理装置通过所述上游端口连接到所述处理器，所述端口代理装置与所述多个第一下游端口相连，所述第一下游端口用于连接PCIe端点设备，
所述方法包括：
所述处理器通过端口代理装置将所述多个第一下游端口与其下的PCIe端点设备的连接设置为相同的总线号。
具体的，系统枚举软件枚举所述PCIe系统中的PCIe端点设备，在枚举过程中设置所述多个第一下游端口与其下的PCIe端点设备的连接的总线号。
第七方面，本申请提供了一种计算机程序产品，所述计算机程序产品包括在计算机可读存储介质中存储的计算机程序，并且所述计算程序通过处理器进行加载来实现上述第一方面或第一方面任意可能的实现方式的功能的指令。
第八方面，本申请提供了一种计算机可读存储介质，用于存储计算机程序，所述计算程序通过处理器进行加载，来实现上述方面或各方面任意可能的实现方式的功能。
附图说明
图1是一种PCIe系统中总线号分配示意图；
图2是与图1对应的PCIe设备管理拓扑示意图；
图3是本申请实施例提供的一种PCIe系统中总线号分配示意图；
图4是与图3对应的PCIe设备管理拓扑示意图；
图5是本申请实施例提供的一种DPA配置空间示意图；
图6是本申请实施例提供的另一种PCIe系统中总线号分配示意图；
图7是与图6对应的PCIe设备管理拓扑示意图；
图8是本申请实施例提供的另一种PCIe系统中总线号分配示意图；
图9A和图9B是本申请实施例提供的一种枚举PCIe设备的流程示意图；
图10是本申请实施例提供的地址窗口分配示意图；
图11是本申请实施例提供的一种ID路由的方法流程示意图；
图12是本申请实施例提供的一种PCIe交换设备1200的装置结构示意图。
具体实施方式
下面将结合附图对本申请实施方式作进一步地详细描述。
PCIe总线作为处理器系统的局部总线，其作用与PCI总线类似，主要目的是为了连接处理器系统中的外部设备。PCIe总线使用端到端的连接方式，在一条PCIe链路的两端只能各连接一个设备，这两个设备互为数据发送端和数据接收端。
典型的PCIe系统包括根复合体(Root Complex，RC)、PCIe交换设备(即PCIe switch)和PCIe端点设备。其中，RC可以位于处理器上，RC本身占用一个bus号。PCIe switch包含上游端口和下游端口，上游端口用于连接RC的根端口(root port，RP)或者其他PCIe switch的下游端口。RC的根端口、PCIe switch的上游端口和下游端口又可以成为桥设备，桥设备用于连接其他桥设备或者端点设备。
需要说明的是，PCIe switch可以是单独的物理设备，例如物理交换机或交换芯片；也可以是在芯片中实现的硬件功能模块，例如，可以为集成在处理器芯片中硬件功能模块，实现PCIe switch的功能。
如图1所示，示例性的，为一种PCIe系统中总线号分配示意图，其中，处理器CPU上存在多个根端口(Root Port，RP)，图中以2个RP为例，多个RP可以位于一个或多个RC(图中未示出)上。示例性的，RP0可以连接到端点设备或者桥设备(图中未示出)，RP1连接到PCIe switch(交换机)的上游端口(Uplink Port，UP)，PCIe switch包括m+1个下游端口(Downlink Port，DP)，标记为DP_0、DP_1、…、DP_m，每个下游端口下连接有一个端点设备，图1所示的端点设备标记为EP_0、EP_1、…、EP_m。示例性的，连接上游端口UP的根端口RP1占用bus X，RP1与UP之间的连接(link)占用bus N。此时，PCIe switch内部上游端口到m+1个下游端口的连接占用bus N+1，各下游端口与对应的端点设备之间的连接分别占用bus N+2、bus N+3、…、bus N+2+m。其中，X为大于等于0的整数，N为大于X的整数。示例性的，当图示的2个RP位于不同的RC，且RP0所在的左侧RC首先枚举到时，X的取值取决于左侧RC下的PCIe结构占用的总线号的情况，例如，当左侧RC下的PCIe结构占用了5个bus号(即bus 0-4被左侧RC及其下的PCIe结构占用)时，X的取值为5，N的取值为6；当图示的2个RP位于同一RC且该RC为处理器首先枚举到的RC时，X取值为0，假如左侧RC下的PCIe结构占用了5个bus号(即bus 0-4被RC及RP0下的PCIe结构占用)时，此时N的取值为5。
PCIe设备可以分为两种，一种是桥设备，例如根端口RP、上游端口、下游端口，一种是端点设备(Endpoint，EP)。在PCI桥的两端，分别连接了两条总线，分别是上游总线(Primary  Bus)和下游总线(Secondary Bus)。PCIe桥可以管理其下的PCIe子树。桥设备具有Subordinate Bus Number、Secondary Bus Number和Primary Bus Number寄存器，Primary Bus Number寄存器存放该桥设备上游的PCIe总线号，Secondary Bus Number寄存器存放该桥设备下游的PCIe总线号，Subordinate Bus Number寄存器存放该桥设备下游的PCIe子树中编号最大的PCIe总线号，Subordinate Bus Number称为从属总线号。
如图2所示，为与图1对应的PCIe设备管理拓扑示意图。RP1的Primary Bus Number为X，BDF为X:0.0，Secondary Bus Number为N，Subordinate Bus Number为N+2+m。与所述RP1相连的PCIe switch上游端口UP的Primary Bus Number为N，BDF为N:0.0，Secondary Bus Number为N+1，Subordinate Bus Number为N+2+m。PCIe switch下游端口DP_0的Primary Bus Number为N+1，BDF为N+1:0.0，Secondary Bus Number和Subordinate Bus Number为N+2。PCIe switch下游端口DP_1的Primary Bus Number为N+1，BDF为N+1:1.0，Secondary Bus Number和Subordinate Bus Number为N+3。PCIe switch下游端口DP_m的Primary Bus Number为N+1，BDF为N+1:m.0，Secondary Bus Number和Subordinate Bus Number为N+2+m。与下游端口DP_0相连的EP_0的BDF为N+2:0.0，与下游端口DP_1相连的EP_1的BDF为N+3:0.0，与下游端口DP_m相连的EP_m的BDF为N+2+m:0.0。前述的Primary Bus Number、Secondary Bus Number以及Subordinate Bus Number可以遵从PCIe协议的定义，其中，Primary Bus Number用于表示上游总线号，Secondary Bus Number用于表示下游总线号，Secondary Bus Number用于表示挂在本级bus下的PCIe子树中最大的bus号。Secondary Bus Number和Secondary Bus Number指示了本级bus下的PCIe子树的bus号范围。
图1和图2仅是给出一种PCIe系统下可能的总线分配和管理拓扑举例，用于表示总线号分配和管理拓扑的情况。由此可见，由于PCIe为端对端的连接方式，一个PCIe端点设备需要占用一个总线号，从而使得PCIe端点设备的数量收到了总线号数量的限制。
如图3所示，本申请实施例提供了一种PCIe系统中总线号分配示意图。图3所示的PCIe系统中包括PCIe switch(即PCIe交换设备)。本申请实施例在PCIe switch中增加下游端口代理(Downlink Port Agent,DPA)，由DPA为其下游的端口提供代理服务，连接到DPA下游端口的各个端点设备共享同一总线，从而节约了bus的数量，提供了系统可以支持的端点设备的数量。在本申请实施例中，DPA又称为端口代理装置，在具体实现场景中，DPA可以为ASIC芯片或者为软件模块，当DPA为软件模块时，则由处理器加载指令实现DPA功能。
在一种可能的实施方式中，PCIe switch可以是单独的硬件设备，例如交换机或者交换芯片；也可以是集成在其他芯片中的硬件功能模块，该硬件功能模块包括上游端口、端口代理装置和多个下游端口，示例性的，PCIe switch可以集成在主机的处理器芯片中。
图3所示的PCIe系统中总线号分配示意图实现了与前述图1和图2不同的总线号分配逻辑。该PCIe系统包括处理器CPU、PCIe交换设备(如图示的PCIe switch)和PCIe端点设备。本申请后续实施例以PCIe switch为例对方案进行说明。
处理器上存在多个根端口(图中以2个RP为例)，多个RP可以位于一个或多个RC(图中未示出)上。示例性的，RP0可以连接到端点设备或者桥设备(图中未示出)，RP1连接到PCIe switch的上游端口UP，PCIe switch包括m+1个下游端口，标记为DP_0、DP_1、…、DP_m-1、DP_m，图3所示的端点设备标记为EP_0、EP_1、…、EP_m-1、EP_m。与图1不同的是，图3所示的PCIe switch中新增了DPA，上游端口UP与DPA相连、DPA与PCIe switch的部分或全部下游端口相连，与DPA相连的下游端口可以被称为DPA的下游端口、从属于DPA 的下游端口、或者使能了代理功能的下游端口，示例性的，图中以DP_0、DP_1、…、DP_m-1作为使能了代理功能的下游端口。
示例性的，连接上游端口UP的根端口RP1占用bus X，RP1与UP之间的连接占用bus N。此时，PCIe switch内部上游端口UP到DPA的连接占用bus N+1，下游端口DP_0、DP_1、…、DP_m-1与EP_0、EP_1、…、EP_m-1之间的m-1个连接共享bus N+2，UP到DP_m的连接占用bus N+3。其中，X为大于等于0的整数，N为大于X的整数。X和N的取值举例与图1相同。示例性的，当图示的2个RP位于不同的RC，且RP0所在的左侧RC首先枚举到时，X的取值取决于左侧RC下的PCIe结构占用的总线号的情况，例如，当左侧RRC下的PCIe结构占用了5个bus号(即bus 0-4被左侧RC及其下的PCIe结构占用)时，X的取值为5，N的取值为6；当图示的2个RP位于同一RC且该RC位处理器首先枚举到的RC时，X取值为0，假如左侧RC下的PCIe结构占用了5个bus号(即bus 0-4被占用)，此时N的取值为5。
如图4所示，本申请实施例还提供了一种与图3相对应的PCIe设备管理拓扑示意图。RP1的Primary Bus Number为X，BDF为X:0.0，Secondary Bus Number为N，Subordinate Bus Number为N+3。与所述RP1相连的PCIe switch上游端口UP的Primary Bus Number为N，BDF为N:0.0，Secondary Bus Number为N+1，Subordinate Bus Number为N+3。PCIe switch中的DPA的Primary Bus Number为N+1，BDF为N+1:0.0，Secondary Bus Number为N+2，Subordinate Bus Number为N+2。由于DP_0、DP_1、…、DP_m-1使能了代理功能，由DPA为这m个DP提供代理，因此，DP_0、DP_1、…、DP_m-1这m个DP的Primary Bus Number、Secondary Bus Number，以及Subordinate Bus Number与DPA相同。EP_0的BDF为N+2:0.0,EP_1的BDF为N+2:1.0,…，EP_m-1的BDF为N+2:m-1.0。DP_m未使能代理功能，DP_m的BDF为N+1:1.0，DP_m的Primary Bus Number为N+1，Secondary Bus Number为N+3，Subordinate Bus Number为N+3。
前述图3和图4给出了本申请实施例提供的PCIe switch中使能了DPA功能后的bus分配和管理拓扑。具体的，DPA可以代理所述PCIe switch的多个下游端口功能，将代理的多个下游端口对系统呈现为一个下游端口，示例性的，一个DPA可以代理32个PCIe switch的下游端口。
DPA在对事务层协议(Transaction Layer Protocol，TLP)包进行路由时，将TLP包携带的BDF中的设备标识(5bits device ID)替换为对应的端点设备连接的下游端口编号后，再路由TLP包到处理器。具体的，当所述PCIe端点设备发送的数据包为请求TLP包时，所述端口代理装置具体用于将所述请求TLP包携带的请求者(Requester)BDF中的设备标识替换为与所述PCIe端点设备相连的第一下游端口的编号，或者，当PCIe端点设备发送的数据包为完成TLP包时，所述端口代理装置具体用于将所述完成TLP包携带的完成者(Completer)BDF中的设备标识替换为与所述PCIe端点设备相连的第一下游端口的编号。通过上述替换操作，DPA将其下的下游端口连接的PCIe端点设备映射到所述DPA的下级总线上。此时，DPA下的各个下游端口连接的多个PCIe端点设备共同使用同一个bus号，从而解决了现有技术中需要为每个PCIe switch下游端口连接的端点设备必须分配一个下级bus的问题，避免了PCIe系统的端点设备数量必须小于256的限制(在实际产品实现时，由于RC和其他桥设备还会占用bus号，总的PCIe端点数量可能会少于248)。更进一步的，本申请实施例还避免了现有技术中，PCIe switch下游端口连接的PCIe设备未使用完所有device(5bits).function(3bits)表示的256个PCIe ID造成的浪费问题。
示例性的，现有技术中，PCIe switch连接m个PCIe端点设备需要占用m+2个bus号，而本申请实施例中，当PCIe switch的m个下游端口均使能代理功能时，PCIe switch连接m个PCIe端点设备仅占用3个bus，从而提高了PCIe系统支持的端口设备的数量，提高bus利用率。
本申请实施例通过在PCIe switch内增加DPA，无需改动现BIOS、OS、PCIe子系统和PCIe设备驱动。
通过本申请实施例公开的上述方案可以大大增加一个PCIe可以连接的端点设备的数量，本申请实施例举例如下：实现一个计算机系统支持7648个只有function 0的PCIe端点设备(如固态硬盘)扩展的方式如下：
8个root port占用1个bus，每个root port级联一个PCIe switch，每个pcie switch有32个DPA，每个PCIe switch连接32*32个只有function 0的PCIe端点，n/32+34*n＝256,计算式中n为pcie switch数量，每个pcie switch有32个DPA，每个DPA可以代理32个下游端口需要使用34个bus。计算得出n＝7.52,取整n＝7。那么通过PCIe switch级联的设备最多为7*32*32＝7168，一共使用了1+7*34＝239个bus，剩余17个bus还可以继续扩展连接设备(17-2)*32＝480,最大设备数量为7168+480)。
如图5所示，示例性的，本申请实施例提供了一种PCIe switch中DPA配置空间示意图，与PCIe协议定义的DP的配置空间的不同之处在于：
DPA配置空间寄存器BAR1中映射了归属于所述DPA的所有DP的的配置空间寄存器的配置空间。DPA驱动可以通过BAR1寄存器间接访问所述DPA下的所有DP的配置空间，处理PCIe switch DP链路、错误和热插拔等。
DPA配置空间新增DPA功能结构，所述DPA功能结构包括Enable使能寄存器和位图寄存器。其中，DPA Enable寄存器bit0用于使能/禁止DPA功能，示例性的，当取值为1时，表示DPA功能被使能。位图寄存器DPA Port Bit Map LSB/MSB实现32位位图(PCIe switch DP数量超过32时可以继续扩充)，用于配置DPA需要代理的各个DP，其中，位图的每一位表示对应编号的DP是否使能了代理功能。LSB全称为Least Significant Bit，在二进制数中是指最低有效位，MSB全称为Most Significant Bit，在二进制数中是指最高有效位。MSB位于二进制数的du最左侧，位于二进制数的最右侧。
示例性的，结合图3，下游端口DP_0、DP_1、…、DP_m中，DP_0、DP_1、…、DP_m-1使能了代理功能，DP_m未使能代理功能，此时，DPA位图寄存器大于等于m+1位，其中，位图中用于表示DP_0、DP_1、…、DP_m-1的位取值可以为1，位图中表示DP_m的位取值为0，取值为1的位表示该位对应的下游端口已经使能了代理功能，取值为0的位表示该位对应的下游端口未使能代理功能。
本申请实施例提供的包含DPA的PCIe switch可以适用于各种PCIe架构。例如，多个包含DPA的PCIe switch可以组成多层架构；包含DPA的PCIe switch也可以与传统的PCIe switch在PCIe系统中混合使用；一个PCIe switch中也可以实现多个DPA；从而在单个DPA下可以挂载的下游端口数量有限的情况下，通过在PCIe switch中用多个DPA来提高PCIe系统可以支持的端点设备的数量。
示例性的，如图6所示，为本申请实施例提供的另一种PCIe系统结构示意图，图示的PCIe系统中，包含DPA的两个PCIe switch串行连接，PCIe switch1的UP连接到PCIe switch 0的DP_m口。此时，bus号的分配请参阅图6。
需要说明的是，前述图3和图6中，DPA下的所有下游端口共享相同的下游总线号。在另一种可能的实施方式中，DPA下的部分(非全部)下游端口共享相同的下游总线号。此时，所述端口代理装置的下游总线号小于归属总线号。归属总线号减去下游总线号的值小于前述第一下游端口的数量。
如图7所示，为本申请实施例提供的与图6对应的PCIe设备管理拓扑示意图。RP1的Primary Bus Number为X，BDF为X:0.0，Secondary Bus Number为N，Subordinate Bus Number为N+6。与所述RP1相连的PCIe switch 0的上游端口UP的Primary Bus Number为N，BDF为N:0.0，Secondary Bus Number为N+1，Subordinate Bus Number为N+6。PCIe switch 0中的DPA的Primary Bus Number为N+1，BDF为N+1:0.0，Secondary Bus Number为N+2，Subordinate Bus Number为N+2。由于DP_0、DP_1、…、DP_m-1使能了代理功能，由DPA为这m个DP提供代理，因此，DP_0、DP_1、…、DP_m-1这m个DP的Primary Bus Number、Secondary Bus Number，以及Subordinate Bus Number与DPA相同。连接到PCIe switch 0的DP_0、DP_1、…、DP_m-1的各端点设备的BDF分别为N+2:0.0、N+2:1.0、…、N+2:m-1.0。PCIe switch 0的DP_m未使能代理功能，DP_m的BDF为N+1:1.0，DP_m的Primary Bus Number为N+1，Secondary Bus Number为N+3，Subordinate Bus Number为N+6。PCIe switch 1的UP连接到PCIe switch 0的DP_m。PCIe switch 1的UP的Primary Bus Number为N+3，BDF为N+3:0.0，Secondary Bus Number为N+4，Subordinate Bus Number为N+6。PCIe switch 1中的DPA的Primary Bus Number为N+4，BDF为N+4:0.0，Secondary Bus Number为N+5，Subordinate Bus Number为N+5。由于PCIe switch 1中的DP_0、DP_1、…、DP_m-1使能了代理功能，由DPA为这m个DP提供代理，因此，PCIe switch 1中的DP_0、DP_1、…、DP_m-1，这m个DP的Primary Bus Number、Secondary Bus Number，以及Subordinate Bus Number与PCIe switch 1的DPA相同。连接到PCIe switch 1的DP_0、DP_1、…、DP_m-1的各端点设备的BDF分别为N+5:0.0、N+5:1.0、…、N+5:m-1.0。PCIe switch 1的DP_m未使能代理功能，DP_m的BDF为N+4:1.0，DP_m的Primary Bus Number为N+4，Secondary Bus Number为N+6，Subordinate Bus Number为N+6。连接到PCIe switch 1的DP_m的端点设备的BDF为N+6:0.0。
示例性的，如图8所示，为本申请实施例提供的另一种PCIe系统结构示意图，图示的PCIe系统中，PCIe switch包含并列的n个DPA，每个DPA下有m个下游端口使能了代理功能，PCIe switch的DP_i未使能代理功能。此时，bus号的分配请参阅图8中的内容。各个DPA下的下游端口数量可以不同，下游端口是否使能代理功能可以灵活配置。
需要说明的是，前述各实施例中使能了代理功能的下游端口可以称为第一下游端口，未使能代理功能的下游端口可以称为第二下游端口。
前述PCIe系统的结构示意图仅是可能的实现方式，可以理解的是，本申请实施例提供的PCIe switch可以灵活的出现在各种PCIe系统结构中，使用新增的DAP为PCIe switch的下游端口提供代理功能。
结合前述的实施例，本申请实施例还提供了DPA在PCIe系统中的工作流程。所述工作流程包括以下的至少一种：PCIe switch上电或者复位初始化；PCIe设备枚举；PCIe内存地址资源分配；TLP包ID路由；TLP包地址路由；TLP包隐式路由；端点设备热插拔；错误检测和处理等等。
PCIe switch上电或者复位初始化：
PCIe switch上电/复位初始化时，PCIe switch通过固件FirmWare或者带电可擦可编程只读存储器(Electrically Erasable Programmable read only memory，EEPROM)等方式，完成DPA功能结构中DPA Enable寄存器bit0和DPA位图寄存器Port Bit Map LSB/MSB的设置。示例性的，DPA Enable寄存器bit0置1，DPA Port Bit Map LSB/MSB两个寄存器对应的位图bit0～bitm-1(示例性的，m小于31)中需要使能DPA代理的端口对应的位置为1，以此使能DPA功能和配置需要DPA代理的下游端口DP。
如图9A和9B所示，结合图3所示的PCIe系统结构，本申请实施例提供的一种枚举PCIe设备的流程示意图，包括：
901：处理器按顺序枚举bus 0、bus 1、…、bus N-1连接的PCIe设备。
示例性的，本申请实施例假设bus 0、bus 1、…、bus N-1下连接的均为传统的PCIe设备，此时枚举过程遵从于PCIe协议的定义。具体的，可以由处理器运行系统枚举软件(BIOS和/或OS)执行枚举过程。本申请实施例中并不限定枚举的触发，在一种可能的实施方式中，可以遵从PCIe协议的定义。
902：处理器对bus N上连接的设备进行枚举，将PCIe switch上游端口的Primary Bus Number寄存器写为N，Secondary Bus Number为N+1，Subordinate Bus Number为N+1。
示例性的，步骤902包括下述过程：
处理器向PCIe switch的上行端口UP发送配置读请求，所述配置读请求用于枚举bus为N，device为0，function为0的设备，获取该设备的Device ID和Vendor ID。PCIe设备配置空间中记录的Device ID和Vendor ID用于记录PCIe设备的设备标识和厂商标识，其中，配置空间中的设备标识为厂商写入的，与BDF中的设备标识不同。
PCIe switch的上行端口UP检测到本端口的Primary Bus Number为0，Secondary Bus Number为0，Subordinate Bus Number为0，向处理器返回上游端口配置空间中记录的Device ID和Vendor ID寄存器值。需要说明的是，在枚举操作之前，各端口的Primary Bus Number、Secondary Bus Number、Subordinate Bus Number的初始值均为0。
在处理器确定接收到PCIe switch上游端口返回的Device ID和Vendor ID的值为非0xFFFF时，判定总线N上存在PCIe设备。处理器发送对bus为N，device为0，function为0的设备PCI Header Type寄存器配置读请求。需要说明的是，Device ID和Vendor ID的值是指的Device ID和Vendor ID组合起来的值。
PCIe switch的上行端口UP接收处理器发出的对PCI Header Type寄存器的配置读请求，检测到本端口的Primary Bus Number为0，Secondary Bus Number为0，Subordinate Bus Number为0，向处理器返回PCIe switch上游端口配置空间PCI Header Type寄存器的值。
根据PCIe协议定义，PCI-PCI桥的PCI Header Type寄存器值为10000001b。处理器根据PCIe协议识别到bus为N，device为0，function为0的设备为PCI-PCI桥。
处理器发送对bus为N，device为0，function为0的设备配置空间PCI Express Capability寄存器的配置读请求。
PCIe switch的上行端口接收对PCI Express Capability寄存器的配置读请求，检测到本端口的Primary Bus Number为0，Secondary Bus Number为0，Subordinate Bus Number为0，向处理器返回PCIe switch上游端口配置空间PCI Express Capability寄存器的值。
处理器读取到PCIe switch上游端口返回的PCI Express Capability寄存器值为非0xFFFF，且Device/Port Type字段为0101b时，处理器根据PCIe协议识别到bus为N，device 为0，function为0的设备为PCIe switch上游端口。
处理器向bus为N，device为0，function为0的设备发送配置写请求，用于将PCIe switch上游端口的Primary Bus Number寄存器写为N，Secondary Bus Number寄存器写为N+1，Subordinate Bus Number寄存器写为N+1。
在完成对bus N连接的设备的枚举后，处理器继续枚举bus N+1连接的设备，在本申请实施例中，即，处理器枚举PCIe switch中的DPA。
步骤903：处理器向PCIe switch的上行端口UP发送配置读请求，所述配置读请求用于枚举bus为N+1，device为0，function为0的设备，用于获取该设备的Device ID和Vendor ID。
步骤904：由于步骤903中配置读请求携带的bus为N+1，PCIe switch上游端口在确定本端口的Secondary Bus Number为N+1，Subordinate Bus Number为N+1后，将该配置读请求转入内部路由，发送给DPA。
步骤905：DPA检测到本端口的Primary Bus Number为0，Secondary Bus Number为0，Subordinate Bus Number为0，DPA进一步根据配置空间enable寄存器bit0确定PCIe switch DPA已使能，根据端口使能记录确定存在已使能了代理功能的下游端口。DPA向处理器返回本端口配置空间中记录的Device ID和Vendor ID寄存器值。
需要说明的是，PCIe设备配置空间记录的vendor ID为厂商的ID，deviceID厂商提供的该PCIe设备的ID，Device ID和Vendor ID用于表示PCIe设备的合法性。与之不同的是，BDF中的Device ID是处理器枚举PCIe设备时分配的，用于在管理PCIe设备，因此BDF中的device ID跟PCIe设备配置空间记录的device ID不同。
其中，所述端口使能记录可以记录在位图寄存器，通过位图寄存器中记录的所述PCIe交换设备的各下游端口对应的位图值来表示各下游端口连接的端点设备是否已经使能代理功能。所述端口使能记录还可以为映射表。所述位图寄存器可以为DPA Port Bit Map LSB/MSB寄存器。
步骤906：处理器接收DPA返回的Device ID和Vendor ID，确定Device ID和Vendor ID的值为非0xFFFF时，判定总线N+1上存在PCIe设备。处理器发送对bus为N+1，device为0，function为0的设备PCI Header Type寄存器配置读请求。
步骤907：DPA接收处理器发出的对PCI Header Type寄存器的配置读请求，检测到本端口的Primary Bus Number为0，Secondary Bus Number为0，Subordinate Bus Number为0，向处理器返回DPA端口配置空间PCI Header Type寄存器的值。
根据PCIe协议定义，PCI-PCI桥的PCI Header Type寄存器值为10000001b。处理器根据PCIe协议识别到bus为N+1，device为0，function为0的设备为PCI-PCI桥。
步骤908：处理器发送对bus为N+1，device为0，function为0的设备配置空间PCI Express Capability寄存器的配置读请求。
步骤909：DPA接收对PCI Express Capability寄存器的配置读请求，检测到本端口的Primary Bus Number为0，Secondary Bus Number为0，Subordinate Bus Number为0，向处理器返回DPA端口配置空间PCI Express Capability寄存器的值。
处理器读取到DPA返回的PCI Express Capability寄存器值为非0xFFFF，且Device/Port Type字段为0110b时，处理器根据PCIe协议识别到bus为N+1，device为0，function为0的设备为PCIe switch下游端口，即处理器将所述DPA识别为PCIe switch下游端口。
步骤910：处理器向bus为N，device为0，function为0的设备发送配置写请求，用 于将PCIe上游端口的Primary Bus Number写为N，Secondary Bus Number为N+1，Subordinate Bus Number为N+2。
步骤911：处理器向bus为N+1，device为0，function为0的设备发送配置写请求，用于将DPA的Primary Bus Number寄存器写为N+1，Secondary Bus Number为N+2，Subordinate Bus Number为N+2。
步骤912：DPA接收处理器向bus为N+1，device为0，function为0的设备发送配置写请求，将Primary Bus Number写为N+1，Secondary Bus Number为N+2，Subordinate Bus Number为N+2。进一步的，DPA根据位图寄存器的取值，将使能了DPA代理功能的下游端口的Primary Bus Number写为N+1，Secondary Bus Number为N+2，Subordinate Bus Number为N+2。示例性的，与图3对应，将下游端口DP_0、DP_1、…、DP_m-1的Primary Bus Number写为N+1，Secondary Bus Number为N+2，Subordinate Bus Number为N+2。
处理器继续枚举bus N+2连接的设备，即处理器枚举DPA下的下游端口0、1、…、m-1连接的各端点设备。
步骤913：处理器发送配置读请求，所述配置读请求用于枚举bus为N+2，device为0，function为0的设备，用于获取该设备的Device ID和Vendor ID。
步骤914：由于步骤913中配置读请求携带的bus为N+1，PCIe switch上游端口在确定本端口的Secondary Bus Number为N+1，Subordinate Bus Number为N+2后，将该配置读请求转入内部路由，发送给DPA。
步骤915：DPA检测到本端口的Primary Bus Number为N+1，Secondary Bus Number为N+2，Subordinate Bus Number为N+2，DPA进一步根据配置空间enable寄存器bit0确定PCIe switch DPA已使能，根据位图寄存器DPA Port Bit Map LSB/MSB两个寄存器的值确定存在已使能了代理功能的下游端口。DPA将配置读请求路由到下游端口DP_0。
下游端口DP_0检测到本端口的Primary Bus Number为N+1，Secondary Bus Number为N+2，Subordinate Bus Number为N+2，将配置读请求路由到与其连接的端点设备EP_0.
步骤916：端点设备EP_0返回完成TLP(Completion TLP)包到下游端口DP_0，所述完成TLP包携带端点设备EP_0的Device ID和Vendor ID寄存器值。下游端口DP_0将所述完成TLP包路由至DPA。
步骤917：DPA接收所述完成TLP包，DPA根据enable寄存器bit0确定DPA功能已使能，根据位图寄存器确定下游端口DP_0已使能代理功能，将所述完成TLP包中设备标识转换为0，然后将完成TLP包通过PCIe switch上游端口路由至处理器。其中，所述完成TLP报文的设备标识携带在完成者BDF中，完成者BDF又称为完成者ID。
步骤918：处理器接收到所述完成TLP包，确定所述完成TLP包中的Device ID和Vendor ID的值为非0xFFFF时，判定总线N+2上存在PCIe设备。处理器发送对bus为N+2，device为0，function为0的设备PCI Header Type寄存器配置读请求。
步骤919：与前述方案类似，PCIe端点设备EP_0返回本设备的PCI Header Type寄存器值。处理器读取到PCIe端点设备EP_0返回的PCI Header Type寄存器值为00000000b。系统枚举软件根据PCIe协议识别到bus为N+2，device为0，function为0的设备为端点设备，且只含有1个function。
步骤920：类似地，处理器继续读取bus为N+2，device为0，function为0的设备配置空间PCI Express Capability寄存器。
步骤921：PCIe端点设备EP_0返回PCI Express Capability寄存器值。系统枚举软件读取到PCIe端点设备EP_0返回的PCI Express Capability寄存器值为非0xFFFF，且Device/Port Type字段为0000b。系统枚举软件根据PCIe协议识别到bus为N+2，device为0，function为0的设备为端点设备。
若端点设备含有多个function，系统枚举软件读取到PCIe端点设备EP_0返回的PCI Header Type寄存器值为10000000b。系统枚举软件根据PCIe协议识别到bus为N+2，device为0，function为0的设备为端点设备，且只含有多个function。则系统枚举软件重复上述步骤将bus为N+2，device为0，function为1～7的端点设备枚举进系统。
在处理器(系统枚举软件)完成对下游端口DP_0的枚举后，需要扫描bus N+2下所有的设备功能。在一种具体的实施方式中，计算机操作系统内核中增加强制扫描的启动参数，示例性的，对于Linux系统而言，可以在Linux内核中增加启动参数pci＝pcie_scan_all，使得处理器可以强制枚举bus N+2下的所有设备功能，即共执行256次扫描。在处理器在接收到EP_0返回的PCI Header Type寄存器值为00000000b时，避免处理器因为发现EP_0为只有1个function的设备，误以为已完成bus N+2的总线扫描，从而遗漏了对bus N+2总线上其他设备功能的扫描。示例性的，还可以参与其他参数设置的方式使得处理器可以对DPA的下级总线连接的设备执行完整扫描，本申请实施例不再赘述。
通过上述设置，处理器在完成对bus为N+2，device为0的设备的枚举后，在步骤922中，处理器继续对bus为N+2，device为1至m-1，function为1至7的设备功能执行枚举，m为大于等于1的整数。具体的枚举过程与处理器对bus为N+2，device为0，function为0的设备的枚举过程相同。本申请实施例不再赘述。
需要说明的是，当使能了代理功能的下游端口未连接PCIe端点设备时，处理器在枚举过程中跳过该端口，同时跳过该端口对应的device ID，继续枚举其他的下游端口。
在处理器完成对bus N+2下所有的端点设备的枚举后，处理器继续枚举，遍历bus N+1。
步骤923：处理器发送针对bus为N+1，device为m，function为0的设备的配置读请求，所述配置读请求用于获取该设备的Device ID和Vendor ID。
示例性的，图3所示的PCIe架构图中，在完成DPA下的下游端口DP_0至DP_m-1连接的端点设备的枚举后，处理器继续枚举下游端口DP_m连接的设备，此时，配置读请求中带的device ID为1(即对应用图3的架构，对DP_m枚举过程中，下述步骤中的m取值为1)。需要说明的是，当存在多个下游端口未使能代理功能时，此处的配置读请求的device ID逐次加1。
步骤924：由于步骤923中配置读请求携带的bus为N+1，PCIe switch上游端口在确定本端口的Secondary Bus Number为N+1，Subordinate Bus Number为N+2后，将该配置读请求转入内部路由，发送给DP_m。
步骤925：PCIe switch下游端口DP_m检测到本端口的Primary Bus Number为0，Secondary Bus Number为0，Subordinate Bus Number为0，向处理器返回下游端口DP_m的Device ID和Vendor ID寄存器值。
处理器确定PCIe switch下游端口DP_m返回的Device ID和Vendor ID的值为非0xFFFF，判定总线bus为N+1，device为m，function为0的设备存在。
步骤926：类似地，处理器继续读取bus为N+1，device为m，function为0的设备配置空间PCI Header Type寄存器。处理器获取下游端口DP_m返回的PCI Header Type寄存 器值为10000001b。系统枚举软件根据PCIe协议识别到bus为N+1，device为m，function为0的设备为PCI-PCI桥。
步骤927：类似地，处理器继续读取bus为N+1，device为m，function为0的设备配置空间PCI Express Capability寄存器。处理器读取到PCIe switch下游端口DP_m返回的PCI Express Capability寄存器值为非0xFFFF，且Device/Port Type字段为0110b。系统枚举软件根据PCIe协议识别到bus为N+1，device为m，function为0的设备为PCIe switch下游端口。
步骤928：处理器向bus为N，device为0，function为0的设备配置写Primary Bus Number寄存器为N，Secondary Bus Number为N+1，Subordinate Bus Number为N+3。即PCIe switch上游端口的Primary Bus Number寄存器为N，Secondary Bus Number为N+1，Subordinate Bus Number为N+3。
步骤929：处理器向bus为N+1，device为m，function为0的设备配置写Primary Bus Number寄存器为N+1，Secondary Bus Number为N+3，Subordinate Bus Number为N+3。即PCIe switch下游端口DP_m的Primary Bus Number寄存器为N+1，Secondary Bus Number为N+3，Subordinate Bus Number为N+3。
通过步骤924-929，处理器完成DP_m的枚举，进一步的，处理器继续枚举bus N+3，进行EP_m的枚举。
当存在其他未使能代理功能的下游端口时，处理器采用相同方式，继续枚举其他下游端口以及其他下游端口下连接的PCIe设备。
通过上述方式，处理器完成了图3所示的PCIe系统的枚举。在其他可能的实施方式中，当PCIe系统存在更多的下级PCIe switch以及PCIe switch存在更多的端口时，重复执行前述的步骤，即可完成各种PCIe系统中的设备枚举。
PCIe内存地址资源分配：
处理器(运行系统枚举软件)为PCIe系统中各PCIe设备分配内存地址，在一种可能的实施方式中，处理器可以根据传统PCIe设备内存(memory)地址资源分配算法进行内存地址分配。
处理器首先遍历各PCIe端点设备声明需要的内存地址资源，然后从系统内存地址资源中申请相应的地址空间，再将起始地址写到PCIe端点设备BAR寄存器，并调整该端点设备至RP的路径上各桥设备的地址窗口。例如，PCI-PCI桥的Memory Limit寄存器和Memory Base寄存器表示的32位地址窗口，以及Prefetchable Memory Limit寄存器、Prefetchable Memory Base寄存器、Prefetchable Memory Upper Base Address寄存器和Prefetchable Memory Upper Limit Address寄存器表示的64位地址窗口。
具体的，PCIe switch DPA在将系统枚举软件针对PCIe端点设备的Memory Base寄存器的配置写TLP包路由至PCIe switch下游端口时，进行记录。PCIe switch可以感知下游端口连接的所有端点设备的32位/64位起始地址。桥设备的地址窗口(基于PCI-PCI桥的32位和64位地址窗口)必须包含其下游全部PCIe设备使用的地址范围，则PCIe switch固件FW可根据各下游端口连接的PCIe端点设备起始地址计算出结束地址，并将其设置到下游端口Memory Limit寄存器和Memory Base寄存器Prefetchable Memory Limit寄存器、Prefetchable Memory Base寄存器、Prefetchable Memory Upper Base Address寄存器和 Prefetchable Memory Upper Limit Address寄存器。示例性的，如图10所示，为本申请实施例提供的一种内存地址资源分配图。PCIe系统架构中，位于上游的设备的地址窗口包含其各层下游设备的地址窗口。图10的示例中，各地址窗口连续。具体的，PCIe switch上游端口的地址窗口为[A,Z],PCIe switch中的DPA的地址窗口为[A,Y]，PCIe switch中的下游端口DP_0、DP_1、…、DP_m-1使能了代理功能，此时，PCIe switch DPA代理的DP_0的地址窗口为[A,B]，DP_1的地址窗口为[B,C]，…，DP_m-1的地址窗口为[X,Y]，未使能代理功能的DP_m的地址窗口为[Y,Z]。在一种可能的实施方式中，当存在足够容量的寄存器记录离散的地址窗口时，各设备的地址窗口也可以采用不连续的地址空间。
PCIe协议定义了ID路由(ID Based Routing)、地址路由(Address Based Routing)以及隐式路由(implicit routing)等工作流程，本申请实施例提供的包含DPA的PCIe switch应用在PCIe架构中时，前述的工作流程也需要做相应的调整。
如图11所示，为本申请实施例提供的一种ID路由的方法流程示意图。ID路由一般应用在配置请求(配置读请求/写请求)的传输流程中。所述方法包括：
步骤1101：PCIe switch上游端口接收来自处理器的配置请求，确定配置请求携带的BDF中的bus号是否落入所述上游端口的下级bus和从属bus的范围，如果是，则将配置请求转入PCIe switch内部路由，发送给DPA。
所述配置请求可以为处理器发出的配置读/写请求TLP包。需要说明的是，本申请实施例是以配置请求发送到DPA为例进行说明，当上游端口根据配置请求携带的BDF需要将配置请求路由到未使能代理功能的下游端口时，可以按照PCIe协议规定的流程执行。
步骤1102：DPA接收所述配置请求，确定所述配置请求携带的目标BDF中的bus号是否落入DPA的下级bus和从属bus的范围，如果是，DPA根据所述配置请求携带的目标BDF中的D(即设备标识，可以用device(5bits)表示)将配置请求路由到对应的下游端口，其中所述配置请求携带设备标识的值作为下游端口的编号，DPA将配置请求携带的目标BDF中的设备标识改成0。
步骤1103：下游端口将配置请求路由到与该下游端口连接的端点设备。
步骤1101-1103中对配置请求处理的具体举例可以参见步骤913至步骤916。
步骤1104：DPA配置请求携带的BDF中的bus号不属于DPA的下级bus和从属bus定义的范围，则说明发生了路由错误。此时DPA可以参考协议定义的桥设备进行错误处理的流程来操作，本申请实施例并不进行限定。
步骤1105：PCIe switch上游端口接收处理器发出的完成TLP包，确定所述完成TLP包携带请求者BDF中的bus号是否落入所述上游端口的下级bus和从属bus的范围，如果是，则将所述完成TLP包转入PCIe switch内部路由，发送给DPA。
步骤1106：DPA接收所述完成TLP包，确定所述完成TLP包携带的请求者BDF中的bus号是否落入DPA的下级bus和从属bus的范围，如果是，DPA将所述所述完成TLP包携带的请求者BDF中的D(即设备标识，可以用device(5bits)表示)更改为0，然后将所述完成TLP包路由到对应的下游端口，其中所述完成TLP包携带的请求者BDF中的原始设备标识的值作为下游端口的编号。
步骤1107：下游端口将所述完成TLP包路由到与该下游端口连接的端点设备。
步骤1108：DPA确定所述完成TLP包携带的请求者BDF中的bus号未落入DPA的下级bus和从属bus的范围时，表示发生了路由错误，执行错误处理操作。
步骤1109：PCIe switch下游端口接收来自端点设备的完成TLP包，将所述完成TLP包发送给DPA。
步骤1110：DPA接收所述完成TLP包，确定发送所述完成TLP包的下游端口的编号，根据enable寄存器bit0确定DPA功能已使能，且根据位图寄存器的值确定发送所述完成TLP包的下游端口已使能代理功能后，DPA将所述完成TLP包中携带的完成者BDF中的设备标识更改为所述下游端口的编号，DPA将所述完成TLP包通过上游端口路由到处理器。
步骤1111：在接收到所述完成TLP包后，如果DPA根据enable寄存器bit0确定DPA功能未使能，或者，根据位图寄存器的值确定发送所述完成TLP包的下游端口未使能代理功能，则DPA不修改所述完成TLP包的设备标识，而是直接将所述完成TLP包通过上游端口发送到处理器。
包含DPA的PCIe switch执行地址路由的方法逻辑包括：
PCIe switch上游端口收到来自处理器的存储器操作请求(例如，memory读/写请求TLP包)时，判断目标地址是否落入PCIe switch上游端口的地址窗口内。如果目标地址落入PCIe switch上游端口的地址窗口内则进入PCIe switch内部路由。PCIe switch内部路由判断目标地址是否落入PCIe switch DPA的地址窗口内。如果目标地址落入PCIe switch DPA的地址窗口范围，则PCIe switch上游端口将收到的存储器操作请求路由至PCIe switch DPA。DPA判断目标地址是否落入DPA代理的下游端口地址窗口。如果目标地址属于PCIe switch DPA代理的下游端口的地址窗口范围，则将存储器操作请求路由至PCIe switch DPA代理的下游端口。PCIe switch下游端口将存储器操作请求路由至与其相连的PCIe端点设备。如果DPA确定目标地址落入PCIe switch未被DPA代理下游端口的地址窗口内，则表示发生了路由错误。
PCIe下游端口收到来自端点设备的内存操作请求(例如，memory读/写请求TLP包)时，将内存操作请求发送给DPA。DPA检测发送所述内存操作请求的下游端口是否使能了代理功能，如果未使能，DPA将内存操作请求通过上游端口路由到处理器，如果已使能，则DPA将内存操作请求中的设备标识更改为发送所述内存操作请求的下游端口的编号，然后将所述内存操作请求通过上游端口路由到处理器。具体的，内存操作请求的设备标识携带在请求中的BDF中。
包含DPA的PCIe switc执行隐式路由的方法逻辑包括：
PCIe switch上游端口收到来自处理器的隐式TLP包(即消息报文)时，根据隐式TLP包中消息路由子字段进行路由。
DPA接收到上游端口发送的隐式TLP包时，确定发送所述隐式TLP包的下游端口是否使能了代理功能，如果是，则将隐式TLP包中的请求者BDF中的设备标识更改为0，然后，如果否，则直接根据TLP包中消息路由子字段进行路由。
被PCIe switch DPA代理的下游端口主动发出的隐式TLP包经PCIe switch DPA时，DPA需要将TLP包中请求者ID/BDF改为PCIe switch DPA的BDF。
未被PCIe switch DPA代理的下游端口主动发出的隐式TLP包则直接根据TLP包中消息路由子字段进行路由。
在DPA代理的下游端口发生端点设备热插拔时，下游端口上报的中断消息报文经过DPA时，由DPA按照前述隐式路由的规则更改BDF，将中断上报至处理器，触发DPA热插拔中断。DPA执行端口扫描，与前述实施例描述的强制设备功能扫描类似，此时，DPA需要强制扫描 device(5bits).function(3bits)表示的256个设备功能。
在DPA代理的下游端口上报错误消息报文时，DPA将错误消息报文的BDF更改为DPA的BDF，并在DPA和下游端口的配置空间寄存器中记录端口的状态信息，处理器在接收到所述错误消息报文时，遍历DPA下的各下游端口的配置空间寄存器，根据前述记录的端口的状态确定检测到错误的下游端口的编号，从而针对该下游端口执行错误处理。具体的记录端口的状态的配置空间寄存器可以为AER(advanced error reporting)功能寄存器。
需要说的是，在本申请各实施例中，处理器与PCIe switch之间可能存在一层或多层的桥设备，处理器通过与PCIe switch之间的桥设备进行通信。为简便描述，本申请上述各实施例以PCIe switch直连到处理器RP为例进行说明。可以理解的是，当处理器与PCIe switch之间存在传统PCIe桥设备时，中间的PCIe桥设备可以根据PCIe协议的定义处理处理器和本申请实施例提供的PCIe switch之间的通信。
本申请实施例为保证与当前PCIe协议的场景兼容，仅允许DPA将其代理的下游端口连接的PCIe端点设备映射到BDF中的设备标识上，而设备标识为5bits，因此，一个DPA最多只能映射32个下游端口连接的PCIe端点设备。若PCIe端点设备的Fuction小于8个，则仍存在PCIe ID浪费问题。为解决该问题，可设置将DPA映射下游端口连接的PCIe端点设备的设备标识由5bits拓展为6bits或者7bits，相应地，function占用的bit则由3bits缩减为2bits或1bit。此时，DPA映射下游端口连接的PCIe端点设备数量和PCIe端点设备function数量表格如下表所示：
Figure PCTCN2021102622-appb-000001
本申请实施例提供了一种包含DPA的PCIe switch，通过DPA为多个下游端口提供代理，在不改动现有BIOS、OS PCIe子系统和PCIe设备驱动情况下，将多个下游端口对上层系统呈现为一个端口，从而减少了bus的占用，提高了PCIe系统支持的端点设备的数量。具体的，本申请实施例可以增加DPA驱动，而不需要更改PCIe系统其它部分的软件，对现有系统影响小。同时，本申请实施例无需增加NTB等隔离两个PCIe域的电路单元，因此成本低。进一步的，当DPA采用ASIC芯片实现时，执行上游端口与下游端口之间的通信转发的性能高。
如图12所示，为本申请实施例提供的一种PCIe交换设备1200的装置结构示意图，PCIe交换设备1200包括处理器1201、内部连接1202、上游端口1204、多个下游端口1205以及存储器1203。其中，所述处理器1201可以为多个处理器核，所述处理器可以包括多个寄存器，用来存储PCIe交换设备的配置信息。图12所示的PCIe交换设备，端点代理装置(前述实施例中的DPA)可以采用软件模块，即由处理器1201执行存储器1203中的指令，实现前述DPA的功能。所述上游端口1202可以为一个或多个。
可选的，上述处理器1201可以是一个通用中央处理器(central processing unit，CPU)，网络处理器(network processor，NP)，微处理器，或一个或多个用于控制本申请方案程序执行的集成电路。
上述内部连接1202可包括一通路，在上述组件之间传送信息。可选的，内部连接1202为总线。
上述存储器1203可以是只读存储器(read-only memory，ROM)或可存储静态信息和指令的其他类型的静态存储设备，随机存取存储器(random access memory，RAM)或者可存储信息和指令的其他类型的动态存储设备，也可以是电可擦可编程只读存储器(electrically erasable programmable read-only memory，EEPROM)、只读光盘(compact disc read-only memory，CD-ROM)或其他光盘存储、光碟存储(包括压缩光碟、激光碟、光碟、数字通用光碟、蓝光光碟等)、磁盘存储介质或者其他磁存储设备、或者能够用于携带或存储具有指令或数据结构形式的期望的程序代码并能够由计算机存取的任何其他介质，但不限于此。存储器可以是独立存在，通过总线与处理器相连接。存储器也可以和处理器集成在一起。
其中，存储器1203用于存储执行本申请方案的应用程序代码，并由处理器1201来控制执行。处理器1201用于执行存储器1203中存储的应用程序代码，以及配合上游端口1204和下游端口1205，从而使得该装置1200实现本申请前述实施例中描述的功能。
在具体实现中，作为一种实施例，处理器1201可以包括一个或多个CPU，例如图12中的CPU0和CPU1。
在具体实现中，作为一种实施例，该PCIe交换设备1200可以包括多个处理器。这些处理器中的每一个可以是一个单核(single-CPU)处理器，也可以是一个多核(multi-CPU)处理器。这里的处理器可以指一个或多个设备、电路、和/或用于处理数据(例如计算机程序指令)的处理核。
本领域普通技术人员可以理解实现上述实施例的全部或部分步骤可以通过硬件来完成，也可以通过程序来指令相关的硬件完成，所述的程序可以存储于一种计算机可读存储介质中，上述提到的存储介质可以是只读存储器，磁盘或光盘等。
以上所述仅为本申请的可选实施例，并不用以限制本申请，凡在本申请的原则之内，所作的任何修改、等同替换、改进等，均应包含在本申请的保护范围之内。

Claims (34)

  1. 一种PCIe交换设备，其特征在于，包括：上游端口、端口代理装置和多个第一下游端口，其中，
    所述上游端口与所述端口代理装置相连，所述端口代理装置与所述多个第一下游端口相连，所述第一下游端口用于连接PCIe端点设备；
    所述多个第一下游端口与其下的PCIe端点设备的连接共享相同的总线号。
  2. 如权利要求1所述的设备，其特征在于，还包括：一个或多个第二下游端口，
    其中，所述一个或多个第二下游端口与所述上游端口相连。
  3. 如权利要求1或2所述的设备，其特征在于，
    所述端口代理装置与所述多个第一下游端口具备相同的归属总线号。
  4. 如权利要求1-3任一所述的设备，其特征在于，
    所述第一下游端口的编号用于表示与所述第一下游端口连接的PCIe端点设备的设备标识。
  5. 如权利要求4所述的设备，其特征在于，
    所述端口代理装置用于从所述第一下游端口接收PCIe端点设备发送的数据包，将所述数据包中的设备标识替换为所述第一下游端口的编号。
  6. 如权利要求5所述的设备，其特征在于，
    当所述PCIe端点设备发送的数据包为请求TLP包时，所述端口代理装置具体用于将所述请求TLP包携带的请求者BDF中的设备标识替换为与所述PCIe端点设备相连的第一下游端口的编号，或者，
    当所述PCIe端点设备发送的数据包为完成TLP包时，所述端口代理装置具体用于将所述完成TLP包携带的完成者BDF中的设备标识替换为与所述PCIe端点设备相连的第一下游端口的编号。
  7. 如权利要求4-6任一所述的设备，其特征在于，
    所述端口代理装置用于接收所述处理器发送的数据包，将所述数据包中设备标识的值作为接收所述数据包的目的端口的编号，并将所述数据包中的设备标识更改为0，将所述数据包发送到所述目的端口。
  8. 如权利要求7所述的设备，其特征在于，
    当所述处理器发送的数据包为配置请求TLP包时，将所述配置请求TLP包携带的目的BDF中的设备标识替换为0，或者，
    当所述处理器发送的数据包为完成TLP包时，将所述完成TLP包携带的请求者BDF中的设备标识替换为0。
  9. 如权利要求1-8任一所述的设备，其特征在于，
    所述端口代理装置的配置空间寄存器中映射有归属于所述端口代理装置的所有第一下游端口的配置空间寄存器。
  10. 如权利要求1-9任一所述的设备，其特征在于，
    所述端口代理装置的使能寄存器的值用于表示所述端口代理装置的代理功能是否使能。
  11. 如权利要求1-10任一所述的设备，其特征在于，
    所述PCIe交换设备在上电或者复位初始化时，通过固件或者读取非易失性存储器的方 式，设置所述端口代理装置的寄存器。
  12. 如权利要求1-11任一所述的设备，其特征在于，
    所述端口代理装置，用于接收所述上游端口转发的第一配置请求，所述第一配置请求携带所述端口代理装置的管理拓扑对应的各总线号；
    所述端口代理装置，还用于根据所述第一配置请求，设置自身的管理拓扑对应的各总线号，并将其下连接的所有第一下游端口的管理拓扑设置为相同的总线号。
  13. 如权利要求1-12任一所述的设备，其特征在于，
    所述端口代理装置，用于接收处理器发送的第二配置请求，所述第二配置请求携带的总线号为所述端口代理装置的下级总线号、携带的设备标识为m，m为大于等于0的整数；
    所述端口代理装置，还用于根据所述第二配置请求携带的设备标识的值m，将所述第二配置请求路由到对应的目的端口，所述目的端口的编号等于m。
  14. 如权利要求2所述的设备，其特征在于，
    所述端口代理装置用于根据端口使能记录确定所述多个第一下游端口使能了代理功能，以及确定所述一个或多个第二下游端口未使能代理功能。
  15. 如权利要求1-14所述的设备，其特征在于，
    所述端口代理装置的地址窗口为其下连接的所有第一下游端口的地址窗口的合集。
  16. 如权利要求1所述的设备，其特征在于，
    所述端口代理装置还用于接收其下的第一下游端口上报的消息报文，将所述消息报文中携带的所述第一下游端口的BDF替换为自身的BDF，将所述消息报文路由到上游端口。
  17. 如权利要求1所述的设备，其特征在于，所述PCIe交换设备不经过其他桥设备连接到处理器的根端口或者通过其他的桥设备连接到处理器的根端口。
  18. 如权利要求4所述的设备，其特征在于，所述PCIe端点设备的设备标识的位数与功能标识的位数之和为8。
  19. 如权利要求18所述的设备，其特征在于，所述PCIe交换设备为专用集成电路ASIC芯片。
  20. 如权利要求1-19所述的设备，其特征在于，所述PCIe交换设备中包含多个并列的端口代理装置，每个端口代理装置与一组所述第一下游端口相连。
  21. 一种PCIe交换设备，其特征在于，包括：处理器、存储器、上游端口和多个第一下游端口，所述存储器中存储有指令，当处理器执行所述指令时实现如权利要求1-20所述的端口代理装置的功能。
  22. 一种扩展PCIe系统的方法，其特征在于，所述PCIe系统包括处理器和PCIe交换设备，所述PCIe交换设备包括：上游端口、端口代理装置和多个第一下游端口，其中，
    所述端口代理装置通过所述上游端口连接到所述处理器，所述端口代理装置与所述多个第一下游端口相连，所述第一下游端口用于连接PCIe端点设备，
    所述方法包括：
    所述处理器通过端口代理装置将所述多个第一下游端口与其下的PCIe端点设备的连接设置为相同的总线号。
  23. 如权利要求21或22所述的方法，其特征在于，
    所述端口代理装置与所述多个第一下游端口具备相同的归属总线号。
  24. 如权利要求21-23任一所述的方法，其特征在于，
    所述第一下游端口的编号用于表示与所述第一下游端口连接的PCIe端点设备的设备标识。
  25. 如权利要求24所述的方法，其特征在于，所述方法还包括：
    所述端口代理装置从所述第一下游端口接收PCIe端点设备发送的数据包，将所述数据包中的设备标识替换为所述第一下游端口的编号。
  26. 如权利要求24或25所述的方法，其特征在于，所述方法还包括：
    所述端口代理装置接收所述处理器发送的数据包，将所述数据包中设备标识的值作为接收所述数据包的目的端口的编号，并将所述数据包中的设备标识更改为0，将所述数据包发送到所述目的端口。
  27. 如权利要求21-26任一所述的方法，其特征在于，
    所述端口代理装置的配置空间寄存器中映射有归属于所述端口代理装置的所有第一下游端口的配置空间寄存器。
  28. 如权利要求21-27任一所述的方法，其特征在于，
    所述端口代理装置的使能寄存器的值用于表示所述端口代理装置的代理功能是否使能。
  29. 如权利要求21-28任一所述的方法，其特征在于，所述方法还包括：
    所述PCIe交换设备在上电或者复位初始化时，通过固件或者读取非易失性存储器的方式，设置所述端口代理装置的寄存器。
  30. 如权利要求21-29任一所述的方法，其特征在于，所述方法还包括：
    所述端口代理装置接收所述上游端口转发的第一配置请求，所述第一配置请求携带所述端口代理装置的管理拓扑对应的各总线号；
    所述端口代理装置根据所述第一配置请求，设置自身的管理拓扑对应的各总线号，并将其下连接的所有第一下游端口的管理拓扑设置为相同的总线号。
  31. 如权利要求21-30任一所述的方法，其特征在于，所述方法还包括：
    所述端口代理装置接收处理器发送的第二配置请求，所述第二配置请求携带的总线号为所述端口代理装置的下级总线号、携带的设备标识为m，m为大于等于0的整数；
    所述端口代理装置根据所述第二配置请求携带的设备标识的值m，将所述第二配置请求路由到对应的目的端口，所述目的端口的编号等于m。
  32. 如权利要求21-31任一所述的方法，其特征在于，所述PCIe设备还包括一个或多个第二下游端口，所述方法还包括：
    所述端口代理装置根据端口使能记录确定所述多个第一下游端口使能了代理功能，以及确定所述一个或多个第二下游端口未使能代理功能。
  33. 一种PCIe系统，其特征在于，包括处理器和如权利要求1-20任一所述的PCIe交换设备。
  34. 如权利要求33所述的系统，其特征在于，所述PCIe交换设备包括第一PCIe交换设备和第二PCIe交换设备，其中，所述第二PCIe设备的上游端口连接到所述第一交换设备的第二下游端口，所述第二下游端口未使能代理功能。
PCT/CN2021/102622 2020-07-13 2021-06-28 扩展PCIe系统的方法、PCIe交换设备及PCIe系统 WO2022012309A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP21842488.5A EP4167100A4 (en) 2020-07-13 2021-06-28 PCIE SYSTEM EXTENSION METHOD, PCIE SWITCH DEVICE AND PCIE SYSTEM
US18/150,991 US20230161728A1 (en) 2020-07-13 2023-01-06 Method for expanding pcie system, pcie switching device, and pcie system

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN202010677632.7 2020-07-13
CN202010677632 2020-07-13
CN202011620807.7 2020-12-30
CN202011620807.7A CN114006875A (zh) 2020-07-13 2020-12-30 扩展PCIe系统的方法、PCIe交换设备及PCIe系统

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/150,991 Continuation US20230161728A1 (en) 2020-07-13 2023-01-06 Method for expanding pcie system, pcie switching device, and pcie system

Publications (1)

Publication Number Publication Date
WO2022012309A1 true WO2022012309A1 (zh) 2022-01-20

Family

ID=79555014

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/102622 WO2022012309A1 (zh) 2020-07-13 2021-06-28 扩展PCIe系统的方法、PCIe交换设备及PCIe系统

Country Status (3)

Country Link
US (1) US20230161728A1 (zh)
EP (1) EP4167100A4 (zh)
WO (1) WO2022012309A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114840465A (zh) * 2022-05-06 2022-08-02 深圳星云智联科技有限公司 提高PCIe卡兼容性的方法、装置、设备及存储介质
WO2023177947A3 (en) * 2022-02-09 2023-12-14 Microchip Technology Incorporated Switching data based on a bus identifier and a device identifier

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116414759B (zh) * 2023-06-09 2023-09-29 深圳中安辰鸿技术有限公司 计算机交换系统及地址分配、枚举、数据分发方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103180817A (zh) * 2012-07-02 2013-06-26 杭州华为数字技术有限公司 存储扩展装置及服务器
US20150186310A1 (en) * 2013-12-30 2015-07-02 Paul S. Levy Systems and methods for automatic root port to non-transparent bridge switching for a pci express interconnect architecture
CN105531684A (zh) * 2013-09-06 2016-04-27 思科技术公司 通用pci express端口
CN107278299A (zh) * 2015-03-26 2017-10-20 英特尔公司 经由可重配置的虚拟交换机实现次级总线功能性的方法、装置和系统

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5598148B2 (ja) * 2010-08-05 2014-10-01 富士通株式会社 スイッチング装置、情報処理装置、及びスイッチング装置の制御方法
JP5541021B2 (ja) * 2010-09-09 2014-07-09 富士通株式会社 スイッチ装置

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103180817A (zh) * 2012-07-02 2013-06-26 杭州华为数字技术有限公司 存储扩展装置及服务器
CN105531684A (zh) * 2013-09-06 2016-04-27 思科技术公司 通用pci express端口
US20150186310A1 (en) * 2013-12-30 2015-07-02 Paul S. Levy Systems and methods for automatic root port to non-transparent bridge switching for a pci express interconnect architecture
CN107278299A (zh) * 2015-03-26 2017-10-20 英特尔公司 经由可重配置的虚拟交换机实现次级总线功能性的方法、装置和系统

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023177947A3 (en) * 2022-02-09 2023-12-14 Microchip Technology Incorporated Switching data based on a bus identifier and a device identifier
CN114840465A (zh) * 2022-05-06 2022-08-02 深圳星云智联科技有限公司 提高PCIe卡兼容性的方法、装置、设备及存储介质
CN114840465B (zh) * 2022-05-06 2023-08-18 深圳星云智联科技有限公司 提高PCIe卡兼容性的方法、装置、设备及存储介质

Also Published As

Publication number Publication date
US20230161728A1 (en) 2023-05-25
EP4167100A1 (en) 2023-04-19
EP4167100A4 (en) 2023-11-08

Similar Documents

Publication Publication Date Title
WO2022012309A1 (zh) 扩展PCIe系统的方法、PCIe交换设备及PCIe系统
JP7118922B2 (ja) スイッチングデバイス、ペリフェラル・コンポーネント・インターコネクト・エクスプレスシステムおよびその初期化方法
RU2640648C2 (ru) Управление ресурсами для доменов высокопроизводительного межсоединения периферийных компонентов
US8095701B2 (en) Computer system and I/O bridge
JP5763873B2 (ja) データ処理システムの複数のルート複合体間の通信のために共用メモリを初期設定するための方法、コンピュータ・プログラム、およびデータ処理システム
US7571273B2 (en) Bus/device/function translation within and routing of communications packets in a PCI switched-fabric in a multi-host environment utilizing multiple root switches
US10698849B2 (en) Methods and apparatus for augmented bus numbering
CN108984465B (zh) 一种消息传输方法及设备
CN114817103A (zh) 用于扩展外围组件互连高速结构的系统和方法
JP2008152787A (ja) データ処理システム内で実行中の通信ファブリックにコンポーネントをホット・プラグするための方法、プログラム、およびシステム（実行中のＰＣＩｅファブリックにおける新しいコンポーネントのホット・プラグ／除去のためのシステムおよび方法）
US11829309B2 (en) Data forwarding chip and server
JP2008171413A (ja) データ処理システム内で第１のホスト・システムと第２のホスト・システムとの間で通信するための方法、プログラム、およびシステム（ソケット接続および共用メモリを使用するホスト・システム間の通信のためのシステムおよび方法）
US11995019B2 (en) PCIe device with changeable function types and operating method thereof
US11928070B2 (en) PCIe device
US20220300442A1 (en) Peripheral component interconnect express device and method of operating the same
US20220327228A1 (en) PCIe FUNCTION AND OPERATING METHOD THEREOF
CN114006875A (zh) 扩展PCIe系统的方法、PCIe交换设备及PCIe系统
US20210165750A1 (en) Packet Forwarding Method, Intermediate Device, and Computer Device
CN116414759B (zh) 计算机交换系统及地址分配、枚举、数据分发方法
CN115576889A (zh) 链式的多芯片系统及通讯方法
CN116932451A (zh) 一种数据处理方法、主机及相关设备
EP1596307A1 (en) I/O configuration messaging within a link-based computing system
US20230350824A1 (en) Peripheral component interconnect express device and operating method thereof
US20230318606A1 (en) Interface device and method of operating the same
CN117978775A (zh) Pci-e非透明桥的地址转换方法、装置、设备及介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21842488

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021842488

Country of ref document: EP

Effective date: 20230111

NENP Non-entry into the national phase

Ref country code: DE