CN113132199B

CN113132199B - PCIe NTB system implementation management method

Info

Publication number: CN113132199B
Application number: CN202110427205.8A
Authority: CN
Inventors: 苏海亮; 宁佐林; 黄柄权; 孙旭
Original assignee: Wuxi Zhongxing Microsystem Technology Co ltd
Current assignee: Wuxi Zhongxing Microsystem Technology Co ltd
Priority date: 2021-04-21
Filing date: 2021-04-21
Publication date: 2022-07-01
Anticipated expiration: 2041-04-21
Also published as: CN113132199A

Abstract

The invention provides a PCIe NTB system implementation management method, which comprises the steps of setting a first NT module in a first PCIe bus controller; providing a second NT module in a second PCIe bus controller; taking the second NT module as a down-hanging device of a first uplink port of a second PCIe bus controller; and completing address translation and ID translation through the first NT module and the second NT module so as to realize NTB interconnection between the first PCIe bus controller and the second PCIe bus controller. The invention realizes the support of NTB function on the basis of PCIe Switch general architecture, meets NTB application of different scenes, improves port configuration flexibility and reduces link establishment cost.

Description

PCIe NTB system implementation management method

Technical Field

The invention belongs to the field of bus design, and particularly relates to a PCIe NTB system implementation management method.

Background

PCIe (PCI express) is a high-speed serial computer expansion bus standard. PCIe is a tree topology, as shown in the exemplary PCIe topology of FIG. 1. There can only be one Host in the whole topology, and the Host can access all the devices in the topology.

In modern communication and storage scenarios, a scenario application with multiple Host sharing devices is usually required, for example, as shown in fig. 2, both Host1 and Host2 need to access all EP devices (i.e., PCIe Endpoint devices) under PCIe Switch1 and PCIe Switch2, for example, Host1 may access EP10, EP11, EP12, and EP13, and Host1 also needs to access devices of EP20, EP21, EP22, and EP 23. However, since multiple hosts cannot coexist in the same PCIe topology, the multiple hosts must be divided into multiple PCIe topology domains, and each PCIe topology domain has one Host. If Host1 were to access a device under PCIe Switch2, the access operation would need to span both PCIe topology domains. To solve the access across the PCIe topology domain, an NTB method (Non-Transparent Bridge) is introduced, and an NTB port is used for interconnection between the PCIe Switch1 and the PCIe Switch2 shown in fig. 3.

In the existing NTB technology, each PCIe Switch is an independent PCIe topology, and the NT Port of each PCIe Switch includes one DP (Downstream Port) and two EPs, as shown in DP10 and DP20 in fig. 3. When the Host1 performs enumeration management, the Host1 can access the DP10 and the EP10, perform address conversion between the EP10 and the EP11, and complete address conversion from the PCIe Switch1 to the PCIe Switch 2. Host2 can access DP20 and EP20 to perform address translation between EP20 and EP21, and complete address translation from PCIe Switch2 to PCIe Switch 1. For EP11 and EP21, both Host sides cannot access and are used only as address translations. The DP10 and the DP20 are interconnected, and the characteristics of crosslink in the PCIe protocol are utilized for interconnection communication.

It can be seen that, in the PCIe IP of DP in the prior art, two EPs need to be implemented, and the functional modules of the two EPs and the DP are tightly coupled, and there are the following problems in practical application:

(1) each NT port needs to contain one DP port, and two EPs are included in the DP port; when each PCIe port is to support an NT port, resource overhead is increased.

(2) When a limited number of ports support NT ports, the usage scenario is limited. Only limited ports support NTB, and other ports cannot support NTB functions. Because an EP can only be bound to a corresponding DP port one by one, flexibility is lacking.

(3) It is necessary to specify that the NT ports on both sides must be DP ports, thereby placing additional restrictions on the functionality of link establishment using Crosslink characteristics.

Disclosure of Invention

The invention aims to provide a PCIe NTB system implementation management method, which comprises the following steps:

providing a first NT module NT0 in the first PCIe bus controller;

providing a second NT module NT1 in a second PCIe bus controller;

the second NT module NT1 is used as a down-hanging device of a first UP port UP11 of a second PCIe bus controller;

address translation and ID translation are accomplished by the first NT module NT0 and the second NT module NT1 to achieve NTB interconnection between the first PCIe bus controller and the second PCIe bus controller.

Preferably, the first and second PCIe bus controllers are a first PCIe Switch and a second PCIe Switch, respectively.

Preferably, the first NT module NT0 and the second NT module NT1 include a virtual downstream port VDP and a virtual endpoint device VEP, respectively;

the apparatus for taking the second NT module NT1 as a drop-down device of the first upstream port UP11 of the second PCIe bus controller further includes:

the virtual downstream port VDP and the virtual endpoint device VEP of the second NT module NT1 are used as the downstream device of the first upstream port UP11 of the second PCIe Switch.

Preferably, the second NT module NT1 further includes a second downstream port DP20 for link negotiation link establishment with the first upstream port UP 11.

Preferably, the first PCIe Switch further includes UP10, and the virtual downstream port VDP and the virtual endpoint device VEP of the first NT module NT0 are as a drop device of the UP 10.

Preferably, the first NT module NT0 is logically connected to a plurality of the second NT modules NT1, respectively, and the plurality of the second NT modules NT1 are configured as downstream devices of upstream ports of a plurality of second PCIe switches, respectively, so as to establish NTB interconnect between the first PCIe Switch and the plurality of second PCIe switches.

Preferably, the method further comprises:

setting a first NT module NT0 in the second PCIe Switch;

setting a second NT module NT1 in a third PCIe Switch;

taking the second NT module NT1 in the third PCIe Switch as a down-hanging device of the first UP port UP21 of the second PCIe Switch;

address translation and ID translation are completed by the first NT module NT0 in the second PCIe Switch and the second NT module NT1 in the third PCIe Switch, so as to implement NTB interconnection among the first PCIe Switch, the second PCIe Switch, and the third PCIe Switch.

Preferably, the first PCIe Switch and the second PCIe Switch include a plurality of upstream ports UP and/or a plurality of downstream DP ports, and the first NT module NT0 and the second NT module NT1 are mapped to any one of the upstream ports UP and downstream DP ports of the first PCIe Switch and the second PCIe Switch, respectively.

Preferably, the first and second PCIe bus controllers are a first Root complex logic RC1 and a second Root complex logic RC2, respectively.

Preferably, the second NT module NT1 includes a virtual downstream port VDP and a virtual endpoint device VEP;

the virtual downstream port VDP and the virtual endpoint device VEP of the second NT module NT1 are used as the drop device of the first upstream port UP11 of the second Root complete logic RC 2.

Preferably, the first NT module NT0 establishes logical connection with the plurality of second NT modules NT1, respectively, and the plurality of second NT modules NT1 serve as drop-down devices for uplink ports of the plurality of second Root complex logic RCs 2, respectively, so as to establish NTB interconnection between the first Root complex logic RC1 and the plurality of second Root complex logic RCs 2.

Compared with the prior art, the scheme of the invention supports the flexible mapping of the NT module to any UP/DP/RP port, realizes the optimal configuration of the NTB function, saves resources and meets the NTB application of different scenes; the NT function module can be configured to different UP/DP/RP ports, so that the flexibility of port configuration is improved; the chain building is performed without using the characteristic of Crosslink, the chain building characteristic between UP and DP/RP is maintained, and the chain building cost is reduced. Compared with the prior art, the invention can support NTB function on the basis of PCIe Switch (or RC) general architecture.

Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 shows a schematic diagram of a typical topology for PCIe according to the prior art.

Fig. 2 shows a typical topology diagram of a PCIe NTB according to the prior art.

Fig. 3 shows a NTB port organization architecture diagram for PCIe Switch according to the prior art.

Fig. 4 shows a diagram of NTB functional modules of a PCIe Switch according to the first embodiment of the present invention.

Fig. 5 shows an NTB functional block diagram of a PCIe RC according to a second embodiment of the present invention.

Fig. 6 shows a diagram of an NTB functional module of a PCIe Switch according to a third embodiment of the present invention.

Fig. 7 shows an NTB functional block diagram of a hybrid port according to a fourth embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The system implementation management method of the PCIe NTB of the invention provides an NT function module on a general PCIe Switch port, wherein the NT function module comprises VDP and VEP equipment, and completes address and ID conversion among a plurality of NTs to realize the function of the PCIe NTB.

The NTB functional block diagram of PCIe Switch of the present invention is shown in FIG. 4. The interconnection between two switches is realized by means of interconnection of DP and UP (Upstream Port, PCIe Upstream Port). Among them, the DP20 port in Host2 and the UP11 port in Host1 are interconnected. DP20 is the Downstream Port of downlink Port defined in PCIe protocol, UP11 is the Upstream Port defined in PCIe protocol, and DP20 and UP11 perform link negotiation link establishment without performing link establishment based on Crosslink characteristics.

In fig. 4, the PCIe Switch includes NT basic structures, including NT0 and NT1, each NT including a VDP (Virtual downlink Port) and a VEP (Virtual EP).

Among them, VDP and VEP in NT0 are in PCIe topology of PCIe Switch1, VDP and VEP of NT0 are suspended devices in UP10, which can be enumerated by Host1, but cannot be accessed by Host 2.

The VDP and VEP in NT1 are in PCIe topology of PCIe Switch2, and the VDP and VEP of NT1 are on-hook devices of UP11, which can be accessed by Host2, but cannot be accessed by Host 1.

Two PCIe switches are isolated by NT0 and NT1, which ensures that PCIe topology domains do not overlap, and in the PCIe topology of Host1, devices in UP10 and NT0, DP10, DP11, … and DP1N all belong to the topology of Host 1. The device of NT1 and the devices in UP11 and PCIe Switch2 both belong to the PCIe topology of Host 2.

The NT0 and NT1 form a complete NT port functional combination, and together realize the function of NTB. The NT function module of the invention is independent and has no fixed binding relation with a specific DP/UP port. For example, in fig. 4, the NT0 and NT1 modules exist independently, but not in a certain UP/DP, and the NT modules can be flexibly mapped to any UP/DP port, so as to implement the optimal configuration of the NTB function, that is, save resources, meet NTB applications of different scenarios, and improve configurability and scalability of the port.

In an alternative embodiment, the NT function module can be configured to different UP/DP ports with great flexibility. For example, NT0 in fig. 4 is configured for use by UP10 as a device hanging down from UP10, and NT1 is configured for use by UP11 as a device hanging down from UP 11. However, the above configuration is only an example, and NT0 and NT1 may be configured to any PCIe Switch port for use according to different application scenarios.

Each port in the PCIe Switch supports UP/DP configurability, and the port used with the NT function module needs to be configured as an UP port.

Only NT0, NT1 are shown in fig. 4, i.e., PCIe Switch has only one NT functional port at most. In addition, in alternative embodiments, the number of NT functional modules may be expanded as needed. The NT function module only needs to be increased according to the number of required expansion, and NT function ports are correspondingly increased. The dotted line of fig. 4 represents the logical connection established between NT0 and NT 1. It will be understood by those skilled in the art that if the number of NT1 is expanded to plural, the above logical connection relationship can be established between NT0 and each NT1, and between each two NT 1.

The above structure of the present invention does not require special configuration for the PCIe Switch of the peer end (e.g. PCIe Switch2 in fig. 4) linked to the NT functional port, that is, only one standard PCIe Switch is required to be maintained, so that the interface compatibility is improved.

Therefore, on the basis of the existing PCIe Switch function, two PCIe switches are linked through the NT port, only the role of the UP/DP port needs to be configured, the characteristic of crossbar is not needed to be used for building the link, the link building characteristic between the UP and the DP is kept, and the link building cost is reduced. By expanding the number of the NT functional ports, 2 PCIe switches can be realized, and the expansion interconnection of a plurality of PCIe topological structures is realized. Compared with the prior art, the invention can support NTB function on the basis of PCIe Switch general architecture.

FIG. 5 illustrates a NTB functional block diagram of a PCIe RC in accordance with the present invention. Besides PCIe Switch, RC (Root Complex, Root management of PCIe) may also use the method of the present invention to perform NTB function expansion. As shown in fig. 5, NT0 and NT1 constitute an NTB function, and the RP (Root Port) corresponding to NT1 is configured as UP 11. Therefore, the scheme of the invention can realize the PCIe bridge extension NTB function of PCIe Switch and can also realize the RP bridge extension NTB function of RC.

Specifically, the RP20 port and the UP11 port are interconnected. RP20 and UP11 do link negotiation link establishment without the need for link establishment based on Crosslink characteristics. The NT0 includes a VRP (Virtual root node) and a VEP (Virtual EP). NT1 includes a VDP (Virtual downlink Port) and a VEP.

Wherein, VRP and VEP in NT0 are in the topology of RC1, and VRP and VEP of NT0 are under-hung devices of Bus1 and cannot be accessed by RC 2. VDP and VEP in NT1 are in RC2 topology, and VDP and VEP of NT1 are on-hook devices of UP11, and can be accessed by RC2 but can not be accessed by RC 1. The two RCs are isolated by NT0 and NT1, so that the RC topological structure domains are prevented from overlapping, and in the topological structure of RC1, devices in NT0, RP10, RP11, RP … and RP1N all belong to the topological structure of RC 1. The device of NT1 and the devices of UP11 and RC2 both belong to the topology of RC 2.

Therefore, on the basis of the existing RC function, two RCs are linked through the NT port, as long as the role of the UP/RP port is configured, the characteristic of crossbar is not needed to be used for link establishment, the link establishment characteristic between the UP and the RP is kept, and the link establishment cost is reduced. By expanding the number of the NT functional ports, any number of RCs can be interconnected through the NT ports, and the expansion interconnection of a multi-RC topological structure is realized.

Fig. 6 illustrates an embodiment of a multi-stage NTB interconnect. Taking PCIe Switch as an example, multiple NTs 1 are respectively built in PCIe Switch1, PCIe Switch2, and PCIe Switch 3. On the basis of the functional architecture of fig. 4, the DP3N port in Host3 and the UP21 port in Host2 are interconnected. DP3N and UP21 do link negotiation link establishment without the need for link establishment based on Crosslink characteristics. VDPs and VEPs of NT0 in Host2 are devices hanging down from UP20, can be enumerated by Host2, but cannot be accessed by Host 3. VDPs and VEPs in NT1 of Host2 are in PCIe topology of PCIe Switch3, and VDPs and VEPs of NT1 are devices hanging down in UP21, and can be accessed by Host3 but can not be accessed by Host 2. Similarly, the DP1N port in Host1 and the UP31 port in Host3 are interconnected. DP1N and UP31 perform link negotiation link establishment. VDPs and VEPs of NT0 in Host3 are hanging devices under UP30, which can be enumerated by Host3, but cannot be accessed by Host 1. VDP and VEP in NT1 of Host3 are in PCIe topology of PCIe Switch1, and VDP and VEP of NT1 are under-hung devices of UP31, and can be accessed by Host1 but can not be accessed by Host3, thereby realizing functions of cascaded PCIe NTB of PCIe Switch1, PCIe Switch2 and PCIe Switch 3. Those skilled in the art will appreciate that any number of NTB interconnects may be constructed in the same manner for more than 3 PCIe switches.

For other types of PCIe bus controllers, including RCs, the architecture of the cascaded NTB may also be constructed in accordance with the example given in fig. 6. Referring to fig. 7, the architecture of the cascaded NTB shown in fig. 7 applies to different types of PCIe bus controllers, as compared to the single type of PCIe bus controller of fig. 6. Namely, NT1 are established in PCIe Switch1, PCIe Switch2, and RC1, respectively. The NTB interconnection between PCIe Switch1 and PCIe Switch2 is the same as that described in fig. 6, and is not described here again. The RP1N port in RC1 and the UP21 port in Host2 are interconnected. RP1N and UP21 perform link negotiation link establishment. The VDP and VEP in NT1 of Host2 are in the PCIe topology of RC1, and the VDP and VEP of NT1 are under-hung devices of UP 21. The DP1N port in Host1 and the UP11 port in RC1 are interconnected. DP1N and UP11 perform link negotiation link establishment. VDP and VEP in NT1 of RC1 are in PCIe topology of PCIe Switch1, and VDP and VEP of NT1 are UP11 devices of RC1, which can be accessed by Host1 but cannot be accessed by RC1, thereby realizing the function of cascaded PCIe NTB of PCIe Switch1, PCIe Switch2 and RC 1.

Therefore, by expanding the number of the NT function ports, the PCIe bus controllers with any number and any type can be interconnected through the NT ports, and the expansion interconnection of various PCIe topological structures is realized.

Those skilled in the art will appreciate that the topology of the elements and the number of functional blocks described in the above embodiments are by way of example only. Those skilled in the art can use any connection method to connect ports such as UP, DP/RP, etc. according to the requirement, and can also arbitrarily specify the number of NT functional modules.

Although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims

1. A system implementation management method for PCIe NTB is characterized by comprising the following steps:

providing a first NT module NT0 in the first PCIe bus controller;

providing a second NT module NT1 in a second PCIe bus controller;

the first NT0 and second NT module NT1 include virtual downstream ports VDP and virtual endpoint devices VEP, respectively;

taking the second NT module NT1 as a down-hanging device of a first UP port UP11 of a second PCIe bus controller;

2. The PCIe NTB system implementation management method of claim 1, wherein the first PCIe bus controller and the second PCIe bus controller are a first PCIe Switch and a second PCIe Switch, respectively.

3. The system implementation management method of PCIe NTB according to claim 2,

the apparatus for hanging the second NT module NT1 as the first upstream port UP11 of the second PCIe bus controller further includes:

4. The system implementation management method for PCIe NTB of claim 3, wherein the second NT module NT1 further comprises a second downstream port DP20 for link negotiation link establishment with the first upstream port UP 11.

5. The PCIe NTB system implementation management method of claim 3, wherein the first PCIe Switch further comprises UP10, and the virtual downstream port VDP and virtual endpoint device VEP of the first NT module NT0 are taken as the down-hanging devices of the UP 10.

6. The PCIe NTB system implementation management method according to claim 3, wherein the first NT module NT0 establishes logical connection with the plurality of second NT modules NT1 respectively, and the plurality of second NT modules NT1 are used as down-hanging devices of the upstream ports of the plurality of second PCIe switches respectively, so as to establish NTB interconnection between the first PCIe Switch and the plurality of second PCIe switches.

7. The system implementation management method of PCIe NTB of claim 3, further comprising:

setting a first NT module NT0 in the second PCIe Switch;

setting a second NT module NT1 in a third PCIe Switch;

8. The PCIe NTB system implementation management method of claim 4, wherein the first PCIe Switch and the second PCIe Switch comprise a plurality of upstream ports UP and/or a plurality of downstream DP ports, and the first NT module NT0 and the second NT module NT1 are mapped to any one of the upstream ports UP and downstream DP ports of the first PCIe Switch and the second PCIe Switch, respectively.

9. The system implementation management method of the PCIe NTB of claim 1, wherein the first PCIe bus controller and the second PCIe bus controller are a first Root complex logic RC1 and a second Root complex logic RC2, respectively.

10. The system implementation management method of PCIe NTB of claim 9,

11. The PCIe NTB system implementation management method of claim 9, wherein the first NT module NT0 establishes logical connection with the plurality of second NT modules NT1, respectively, and the plurality of second NT modules NT1 serve as drop-down devices for the upstream ports of the plurality of second Root complex logic RCs 2, respectively, so as to establish NTB interconnection between the first Root complex logic RC1 and the plurality of second Root complex logic RCs 2.