CN112416834A

CN112416834A - Interconnection method between devices compatible with PCIe structure and related devices

Info

Publication number: CN112416834A
Application number: CN202011379794.9A
Authority: CN
Inventors: 不公告发明人
Original assignee: Shanghai Biren Intelligent Technology Co Ltd
Current assignee: Shanghai Bi Ren Technology Co ltd
Priority date: 2020-11-30
Filing date: 2020-11-30
Publication date: 2021-02-26
Anticipated expiration: 2040-11-30
Also published as: CN115374038A; CN112416834B

Abstract

The invention provides an interconnection method and related equipment among equipment compatible with a PCIe (peripheral component interconnect express) structure, wherein the interconnection method is applied to a heterogeneous system, the heterogeneous system comprises a host and a plurality of pieces of equipment connected with the host, and the interconnection method comprises the following steps: the first equipment changes the type of a data path of the first equipment according to the received preset control character sent by the second equipment or the host; the first equipment and the second equipment after the type is changed are interconnected; the preset control character is an instruction which accords with a PCIe communication protocol and can control the first device to change the type of a data path of the first device, and the type of the data path is a root complex RC type or an end point device EP type. The invention can realize the interconnection among the devices by switching the device types, and can reduce the data transmission delay of the interconnection among the devices in the heterogeneous computing system.

Description

Interconnection method between devices compatible with PCIe structure and related devices

Technical Field

The present invention relates to the field of computer technologies, and in particular, to an inter-device interconnection method and related devices compatible with a PCIe (Peripheral Component Interconnect Express, PCIe for short) structure.

Background

With the rapid development of general high-performance computing systems, the field of artificial intelligence and multimedia image processing, the demand of modern society for data centers and cloud computing is increasingly dependent. For a data center, a core engine of cloud computing has a further requirement on a high-performance general computing chip. This dramatic expansion of data and computational volume is now far beyond the capabilities of conventional CPUs.

In order to make up for the deficiency of the computing power of the CPU, a general solution is to adopt a heterogeneous computing system of a co-processing chip to improve the computing performance. A typical coprocessor chip has many options such as GPU (Graphics Processing Unit, abbreviated GPU, Chinese: Graphics processor), ASIC (Application Specific Integrated Circuit, abbreviated ASIC, Chinese: Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array, abbreviated FPGA).

How to further expand and improve heterogeneous computing performance and reduce data transmission delay required by computing-intensive workloads becomes a major problem at present.

Disclosure of Invention

The invention provides a multi-device interconnection method compatible with a PCIe (peripheral component interconnect express) structure and related devices, which are used for solving the problem that data transmission delay is overlarge due to the fact that pairwise interconnection paths are lacked between the devices in a heterogeneous computing system based on the PCIe topological structure, so that interconnection between the devices is realized.

The invention provides an interconnection method between devices compatible with a PCIe structure, which is applied to a heterogeneous system, wherein the heterogeneous system comprises a host and a plurality of devices connected with the host, and the interconnection method comprises the following steps:

the first equipment changes the type of a data path of the first equipment according to the received preset control character sent by the second equipment or the host;

the first equipment and the second equipment after the type is changed are interconnected;

the preset control character is an instruction which accords with a PCIe communication protocol and can control the first device to change the type of a data path of the first device, and the type of the data path is a root complex RC type or an end point device EP type.

According to the method for interconnecting devices compatible with the PCIe structure, the first device changes the type of a data path of the first device according to the received preset control instruction sent by the second device, and the method comprises the following steps:

the first equipment sends the training characters containing the first preset link codes to the second equipment;

the second equipment switches the type of a data path which receives continuous training characters containing first preset link codes firstly;

the second device sends training characters containing second preset link codes to the first device through the data path after type switching so as to realize interconnection of the first device and the second device;

the first preset concatenated coding is used for defining the type of a data path of the first device, the second preset concatenated coding is used for defining the type of a data path of the second device, and the type of the data path is a root complex RC type or an end point device EP type.

According to the method for interconnecting devices compatible with the PCIe architecture of the present invention, after the second device sends the training character including the second preset link code to the first device through the data path after the type conversion, the method includes:

and when the remaining data path of the second equipment receives continuous training characters containing the first preset link code, switching the types of the remaining data path of the second equipment according to the type of the data path.

According to the method for interconnecting devices compatible with the PCIe structure, before the first device sends the training characters containing the first preset link code to the second device, the method comprises the following steps:

and setting a timeout timing mechanism for the first device and the second device to ensure that the PCIe link establishment is carried out again after all data paths of the second device are switched in type.

According to the interconnection method between the devices compatible with the PCIe structure, the working step of the timeout timing mechanism comprises the following steps:

the host computer confirms whether the peer-to-peer interconnection link between the devices is successful or not through a link state register in the detection device;

if the connection state is always a False state within a preset time, the peer-to-peer interconnection link between the devices is considered to be unsuccessful;

resetting the timing value and re-trying to establish a peer-to-peer interconnection link between the devices;

the host machine positions and monitors the position and state information among the devices through a preset PCIe BDF code, and changes or monitors the link state registers among the devices in real time.

According to the interconnection method between the devices compatible with the PCIe structure, provided by the invention, under the condition that the data path of the first device is of an RC type and the data path of the second device is of an RC type structure, the interconnection method comprises the following steps:

the second equipment sends the training characters containing the second preset link codes to the first equipment;

the first equipment switches the type of a data path which receives continuous training characters containing first preset link codes from an RC type to an EP type;

and the first equipment sends the training characters containing the first preset link codes to the second equipment through the data path after the type switching so as to realize the interconnection of the second equipment and the first equipment.

According to the interconnection method between the devices compatible with the PCIe structure, provided by the invention, under the condition that the first device is of an EP type and the second device is of an EP type structure, the interconnection method comprises the following steps:

the second equipment switches the type of a data path which is received continuously at first and contains the training characters of the first preset linking code from an EP type to an RC type;

and the second equipment sends the training characters containing a second preset link code to the first equipment through the data path after the type switching so as to realize the interconnection of the first equipment and the second equipment.

According to the method for interconnecting devices compatible with the PCIe structure, when the first device is of an RC type and the second device is of an EP type structure, or when the first device is of an EP type and the second device is of an RC type structure, the re-establishing the PCIe link includes:

the first equipment sends the training characters containing the first link codes to the second equipment through a sending end of a data path;

after the second equipment receives the training characters, the second equipment identifies the type of the first equipment according to the first link code and configures a second link code of the second equipment;

the second device sends the training characters containing the second link codes to the first device through a sending end of a data path so as to inform the first device of the second link codes;

and after the first equipment receives the training characters and confirms the second link codes, the PCIe link establishment of the first equipment and the second equipment is completed.

According to the method for interconnecting devices compatible with the PCIe structure, the first device changes the type of a data path of the first device according to a received preset control instruction sent by a host, and the method comprises the following steps:

the host computer distributes bus function equipment codes for each equipment to determine the coordinate position of each equipment in the whole mechanism system;

and according to the bus function equipment code, the host machine realizes the switching of the type of the control peer-to-peer interconnection equipment.

The invention also provides an inter-device interconnection system compatible with PCIe structure, comprising a CPU, a root complex RC, a memory, a switch and a plurality of devices, wherein the CPU is respectively connected with the memory, the switch and the devices through the root complex, the devices comprise a first device and a second device, wherein,

the first equipment sends a training character containing a first preset link code to the second equipment; the second equipment switches the type of a data path which receives continuous training characters containing first preset link codes firstly; the second device sends training characters containing second preset link codes to the first device through the data path after type switching so as to realize interconnection of the first device and the second device;

the first preset concatenated coding is used for defining the type of the first device, the second preset concatenated coding is used for defining the type of the second device, and the type of the data path is a root complex RC type or an end point device EP type.

The invention also provides an interconnection chip between devices compatible with the PCIe structure, which is applied to a heterogeneous computing system, wherein the interconnection chip comprises a first chip and a second chip, wherein:

the first chip sends a training character containing a first preset link code to the second chip;

the second chip switches the types of the data paths which firstly receive continuous training characters containing the first preset link codes;

the second chip sends training characters containing a second preset link code to the first chip through the data path after type switching so as to realize interconnection of the first chip and the second chip;

the first preset chaining coding is used for defining the type of a data path of the first chip, the second preset chaining coding is used for defining the type of a data path of the second chip, and the type of the data path is a root complex RC type or an end point device EP type.

The invention also provides an electronic device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor executes the program to realize the steps of the PCIe structure compatible device interconnection method.

The present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the method for interconnecting devices compatible with a PCIe fabric as described in any one of the above.

According to the interconnection method and the related equipment compatible with the PCIe structure, provided by the invention, interconnection among the equipment can be realized by switching the equipment types, so that data reading and writing among the equipment are realized, the data transmission delay of interconnection among the equipment in a heterogeneous computing system can be reduced, the data bandwidth of interconnection among multiple equipment is expanded, and the bottleneck of the heterogeneous computing system is further eliminated.

Drawings

In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.

FIG. 1 is a schematic diagram of a heterogeneous computing system provided by the prior art;

FIG. 2 is a flow chart illustrating a method for interconnecting devices compatible with a PCIe fabric according to the present invention;

FIG. 2a is a second flowchart of the method for interconnecting devices compatible with PCIe structure according to the present invention;

FIG. 3 is a schematic diagram of a PCIe interconnect topology provided in the present invention;

FIG. 4 is a schematic diagram of a PCIe link establishment provided by the present invention;

FIG. 5 is a schematic diagram of the interconnection between devices of the RC/RC structure provided by the present invention;

FIG. 6 is a schematic illustration of the interconnection between devices of the EP/EP architecture provided by the present invention;

FIG. 6a is a schematic diagram of the Timeout timing mechanism provided by the present invention;

FIG. 7a is a schematic structural diagram of an inter-device interconnect system compatible with PCIe fabric according to an embodiment of the present invention;

FIG. 7b is a schematic diagram of an inter-device interconnect system compatible with PCIe fabric according to another embodiment of the present invention;

FIG. 8 is a flow chart of a method for interconnecting devices compatible with a PCIe fabric according to the present invention;

fig. 9 is a schematic structural diagram of an electronic device provided by the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Artificial Intelligence (AI) has three elements: algorithm, computational power, data. The calculation force root is provided by a bottom chip, and can be divided into isomorphic calculation and heterogeneous calculation according to the composition mode of the calculation chip, as follows:

isomorphic calculation: and the computing units using the same type of instruction sets and architectures form a computing mode of the system.

And (3) heterogeneous calculation: and the computing mode of the system is formed by using different types of instruction sets and the computing units of the architecture. Common classes of computing units include CPUs, GPUs, ASICs, FPGAs, and the like.

Such as: when a program is run, a CPU can be regarded as a director, i.e., a Host, and a GPU (Graphics processing Unit) that performs a large amount of computation is a computing Device, i.e., a Device. The CPU and the GPU are connected through a PCIe bus and used for transmitting instructions and data.

FIG. 1 is a schematic diagram of a heterogeneous computing system provided by the prior art, as shown. The heterogeneous computing systems, such as CPUs, are shown interconnected with multiple Device computing systems via a high speed IO bus, such as PCIe.

Data transmission between the CPU and the Device, and between the Device and the Device, all follow the PCIe protocol specification. Host and Device share IO bus bandwidth between Device and Device. In the general heterogeneous computing System 100, a CPU 101 is interconnected with a System Memory (System Memory)103, a switch 104, a switch 105, and a plurality of devices (Device 0106, Device 1107, Device 2108, and Device 3109) through an RC (Root Complex) 102, and the whole heterogeneous System follows a PCIe bus protocol topology.

In the above, the RC is the root of the PCIe bus, and the RC connects the bus, the memory, and the processor to the bus path. EP (Endpoint device, abbreviated EP) is a generic name of PCI/PCIe devices, such as PCIe network card, PCIe video card, and the like. Switch: simply a PCIe switch, for extending the PCIe interface.

Based on the heterogeneous computing system 100 of fig. 2a, each Device (Device 0106, Device 1107, Device 2108, Device 3109) and the CPU 01 have independent data transmission paths, such as the transmission path 110 between the Device 1107 and the CPU 101 and the transmission path 112 between the Device 2108 and the CPU 101. Based on the existing PCIe topology, there is no independent data path between Device and Device, for example, there is no independent data path between Device 1107 and Device 2108, and the data path 111 is highly shared with the CPU transmission paths 110 and 112. And the data exchange path between the Device 1107 and the Device 2108 is extremely long, the data exchange between the Device 1107 and the Device 2108 must pass through the RC 102 and the switch 104.

Therefore, in the PCIe-based heterogeneous system with the topology, data transmission delay between the Device and the Device is large, and the structure sharing the data path with the CPU has a significant influence on the data transmission bandwidth between the Device and the Device, so that the data transmission bandwidth between each Device and the Device becomes a bottleneck of the overall heterogeneous computing system. In addition, the multiple Device interconnection system based on the strict PCIe topology generally needs extra Switch auxiliary connection outside the chip, and in such a topology, not only the system complexity and the hardware cost are increased, but also a high-efficiency pairwise interconnection structure under multiple devices cannot be realized.

Generally, the interconnection of multiple devices in a pairwise interconnection structure based on a PCIe topology structure needs to determine the ID and the location of each Device in a system in advance, and then Host performs initialization configuration on each Device according to the obtained ID and location information, and then the connection between the devices can be established. Once a Device joins or leaves the system, the ID number and location information of each Device needs to be recalculated and assigned by the Host. The method greatly limits the flexibility of interconnection of multiple devices and the difficulty of software implementation.

Therefore, the invention provides a more flexible method for realizing pairwise interconnection between devices based on self-adaptation (auto-negotiation) under the condition of being compatible with the traditional PCIe topological structure, realizes interconnection between the devices under the condition of being compatible with the PCIe topological structure such as an RC/EP interconnection structure, and also supports interconnection between the devices under the condition of an unconventional RC/RC and EP/EP interconnection topological structure.

Fig. 2 is a flowchart illustrating an inter-device interconnection method compatible with the PCIe fabric according to the present invention, as shown in the figure. An interconnection method between devices compatible with a PCIe (peripheral component interconnect express) structure is applied to a heterogeneous system, wherein the heterogeneous system comprises a host and a plurality of devices connected with the host, and the interconnection method comprises the following steps:

step 200, the first device changes the type of its own data path according to the received preset control character sent by the second device or the host.

Step 201, interconnecting the first device and the second device after the type is changed;

Fig. 2a is a second schematic flowchart of the method for interconnecting devices compatible with the PCIe fabric according to the present invention, as shown in the figure. The method for changing the type of the data path of the first device according to the received preset control instruction sent by the second device comprises the following steps:

in step 210, the first device sends a training character containing a first predetermined linking code to the second device.

Optionally, the first preset link code may be set as a PAD link number character.

Step 220, the second device switches the type of the data path that receives the continuous training characters containing the first preset concatenated code first.

Step 230, the second device sends the training characters containing the second preset link codes to the first device through the data path after the type switching, so as to implement interconnection between the first device and the second device.

Optionally, the second preset link code may be set to a non-PAD link number.

The first preset link code is used for defining the type of the first device, and the second preset link code is used for defining the type of the second device; the type of the data path is an RC type or an EP type.

It should be noted that the first device and the second device each have one or more data paths.

Optionally, the first predetermined concatenated coding may be defined as the type of the data path being an EP type, and the second predetermined concatenated coding may be defined as the type of the data path being an RC type.

Therefore, the method for interconnecting devices compatible with the PCIe structure can be applied to a heterogeneous computing system, is suitable for architectures of CPUs and multiple devices, particularly for interconnection among the multiple devices, and can reduce data transmission delay of interconnection among the multiple devices in the heterogeneous computing system, expand data bandwidth of interconnection among the multiple devices and further eliminate bottleneck of the heterogeneous computing system.

Particularly, the Device pairwise interconnection structure described in the invention can avoid introducing multiple switches (switches) to increase the complexity and cost of the system and reduce the cost of the system.

Specifically, under the condition of compatibility with a complete PCIe topological structure and rules, complexity of realizing interconnection among multiple devices can be reduced by introducing a Host programming mode, and difficulty in realizing software and hardware is balanced.

Specifically, the interconnection system between the devices compatible with the PCIe structure does not need to determine the ID and the position information of each Device in advance, each Device is connected by completely depending on hardware self-adaptation, the software implementation difficulty is simplified, and the system flexibility is greatly enhanced.

The following description will be directed to a specific structure of the PCIe fabric compatible inter-device interconnection method according to the present invention.

FIG. 3 is a schematic diagram of a PCIe interconnect topology provided by the present invention, as shown. Under the unconventional RC/RC or EP/EP interconnection structure, devices interconnected pairwise make adaptive Device Type conversion between RC/RC structures and between EP/EP structures by realizing an adaptive (auto-negotiation) mechanism in the link establishment process. As shown in the diagram 300, the unconventional RC/RC, EP/EP interconnect structure is finally realized to be converted into an RC/EP structure, EP/RC structure, so that all rule requirements under the PCIe framework are met.

Specifically, the RC/RC structure of 301 is converted into the RC/EP structure of 302 or the EP/RC structure of 303 through adaptive Device type switching to carry out interconnection among devices. Because the structure of a general RC/RC and the structure of an EP/EP cannot be interconnected according to the rules of the existing PCIe architecture.

FIG. 4 is a schematic diagram of a PCIe link establishment provided by the present invention, as shown. In a link setup procedure of conventional PCIe, two-by-two interconnected devices (such as RC409 and EP 408) define their own type by identifying PAD link number (first preset link code) of the opposite end Training symbol. Lane _ a0 represents one data path on the RC409 side, and Lane _ b0 represents one data path on the EP 408 side.

Generally, under the conventional PCIe topology, the RC409 first sends a training symbol (containing link number) to the EP 408 at the opposite end through the TX 401 of the data path, and informs the EP 408 at the link number of the RC409 at the opposite end, and the EP 408 passively receives the training symbol (containing the link number) through the RX 410 of the data path.

EP 408 configures its own Link number according to the Link number sent by RC409, and feeds back to RC409 through TX 411 of the data path, that is, sends a tracking symbol containing its own configured Link number to RC409, and informs RC409 of the Link number configured by EP 408 itself, and after RC409 passively receives the Link number sent by EP 408 through RX 412 of the data path and confirms this data, the interconnection of RC409 and EP 408 under the PCIe topology is primarily completed.

FIG. 5 is a schematic diagram of the interconnection between devices of the RC/RC structure provided by the present invention, as shown. Under the interconnection structure of the RC 509 and the RC 508, the RC 508 switches the type of the data path lane _ b 1501 which receives the consecutive training symbol 502 containing the second predetermined link number (non-PAD link number) to the data path type of the EP attribute.

After all the data paths (such as lane _ b 1701, lane _ b 0703, lane _ b 2704 and lane _ b 3705) with the RC attribute on the side where the RC 508 is located are all converted into the data path type with the EP attribute, that is, the Device type (Device type) is converted from the RC type into the EP type, the PCIe Link establishment is performed again after the preset Timeout timing time of the Training state machine, and the PCIe Link establishment is as shown in fig. 4.

Therefore, after an Auto connectivity mechanism, two devices (RC 509/RC 508) are converted into an RC/EP interconnection topological structure from an RC/RC interconnection structure, namely, the RC type of the RC 508 is switched into the EP type.

It should be noted that, in order to identify the RC and the EP, the embodiment of the present invention sends a non-PAD link number link code to the RC type definition, and sends a PAD link number link code to the EP type definition, but the present invention is not limited thereto, and any definition mode beneficial to identifying the RC and the EP may be used.

Fig. 6 is a schematic diagram of the interconnection between devices of the EP/EP architecture provided by the present invention, as shown. Under the interconnected architecture of EP 609 and EP 608, EP 608 sends a training symbol containing the PAD link number to EP 609 via data path 602.

EP 609 switches the type of Lane _ a 2604 of the data path receiving the first consecutive training symbol containing the PAD link number to RC type, and sets the link number of all data paths (e.g. Lane _ a 0603, Lane _ a 1601, Lane _ a 3605) on the same side as this data path to default value, which can be controlled by a register and can be allocated for power-up initialization. Then, EP 609 sends a tracking symbol 606 containing a non-PAD link number to the data path Lane _ b 2607 of EP 608 through the data path 606 which has been switched to the RC type.

Meanwhile, the EP 609 does not receive other data paths (such as Lane _ a 0603, Lane _ a 1601 and Lane _ a 3605) of the continuous PAD link number sent by the EP 608, and the EP 608 keeps sending the PAD link number to other data paths of the EP 609, and the other data paths keep the data path type of the EP attribute until the other data paths also receive the continuous PAD link number sent by the EP 608 and switch the data path type from the EP type to the RC type.

It should be noted that, the switching of the data path type is based on a preset Timeout timing mechanism, and the PCIe link establishment is resumed while keeping the switched Device type (Device type).

The Timeout timing mechanism described above is further described below as follows:

the introduction of the Timeout timing mechanism may further prevent dead lock between links. HOST monitors a link up status register in the Device to confirm whether peer-to-peer interconnection links between the devices are successful, and if the link up status in a certain peer-to-peer interconnection link is always False within a certain time, the HOST considers that auto connectivity is sunk into the dead load in the existing state. It is necessary to set a new time out value and retry auto connectivity to establish peer-to-peer interconnection links. The PCIe BDF number (namely Bus number Bus coding, Device number Device coding and Function number Function coding) is introduced, so that the Host can conveniently and rapidly position and monitor the position and state information of each Device in the system. And changing or detecting peer-to-peer interconnection state control registers among the devices in the heterogeneous system in real time.

Host first generates random unequal values according to some algorithm, the values of which are unequal and meet the auto negotiation rule. Then, the Host can perform coordinate positioning on multiple devices at the system application level through the enumerated BDF number (Bus number, Device number, Function number), so as to distinguish the devices 0, 1, 2 and 3 in detail. And sequentially sets a pre-generated random time out value for devices of different BDF numbers. As shown in fig. 6a, the Host read/write control command reaches the Device internal/external bus control unit 612a through the external hub control bus 618a, and then forwards the received random time out value to the Tr timer in each peer interconnection unit EP 616a,

EP

614b, 615c through the Device internal data control bus. The Tr timer performs a timing operation by combining the Time out value set by the host and a Free running clock generated by a PLL (phase locked loop) of an SMU (System management unit) in the Device, thereby completing the whole Time out mechanism.

Therefore, the invention can realize peer-to-peer interconnection between devices by introducing an Auto negotiation mechanism, and is compatible with PCIe link establishment rules and topological structures. Furthermore, the timeout timing mechanism can both prevent deadlock between RC/RC and EP/EP in Auto connectivity and indirectly control type switching of EP/RC.

Another embodiment of the present invention further provides an inter-device interconnection method compatible with the PCIe fabric, that is, a method for directly controlling establishment of a peer-to-peer interconnection link in a top-down structure through a Host software layer.

Generally, under the PCIe legacy topology, as shown in fig. 7b, the Host RC 302 is connected to each Device 0/1/2/3 via Switch 304 or directly via 318a/318b/318c/318d external hub control Bus, and after establishing the transmission link, the Host RC enumerates each Device, thereby allocating BFD (Bus, Function, Device) Number, i.e. preset Bus Function Device code. Thereby determining the coordinate position of each Device throughout the heterogeneous system. The system software layer can identify each Device existing in the system through the BFD number, and integrally control the attribute of any peer-to-peer interconnection control unit in the heterogeneous system.

For example, Device0/Device1/Device2/Device3 are multiple devices with identical equivalent functions. The default attribute (or "type") of the initial peer-to-peer interconnection control unit is EP, the software layer can control the peer-to-peer interconnection unit 313a/314a/315a in the Device 0306 a to keep the EP attribute unchanged by reading the BFD number allocated to each Device, and for the control unit 313b/314d/315c connected to the peer-to-peer interconnection unit 313a/314a/315a, the Host can change the attribute in a manner of directly controlling the internal register control of each Device according to some rule or algorithm, so that the control unit is switched to RC. Thereby establishing RC/EP, EP/RC peer-to-peer interconnect bus data links conforming to the PCIe topology.

Fig. 7a is a schematic structural diagram of an inter-device interconnection system compatible with a PCIe fabric according to an embodiment of the present invention, as shown in the figure. The heterogeneous computing system 700a is composed of a CPU201 and multiple devices 0206 to 3209, the Host CPU201 is connected to a system memory 203 through an RC 202, and the multiple compute-intensive devices 0206, 2207, 1208, 3209 interconnect Host and devivce.

In the data path of the Host CPU201, each Device (Device) follows the PCIe protocol, exists in the PCIe topology as each endpoint (ep)210, and receives various data types between the transmission and the Host CPU201, such as Memory, configuration, Message, IO, and the like.

For the data transmission path between Device and Device, the present invention completes the multi-Device interconnect structure by introducing independent data paths between Device and Device, such as by adding Device0 and Device2 data path 211, Device0 and Device1 data path 214, Device0 and Device3 data path 216.

Based on the interconnection structure, the difference between any two connected devices and Host CPU data paths is that an independent data transmission path is provided, and Host CPU data and other Device data are separated. In the process of computation intensive operation, such as certain deep learning training, in order to perfect branch prediction or model convergence, the devices 0206 to 1208 need a large amount of data to be mutually transmitted and shared, the invention can realize pairwise interconnection between the devices 0206 to 1208, and a data transmission read-write mechanism with higher efficiency and low delay can be provided through separated data paths (such as the data paths 211 to 216), so that the instant data transmission characteristic is ensured, and the processing speed of the high-performance computation coprocessor is further enhanced.

Fig. 7b is a schematic structural diagram of an inter-device interconnect system compatible with PCIe fabric according to another embodiment of the present invention, as shown in the figure. In a complete heterogeneous computing system 700b with multi-core CPUs (e.g., CPU core 0301 a, CPU core 1301 b, CPU core 2301 c, CPU core 3301 d) and multi-devices (e.g., Device 0306 a, Device 1306 c, Device 2306 b, Device 3306 d), the multiple CPU cores 301a, 1301 b, 2301 c, 3301 d in the Host CPU access the system Memory 305 through the Host Memory Controller (HMC) 303 in RC 302, respectively.

The multiple compute intensive devices 0306 a, 1306 c, 2306 b and 3306 d can realize data exchange management between CPU and Device directly through the external hub control bus (318 a-318 c) or indirectly through the Switch (304a, 304 b).

The external bus control unit in each Device (Device 0306 a, Device 1306 c, Device 2306 b, Device 3306 d) is of EP type, i.e. EP 310a, EP 310b, EP 310c, EP 310d, controlled by RC 302 in the Host CPU, and the Host RC 302 mainly completes a series of control tasks such as PCIe link establishment, EP configuration enumeration, interrupt vector distribution reception, and power consumption management.

After the peripheral hub control bus 318(318a, 318b, 318c, 318d) is initialized, data between Host and Device is exchanged via this bus. Such as the Host CPU 301(CPU core 301a, CPU core 1301 b, CPU core 2301 c, CPU core 3301 d) can read and write data in the Memory cells 312(312a, 312b, 312c, 312d) of the Device 306(Device 0306 a, Device 1306 c, Device 2306 b, Device 3306 d), or a register of a client in the Device such as the Memory Controller 311 a. The Device can read and write data in the System Memory 305 via the peripheral hub control bus 318(318a, 318b, 318c, 318 d).

The peripheral hub control bus 318(318a, 318b, 318c, 318d) line is a dedicated bus between Host and Device (Device 0306 a, Device 1306 c, Device 2306 b, Device 3306 d), through which all data between Host and Device is transferred. The dedicated peripheral hub control assembly 318 avoids the disadvantage of Device-to-Device and Device-to-Host data path aliasing and bandwidth mutual occupation, thereby degrading the performance of heterogeneous computing systems.

In Device 306 (e.g., Device 0306 a, Device 1306 c, Device 2306 b, Device 3306 d), the outer hub control unit EP 310 (e.g., EP 310a, EP 310b, EP 310c, EP 310d) is controlled by the RC 302 in Host. The EP 310 (e.g., EP 310a, EP 310b, EP 310c, EP 310d) receives and translates the Host CPU data register read/write control instructions to the Device internal high speed on-chip bus 316 (e.g., 316a, 316b, 316c, 316 d). The layered high-speed on-chip bus routes the translated Host CPU read/write command to a Device on-chip memory controller 311 (e.g., 311a, 311b, 311c, 311 d). The on-chip memory unit 312 is read and written via the memory controller 311. And feeds back to the Host end.

The peer-to-peer interconnection control unit EP (313/314/315) in the Device 306 (e.g., Device 0306 a, Device 1306 c, Device 2306 b, Device 3306 d) performs PCIe Link establishment and data read/write with the peer control unit through auto connectivity mechanism. The peer-to-peer interconnection bus realizes physical data path between two connected devices, and data exchange can be carried out between any connected devices through the bus. For example, Device 306a data read/write control commands may reach the peer Device 306d via the peer-to-peer interconnect bus data path 317 e. EP 314d receives the translated data read/write control instruction from Device 306a, which is routed to memory control unit 311d via its internal high-speed on-chip bus 319 d. The memory control unit 311d reads and writes the on-chip memory unit 312d according to the translated read/write control command, and returns data to the Device 306 a.

Therefore, based on the method for interconnecting devices compatible with the PCIe structure, the Device (such as Device 0306 a, Device 1306 c, Device 2306 b and Device 3306 d) can be interconnected pairwise, and based on the interconnection, PCIe link connection is established, so that data can be read and written between the devices, data transmission delay of interconnection of multiple devices in a heterogeneous computing system can be reduced, data bandwidth of interconnection of multiple devices is expanded, and bottleneck of the heterogeneous computing system is further eliminated.

Fig. 8 is a flowchart of an inter-Device interconnection method compatible with PCIe fabric according to still another embodiment of the present invention, where as shown in the flowchart, the method for establishing an adaptive peer-to-peer connection between multiple devices (devices) according to the present invention includes the following steps:

step 801, initializing a Host path link between a Host CPU and each Device, including initializing a physical link.

Step 802, establishing a connection between the Host and each Device in a higher data transmission layer link and completing an enumeration process, thereby completing the whole Host path link, and then performing data interaction between the Host and each Device.

In step 803, the CPU configures registers of peer-to-peer interconnection control unit controllers in the devices via the Host path.

In step 804, the devices initially attempt to connect to the physical layer link.

Step 805, auto connectivity (adaptive) interconnection is performed between devices during link establishment.

Step 806, according to the auto connectivity result, the peer-to-peer interconnection control unit in each device automatically switches the device type.

In step 807, each Device completes the connection established by the PCIe link again according to the switched Device type.

Step 808, completing the connection of the adaptive peer-to-peer interconnection link, and transmitting and receiving data between the devices through the link.

The following describes the PCIe fabric compatible multi-device interconnect system according to the present invention, and the PCIe fabric compatible multi-device interconnect system described below and the PCIe fabric compatible multi-device interconnect method described above may be referred to correspondingly.

The invention provides an inter-device interconnection system compatible with a PCIe structure, which comprises a CPU, a root complex RC, a memory, a switch and a plurality of devices, wherein the CPU is respectively connected with the memory, the switch and the devices through the root complex, the devices comprise a first device and a second device,

the first chip sends the training characters containing the PAD link number to the second chip;

the second chip switches the types of the data paths which firstly receive continuous training characters containing PAD link numbers;

the second chip sends the training characters containing non-PAD link number to the first chip through the data path after type switching so as to realize interconnection among chips;

the PAD link number is used for defining the type of the first chip, and the non-PAD link number is used for defining the type of the second chip; the first chip and the second chip both comprise a plurality of data paths, and the type of the data paths is RC type or EP type.

Optionally, the first chip and the second chip may be GPU chips.

Fig. 9 illustrates a physical structure diagram of an electronic device, and as shown in fig. 9, the electronic device may include: a processor (processor)810, a communication Interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication Interface 820 and the memory 830 communicate with each other via the communication bus 840. The processor 810 may invoke logic instructions in the memory 830 to perform the steps of the PCIe fabric compliant multi-device interconnect method.

In addition, the logic instructions in the memory 830 may be implemented in software functional units and stored in a computer readable storage medium when the logic instructions are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

In another aspect, the present invention also provides a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the steps of the PCIe fabric compliant multi-device interconnect method provided by the above methods.

In yet another aspect, the present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program that, when executed by a processor, is implemented to perform the steps of the PCIe fabric compatible multi-device interconnect method provided above.

The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.

Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.

Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims

1. An interconnection method between devices compatible with a PCIe (peripheral component interconnect express) structure, which is applied to a heterogeneous system, wherein the heterogeneous system comprises a host and a plurality of devices connected with the host, and the interconnection method comprises the following steps:

2. The method of claim 1, wherein the first device changes the type of its own data path according to receiving a preset control instruction sent by the second device, and the method comprises:

3. The method of claim 1, wherein after the second device sends the training characters containing the second predetermined concatenated codes to the first device through the type-converted data path, the method comprises:

4. The method of claim 1, wherein before the first device sends the training characters containing the first predetermined concatenated code to the second device, the method comprises:

5. The method of claim 4, wherein the timeout timing mechanism operates by the steps of:

the host machine positions and monitors the position and state information among the devices through presetting bus function device codes, and changes or monitors the link state registers among the devices in real time.

6. The method of claim 1, wherein in the case that the data path of the first device is of an RC type and the data path of the second device is of an RC type structure, the interconnecting method comprises:

7. The method according to claim 1, wherein in the case where the first device is of an EP type and the second device is of an EP type configuration, the interconnecting method comprises:

8. The method of claim 3, wherein the resuming PCIe link establishment under the RC-type and EP-type structures of the first device or under the RC-type and EP-type structures of the second device comprises:

9. The method according to claim 1, wherein the first device changes the type of its own data path according to receiving a preset control command sent by the host, and the method comprises:

and controlling the switching of the types of the peer-to-peer interconnection equipment by the host according to the preset bus function equipment code.

10. An inter-device interconnection system compatible with PCIe structure, comprising a CPU, a root complex RC, a memory, a switch and a plurality of devices, wherein the CPU is connected with the memory, the switch and the plurality of devices through the root complex, respectively, the devices comprise a first device and a second device,

11. The utility model provides an interconnection chip between equipment of compatible PCIe structure, is applied to heterogeneous computing system, its characterized in that, the interconnection chip includes first chip and second chip, wherein:

12. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor when executing the program implements the steps of the PCIe fabric compatible inter-device interconnect method of any of claims 1 to 9.

13. A non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the PCIe fabric compatible inter-device interconnect method of any one of claims 1 to 9.