CN109240832B

CN109240832B - Hardware reconfiguration system and method

Info

Publication number: CN109240832B
Application number: CN201811113981.5A
Authority: CN
Inventors: 袁柳; 魏星; 涂吉; 谢海永
Original assignee: China Academy of Electronic and Information Technology of CETC
Current assignee: China Academy of Electronic and Information Technology of CETC
Priority date: 2018-09-25
Filing date: 2018-09-25
Publication date: 2022-04-19
Anticipated expiration: 2038-09-25
Also published as: CN109240832A

Abstract

The invention discloses a hardware reconstruction system and a hardware reconstruction method, which can decouple computational and storage resources such as GPU, FPGA, GPU, IPU, SSD and the like from a server through PCIe multi-bus and cross switch technology, and realize hierarchical interconnection such as interconnection in a pool, interconnection among pools and the like of heterogeneous computational and storage resource pools. Connection reconfiguration capabilities for standard PCIe computing resources are provided through a pool-based and remote PCIe crossbar configuration. Through the introduction of a PCIe exchange mechanism, the expandability and performance advantages of PCIe communication are fully utilized, a high-performance-ratio high-availability system can be created, most heterogeneous devices can be supported, the universality is higher, and the data transmission delay, the system complexity and the power consumption can be reduced.

Description

Hardware reconfiguration system and method

Technical Field

The invention relates to the technical field of computers, in particular to a hardware reconfiguration system and a hardware reconfiguration method.

Background

In the use process of the existing massive node server, resource waste and energy waste caused by unbalanced utilization rate of each component resource (CPU, memory, magnetic disk, SSD, network IO, GPU, FPGA and the like) of the server exist, and high cost caused by frequent upgrade of server hardware caused by different upgrade and update cycles of each component resource exists. Most of hardware resources of the domestic data center server are connected through Ethernet, so that the problems of protocol bloat and high power consumption exist, and the connection of InfiniBand and the like needs special equipment, so that the equipment dependence is strong. Enterprises of hundred degrees, Tencent, Huashi and the like begin to conduct architecture research in the direction of decoupling and pooling of component resources such as server computing, storage, transmission and the like, and the architecture research comprises pooling storage service, heterogeneous computing resource (FPGA, GPU, CPU, IPU) pooling service and the like. However, the PCIe-based hardware resource pooling is currently limited to the complete machine hierarchy, is still being explored, and does not form a standard. And only PCIe expansion capability is utilized, and the advantages of the PCIe expansion capability in communication and reconfiguration scheduling are ignored.

Disclosure of Invention

The invention provides a hardware reconstruction system and a hardware reconstruction method, which aim to solve the problem of unbalanced utilization rate of each part in hardware reconstruction in the prior art.

In one aspect, the present invention provides a hardware reconfiguration system including:

the multi-path interface connection based on a remote PCIe crossbar switch is arranged between each resource pool and the adjacent resource pool, the multi-path interface connection based on the PCIe crossbar switch is arranged in the resource pools, the remote PCIe crossbar switch is connected with the PCIe crossbar switch in the resource pools through a shared interface, and a double-layer crossbar switch configuration link based on the PCIe crossbar switch and the remote PCIe crossbar switch is formed;

the resource pool comprises a computing resource pool and/or a storage resource pool, processors in the resource pool are interconnected through PCIe cross switches in the resource pool, when connection configuration is carried out, the PCIe cross switch connection relation in the resource pool is configured through configuration information, computing and storage resources in the resource pool are distributed for a designated PCIe bus of a CPU, direct connection between the processors in the computing resource pool and the storage resource pool is realized, and dynamic allocation of the computing resources and the storage resources as required is supported;

the remote PCIe crossbar switch is interconnected with each resource pool through a sharing interface, peer-to-peer exchange between the resource pools based on the PCIe protocol is realized, when connection configuration is carried out, the connection relationship of the remote PCIe crossbar switch is configured through configuration information, processor resources of adjacent resource pools are distributed for a specified PCIe bus of a CPU, and calculation and storage resource dynamic distribution among the resource pools are carried out.

Preferably, the shared interface supports two interface modes of a transparent bridge and a non-transparent bridge, and can be configured as a master port or a slave port, and the interconnection between the ports is realized through a PCIe shared interface cable.

Preferably, the PCIe crossbar in the resource pool is implemented as a PCIe switch backplane, and the PCIe switch backplane includes multiple PCIe bus slots and a PCIe switch chip.

Preferably, the remote PCIe crossbar is implemented by a PCIe switch, and implements a point-to-point switching manner between resource pools, including one or more of the following: packet switching, circuit switching, and virtual channel switching.

Preferably, the virtual channel switch is configured to form a plurality of virtual circuit switch connections that can share the same physical channel through the virtual channel, and the virtual circuit switch connections may cooperate with the packet switch connections and the circuit switch connections to transmit the streaming data.

Preferably, a compiler is also included; the compiler is used for distributing paths for each known flow communication according to the known flow communication condition between cores, determining virtual circuit switching connection, circuit switching connection and packet switching connection, establishing each switching connection through prestored connection information according to the communication compiling result during operation, and transmitting the communication between different resource pools on the corresponding switching connection.

Preferably, the remote PCIe crossbar comprises a plurality of input units, a crossbar crossing unit, a path computation unit, a virtual channel assignment unit, a crossbar assignment unit, and a circuit configuration unit;

the input unit is used for configuring and outputting arbitrated virtual channel numbers for connected PCIe devices, each input unit comprises n output virtual channels VC1-VCn, a bypass channel, a PS state storage and a VCS state storage, the input unit is connected with input virtual channel exchange VCS signals and port input signals of the PCIe devices, the PS state storage corresponds to the virtual channel state of the basic packet switch router, the VCS state storage corresponds to the virtual circuit exchange connection state, when the input is the VCS signals, the output virtual channel numbers corresponding to the input virtual channels are found directly through information in the VCS state storage, the VCS signals are directly output to the PCIe devices corresponding to the output virtual channel numbers, and each input unit is additionally provided with a bypass channel to allow packet switch data sheets to be directly input to the cross-over switch unit;

the path calculation unit is used for establishing an ID-port routing path or an address-port routing path and providing a path arbitration result for the data sheet transmitted in the packet switching connection;

the virtual channel allocation unit is used for allocating and outputting a virtual channel number for the input data sheet according to the path arbitration result of the path calculation unit or according to the PS state storage information;

the circuit configuration unit is used for providing pre-stored cross switch configuration information for the data sheet transmitted in the circuit switching connection;

the cross switch distribution unit is used for storing preset circuit configuration storage information through circuit configuration and directly configuring the cross switch so that data pieces connected in circuit switching can directly enter the cross switch spanning unit through the bypass switch;

and the cross switch crossing unit is used for outputting the data packet to PCIe equipment corresponding to the output virtual channel number according to the arbitration scheduling result of the input unit.

Preferably, the pool of computing resources comprises one or more of: CPU, GPU, FPGA, IPU.

Preferably, the storage resource pool comprises one or more of: SSD, HDD.

In another aspect, the present invention provides a hardware reconfiguration method, which applies any one of the above hardware reconfiguration systems, including:

step 1: scanning a PCIe bus through a management node, and acquiring PCIe equipment port information connected to a PCIe multi-bus and a PCIe crossbar in a pool, wherein the management node is a preset CPU node;

step 2: the management node acquires PCIe equipment port information of a remote PCIe crossbar connected with a PCIe crossbar sharing interface in the pool;

and step 3: analyzing the types and the quantity of the CPU, the GPU, the FPGA, the IPU, the SSD, the HDD and other hardware equipment connected with the PCIe bus according to the user requirements;

and 4, step 4: judging whether the resources in the local computing and storing resource pool are enough or not according to the types and the number of the required hardware equipment in the step 3, if yes, turning to a step 5, and if not, turning to a step 7;

and 5: the management node configures the configuration space of the PCIe crossbar switch in the resource pool, configures and distributes required processors for a specified PCIe bus according to a depth-first search algorithm, configures the PCIe crossbar switch in the pool, and configures information such as a master-slave bus, a command register and the like of each processor port connected with the PCIe crossbar switch in the pool;

step 6: the PCIe data packet required by the invention is constructed through a packet format of a PCIeDMA or a standard PCIe link layer protocol, the equipment connected with a designated PCIe bus is accessed, and the PCIe hardware reconfiguration configuration is finished;

and 7: the management node configures the configuration space of the PCIe crossbar switch in the resource pool and allocates available computing and storage resources for the appointed PCIe bus;

and 8: the management node configures a remote PCIe crossbar switch, configures a virtual switch channel, a packet switch channel and a circuit switch channel according to the remote PCIe crossbar switch, supports two connection interfaces of a transparent bridge and a non-transparent bridge, and connects the required adjacent computing and storage resource pools;

and step 9: the management node allocates processors required by the local resource pool for the appointed PCIe bus according to a depth-first search algorithm, selects processors required by the adjacent resource pool, configures the in-pool/remote PCIe crossbar, and information such as master-slave buses, command registers and the like connected to each processor port of the PCIe crossbar, and goes to step 6.

The invention has the following beneficial effects:

the invention can decouple the calculation and storage resources such as GPU, FPGA, GPU, IPU, SSD and the like from the server by PCIe multi-bus and cross switch technology, and realize hierarchical interconnection such as interconnection in the pool, interconnection among the pools and the like of heterogeneous calculation and storage resource pools. Connection reconfiguration capabilities for standard PCIe computing resources are provided through PCIe crossbar-based configuration. Through the introduction of a PCIe exchange mechanism, a high-performance-price-ratio high-availability system can be created, the expandability and performance advantages of PCIe communication are fully utilized, most of heterogeneous devices can be supported, and the universality is stronger; the PCIe configuration function is provided, calculation, storage and network can be dynamically switched according to the performance requirement of the data center server, a proper number of calculation nodes can be distributed for application, and calculation and storage resources can be uniformly managed and flexibly distributed to support connection reconstruction; data transmission delays, system complexity and power consumption can be reduced, with power savings of up to 50% in data-intensive environments.

The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.

Drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:

FIG. 1 is a diagram illustrating a hardware reconfiguration system according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a specific architecture of a hardware reconfiguration system based on an in-pool/remote crossbar according to an embodiment of the present invention;

FIG. 3 is a schematic representation of a PCIe crossbar-based reconfiguration within a resource pool in accordance with an embodiment of the present invention;

FIG. 4 is a schematic representation of remote PCIe crossbar-based reconfiguration between resource pools in accordance with an embodiment of the present invention;

FIG. 5 is a schematic diagram of a remote PCIe crossbar architecture of an embodiment of the present invention;

FIG. 6 is a schematic diagram of a PCIe transaction packet header format according to an embodiment of the invention;

FIG. 7 is a flow chart illustrating a PCIe hardware reconfiguration configuration according to an embodiment of the invention.

Detailed Description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

A first embodiment of the present invention provides a hardware reconfiguration system, referring to fig. 1, including:

the resource pool comprises a computing resource pool and/or a storage resource pool, processors in the resource pool are interconnected through PCIe cross switches in the resource pool, when connection configuration is carried out, the PCIe cross switch connection relation in the resource pool is configured through configuration information, computing and storage resources in the resource pool are distributed for a specified PCIe bus of a CPU, direct connection between the processors in the computing resource pool and the storage resource pool is realized, and dynamic allocation of the computing resources and the storage resources as required is supported;

The invention can decouple the calculation and storage resources such as GPU, FPGA, GPU, IPU, SSD and the like from the server by PCIe multi-bus and cross switch technology, and realize hierarchical interconnection such as interconnection in the pool, interconnection among the pools and the like of heterogeneous calculation and storage resource pools. Connection reconfiguration capabilities for standard PCIe computing resources are provided through PCIe crossbar-based configuration. Through the introduction of a PCIe exchange mechanism, the expandability and performance advantages of PCIe communication are fully utilized, a high-availability system with high cost performance can be created, most of heterogeneous devices can be supported, and the universality is stronger; the PCIe configuration function is provided, calculation, storage and network can be dynamically switched according to the performance requirement of the data center server, a proper number of calculation nodes can be distributed for application, and calculation and storage resources can be uniformly managed and flexibly distributed to support connection reconstruction; data transmission delays, system complexity and power consumption can be reduced, with power savings of up to 50% in data-intensive environments.

It should be noted that the shared interface according to the embodiment of the present invention supports two interface modes, namely, a transparent bridge and a non-transparent bridge, and can be configured as a master port or a slave port, and the interconnection between the ports is realized through a PCIe shared interface cable.

In the embodiment of the invention, the PCIe crossbar switch in the resource pool is realized by a PCIe exchange bottom plate, and the PCIe exchange bottom plate comprises a plurality of PCIe bus slots and a PCIe exchange chip.

In the embodiment of the present invention, the remote PCIe crossbar is implemented by a PCIe switch, and a point-to-point switching manner between resource pools is implemented, where the method includes one or more of the following: packet switching, circuit switching, and virtual channel switching.

The virtual channel switching in the embodiments of the present invention is used to form a plurality of virtual circuit switched connections that can share the same physical channel through the virtual channels, and the virtual circuit switched connections may be matched with the packet switched connections and the circuit switched connections to transmit stream data.

The system of the embodiment of the invention also comprises a compiler; the compiler is used for distributing paths for each known flow communication according to the known flow communication condition between cores, determining virtual circuit switching connection, circuit switching connection and packet switching connection, establishing each switching connection through prestored connection information according to the communication compiling result during operation, and transmitting the communication between different resource pools on the corresponding switching connection.

In the embodiment of the invention, the remote PCIe crossbar switch comprises a plurality of input units, a crossbar crossing unit, a path calculation unit, a virtual channel distribution unit, a crossbar distribution unit and a circuit configuration unit;

It should be noted that, in the embodiment of the present invention, the computing resource pool includes one or more of the following: CPU, GPU, FPGA, IPU, etc. The storage resource pool comprises one or more of: SSD, HDD, etc. Those skilled in the art can configure the device according to actual needs, and the invention is not limited to this.

The system according to an embodiment of the invention will be explained and illustrated in detail below with reference to fig. 1-6:

as shown in fig. 1, the invention firstly constructs a series of resource pools, wherein each resource pool is connected with an adjacent resource pool through a multi-path interface based on a remote PCIe crossbar switch, the resource pools are connected through multi-path interfaces based on a PCIe crossbar switch, the remote PCIe crossbar switch is connected with the PCIe crossbar switch in the resource pools through a shared interface to form a double-layer crossbar switch configuration link based on the crossbar switch and the remote PCIe crossbar switch, and the resource pools support a plurality of connection relationships such as a star topology, a grid topology, a tree topology, a full connection topology, and the like;

the resource pool comprises a computing resource pool and/or a storage resource pool, and the processors in the resource pool are interconnected through a cross switch in the resource pool. The computing resource pool comprises heterogeneous computing resources such as a CPU, a GPU, an FPGA, an IPU and the like, and the storage resource pool comprises storage resources such as an SSD, an HDD and the like. Each computing storage resource pool comprises at least one resource pool such as a CPU pool, a GPU pool, an FPGA pool, an IPU pool, a storage pool and the like. Wherein: the CPU pool comprises a plurality of CPU processors, the GPU pool comprises a plurality of GPU processors, the FPGA pool comprises a plurality of FPGA processors, the IPU pool comprises a plurality of IPU processors such as the cambrian era, the storage pool comprises a plurality of hard disk storage resources such as SSD and HDD, and each computing and storage resource is externally connected with the cross switch through a PCIe bus. When connection configuration is carried out, the cross switch connection relation in the resource pool is configured through the configuration information, computing and storage resources in the resource pool are distributed for the appointed PCIe bus of the CPU, direct connection between processors in the computing resource pool and the storage resource pool is achieved, and dynamic demand distribution of the computing resources and the storage resources is supported.

The cross switch in the resource pool comprises a plurality of sharing interfaces, the sharing interfaces support two interface modes of a transparent bridge and a non-transparent bridge, the sharing interfaces can be configured to be master ports or slave ports, the interconnection among the ports is realized through a PCIe sharing interface cable, and PCIe equipment with any standard PCIe3.0 interface can be connected. The shared interface cross switch is directly connected with each processor in the CPU pool, the GPU pool, the FPGA pool, the IPU pool and the storage pool through a plurality of PCIe buses, the direct connection relation between the designated input interface and the output interface can be controlled through configuration information, the calculation and storage resources allocated by the PCIe buses through the cross switch can be configured by inputting the reconstruction information into the cross switch, and the calculation and storage resources such as the CPU, the FPGA, the GPU, the IPU, the storage and the like cascaded on the buses are allocated to the plurality of PCIe buses.

The remote PCIe crossbar enables peer-to-peer exchanges between PCIe protocol based resource pools. The remote cross switch is connected with the cross switch in the resource pool through the sharing interface, and communication interconnection among the resource pools is achieved. When connection configuration is carried out, the connection relationship of the remote cross switch is configured through configuration information, processor resources of adjacent resource pools are distributed for a specified PCIe bus of the CPU, and dynamic distribution of calculation and storage resources among the resource pools is supported.

The configuration information is input and reconstructed to the cross switch, and the cross switch changes the connection relation of the virtual PCI bridge in the cross switch according to the user requirement and the current system occupation condition through the configuration information, and allocates the computing and storage resources connected to the appointed PCIe bus. For example, the system 1 on the PCIe bus is designated as 4 GPUs, 4 CPUs, 4 FPGAs, and 8 SSDs at one time; at the next moment, the cross switch is configured through the reconfiguration information, and a computing system connected to the PCIe bus becomes a system 2 which comprises 2 CPUs, 1 GPU, 1 FPGA and 2 SSDs; at the next moment, the cross switch is configured through the reconfiguration information, and a computing system connected to the PCIe bus becomes a system 3 which comprises 1 CPU, 1 GPU, 1 FPGA and 2 SSD. Therefore, the connection relation between the computing and storage resources is changed, so that the hardware system supports the reconstruction of the computing capacity and the connection capacity, and the dynamic demand allocation of the computing and storage resources is formed.

As shown in fig. 2, an embodiment of the invention includes 6 compute and storage resource pools, 1 remote PCIe crossbar and 1 master node. Wherein: each resource pool comprises 1 PCIe crossbar switch back plate, each PCIe crossbar back plate comprises a plurality of PCIe bus slots and 1 PEX8796 exchange chip, and computing and storage resources such as a CPU, a GPU, an FPGA, an IPU, an SSD and the like are inserted into the slots of the PCIe crossbar switch back plate to realize interconnection reconstruction in the resource pool. The resource pools are connected with a remote cross switch through PCIe shared interface connecting lines, and the shared interface connecting lines in the case are realized by selecting MiniSAS interface cables of PCIex 4. The remote crossbar switch is implemented in the form of a PCIe switch, the PCIe switch in this embodiment includes 12 ports, and can support peer-to-peer switching of 12 computing and storage resource pools, and each of the 12 ports can be configured in a transparent bridge mode and a non-transparent bridge mode, where the transparent bridge mode is used to connect a computing/storage resource pool without a CPU, and the non-transparent bridge mode is used to connect a computing/storage resource pool containing a host. The master control node is connected with a plurality of computing and storage resource pools through a remote PCIe crossbar switch and is used for reconfiguration and data transmission control. The master control node scans the connected hardware equipment through the remote PCIe crossbar and the PCIe crossbar in the resource pool, and allocates different computing and storage resources for the CPU through the PCIe crossbar and the remote PCIe crossbar in the resource pool according to the reconfiguration configuration information to realize hardware system reconfiguration.

As shown in fig. 3, another embodiment of the present invention is a crossbar-based reconfiguration system in a resource pool, wherein the crossbar is implemented in the form of a PCIe switch backplane, and the PCIe switch backplane includes multiple PCIe bus slots and PCIe switch chips. Different compute and memory resources are inserted into the cross-bar backplane slots within the pool. The hardware architecture of the resource pool is dynamically constructed by adopting a method based on a PCIe multi-path bus and a cross switch, and configuration information controls the interconnection relation between an input interface and an output interface of the cross switch. In the figure, hardware resources such as different CPUs, GPUs, FPGAs, IPUs, SSDs and HDDs are connected to PCIe bus slots of the cross bar switch, and the CPUs can select hardware devices connected to the appointed bus slots to form different computing systems. By configuring the crossbar within the resource pool, hardware devices connected on the PCIe bus may be changed. For example: at the next moment, by changing the configuration of the cross switch, the hardware resources connected with the PCIe buses such as the PCIe bus 1, the PCIe bus 2 and the like can be changed by exchanging, adding, deleting and the like, so that a new computing system is formed.

The mechanism for reconfiguring hardware resources such as CPU, GPU, FPGA, IPU, SSD, HDD, etc. on different computing and storage resource pools is shown in fig. 4. And adjacent resource pools are interconnected through a remote PCIe cross switch, so that the communication of the processors among the resource pools is realized. The remote PCIe crossbar switch is realized in the form of a PCIe switch and comprises a plurality of PCIe shared ports, and the hardware architecture of different resource pools is dynamically constructed and realized based on the control of the remote PCIe crossbar switch.

For example: in the figure, the hardware resources of the resource pool 1 and the resource pool 2 are connected through a remote PCIe crossbar shown in fig. 4, so that the hardware resources of the resource pool 1 and the resource pool 2 can directly realize data interaction through PCIe transmission. The remote PCIe crossbar is connected to the crossbar in the resource pool through a shared interface. A two-tier crossbar configuration link based on a crossbar and a remote PCIe crossbar is formed. When connection configuration is carried out, firstly, the cross switch connection relation in the computing and storage resource pool is configured through configuration information, and computing and storage resources are distributed for a specified PCIe bus in the computing and storage resource pool; and then configuring a remote PCIe cross switch among the resource pools to realize the direct connection from the resource pools to the resource pools. As shown in fig. 4, the hardware reconfiguration mechanism between resource pools can configure the crossbar switch and the remote PCIe crossbar switch in the pool, so that the CPU and the FPGA in the motherboard 1 and the SSD and the CPU in the motherboard 2 can communicate with each other without affecting the use of other hardware resources.

Fig. 5 is a schematic structural diagram of a remote crossbar switch, and as shown in fig. 5, the embodiment of the present invention supports one or more of the following switching modes: packet switched PS, circuit switched CS and virtual channel switched VCS. Virtual channel switching for forming a plurality of virtual circuit switched connections that can share the same physical channel through the virtual channels, and the virtual circuit switched connections can cooperate with the packet switched connections and the circuit switched connections to transmit streaming data. According to the known inter-core flow communication condition, a compiler distributes a path for each known flow communication, virtual circuit switching connection, circuit switching connection and packet switching connection are determined, connection information is stored in each PCIe exchanger through prestored connection information according to a communication compiling result during operation to establish each switching connection, and communication among different mainboard resource pools is transmitted on the corresponding switching connection.

Compared to a basic packet switch, the hybrid-policy switch of the present invention requires some additional hardware: including bypass channels, circuit configuration storage, and VCS state storage. The remote PCIe crossbar switch comprises a plurality of input units, a crossbar switch spanning submodule, a path calculation submodule, a virtual channel distribution submodule, a crossbar switch distribution submodule, a circuit configuration submodule and the like.

Wherein: the input units are used for configuring output arbitrated virtual channel numbers for connected PCIe devices, and each input unit comprises n output virtual channels VC1-VCn, a bypass channel, a PS state storage and a VCS state storage. The input unit is connected with an input Virtual Channel Switching (VCS) signal and a port input signal of PCIe equipment, and the serial numbers of the VCS signal and the input signal correspond to the port number of the PCIe equipment one by one. The PS state stores the virtual tunnel state of the corresponding basic packet-switched router. The VCS state stores a corresponding virtual circuit switched connection state. When the input is a VCS signal, an output virtual channel number corresponding to the input virtual channel is found directly through information in VCS state storage, and the VCS signal is directly output to PCIe equipment corresponding to the output virtual channel number; the PS state and the VCS state each contain n fields, corresponding to the n virtual channels. And the n virtual channels are shared by the virtual circuit-switched connection and the packet-switched connection. And each input unit is added with a bypass channel to allow the data sheet to be directly input to the cross-bar sub-module without passing through a virtual channel.

The path calculation module is connected with the input unit and used for establishing an ID-port routing path or an address-port routing path and providing a path arbitration result for the data sheet transmitted in the packet switching connection. Wherein the ID-port routing table provides a path mapping relationship between PCIe device numbers and port numbers; the address-port routing table provides a path mapping relationship between PCIe device physical addresses and port numbers. And the operation carries out communication path arbitration according to the ID and the address in the PCIe transaction packet.

The virtual channel allocation unit is used for allocating output virtual channel numbers for the input data pieces according to the path arbitration result of the path calculation unit or according to the PS state storage information;

the circuit configuration unit is used for providing pre-stored cross switch configuration information for the data sheet transmitted in the circuit switching connection, storing the connection information used for storing the physical channel for the circuit switching connection, connecting with the cross switch distribution module and the virtual channel distribution module, and establishing the circuit switching connection by setting the circuit configuration information.

The cross switch distribution unit is used for storing preset circuit configuration storage information through circuit configuration and directly configuring the cross switch so that data sheets connected in circuit switching can directly enter the cross switch spanning sub-module through the bypass switch;

and the cross switch crosses the sub module and is used for outputting the data packet to PCIe equipment corresponding to the output virtual channel number according to the arbitration scheduling result of the input unit.

In order to implement a hybrid strategy supporting circuit switching, packet switching, and virtual channel switching, the present patent modifies the PCIe transaction layer packet header of the PCIe transaction packet, and adds a switching flag bit in the reserved field, as shown in fig. 6. And when the hardware reconfiguration configuration of the PCIe cross switch is carried out, selecting a corresponding exchange strategy according to the exchange zone bit of the PCIe transaction layer packet header.

If the exchange flag bit of the input signal PCIe transaction layer packet header is an input virtual channel VCS signal, an output virtual channel number corresponding to the input virtual channel is found directly through VCS state information, and the VCS signal is directly output to PCIe equipment corresponding to the output virtual channel number.

If the input signal is the input signal of the PCIe port, the arbitration scheduling needs to be performed on the input unit according to the input signal of the PCIe port. If the exchange flag bit of the PCIe transaction layer packet header is a circuit exchange signal, the data sheet transmitted in the circuit exchange connection reaches the input unit, the cross switch crossing sub-module is immediately configured according to the connection information pre-stored in the circuit configuration, then the data sheet directly enters the cross switch crossing sub-module through the bypass channel, and the cross switch crossing sub-module directly outputs the data packet to the corresponding PCIe device according to the configuration result.

If the input signal is the input signal of the PCIe port, the arbitration scheduling needs to be performed on the input unit according to the input signal of the PCIe port. If the exchange flag bit of the PCIe transaction layer packet header is a packet exchange signal, the data sheet transmitted in the packet exchange connection reaches the input unit, firstly, an output virtual channel number is distributed to the input data sheet according to the path arbitration result of the path calculation unit or according to the PS state information, then, the output virtual port number is sent to the cross switch crossing sub-module, and the cross switch crossing sub-module outputs the data packet to PCIe equipment corresponding to the output virtual channel number according to the arbitration scheduling result.

A second embodiment of the present invention provides a hardware reconfiguration method, referring to fig. 7, which applies the hardware reconfiguration system according to any one of the first embodiments of the present invention, including:

The embodiment of the invention realizes the reconstruction of computing capability and the reconstruction of connection capability by reconstructing and configuring the cross switch in the resource pool and the remote PCIe cross switch. Reference may be made in detail to the first embodiment of the present invention, which will not be discussed in detail.

The embodiment of the invention can at least bring the following beneficial effects:

1. the embodiment of the invention supports heterogeneous computing resources such as CPU, GPU, FPGA, IPU and the like, storage resources such as SSD, HDD and the like are distributed in each computing and storage resource pool, the storage resource pools are interconnected through a remote PCIe cross switch, and the CPU, GPU, FPGA, IPU and storage resources in the storage resource pools are interconnected through PCIe multi-buses and cross switches, so that hardware resources can be decoupled, and the dynamic configuration capability of a network is improved;

2. the invention provides a hierarchical interconnection scheme of a PCIe crossbar switch and a remote PCIe crossbar switch in a pool, which can allocate proper amount of computing nodes and storage nodes for application according to the performance requirement of a data center server by configuring the PCIe crossbar switch in the pool and the remote PCIe crossbar switch, wherein each computing storage resource can be uniformly managed and flexibly allocated, and the hardware architecture supports the construction of computing capacity and connection capacity as required by reconstructing the connection between each hardware resource, thereby effectively improving the performance and energy efficiency of the hardware architecture;

3. the invention provides a remote PCIe cross switch configuration scheme of a hybrid switching mechanism, which simultaneously supports three switching modes of packet switching, circuit switching and virtual channel switching, wherein the virtual channel switching can directly pass through the cross switch, the configuration is simpler and more convenient, and simultaneously supports the use of the packet switching or the circuit switching when links conflict, thereby ensuring a high-efficiency communication link based on a PCIe multi-path bus and the cross switch.

It should be noted that the PCIe crossbar switch according to the embodiment of the present invention may be implemented by a PCIe switch chip, and may also implement the crossbar switch function by an FPGA.

The algorithms and displays presented herein are not inherently related to any particular computer, virtual machine, or other apparatus. Various general purpose systems may also be used with the teachings herein. The required structure for constructing such a system will be apparent from the description above. Moreover, the present invention is not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any descriptions of specific languages are provided above to disclose the best mode of the invention.

In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.

The various component embodiments of the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or Digital Signal Processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in a distributed file system data import apparatus according to embodiments of the present invention. The present invention may also be embodied as apparatus or device programs (e.g., computer programs and computer program products) for performing a portion or all of the methods described herein. Such programs implementing the present invention may be stored on computer-readable media or may be in the form of one or more signals. Such a signal may be downloaded from an internet website or provided on a carrier signal or in any other form.

It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The usage of the words first, second and third, etcetera do not indicate any ordering. These words may be interpreted as names.

Claims

1. A hardware reconfiguration system, comprising:

the system comprises a resource pool, a remote PCIe crossbar switch, a double-layer crossbar switch configuration link and a double-layer crossbar switch configuration link, wherein the resource pool is connected with an adjacent resource pool through a multi-path interface based on the remote PCIe crossbar switch;

the remote PCIe crossbar switch is interconnected with the shared interfaces of all resource pools through the shared interfaces on the remote PCIe crossbar switch, peer-to-peer exchange between the resource pools based on the PCIe protocol is realized, when connection configuration is carried out, the connection relationship of the remote PCIe crossbar switch is configured through configuration information, processor resources of adjacent resource pools are distributed for a designated PCIe bus of a CPU, and calculation and storage resources among the resource pools are dynamically distributed.

2. The system of claim 1,

the shared interface supports two interface modes of a transparent bridge and a non-transparent bridge, can be configured as a master port or a slave port, and realizes interconnection between the ports through a PCIe shared interface cable.

3. The system of claim 1,

the PCIe crossbar switch in the resource pool is realized by a PCIe exchange bottom plate, and the PCIe exchange bottom plate comprises a plurality of PCIe bus slots and a PCIe exchange chip.

4. The system of claim 1,

the remote PCIe crossbar is realized by PCIe switches, realizes a point-to-point switching mode among resource pools, and comprises one or more of the following modes: packet switching, circuit switching, and virtual channel switching.

5. The system of claim 4,

the virtual channel switch is used for forming a plurality of virtual circuit switch connections which can share the same physical channel through the virtual channel, and the virtual circuit switch connections can be matched with the packet switch connections and the circuit switch connections to transmit stream data.

6. The system of claim 4, further comprising a compiler;

the compiler is used for distributing paths for each known flow communication according to the known flow communication condition between cores, determining virtual circuit switching connection, circuit switching connection and packet switching connection, establishing each switching connection through prestored connection information according to the communication compiling result during operation, and transmitting the communication between different resource pools on the corresponding switching connection.

7. The system of claim 1,

the remote PCIe crossbar switch comprises a plurality of input units, a crossbar crossing unit, a path calculation unit, a virtual channel distribution unit, a crossbar distribution unit and a circuit configuration unit;

8. The system of claim 1,

the pool of computing resources includes one or more of: CPU, GPU, FPGA, IPU.

9. The system of claim 1,

the storage resource pool comprises one or more of: SSD, HDD.

10. A hardware reconfiguration method, wherein the hardware reconfiguration system according to any one of claims 1 to 9 is applied, comprising:

step 6: the PCIe data packet required by the invention is constructed through the packet format of PCIe DMA or a standard PCIe link layer protocol, the equipment connected with the appointed PCIe bus is accessed, and the PCIe hardware reconfiguration configuration is finished;