CN116049079A

CN116049079A - Method, system, device and storage medium for interconnecting multi-chip cluster servers

Info

Publication number: CN116049079A
Application number: CN202310077841.1A
Authority: CN
Inventors: 赵元; 苏康; 李灯伟
Original assignee: Shandong Yunhai Guochuang Cloud Computing Equipment Industry Innovation Center Co Ltd
Current assignee: Shandong Yunhai Guochuang Cloud Computing Equipment Industry Innovation Center Co Ltd
Priority date: 2023-01-30
Filing date: 2023-01-30
Publication date: 2023-05-02

Abstract

The invention provides a method, a system, equipment and a storage medium for interconnecting multi-chip cluster servers, wherein the method comprises the following steps: setting a custom design chip comprising a plurality of PCIe terminal devices and a first PCIe bridge device, and setting a custom exchange chip comprising a ROOT port function module of PCIe and a second PCIe bridge device; setting a custom design chip in each server cluster, establishing connection between each SOC server in the server cluster and the custom design chip through a plurality of PCIe terminal devices, and establishing connection between the second PCIe bridge device and each first PCIe bridge device; responding to data interaction in the cluster, establishing a first routing channel between two servers through a first PCIe bridge device, and carrying out data interaction through the first routing channel; and responding to the data interaction between the clusters, establishing a second routing channel between the two clusters through the first PCIe bridge device and the second PCIe bridge device, and carrying out the data interaction through the second routing channel.

Description

Method, system, device and storage medium for interconnecting multi-chip cluster servers

Technical Field

The present invention relates to the field of chip design, and in particular, to a method, a system, an apparatus, and a storage medium for interconnecting multi-chip cluster servers.

Background

Along with the rapid development of cloud computing in recent years, the scale is continuously increased, the application of a multipath server is becoming wider and wider, the stability performance is good, the safety is high, the operation efficiency is high, and the multipath server has been widely applied in almost all production and living fields. With the continuous development of the application field of the server, the application requirement of the high-end server has entered an important stage. The complex architecture implementation supports the high-performance index, i.e. the characteristics of high safety, high availability, high reliability, and the like, of the high-end server system.

In the current implementation of shared memory based on the cluster interconnection of common processors, as shown in fig. 1, the common processor clusters are mainly formed by interconnection of standard ethernet through a multi-level network switch, and the task scheduling of system software is used for implementing execution of various system tasks. In particular, as the SOC (System On Chip) of the processor interconnection to the mobile phone System Chip generally does not have a standard ethernet interface, as shown in fig. 2, the processor interconnection is often realized by externally connecting an expansion network conversion Chip through PCIe (Peripheral Component Interconnect Express) or a USB interface, converting the interface into a standard ethernet interface, and then, interconnecting the interfaces through a network switch, so as to form a cluster System consistent with the common processor interconnection architecture through the ethernet interface.

For a general server adopting a server-level processor chip, enough network interfaces can be connected to all levels of Ethernet switches so as to realize multi-machine interconnected clusters, but for a cluster realized by a low-cost mobile phone SOC chip, as no standard network interfaces exist, as can be seen from FIG. 2, an external network interface adaptation conversion chip is needed to realize the network cluster of multiple processors. Because the purpose of clustering by adopting the low-cost mobile phone SOC chip is to reduce the cost of the whole processor cluster, the additional network interface adaptation conversion chip for each mobile phone SOC chip greatly increases the cost of the whole system, thereby losing the original system design goal of reducing the cost of the whole processor cluster. In addition, multistage forwarding across a plurality of different interfaces is needed through interface conversion, the interconnection efficiency between mobile phone SOC chips is limited by the interfaces, and the communication and data interaction efficiency is obviously reduced. Therefore, the performance of the conventional mobile phone SOC chip cluster interconnection scheme is greatly reduced, and meanwhile, the cost of the cluster system cannot be greatly reduced, so that the cost performance and the market competitiveness of the whole system solution are obviously affected.

Disclosure of Invention

In view of the above, an object of the embodiments of the present invention is to provide a method, a system, a computer device, and a computer readable storage medium for interconnecting multi-chip cluster servers, which can simplify the interconnection manner of SOC processors, thereby greatly reducing the number of external expansion chips and greatly reducing the product cost; the invention creatively adopts a new mode to realize the clustering of the SOC processor, and provides a new thought and a new product mode for the design and realization of the micro cluster server; the invention greatly simplifies the complexity of SOC processor cluster interconnection, reduces the cost of design resources, thereby effectively reducing the production test cost, simplifying the function realization and effectively reducing the realization complexity in the development test process.

Based on the above objects, an aspect of the embodiments of the present invention provides a method for interconnecting multi-chip cluster servers, including the following steps: setting a custom design chip comprising a plurality of PCIe terminal devices and a first PCIe bridge device, and setting a custom exchange chip comprising a ROOT port function module of PCIe and a second PCIe bridge device; setting a custom design chip in each server cluster, establishing connection between each SOC server in the server cluster and the custom design chip through a plurality of PCIe terminal devices, and establishing connection between the second PCIe bridge device and each first PCIe bridge device; responding to data interaction in the cluster, establishing a first routing channel between two servers through the first PCIe bridge device, and carrying out data interaction through the first routing channel; and responding to the data interaction between the clusters, establishing a second routing channel between the two clusters through the first PCIe bridge device and the second PCIe bridge device, and carrying out the data interaction through the second routing channel.

In some embodiments, the establishing, by the first PCIe bridge device, a first routing path between two servers includes: responding to the data writing of the starting server to the destination server, initiating a data reading request according to the appointed memory address of the starting server, and initiating a DMA memory writing request to the first PCIe bridge device; the first PCIe bridge device routes the DMA write memory request to PCIe terminal device of the destination server according to the configuration information, and the PCIe terminal device of the destination server initiates the DMA write request to the destination server and receives the return information of the destination server; and sending, by the first PCIe bridge device, the write memory address in the return message to the originating server to establish a first routing channel from the originating server address space to the destination server address space.

In some embodiments, the establishing, by the first PCIe bridge device, a first routing path between two servers includes: responding to the reading of data from an initial server to a destination server, configuring a DMA (direct memory access) write memory instruction for PCIe (peripheral component interconnect express) terminal equipment of the initial server, and distributing write memory address space, data size and address information of the destination server; and the first PCIe bridge device routes the DMA read memory request to the PCIe terminal device of the destination server according to the configuration information, and the PCIe terminal device of the destination server initiates the DMA read request to the destination server and receives the returned data of the destination server.

In some embodiments, the establishing a second routing channel between two clusters through the first PCIe bridge device and the second PCIe bridge device includes: responding to the initial cluster to write data into the target cluster, initiating a read data request according to the appointed memory address of the initial server of the initial cluster, and initiating a DMA write memory request to the first PCIe bridge device of the initial server; the first PCIe bridge device routes the DMA write memory request to the first PCIe bridge device of the target cluster through the second PCIe bridge device, and the first PCIe bridge device of the target cluster routes the DMA write memory request to PCIe terminal devices of the target server according to the memory space configuration information; and receiving the return information of the destination server through the PCIe terminal equipment of the destination server, and routing the write memory address in the return information to the PCIe terminal equipment of the starting server through the second PCIe bridge equipment, the first PCIe bridge equipment of the starting cluster and the first PCIe bridge equipment of the destination cluster.

In some embodiments, the establishing a second routing channel between two clusters through the first PCIe bridge device and the second PCIe bridge device includes: responding to the initial cluster to read data from the target cluster, the initial cluster configures a DMA memory writing instruction to the connected PCIe terminal equipment and distributes the address information of the target server of the target cluster and the address space and the data size of the written memory; and the first PCIe bridge device of the starting server routes the DMA read memory request to the PCIe terminal device of the destination server of the destination cluster through the second PCIe bridge device according to the configuration information, and the PCIe terminal device of the destination server initiates the DMA read request to the destination server and receives the returned data of the destination server.

In some embodiments, the method further comprises: and forming a basic PCIe system by each PCIe terminal device and the corresponding server, independently initializing each basic PCIe system, distributing corresponding ID and address space and binding with the corresponding custom design chip for use.

In some embodiments, the method further comprises: and forming a tree PCIe system by each PCIe terminal device, the first PCIe bridge device, the second PCIe bridge device and the ROOT module with the ROOT port, and distributing independent and different buses, device IDs and address spaces for each class of bridge devices and all PCIe terminal devices of each tree PCIe system.

In another aspect of the embodiments of the present invention, there is provided a system for interconnecting multi-chip cluster servers, including: the device comprises a setting module, a first PCIe bridge device and a second PCIe bridge device, wherein the setting module is configured to set a custom design chip comprising a plurality of PCIe terminal devices and the first PCIe bridge device, and set a custom exchange chip comprising a PCIe ROOT port function module and the second PCIe bridge device; the connection module is configured to set a custom design chip in each server cluster, establish connection between each SOC server in the server cluster and the custom design chip through a plurality of PCIe terminal devices, and establish connection between the second PCIe bridge device and each first PCIe bridge device; the first interaction module is configured to respond to the data interaction inside the cluster, establish a first routing channel between two servers through the first PCIe bridge device, and conduct the data interaction through the first routing channel; and the second interaction module is configured to respond to the data interaction between the clusters, establish a second routing channel between the two clusters through the first PCIe bridge device and the second PCIe bridge device, and conduct the data interaction through the second routing channel.

In yet another aspect of the embodiment of the present invention, there is also provided a computer apparatus, including: at least one processor; and a memory storing computer instructions executable on the processor, which when executed by the processor, perform the steps of the method as above.

In yet another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium storing a computer program which, when executed by a processor, implements the method steps as described above.

The invention has the following beneficial technical effects:

1. through the custom design of a multi-level PCIe bridge, the data exchange among the mobile phone SOC processor chips is realized, the additional conversion chip on each interface is replaced, and the data exchange in the cluster node does not need to be accessed to a network switch by using each mobile phone SOC processor chip as a single node, so that the implementation cost of the cluster is greatly reduced;

2. through the interconnection use of PCIe interfaces, PCIe EP device P2P memory sharing and data exchange of a plurality of mobile phone SOC processors are directly carried out, so that larger delay caused by repeated interface forwarding of the system is reduced, and the overall system performance is improved;

3. The method realizes the quick data exchange among a plurality of PCIe root nodes, and provides a new quick data exchange scheme for the interconnection of processors with PCIe root interfaces;

4. the cluster number of the SOC processors is easily expanded, and processors in different nodes can realize fast data exchange across the nodes through a multi-PCIe tree system consisting of terminal EP devices.

Drawings

In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are necessary for the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention and that other embodiments may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic diagram of a cluster interconnect architecture for a conventional processor in the prior art;

FIG. 2 is a schematic diagram of a prior art network switch and SOC interconnected architecture;

FIG. 3 is a schematic diagram of an embodiment of a method for interconnecting multi-chip cluster servers according to the present invention;

FIG. 4 is a schematic diagram of an interconnection architecture of a multi-chip cluster server according to the present invention;

FIG. 5 is a schematic diagram of an embodiment of a system for interconnecting multi-chip cluster servers according to the present invention;

FIG. 6 is a schematic diagram of a hardware architecture of an embodiment of a computer device interconnected by a multi-chip cluster server according to the present invention;

fig. 7 is a schematic diagram of an embodiment of a computer storage medium interconnected by a multi-chip cluster server according to the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention will be described in further detail with reference to the accompanying drawings.

It should be noted that, in the embodiments of the present invention, all the expressions "first" and "second" are used to distinguish two entities with the same name but different entities or different parameters, and it is noted that the "first" and "second" are only used for convenience of expression, and should not be construed as limiting the embodiments of the present invention, and the following embodiments are not described one by one.

In a first aspect of the embodiment of the present invention, an embodiment of a method for interconnecting multi-chip cluster servers is provided. Fig. 3 is a schematic diagram of an embodiment of a method for interconnecting multi-chip cluster servers according to the present invention. As shown in fig. 3, the embodiment of the present invention includes the following steps:

S1, setting a custom design chip comprising a plurality of PCIe terminal devices and a first PCIe bridge device, and setting a custom exchange chip comprising a ROOT port function module of PCIe and a second PCIe bridge device;

s2, setting a custom design chip in each server cluster, establishing connection between each SOC server in the server cluster and the custom design chip through a plurality of PCIe terminal devices, and establishing connection between the second PCIe bridge device and each first PCIe bridge device;

s3, responding to data interaction in the cluster, establishing a first routing channel between two servers through the first PCIe bridge device, and carrying out data interaction through the first routing channel; and

s4, responding to data interaction between the clusters, establishing a second routing channel between the two clusters through the first PCIe bridge device and the second PCIe bridge device, and performing data interaction through the second routing channel.

According to the invention, task distribution and data exchange among a plurality of processors are realized by taking the mobile phone SOC chip based on the ARM processor into consideration, the application scene of the mobile phone SOC chip based on the ARM processor can be greatly expanded, the server miniaturization clustering of the mobile phone SOC chip based on the ARM processor is realized, and the processing function of the universal ARM server is realized under the condition of smaller physical and mechanical size. Because the yield and market share of the mobile phone SOC chip based on the ARM processor are huge, the processor realized by the method has the natural chip cost advantage under large-scale mass production, and the server cluster of the mobile phone SOC chip based on the ARM processor also has the natural advantage of compatibility and adaptability to the service of mobile phone application running at a server side.

The invention provides a novel quick data interaction method for secondary cluster interconnection of multiple SOC chips on a single board card, which is beneficial to directly realizing quick data interaction of multiple SOC chips by utilizing a mobile phone SOC chip interface based on an ARM processor, so that task dispatch can be quickly carried out on the multiple SOC chips, and stable operation of application tasks is realized. Under the condition of keeping the SOC chip interface unchanged, the mobile phone terminal and the server terminal can be compatible, the application scene of the mobile phone SOC chip based on the ARM processor is expanded, the iteration speed of the mobile phone SOC chip can be matched with that of the mobile phone terminal SOC chip, and the server cluster equipment of the mobile phone SOC chip based on the ARM processor can be rapidly produced in quantity and marketed.

Fig. 4 is a schematic diagram of an architecture of interconnection of multiple chip cluster servers provided by the present invention, as shown in fig. 4, the present invention includes m cluster servers, each server cluster includes n SOC server systems, the SOC servers are connected to a cluster custom-designed chip of the present invention through PCIe interfaces, an interface of the chip connected to each SOC server is presented as an EP (terminal) device, and a ROOT port of the SOC server is connected to the EP device through PCIe interfaces. Each cluster is provided with one customized design chip of the invention, which is connected with each SOC server in the cluster through a PCIe interface, and m customized design chips of the invention exist in the system and are respectively connected with each SOC server in the m clusters through the PCIe interface.

The custom designed chip of the present invention in each cluster contains a total of PCIe Bridge devices (Bridge 1 in fig. 4) containing n PCIe EP (terminal) devices. Each EP device has a PCIe interface in two directions, one connected to the ROOT interface of the corresponding SOC server as described above, and the other connected to the downstream port of the bridge device in the chip. The bridge device in the chip contains n downstream ports and one upstream port.

The M server clusters are interconnected through the customized switching chip in the present invention, and the lowest block diagram in fig. 4 is the customized switching chip, where the customized switching chip includes a PCIe ROOT port function module and a PCIe Bridge device (Bridge 0 in fig. 4). The Bridge0 Bridge device has m downstream ports, and is connected with upstream ports of Bridge1 Bridge devices of custom chips in m clusters respectively. The upstream port of Bridge0 Bridge device is connected to the ROOT module.

The data in the memory of SOC0 is written into the memory of SOC 1. The SOC0 configures a DMA memory reading instruction to the corresponding connected EP0 through a ROOT0 port, the EP0 starts a built-in DMA, a read data request is initiated according to a specified memory address of the SOC0, meanwhile, the EP0 initiates a DMA memory writing request to the lower Brigde1, the bridge1 routes the DMA memory writing request to the EP1 according to configuration information, the EP1 initiates the DMA memory writing request to the SOC1, the SOC1 responds to the request, a corresponding data storage address space is allocated, and the address space information is returned to the DMA of the EP1, after the EP1 receives the information, the corresponding memory writing address information is sent to the EP0 through the bridge1, and a routing channel from the SOC0 address space to the SOC1 address space is established. Then, EP0 reads data from the corresponding address space of SOC0, writes data into the corresponding address space of SOC1 after receiving the data, and after the data writing is completed, EP0 and EP1 send interrupts to the systems of SOC0 and SOC1, respectively, informing that the reading and writing of data are completed.

The memory from which SOC0 reads data from the memory of SOC1 to SOC0 will be described as an example. The SOC0 configures a DMA memory writing instruction to the corresponding connected EP0 through a ROOT0 port, allocates a writing memory address space, a data size and address information of the SOC1 to be read, the EP0 initiates a DMA memory reading request to the lower Brigde1, the bridge1 routes the DMA memory reading request to the EP1 according to the address configuration information, the EP1 initiates the DMA memory reading request to the SOC1, the SOC1 responds to the request, the read memory data is returned to the EP1, the EP1 sends the read data to the EP0, the DMA of the EP0 initiates the writing request, and the data is written into the corresponding memory space of the SOC 0. After the data writing is completed, the EP0 and the EP1 send interrupts to the systems of the SOC0 and the SOC1 respectively, and the completion of the data writing and the data reading is notified.

Taking the example that the data in the memory of SOC0 of cluster 1 is written into the memory of SOC1 of cluster 2 as an example. The SOC0 of the cluster 1 configures a DMA memory reading instruction to the corresponding connected EP0 through a ROOT0 port, the EP0 starts a built-in DMA, a read data request is initiated according to a designated memory address of the SOC0 of the cluster 1, meanwhile, the EP0 initiates a DMA memory writing request to the lower bridge1, the bridge1 sends the request to the bridge0 in an ascending mode according to the configuration information of a memory space, the bridge0 routes the DMA memory writing request to the bridge1 of the cluster 2 according to the configuration information of the memory space, the bridge1 then initiates a DMA write request to the SOC1 of the connected cluster 2 below the downstream port of the DMA memory writing request according to the configuration information of the memory space, the SOC1 of the cluster 2 responds to the request, a corresponding data storage address space is allocated, and the address space information is returned to the EP1, after the EP1 receives the information, the corresponding write memory address information is routed to the EP0 of the cluster 0 through the bridge1 and the bridge0, and a routing channel from the SOC address space of the cluster 1 to the SOC1 of the cluster 2 is established. Then, EP0 reads data from the corresponding address space of SOC0 of cluster 1, writes the data into the corresponding address space of SOC1 of cluster 2 after receiving the data, and after the data writing is completed, sends interrupts to the systems of SOC0 of cluster 1 and SOC1 of cluster 2, respectively, informing that the reading and writing of the data are completed.

Taking SOC0 of cluster 1 as an example, reading data from the memory of SOC1 of cluster 2 to the memory of SOC0 of cluster 1 is described. The SOC0 of the cluster 1 configures a DMA write memory instruction to the corresponding connected EP0 through a ROOT0 port and distributes address information written into a memory address space and data size and SOC1 of the cluster 2 to be read, the EP0 initiates a DMA read memory request to the lower bridge1, the bridge1 sends the DMA read memory request to the upstream port according to the address configuration information, the bridge0 routes the DMA read memory request to the bridge1 of the cluster 2 according to the address configuration information, the bridge1 of the cluster 2 routes the DMA read memory request to the EP1 of the cluster 2 according to the address configuration information, the EP1 initiates a DMA read request to the SOC1 of the cluster 2, the SOC1 of the cluster 2 responds to the request, the read memory data is returned to the EP1, the EP1 sends the read data to the bridge1 of the cluster 2, the bridge1 forwards the data to the bridge0, the bridge0 forwards the data to the EP0 of the cluster 1, and after the EP0 receives the data, the DMA write request is initiated through the built-in memory request, and the data is written into the corresponding SOC1 of the cluster 0. After the data writing is completed, the EP0 of the cluster 1 and the EP1 of the cluster 2 send interrupts to the system of the SOC0 of the cluster 1 and the system of the SOC1 of the cluster 2 respectively, and the completion of the data writing and the data reading is notified.

When the system is initialized, each EP in the system in the figure 4 is respectively presented in two sets of PCIe systems through two PCIe interfaces of the system, and one system is a ROOT-EP basic PCIe system formed by the EP and a ROOT port of an SOC server above, so that the data interaction communication between the SOC server and the EP port is realized; another system is that the EP is connected to bridge0 of the switch chip through bridge1 below and then to the ROOT module with ROOT port below through bridge 0. A full PCIe tree system structure including a lower ROOT-bridge0-brgidg 1-EP m n is formed.

In some embodiments, the method further comprises: and forming a basic PCIe system by each PCIe terminal device and the corresponding server, independently initializing each basic PCIe system, distributing corresponding ID and address space and binding with the corresponding custom design chip for use. In total, m×n PCIe systems exist above FIG. 4, each PCIe system is independently initialized, and is allocated with respective ID numbers and address spaces and is used by binding with chip numbers through custom design.

In some embodiments, the method further comprises: and forming a tree PCIe system by each PCIe terminal device, the first PCIe bridge device, the second PCIe bridge device and the ROOT module with the ROOT port, and distributing independent and different buses, device IDs and address spaces for each class of bridge devices and all PCIe terminal devices of each tree PCIe system. In fig. 4, a tree PCIe system is shown below, and during initialization, a system traversal is initiated by a ROOT port below, and configuration information such as an independent and different bus, a device ID number, an address space, and the like is allocated to each level of bridge device and all EP terminal devices.

The invention can simplify the interconnection mode of the SOC processors, thereby greatly reducing the number of external expansion chips and achieving the purpose of greatly reducing the product cost; the invention creatively adopts a new mode to realize the clustering of the SOC processor, and provides a new thought and a new product mode for the design and realization of the micro cluster server; the invention greatly simplifies the complexity of SOC processor cluster interconnection, reduces the cost of design resources, thereby effectively reducing the production test cost, simplifying the function realization and effectively reducing the realization complexity in the development test process.

It should be noted that, the steps in the embodiments of the method for interconnecting multi-chip cluster servers may be intersected, replaced, added and deleted, so that the method for interconnecting multi-chip cluster servers by the reasonable permutation and combination should also belong to the protection scope of the present invention, and should not limit the protection scope of the present invention to the embodiments.

Based on the above object, a second aspect of the embodiments of the present invention provides a system for interconnecting multi-chip cluster servers. As shown in fig. 5, the system 200 includes the following modules: the device comprises a setting module, a first PCIe bridge device and a second PCIe bridge device, wherein the setting module is configured to set a custom design chip comprising a plurality of PCIe terminal devices and the first PCIe bridge device, and set a custom exchange chip comprising a PCIe ROOT port function module and the second PCIe bridge device; the connection module is configured to set a custom design chip in each server cluster, establish connection between each SOC server in the server cluster and the custom design chip through a plurality of PCIe terminal devices, and establish connection between the second PCIe bridge device and each first PCIe bridge device; the first interaction module is configured to respond to the data interaction inside the cluster, establish a first routing channel between two servers through the first PCIe bridge device, and conduct the data interaction through the first routing channel; and the second interaction module is configured to respond to the data interaction between the clusters, establish a second routing channel between the two clusters through the first PCIe bridge device and the second PCIe bridge device, and conduct the data interaction through the second routing channel.

In some embodiments, the first interaction module is configured to: responding to the data writing of the starting server to the destination server, initiating a data reading request according to the appointed memory address of the starting server, and initiating a DMA memory writing request to the first PCIe bridge device; the first PCIe bridge device routes the DMA write memory request to PCIe terminal device of the destination server according to the configuration information, and the PCIe terminal device of the destination server initiates the DMA write request to the destination server and receives the return information of the destination server; and sending, by the first PCIe bridge device, the write memory address in the return message to the originating server to establish a first routing channel from the originating server address space to the destination server address space.

In some embodiments, the first interaction module is configured to: responding to the reading of data from an initial server to a destination server, configuring a DMA (direct memory access) write memory instruction for PCIe (peripheral component interconnect express) terminal equipment of the initial server, and distributing write memory address space, data size and address information of the destination server; and the first PCIe bridge device routes the DMA read memory request to the PCIe terminal device of the destination server according to the configuration information, and the PCIe terminal device of the destination server initiates the DMA read request to the destination server and receives the returned data of the destination server.

In some embodiments, the second interaction module is configured to: responding to the initial cluster to write data into the target cluster, initiating a read data request according to the appointed memory address of the initial server of the initial cluster, and initiating a DMA write memory request to the first PCIe bridge device of the initial server; the first PCIe bridge device routes the DMA write memory request to the first PCIe bridge device of the target cluster through the second PCIe bridge device, and the first PCIe bridge device of the target cluster routes the DMA write memory request to PCIe terminal devices of the target server according to the memory space configuration information; and receiving the return information of the destination server through the PCIe terminal equipment of the destination server, and routing the write memory address in the return information to the PCIe terminal equipment of the starting server through the second PCIe bridge equipment, the first PCIe bridge equipment of the starting cluster and the first PCIe bridge equipment of the destination cluster.

In some embodiments, the second interaction module is configured to: responding to the initial cluster to read data from the target cluster, the initial cluster configures a DMA memory writing instruction to the connected PCIe terminal equipment and distributes the address information of the target server of the target cluster and the address space and the data size of the written memory; and the first PCIe bridge device of the starting server routes the DMA read memory request to the PCIe terminal device of the destination server of the destination cluster through the second PCIe bridge device according to the configuration information, and the PCIe terminal device of the destination server initiates the DMA read request to the destination server and receives the returned data of the destination server.

In some embodiments, the system further comprises an initialization module configured to: and forming a basic PCIe system by each PCIe terminal device and the corresponding server, independently initializing each basic PCIe system, distributing corresponding ID and address space and binding with the corresponding custom design chip for use.

In some embodiments, the system further comprises an allocation module configured to: and forming a tree PCIe system by each PCIe terminal device, the first PCIe bridge device, the second PCIe bridge device and the ROOT module with the ROOT port, and distributing independent and different buses, device IDs and address spaces for each class of bridge devices and all PCIe terminal devices of each tree PCIe system.

In view of the above object, a third aspect of the embodiments of the present invention provides a computer device, including: at least one processor; and a memory storing computer instructions executable on the processor, the instructions being executable by the processor to perform the steps of: s1, setting a custom design chip comprising a plurality of PCIe terminal devices and a first PCIe bridge device, and setting a custom exchange chip comprising a ROOT port function module of PCIe and a second PCIe bridge device; s2, setting a custom design chip in each server cluster, establishing connection between each SOC server in the server cluster and the custom design chip through a plurality of PCIe terminal devices, and establishing connection between the second PCIe bridge device and each first PCIe bridge device; s3, responding to data interaction in the cluster, establishing a first routing channel between two servers through the first PCIe bridge device, and carrying out data interaction through the first routing channel; and S4, responding to data interaction between the clusters, establishing a second routing channel between the two clusters through the first PCIe bridge device and the second PCIe bridge device, and performing data interaction through the second routing channel.

In some embodiments, the steps further comprise: and forming a basic PCIe system by each PCIe terminal device and the corresponding server, independently initializing each basic PCIe system, distributing corresponding ID and address space and binding with the corresponding custom design chip for use.

In some embodiments, the steps further comprise: and forming a tree PCIe system by each PCIe terminal device, the first PCIe bridge device, the second PCIe bridge device and the ROOT module with the ROOT port, and distributing independent and different buses, device IDs and address spaces for each class of bridge devices and all PCIe terminal devices of each tree PCIe system.

Fig. 6 is a schematic hardware structure of an embodiment of the computer device with interconnected multi-chip cluster servers according to the present invention.

Taking the example of the apparatus shown in fig. 6, a processor 301 and a memory 302 are included in the apparatus.

The processor 301 and the memory 302 may be connected by a bus or otherwise, for example in fig. 6.

The memory 302 is used as a non-volatile computer readable storage medium, and may be used to store non-volatile software programs, non-volatile computer executable programs, and modules, such as program instructions/modules corresponding to the method of interconnecting multi-chip cluster servers in the embodiments of the present application. Processor 301 executes various functional applications of the server and data processing, i.e., a method of implementing multi-chip cluster server interconnection, by running non-volatile software programs, instructions, and modules stored in memory 302.

Memory 302 may include a storage program area that may store an operating system, at least one application program required for functionality, and a storage data area; the storage data area may store data created according to the use of the method of multi-chip cluster server interconnection, etc. In addition, memory 302 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 302 may optionally include memory located remotely from processor 301, which may be connected to the local module via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

Computer instructions 303 corresponding to a method of interconnecting one or more multichip cluster servers are stored in memory 302, which when executed by processor 301, perform the method of interconnecting multichip cluster servers in any of the method embodiments described above.

Any one embodiment of a computer device that performs the method for interconnecting multi-chip cluster servers described above may achieve the same or similar effects as any one of the method embodiments described above.

The present invention also provides a computer readable storage medium storing a computer program which when executed by a processor performs a method of interconnecting multi-chip cluster servers.

Fig. 7 is a schematic diagram of an embodiment of a computer storage medium interconnected by the multi-chip cluster server according to the present invention. Taking a computer storage medium as shown in fig. 7 as an example, the computer readable storage medium 401 stores a computer program 402 that when executed by a processor performs the above method.

Finally, it should be noted that, as will be appreciated by those skilled in the art, all or part of the processes in the methods of the embodiments described above may be implemented by a computer program to instruct related hardware, and the program of the method for interconnecting the multi-chip cluster servers may be stored in a computer readable storage medium, where the program may include the processes in the embodiments of the methods described above when executed. The storage medium of the program may be a magnetic disk, an optical disk, a read-only memory (ROM), a random-access memory (RAM), or the like. The computer program embodiments described above may achieve the same or similar effects as any of the method embodiments described above.

The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the disclosed embodiments may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.

It should be understood that as used herein, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.

The foregoing embodiment of the present invention has been disclosed with reference to the number of embodiments for the purpose of description only, and does not represent the advantages or disadvantages of the embodiments.

It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program for instructing relevant hardware, and the program may be stored in a computer readable storage medium, where the storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

Those of ordinary skill in the art will appreciate that: the above discussion of any embodiment is merely exemplary and is not intended to imply that the scope of the disclosure of embodiments of the invention, including the claims, is limited to such examples; combinations of features of the above embodiments or in different embodiments are also possible within the idea of an embodiment of the invention, and many other variations of the different aspects of the embodiments of the invention as described above exist, which are not provided in detail for the sake of brevity. Therefore, any omission, modification, equivalent replacement, improvement, etc. of the embodiments should be included in the protection scope of the embodiments of the present invention.

Claims

1. A method for interconnecting multi-chip cluster servers, comprising the steps of:

setting a custom design chip comprising a plurality of PCIe terminal devices and a first PCIe bridge device, and setting a custom exchange chip comprising a ROOT port function module of PCIe and a second PCIe bridge device;

setting a custom design chip in each server cluster, establishing connection between each SOC server in the server cluster and the custom design chip through a plurality of PCIe terminal devices, and establishing connection between the second PCIe bridge device and each first PCIe bridge device;

Responding to data interaction in the cluster, establishing a first routing channel between two servers through the first PCIe bridge device, and carrying out data interaction through the first routing channel; and

and responding to data interaction between the clusters, establishing a second routing channel between the two clusters through the first PCIe bridge device and the second PCIe bridge device, and performing data interaction through the second routing channel.

2. The method of claim 1, wherein the establishing a first routing channel between two servers through the first PCIe bridge device comprises:

responding to the data writing of the starting server to the destination server, initiating a data reading request according to the appointed memory address of the starting server, and initiating a DMA memory writing request to the first PCIe bridge device;

the first PCIe bridge device routes the DMA write memory request to PCIe terminal device of the destination server according to the configuration information, and the PCIe terminal device of the destination server initiates the DMA write request to the destination server and receives the return information of the destination server; and

and sending the write memory address in the return information to the starting server through the first PCIe bridge device so as to establish a first route channel from the starting server address space to the destination server address space.

3. The method of claim 1, wherein the establishing a first routing channel between two servers through the first PCIe bridge device comprises:

responding to the reading of data from an initial server to a destination server, configuring a DMA (direct memory access) write memory instruction for PCIe (peripheral component interconnect express) terminal equipment of the initial server, and distributing write memory address space, data size and address information of the destination server; and

the first PCIe bridge device routes the DMA read memory request to PCIe terminal device of the destination server according to the configuration information, and the PCIe terminal device of the destination server initiates the DMA read request to the destination server and receives the returned data of the destination server.

4. The method of claim 1, wherein the establishing a second routing channel between two clusters through the first PCIe bridge device and the second PCIe bridge device comprises:

responding to the initial cluster to write data into the target cluster, initiating a read data request according to the appointed memory address of the initial server of the initial cluster, and initiating a DMA write memory request to the first PCIe bridge device of the initial server;

the first PCIe bridge device routes the DMA write memory request to the first PCIe bridge device of the target cluster through the second PCIe bridge device, and the first PCIe bridge device of the target cluster routes the DMA write memory request to PCIe terminal devices of the target server according to the memory space configuration information; and

And receiving the return information of the destination server through the PCIe terminal equipment of the destination server, and routing the write memory address in the return information to the PCIe terminal equipment of the starting server through the second PCIe bridge equipment, the first PCIe bridge equipment of the starting cluster and the first PCIe bridge equipment of the destination cluster.

5. The method of claim 1, wherein the establishing a second routing channel between two clusters through the first PCIe bridge device and the second PCIe bridge device comprises:

responding to the initial cluster to read data from the target cluster, the initial cluster configures a DMA memory writing instruction to the connected PCIe terminal equipment and distributes the address information of the target server of the target cluster and the address space and the data size of the written memory;

and the first PCIe bridge device of the starting server routes the DMA read memory request to the PCIe terminal device of the destination server of the destination cluster through the second PCIe bridge device according to the configuration information, and the PCIe terminal device of the destination server initiates the DMA read request to the destination server and receives the returned data of the destination server.

6. The method according to claim 1, wherein the method further comprises:

And forming a basic PCIe system by each PCIe terminal device and the corresponding server, independently initializing each basic PCIe system, distributing corresponding ID and address space and binding with the corresponding custom design chip for use.

7. The method according to claim 1, wherein the method further comprises:

and forming a tree PCIe system by each PCIe terminal device, the first PCIe bridge device, the second PCIe bridge device and the ROOT module with the ROOT port, and distributing independent and different buses, device IDs and address spaces for each class of bridge devices and all PCIe terminal devices of each tree PCIe system.

8. A system for interconnecting multi-chip cluster servers, comprising:

the device comprises a setting module, a first PCIe bridge device and a second PCIe bridge device, wherein the setting module is configured to set a custom design chip comprising a plurality of PCIe terminal devices and the first PCIe bridge device, and set a custom exchange chip comprising a PCIe ROOT port function module and the second PCIe bridge device;

the connection module is configured to set a custom design chip in each server cluster, establish connection between each SOC server in the server cluster and the custom design chip through a plurality of PCIe terminal devices, and establish connection between the second PCIe bridge device and each first PCIe bridge device;

The first interaction module is configured to respond to the data interaction inside the cluster, establish a first routing channel between two servers through the first PCIe bridge device, and conduct the data interaction through the first routing channel; and

and the second interaction module is configured to respond to data interaction between the clusters, establish a second routing channel between the two clusters through the first PCIe bridge device and the second PCIe bridge device, and perform data interaction through the second routing channel.

9. A computer device, comprising:

at least one processor; and

a memory storing computer instructions executable on the processor, which when executed by the processor, perform the steps of the method of any one of claims 1-7.

10. A computer readable storage medium storing a computer program, characterized in that the computer program when executed by a processor implements the steps of the method of any one of claims 1-7.