WO2012119369A1

WO2012119369A1 - Message processing method, device and system based on cc-numa

Info

Publication number: WO2012119369A1
Application number: PCT/CN2011/077898
Authority: WO
Inventors: 程永波; 贺成洪; 兰可嘉
Original assignee: 华为技术有限公司
Priority date: 2011-08-02
Filing date: 2011-08-02
Publication date: 2012-09-13
Also published as: CN102318275A; CN102318275B

Abstract

Disclosed are a message processing method, device and system based on CC-NUMA. A certain node controller in the present node performs address resolution on a message sent by another node, and if the destination address of the message is the present node and the catalogue of the address space corresponding to the message is maintained by the node controller, it determines a processor to which the address space corresponding to the message belongs; if the link of a quick path interconnect bus between the node controller and the processor is normal, then it sends the message to the processor via the quick path interconnect bus, so that the processor processes the message; if the link of the quick path interconnect bus between the node controller and the processor has failed, or the catalogue of the address space corresponding to the message is not maintained by the node controller, then it forwards the message to another node controller in the present node and this other node controller processes the message. The method can improve the efficiency and reliability of accessing system resources.

Description

Message processing method, device and system based on CC-NUMA

TECHNICAL FIELD The present invention relates to the field of non-uniform storage access technologies, and in particular, to a CC-NUMA-based message processing method, apparatus, and system. Background technique

Cache Coherence Non-uniform Memory Access (CC-NUMA) is an important system structure currently used in large-scale parallel computer design. In the CC-NUMA system, each node is composed of a node controller and a plurality of central processing unit CPUs, each node is interconnected through a network, and each central processing unit CPU can access both local memory resources and other nodes in the entire system. On the resource, the system is called a "non-uniform" access system because each central processor accesses local memory resources faster than accessing other nodes' memory resources.

In a CC-NUMA system, there is a node controller in each node, and the node controller can complete distributed memory sharing and cache consistency maintenance. Referring to FIG. 1 , which is a basic structural diagram of a node in the prior art, it can be seen that the node controller in the node has two processor CPUs, and each processor passes through a Quick Path Interconnect (QPI) bus and the node. The controller is connected; the node controller is provided with a network interface (NI, Network Interface), and each node in the system is extended by the network interface NI interconnection, so that the memory resources in the entire system are shared.

When the node controller in the node receives the packet sent by the other node controller through the network interface, the node controller performs address resolution on the received packet, and sends the packet according to the parsed packet address. The corresponding processor processes to complete access to its memory resource data. Similarly, when the processor needs to access the remote resource, the processor can send the message to the node controller through the QPI bus, and the node controller queries the routing table according to the destination node address of the packet, and passes the network interface corresponding to the destination node. The message is sent to the next hop node, and finally the message is sent to the destination node to complete the access of the resource.

In order to complete the sharing of resources within the system, the node controller needs to maintain the address space within the entire node. The directory, and the routing of the packets accessing the two central processors, the system resource access efficiency is low. At the same time, when the QPI bus connection between the processor and the node controller fails, or the node controller itself fails, other nodes may not be able to send a message to the node, cannot access the resources in the node, or cause the entire node to report. The processing of the file is interrupted, which in turn affects the resource access of the entire system, and the reliability of system resource access is low.

Summary of the invention

In view of this, the present invention provides a CC-NUMA-based message processing method, apparatus, and system to improve resource access efficiency and reliability.

To achieve the above object, the present invention provides the following technical solutions as follows:

A packet processing method based on CC-NUMA, in which two node controllers are configured in a node, and each node controller maintains a directory of its corresponding address space, and the method includes:

Receiving a packet sent by another node, and performing address resolution on the packet;

When the destination address of the message is the node, it is determined whether the directory of the address space corresponding to the message is maintained by the node controller;

If yes, determining, by the processor to which the address space corresponding to the message belongs, sending the message to the processor through a fast channel interconnect bus connected to the processor, so that the processor is The message is processed;

If not, forwarding the packet to another node controller in the node, so that the another node controller determines a processor to which the address space corresponding to the packet belongs, and sends the packet The processing is performed on the processor corresponding to the file.

When the destination address of the packet is the node, determining whether the address space corresponding to the packet is in the address space maintained by the packet;

If yes, determining, by the processor to which the address space corresponding to the message belongs, when the link of the fast channel interconnection bus between the node controller and the processor fails, the packet and the local node are controlled. a directory maintained by the device and containing at least an address space corresponding to the message, sent to another node controller, so that the another node controller forwards the message to the processor, and maintains the The directory of the address space corresponding to the text.

A packet processing method based on CC-NUMA, in which two node controllers are configured in a node, and each node controller maintains a directory of its corresponding address space, including:

Obtaining fault information of another node controller;

Obtaining, by broadcast snooping, a directory of an address space maintained by the another node controller; receiving a packet sent by another node, and performing address resolution on the packet;

When the destination address of the message is the node, the processor to which the address space corresponding to the message is located is determined, and the message is sent to the processor.

The present invention also provides a CC-NUMA-based message processing apparatus, in which two node controllers are configured in a node, and each node controller maintains a directory of its corresponding address space, and the message processing apparatus includes:

a message receiving unit, configured to receive a packet sent by another node, and perform address resolution on the packet; and an address analyzing unit, configured to determine, when the destination address of the packet is a node, Whether the directory of the address space is maintained by the local node controller, and if so, the operation of the processor address determining unit is performed;

a processor address determining unit, configured to determine a processor to which the address space corresponding to the text belongs;

And a message sending unit, configured to send the message to the processor through a fast channel interconnect bus connected to the processor, so that the processor processes the “3⁄4 text”.

A CC-NUMA-based message processing device is configured with two node controllers in a node, and each node controller maintains a directory of its corresponding address space, and the message processing device includes: a link failure acquiring unit , for obtaining a link connection failure of the fast channel interconnection bus in the system;

a message receiving unit, configured to receive a message sent by another node, and perform address resolution on the message; and an address analyzing unit, configured to determine, when the destination address of the text is a node, the “3⁄4 text corresponding to the message Whether the directory of the address space is maintained by the local node controller, and if so, the operation of the processor address determining unit is performed;

a processor address determining unit, configured to determine a processing crying port to which the address space corresponding to the text belongs. a fault processing unit, configured to: when the link between the local node controller and the processor is faulty, the message and a directory maintained by the local node controller and including at least an address space corresponding to the packet, Sending to another node controller, so that the other node controller forwards the message to the processor and maintains a directory of the address space corresponding to the text.

A CC-NUMA-based message processing device is configured with two node controllers in a node, and each node controller maintains a directory of its corresponding address space, and the message processing device includes: a controller fault acquiring unit , used to obtain fault information of another node controller;

An address space data obtaining unit, configured to acquire, by broadcast snooping, a directory of an address space maintained by the another node controller;

a message receiving unit, configured to receive a packet sent by another node, and perform an address resolution analysis unit on the packet, where the packet is determined to be a destination address of the packet, and the packet is determined to be corresponding to the packet. The processor to which the address space belongs;

a message sending unit, configured to send the message to the processor.

A CC-NUMA based message processing system includes two node controllers and at least two processors;

The node controller and the processor are connected by a fast channel interconnect bus;

The two node controllers are connected through a network interface;

The node controller has built-in CC-NUMA-based message processing apparatus as described above. As can be seen from the foregoing technical solution, the embodiment of the present invention discloses a CC-NUMA-based message processing method, apparatus, and system, in which a node controller receives a destination address sent by another node as a node of the node. In the text, it is determined whether the directory of the address space corresponding to the message is maintained by the node controller. If yes, the node controller determines the processor to which the address space corresponding to the message belongs, and is connected to the processor. The fast channel interconnect bus sends the message to the processor; if the address space corresponding to the message does not belong to the address space maintained by the node controller, the node controller forwards the message to another node controller In order for another node controller to process the message, since two node controllers in the node respectively maintain data of a part of the address space, both node controllers can receive the message sent by other nodes, and receive the message. Address resolution of the received message, so that the message is sent to the corresponding processor to complete resource access and improve resource access. speed, This improves system performance.

At the same time, when a QPI link fails, the node controller can obtain the QPI link failure information, when the processor corresponding to the address space corresponding to the received message and the node controller

When the QPI bus fails, the node controller can forward the packet and the directory information of the address space corresponding to the packet to another node controller, and the other node controller processes the packet and sends the packet. To the corresponding processor, communication interruption due to link failure is avoided.

After one node controller in the node fails, the other node controller can obtain the node controller failure information, and obtain the directory information of the address space maintained by the node controller through broadcast interception, and another A node controller processes all the "3" messages sent to the node, avoiding communication interruption of the entire CC-NUMA system due to the failure of the node controller itself in the node. BRIEF DESCRIPTION OF THE DRAWINGS In order to more clearly illustrate the present invention The embodiments or the prior art solutions will be briefly described below, and the drawings used in the description of the prior art will be briefly described. It is obvious that the drawings in the following description are only some embodiments of the present invention. For those skilled in the art, other drawings may be obtained based on these drawings without any creative work.

1 is a schematic diagram of a basic structure of a node in the prior art;

2 is a schematic diagram showing the basic structure of a node in the present invention;

FIG. 3 is a flowchart of an embodiment of a method for processing a message based on CC-NUMA according to an embodiment of the present invention;

FIG. 4 is a flowchart of another embodiment of a CC-NMUA-based packet processing method according to an embodiment of the present invention;

FIG. 5 is a flowchart of another embodiment of a CC-NUMA-based message processing method according to an embodiment of the present invention;

FIG. 6 is a flowchart of another embodiment of a CC-NMUA-based packet processing method according to an embodiment of the present invention;

FIG. 7 is a schematic structural diagram of a CC-NUMA-based message processing apparatus according to an embodiment of the present invention;

FIG. 8 is a CC-NUMA-based message processing apparatus according to an embodiment of the present invention. Schematic;

FIG. 9 is a schematic structural diagram of a CC-NUMA-based message processing apparatus according to an embodiment of the present invention. The technical solutions in the embodiments of the present invention are clearly and completely described in the following with reference to the accompanying drawings in the embodiments of the present invention. It is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. example. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present invention without the creative work are all within the scope of the present invention.

In order to solve the problems that may occur when the CC-NUMA system in the prior art performs resource access, the present invention configures two node controllers (NCs) in one node, and each node controller maintains its corresponding address. The directory of the space. The address space is also called the system address and is used to locate the specific location of the resources in the CC-NUMA system.

In the CC-NUMA system, each node in the system is assigned an address space of a region, and the address space of the node is separately assigned to the processor in the node. For example, suppose that there are two processor CPUs in a node, node 1 in the system allocates 0-1TB of address space, and processor 1 and processor 2 in node 1 respectively allocate 512 GB of address space; Node 2 allocates 1TB-2TB of address space, in which processor 1 and processor 2 of node 2 respectively allocate 512 GB of address space, and the allocation of address space of other nodes in the system is similar.

In the prior art, a node controller is provided in a node, and the node controller needs to maintain a directory of the address space of the entire node. In the present invention, two node controllers are disposed in one node, and FIG. 2 is a schematic diagram of a basic structure of a node in the present invention. Taking a two-processor CPU in a node as an example, the node includes a node controller. NC1, node controller NC2, first processor CPU1 and second processor CPU2, wherein each processor is connected to node controller NC1 and node controller NC2 via first fast channel interconnect bus QPI0 and second fast interconnect QPI1, respectively The node controller NC1 and the node controller NC2 are connected through the network interface NI. Each node controller has several network interfaces NI, and the node controllers in the node can be interconnected with the node controllers in other nodes through the network interface NI. The invention divides the address space of the node into two regions, and each node controller maintains a directory of the address space of one region, for example, the node of the node The address space can be divided into a first address space and a second address space, wherein one node controller maintains a directory of the first address space, and another node controller maintains a directory of the second address space. It should be noted that the first address space may include a part of the address space allocated to the address space of the first processor and a part of the address space allocated to the address space of the second processor, and the second address space is also It is possible to include both a portion of the address space in the address space allocated to the first processor and a portion of the address space in the address space allocated to the second processor.

The directory that maintains the address space refers to the data access situation of the node controller to the address space and the data state of the address space. For example, when a processor in another node needs to access a resource saved on a processor in the node, the processor of the other node sends a resource access request in the form of a packet, and the node controller of the node receives the packet. After that, the processing status of the resource data that the packet requests to access is recorded in the directory of the address space (for example, the data state of the address space requested to be accessed may be a modified state, an exclusive state, a shared state, etc.), and a node that sends a message. Information such as ID and processor ID.

In the present invention, each node is provided with two node controllers, and each node controller is provided with a plurality of network interfaces NI connected to the node controllers of other nodes, compared with the prior art, in the CC-NUMA system. The number of NI interfaces provided by each node for connecting to other nodes is twice the number of NI interfaces provided by each node in the prior art for interconnecting between nodes. Therefore, when each CC-NUMA system Nodes are connected to each other to form a larger topology network. When communication between different nodes in the whole system, the average number of hops of the intermediate nodes is reduced during the process of sending packets from the source node to the destination node, which improves system performance.

Based on the node structure of the present invention, FIG. 3 is a schematic flowchart of an embodiment of a CC-NUMA-based packet processing method according to the present invention. In the present invention, two node controllers and two nodes are configured in one node. The controllers are interconnected through a network interface, and each node controller maintains a directory of its corresponding address space, and the method includes:

Step 301: The first node controller receives a packet sent by another node, and performs address resolution on the packet.

When a processor in another node needs to access a resource in a processor CPU of the node, the processor of the other node may send a message to the node, where the packet includes the address of the destination node to be accessed (destination node ID) ), the address of the destination processor (processor ID), and the address space accessed Data requests, etc. After receiving the packet sent by other nodes through the network interface NI, the node controller in the node generally performs a cyclic redundancy check (CRC), and after receiving the CRC, the received message is received. Perform address resolution, that is, decode the address of the packet to determine whether the destination address of the packet is the local node. If the destination address of the packet is not the local node, the node controller queries the routing table according to the routing table. The packet is forwarded to the node controller in the other node. When the destination address of the packet is the node, the node controller needs to perform corresponding processing, so as to finally send the packet to the node and the packet. The processor requests access to the address space data corresponding to the processor.

Step 302: When the destination address of the packet is the node, the first node controller determines whether the directory of the address space corresponding to the packet is maintained by the first node controller, and if yes, proceeds to step 303; Then, the process proceeds to step 304.

When the first node controller determines that the destination address of the packet is the node according to the parsed address, the first node controller needs to determine the address space corresponding to the packet according to the parsed address, that is, the packet. Whether the address space to be accessed and the directory of the address space to be accessed by the message are maintained by the first node controller.

In order to determine whether the address space corresponding to the message is maintained by the first node controller or the second node controller, the parity bit of the address space may be preset, and the parity bit and the node controller are preset. Correspondence between the two. Specifically, setting the first bit in the address space to the parity bit can be set as needed. For example, the sixth bit in the address space can be set to the parity bit, and set to the parity in the address space. When the parity bit is "0", the directory of the corresponding address space is maintained by the first node controller; when the parity bit in the address space is "Γ, the directory of the corresponding address space is controlled by the second node controller. Maintenance, so the directory of 32 consecutive address spaces in front of the address space is maintained by the first node controller, and the 32 consecutive address spaces are sequentially returned to the directory of 32 consecutive address spaces by the second node controller. Maintenance, and directory maintenance of subsequent address spaces are also maintained by the first node controller or the second node controller in an alternating manner.

Specifically, setting the first bit in the address space to the parity bit can be set as needed. In order to reasonably allocate the directory of the address space maintained by the first node controller and the second node controller, one of the 6th to the 10th bits in the address space may be selected as the parity bit. Of course, you can also select consecutive two bits as the parity bit in the address space. For example, you can put the fifth bit in the address space. Together with the sixth bit as a parity bit, when the fifth and sixth bits in the address space are both "0" or both "1", the directory of the corresponding address space is maintained by the first node controller. When the numbers of the fifth and sixth digits in the address space are different, the directory of the corresponding address space is maintained by the second node controller. Of course, there are other ways to distinguish the directory of the address space maintained by the first node controller and the second node controller, which is not enumerated here.

After performing the address resolution on the received packet, the first node controller may determine the address space corresponding to the packet according to the correspondence between the parity bit of the address space corresponding to the packet and the node controller. Whether the parity bit corresponds to the first node controller, and if so, the directory of the address space corresponding to the message is maintained by the first node controller. For example, after receiving the packet, the first node controller performs address resolution on the packet, and determines that the parity bit (such as the fifth digit in the address) in the address of the data resource that the packet requests to access is "0". The directory of the address space that the message needs to access is maintained by the first node controller, otherwise it is maintained by the second node controller.

Step 303: The first node controller determines, by the processor to which the address space corresponding to the packet belongs, the packet is sent to the processor through a fast channel interconnect bus connected to the processor, so that the processor reports the packet. The text is processed.

When the directory corresponding to the address space requested by the packet is maintained by the first node controller, the first node controller may determine the required access of the packet according to the result of the address resolution of the packet. The processor to which the address space belongs. After determining the processor in which the address space requested to be accessed is located, the first node controller can query the routing table, and the fast channel connected to the processor by the first node controller according to the pre-configured port routing path The interconnect bus QPI sends the message to the processor for the processor to process the message.

Step 304: The first node controller forwards the packet to the second node controller in the local node, so that the second node controller determines the processor to which the address space corresponding to the packet belongs, and the packet is sent. The processor corresponding to the packet is sent to perform packet processing.

If the directory corresponding to the address space that the message needs to access is not maintained by the first node controller, the first node controller passes the network interface NI between the first node controller and the second node controller. The message is forwarded to the second node controller. After receiving the packet, the second node controller performs address resolution on the packet, and determines which processor in the node is required to access the address space, and the second node controller is based on the Configured port path through the second node controller A fast channel interconnect bus with the processor forwards the message to the processor, and the processor processes the message.

In the method of this embodiment, after receiving a packet sent by another node, a node controller in the node performs address resolution on the packet, and the directory corresponding to the address space of the packet is maintained by the node controller. In the address space, the node controller further determines the processor to which the address space corresponding to the message belongs, and sends the message to the processor through the fast channel interconnect bus connected to the processor; if the message corresponds to The directory of the address space is not maintained by the node controller, and the node controller forwards the message to another node controller in the node, so that the other node controller processes the message. Since the directory of the address space in the node can be divided into two parts, each node controller maintains data of a part of the address space, so that other nodes in the CC-NUMA system need to access resources in the processor of the node, according to the routing path. Selecting which node controller in the node to send the message, the two node controllers in the node can receive different message requests, and respectively perform address resolution of the 3⁄4 text, and maintain the directory of the corresponding address space. And performing the operation of sending the message to the corresponding processor, because the two node controllers work simultaneously to complete the resource access, and the directory of the address space maintained by each node controller is reduced compared with the prior art, The packet processing speed of the node controller is increased, and the system resource access speed is increased, thereby improving the system performance. In the step 303, the first node controller sends the packet to the packet corresponding to the packet. After the processor in which the address space is located, the processor depends on the address space that the message needs to access. The packet is processed according to the resource, and the processed packet is sent to the first node controller, so that the first node controller returns the processed packet to the source node that sends the packet, that is, sends the resource. Accessing the requested node. Similarly, after the second node controller sends the message to the processor where the address space corresponding to the message is located, the processor also returns the processed message to the second node controller, The two-node controller can query the routing table and select the network interface NI to return the processed message to the source node that initiated the resource access request.

FIG. 4 is a flowchart of another embodiment of a CC-NUMA-based message processing method according to an embodiment of the present invention. The node of the present invention is configured with two node controllers, and each node controller is respectively configured. The directory of the corresponding address space is maintained. The method in this embodiment includes:

Step 401: The first node controller receives a packet sent by another node, and performs an address on the packet. Analysis.

Step 402: When the destination address of the packet is the node, the first node controller determines whether the directory of the address space corresponding to the packet is maintained by the first node controller, and if yes, proceeds to step 403; Then, proceed to step 404.

Step 403: The first node controller determines, by the processor to which the address space corresponding to the packet belongs, the packet to be sent to the processor through a fast channel interconnect bus connected to the processor.

Step 404: The first node controller forwards the packet to the second node controller, and the second node controller receives the packet, and performs address resolution on the packet to determine a processor to which the address space corresponding to the packet belongs. The directory of the address space corresponding to the message is maintained, and the message is sent to the processor.

Step 405: After receiving the packet sent by the first node controller or the second node controller, the processor processes the packet, and determines whether the directory of the address space corresponding to the packet is maintained by the first node controller. If yes, the processor sends the processed message to the first node controller, so that the first node controller returns the processed message to the source node; if not, the processor processes the processed message. The text is sent to the second node controller, so that the second processor returns the processed message to the source node.

After receiving the packet, the processor processes the packet according to the data of the address space requested by the packet. Since the node controller needs to record information such as the data state of the address space accessed by the packet, the processing is performed. The subsequent message still needs to be returned to the corresponding node controller, so that the node controller maintains the directory of the address space requested by the message. Therefore, after the processor finishes processing the packet, it needs to determine which node controller maintains the directory of the address space corresponding to the packet, and returns the processed packet to the node controller. Finally, the node controller returns the message after the processor to the source node that sent the message.

It should be noted that, after the node controller receives the received message, it is required to determine whether the directory of the address space corresponding to the packet is maintained by the local node controller, and if yes, the local node controller sends the message to the node. The processor corresponding to the address space of the message, therefore, the processor can directly return the message to the node controller in the node to send the message to the node, that is, when the first node controller sends the message After going to a processor in the node, the processor processes the "3⁄4 text" and returns the processed message to the first node controller; when the second node controller sends the message to the node After a processor in the processor, the processor processes the packet and returns the processed packet to the second node controller. Step 406: After receiving the processed text returned by the processor, the first node controller or the second node controller updates the directory of the address space corresponding to the processed message, and the processed message is processed. Returns to the source node that sent the message.

It should be noted that, in the present invention, when the node controller receives the packet sent by the other node, it determines whether the directory of the address space corresponding to the packet is maintained by the node, and if so, the node controller needs to update the packet. The directory of the address space corresponding to the packet, for example, the source node that sends the request for the message, and the like. When the processor that the packet requests to access returns the processed packet, the node controller needs to request the packet. The data status is recorded, etc., so when the node controller determines that the directory of the address space corresponding to the message sent by the other node is maintained by the local node controller, and the node controller returns the processed message to the source. Before the node, the node controller needs to maintain the directory of the address space corresponding to the packet.

The above two embodiments are described by taking the packet that the other node requests the resource access of the other node as an example, and the process of the node controller and the processor after receiving the packet in the node. When a processor in the node needs to access resources on a processor in another node, the processor in the node may also generate a resource access request packet, query the routing table, and select an optimal routing path. The optimal routing path determines whether the generated packet is sent to the first node controller or the second node controller, and after the first node controller or the second node controller receives the packet sent by a processor in the node, It also queries its own routing table to determine which externally connected NI interface in the node controller forwards the message. This process is the same as the process in which the processor of the local node sends the packets of the resources in the processor of the other node in the prior art, and details are not described herein again. In the CC-NUMA system, the fast-path interconnect bus QPI between a node controller and a processor in a node may be faulty. In the prior art, when the node controller and a processor are connected by a fast channel, If the bus QPI fails, the link fails, and the message transmission processing cannot be performed, and the communication that causes the resource access is interrupted.

In order to avoid the communication interruption caused by the QPI link failure, FIG. 5 is a schematic flowchart diagram of another embodiment of a CC-NUMA-based packet processing method according to the present invention. The embodiment is applied to a node controller and a processor. In the case where the fast channel interconnect bus fails, as in the above embodiment, two node controllers are configured in the node, and each node controller maintains its corresponding address. The directory of the space. The text processing method of this embodiment includes:

Step 501: The first node controller receives the packet sent by the other node, and performs address resolution on the packet. Step 502: When the first node controller determines that the destination address of the packet is the node and the address space corresponding to the packet When the address space maintained by the first node controller is within the address space, the processor to which the address space corresponding to the message is located is determined.

The operations of the steps 501 and 502 are the same as the corresponding operations described in the foregoing embodiments, and details are not described herein again.

Step 503: If the link of the fast path interconnection bus between the first node controller and the processor fails, the first node controller corresponding to the packet and the first node controller, including at least the packet The directory of the address space is sent to the second node controller, so that the second node controller maintains the directory of the address space corresponding to the message, and forwards the message to the processor to which the address space corresponding to the message belongs.

In order to determine the link failure, the node controller in the node needs to obtain the link failure of the fast channel interconnection bus QPI between the node controller and the processor. When any of the nodes in the node and the fast interconnect channel of the processor fail, both processors in the node can obtain information about the link failure.

When the fast-path interconnect bus in the node fails, the interrupt source of the fault reports the interrupt fault information to the processor, and the processor notifies the node controller of the fast-path interconnect bus to the node controller in the node. The controller receives link fault information of the fast channel interconnect bus sent by the processor. For example, when the fast channel interconnection bus between the first node controller and the certain processor in the node fails, the message data sent by the first node controller received by the processor generates an error, when the generated error data When the number of packets exceeds a preset value, the processor determines that the fast path interconnection bus between the first node controller and the processor is faulty, and the processor notifies the first node controller and the second link of the detected link failure. Node controller.

Of course, when there is a problem with the link failure connected to the node controller, the node controller can also detect the QPI link failure connected thereto, and report the fault to a processor in the node, the processor links the link. The failure notifies other processors and another node controller. It is also possible to notify the node controller of the fast-path interconnect bus link that the link failure of the fast-path interconnect bus link is faulty to another A node controller.

When the link between the first node controller and the processor fails, the processor or the first node controller invokes the basic input/output system BIOS program, performs port routing path configuration, and changes the first node controller to send the message to The port routing path of the processor changes the path between the first node controller and the processor to a routing path through the network interface NI between the first node controller and the second node controller.

For example, when the fast path interconnection bus between the first node controller and the first processor in the node fails, the routing path between the first node controller and the first processor is reconfigured, when the first node controls After receiving the packet, the device determines that the directory of the address space corresponding to the packet is maintained by itself and the packet is sent to the first processor, because the fast channel is interconnected between the first node controller and the first processor at this time. If the bus fails, the first node controller executes an interrupt handler, and according to the reconfigured port routing path, through the network interface NI between the second node controller, the message and the first node controller maintain at least A directory containing the address space corresponding to the message is sent to the second node controller.

After the second node controller receives the packet from the first node controller, the second node controller also needs to perform address resolution on the packet to determine which processor in the node belongs to the address space of the packet, and The message is sent to the processor through the fast channel interconnect bus connected to the processor, so that the processor processes the message. Of course, after the processor finishes processing the packet, the processed packet is returned to the second node controller. During the processing of the entire packet, the second node controller will catalog the address space corresponding to the packet. Perform maintenance, and finally return the processed message returned by the processor to the source node that sent the data request message.

It should be noted that when the link of the fast path interconnection bus between the first node controller and the processor fails, the first node controller may also send the message only to the second node controller, the second node. The controller performs address resolution on the packet, and sends the packet to the processor corresponding to the requested access. When the second node controller needs to perform read/write modification or write back to the directory content of the address space of the packet, the request for the directory modification is sent to the first node control through the NI interface with the first node controller. And the first node controller maintains the directory of the address space corresponding to the message.

In this embodiment, when a certain QPI link in the node fails, the node controller can obtain the QPI link fault information, and when the first node controller receives the packet, determines the address corresponding to the packet. The processor to which the space belongs, when the QPI bus between the processor and the first node controller fails, the first node controller may forward the message and the directory information of the address space corresponding to the packet to the second node. The controller performs address resolution on the packet by the second node controller, determines which processor in the node belongs to the address space corresponding to the packet, and uses the fast channel interconnection bus connected to the processor to report the packet. The text is sent to the processor, so even if a QPI link in the node fails, the new routing path can be selected, and the packet is sent to the corresponding processor, thereby avoiding the communication interruption caused by the link failure. . In the prior art, when the node controller in the node fails, the node controller cannot establish a directory of the address space. For example, if the memory of the address space directory in the node controller fails, the node controller cannot reacquire. The directory of the address space. In this case, other nodes may not be able to access the data resources saved in the processor of the node, and the node cannot receive, process, or forward the packets sent by other nodes, thereby affecting the resource access of the entire CC-NUMA system. In order to solve this problem, referring to FIG. 6, FIG. 6 is a schematic flowchart diagram of another embodiment of a CC-NUMA-based packet processing method according to the present invention. In this embodiment, two node controllers are configured in a node, and each The node controller maintains a directory of its corresponding address space. In this embodiment, when a node controller in a node is faulty, the method in this embodiment includes:

Step 601: The first node controller acquires fault information of the second node controller.

The second node controller's own directory is lost, or the memory of the second node controller's internal storage address space directory fails, causing the second node controller to fail to perform directory maintenance of the address space, and the failure interrupt source will be the second node controller. The fault information is sent to a processor in the node, and the processor notifies the other nodes in the node and the first node controller of the fault information of the second node controller. The processor also sends the fault information of the second node controller to other nodes in the system. After the other node obtains the fault information of the second node controller, the packet routing path sent to the node is re-configured, and the second node controller will not receive the packet information sent by other nodes.

Of course, when the second node controller detects its own fault and cannot establish or maintain a directory of the address space, the second node controller can directly notify the first node controller of its own fault information.

Step 602: The first node controller acquires a directory of an address space maintained by the second node controller by using broadcast snooping. In order to prevent the resource access request of other nodes in the event that the second node controller fails, the first node controller sends the interception to the other nodes in the system by means of broadcast snooping to obtain the maintenance of the first node controller. The data information of the address space, the processor in the other node returns the corresponding data information to the first node controller, so that the first node controller establishes the directory information of the address space originally maintained by the second node controller.

Step 603: When the first node controller receives the packet sent by the other node, the address resolution is performed on the packet.

Step 604: When the destination address of the packet is the node, the first node controller determines the processor to which the address space corresponding to the packet belongs, and sends the packet to the address space corresponding to the packet. processor.

After the second node controller fails, the first node controller acquires the directory of the address space maintained by the original second node controller by means of broadcast interception. At this time, the first node controller maintains the entire node. When the first node controller receives the packet sent by the other node, the first node controller does not need to determine whether the address space corresponding to the packet is maintained by the local node controller, and the first node controller can directly determine the packet. The processor to which the corresponding address space belongs, and the message is sent to the processor through a fast channel interconnect bus connected to the processor.

In this embodiment, after one of the node controllers fails, the other node controller can obtain the node controller failure information, and obtain the data information of the address space maintained by the node controller through broadcast interception, and All the messages sent to the node are processed by another node controller, which avoids the communication interruption of the entire CC-NUMA system due to the failure of the node controller itself in the node. Referring to FIG. 7, an embodiment of a CC-NUMA-based message processing apparatus according to the present invention is configured. In this embodiment, two node controllers are configured in a node, and each node controller maintains its corresponding address. The message processing device may be a node controller or a part of a node controller. The message processing device includes: a message receiving unit 701, an address analyzing unit 702, a processor address determining unit 703, and a message sending. Unit 704.

The message receiving unit 701 is configured to receive a packet sent by another node, and perform address solution on the packet. The address analyzing unit 702 is configured to determine, when the destination address of the packet is the node, whether the directory of the address space corresponding to the packet is maintained by the local node controller, and if yes, perform the operation of the processor address determining unit.

The processor address determining unit 703 is configured to determine a processor to which the address space corresponding to the >3⁄4 text belongs.

The message sending unit 704 is configured to send the message to the processor through a fast channel interconnect bus connected to the processor, so that the processor processes the message.

The operation of the message receiving unit and the address analyzing unit in the node controller may be part of the Rbox module or the Rbox module in the node controller. When the node controller determines that the directory of the address space corresponding to the received message is maintained by the local node controller, the packet may be sent to the address determining unit to perform the operations of the processor determining unit and the message sending unit. The CCM (Cache Coherence Module) may be a part of the CCM module, and the CCM (Cache Coherence Module) may be executed in the node controller. After determining the processor corresponding to the message, the corresponding QPI bus can be selected to send the message to the corresponding processor, and the QPI link layer management module connected to the processor needs to be completed when the message is sent to the corresponding processor. QPIL selects the corresponding QPI bus and sends the message to the corresponding processor.

It should be noted that, in the directory of the address space corresponding to the packet, it is determined that the node controller is protected by a plurality of methods. The address analysis unit 702 may include:

An address analysis sub-unit, configured to determine, according to a correspondence between a parity bit of the preset address space and a node controller, whether a parity bit of the address space corresponding to the packet corresponds to the node controller, and if The directory of the address space corresponding to the message is maintained by the local node controller.

When the node controller receives the packet whose destination address is the local node, and determines that the directory of the address space corresponding to the packet is maintained by the local node controller, the node controller also needs to record and send the packet to access the address space. The address processing device of the present embodiment further includes: an address space maintenance unit, configured to maintain a directory of an address space corresponding to the received message.

Further, after the node controller sends the packet to the processor corresponding to the packet, the processor processes the packet according to the request information of the packet, and returns the processed packet to the node controller. Correspondingly, the message processing apparatus further includes: a message returning unit, configured to receive, returned by the processor The processed packet is sent to the source node corresponding to the packet. FIG. 8 is a schematic structural diagram of another embodiment of a CC-NUMA-based message processing apparatus according to the present invention. The apparatus of this embodiment is applied to a fast channel interconnection bus between a node controller and a processor. In this embodiment, the node is configured with two node controllers in the node, and each node controller maintains its corresponding address space. The packet processing device includes: a link fault obtaining unit 801 and a message receiving unit 802. The address analyzing unit 803, the processor address determining unit 804, and the fault handling unit 805.

The link fault obtaining unit 801 is configured to acquire a link connection fault of the fast channel interconnect bus in the system.

The message receiving unit 802 is configured to receive a packet sent by another node, and perform address resolution on the packet. The address analyzing unit 803 is configured to: when the destination address of the packet is the node, determine whether the directory of the address space corresponding to the packet is maintained by the node controller, and if yes, perform the operation of the processor address determining unit.

The processor address determining unit 804 is configured to determine a processor to which the address space corresponding to the text belongs.

The fault processing unit 805 is configured to: when the link between the local node controller and the processor is faulty, the message and the directory maintained by the local node controller and including at least the address space corresponding to the packet, Sending to another node controller, so that the other node controller forwards the message to the processor, and maintains the directory of the address space corresponding to the "3⁄4".

The link fault obtaining unit 801 includes: a link fault information receiving unit, configured to receive link fault information of a fast channel interconnect bus between the node controller and the processor sent by the processor, according to a manner of obtaining a link fault. .

The link fault detecting unit is configured to detect a link of the fast channel bus connected to the processor, and obtain a link fault of the fast channel interconnect bus.

It should be noted that, after the other node controller receives the packet forwarded by the local node controller, the other node controller also needs to perform address resolution on the packet, and determines that the address space of the packet belongs to the local node. Which processor, and through the fast-channel interconnect bus connected to the processor, sends a message to the processor for the processor to process the message. Of course, the processor completes the message processing. After that, the processed message is returned to the other node controller, and the other node controller maintains the directory of the address space corresponding to the message, and returns the processed message to the sent message. Source node. FIG. 9 is a schematic structural diagram of another embodiment of a CC-NUMA-based message processing apparatus according to the present invention. When the device in this embodiment is applied to a node controller, the embodiment is configured in the node. Two node controllers, each of which maintains its corresponding address space. The message processing apparatus includes: a controller fault acquiring unit 901, an address space data acquiring unit 902, a message receiving unit 903, and an address analyzing unit 904. And a message transmitting unit 905.

The controller fault acquiring unit 901 is configured to acquire fault information of another node controller.

The address space data obtaining unit 902 is configured to obtain, by broadcast snooping, a directory of an address space maintained by the another node controller.

The message receiving unit 903 is configured to receive a message sent by another node, and perform an address de-address analysis unit 904 on the message, where the address address corresponding to the message is determined when the destination address of the message is the node. The processor to which it belongs.

The message sending unit 905 is configured to send the message to the processor to which the address space corresponding to the message belongs.

The controller fault acquiring unit 901 includes: a first fault information receiving unit, configured to receive fault information of the another node controller sent by a processor.

The method includes: a second fault receiving unit, configured to receive fault information sent by the another node controller. In addition, the present invention further provides a CC-NUMA-based message processing system, including: two node controllers, and at least two processors; each node controller is interconnected with each processor through a fast channel. The bus is connected. The two node controllers are connected through a network interface.

Each node controller has built-in CC-NUMA-based message processing apparatus described in the above embodiments of the present invention. The various embodiments in the present specification are described in a progressive manner, and each embodiment focuses on differences from other embodiments, and the same similar parts between the various embodiments may be referred to each other. For For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant part can be referred to the method part.

A person skilled in the art will further appreciate that the elements and algorithm steps of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software or a combination of both, in order to clearly illustrate the hardware and software. Interchangeability, the composition and steps of the various examples have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the solution. A person skilled in the art can use different methods for implementing the described functions for each particular application, but such implementation should not be considered to be beyond the scope of the present invention.

The steps of a method or algorithm described in connection with the embodiments disclosed herein can be implemented directly in hardware, a software module executed by a processor, or a combination of both. The software module can be placed in random access memory (RAM), memory, read only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or technical field. Any other form of storage medium known.

The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments are obvious to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded to the broadest scope of the principles and novel features disclosed herein.

Claims

Rights request

A CC-NUMA-based packet processing method, wherein two node controllers are configured in a node, and each node controller maintains a directory of its corresponding address space, and the method includes:

If yes, determining, by the processor to which the address space corresponding to the message belongs, sending the message to the processor through a fast channel interconnect bus connected to the processor, so that the processor is Processing

If not, forwarding the packet to another node controller in the node, so that the another node controller determines a processor to which the address space corresponding to the packet belongs, and sends the packet The packet processing is performed on the processor corresponding to the packet.

The method according to claim 1, wherein the determining whether the directory of the address space corresponding to the packet is maintained by the node controller comprises:

Determining, according to a correspondence between the parity bit of the preset address space and the node controller, whether the parity bit of the address space corresponding to the packet corresponds to the local node controller, and if yes, the packet The directory of the corresponding address space is maintained by the local node controller.

The method according to claim 1, wherein after determining that the address space corresponding to the message is maintained by the local node controller, the method further includes: maintaining directory information of the address space corresponding to the >3⁄4 text .

The method according to claim 1, wherein after the node controller sends the message to the processor, the method further includes:

Receiving the processed message returned by the processor, and sending the processed message to the source node corresponding to the file.

5. A CC-NUMA-based message processing method, wherein two node controllers are configured in a node, and each node controller maintains a directory of its corresponding address space, and the method includes:

Receiving a packet sent by another node, and performing address resolution on the packet; When the destination address of the packet is the node, determining whether the address space corresponding to the packet is in the address space maintained by the packet;

If yes, determining, by the processor to which the address space corresponding to the message belongs, when the link of the fast channel interconnection bus between the node controller and the processor fails, the packet and the local node are controlled. a directory maintained by the device and containing at least an address space corresponding to the message, sent to another node controller, so that the other node controller forwards the message to the processor, and maintains the The directory of the address space corresponding to the text.

The method according to claim 5, wherein the link failure of the fast channel interconnection bus between the node controller and the processor is obtained, including:

Receiving link fault information of the fast channel interconnection bus between the node controller and the processor sent by the processor.

A link of the fast track bus between the processor is detected to obtain a link failure of the fast track interconnect bus.

The method according to claim 5, wherein the forwarding, by the another node controller, the message to the processor includes:

The another node controller performs address resolution on the packet, determines a processor to which the address space of the packet belongs, and sends the packet to the server through a fast channel interconnection bus connected to the processor. a processor, such that the processor processes the message.

9. A CC-NUMA-based message processing method, wherein two node controllers are configured in a node, and each node controller maintains a directory of its corresponding address space, including: acquiring another node Controller fault information;

The method according to claim 9, wherein the acquiring the fault information of the other node controller comprises: Receiving fault information of the another node controller sent by a processor in the node.

The method according to claim 9, wherein the acquiring the fault information of the other node controller comprises:

Receiving failure information of the another node controller sent by the another node controller.

12. A CC-NUMA-based message processing apparatus, wherein two node controllers are configured in a node, and each node controller maintains a directory of its corresponding address space, and the message processing apparatus includes :

And a message sending unit, configured to send the message to the processor through a fast channel interconnect bus connected to the processor, so that the processor processes the >3⁄4 text.

The device according to claim 12, wherein the address analyzing unit comprises:

An address analysis subunit, configured to determine, according to a correspondence between a parity bit of the preset address space and the node controller, whether the parity bit of the address space corresponding to the packet corresponds to the node controller, if If yes, the directory of the address space corresponding to the >3⁄4 text is maintained by the local node controller.

The device according to claim 12, further comprising: an address space maintenance unit, configured to maintain a directory of the address space corresponding to the text.

The device according to claim 12, further comprising:

And a packet returning unit, configured to receive the processed packet returned by the processor, and send the processed packet to the source node corresponding to the packet.

16. A CC-NUMA-based message processing apparatus, wherein two node controllers are configured in a node, and each node controller maintains a directory of its corresponding address space, and the message processing apparatus includes :

a link fault obtaining unit, configured to acquire a link connection of a fast channel interconnect bus in the system, Barrier

a fault processing unit, configured to: when the link between the local node controller and the processor is faulty, the message and a directory maintained by the local node controller and including at least an address space corresponding to the packet, Sending to another node controller, so that the other node controller forwards the message to the processor and maintains a directory of the address space corresponding to the text.

The device according to claim 16, wherein the link failure acquiring unit comprises:

And a link failure information receiving unit, configured to receive link fault information of the fast channel interconnection bus between the node controller and the processor sent by the processor.

The link fault detecting unit is configured to detect a link of the fast channel bus with the processor, and obtain a link fault of the fast channel interconnect bus.

19. A CC-NUMA-based message processing apparatus, wherein two node controllers are configured in a node, and each node controller maintains a directory of its corresponding address space, and the message processing apparatus includes :

a controller fault acquiring unit, configured to acquire fault information of another node controller;

a message receiving unit, configured to receive a packet sent by another node, and perform an address resolution analysis unit on the packet, where the packet is determined to be a destination address of the packet, and the packet is determined to be corresponding to the packet. The processor to which the address space belongs; a message sending unit, configured to send the message to the processor.

The device according to claim 19, wherein the controller fault acquiring unit comprises:

The first fault information receiving unit is configured to receive fault information of the another node controller sent by a certain processor.

The device according to claim 19, wherein the controller fault acquiring unit comprises: a second fault receiving unit, configured to receive fault information sent by the another node controller.

22. A CC-NUMA based message processing system, comprising: two node controllers; and at least two processors;

The two node controllers are connected through a network interface;

The node controller incorporates the CC-NUMA-based message processing apparatus according to any one of claims 12 to 21.