JP4117621B2 - Data batch transfer device - Google Patents

Description

The present invention relates to a data batch transfer apparatus , and more particularly, a calculation system or distance transfer that connects a plurality of computers (nodes) to perform calculation while performing high-speed data transfer and enables large-capacity transfer between nodes with one command. The present invention relates to a data batch transfer apparatus for batch transfer of data in a cluster.

In a large-scale computing system spanning a plurality of nodes, parallel processing is performed in order to increase processing speed. A parallel execution program that spans between nodes can be divided into a calculation phase mainly for each node and a transfer phase for synchronous transfer between nodes. It is important to reduce the transfer time of this transfer phase for rapidly increasing computing performance.

In particular, in the field of scientific calculation (for example, various simulations)
(1) Prior to calculation at each node, a large array data is constructed at a certain node, this is divided and assigned to each node, and the data is transferred to each node. (2) Transfer in the transfer phase during the calculation phase (transfer between each node to perform addition operation of data at the boundary of adjacent calculation areas)
Two types of transfers occur. Therefore, it is required to reduce these transfer times.

Conventional techniques in such technical fields are disclosed in various technical documents. A data processing apparatus and a data processing method for interconnecting a plurality of clusters each including a plurality of CPUs (arithmetic processing apparatuses), data transfer apparatuses, and a shared memory connected to a local area network through an inter-cluster network are disclosed ( For example, see Patent Document 1.) In addition, when accessing the entire subarray data in the two-dimensional array data, it is not necessary to issue a data transfer instruction by dividing the total number of transfer elements by the number of subarray data elements , regardless of the number of subarray data elements. An information processing apparatus and an information processing system that improve transfer efficiency during data transfer are disclosed (for example, see Patent Document 2).

JP 2000-322392 A (page 4, FIG. 1) JP-A-11-134310 (page 2-3, FIG. 2)

  The prior art as described above will be described with reference to FIG. As shown in the upper left of FIG. 8, the prior art data (or information) processing apparatus 100 includes a plurality of CPUs (abbreviated as CP in the figure) 110, a memory 120 and an RCU (between nodes) common to the plurality of CPUs 110. A plurality of nodes 0 to n including a transfer control unit 130. The plurality of nodes are configured to be switched and selected by the inter-node switch 140, and inter-node transfer is performed in units of RCUs 130. The image of the memory address of the memory 120 viewed from the RCU 130 at this time is shown in the lower left in FIG.

Next, conventional distance transfer will be described with reference to FIGS.
FIG. 9 is an image diagram of distance transfer. When transferring between memories, the address of each element is changed according to a rule and transferred. In the figure, BL is the local node main memory transfer start address , BR is the remote node main memory transfer start address, FL1 is the first distance transfer element number, FL2 is the second distance transfer block number, and TL is the total transfer element. DL1, local node first distance, DL2 local node second distance, DL3 local node third distance, DR1 remote node first distance, DR2 remote node second distance, DR3 remote node third distance Each is shown.

  FIG. 10 is a principle diagram for explaining the mechanism of distance transfer. An operation is performed on arrays stored at consecutive addresses, and arrays that are regularly scattered (addresses fly) are arranged in another rule. It is corrected and stored in the transfer destination. The state in the transfer source memory and the state in the transfer destination node memory are shown. Here, the start address of the array is displayed as 0x80000.

  FIG. 11 is an explanatory diagram of a method of using distance transfer. In FIG. 9, each piece of information is transfer instruction information. Among these, information on the local node side is transfer source arrangement information, and remote node side information is transfer destination information. FIG. 10 shows a configuration of the array and an arrangement image in the memory. In order to divide a large array and perform calculation at each node, data scattered in the array is collected and transferred with a certain rule, and is collected as one array for calculation. Further, as shown in FIG. 11, not only the above-described division / transfer (see FIG. 11A), but also reverse transfer / integration (see FIG. 11B), compression / enlargement (see FIG. C11). In addition, various uses such as transposition transfer (deformation) (see FIG. 11D) are performed.

  In the conventional example shown in FIG. 8, both the CPU 110 and the RCU 130 can access only the memory 120 of the own node. Under these conditions, in the parallel execution program transfer, contention arbitration occurs at the time of transfer for each node, so that a transfer waiting time to the corresponding node is added to the transfer time. In particular, when a node that performs transfer to a plurality of nodes first waits for transfer due to contention arbitration, the subsequent transfer also waits for transfer together (so-called head blocking phenomenon). It has the problem of increasing.

The present invention has been made in view of the above-described problems of the prior art, and by having a batch transfer mechanism in a cluster, this contention arbitration time can be reduced to one time or less even in a parallel execution program or the like. It is a main object of the present invention to provide a calculation system, that is, a data batch transfer device . Another object of the present invention is to provide a data batch transfer device that is realized as software control without changing the principle of distance transfer, and that can execute a program efficiently and at high speed without performing new complicated control. And

In order to solve the above-described problem, the data batch transfer apparatus according to the present invention employs the following characteristic configuration.

(1) n nodes (computers ) each having a memory common to a plurality of CPUs (where n is an integer of 2 or more), wherein the CPU issues a transfer instruction and addresses are regularly An instruction for rearranging scattered data arrays to another rule and transferring them to a transfer destination, and rearranging and transferring the data arrays according to a memory capacity distance corresponding to the memory size of the memory In a data batch transfer apparatus that issues a distance instruction to indicate, and batch stores (accumulates) and transfers data of each memory of the plurality of nodes to a continuous memory according to the transfer instruction,
An inter-node transfer control unit (RCU) commonly connected to the memories of the plurality of nodes;
The RCU has a data store unit for storing data in the memory of each node, a value obtained by multiplying each real address of the memory of each node by the node number n and the memory capacity distance. It holds as a global address calculated by adding to the real address of the memory of the nth node, and the address of the transfer target data specified by the global address in the transfer instruction is determined by the memory capacity distance. The real address generated by the decomposition and the destination memory are shown with reference to an address conversion table that stores the correspondence relationship of the node to which the real address is to be transmitted by decomposing into the real address An address conversion unit for notifying the data store unit of information,
The data store unit corresponds to the real address and the real address generated by disassembling the global address specified by one distance instruction for the transfer target data collectively stored based on the global address. And a data batch transfer device for transferring between the nodes based on the information indicating the memory.

According to the data batch transfer device of the present invention, the following practical effects can be obtained. In other words, by carrying out batch transfer with the address conversion means, it is possible to reduce the transfer contention arbitration time that occurs for each node transfer within a cluster or for a designated cluster to one time for all node transfers within the cluster. Further, since the software instruction interface is realized without changing, it is possible to bring out the original performance of the hardware without complicating the software control and without complicating the software control of the transfer procedure.

Hereinafter, the configuration and operation of a preferred embodiment of a data batch transfer apparatus according to the present invention will be described in detail with reference to the accompanying drawings.

  First, the present invention can be summarized into three points as will be apparent from the following description. First, conventionally, each node (computer) has a node-to-node transfer control unit (hereinafter referred to as RCU) for a plurality of nodes (hereinafter referred to as a cluster). In addition, the RCU has an address translation mechanism in the inter-node cluster that can be accessed from the RCU using the memory area of each intra-cluster node as a continuous address. In this specification, “cluster” means that the memory of a node in each cluster can be accessed from each CPU in the node, but transfer from one node to another is performed from the CPU of another node via the RCU. Configured so that the data cannot be referred to unless the data is transferred to the memory in the own node, and a plurality of nodes in which each node cannot directly refer to (and update) the memory of the other node are configured. Means things.

  Second, when a transfer instruction (one instruction) is issued from one CPU of a certain node, the address designated by the CPU is converted through the above-described address conversion mechanism, and data is continuously loaded by loading data from each memory. It has a data load mechanism that takes it into the RCU as data. Then, the data collected in one place by this data load function is converted into continuous data in accordance with the transfer instruction of the CPU, and the data is stored (accumulated) in each memory divided into each node. Has a store mechanism.

  Thirdly, a transfer instruction, which is an instruction from the CPU, is specified by a distance instruction (array data regularly scattered on the memory is specified by its address skip rule [placement rule for transfer source array and placement rule for transfer destination array]). Is issued by a batch command between nodes). As a result, it is possible to perform batch transfer of all nodes in the cluster while performing software control of distance transfer between single nodes in which a jump distance of addresses across nodes in the cluster is added as a new distance.

  FIG. 1 is a block diagram showing the basic configuration of a preferred embodiment of the data batch transfer apparatus of the present invention. The data batch transfer apparatus 10 shown in FIG. 1 includes a plurality of notes (nodes 0 to n) each including a plurality of CPUs 12, a memory 14 provided for each node, and a memory 14 connected to the CPU and all memories of the plurality of nodes. 14 and the RCU 20 connected to 14.

The RCU 20 includes a transfer instruction information notification unit 21, a data load unit 22, a data store unit 23, an address conversion unit 24, and a data storage buffer 25 from the CPU, as shown on the right side of FIG. Here, the transfer instruction information notification unit 21, the data load unit 22, and the data store unit 23 from the CPU are interconnected via a bus. The address conversion unit 24 obtains an output from the transfer instruction information notification unit 21 from the CPU, and outputs it to the data load unit 22 and the data store unit 23. The data storage buffer 25 is connected between the data load unit 22 and the data store unit 23.

  As described above, the data batch transfer apparatus 10 includes the RCU 20 connected to a plurality of nodes. The RCU 20 includes the address conversion unit 24 and is connected to one node among the plurality of CPUs 12 connected to the plurality of nodes. Transfer between batch nodes by specifying transfer instructions from the underlying CPU (array data regularly scattered in the memory by the address skip rule [placement rule for the transfer source array and placement rule for the transfer destination array]) The transfer instruction information is received by the command. Inside the inter-node transfer control unit 20, transfer source data is fetched in parallel from a plurality of nodes in accordance with transfer instruction information (including transfer source arrangement information and transfer destination arrangement information) from the CPU 12, and each node receives this data. Arranged in the memory 14.

  Next, main functions of each unit constituting the data batch transfer apparatus 10 described above will be described. The address conversion unit 24 disassembles the data into continuous data chunks based on data dotted rules such as the address start position and address jump width indicated in the transfer source information of the transfer instruction information from the CPU 12. Further, it is uniquely converted by an address conversion table (see FIG. 5 described later) indicating which data is actually brought from which address of which node, and this is notified. Similarly, which data is actually written to which address of which node based on the data interspersed rules such as the address start position and address jump width indicated in the transfer destination information of the transfer instruction information from the CPU 12 It has a function to convert uniquely. As a result, the address conversion unit 24 specifies the node that actually accesses the memory and its memory address (for each block), and transmits this information to the data load unit 22.

  The data load unit 22 accesses the specified memory address (start address) of the specified node, and loads the data for each block at once. The data load unit 22 holds the order of addresses converted by the address conversion unit 24 when loading, and stores data returned asynchronously from each node in a data storage buffer so that the requested order of addresses is guaranteed. 25 is instructed. Specifically, a sequence ID for order is provided, and this is guaranteed by using this as a write address of the data storage buffer 25. The data store unit 23 retrieves data from the data storage buffer 25, and designates the address (start) designated by the designated node according to the transfer destination node and the transfer destination address converted by the address conversion unit 24 based on the notified transfer destination arrangement information. Store the data for each block at the same time.

  In this way, in the data batch transfer method and apparatus according to the present invention, the address conversion unit 24, the data load unit 22 and the data store unit 23 which process an array scattered in a plurality of nodes by a distance instruction for each cluster. Have. Therefore, the contention arbitration that occurs in the node unit transfer becomes one time in the cluster unit, so the contention arbitration time can be reduced, and the transfer is realized based on the distance transfer command, so that the batch control can be performed without complicating the SW control. realizable.

  Next, FIG. 2 shows a block diagram of a cluster configuration in which a total of four nodes of node 0 to node 3 as a specific example of FIG. 1 are bundled. In the specific example shown in FIG. 2, the RCU (internode transfer control unit) 20 is connected to a plurality of nodes (node 0 to node 3). Each node includes a memory 14 and one or more CPUs 12. The RCU 20 transmits / receives data to / from each memory 14 by connecting to a plurality of nodes. Each node notifies the RCU 20 of information from the CPU 12 through the memory 14 as a route. Here, the RCU 20 includes a transfer instruction information notification unit 21 from the CPU, a data load unit 22, a data store unit 23, address conversion units 24a and 24b, data storage buffers 25a and 25b, address instruction information buffers 26a and 26b, and between clusters. A data sending unit 27 and an intercluster data receiving unit 28 are included.

  CPU 12 that resides in one of a plurality of nodes transfers a transfer command (a batch transfer command (see FIG. 3) by specifying an address skip rule for array data regularly scattered in memory). Receive instruction information. The transfer instruction information notification unit 21 from the CPU 12 retains (queues) a plurality of transfer instructions and, from time to time, from the oldest instruction information, transfers the transfer source data arrangement information to the address conversion unit 24 and transfers the transfer destination data arrangement information to the address. The instruction information buffer 26 is notified.

  The address conversion unit 24 loads data in advance and has an address conversion table (see FIG. 5) for holding the data. The transfer unit data arrangement information notified from the transfer instruction information notification unit 21 from the CPU 12 is stored in the address conversion unit 24. It is broken down into addresses for each node and data block (referred to as a block), and this is notified to the data load unit 22. The data load unit 22 loads data from the designated address in the memory 14 of each node for each block, and stores the data in the data accumulation buffer 25 in the order in which the addresses are notified from the address conversion unit 24.

  The inter-cluster data sending unit 27 performs contention arbitration of data transfer between clusters and controls data transfer. If data transfer is possible, each information and data is taken out from the address indication information buffer 26 and the data storage buffer 25 and transferred. In the case of transfer from the own cluster to the own cluster in the cluster, the data is transmitted to the inter-cluster data receiving unit 28. The intercluster data receiving unit 28 stores the transfer destination data arrangement information and data in the address instruction information buffer 26 and the data storage buffer 25. The address instruction information buffer 26 notifies the address conversion unit 24 of transfer destination data arrangement information. The address conversion unit 24b has an address conversion table (refer to FIG. 5) which loads data therein in advance and holds the data, and transfers the transfer destination data arrangement information notified from the address instruction information buffer 26 to the node and the data Are divided into addresses for each block (block) and notified to the data store unit 23. The data store unit 23 stores data at a designated address in the memory 14 of each node for each block.

The configuration of the embodiment of the present invention and the function of each unit have been described in detail. However, the CPU 12, the memory 14, and the intercluster switch 30 in FIG. 2 are well known to those skilled in the art and are not directly related to the present invention. The detailed configuration is omitted. In the above-described embodiment, there may be a single cluster or a plurality of clusters. Further, the number of CPUs 12 is not particularly limited. The address conversion table (FIG. 5 ) is a specific example in the case of a page size of 64 MB, a main storage of 1 TB, and the number of nodes in a cluster of 4, but these are only examples, and there is no particular limitation on these numbers. The page management of the main memory is well known to those skilled in the art and is not directly related to the present invention, so that the detailed configuration is omitted.

  Next, the principle of the new distance transfer according to the present invention will be described with reference to FIG. In the conventional distance transfer, the size of the memory in the node is defined as one distance, and an image in which a part of one large memory is allocated to one node for each designated distance (ie, simultaneous access to each node) Transfer data in batches. In FIG. 3, distance access is performed by defining distance across nodes = in-node memory size (assuming 1 TB). In this example, data distributed to a plurality of nodes is integrated into one node. Here, Sub and A are described as one-dimensional arrays for simplicity.

  FIG. 5 is an explanatory diagram of the address conversion unit 24 in FIG. As shown in FIG. 5A, the address conversion unit 24 includes a conventional structure of a distance address decomposition circuit 241 including a plurality of adders and an address conversion table 242 provided on the output side thereof. FIG. 5B is a description of a specific example of the address conversion table 242.

  Next, regarding the data transfer in FIG. 4, the operation of the system in FIG. 2 will be described with reference to timing charts shown in FIGS. First, the address table is written to the address conversion unit 24 when the node system is started up. This is a value set by the CPU 12 or the like in the form of data writing to a part of the RCU 20. In this way, it is set in advance prior to actual transfer. Thereafter, at the timing (1), the CPU 12 issues a distance transfer instruction. This transfer instruction notifies the memory 14 of transfer instruction information. The memory 14 passes this to the RCU 20. Next, the transfer instruction information arrives at the transfer instruction information notification unit 21 from the CPU in the RCU 20 at the timing (2). In FIG. 4, the array portion adjacent to Asb_n in the subspace (Asub_n + 1, etc.) of array A adjacent to Asb_n is transferred to the adjacent nodes all at once with 1 to 3 instructions (3 instructions for a three-dimensional array). Can do.

  In the transfer instruction information notification unit 21 from the CPU, the transfer source data arrangement information in the transfer instruction information is transferred to the address conversion unit 24 from the oldest instruction information as needed while holding (queueing) a plurality of transfer instructions. The data arrangement information is notified to the address instruction information buffer 26 at the timing (3). The address conversion unit 24 decomposes the transfer source data arrangement information notified from the transfer instruction information notification unit 21 from the CPU into addresses for each data block (block) by operating an address decomposition circuit in distance transfer. (Timing 4 to 7), this is converted into the port number of the RCU 20 and the memory address and data length of the memory access request to be sent out by converting this with the address conversion table. After the conversion, this is notified to the data load unit 22 (timing 8 to 11).

  The data load unit 22 outputs to the port designated for each block (timing 12 to 15). Data returned at each timing (data from the designated address in the memory of each node) is loaded (timing 16 to 19). Data is stored in the data storage buffer 25a in the order in which the addresses are notified from the address conversion unit 24 (timing 20). The data storage buffer 25 determines whether or not all data has been received based on the number of loaded data.

  Next, when all the data is collected in the data storage buffer 25a, a transfer enable notification is issued to the inter-cluster data sending unit 27 (timing 21). As a result, the inter-cluster data sending unit 27 performs contention arbitration for inter-cluster data transfer and controls data transfer (timing 22). If data transfer is possible, each information and data is taken out from the address indication information buffer 26a and the data storage buffer 25a and transferred (timing 23). In the case of transfer from the own cluster to the own cluster in the cluster, the data is transmitted to the inter-cluster data receiving unit 28 (timing 24).

  The inter-cluster data receiving unit 28 stores the transfer destination data arrangement information and data in the address instruction information buffer 26b and the data storage buffer 25b (timing 25). The address instruction information buffer 26b notifies the address conversion unit 24b of the transfer destination data arrangement information (timing 26). The address conversion unit 24b decomposes the transfer destination data arrangement information notified from the address instruction information buffer 26b into addresses for each data block (block) by operating an address decomposition circuit in distance transfer (timing 27 to 30). ). Then, by converting this using the address conversion table, the port number of the RCU 20 and the memory address and data length of the memory access request to be transmitted are converted. After this conversion, this is notified to the data store unit 23. The data store unit 23 stores data in the designated address in the memory 14 of each node for each block (timing 31 to 34). As a result, as shown in FIG. 4, transfer data from each node is transferred from a plurality of nodes to a plurality of nodes at once.

Next, a second embodiment of the present invention will be described with reference to FIG. This basic configuration is as described above with reference to FIG. 2, but further devised for the notification of the distance command. As an example, the embodiment described up to now takes the form of transferring data to the destination cluster (including its own cluster) from essentially the source cluster starts transferring the notification from the CPU of the source cluster. 7, the transfer instruction information notification unit 21 of the CPU has a request transfer means and the request receiving means for notifying a request from another cluster of CPU, when the request receiving means has received the transfer instruction information, which CPU Is notified to the transfer instruction information notification unit 21. Thereafter, the processing from the transfer instruction information notification unit 21 from the CPU is the same as in the first embodiment described above.

  As described above, in the second embodiment, batch transfer within a cluster is possible with respect to other clusters. Therefore, in a system composed of a plurality of clusters, from one CPU of one node that supervises operation to all clusters. The execution time can be further shortened by performing batch transfer processing in the cluster in parallel.

  FIG. 8 shows the data batch transfer device 10 according to the present invention and the conventional data batch transfer device 100 together with the image of the memory address as seen from the RCU side. According to the data batch transfer device 10 according to the present invention shown on the right side in FIG. 8, a common RCU 20 is used for a plurality of memories 14 of a plurality of nodes each including a plurality of CPUs 12 and memories 14, and the inter-cluster switch 30 is connected from this RCU 20. Connected to other clusters via Therefore, it can be seen that the memory address viewed from the RCU 20 by making the cluster global address can collect data of scattered addresses according to a certain rule and perform batch transfer once.

  The configuration and operation of the preferred embodiment of the present invention have been described in detail above. However, it should be noted that such examples are merely illustrative of the invention and do not limit the invention in any way. Those skilled in the art will readily understand that various modifications and changes can be made according to a specific application without departing from the gist of the present invention.

It is a block diagram which shows the basic composition of 1st Example of the data batch transfer apparatus by this invention. It is a block diagram which shows the specific example of the data batch transfer apparatus shown in FIG. It is a conceptual diagram explaining the new distance transfer by this invention. It is explanatory drawing of the data transfer of the adjacent area | region in the transfer phase by this invention. FIG. 3 is a block diagram illustrating an internal configuration of an example of an address conversion unit in FIG. 2. It is explanatory drawing of the dress conversion table in FIG. 3 is a timing chart for explaining the operation of the present invention. 3 is a timing chart for explaining the operation of the present invention. It is explanatory drawing of 2nd Example of this invention. It is explanatory drawing which contrasts the prior art example (left side) of a data batch transfer apparatus, and this invention (right side). It is a figure explaining the specification of the conventional distance transfer instruction. It is explanatory drawing of the mechanism of the distance transfer in the case of a division | segmentation, and a calculation. It is explanatory drawing of the main usage method of distance transfer.

Explanation of symbols

10 Data batch transfer device 12 CPU
14 Memory 20 RCU (Internode Transfer Control Unit)
21 Transfer instruction information notification unit 22 from CPU CPU 22 Data load unit 23 Data store unit 24 Address conversion unit 25 Data storage buffer 26 Address instruction information buffer 27 Intercluster data transmission unit 28 Intercluster data reception unit 30 Intercluster switch

Claims (1)

N nodes (computers) each having a memory common to a plurality of CPUs, where the CPU gives a transfer instruction and a data array in which addresses are regularly scattered An instruction that instructs to transfer to the transfer destination after rearranging to a rule, and is issued by a distance instruction that instructs to transfer the data array by rearranging and transferring according to the memory capacity distance corresponding to the memory size of the memory, In a data batch transfer device that batch-stores (accumulates) and transfers data of each memory of the plurality of nodes to a continuous memory according to a transfer instruction,
An inter-node transfer control unit (RCU) commonly connected to the memories of the plurality of nodes;
The RCU has a data store unit for storing data in the memory of each node, a value obtained by multiplying each real address of the memory of each node by the node number n and the memory capacity distance. It holds as a global address calculated by adding to the real address of the memory of the nth node, and the address of the transfer target data specified by the global address in the transfer instruction is determined by the memory capacity distance. The real address generated by the decomposition and the destination memory are shown with reference to an address conversion table that stores the correspondence relationship of the node to which the real address is to be transmitted by decomposing into the real address An address conversion unit for notifying the data store unit of information,
The data store unit corresponds to the real address and the real address generated by disassembling the global address specified by one distance instruction for the transfer target data collectively stored based on the global address. And transferring data between the nodes based on the information indicating the memory.

Images