KR102035843B1 - DATA TRANSMISSION SYSTEM AND METHOD CONSIDERING Non-Uniformed Memory Access - Google Patents

Abstract

A data transfer system for transmitting data loaded in memory buffers of a transmitter to memory buffers of a receiver, wherein each of the transmitter and the receiver is a CPU core to which one of a master thread, a communication thread, and an input / output thread is allocated. CPU sockets including at least one, and memory buffers divided by the number of CPU sockets and disposed in each of the CPU sockets.

Description

DATA TRANSMISSION SYSTEM AND METHOD CONSIDERING NON-UNIFORMED MEMORY ACCESS}

The present invention relates to data transmission techniques.

In the coming big data era, scientists often have to share or transfer large amounts of data to and from labs and computing centers around the world. However, existing data transfer frameworks have limitations that do not reflect the characteristics of parallel file systems (PFS) in laboratories and computing centers.

To solve this problem, the Layout-Aware Data Scheduling (LADS) data transfer framework was designed. The LADS framework improves the efficiency of end-to-end data transfer by allowing multiple threads to work on a per-object basis by transferring data on a per-object basis, and by scheduling data by recognizing the data layout in a parallel file system. Optimized performance of data communication in terabit-network.

However, there is a drawback that the LADS framework does not address some of the problems that may arise in a Non-Uniformed Memory Access (NUMA) architecture. Specifically, end-to-end data transfer can be a bottleneck in storage, CPU, and memory. The LADS framework uses a data-layout-aware multithreaded architecture to address bottlenecks in storage and CPU. However, the LADS framework uses a single RMA buffer, which causes memory controller overload and access to memory on other sockets.

An object of the present invention is to provide a data transmission system for arranging a memory buffer for each CPU socket and a data transmission method using the same.

Another problem to be solved by the present invention is to provide a memory-aware thread scheduling technique that allows a thread allocated to each CPU socket to access a memory buffer disposed in that CPU socket.

A data transfer system according to an embodiment of the present invention includes a plurality of CPU sockets and a memory buffer disposed for each CPU socket, each CPU socket having a CPU core assigned with a master thread and a CPU core assigned with a communication thread. And at least one CPU core to which the input / output thread is allocated.

Threads allocated to CPU cores contained in each CPU socket only access memory buffers located in that CPU socket.

Master threads, communication threads, and I / O threads assigned to CPU cores included in each CPU socket interact to process files transferred through memory buffers disposed in the corresponding CPU sockets.

If each CPU socket includes multiple NUMA nodes composed of CPU cores contained in the corresponding CPU socket, the master thread and the communication thread are allocated to the CPU cores included in the same NUMA node.

I / O threads are allocated to CPU cores adjacent to the CPU core to which the master thread is assigned.

In a method of transmitting data by a data transmission system according to an embodiment of the present invention, a master thread and a communication thread of a transmitting end interact with each other to activate an input / output thread of the transmitting end, wherein the activated input / output thread is a memory buffer of the transmitting end. Loading data into a first memory buffer, a communication thread of a receiver accessing the data loaded in the first memory buffer, and loading the data into a second memory buffer of the memory buffers of the receiver; and And storing data loaded in the second memory buffer in storage by the input / output thread of the receiving end, wherein the first memory buffer comprises a master thread of the transmitting end, a communication thread of the transmitting end, and an input / output thread of the transmitting end. Including allocated CPU cores And a memory buffer arranged to CPU socket, said second memory buffer is a memory buffer arranged to CPU socket including a CPU core input and output threads are allocated for communication threads, and the receiving end of the receiving end.

When the CPU socket of the transmitting end includes a plurality of NUMA nodes composed of CPU cores included in the CPU socket, the master thread and the communication thread are allocated to the CPU cores included in the same NUMA node.

I / O threads are allocated to CPU cores adjacent to the CPU core to which the master thread is assigned.

When the CPU socket of the receiving end includes a plurality of NUMA nodes composed of CPU cores included in the CPU socket, the master thread and the communication thread are allocated to the CPU cores included in the same NUMA node.

I / O threads are allocated to CPU cores adjacent to the CPU core to which the master thread is assigned.

According to the present invention, since a memory buffer is disposed in each CPU socket, the load generated on the memory controller can be reduced compared to the case where a plurality of CPU sockets use a single memory buffer.

In addition, according to the present invention, since each CPU socket needs to access a memory buffer disposed therein, it is possible to reduce a memory access delay time caused by accessing a memory buffer disposed in another CPU socket.

1 is a structural diagram of a data transmission system according to an embodiment.
2 is a diagram illustrating a process of transmitting data by a data transmission system according to an exemplary embodiment.
3 is a diagram illustrating a comparison between a multiple memory buffer implemented in a data transmission system and a conventional single memory buffer, according to an exemplary embodiment.
4 is a diagram illustrating a method of implementing socket-based thread scheduling in a data transmission system according to an embodiment.
5 is a diagram illustrating a method of implementing thread scheduling based on NUMA node in a data transmission system according to an embodiment.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and like reference numerals designate like parts throughout the specification.

Throughout the specification, when a part is said to "include" a certain component, it means that it can further include other components, without excluding other components unless specifically stated otherwise.

Hereinafter, a data transmission system and method according to an embodiment of the present invention will be described with reference to the drawings.

1 is a structural diagram of a data transmission system according to an embodiment.

Referring to FIG. 1, the data transmission system 100 includes a transmitting end 100 and a receiving end 200, and each of the transmitting end 100 and the receiving end 200 includes a first CPU socket 110 and 210 and a second CPU. Sockets 120 and 220.

Although FIG. 1 illustrates that the transmitter 100 and the receiver 200 include only two CPU sockets, the transmitter 100 and the receiver 200 may include two or more CPU sockets according to various embodiments.

In addition, the respective configurations and functions of the CPU sockets 110 and 120 included in the transmitter 100 and the CPU sockets 210 and 220 included in the receiver 200 are the same, hereinafter, the first CPU socket of the transmitter 100. Reference will be made to the reference numeral 110 and the first CPU socket 210 of the receiver 200.

The first CPU socket 110 of the transmitter 100 includes a first NUMA node 111, a second NUMA node 112, and a first memory buffer 113, and includes a first CPU socket ( 210 also includes a first NUMA node 211, a second NUMA node 212, and a first memory buffer 213. The number of NUMA nodes included in each CPU socket shown in FIG. 1 is merely an example, and each CPU socket may include two or more NUMA nodes according to various embodiments.

The first NUMA node 111 and the second NUMA node 112 of the transmitting end 100 include CPU cores 111a to 111d and 112a to 112d, each CPU core having a master thread, a communication thread or an input / output thread. Either of which is assigned.

For example, as shown in FIG. 1, three CPU cores among the CPU cores 111a to 111d included in the first NUMA node 111 of the transmitting end 100 are respectively a master thread, a communication thread, and an I / O thread. May be allocated, and I / O threads may be allocated to three CPU cores among the CPU cores 112a to 112d included in the second NUMA node 112 of the transmitter 100.

The first memory buffer 113 of the transmitter 100 is disposed for each CPU socket included in the transmitter 100. For example, as shown in FIG. 1, there are two CPU sockets 110 and 120 in the transmitting end 100, so that two memory buffers 113 and 123 of the same size are provided with the CPU sockets ( 110 and 120), respectively.

According to the present invention, since a memory buffer is disposed in each CPU socket, the load generated on the memory controller can be reduced compared to the case where a plurality of CPU sockets use a single memory buffer.

In this case, a thread assigned to a CPU core included in each of the CPU sockets 110 and 120 only accesses a memory buffer disposed in that socket.

For example, in FIG. 1, threads allocated to the CPU nodes 111a to 111d and 112a to 112d included in the first CPU socket 110, respectively, access the first memory buffer 113 and the second CPU. Threads allocated to the CPU nodes 121a to 121d and 122a to 122d respectively included in the socket 110 access the second memory buffer 123 in the same manner.

That is, all threads and connections of each CPU socket operate independently of the other sockets, and the master thread, communication thread, and I / O thread assigned to the CPU cores included in each CPU socket will have a memory buffer placed on that CPU socket. Interact with each other to handle files sent through it.

According to the present invention, since the threads included in each CPU socket need to access a memory buffer disposed in the corresponding CPU socket, the memory access delay time caused by accessing the memory buffer disposed in the other CPU socket can be reduced.

The receiving end 200 may have the same structure as the transmitting end 100. That is, the number of CPU sockets included in the receiver 200 is the same as the number of CPU sockets included in the transmitter 100, and the structure of the CPU sockets included in the receiver 200 corresponds to the number of CPU sockets included in the transmitter 100. It can be configured in the same manner as the structure of the CPU socket.

2 is a diagram illustrating a process of transmitting data by a data transmission system according to an embodiment.

In FIG. 2, each of the CPU sockets included in the transmitter 100 and the receiver 200 may include one master thread and one communication thread, and may include a plurality of input / output threads, each of which will be described below. The data loaded in the memory buffers 113 and 123 of the transmitter 100 is transmitted to the receiver 200 by performing the same function.

In this case, the respective configurations and functions of the CPU sockets 110 and 120 included in the transmitter 100 and the CPU sockets 210 and 220 included in the receiver 200 are the same, hereinafter, the first CPU socket of the transmitter 100. A description will be given focusing on data communication between the 110 and the first CPU socket 210 of the receiver 200.

Referring to FIG. 2, in order to activate the input / output threads 111b, 112a, 112b, and 112d of the transmitter 100, the master thread 111a and the communication thread 111d of the transmitter 100 interact with each other.

In detail, the master thread 111a of the transmitting end 100 determines the data chunk layout of the file to be transmitted (S100), and transmits the determined layout to the work queue of the communication thread 111d (S101).

The communication thread 111d receiving the layout transmits a new file request to the communication thread 211d of the receiving end 200 (S103), and the communication thread 211d transmits the received request to the work queue of the master thread 211a. It transmits (S105).

The master thread 211a generates a new file and an identifier for the new file (S107), and transmits an identifier request for the new file to the work queue of the communication thread 211d (S109).

The communication thread 211d transmits the request to the communication thread 111d of the transmitting end 100 (S111), and the communication thread 111d transmits the request to the work queue of the master thread 111a (S113). The thread 111a transmits chunk information of the file to the OST queue (S115).

The master thread 111a activates the input / output threads 111b, 112a, 112b, and 112d by the number of OSTs in which the file is striped (S117).

The activated I / O threads 111b, 112a, 112b, and 112d traverse the OST queues, receive the chunk information, and load the data chunks into the first memory buffer 113 (S119), where the data is stored in the first memory buffer 113. The chunk is stored (S121). That is, the activated I / O threads 111b, 112a, 112b, and 112d load data into the first memory buffer 113 disposed in the first CPU socket 110, and are disposed in the second CPU socket 110. No data is loaded into the second memory buffer 123.

In addition, the activated I / O threads 111b, 112a, 112b, and 112d transmit a request to transmit the data chunk information to the receiving end 200 to the work queue of the communication thread 111d (S123).

The communication thread 111d transmits the request to the communication thread 211d of the receiving end 200 (S125), and the communication thread 211d of the receiving end 200 transmits the request to the first memory buffer 113 of the transmitting end 100. The data chunk is accessed (S127), and the data chunk is loaded from the first memory buffer 113 of the transmitter 100 to the first memory buffer 213 of the receiver 200 (S129). That is, like the transmitter 100, the communication thread 211d accesses only data loaded in the first memory buffer 213. Thereafter, the data chunk is stored in the first memory buffer 113 (S131).

Thereafter, when the communication thread 211d places the input / output threads 211b, 212a, 212b, and 212d in the respective OST queues (S133), the input / output threads 211b, 212a, 212b, and 212d are arranged in the first memory buffer 213. ) Is written to the OST storage (S135). In this way, the transmission for one data chunk is completed and the above steps are repeated until the transmission for all data chunks is completed.

3 is a diagram illustrating a comparison between a multiple memory buffer implemented in a data transmission system and a conventional single memory buffer, according to an exemplary embodiment.

Referring to FIG. 3, in the conventional single memory buffer system 300, a single memory buffer 325 is disposed for a plurality of CPU sockets 310 and 320.

In this case, the threads 321 to 324 allocated to the CPU cores included in the second CPU socket 320 may access the memory buffer 325 allocated to the second CPU socket 320. However, since the threads 311 to 314 allocated to the CPU cores included in the first CPU socket 310 must access the memory buffer 325 disposed far away, this causes a memory access delay time.

In addition, when the threads 311 to 314 and 321 to 324 access one memory buffer 325, a problem occurs that the second CPU socket 320 hosting the memory buffer 325 becomes overloaded. .

In contrast, in the multiple memory buffer system 400 of the present invention, a first memory buffer 415 is disposed for the first CPU socket 410, and a second memory buffer 425 for the second CPU socket 420. Is placed.

In this case, the threads 411 to 414 allocated to the CPU cores included in the first CPU socket 410 may access the first memory buffer 415 and the CPU included in the second CPU socket 420. The threads 421 to 424 allocated to the cores may access the first memory buffer 425, and thus, unlike the conventional terminal memory buffer system 300, the memory access delay time does not occur.

In particular, even when the I / O threads 413, 414, 423, and 424 increase, each I / O thread needs to access a memory buffer in which a CPU socket in which the I / O thread is located exists. (Not shown) can be significantly reduced.

4 is a diagram illustrating a method of implementing socket-based thread scheduling in a data transmission system according to an embodiment.

Referring to FIG. 4, even if the multiple memory buffer system 500 includes the memory buffers 515 and 525 in each of the plurality of CPU sockets 510 and 520, it may be included in the first CPU socket 510 in some cases. The threads 511 to 514 allocated to the assigned CPU cores may access the second memory buffer 525, and conversely, the threads 521 to 514 allocated to the CPU cores included in the second CPU socket 520. 524 may access the first memory buffer 515.

To prevent this, each of the CPU sockets 510 and 520 includes at least one master thread, communication thread, and input / output thread.

For example, as shown in FIG. 4, the first CPU socket 510 includes a first master thread 511, a first communication thread 512, and a first input / output thread 513 and 514. The two CPU sockets 520 include a second master thread 521, a second communication thread 522, and second input / output threads 523 and 524.

In addition, the threads included in each CPU socket interact to process files transmitted through the corresponding CPU socket, and do not process files transmitted through the other CPU socket. That is, all threads work fixedly in their respective CPU sockets.

In detail, the first master thread included in the first CPU socket 510 may include a first communication thread included in the first CPU socket 510 to process a file transmitted through the first memory socket 510. Interacting with 512, the first communication thread 512 operates only the first memory buffer 515. In addition, the first input / output threads 513 and 514 included in the first CPU socket 510 work only on objects allocated through the first CPU socket 510. The same applies to the second CPU socket 520.

5 is a diagram illustrating a method of implementing thread scheduling based on NUMA node in a data transmission system according to an embodiment.

Referring to FIG. 5, when each CPU socket includes a plurality of heterogeneous storage access nodes (hereinafter, NUMA nodes) composed of CPU cores included in the corresponding CPU small, the interaction of the threads included in the CPU socket And fair core usage is important. If a master thread and a communication thread are allocated to different NUMA nodes, performance decreases due to different NUMA node memory accesses when the threads interact.

Thus, the master thread and communication thread included in the CPU socket are allocated to the CPU cores included in the same NUMA nodes, respectively. For example, as shown in FIG. 5, the master thread 611a and the communication thread 611b allocated to the CPU socket 610 may be allocated to the CPU cores included in the first NUMA node 611, respectively. have.

In addition, in order to place the master thread and the I / O thread as close as possible to optimize the performance of the CPU socket, the I / O thread is allocated to the CPU core adjacent to the CPU core to which the master thread is allocated. For example, as shown in FIG. 5, among the CPU cores included in the second NUMA node 612, an input / output thread may be allocated to the CPU cores 612a and 612c that are closer to the master thread 611a, respectively. .

According to the present invention, since a memory buffer is disposed in each CPU socket, the load generated on the memory controller can be reduced compared to the case where a plurality of CPU sockets use a single memory buffer.

In addition, according to the present invention, since each CPU socket needs to access a memory buffer disposed therein, it is possible to reduce a memory access delay time caused by accessing a memory buffer disposed in another CPU socket.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements of those skilled in the art using the basic concepts of the present invention defined in the following claims are also provided. It belongs to the scope of rights.

Claims (10)

As a data transmission system,
Multiple CPU sockets, and
Contains a memory buffer disposed for each CPU socket,
Each CPU socket includes at least one CPU core assigned a master thread, a CPU core assigned a communication thread, and a CPU core assigned an I / O thread,
Each CPU socket includes a plurality of NUMA nodes composed of CPU cores contained in the corresponding CPU socket, and a master thread and a communication thread are allocated to CPU cores included in the same NUMA node. In claim 1,
A data transfer system in which threads allocated to CPU cores contained in each CPU socket access only memory buffers placed in that CPU socket. In claim 1,
A master thread, a communication thread, and an I / O thread assigned to CPU cores included in each CPU socket interact with each other to process a file transferred through a memory buffer disposed in the CPU socket. delete In claim 1,
I / O threads are data transfer systems in which CPU threads are assigned to adjacent CPU cores. As a data transmission system transmits data,
Activating an input / output thread of the transmitting end by interacting with a master thread of the transmitting end and a communication thread;
Loading the data into a first memory buffer of the memory buffers of the transmitting end by the activated input / output thread;
A communication thread of a receiver accessing the data loaded in the first memory buffer to load the data into a second memory buffer among memory buffers of the receiver; and
Storing the data loaded in the second memory buffer in storage by an input / output thread of the receiving end;
The first memory buffer is a memory buffer disposed in a CPU socket including CPU cores to which a master thread of the transmitting end, a communication thread of the transmitting end, and an input / output thread of the transmitting end are allocated;
The second memory buffer is a memory buffer disposed in a CPU socket including CPU cores to which a communication thread of the receiving end and an input / output thread of the receiving end are allocated;
The CPU socket of the transmitting end includes a plurality of NUMA nodes composed of CPU cores included in the CPU socket, and a master thread and a communication thread are allocated to CPU cores included in the same NUMA node. delete In claim 6,
I / O thread is a data transfer method in which a master thread is allocated to a CPU core adjacent to an assigned CPU core. In claim 6,
If the CPU socket of the receiving end includes a plurality of NUMA nodes composed of CPU cores included in the CPU socket, a master thread and a communication thread are allocated to CPU cores included in the same NUMA node. In claim 9,
I / O thread is a data transfer method in which a master thread is allocated to a CPU core adjacent to an assigned CPU core.

Images