JP6070078B2 - Hybrid parallel processing system, hybrid parallel processing method, and computer program - Google Patents

Description

  The present invention relates to parallel processing by a computer, and more particularly to a load distribution technique in parallel processing by a computer.

  Parallel processing by a computer (sometimes referred to as parallel computing or parallel computing) is not a computer program (hereinafter simply referred to as “program”) execution, data processing, calculation, or the like, This process is performed in parallel. The parallel processing classification by computer includes thread parallel processing and process parallel processing. The thread parallel processing is sometimes called shared memory parallel processing or intra-node parallel processing. Process parallel processing is sometimes called distributed memory type parallel processing or inter-node parallel processing. The thread parallel processing is realized mainly by using, for example, OpenMP on a shared memory type computer. Process parallel processing is realized by using, for example, MPI (Message Passing Interface).

  When performing process parallel processing, parallel processing for performing thread parallel processing within each process is called hybrid parallel processing. In the hybrid parallel processing, the processing time of each process varies depending on the processing amount (calculation load) of each process and the number of threads. For example, if the number of threads is the same even though the processing amount of each process is not uniform, a difference occurs in the processing time of each process. In such a case, at the timing of the synchronization process or the data transfer process between the processes, a time occurs in which the process that has ended first waits for the end of another process being executed. This decreases the execution performance of parallel processing (that is, the execution time of the entire parallel processing increases). In particular, in an application program used in the HPC (High Performance Computing) field, the execution time is long, so the influence of a decrease in execution performance is great. In other words, accumulation of the effect of the decrease in execution performance can be a major factor that hinders high-speed execution of application programs.

  In order to prevent the above problems, it is necessary to set an appropriate number of threads for each process. That is, the effect of parallel processing can be improved by increasing the number of threads of a process with a large amount of processing and reducing the number of threads of a process with a small amount of processing. For example, in OpenMP, the user can explicitly specify the number of threads for each process.

  However, it is very difficult to estimate and determine an appropriate number of threads because it is necessary to consider hardware configuration such as the number of processors, software settings such as the number of processes, input data at the time of execution, and the like.

  A technique for solving such problems is described in Patent Document 1. The hybrid parallel processing system described in Patent Document 1 dynamically adjusts the number of threads between processes based on the processing time of the latest shared memory type parallel processing (thread parallel processing).

  Moreover, the analysis apparatus described in Patent Document 2 evaluates the distribution state of processing for a plurality of processes based on the standby time for interprocess communication.

JP 2011-180725 A JP 2011-203994 A

  The hybrid parallel processing system described above needs to measure the processing time each time the thread parallel processing is executed. That is, time measurement processing occurs in all processes. Further, for example, the processing time is uniform among the processes, so that the time measurement process occurs even when the number of threads does not need to be adjusted. For this reason, there is a problem in that the performance of the entire parallel processing deteriorates due to the accumulation of time measurement processing overhead.

  The main object of the present invention is to solve the above-described problems and realize efficient parallel processing in hybrid parallel processing.

The hybrid parallel processing system according to the present invention that achieves the above object is as follows.
With multiple processes in parallel,
Each of the processes
One or more threads that perform the processing assigned to the process;
Data transfer processing means for performing data transfer processing between processes;
Thread number resetting means for resetting the number of threads,
The data transfer processing means includes a transfer waiting time measuring means for measuring a transfer waiting time in the data transfer processing between processes, and transmits / receives the transfer waiting time to / from another process,
The thread number resetting means resets the number of threads based on the transfer waiting time.

The hybrid parallel processing method according to the present invention for achieving the above object is
Process multiple processes in parallel,
Each of the processes performs the assigned processing by one or more threads,
Measure the transfer waiting time in the data transfer process between processes,
The process sends and receives the transfer waiting time to and from other processes,
Reset the number of threads based on the transfer waiting time.

And the program concerning the present invention which achieves the above-mentioned object is
Using a processor that can process multiple processes in parallel,
Each of the processes performs the assigned processing by one or more threads,
Measure the transfer waiting time in the data transfer process between processes,
The process sends and receives the transfer waiting time to and from other processes,
Causes the computer to execute processing for resetting the number of threads based on the transfer waiting time.

  This object can also be achieved by a computer-readable storage medium storing such a program.

  According to the present invention, efficient parallel processing in hybrid parallel processing is realized.

It is a block block diagram showing the structure of the hybrid parallel processing system which concerns on the 1st Embodiment of this invention. It is a flowchart explaining the thread number default value determination process of each process in the hybrid parallel processing system S1 which concerns on the 1st Embodiment of this invention. It is a flowchart explaining the thread number adjustment process (interprocess communication process and thread number reset process) of each process in the hybrid parallel processing system S1 according to the first embodiment of the present invention. It is a figure explaining comparatively the process of the hybrid parallel processing system S1 which concerns on the 1st Embodiment of this invention, and the hybrid parallel processing system which does not perform a thread number adjustment process. It is a block block diagram showing the structure of the hybrid parallel processing system which concerns on the 2nd Embodiment of this invention. And FIG. 11 is a block diagram illustrating an example of elements constituting a computer.

  Next, embodiments of the present invention will be described in detail with reference to the drawings.

[First Embodiment]
FIG. 1 is a block diagram showing the configuration of the hybrid parallel processing system according to the first embodiment of the present invention. The hybrid parallel processing system S1 according to the first embodiment of the present invention includes processes 10-1 to 10-m. Each of the processes 10-1 to 10-m includes threads 11-1 to 11-n, a thread number default value setting unit 12, a thread number resetting unit 13, and a data transfer processing unit 14. The data transfer processing unit 14 includes a transfer waiting time measuring unit 141.

  Processes 10-1 to 10-m are a plurality of processes executed on the same host. The threads 11-1 to 11-n are threads assigned to each process in order to execute processing assigned to the process. The number of threads included in each of the processes 10-1 to 10-m may be the same or different. The thread number default value setting unit 12 determines a default value of the thread number at the start of process execution. The thread number resetting unit 13 resets the thread number of each process based on a transfer waiting time (period) described later. The data transfer processing unit 14 performs data transfer processing and inter-process synchronization processing. The transfer waiting time measuring unit 141 measures the transfer waiting time in the data transfer process and the synchronization waiting time in the inter-process synchronization process.

  FIG. 2 is a flowchart for explaining the thread number default value determination process of each process in the hybrid parallel processing system S1 according to the first embodiment of the present invention. FIG. 3 is a flowchart for explaining thread number adjustment processing (inter-process communication processing and thread number resetting processing) of each process in the hybrid parallel processing system S1 according to the first embodiment of the present invention. Hereinafter, the processing operation of the hybrid parallel processing system S1 will be described in detail with reference to the drawings.

  At the start of program execution by hybrid parallel processing, the thread number default value setting unit 12 determines the default value of the thread number for each process. Referring to FIG. 2, first, the thread number default value setting unit 12 refers to an environment variable for specifying the thread number (step S201). For example, in OpenMP, the user can specify the number of threads via an environment variable.

  If an environment variable is specified (YES in step S202), the number of threads is set based on this environment variable (step S203). If no environment variable is specified (NO in step S202), the number of threads is set from the system default value (step S204).

  After the number of threads is set, process parallel processing starts. Each process performs thread parallel processing, and then performs inter-process communication processing (inter-process synchronization processing or inter-process data transfer processing). In the following, the case of performing the data transfer process between processes will be described, but the process of measuring the transfer waiting time and resetting the number of threads is the same when performing the inter-process synchronization process.

  Referring to FIG. 3, when performing the inter-process communication process, the data transfer processing unit 14 determines whether or not the data transfer process can be started (step S301). If the data transfer process can be started immediately (YES in step S301), the transfer waiting time measuring unit 141 sets the transfer waiting time to zero (step S302). Then, the process proceeds to step S307.

  On the other hand, when the data transfer process cannot be started (NO in step S301), the transfer waiting time measuring unit 141 acquires the transfer waiting start time (S) (step S303). That is, the transfer waiting time measuring unit 141 stores the time at this time as the transfer waiting start time (S). The reason why data transfer processing cannot be started is that the process to be transferred (transfer partner process) has not completed the processing in the process (thread parallel processing, etc.) and is not ready for data transfer. Etc.

  Next, the data transfer processing unit 14 and the transfer waiting time measuring unit 141 stand by until data transfer is possible (step S304).

  When it is detected that the standby state has been solved and data transfer can be started, the transfer waiting time measuring unit 141 acquires the transfer waiting end time (E) (step S305). Then, the transfer waiting time measuring unit 141 calculates a transfer waiting time (ES) (step S306).

  Thereafter, the data transfer processing unit 14 executes a data transfer process (step S307). After the end of the data transfer process, each process exchanges the transfer waiting time and the current number of threads. That is, the transfer waiting time and the current number of threads are transmitted to and received from each other. Then, the thread number resetting unit 13 of each process calculates the number of threads for executing subsequent thread parallel processing based on the transfer waiting time and the number of threads of the own process and other processes, and the calculated number of threads is set as a new thread. Set as a number.

Hereinafter, a procedure and a calculation formula for calculating the new number of threads will be described in detail. The outline (policy) of these procedures and calculation methods is as follows.
The product of the transfer waiting time of each process and the number of threads of each process is aggregated (calculated), and the total of the transfer waiting time is evenly redistributed to all processes. That is, a value (average waiting time) obtained by dividing the total transfer waiting time of each thread by the total number of processes is obtained and transmitted to each process. In each process, the transfer waiting time of each process is compared with the redistributed transfer waiting time (average waiting time).
When the transfer waiting time of the own process exceeds the redistributed transfer waiting time, it is considered that the number of currently allocated threads is excessive (the number of threads is large for a small amount of processing). Then, some threads are transferred to another process.
When the transfer waiting time of the own process is less than the redistributed transfer waiting time, the number of currently assigned threads is considered to be too small (the number of threads is small for the large amount of processing). Then, a part of the thread is acquired from another process.

  Each variable related to the formula for calculating the new number of threads is defined as shown in the table below.

W ^ave and D _n are calculated according to the following formulas, respectively.

Here, if D _n > 0, the process n is considered to have an excessive number of assigned threads because the transfer waiting time is longer than the average waiting time W ^ave . In this case, a part of the thread assigned to the process n is transferred to another process.

The number of processes T _n ^new after resetting a process with D _n > 0 is obtained as follows. First, T _n ^min which is the minimum value of the number of threads reset to the process n is obtained by the following calculation formula.

_Then, by calculating the average values from ^{T n min} to _{T n,} determine the ^{T n new new.} _{Alternatively,} by determining the median of the numerical values of ^{T n min} to a value of _{T n,} may be calculated ^{T n new new.} For example, T _n ^new is obtained by the following calculation formula.

On the other hand, if D _n <0, the process n is considered to have an insufficient number of threads because the transfer waiting time is shorter than the average waiting time W ^ave . In this case, the process n receives a thread transfer from another process. The number of threads to be transferred is obtained by distributing all the threads transferred from the process with D _n > 0 according to the size (ratio) of D _n among the processes with D _n <0.

For example,

In this case, the number of processes T _n ^new after resetting a process where D _n <0 is obtained by the following calculation formula.

  Hereinafter, an example of thread number adjustment processing (calculation of a new thread number) when the number of processes is 2 (p = 2) will be described.

In the following description, the two processes are referred to as “process 1” and “process 2”, respectively. The number of threads of process 1 and process 2 at the start is 8 (T ₁ = T ₂ = 8). Since the calculation load of the process 1 is low, a transfer waiting time of 4 seconds occurs due to the start of inter-process communication processing prior to the process 2 (W ₁ = 4). On the other hand, since the process 2 has a high calculation load, the data transfer can be performed immediately after the start of the inter-process communication process, so that the transfer waiting time is zero (W ₂ = 0). With the above,

It is. Since D ₁ > 0 and D ₂ <0, the thread is transferred from the process 1 to the process 2.

Adjustment of the number of threads of process 1 (calculation of a ^new number of threads T ₁ ^new ) is performed as follows.

  That is, the number of threads in process 1 is reduced from 8 to 6. Then, the reduced two threads are transferred to another process.

The adjustment of the number of threads of the process 2 (calculation of a ^new number of threads T ₂ ^new ) is performed as follows.

  That is, the number of threads in process 2 is increased from 8 to 10 by assigning 2 threads transferred from process 1 to process 2.

  FIG. 4 is a diagram for comparing and explaining the processing of the hybrid parallel processing system S1 according to the first embodiment of the present invention and the hybrid parallel processing system that does not perform the thread number adjustment processing. (A) on the left side of FIG. 4 represents processing of a hybrid parallel processing system that does not perform thread number adjustment processing. 4B on the right side of FIG. 4 represents processing of the hybrid parallel processing system S1 according to the first embodiment of the present invention. In FIGS. 4A and 4B, it is assumed that the number of processes is two.

  Referring to FIG. 4A, in the hybrid parallel processing system that does not perform the thread number adjustment processing, the thread parallel processing I of the process 1 with a small amount of processing ends first. For this reason, in the process 1, a communication waiting time (transfer waiting time) occurs until the thread parallel processing I of the process 2 is completed at the stage of the interprocess communication processing. Also in the subsequent processing, since the processing amount of the process 1 is small, the thread parallel processing II of the process 1 having the small processing amount is completed first as before. For this reason, the process 1 always has a waiting time.

  On the other hand, referring to FIG. 4B, the hybrid parallel processing system S <b> 1 according to the first embodiment of the present invention exchanges communication waiting time at the stage of inter-process communication processing. Based on the communication waiting time, the number of threads is reset according to the load of each process. That is, the number of threads of process 1 with a large amount of processing increases, and the number of threads of process 2 with a small amount of processing decreases. As described above, the number of threads of each process is adjusted to a number corresponding to the processing amount, so that the processing time of each process (processing time of the thread parallel processing II) is leveled in the subsequent processing. As a result, the transfer waiting time is reduced, so that the execution time of the entire parallel processing is shortened. That is, the efficiency of parallel processing is improved.

  As described above, in the hybrid parallel processing system S1 according to the first embodiment of the present invention, the thread number resetting unit 13 determines the number of threads of each process based on the transfer waiting time measured by the transfer waiting time measuring unit 141. Is reset to a number according to the processing amount. As a result, the processing time of each process is leveled, the transfer waiting time is reduced, and the execution time of the entire parallel processing is reduced. As described above, the hybrid parallel processing system S1 according to the first embodiment of the present invention improves the efficiency of parallel processing by performing dynamic load distribution between processes.

  Further, the transfer waiting time measuring unit 141 does not perform the time measuring process when there is no transfer waiting time (when the data transfer process can be started immediately when performing the inter-process communication process). For this reason, compared with the hybrid parallel processing system described in Patent Document 1 in which time measurement processing occurs in all processes, the hybrid parallel processing system S1 according to the first embodiment of the present invention has a performance degradation due to time measurement processing. Few.

[Second Embodiment]
Next, a second embodiment of the present invention will be described.

  FIG. 5 is a block diagram showing the configuration of the hybrid parallel processing system according to the second embodiment of the present invention. The hybrid parallel processing system S2 according to the second embodiment of the present invention includes processes 10-1 to 10-m that are a plurality of processes to be processed in parallel. Each of the processes 10-1 to 10-m is a thread 11-1 to 11-n that is one or more threads that execute processing assigned to each process, and a thread number resetting that resets the number of threads. Unit 13 and a data transfer processing unit 14 that performs data transfer processing between processes. The data transfer processing unit 14 includes a transfer waiting time measuring unit 141 that measures a transfer waiting time in data transfer processing between processes.

  The transfer waiting time measuring unit 141 transmits / receives the transfer waiting time to / from another process. Then, the thread number resetting unit 13 resets the thread number based on the transfer waiting time.

  As described above, in the hybrid parallel processing system S2 according to the second embodiment of the present invention, the thread number resetting unit 13 determines the number of threads of each process based on the transfer waiting time measured by the transfer waiting time measuring unit 141. To reset. As a result, the processing time of each process is leveled, the transfer waiting time is reduced, and the execution time of the entire parallel processing is reduced. Thus, the hybrid parallel processing system S1 according to the second embodiment of the present invention improves the efficiency of parallel processing.

  FIG. 6 is a block diagram illustrating an example of elements constituting the computer. 6 includes a CPU (Central Processing Unit) 910, a RAM (Random Access Memory) 920, a ROM (Read Only Memory) 930, a hard disk drive 940, and a communication interface 950. The CPU 910 includes CPU cores 911 and 912.

  The computer 900 may be connected to another computer 900 via a communication interface 950 via a network (not shown). In this case, the plurality of computers 900 are configured to be able to transmit / receive data to / from each other.

  The CPU 910 can process a plurality of processes in parallel. The number of CPU cores included in the CPU 910 need not be two. The CPU 910 may be a multiprocessor including a plurality of CPUs instead of a single CPU.

  The components of the hybrid parallel processing systems S1 and S2 described above may be realized by executing a program in the CPU 910 of the computer 900. Specifically, the thread number default value setting unit 12, the thread number resetting unit 13, the data transfer processing unit 14, and the transfer waiting time measuring unit 141, which are the components described in FIG. The program may be realized by reading a program from the ROM 930 or the hard disk drive 940 and executing the read program by the CPU 910 as shown in the flowcharts of FIGS. 3 and 4, for example. In such a case, the present invention described with reference to the above-described embodiment is an example of a computer-readable storage medium (for example, hard disk drive 940 or the like) that stores a code representing the computer program or a code representing the computer program. It can be understood that it is constituted by a removable magnetic disk medium, optical disk medium, memory card, etc. (not shown).

  Alternatively, the thread number default value setting unit 12, the thread number resetting unit 13, the data transfer processing unit 14, and the transfer waiting time measuring unit 141 may be realized by dedicated hardware. Further, the hybrid parallel processing systems S1 and S2 may be dedicated hardware including these components.

S1, S2 Hybrid parallel processing system 10-1 to 10-m Process 11-1 to 11-n Thread 12 Thread number default value setting unit 13 Thread number resetting unit 14 Data transfer processing unit 141 Transfer waiting time measuring unit 900 Computer 910 CPU
911, 912 CPU core 920 RAM
930 ROM
940 Hard disk drive 950 Communication interface

Claims (7)

With multiple processes in parallel,
Each of the processes
One or more threads that perform processing assigned to the process;
Data transfer processing means for performing data transfer processing between the processes;
Thread number resetting means for resetting the number of threads,
The data transfer processing means includes a transfer waiting time measuring means for measuring a transfer waiting time in the data transfer processing between the processes, and transmits and receives the transfer waiting time to and from another process;
The thread number resetting means calculates an average waiting time that is a value obtained by dividing the total transfer waiting time of each thread by the number of processes, and calculates the product of the number of threads of the process and the transfer waiting time. Calculating a difference obtained by subtracting an average waiting time, and transferring a part of a thread of a process having a positive value to the calculated difference to a process having a negative value in the calculated difference. A hybrid parallel processing system that resets the number. In each of the processes in which the calculated difference is a positive value, the thread number resetting means is based on a quotient obtained by dividing the average waiting time by the transfer waiting time of the process and the number of threads before resetting. Determining the number of threads to be transferred, and distributing each of the transferred threads according to a ratio of the calculated difference values in each of the processes in which the calculated difference is a negative value. The hybrid parallel processing system according to claim 1 , wherein the number of threads is reset. The process is a hybrid parallel processing system according to claim 1 or 2 including a number of threads default value setting means for determining a default value for the number of threads at the start of the execution of the process. Using a processor that can process multiple processes in parallel,
Each of the processes executes the assigned process by one or more threads;
Measure the transfer waiting time in the data transfer process between the processes,
The process sends / receives the transfer waiting time to / from other processes,
Calculate the average waiting time which is a value obtained by dividing the total transfer waiting time of each thread by the number of processes.
Calculate the difference obtained by subtracting the average waiting time from the product of the number of threads of the process and the transfer waiting time,
A hybrid parallel processing method for resetting the number of threads by transferring a part of a thread of a process whose calculated difference is a positive value to a process whose calculated difference is a negative value . In each process where the calculated difference is a positive value, determine the number of threads to be transferred based on the quotient obtained by dividing the average waiting time by the transfer waiting time of the process and the number of threads before resetting;
In each process the calculated difference is a negative value, according to claim 4 for resetting the number of the threads by dispensing in accordance with all threads that are the transfer to the ratio of the value of the calculated difference The hybrid parallel processing method described in 1. Process multiple processes in parallel,
Each of the processes executes the assigned process by one or more threads;
Measure the transfer waiting time in the data transfer process between the processes,
The process sends / receives the transfer waiting time to / from other processes,
Calculate the average waiting time which is a value obtained by dividing the total transfer waiting time of each thread by the number of processes.
Calculate the difference obtained by subtracting the average waiting time from the product of the number of threads of the process and the transfer waiting time,
A program for causing a computer to execute a process of resetting the number of threads by transferring a part of a thread of a process whose calculated difference is a positive value to a process whose calculated difference is a negative value . In each process where the calculated difference is a positive value, determine the number of threads to be transferred based on the quotient obtained by dividing the average waiting time by the transfer waiting time of the process and the number of threads before resetting;
In each of the processes in which the calculated difference is a negative value, a process for resetting the number of threads by distributing all of the transferred threads according to the ratio of the calculated difference values The program according to claim 6 , wherein the program is executed.

Images