KR20180124692A - Method for processing task in respect to distributed file system - Google Patents

Abstract

The present invention relates to a task processing method for a distributed file system. The method according to an embodiment of the present invention comprises the steps of: receiving information on a plurality of tasks to be processed for distributed data from a distributed file system; loading first distributed data from the distributed file system based on the information and storing the first distributed data in a Java virtual machine (JVM) heap CPU memory when a task related to a GPU kernel operation is included among information on the plurality of tasks; storing second distributed data obtained by copying the first distributed data in an off-heap CPU memory; storing third distributed data obtained by copying the second distributed data in a GPU memory; and performing a GPU kernel operation on the third distributed data. Therefore, the number of transmitting data is minimized.

Description

{METHOD FOR PROCESSING TASK IN RESPECT TO DISTRIBUTED FILE SYSTEM}

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a task processing method for a distributed file system, and more particularly, to a task processing method for a distributed file system in which a GPU executes a task of a distributed file system by storing distributed data in an off- will be.

Hadoop (Hadoop), a high-availability distributed object-oriented platform, is a distributed processing framework that works in a clustered computing environment to handle large amounts of data.

The Hadoop framework manages data based on the Hadoop distributed file system (HDFS), a distributed file system, and can process data in parallel using map-reduce. In addition, the Hadoop framework divides the data to be processed into distributed data, executes the task on a plurality of nodes, and merges the result of the task processing on each node so as to output the result.

On the other hand, the spark framework is a framework that enables distributed data processing to be executed at high speed in a distributed cluster. Sparks can be faster than Hadoop for tasks that require repeated computations, such as machine learning or charting, because they can read the distributed data to be processed from the Hadoop distributed file system and then perform the task's execution in GPU memory .

On the other hand, the Spark Framework is loaded from the Hadoop distributed file system when GPU kernel operation is performed, and is stored in the main memory or disk of the computer, and is transferred to the GPU whenever necessary to be used for calculation.

However, when the GPU completes the tasks for the distributed data in the above manner, the distributed data of the GPU copied from the main memory or the disk of the computer disappears. Therefore, the spark framework performs repetitive operations on the distributed data in the GPU , The distributed data is transferred from the main memory or disk to the GPU again whenever one task is completed.

Thus, since the time required for periodically transmitting distributed data from the main memory to the GPU is much larger than the GPU operation speed, the task processing method of the distributed data using the existing GPU can sufficiently utilize the computation performance of the GPU It is not suitable for doing.

Korean Patent Laid-Open Publication No. 10-1794696: Task Scheduling Method Considering Heterogeneous Processing Types and Distributed Processing System

A problem to be solved by an embodiment of the present invention is to provide a technique for minimizing the number of times data transmission occurs by storing distributed data used for iterative operations in a distributed file system in an off-heap CPU memory.

We also provide a technique that allows distributed data stored in the off-heap CPU memory to be shared directly with other nodes based on the MPI (message passing interface) protocol used between the off-heap CPU memories.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to provide further explanation of the invention as claimed.

A task processing method for a distributed file system according to an exemplary embodiment of the present invention includes receiving information on a plurality of tasks to be processed for distributed data from a distributed file system, processing unit), a first distributed data among the distributed data is loaded from the distributed file system based on the information, and the first distributed data is loaded into a Java virtual machine (JVM) heap central processing unit (CPU) memory Storing the second distributed data copied from the first distributed data in an off-heap CPU memory, storing third distributed data copied from the second distributed data in the GPU memory, And performing the GPU kernel operation on the distributed data.

In this case, the distributed file system is a Hadoop distributed file system (HDFS). The step of loading the first distributed data and storing the first distributed data in the JVM heap CPU memory is performed based on an API of a spark framework .

In addition, storing the third distributed data in the GPU memory may include declaring a GPU context.

In addition, the off-heap CPU memory may share the second distributed data with nodes included in the distributed file system according to a message passing interface (MPI) protocol.

Storing the second distributed data in the off-heap CPU memory includes copying the first distributed data from the JVM heap CPU memory to the off-heap CPU memory through UDS (unix domain socket) communication .

Storing the third distributed data in the GPU memory includes copying the second distributed data from the off-heap CPU memory to the GPU memory via a compute unified device architecture (CUDA) memcpy command .

According to the embodiment of the present invention, since the GPU performs the task on the distributed data transmitted from the off-heap CPU memory and periodically transmits the distributed data from the main memory to the GPU, To maximize GPU performance.

1 is a diagram illustrating an overall system including a task processing apparatus and a distributed file system according to an embodiment of the present invention.
2 is a flowchart showing a process of a task processing method for a distributed file system according to an embodiment of the present invention.
FIG. 3 is a diagram illustrating a process of performing a task according to a task processing method for a distributed file system according to an embodiment of the present invention. Referring to FIG.
4 is an exemplary diagram for explaining halos generated when a domain decomposition method is performed.
5 is an exemplary diagram illustrating a process of performing a region partitioning method according to a task processing method for a distributed file system according to an embodiment of the present invention.
FIG. 6 is a graph illustrating an experimental result of comparing a task processing method for a distributed file system and an existing framework according to an exemplary embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention and the manner of achieving them will become apparent with reference to the embodiments described in detail below with reference to the accompanying drawings. It should be understood, however, that the invention is not limited to the disclosed embodiments, but may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, To fully disclose the scope of the invention to a person skilled in the art, and the scope of the invention is only defined by the claims.

In describing embodiments of the present invention, a detailed description of well-known functions or constructions will be omitted unless otherwise described in order to describe embodiments of the present invention. The following terms are defined in consideration of the functions in the embodiments of the present invention, which may vary depending on the intention of the user, the intention or the custom of the operator. Therefore, the definition should be based on the contents throughout this specification.

The functional blocks shown in the drawings and described below are merely examples of possible implementations. In other implementations, other functional blocks may be used without departing from the spirit and scope of the following detailed description. Also, while one or more functional blocks of the present invention are represented as discrete blocks, one or more of the functional blocks of the present invention may be a combination of various hardware and software configurations that perform the same function.

Also, to the extent that the inclusion of certain elements is merely an indication of the presence of that element as an open-ended expression, it should not be understood as excluding any additional elements.

Further, when a component is referred to as being connected or connected to another component, it may be directly connected or connected to the other component, but it should be understood that there may be other components in between.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

1 is a diagram illustrating an overall system including a task processing apparatus 100 and a distributed file system 10 according to an embodiment of the present invention.

The distributed file system 10 is composed of a plurality of nodes, and a plurality of nodes can transmit and receive data through a communication network. At this time, each node processes tasks for distributed data based on the distributed file framework. At this time, the distributed file system 10 may include a Hadoop Distributed File System (HDFS), and the distributed file framework may include an API of a spark framework.

Here, the nodes of the distributed file system 10 may include devices capable of performing operations including, for example, one or more processors, such as computers, workstations, and the like.

In addition, the node of the distributed file system 10 includes a client node for generating a task to be processed with respect to distributed data and requesting the processing of the task to another node, a task requested from the client node A job tracker node for allocating and scheduling tasks to be handled by other nodes, a task tracker node for performing tasks according to instructions of a job tracker node and generating a task processing result have.

At this time, the plurality of nodes constituting the distributed file system 10 are not limited to one role, and one node may perform at least one of a client node, a job tracker node, and a task tracker node at the same time.

The task processing apparatus 100 includes one or more processors and can be used as a node constituting the Hadoop distributed file system 10. [ The task processing apparatus 100 can simultaneously perform one or more roles of a client node, a job tracker node, and a task tracker node. In performing a task on distributed data, the task processing apparatus 100 can execute distributed data stored in the off-heap CPU memory 120 Gt; 121, < / RTI > the GPU can perform the task. Accordingly, the task processing method for the distributed file system performed by the task processing apparatus 100 will be described with reference to FIG.

2 is a flowchart showing a process of a task processing method for a distributed file system according to an embodiment of the present invention. Each step of the task processing method for the distributed file system according to FIG. 2 can be performed by the task processing apparatus 100 including one or more processors described with reference to FIG. 1, and each step will be described below.

First, information on a plurality of tasks to be processed for distributed data is received from the distributed file system 10 (S210). In this case, the distributed data may include resilient distributed dataset (RDD), and the task is one of the operations for managing distributed data, for example, a map task generated based on the spark framework, Reduce tasks, and the like. ≪ RTI ID = 0.0 >

On the other hand, the spark framework is a framework that enables distributed data processing to be executed in a distributed cluster at a high speed. The spark framework reads distributed data to be processed from the Hadoop distributed file system 10 and executes the tasks in the GPU memory 130 .

On the other hand, the spark framework loads the distributed data 111 from the Hadoop distributed file system 10 when the GPU kernel operation is performed, stores the distributed data in the main memory or disk of the computer, and transmits the distributed data to the GPU And the like.

The task processing method for the distributed file system according to an embodiment of the present invention includes storing the distributed data 121 in the off-heap CPU memory 120 and declaring the GPU context so that the GPU memory 130 By allowing the storage of the distributed data 131 to be maintained, the distributed data is not lost in the GPU memory 130 even if one task in the iterative operation is completed, thereby minimizing time loss associated with transmission of distributed data between hardware , Allowing the GPU to take full advantage of its high computing power.

To this end, when the task processing method for the distributed file system according to an embodiment of the present invention includes a GPU task designated to perform a GPU kernel operation among a plurality of tasks received from the distributed file system 10, The following steps can be performed.

First, predetermined distributed data 111 designated as a task to be processed from the distributed file system 10 is loaded and stored in a JVM (java virtual machine) heap CPU memory (S220). In this case, step S220 may be performed based on the API included in the spark framework.

Thereafter, the distributed data 111 stored in the JVM heap CPU memory 110 is copied and stored in the off-heap CPU memory 120 (S230), and the distributed data 121 stored in the off- And stores it in the GPU memory 130 (S240). For this, an embodiment of the present invention may additionally include an API having the functions of Table 1 below, in comparison with the API included in the spark framework. At this time, the transmission of the distributed data 111 between the JVM heap CPU memory 110 and the off-heap CPU memory 120 can be performed based on UDS (unix domain socket) communication. Accordingly, .

Also, the transmission of the distributed data 121 between the off-heap CPU memory 120 and the GPU memory 130 can be performed on the basis of a CUDA command (for example, memcpy), thereby implementing API functions of htod and dtoh .

Meanwhile, exec, cache, and uncache are functions to manage GPU, and APIs can be implemented using existing technologies, and halo_extract, halo_shuffle, and halo_append can implement API through existing algorithms used in domain decomposition method field.

Therefore, in step S230, send of the API according to Table 1 can be used, and in step S240, htod can be used.

At this time, copying the distributed data 121 stored in the off-heap CPU memory 120 and storing the copied data 121 in the GPU memory 130 (S240) may include declaring the GPU context.

The GPU context is a command for the GPU to copy and store the distributed data 121 from the off-heap CPU memory 120 to perform the kernel operation. When the GPU context is declared in the off-heap CPU memory 120, There is an advantage that the distributed data 131 does not disappear in the GPU memory 130 even if the task for the distributed data 131 is completed. To do this, the GPU context can be declared in the off-heap CPU memory 120 as shown in Table 2. [

Accordingly, the GPU task designated to perform the GPU kernel operation among the plurality of tasks is performed on the distributed data 131 stored in the GPU memory 130 (S250). In this case, when the iterative operation is required, additional operations can be performed by continuing to store the distributed data in the GPU memory 130 by not releasing the GPU context through the cache function shown in Table 1. When all operations are completed it is possible to delete the distributed data 131 stored in the GPU memory 130 through the uncache function and to release the declaration of the GPU context. The distributed data 131 in which the task execution is completed in the GPU can be returned to the off-heap CPU memory 120 and the JVM heap CPU memory 110 and uploaded to the distributed file system 10.

Accordingly, the embodiment of the present invention allows the GPU to perform a task based on the distributed data stored in the off-heap CPU memory 120, so that the data between the main memory and the GPU Thereby solving the transmission problem, thereby maximizing the performance of the GPU in the distributed file system 10. [

FIG. 3 is a diagram illustrating a process of performing a task according to a task processing method for a distributed file system according to an embodiment of the present invention. Referring to FIG.

Referring to FIG. 3, the first row indicates information about a plurality of tasks to be processed for distributed data. At this time, the distributed data 111 stored in the JVM heap CPU memory 110 includes distributed data 121 stored in the resilient distributed dataset (RDD), off-heap CPU memory 120, and distributed data 121 stored in the JVM heap CPU memory 110 (131) is a virtual resilient distributed dataset (VRDD) as data copied from the JVM heap CPU memory (110). On the other hand, send, htod, cache, exec, dtoh, and recv are APIs included in Table 1, and TextFile, map, and reduce are APIs included in the spark framework. Meanwhile, the task used in the example shown in FIG. 3 is as shown in Table 3 below, and line 1 - 10 in Table 3 corresponds to line 1-10 shown in FIG. 3.

3 and Table 3, a task processing method for a distributed file system according to an embodiment of the present invention is a method for processing a plurality of distributed data files 111, 121, 131 to be processed from a distributed file system 10 Information on the task (the first row in Fig. 3) can be received.

Accordingly, the TextFile task may be executed based on the API included in the spark framework to load the distributed data RDD1 from the Hadoop distributed file system 10 (HDFS) and store the distributed data RDD1 in the JVM heap CPU memory 110. [ Also, the distributed data RDD2 can be generated by performing a map task on the distributed data RDD1 based on the API included in the spark framework. Thereafter, the distributed data RDD2 may be copied from the JVM heap CPU memory 110 to the off-heap CPU memory 120 and stored (VRDD3) by performing a send task based on the API generated according to Table 1, (VRDD5) stored in the GPU memory 130 by copying the distributed data (VRDD3) stored in the off-heap CPU memory 120 by performing the htod task and then executing the cache task to distribute It is possible to maintain the storage of the data VRDD5.

Accordingly, since the GPU can store the distributed data (VRDD5) repeatedly used for the calculation in performing the iterative calculation (Iteration 1, Iteration 2, ...), data transfer between the GPU and the CPU is minimized GPU's high computing power can be fully utilized.

4 is an exemplary diagram for explaining halos generated when a domain decomposition method is performed.

The domain segmentation method is a well-known algorithm for solving the boundary value problem occurring at the boundary of a partition while dividing the domain into sub-domains. At this time, a task processing method for a distributed file system according to an embodiment of the present invention is a method for processing a task in a node (Halo), which is an overlapping area between sub-domains (domain 0 and domain 1) You can share it directly.

5 is an exemplary diagram illustrating a process of performing a region partitioning method according to a task processing method for a distributed file system according to an embodiment of the present invention.

Referring to FIG. 5, the first row indicates information about a plurality of tasks to be processed for distributed data. At this time, the distributed data (VRDD1) stored in the GPU memory (130) can be used for iterative calculation while the storage is maintained. Meanwhile, in FIG. 4, the distributed data for domain 0 and domain 1 may be stored in different nodes.

For example, assume that the current node is storing distributed data (VRDD1) for domain 0, and the distributed data for domain 1 is stored in another node. In this case, in order for the current node to perform operation on the halo part of domain 0, information about the halo part of domain 1 of another node should be shared. To this end, the task processing method for the distributed file system according to an embodiment of the present invention can perform the area partitioning method through the command shown in Table 4 below.

5 and Table 4, an embodiment of the present invention performs an extract task based on the API generated according to Table 1 to calculate a part of the distributed data (VRDD1) stored in the GPU memory 130 Heap CPU memory 120, and stores the copied portion. Thereafter, the shuffle task is performed to share the distributed data (VRDD2) for the halo portion stored in the off-heap CPU memory 120 with the distributed data for the halo portion stored at the other node to perform the area division method for halo And can thereby generate the divided data VRDD3 in which the area division method operation has been performed. This is because the off-heap CPU memory 120 can share distributed data with other nodes of the distributed file system 10 in accordance with the MPI (message passing interface) protocol.

Thereafter, the GPU memory 130 may perform an append task based on the API generated according to Table 1 to generate information on domain 0, which is calculated once by the area division method, from VRDD1 and VRDD3, Can be performed.

Therefore, according to the embodiment of the present invention, since the off-heap CPU memory 120 is used, only the GPU memory 130 is used for the calculation, and unlike the existing technology in which information on distributed data can not be directly shared, Distributed data can be directly communicated and shared.

FIG. 6 is a graph illustrating an experimental result of comparing a task processing method for a distributed file system and an existing framework according to an exemplary embodiment of the present invention.

6 is based on the spark-C framework, the spark-G framework, and the GPU memory 130, which performs time and exec tasks to copy / store distributed data from the CPU memory to the GPU memory 130 according to embodiments of the present invention. As compared to the kernel operation time, it can be seen that the embodiment of the present invention shows remarkably improved speed compared to the existing spark framework.

Therefore, according to the above-described embodiment, since the GPU performs the task on the distributed data transmitted from the off-heap CPU memory 120 and the distributed data is periodically transmitted from the main memory to the GPU, So that GPU performance in the distributed file system 10 can be maximized.

The above-described embodiments of the present invention can be implemented by various means. For example, embodiments of the present invention may be implemented by hardware, firmware, software, or a combination thereof.

In the case of hardware implementation, the method according to embodiments of the present invention may be implemented in one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs) , FPGAs (Field Programmable Gate Arrays), processors, controllers, microcontrollers, microprocessors, and the like.

In the case of an implementation by firmware or software, the method according to embodiments of the present invention may be implemented in the form of a module, a procedure or a function for performing the functions or operations described above. A computer program recorded with a software code or the like may be stored in a computer-readable recording medium or a memory unit and may be driven by a processor. The memory unit is located inside or outside the processor, and can exchange data with the processor by various known means.

Combinations of the individual blocks of the block diagrams and flowchart illustrations attached to the present invention may also be performed by computer program instructions. These computer program instructions may be embedded in an encoding processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, so that the instructions, performed through the encoding processor of a computer or other programmable data processing apparatus, Thereby creating means for performing the functions described in each step of the flowchart. These computer program instructions may also be stored in a computer usable or computer readable memory capable of directing a computer or other programmable data processing apparatus to implement the functionality in a particular manner so that the computer usable or computer readable memory It is also possible for the instructions stored in the block diagram to produce a manufacturing item containing instruction means for performing the functions described in each block or flowchart of the block diagram. Computer program instructions may also be stored on a computer or other programmable data processing equipment so that a series of operating steps may be performed on a computer or other programmable data processing equipment to create a computer- It is also possible that the instructions that perform the processing equipment provide the steps for executing the functions described in each block of the block diagram and at each step of the flowchart.

In addition, each block or each step may represent a portion of a module, segment, or code that includes one or more executable instructions for executing the specified logical function. It should also be noted that in some alternative embodiments, the functions mentioned in the blocks or steps may occur out of order. For example, two blocks or steps shown in succession may in fact be performed substantially concurrently, or the blocks or steps may sometimes be performed in reverse order according to the corresponding function.

Thus, those skilled in the art will appreciate that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. It is therefore to be understood that the embodiments described above are to be considered in all respects only as illustrative and not restrictive. It is intended that the present invention covers the modifications and variations of this invention provided they come within the scope of the appended claims and their equivalents. .

10: Distributed File System
100: Task processing device
110: JVM heap CPU memory
120: off-heap CPU memory
130: GPU memory
111, 121, 131: distributed data

Claims (9)

1. A task processing method for a distributed file system performed by one or more processors,
Receiving information on a plurality of tasks to be processed for distributed data from a distributed file system;
When a task related to a graphics processing unit (GPU) kernel operation is included in the information on the plurality of tasks,
Loading first distributed data of the distributed data from the distributed file system based on the information and storing the first distributed data in a Java virtual machine (JVM) heap central processing unit (CPU) memory;
Storing second distributed data obtained by copying the first distributed data in an off-heap CPU memory;
Storing third distributed data having copied the second distributed data in a GPU memory; And
And performing the GPU kernel operation on the third distributed data
How to handle tasks for distributed file systems.
The method according to claim 1,
The distributed file system includes:
Hadoop distributed file system (HDFS)
The step of loading the first distributed data and storing the first distributed data in the JVM heap CPU memory includes:
Based on the API of the spark framework
How to handle tasks for distributed file systems.
The method according to claim 1,
Wherein the storing the third distributed data in the GPU memory comprises:
Declaring a GPU context (GPU context)
How to handle tasks for distributed file systems.
The method according to claim 1,
The off-heap CPU memory includes:
Sharing the second distributed data with a node included in the distributed file system according to an MPI (message passing interface) protocol
How to handle tasks for distributed file systems.
The method according to claim 1,
Storing the second distributed data in the off-heap CPU memory comprises:
Copying the first distributed data from the JVM heap CPU memory to the off-heap CPU memory via UDS (unix domain socket) communication
How to handle tasks for distributed file systems.
The method according to claim 1,
Wherein the storing the third distributed data in the GPU memory comprises:
Copying the second distributed data from the off-heap CPU memory to the GPU memory via a memcpy command of a compute unified device architecture (CUDA)
How to handle tasks for distributed file systems.
7. An arithmetic unit comprising one or more processors for performing the method of any one of claims 1-6.
A computer program comprising instructions for causing one or more processors to perform the method of any one of claims 1 to 6.
A computer program stored on a computer-readable medium for causing the one or more processors to perform the method of any one of claims 1 to 6.

Images