KR20190109638A - Method for scheduling task in big data analysis platform based on distributed file system, program and computer readable storage medium therefor - Google Patents

Abstract

The present invention relates to a task scheduling method in a big data processing platform based on a distributed file system. The task scheduling method in a big data processing platform based on a distributed file system including a name node and a data node, comprises the following steps: receiving information on a data block stored in an in-memory cache of a data node from a name node when a map-reduce job is executed; determining whether the data block used for the map-reduce job exists in the in-memory cache on the basis of the information on the data block; and scheduling a task of the map-reduce job using the cached data block when the data block used for the map-reduce job exists in the in-memory cache.

Description

Task scheduling method of big data processing platform based on distributed file system, computer program and computer readable recording medium for this purpose.

The present invention relates to a task scheduling technique of a big data processing platform based on a distributed file system.

In the Yarn of hadoop 2.0, one of the big data processing platforms based on the distributed file system, the map reduce job consists of at least one task. Yarn's scheduler then schedules these tasks at compute nodes running node managers. When tasks are scheduled on compute nodes, containers are created on those compute nodes. The task is then executed in the container created in this way. At this time, if the compute node on which the task is executed is the same as the data node that stores the data block that this task must process, then this task is referred to as a data local task.

Recently, an in-memory cache has been added to the Hadop Distribution File System (HDFS). In executing the task, in addition to the above-described data local task, the task may be executed by reading a block of data stored in the in-memory cache of the Hadoop distributed file in-memory cache system, which is called a cache local task. In this case, there is a need for a method of improving the utility of the Hadoop distributed file in-memory cache system as compared to the conventional method while fully utilizing the data blocks stored in the in-memory cache.

Korea Patent Registration No. 10-1661475, 2016.09.26 registration

In an embodiment of the present invention, a virtualization environment and a cache locality in a big data processing platform based on a distributed file system, particularly a big data processing platform including an in-memory cache based Hadoop distributed file system (HDFS) We propose a task scheduling technique that considers Cache Locality.

The problem to be solved by the present invention is not limited to the above-mentioned, another problem to be solved is not mentioned can be clearly understood by those skilled in the art from the following description. will be.

According to an embodiment of the present invention, in a task scheduling method in a big data processing platform based on a distributed file system including a name node and a data node, when a map reduce job is executed, the data node is executed. Receiving information from the name node regarding a data block stored in an in memory cache; Determining whether a data block used for the map reduce operation exists in the in-memory cache based on the information about the data block; And scheduling a task of the map reduce job by using a cached data block when a data block used for the map reduce job exists in the in-memory cache. A task scheduling method in a processing platform may be provided.

The scheduling may include scheduling a task of the map reduce job by using a data block cached in a virtual machine in a plurality of physical machines on which the task of the map reduce job is executed. .

The scheduling may include: when a plurality of virtual machines for each physical machine co-resident in the plurality of physical machines, a physical machine in which the virtual machine on which the task of the map reduce operation is executed resides; A task of the map reduce job may be scheduled by using a data block cached in a virtual machine in another physical machine.

In addition, when the data block of the map reduce operation does not exist in the in-memory cache or the waiting time of the scheduling step exceeds a preset time, the map is utilized by using the data block stored in the disk of the virtual machine. The method may further include scheduling a task of the reduce job.

In addition, in the scheduling, the task of the map reduce job may be scheduled by using data blocks cached in virtual machines in different physical machines in the big data processing platform based on the distributed file system. .

According to an embodiment of the present invention, a virtual data environment and cache locality are considered together in a big data processing platform based on a distributed file system, particularly a big data processing platform including an in-memory cache-based Hadoop distributed file system (HDFS). By scheduling the task of the map reduce job, the map reduce job can efficiently use the cached input data.

1 is a system block diagram of a big data processing platform based on a distributed file system for task scheduling according to an embodiment of the present invention, in particular, an in-memory cache-based Hadoop distributed file system and a big data processing platform.
FIG. 2 is a diagram exemplarily illustrating task scheduling considering a virtualization environment performed in a big data processing platform including the in-memory cache-based Hadoop distributed file system of FIG. 1.
3 is a flowchart illustrating a task scheduling process in a big data processing platform including a distributed file system, for example, an in-memory cache-based Hadoop distributed file system according to an embodiment of the present invention.

Advantages and features of the present invention and methods for achieving them will be apparent with reference to the embodiments described below in detail with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but can be implemented in various forms, only the embodiments are to make the disclosure of the present invention complete, and those skilled in the art to which the present invention pertains. It is provided to fully inform the scope of the invention, and the scope of the invention is defined only by the claims.

In describing the embodiments of the present invention, detailed descriptions of well-known functions or configurations will be omitted unless they are actually necessary in describing the embodiments of the present invention. In addition, terms to be described below are terms defined in consideration of functions in the embodiments of the present invention, which may vary according to intentions or customs of users and operators. Therefore, the definition should be made based on the contents throughout the specification.

An embodiment of the present invention provides a virtualization environment in a big data processing platform including a big data processing platform based on a distributed file system, in particular, a Hadop Distribution File System (HDFS) based on an in-memory cache. By scheduling the task of the map reduce task in consideration of the locality and cache locality, we propose a technique for efficiently using the input data cached by the map reduce task of HDFS.

Hereinafter, with reference to the accompanying drawings will be described in detail an embodiment of the present invention.

1 is a system block diagram of a big data processing platform including a distributed file system for task scheduling, eg, a big data processing platform including an in-memory cache-based HDFS 10.

Referring to FIG. 1, the in-memory cache-based HDFS 10 may include a name node 300 and a data node 400. When the application is executed in Hadoop, which is one of the big data processing platforms including the in-memory cache-based HDFS 10, the task is executed in the created container 500. In addition, the scheduling apparatus 100 for scheduling the task in the container of the calculation node is also included in the big data processing platform. Here, the above-described computing node, data node 400 and in-memory cache 420 can be executed in a cluster composed of a plurality of servers (computers), such servers may be referred to as nodes below. In some cases, the node may perform a plurality of functions of the calculation node and the data node 400, and may perform only one function of the calculation node and the data node 400.

The data node 400 may be configured in plural and stores a data block, which is a result of dividing a file into blocks, on a local disk 410 which is a kind of storage.

The in-memory cache 420 may be formed in a portion of memory included in a plurality of servers constituting the in-memory cache-based HDFS 10, for example, random access memory (RAM).

The data node 400 periodically reports to the name node 300 which data block is stored in its in-memory cache 420. For example, the data node 400 may periodically transmit a heartbeat of 3 seconds to the name node 300.

The name node 300 stores and manages basic information of a file or a data block and metadata about which data node 400 among the plurality of data nodes 400 is stored. In addition, the name node 300 manages information about data blocks stored in the local disk 410 and the in-memory cache 420, particularly information stored in the HDFS, among a plurality of data blocks related to the map reduce application. The information about the data blocks stored in the in-memory cache 420 among the plurality of data blocks may include a ratio of the number of data blocks stored by the in-memory cache 420 to the number of the plurality of data blocks, but is not limited thereto. It doesn't happen. In addition, the name node 300 periodically transmits a ratio of the input data stored in the in-memory cache 420 to the scheduling apparatus 100 for each map reduce job. Meanwhile, the cache manager 200 and the virtual machine topology 210 included in the name node 300 will be described later.

The container 500 is configured to execute a task. When the container 500 executes a task, the container 500 transmits a request to the name node 300 to cache a block of data to be processed by the task in the in-memory cache 420. In this case, the request is delivered to the name node 300 even if the corresponding data block is already stored in the in-memory cache 420.

The scheduling apparatus 100 receives a ratio of the number of data blocks stored in the in-memory cache 420 among the total number of data blocks related to the map reduce operation from the name node 300.

Next, the scheduling apparatus 100 is based on whether the first waiting time, which is the maximum waiting time to wait to be scheduled by the cache local task, and the second waiting time, which is the maximum waiting time waiting to be scheduled by the data local task, has passed. To schedule tasks. Here, the first waiting time may be shorter than the second waiting time.

According to an embodiment of the present invention, the scheduling apparatus 100 may schedule a task in consideration of a virtualization environment and cache locality when executing multiple map reduce tasks simultaneously. In detail, the scheduling apparatus 100 may schedule a task to utilize a data block cached or stored in a virtual machine in the same physical machine in consideration of the virtual machine topology.

A detailed process of scheduling the task by the scheduling apparatus 100 will be described in more detail with reference to FIGS. 2 and 3, but the scheduling apparatus 100 stores a memory that stores instructions programmed to perform such a process, and such instructions. It can be implemented by a microprocessor that performs.

The cache manager 200 stores data blocks previously stored in the in-memory cache 420 based on the intimacy with the in-memory cache 420 of the map reduce application when there is no space to additionally store the data blocks in the in-memory cache 420. Replace with the data block to add and save.

In addition, the cache manager 200 adds the data block in consideration of the probability that the data block is stored in the in-memory cache 420 when the data block is added to the in-memory cache 420. The probability of this storage is determined by the cache affinity of the map reduce application, the size of the in-memory cache 420, and the size of the input data of the application.

The virtual machine topology 210 may be included in the name node 300, and may be composed of a plurality of virtual machines in a virtualized environment, and data blocks may be cached or stored in such a virtual machine.

FIG. 2 is a diagram exemplarily illustrating task scheduling of the scheduling apparatus 100 in consideration of a virtualization environment performed by the HDFS 10 of FIG. 1.

As shown in FIG. 2, the virtualization environment includes a plurality of virtual machines, for example, VM1, VM2, VM3, VM4, VM5, and VM6, and some of the plurality of virtual machines are physical machines PM1, PM2, PM3. ) May be included. And, some physical machines, for example PM1 and PM2, may be included in a rack, for example Rack1.

The map reduce job has the shortest execution time when the input data is cached in the memory of the virtual machine where the task runs, for example, VM1. This state will be referred to as VM-ML (Virtual Machine Memory Locality).

However, if the task can't be scheduled with VM-ML and the input data is cached in the memory of another VM running in the same PM, then the input data cached can be utilized only by inter-VM communication without disk I / O. It has a performance effect similar to utilizing input data cached in the memory of a running VM. This state will be referred to as physical machine memory locality (PM-ML).

In addition, the input data cached in the memory of the VM running in the same rack can utilize the input data cached without the I / O of the disk only by inter-PM communication. Will be shortened. This state will be referred to as rack machine memory locality (RM-ML).

Therefore, when scheduling a task, the task execution time is scheduled in the shortest VM-ML, and if the waiting time for scheduling in the VM-ML exceeds the set time, the scheduling is attempted in the PM-ML. If the waiting time for scheduling with PM-ML exceeds the set time, task scheduling is attempted with RM-ML.

If there is no input data cached in the memory or the waiting time for scheduling the task in the RM-ML exceeds the set time, the task scheduling is attempted to utilize the input data stored in the disk of the VM where the task is executed. This state will be referred to as virtual machine disk locality (VM-DL).

In addition, as in PM-ML, when using data read from the disk of another VM in service in the same PM, there is less overhead than using data stored in the disk of the VM in service in other PMs in the same rack. . Therefore, if the waiting time for scheduling with the VM-DL exceeds the set time, the task scheduling is attempted to utilize the input data stored in the disk of another VM of the same PM. This state will be referred to as physical machine disk locality (PM-DL).

Finally, if the scheduling fails, the task is scheduled to utilize input data stored in the disk of the VM in service in another PM of the same rack. This state will be referred to as a rack machine disk locality (RM-DL).

In summary, scheduling in a virtualization environment according to an embodiment of the present invention may be performed in the following order.

VM-ML-> PM-ML-> RM-ML-> VM-DL-> PM-DL-> RM-DL

3 is a flowchart illustrating a task scheduling process in a big data processing platform including a distributed file system, for example, an in-memory cache-based HDFS according to an embodiment of the present invention.

Referring to FIG. 3, first, the scheduling apparatus 100 receives information about a data block stored in an in-memory cache 420 of a node from a name node 300 when a map reduce operation is executed (S100 and S110). . Here, the information about the data blocks stored in the in-memory cache 420 may be, for example, a ratio of the number of data blocks stored in the in-memory cache 420 among the total number of data blocks related to the map reduce operation.

Next, the scheduling apparatus 100 performs a virtual execution of the task of the map reduce job according to whether the data block used in the map reduce job exists in the in-memory cache 420 based on the information about the data block. After deciding whether to utilize the data blocks cached on the machine or the data blocks stored on the disk of the virtual machine, you can schedule the task of the map reduce job.

In this case, the scheduling apparatus 100 calculates a first waiting time, which is a limit of a waiting time until scheduled by the cache local task. The first waiting time may be calculated by multiplying a predetermined maximum waiting time C by a ratio received from the name node 300.

As a result of the determination in step S120, if a data block used for the map reduce operation exists in the in-memory cache, the scheduling apparatus 100 may delete the data block cached in the virtual machine on which the task of the map reduce operation is executed. In operation S130, the task of the map reduce job may be scheduled.

Here, when the data block used for the map reduce operation exists in the in-memory cache, the task may be scheduled in the order of VM-ML-> PM-ML-> RM-ML as described above.

In this case, in the scheduling step S130, the task of the map reduce task may be scheduled by utilizing data blocks cached in the virtual machines in the plurality of physical machines on which the task of the map reduce task is executed. For example, as shown in FIG. 2, when a plurality of virtual machines, for example, VM1 and VM2 co-resident in each of the plurality of physical machines, for example, PM1, map reduce. Maps utilizing data blocks cached in the virtual machine (for example, VM3) in the physical machine (PM1) in which the virtual machine (VM1) on which the task's task is executed resides and in another physical machine (for example, PM2). You can schedule the task of the reduce job.

On the other hand, if the data block of the map reduce operation does not exist in the in-memory cache 420, or if the waiting time of the scheduling step (S130) exceeds the predetermined time (S140), the scheduling apparatus 100 of the virtual machine The task of the map reduce job may be scheduled by using the data block stored in the disk (S150).

If the data block used for the map reduce operation does not exist in the in-memory cache, the task may be scheduled in the order of VM-DL-> PM-DL-> RM-DL as described above.

As described above, according to the exemplary embodiment of the present invention, a map data is considered in consideration of a virtualization environment and cache locality in a big data processing platform based on a distributed file system, particularly a big data processing platform including an in-memory cache-based HDFS. By scheduling the task of the deuce job, the map reduce job is implemented to efficiently use the input data cached in HDFS.

On the other hand, the combination of each block in the accompanying block diagram and each step in the flowchart may be performed by computer program instructions. These computer program instructions may be mounted on a processor of a general purpose computer, special purpose computer, or other programmable data processing equipment, such that the instructions executed by the processor of the computer or other programmable data processing equipment are described in each block of the block diagram. It creates a means to perform the functions.

These computer program instructions may be stored on a computer usable or computer readable recording medium (or memory) or the like that may be directed to a computer or other programmable data processing equipment to implement functionality in a particular manner, thereby making the computer available. Alternatively, instructions stored on a computer readable recording medium (or memory) may produce an article of manufacture containing instruction means for performing the functions described in each block of the block diagram.

In addition, computer program instructions may be mounted on a computer or other programmable data processing equipment, such that a series of operating steps may be performed on the computer or other programmable data processing equipment to create a computer-implemented process to generate a computer or other program. Instructions that perform possible data processing equipment may also provide steps for performing the functions described in each block of the block diagram.

In addition, each block may represent a portion of a module, segment, or code that includes at least one or more executable instructions for executing a specified logical function (s). It should also be noted that in some alternative embodiments, the functions noted in the blocks may occur out of order. For example, the two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the corresponding function.

100: scheduling device
200: cache management unit
210: virtual machine topology
300: Name Node
400: data node
410: local disk
420: In-memory cache
500: container

Claims (7)

A task scheduling method in a big data processing platform based on a distributed file system including a name node and a data node,
Receiving information about a data block stored in an in memory cache of the data node from the name node when a map reduce job is executed;
Determining whether a data block used for the map reduce operation exists in the in-memory cache based on the information about the data block; And
Scheduling a task of the map reduce job by using a cached data block if a data block used for the map reduce job exists in the in-memory cache;
Task scheduling method in big data processing platform based on distributed file system. The method of claim 1,
The scheduling step,
Scheduling a task of the map reduce job by using a data block cached in a virtual machine in a plurality of physical machines on which the task of the map reduce job is executed
Task scheduling method in big data processing platform based on distributed file system. The method of claim 2,
The scheduling step,
When a plurality of virtual machines for each physical machine co-resident in the plurality of physical machines, a cache is stored in a virtual machine in a physical machine different from the physical machine in which the virtual machine on which the task of the map reduce job is executed resides. Scheduling a task of the map reduce job using a data block
Task scheduling method in big data processing platform based on distributed file system. The method of claim 1,
If the data block of the map reduce operation does not exist in the in-memory cache or if the waiting time of the scheduling step exceeds a preset time, the map reduce operation is utilized by using the data block stored in the disk of the virtual machine. Further comprising the step of scheduling a task of the job
Task scheduling method in big data processing platform based on distributed file system. The method of claim 1,
The scheduling step,
Scheduling a task of the map reduce job using a data block cached in a virtual machine in different physical machines of the big data processing platform
Task scheduling method in big data processing platform based on distributed file system. A task scheduling method in a big data processing platform based on a distributed file system including a name node and a data node,
Receiving information about a data block stored in an in-memory cache of the data node from the name node when a map reduce operation is executed;
Determining whether a data block of the map reduce operation exists in the in-memory cache based on the information about the data block; And
If a data block of the map reduce job exists in the in-memory cache, a program including an instruction for performing a task of scheduling a task of the map reduce job by using the cached data block is recorded.
Computer-readable recording media. A task scheduling method in a big data processing platform based on a distributed file system including a name node and a data node,
Receiving information about a data block stored in an in-memory cache of the data node from the name node when a map reduce operation is executed;
Determining whether a data block of the map reduce operation exists in the in-memory cache based on the information about the data block; And
If a data block of the map reduce job exists in the in-memory cache, scheduling a task of the map reduce job by using a cached data block
Computer program stored on a computer readable recording medium.

Images