KR20150093979A - Method and apparatus for assigning namenode in virtualized cluster environments - Google Patents

Abstract

Disclosed are a method and an apparatus for selecting an optimum name node in a virtual cluster environment, wherein a plurality of name nodes exist. The method for allocating a name node in a virtual cluster environment includes a step of receiving a job; and a step of determining a name node, wherein the job is to be allocated, among at least two name nodes by considering a function of at least two name nodes or a processing speed as to the job, which are included in a virtual cluster. Accordingly, the optimum name node in the virtual cluster environment utilizing a plurality of name nodes may be effectively chosen.

Description

[0001] METHOD AND APPARATUS FOR ASSIGNING NAMENODE IN VIRTUALIZED CLUSTER ENVIRONMENTS [0002]

The present invention relates to a technique for distributing large-scale data, and more particularly, to a method and apparatus for selecting an optimal name node in a virtual cluster environment in which a plurality of name nodes exist.

Recently, virtualization technology has been introduced and widely used to make the best use of the hardware resources of computers. Virtualization is a technology that abstracts physical resources into multiple virtual resources. By running different operating systems for each virtual resource, it increases the utilization of multiplexed computing resources and provides efficient provisioning. In addition, it can isolate a specific virtual server when a fault occurs and provide non-stop service through another virtual server, thereby improving system availability and reliability.

On the other hand, parallel and distributed databases have been actively studied for large-scale data processing in the field of databases. In other words, as a software technology to maximize the parallelism of the hardware computing resources provided by the ManiCORE system, parallel DB technology is recently attracting attention with virtualization technology.

Parallel DB technology constructs data as a key-value pair and processes data in parallel to improve processing performance. Examples are Google's Google File System (GFS) and Bigtable, open source Hadoop, Hbase, and Hive, and programming models such as Map-Reduce, DryadLINQ, and MPI.

Hadoop is a Java software framework that supports distributed applications running on computer clusters to handle large amounts of data. It is largely composed of Hadoop Distributed File System (HDFS) and Map-Reduce ).

When constructing a Hadoop cluster according to the conventional Hadoop distributed file system (HDFS), the name node is only one node, which causes various problems. For example, there is a limitation on the name node memory, which limits the number of files and directories managed by the name node, and there is a limitation on the throughput of the file I / O because there is only one name node. There is a problem that the name node must be utilized.

In addition, in recent years, Hadoop has been applied to the concept of NameNode Federation, which is a set of a plurality of independent name nodes. However, when naming node federation is applied in a virtual machine environment having heterogeneous computing characteristics, The performance is different from each other. As a result, assigning a job to a name node according to the conventional method causes a problem due to a performance imbalance.

In order to solve the above problems, an object of the present invention is to provide a method of assigning a job to a name node in a virtualized cluster environment.

It is another object of the present invention to provide a system for parallelly distributing large-scale data.

According to another aspect of the present invention, there is provided a method of allocating a name node in a virtual cluster environment, the method comprising: receiving a job; performing a performance or task of at least two name nodes included in the virtual cluster Determining a name node to which to allocate a job among at least two name nodes considering the processing speed for the name node.

Here, the virtualization cluster may be composed of a plurality of virtual nodes existing in each of a plurality of physical machines.

Here, each of the at least two name nodes may be located one by one in each of a plurality of physical machines.

Here, the name node allocation method may be applied based on a Hadoop Distributed File System (HDFS).

Wherein determining the name node to assign the task comprises randomly selecting two name nodes out of at least two name nodes and selecting the selected two name nodes in consideration of performance and workload of the selected two name nodes. It is possible to determine the name node to which the task is assigned among the name nodes.

According to another aspect of the present invention, there is provided a system for parallel distributed processing of large-scale data according to an exemplary embodiment of the present invention includes: a name node federation configured by a set of at least two name nodes included in a virtualization cluster; And a name node control unit for determining a name node to allocate a task among at least two name nodes in consideration of performance or processing speed of at least two name nodes.

In the virtual cluster environment according to the present invention, when a method for assigning a name node and a system for parallelly distributing large-scale data are used, an optimal name node can be effectively selected in a virtual cluster environment utilizing a plurality of name nodes .

Also, according to the present invention, there is an advantage that consistent performance can be maintained while minimizing a check on performance and workload of a name node.

1 is a conceptual diagram for explaining a distributed file system.
2 is a conceptual diagram illustrating a cluster to which a plurality of name nodes are applied according to an embodiment of the present invention.
3 is a conceptual diagram for explaining a system for parallel-distributed processing large-scale data according to an embodiment of the present invention.
4 is a flowchart illustrating a method of assigning a name node in a virtual cluster environment according to an embodiment of the present invention.
5 is a graph illustrating a frequency of a name node to which tasks are allocated in a virtualization cluster to which a plurality of name nodes are applied according to an embodiment of the present invention.
6 is a graph for comparing the processing time of a task when a method of assigning a name node is applied in a virtual cluster environment according to an embodiment of the present invention.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. Like reference numerals are used for like elements in describing each drawing.

The terms first, second, A, B, etc. may be used to describe various elements, but the elements should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In the present application, the terms "comprises" or "having" and the like are used to specify that there is a feature, a number, a step, an operation, an element, a component or a combination thereof described in the specification, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the contextual meaning of the related art and are to be interpreted as either ideal or overly formal in the sense of the present application Do not.

First, terms used in the present invention will be briefly described as follows.

Because the Hadoop Distributed File System (HDFS) is a system designed for high availability and low cost, it always copies several files and distributes the copied files. In addition, information about the contents and location of the file is also distributed and stored in a plurality of copies. Because the contents and information of the files are distributed and stored on several computers, the search time is shortened.

Map-Reduce is a distributed data processing technology that utilizes multiple computers for efficient data processing. First, in the Map phase, large-scale data is distributed to several computers in parallel, processed in parallel, Results), and in the Reduce step, the resulting intermediate results are combined to produce the final desired result.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram for explaining a distributed file system.

First, a conventional distributed file system will be described in order to clarify the technical characteristics of the present invention.

Referring to FIG. 1, the Hadoop distributed file system (HDFS) includes a name node 110, which is a master node, and a plurality of data nodes 120-1 to 120-n. HDFS has a structure that supports high-capacity file management and ensures availability by storing multiple file copies. The data nodes 120-1 through 120-n store a copy of the file, and the name node 110 stores meta data of the file.

Specifically, the name node 110 manages a file system access request from the client 130 while managing the namespace of the file system. In HDFS, file data is divided into blocks and stored in the plurality of data nodes 120-1 to 120-n in a distributed manner. Here, the client 130 may refer to various types of user terminals.

The data nodes 120-1 through 120-n process data input and output requests from the client 130 and the naming node 110 periodically receives heartbeats from the data nodes 120-1 through 120- The state of the data nodes 120-1 to 120-n can be checked.

In order to ensure the availability of data, HDFS divides the relational data into blocks and then replicates the data to the data nodes 120-1 to 120-n for distribution. This is a technique of distributing one piece of data into n complete copies. It is possible to completely recover the original data even if only one of the nodes survive, but as the number of copies increases, the storage increases.

However, it has been described above that the HDFS according to FIG. 1 may cause various problems due to the parallel distributed processing of data using one name node.

2 is a conceptual diagram illustrating a cluster to which a plurality of name nodes are applied according to an embodiment of the present invention.

Referring to FIG. 2, a plurality of independent name nodes 210-1 to 210-k, ... 210-n exist in one cluster 20, and the name nodes 210-1, ... 210-k, ... 210-n may have respective unique namespaces (NS: NameSpace).

In addition, the block pools 230 have information on the entire blocks of the cluster 20, and each of the name nodes 210-1, ..., 210-k, ... 210-n And may divide a particular portion of such total block pools (Block Pools) 230. For example, NameNode 1 210-1 has Pool 1 230-1, NameNode k 210-k has Pool k 230-k, and NameNode n 210-n has Pool n (230-n).

For example, the cluster 20 may have N name nodes 210-1, ... 210-k, ... 210-n, and each of the name nodes 210-1, ..., 210- 210-k, ... 210-n may have a namespace. The cluster 20 may also have M data nodes 220-1 ... 220-k, ... 220-n and the data nodes 220-1 ... 220- ... 220-n can interwork with the name nodes.

Here, a name space (NS) is a conceptual area that can support create, delete, modify, and list of files and directories. can do.

In addition, block storage manages the membership of a data node, and it is a conceptual concept that supports and manages the creation, deletion and modification of blocks. It can mean area. Here, a block may mean a unit of divided data that is distributedly stored in a plurality of data nodes.

On the other hand, a plurality of sets of independent name nodes can be named " Name Node Federation ".

3 is a conceptual diagram for explaining a system for parallel-distributed processing large-scale data according to an embodiment of the present invention.

If the Name Node Federation 310 is implemented considering only the cluster environment in which the Hadoop cluster is constructed only as a physical node, a performance problem may occur in a virtualized Hadoop cluster environment having virtual nodes.

In other words, since the performance of the virtual nodes is most affected by the performance of the physical node, the virtual Hadoop cluster becomes a cluster of the heterogeneous environment in which the performance of the virtual nodes is different according to the performance of each physical node.

For example, when placing a job in the name node federation 310, it is possible to arrange a job assuming that the performance of all the name nodes 311 is the same.

However, when the name node federation 310 is applied in a virtual machine environment having heterogeneous computing characteristics, if assigning a job to a name node according to the conventional method due to the different performance of each name node, And you will see damage caused by.

A system for parallel distributed processing of large-scale data according to an embodiment of the present invention provides a technique for efficiently assigning jobs to a plurality of name nodes in a virtual cluster environment.

A system for parallelly distributing large-scale data may include a name node federation 310 and a name node control unit 330.

The name node federation 310 may be composed of a set of at least two name nodes included in the virtualization cluster.

Here, the virtualization cluster may be composed of a plurality of virtual nodes existing in each of a plurality of physical machines, and each of the name nodes may be located in each of a plurality of physical machines.

The name node control unit 330 determines a name node to allocate a job among at least two name nodes considering the performance or processing speed of at least two name nodes included in the name node federation 310 .

In detail, two name nodes among at least two name nodes are selected randomly, and a name node to which a task is allocated among two selected name nodes is determined in consideration of performance and workload of the selected two name nodes .

Referring to FIG. 3, a system for parallel processing large-scale data based on the Hadoop Distributed File System (HDFS) will be described.

In FIG. 3, each physical machine 30 may include one Hadoop name node 311 and a plurality of Hadoop data nodes 321. In addition, the name node federation 310 can be configured as a set of Hadoop name nodes 311 included in each physical machine 30.

The name node control unit 330 can control and manage the Hadoop name nodes 311 included in the name node federation 310. [

The name node controller 330, which may be called a High Ability NameNode, has information on the entire namespace and the entire block pool, and can perform overall management and control of the cluster.

For example, when a plurality of jobs are executed, the name node control unit 330 selects the Hadoop name node 311 considering the performance of the physical nodes and the current work amount, 311) can also be assigned a job.

The Hadoop name node 311 exists for each physical node and can have information on the blocks of the virtual Hadoop data nodes 321 in the physical node and individual namespaces.

An example in which the name node control unit 330 assigns a job to a Hadoop name node will be described below.

1) Select all the name nodes sequentially - (first technique)

According to the first technique, jobs can be sequentially allocated to all the name nodes. The first technique is to place jobs in the node node federation of the existing physical environment. If this method is applied to a virtualized Hadoop cluster, the performance of the whole cluster may be degraded due to the imbalance of Hadoop NameNodes having different performance have.

2) Selection based on performance and workload of all name nodes - (second technique)

According to the second technique, jobs can be allocated by checking the performance and workload of all the name nodes. The second technique will consistently maintain consistency of performance, but it will incur costs to check the performance and workload of Hadoop NameNodes, which may cause performance degradation.

3) randomly select one of all the name nodes - (third technique)

According to the third technique, one name node among all the name nodes can be randomly selected and assigned a job. The third technique does not incur the same cost and performance as the second technique. However, as in the first technique, the performance of the entire cluster may be degraded due to the unbalance of the Hadoop NameNodes.

4) Randomly selecting 2 of all the name nodes and finally selecting one considering performance and workload - (4th technique)

According to the fourth technique, two name nodes among all the name nodes are randomly selected, and the performance and the workload of the selected two name nodes are checked to determine the final name node and assign a job. The fourth technique is a consensus on the consistency of performance and the check cost, which are the problems raised in the first to third techniques described above, and can show a consistent performance to some extent, and the check cost can be greatly reduced.

4 is a flowchart illustrating a method of assigning a name node in a virtual cluster environment according to an embodiment of the present invention.

Referring to FIG. 4, a method of assigning a name node in a virtualized cluster environment according to an embodiment of the present invention may include receiving a job and determining a name node to which a task is assigned have.

First, a job allocated from a client can be received (S410).

A name node to assign a task among at least two name nodes may be determined in consideration of the performance of at least two name nodes included in the virtualization cluster or the processing speed of the task.

Here, the virtualization cluster may consist of a plurality of virtual nodes in each of a plurality of physical machines, and each of the at least two name nodes may be located in each of a plurality of physical machines.

In operation S420, two name nodes among at least two name nodes may be randomly selected.

Among the selected two name nodes, a name node to which a job is to be assigned can be determined (S430). That is, considering the performance and workload of the selected two name nodes, it is possible to finally determine the name node to which the job is to be allocated among the two selected names.

Finally, the task can be assigned to the finally determined name node (S440).

In addition, the naming node allocation method can be applied based on the HADOP Distributed File System (HDFS), but is not limited thereto.

5 is a graph illustrating a frequency of a name node to which tasks are allocated in a virtualization cluster to which a plurality of name nodes are applied according to an embodiment of the present invention.

That is, FIG. 5 is a graph showing the simulation results that can compare the performance of the four techniques described above.

The assumptions for the simulation are as follows.

A) Virtualization Hadoop cluster has 10 Hadoop NameNodes, 5 namesnodes 1 through 5 are poor performance group, and 5 namesnodes 6 to 10 are distinguished as good group B .

B) The name node controller selects a Hadoop NameNode according to the above-mentioned four techniques to distribute a total of 100 jobs. And it calculates the time that all jobs in the cluster are processed and finished.

C) Assume that Group A's Hadoop NameNode takes two seconds to process one job, and Group B's Hadoop NameNode takes one second to process one job.

D) Assume the cost of checking the workload and performance of the Hadoop NameNode is 0.01 seconds per job.

FIG. 5A shows the frequency with which a job is assigned to a name node in accordance with a first technique, FIG. 5B shows a frequency with which a job is assigned to a name node in accordance with a second technique, FIG. 5D shows the frequency with which a job is assigned to a name node when the fourth technique is used, and FIG. 5D shows a frequency with which a job is assigned to a name node. Here, the third technique and the fourth technique in which the random selection is performed were performed three times in total.

Referring to FIG. 5A, regardless of the performance of the name node, jobs are uniformly allocated to all the name nodes, and the performance of the entire cluster may be degraded.

Referring to FIG. 5B, it can be seen that the job assignment frequency is low in the group A with low performance. However, the cost for checking the performance and workload of the name nodes occurs, and the performance degradation may occur.

Referring to FIG. 5c, it can be seen that a job is assigned to a name node inconsistently. That is, irrespective of the performance of the name node, a job is randomly assigned, so that the performance of the entire cluster may be deteriorated.

Referring to FIG. 5D, it can be seen that the frequency of assigning jobs to the low performance group A is low. In addition, since the performance and the workload of two randomly selected name nodes are compared, it is expected that the added cost will not be large.

6 is a graph for comparing the processing time of a task when a method of assigning a name node is applied in a virtual cluster environment according to an embodiment of the present invention.

FIG. 6 shows the time required to process 100 jobs according to the above-described four techniques based on the assumptions described above with reference to FIG.

Referring to FIG. 6, it can be seen that the shortest processing time is required according to the fourth technique, and the longest processing time is required according to the third technique.

In other words, the fourth technique is a method of compromising the consistency of performance and the check cost, and it has a merit that it can show a consistent performance to some extent and the check cost can be greatly reduced.

In the virtual cluster environment according to an embodiment of the present invention, a method for assigning a name node and a system for parallelly distributing large-scale data can effectively select an optimal name node in a virtual cluster environment utilizing a plurality of name nodes . That is, there is an advantage that consistent performance can be maintained while minimizing the check of the performance and the workload of the name node.

Therefore, the present invention can contribute to the overall performance improvement of the virtualization cluster.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the present invention as defined by the following claims It can be understood that

Claims (10)

In a name node assignment method,
Receiving a job; And
Determining a name node to which the task is to be allocated among the at least two name nodes in consideration of the performance of at least two name nodes included in the virtualization cluster or the processing speed for the task, / RTI > The method according to claim 1,
The virtual cluster includes:
And a plurality of virtual nodes existing in each of the plurality of physical machines. The method according to claim 1,
Wherein each of the at least two name nodes comprises:
Wherein the plurality of physical machines are located one by one in each of the plurality of physical machines. The method according to claim 1,
The method of claim 1,
Wherein the method is applied based on a Hadoop Distributed File System (HDFS). The method according to claim 1,
Wherein the step of determining a name node to which the task is assigned comprises:
Randomly selecting two name nodes out of the at least two name nodes,
Wherein a name node to which the job is to be allocated is determined from among the selected two name nodes in consideration of performance and workload of the selected two name nodes. In a system for processing large-scale data,
Node node federation consisting of a set of at least two name nodes included in a virtualization cluster; And
And a name node control unit for determining a name node to which the job is to be allocated among the at least two name nodes in consideration of a performance or a processing speed of a job of the at least two name nodes included in the name node federation A system that distributes large-scale data in parallel. The method of claim 6,
The virtual cluster includes:
And a plurality of virtual nodes existing in each of a plurality of physical machines. The method of claim 6,
Wherein each of the at least two name nodes comprises:
Wherein the plurality of physical machines are located one by one in each of the plurality of physical machines. The method of claim 6,
A system for parallel-distributing large-scale data,
A system for parallel distributed processing of large-scale data characterized by being based on the Hadoop Distributed File System (HDFS). The method of claim 6,
The name node control unit,
Randomly selecting two name nodes out of the at least two name nodes,
And determines a name node to which the job is to be allocated among the selected two name nodes considering performance and workload of the selected two name nodes.

Images