KR101638727B1 - Cluster system - Google Patents

Abstract

Disclosed is a cluster system. The cluster system, which includes a plurality of nodes and is used for combined usage of a Unix file system and a distribution file system, comprises a data management part. When there is a request to input and output data of the Unix file system, the data management part accesses each of the nodes and searches and provides a required data in a disc cache. The cluster system is able to reduce costs for a storage system.

Description

CLUSTER SYSTEM {CLUSTER SYSTEM}

The present invention relates to a cluster system, and more particularly, to a cluster system in which a Unix system and a distributed file system are mixed.

With the recent development of cloud computing technology, users can apply high-performance computing tasks simply by applying services without having to directly access and control high-performance cluster system. Hadoop is one of the most popular cloud computing related technologies, enabling parallel distributed processing and building a high-performance cluster environment using relatively inexpensive servers. However, in order to utilize MapReduce, a parallel distributed processing method supported by Hadoop, it is necessary to convert the programs and tools that can be executed on existing Linux / Unix (Linux / Unix) into the Hadoop method capable of MapReduce, And karate are needed.

As a result, existing Linux / Unix programs do not need to be modified in the current way, and developing newly developed programs as Hadoop programs using MapReduce is the ideal form of cloud system using Hadoop. To do this, we need to share HDFS, a file storage system from Hadoop, into a file storage system that can be used by general Linux / Unix. At this time, local storage on multiple individual nodes is recognized as a combined storage and shared There is a problem that the data input / output speed is remarkably lowered.

SUMMARY OF THE INVENTION The present invention provides a cluster system capable of mixing a cluster system using a distributed file system such as a Hadoop Distributed Fil System (HDFS) and a general cluster system such as Linux / Unix .

Another object of the present invention is to provide a cluster system capable of preventing a general cluster system from accessing a distributed file system and slowing down data input / output.

The present invention also provides a cluster system capable of reducing the cost of a storage system by integrating storage systems of a general cluster system and a distributed file system.

Another object of the present invention is to provide a cluster system suitable for large data analysis by using a common cluster system and a distributed file system.

According to an aspect of the present invention, there is provided a cluster system for a mixed use of a UNIX file system and a distributed file system including a plurality of nodes, wherein each of the nodes accesses the plurality of nodes when requesting data input / output of the UNIX file system, And a data management unit for searching and providing the cluster system.

Wherein the data management unit comprises: a retrieval unit for retrieving the required data from the disk cache of the plurality of nodes in response to a data input / output request of the UNIX file system; A data request unit for requesting data to the distributed file system when the required data is not retrieved from the disk cache of the plurality of nodes; An update unit for receiving data from the distributed file system and updating the data in the disk cache; And a data processor for collecting the required data from the disk cache and providing the collected data to the UNIX file system.

The search unit may search the required data using index information provided by at least one of the plurality of nodes.

The update unit may generate a data list according to the priority and compare the disc cache data information to update the missing data.

The update unit may create a data list according to the priority using at least one of a file use date, a file use frequency, a file creation date, and a file modification date.

The cluster system of the present invention can be used with a general cluster system such as Linux / Unix and a distributed file system such as Hadoop Distributed Fil System (HDFS) It is possible to prevent a speed reduction in data input / output by accessing the system.

In addition, by integrating the storage system of the general cluster system and the Hadoop distributed file system, the cost of the storage system can be reduced, and it is suitable for the big data analysis by using the general cluster system and the Hadoop distributed file system.

1 is a conceptual diagram of a cluster system according to an embodiment of the present invention;
2 is a block diagram of a data management unit according to an embodiment of the present invention.
3 is a flowchart illustrating an operation of a data management unit according to an embodiment of the present invention.
4 is a flowchart illustrating an operation of the update unit according to an embodiment of the present invention;
5 to 7 are diagrams for comparing performance of a cluster system and an existing cluster system according to an embodiment of the present invention.

The present invention is capable of various modifications and various embodiments, and specific embodiments are illustrated and described in the drawings. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

The terms including ordinal, such as second, first, etc., may be used to describe various elements, but the elements are not limited to these terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the second component may be referred to as a first component, and similarly, the first component may also be referred to as a second component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In the present application, the terms "comprises" or "having" and the like are used to specify that there is a feature, a number, a step, an operation, an element, a component or a combination thereof described in the specification, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the contextual meaning of the related art and are to be interpreted as either ideal or overly formal in the sense of the present application Do not.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings, wherein like or corresponding elements are denoted by the same reference numerals, and redundant description thereof will be omitted.

FIG. 1 is a conceptual diagram of a cluster system according to an embodiment of the present invention, and FIG. 2 is a configuration block diagram of a data management unit 100 according to an embodiment of the present invention.

1 and 2, a cluster system according to an embodiment of the present invention includes a distributed file system 10 and a UNIX file system 20 that distribute and store data to a plurality of nodes 31 to 35 including a disk cache, And a data management unit 100 for retrieving and providing required data from the (31-35) disk cache of a plurality of nodes in response to a data input / output request of the node.

The cluster system according to an embodiment of the present invention includes a UNIX file system 20 based on a common file system of a shared type such as Linux / Unix and a distributed file system 10 such as a Hadoop system They are mixed and run through programs or tools under each operating system.

The plurality of nodes 31 to 35 distribute and store data. The plurality of nodes 31 to 35 may include a storage unit and a data cache. The plurality of nodes 31 to 35 may function as a master node. The data cache may include a solid state drive (SSD) ), A hard disk drive (HDD), or the like. In an embodiment of the present invention, the data cache can be used as a concept of a buffer for preloading data in a storage unit.

The distributed file system 10 such as the Hadoop file system (HDFS) distributes and stores data redundantly to a plurality of nodes 31 to 35. When receiving a data input / output request from the cluster system, the nodes 31 to 35 are accessed The requested action can be handled. In this case, by processing the corresponding data in the node where each data is stored, the effect of distributed processing can be enhanced while minimizing file input / output.

The data management unit 100 may include a search unit 110, a data request unit 120, an update unit 130, and a data processing unit 140.

The retrieving unit 110 retrieves required data from the disk cache of the plurality of nodes 31 to 35 in response to a data input / output request of the UNIX file system 20. [ At this time, the search unit 110 can search for required data by using the index information provided by at least one of the nodes 31 to 35. The index information is a list of data existing in the disk cache, for example, can be provided through a node selected as a master node among the plurality of nodes 31 to 35.

The data requesting unit 120 can request data from the distributed file system 10 when required data can not be retrieved from the disk cache of the plurality of nodes 31 to 35. [

The update unit 130 receives data from the distributed file system 10 and can update the disk cache. Upon receiving data from the distributed file system 10 at the request of the data requesting unit 120, the updating unit 130 copies the data to the disk cache and corrects the data list of the index information.

In addition, the update unit 130 may periodically update data in the disk cache. The update unit 130 may generate a data list according to the priority and compare the data list with the data information of the disk cache to update the missing data. At this time, the updating unit 130 may compare the index information of the master node to extract the missing data.

The updating unit 130 may generate a data list according to the priority by using at least one of a file use date, a file use frequency, a file creation date, and a file modification date. The updating unit 130 may create a priority list in the order of recently used files, frequently used files, first generated files, recently modified files, etc. using the listed elements, You can also create a ranking list.

3 is a flowchart illustrating an operation of the data management unit 100 according to an embodiment of the present invention.

First, the data management unit 100 checks whether there is a data input / output request from the UNIX file system 20.

When the data input / output request is confirmed from the Unix file system 20, the data management unit 100 accesses the disk caches included in the plurality of nodes to check whether the required data exists. At this time, the data management unit 100 can check whether there is necessary data in the disk cache by using the index information provided by the master node.

The data management unit 100 extracts the required data if the required data exists in the disk cache, and provides the extracted data to the UNIX file system 20.

The data management unit 100 requests the distributed file system 10 for the required data if the required data does not exist in the disk cache.

The data management unit 100 copies the required data requested from the distributed file system 10, stores the copied data in the disk cache, and modifies the data list of the disk cache.

4 is a flowchart illustrating an operation of the update unit 130 according to an embodiment of the present invention.

First, the update unit 130 checks whether a period for performing an update according to a preset time has arrived.

The update unit 130 generates a priority file list when it is determined that the update period has arrived.

The updating unit 130 compares the priority file list with the data information of the disk cache to check whether there is missing data.

The update unit 130 copies the data from the distributed file system 10 according to the priority file list and stores the data in the disk cache when the missing data exists. The update unit 130 updates the data list of the disk cache after the disk cache update.

5 to 7 are diagrams for comparing performance of a cluster system and an existing cluster system according to an embodiment of the present invention.

5 to 7, the NFS3 gateway refers to a cluster system in which a distributed file system is used in the form of an NFS to mix a UNIX file system 20 and a distributed file system 10. OpenBio is an embodiment of the present invention Quot; cluster system " The hit rate of OpenBio means the ratio of data to disk cache. If hit rate is 0%, there is no data in disk cache. If Hit rate is 100% Are all present.

Referring to FIG. 5, it can be seen that the time consumed for data output in all the data intervals of 5G to 50G is reduced as compared with the conventional cluster system.

Referring to FIG. 6, it can be seen that the performance improvement ratio with time decreases. When the Hit rate is 0%, the improvement rate is close to 200 to 380%. When the Hit rate is 100%, the improvement rate is about 480% To 810%.

FIG. 7 is a graph showing time and performance improvement rates of a conventional cluster system and a cluster system according to an embodiment of the present invention, according to data capacity.

As used in this embodiment, the term " portion " refers to a hardware component such as software or an FPGA (field-programmable gate array) or ASIC, and 'part' performs certain roles. However, 'part' is not meant to be limited to software or hardware. &Quot; to " may be configured to reside on an addressable storage medium and may be configured to play one or more processors. Thus, by way of example, 'parts' may refer to components such as software components, object-oriented software components, class components and task components, and processes, functions, , Subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. The functions provided in the components and components may be further combined with a smaller number of components and components or further components and components. In addition, the components and components may be implemented to play back one or more CPUs in a device or a secure multimedia card.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the present invention as defined by the following claims It can be understood that

10: Distributed File System
20: Unix file system
30: node
100:

Claims (5)

CLAIMS What is claimed is: 1. A cluster system including a plurality of nodes and for mixing a UNIX file system and a distributed file system,
And a data manager for accessing the plurality of nodes in parallel when a data input / output request of the UNIX file system is requested, and retrieving and providing required data from the disk cache,
The data management unit,
A retrieval unit retrieving the required data from the disk cache of the plurality of nodes in response to a data input / output request of the UNIX file system;
A data request unit for requesting data to the distributed file system when the required data is not retrieved from the disk cache of the plurality of nodes;
An update unit for receiving data from the distributed file system and updating the data in the disk cache; And
And a data processor for collecting the required data from the disk cache and providing the collected data to the UNIX file system.
delete The method according to claim 1,
Wherein the searching unit searches the required data using index information provided by at least one of the plurality of nodes.
The method according to claim 1,
Wherein the update unit creates a data list according to the priority and compares the data list with the required data of the disk cache to update the missing data.
5. The method of claim 4,
Wherein the update unit creates a data list according to the priority using at least one of a file use date, a file use frequency, a file creation date, and a file modification date.

Images