KR102167791B1 - Apparatus for Managing In-memory and Computer-Readable Recording Medium with Program therefor - Google Patents

Abstract

In a device that manages an in-memory layer in a distributed processing system that includes one name node storing block mapping information and a plurality of data nodes that distribute and store large-capacity data, it is required to store data blocks in in-memory. A request receiving unit for receiving an in-memory request as a requester; A mapping information transmission unit for analyzing the in-memory request and obtaining mapping information for the data block; And a storage unit for storing the data block as an in-memory by receiving an in-memory command to store the data block as an in-memory from a client receiving the mapping information, and a computer therefor. Provides a recording medium that can be read by.

Description

In-memory layer management device and computer-readable recording medium therefor {Apparatus for Managing In-memory and Computer-Readable Recording Medium with Program therefor}

The present embodiment relates to an in-memory layer management apparatus and a computer-readable recording medium therefor.

The content described in this section merely provides background information on the present embodiment and does not constitute the prior art.

Existing Hadoop-based data processing depends on the Hadoop Distributed File System (HDFS), a Hadoop-based system. When data to be processed by the Hadoop system is executed through several steps, intermediate files are written to HDFS at the end of each step, causing disk I/O to affect the performance of the entire Hadoop system. Can give
[Prior technical literature] Korean Patent Application Publication 10-2012-0085400 (2012.08.01)

The main object of this embodiment is to reduce disk I/O and improve processing speed by implementing distributed in-memory in software and storing intermediate files in in-memory.

Its purpose is to provide an environment in which a query language similar to SQL (Structured Query Language) can be used, and to dynamically create various data schemas.

According to an aspect of the present embodiment, there is provided an apparatus for managing an in-memory layer in a distributed processing system including one name node for storing block mapping information and a plurality of data nodes for distributing and storing large-capacity data. A request receiving unit for receiving an in-memory request, which is a request to store a block in the in-memory; A mapping information transmission unit for analyzing the in-memory request and obtaining mapping information for the data block; And a storage unit for storing the data block as an in-memory by receiving an in-memory command for storing the data block as an in-memory from a client receiving the mapping information. Provides.

According to another aspect of the present embodiment, there is provided a process of receiving an in-memory request, which is a request to store a data block in an in-memory, to a computer; Analyzing the in-memory request to obtain mapping information for the data block; And a computer-readable recording in which a program for executing a process of storing the data block as an in-memory by receiving an in-memory command to store the data block in in-memory from a client receiving the mapping information. Provide the medium.

As described above, according to the present embodiment, it is possible to increase processing efficiency by loading a small file that is frequently accessed into the memory, and loads commonly required data in the memory when processing all data such as common code and meta information. There is an effect of increasing the efficiency.

In addition, when an application is implemented so that an intermediate file that occurs in the middle of data processing can be loaded into a memory, disk I/O is reduced, which improves the overall processing speed of the application.

1 is a block diagram illustrating an HDFS system including an in-memory layer management apparatus 100 according to an embodiment of the present invention.
2 is a block diagram illustrating a system including the in-memory layer management apparatus 100 according to an embodiment of the present invention.
3 is a flowchart illustrating an in-memory layer management method according to another embodiment of the present invention.

Hereinafter, some embodiments of the present invention will be described in detail through exemplary drawings. In describing an embodiment of the present invention, when it is determined that a detailed description of a related known configuration or function may obscure the subject matter of the present invention, a detailed description thereof will be omitted.

1 is a block diagram illustrating an HDFS system including an in-memory layer management apparatus 100 according to an embodiment of the present invention.

As shown in FIG. 1, the HDFS system 10 generally includes a plurality of data nodes 11, 12, and 13 that distribute and store large amounts of data, and data nodes 11 and 12 in which data blocks are stored. It is implemented by including one name node 14 and an app client 130 that stores mapping information, which is information on the location of 13).

The in-memory layer management apparatus 100 may operate within one data node (eg, data node 1) among a plurality of data nodes 11, 12, and 13, but is not limited thereto. It can also be operated distributed across nodes.

2 is a block diagram illustrating a system including the in-memory layer management apparatus 100 according to an embodiment of the present invention.

The in-memory layer management apparatus 100 according to an embodiment of the present invention includes a request receiving unit 111, a mapping information transmitting unit 112, a command transmitting unit 130, a storage unit 140, and an in-memory client ( 150) may be implemented. According to the embodiment, the in-memory layer management apparatus 100 includes a request receiving unit 111, a mapping information transmitting unit 112, a command transmitting unit 130, a storage unit 140, and an in-memory client 150. Some of the elements may be omitted or may be implemented by additionally including other elements.

The in-memory layer management apparatus 100 is implemented to be executed in the Hadoop distributed processing system 10 including one name node that stores mapping information of a block and a plurality of data nodes that distribute and store large amounts of data.

The request receiving unit 111 receives an in-memory request, which is a request to store a data block in an in-memory. The in-memory request is generated by a user input from the command transmission unit 130 using a command line interface (CLI).

The mapping information transmission unit 112 analyzes the in-memory request received from the request reception unit 111 to obtain mapping information for a data block to be stored, and transmits it to the command transmission unit 130.

The command transmission unit 130 generates an in-memory request transmitted to the request reception unit 111, receives mapping information from the mapping information transmission unit 112, accesses a data node corresponding to the received mapping information, and A command to store the block in the in-memory is transmitted to the storage unit 140. Here, the command transmission unit 130 may be an app client that exists outside the in-memory layer management apparatus 100, or may be another application executed within the HDFS 10, and in the following description, the command transmission unit ( 130) instead of the app client 130 will be described as an example.

Here, the request receiving unit 111 and the mapping information transmitting unit 112 may be implemented as a single driver 110 module.

The storage unit 140 receives a command to store the data block in the in-memory from the app client 130 that has received the mapping information, and stores the data block in the in-memory. Here, the storage unit 140 may be stored in a data node different from the request receiving unit 111 and the mapping information transmitting unit 112.

When the corresponding meta information is not stored, the mapping information transmission unit 112 requests the corresponding meta information from the name node 14 to obtain the mapping information.

For reference, when the mapping information transmission unit 112 requests the meta information from the name node 14 to obtain the mapping information, the driver 110 stores the meta information in the in-memory and stores the meta information next time. When the information is needed, use the one stored in memory.

The app client 130 accesses the driver 110 of the in-memory layer management device 100 existing in any one node 11 to receive information necessary for in-memory setting, and the app client 130 An in-memory area is set by sending a command to the storage unit 140 existing in one data node, and data is stored in the set in-memory area. For example, the driver 110 and the storage unit 140 may exist in the same data nodes 11, 12, and 13 or may exist in different data nodes 11, 12, and 13.

In HDFS, YARN (Yet Another Resource Negotiator) is generally used as a resource manager. Hadoop YARN framework is a resource manager that allocates resources for each application in Hadoop and distributes resources according to rules. Hadoop applications are implemented in the form of sending commands to YARN to allocate or release resources such as memory or storage space.

The in-memory command generated by the app client 130 is written in the HDFS system 10 during the execution of a specific application, so that the specific data generated as an intermediate file on the disk of the data node on the HDFS is together in a specific in-memory area. It could be an order to save. In this case, the specific in-memory area in which this intermediate file is stored is reserved in advance. In order to secure space in the in-memory area, a specific in-memory area is secured by designating an intermediate file created by a specific application to be stored in a specific in-memory area.

Here, the generated intermediate file may be generated for the next use of the application during the operation of one application, or another application may use the intermediate file generated by one application during the execution of a plurality of applications. .

The in-memory client 150 allows in-memory access to any one application created in the HDFS distributed processing system. For example, one in-memory client 150 is generated for each data node accessed by one application.

The in-memory client 150 communicates with the driver 110 to determine in which data node the in-memory used by the application is located, and directly communicates with the corresponding data node to obtain necessary in-memory data. If there is no desired data in the in-memory data area, it is read from HDFS, cached, and stored in in-memory. Here, the expression that the in-memory client 150 communicates with the data node is used, but actually functions to communicate with a worker for in-memory management in the data node. In addition, in the present embodiment, even when a data node and a certain operation are described as being performed, a worker performing the corresponding operation may be provided in the data node.

In order to secure a specific in-memory space in a certain data node, when a command to secure an in-memory space for the data node is issued from the app client 130, the in-memory space may be secured by requesting YARN. . These commands may be generated by a user command, or may be generated by any running application.

As a method of loading specific data into the in-memory, it may be set to be stored in the in-memory space when specific data (eg, meta data) is generated by an application, such as when storing an intermediate file. In addition, when a command to load a specific HDFS storage block into the secured in-memory space is generated by a user command, the mapping information for the corresponding block is obtained through the driver 110 to the worker of the data node managing the block. You can also issue a memory load command.

On the other hand, when a certain data block is stored in the in-memory, not only one data node may be repeatedly stored in another data node. When an in-memory area is allocated, the number of copies and an area release period for the area can be set. Allocation and deallocation of the in-memory area can be allocated and deallocated on a file or directory basis.

Meanwhile, in order to manage the capacity of the in-memory area, periodically or aperiodically, the app client 130 sends a resource monitoring command (or query) to the driver 110. When the driver 110 receives the resource monitoring command, it transmits the corresponding resource monitoring result to the app client 130. When the app client 130 determines that the available in-memory capacity is insufficient, it requests the YARN to additionally allocate a certain amount of in-memory. Here, the command to be additionally allocated may receive a command by the user, and in some cases, when a resource is insufficient while monitoring a resource by a specific daemon called the resource management unit 160 (e.g., in-memory When the usable capacity is less than the threshold), additional resources may be allocated from YARN by automatically providing an in-memory request command for requesting additional allocation of in-memory of a preset size to YARN.

When the driver 110 is driven for the first time, it informs the driver ID to YARN and requests a resource. The driver 110 sets information on how much memory to allocate for each worker and the number of workers. Workers can be distributed across multiple data nodes.

The app client 130 sends a driver id to YARN, receives the driver address information from YARN, communicates with the corresponding driver 130, and transmits a query to the driver 130.

3 is a flowchart illustrating an in-memory layer management method according to another embodiment of the present invention.

A dynamic schema query processing method according to another embodiment of the present invention includes an in-memory request generation process (S310), an in-memory request reception process (S320), a mapping information generation process (S330), and an in-memory command transmission process ( S340) and an in-memory storage process (S350).

In the process of generating an in-memory request (S310), a user or an application generates an in-memory request to store a data block in the in-memory.

In the in-memory request receiving process (S320), an in-memory request, which is a request to store a data block in the in-memory, is received.

In the mapping information generation process (S330), the in-memory request is analyzed to obtain mapping information for a data block to be stored, and it is transmitted to the app client 130.

In the in-memory command transmission process (S340), the app client 130 receives the mapping information, accesses the data node corresponding to the received mapping information, and asks to store the data block as an in-memory together with information on the data block to be stored. Send in-memory commands.

In the in-memory storage process (S350), the data block is stored in the in-memory storage space by receiving the in-memory command.

The operations in the in-memory request generation process (S310) and the in-memory command transmission process (S340) correspond to the operation of the app client 130, and the in-memory request reception process (S320) is performed by the request receiving unit 111 Since the operation corresponds to the operation, the mapping information generation process (S330) corresponds to the operation of the mapping information transmission unit 112, and the in-memory storage process (S350) corresponds to the operation of the storage unit 150, further detailed description Omit it.

In FIG. 3, it is described that processes S310 to S350 are sequentially executed, but this is merely illustrative of the technical idea of an embodiment of the present invention. In other words, a person of ordinary skill in the art to which an embodiment of the present invention belongs can change the order described in FIG. 3 and execute one or more of the processes S310 to S350 without departing from the essential characteristics of the embodiment of the present invention. Since the process is executed in parallel and can be applied by various modifications and variations, FIG. 3 is not limited to a time series order.

The in-memory layer management apparatus 100 according to an embodiment of the present invention executes a communication device such as a communication modem for performing communication with various devices or wired/wireless communication networks, a memory for storing data for executing a program, and a program. Thus, it may mean various devices including a microprocessor for calculation and control. According to at least one embodiment, the memory is a computer such as a random access memory (RAM), a read only memory (ROM), a flash memory, an optical disk, a magnetic disk, or a solid state disk (SSD). It may be a readable recording/storing medium. According to at least one embodiment, the microprocessor may be programmed to selectively perform one or more operations and functions described in the specification. According to at least one embodiment, the microprocessor may be implemented entirely or partially as hardware such as an Application Specific Integrated Circuit (ASIC) having a specific configuration.

As described above, the in-memory layer management method described in FIG. 3 may be implemented as a program and recorded on a computer-readable recording medium. A recording medium in which a program for implementing the in-memory layer management method according to the present embodiment is recorded and the computer-readable recording medium includes all kinds of recording devices in which data that can be read by a computer system is stored. Examples of such computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. In addition, the computer-readable recording medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the present embodiment may be easily inferred by programmers in the technical field to which the present embodiment belongs.

The above description is merely illustrative of the technical idea of the present embodiment, and those of ordinary skill in the technical field to which the present embodiment belongs will be able to make various modifications and variations without departing from the essential characteristics of the present embodiment. Accordingly, the present exemplary embodiments are not intended to limit the technical idea of the present exemplary embodiment, but are illustrative, and the scope of the technical idea of the present exemplary embodiment is not limited by these exemplary embodiments. The scope of protection of this embodiment should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present embodiment.

10: HDFS system
11: data node 1
12: data node 2
13: data node n
14: Name node
100: in-memory layer management device
110: driver
111: request receiver
112: mapping information transmission unit
130: command transmission unit, app client
140: storage unit
150: in-memory client

Claims (11)

In a distributed file system that includes a name node that stores mapping information, which is information about the location of a data node where data blocks are stored, and a plurality of data nodes that distribute and store large amounts of data, In the device for managing the memory layer,
A request receiving unit for receiving an in-memory request, which is a request to store the data block in an in-memory;
A mapping information transmission unit that obtains the mapping information from the name node based on the in-memory request; And
An in-memory command to receive the mapping information from the mapping information transmission unit, access a first data node, which is a data node corresponding to the mapping information among the plurality of data nodes, to store the data block as an in-memory Command transmission unit for transmitting to the first data node
In-memory layer management apparatus comprising a. delete The method of claim 1,
Wherein the first data node includes a storage unit for receiving the in-memory command and storing the data block as an in-memory. delete The method of claim 1,
Wherein the in-memory command is a command to store specific data to be written in the distributed file system in an in-memory area. The method of claim 5, wherein when it is designated to store an intermediate file generated by one application among applications operating in the distributed file system in the in-memory area, the intermediate file is stored in the in-memory area. In-memory layer management device. The method of claim 1,
And an in-memory client for accessing the in-memory for an application generated in the distributed file system. The method of claim 1,
Including a driver module including the request receiving unit and the mapping information transmission unit,
The driver module receives a request for an in-memory area for storing data as an in-memory from a user and transmits a request for the in-memory area to a resource management device of the distributed file system, and the in-memory area In-memory layer management device, characterized in that the allocation of. The method of claim 8,
When the available capacity of the in-memory area is less than or equal to a threshold, the in-memory layer management apparatus further comprises a resource management unit for additionally allocating an in-memory area. delete On the computer,
In a distributed file system that includes a name node that stores mapping information, which is information about the location of a data node where data blocks are stored, and a plurality of data nodes that distribute and store large amounts of data, In a method for a device managing a memory layer to manage an in-memory layer,
Receiving an in-memory request, which is a request to store the data block in in-memory, from a user;
Obtaining the mapping information from the name node based on the in-memory request; And
Receiving the mapping information, accessing a first data node corresponding to the mapping information among the plurality of data nodes, and sending an in-memory command to store the data block as an in-memory to the first data node Transfer process
A recording medium that can be read by a computer on which a program to run is recorded.

Images