KR101772108B1 - Memory management system and method based on spark - Google Patents

Abstract

The present invention relates to a system to manage a memory based on a spark environment, capable of solving a problem of performance deterioration severely occurring in memory shortage in a spark environment. According to the present invention, the system comprises: a master to execute a driver process dividing a job into a plurality of tasks and allocating the tasks; and a worker node to execute an execution process executing the task allocated by the driver process and transmitting an execution result to the driver process. The worker node includes a worker manager to determine whether or not bottleneck is generated in the spark environment, to perform a process solving generation of the bottleneck in accordance with a determination result, and to transmit change information changed as the process is performed to the master. The master includes a driver manager to update the change information received from the worker manager to the driver process.

Description

[0001] SPARK-BASED MEMORY MANAGEMENT SYSTEM AND METHOD [0002]

The present invention relates to a spark-based memory management system and method.

Recently, Spark, which is used as a big data distribution processing platform, is attracting much attention because it stores the RDD dataset in the memory resource and improves the speed by reading / writing at the memory speed.

RDD is an abbreviation of Resilient Distributed Dataset. It has the advantage of not having to read / write data to a disk because it can not be modified once it is created and stored in memory, and has good locality in repeated big data distribution processing. .

These sparks are computed with in-memory and use RDD data structures to address performance degradation caused by iterative operations in Hadoop.

However, in a spark, when a large amount of data is stored and processed in a limited memory capacity in a cluster configuration, a bottleneck occurs when the memory becomes insufficient, and the bottleneck causes a severe performance drop in the spark.

The background technology of the present application is disclosed in Korean Patent Laid-Open Publication No. 2015-0089538 (Publication Date: 2015.08.05).

SUMMARY OF THE INVENTION It is an object of the present invention to provide a spark-based memory management system and method capable of solving the problem of a performance degradation which is severe when a memory is short in a spark environment.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a spark-based memory management system and method capable of improving spark performance.

It is to be understood, however, that the technical scope of the embodiments of the present invention is not limited to the above-described technical problems, and other technical problems may exist.

According to an aspect of the present invention, there is provided a memory management system in a spark environment, comprising: a master for executing a driver process for dividing and assigning a job to a plurality of tasks; Wherein the worker node determines whether or not the bottleneck phenomenon occurs in the spark environment, and transmits the result of the execution to the driver process based on the determination result And a worker manager for performing processing for eliminating the occurrence of the bottleneck and for transmitting change information of the changed execution process to the master by performing the processing, wherein the master notifies the change information received from the walker manager Updating for the driver process That agent may include a driver manager.

Also, the execution process may include generating an RDD for performing the task and storing the RDD in a memory or a fast storage medium, wherein the worker manager moves some RDDs stored in the memory to the fast storage medium, Can be eliminated.

In addition, the walker manager can determine whether the bottleneck phenomenon has occurred by checking whether at least one of the full garbage collection and the shuffle spill has occurred at a predetermined cycle.

The walker manager may identify a first RDD to be transferred to the fast storage medium among the RDDs stored in the memory as the partial RDD based on a predetermined RDD replacement algorithm when it is determined that the full garbage collection has occurred, The first RDD may be copied and stored in the high-speed storage medium, and the first RDD stored in the memory may be deleted.

In addition, the walker manager may determine whether the predetermined hip size is exceeded in order to prevent the bottleneck due to the full garbage collection, and if the predetermined hip size is exceeded, The execution process may be controlled to be stored in the high-speed storage medium instead of the memory.

The walker manager may determine whether to increase the specific gravity of the shuffle space by determining whether the specific gravity of the shuffle space is less than a preset value in the memory when it is determined that the shuffle spill has occurred.

In addition, the walker manager may further include, when it is determined that the specific gravity of the shuffle space is less than the predetermined value, the RDD among the RDDs stored in the memory based on a preset RDD replacement algorithm to increase the specific gravity of the shuffle space The second RDD to be moved to the first RDD may be identified as the partial RDD, the second RDD may be copied and stored in the fast storage medium, and the second RDD stored in the memory may be deleted.

Also, the walker manager may increase the weight of the shuffle space by deleting the second RDD from the memory, and the maximum increase specific gravity value of the shuffle space that can be increased at a time may be set by user input.

In addition, the worker manager may move the partial RDD to the high-speed storage medium using a spark API.

The walker manager receives information of the RDD and information of the memory from the execution process and controls the storage location of the RDD and the setting of the memory by the execution process, The change information of the storage location of the RDD and the change information of the memory setting may be transmitted to the driver manager as change information of the execution process.

Meanwhile, a memory management method in a spark environment according to an embodiment of the present invention includes the steps of (a) executing a driver process included in a master to divide a job into a plurality of tasks and allocating them, (b) Executing a task executed by the driver process, and transmitting a result of the execution to the driver process, wherein the step (b) includes the steps of: (b1) The manager determines whether or not a bottleneck phenomenon occurs in the spark environment, performs processing for eliminating the occurrence of the bottleneck phenomenon according to the determination result, and transmits the changed change information of the execution process to the master (B2) the driver manager included in the master changes the change information received from the walker manager It may include the step of updating with respect to the driver process.

In the step (b), the execution process generates an RDD for performing the task and stores the RDD in a memory or a high-speed storage medium. In the step (b1), the worker manager registers RDD The occurrence of the bottleneck phenomenon can be solved by moving some RDDs to the high-speed storage medium.

In the step (b1), the walker manager can determine whether the bottleneck phenomenon has occurred by checking whether at least one of full garbage collection and shuffle spill has occurred at a predetermined period.

In the step (b1), if the walker manager determines that the full garbage collection has occurred, the first RDD to be transferred to the fast storage medium among the RDDs stored in the memory, based on the preset RDD replacement algorithm, It may be identified as some RDD, the first RDD may be copied and stored in the fast storage medium, and the first RDD stored in the memory may be deleted.

In the step (b), the walker manager may determine whether the predetermined heap size is exceeded in order to prevent the occurrence of the bottleneck due to the full garbage collection, and if the predetermined heap size The RDD may be controlled to be stored in the high-speed storage medium instead of the memory.

In the step (b1), when the walker manager determines that the shuffle spill has occurred, it determines whether the specific gravity (%) of the shuffle space is less than a predetermined value in the memory, Or not.

If it is determined in step (b1) that the weight of the shuffle space is less than the predetermined value, the walker manager may set the shuffle space in the memory based on a preset RDD replacement algorithm to increase the weight of the shuffle space The second RDD to be transferred from the stored RDD to the high-speed storage medium may be identified as the partial RDD, the second RDD may be copied and stored in the high-speed storage medium, and the second RDD stored in the memory may be deleted.

In the step (b1), the walker manager increases the specific weight of the shuffle space by deleting the second RDD from the memory, and the maximum increase specific gravity value of the shuffle space that can be increased at one time is It can be settable.

In the step (b1), the worker manager may move the partial RDD to the high-speed storage medium using a spark API.

In the step (b1), the walker manager receives information of the RDD and information of the memory from the execution process, and controls the storage location of the RDD by the execution process and the setting of the memory, The change information of the storage location of the RDD and the change information of the memory setting may be transmitted to the driver manager as change information of the execution process when it is determined that a change has occurred in the storage location of the RDD and the setting of the memory.

The above-described task solution is merely exemplary and should not be construed as limiting the present disclosure. In addition to the exemplary embodiments described above, there may be additional embodiments in the drawings and the detailed description of the invention.

According to the above-mentioned problem solving means of the present invention, the cause of bottleneck in the spark environment is identified and the bottleneck phenomenon confirmed is solved, thereby solving the factor of the decline in the performance which is drastically caused when the memory is short in the spark environment.

According to the above-mentioned problem solving means of the present invention, the performance of an in-memory distributed processing framework can be improved by using a high-speed storage medium in a spark environment.

According to the above-mentioned problem solving means of the present invention, by using a real-time analysis tool that does not affect the performance of the distributed framework, the spark performance can be improved more effectively.

FIG. 1 is a block diagram of a spark cluster system for applying a spark-based memory management system according to an embodiment of the present invention. Referring to FIG.
FIG. 2 is a diagram illustrating a schematic configuration of a spark-based memory management system according to an embodiment of the present invention.
FIG. 3 is a flowchart illustrating a process of eliminating a bottleneck phenomenon through a walker manager in a spark-based memory management system according to an embodiment of the present invention.
4A and 4B are schematic operation flowcharts of a spark-based memory management method according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. It should be understood, however, that the present invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In the drawings, the same reference numbers are used throughout the specification to refer to the same or like parts.

Throughout this specification, when an element is referred to as being "connected" to another element, it is intended to be understood that it is not only "directly connected" but also "electrically connected" or "indirectly connected" "Is included.

It will be appreciated that throughout the specification it will be understood that when a member is located on another member "top", "top", "under", "bottom" But also the case where there is another member between the two members as well as the case where they are in contact with each other.

Throughout this specification, when an element is referred to as "including " an element, it is understood that the element may include other elements as well, without departing from the other elements unless specifically stated otherwise.

FIG. 1 is a block diagram of a spark cluster system for applying a spark-based memory management system according to an embodiment of the present invention. Referring to FIG.

Referring to FIG. 1, a typical spark cluster system 100 may include a network 10, a master 20, and a plurality of worker nodes 30a, 30b, ..., 30x.

The master 20 and the plurality of worker nodes 30a, 30b, ..., and 30x may be connected through the network 10 and may communicate with each other through the network 10. [

Examples of the network 10 include a 3rd Generation Partnership Project (3GPP) network, a Long Term Evolution (LTE) network, a World Interoperability for Microwave Access (WIMAX) network, the Internet, a LAN (Local Area Network) A wireless LAN, a WAN (Wide Area Network), a PAN (Personal Area Network), a Bluetooth network, an NFC (Near Field Communication) network, a satellite broadcasting network, an analog broadcasting network, a DMB (Digital Multimedia Broadcasting) But is not limited thereto.

The master 20 and the plurality of worker nodes 30a, 30b ... 30x can be a computing device capable of being a desktop PC, a notebook, a tablet, a mobile phone, a smart phone, a mobile communication terminal, a personal digital assistant , But is not limited thereto.

When a job for analyzing big data is performed in a spark environment, the master 20 executes a driver process 21 which is a Java process, and a plurality of worker nodes Each of the processors 30a, 30b, ..., 30x can execute an executor process 31a, 31b, ..., 31x, which is a Java process for processing a task. In this case, since the configurations included in each of the plurality of worker nodes 30a, 30b, ..., and 30x and the processing procedures performed by the respective ones are the same, a plurality of worker nodes 30a, 30b, The communication between one worker node 30a and the master 20 will be described. That is, the description of one worker node 30a may be applied to the other worker nodes 30b, ..., 30x.

First, the master 20 divides a job into a plurality of tasks for distributed processing, and divides the divided tasks into a plurality of worker nodes 30a, 30b, ..., 30x) of the driver. At this time, in consideration of the situation of the plurality of worker nodes 30a, 30b, ..., 30x, the driver process 21 divides the divided tasks into a plurality of worker nodes 30a, 30b, (31a, 31b, ..., 31x).

The worker node 30a to which the task has been assigned can execute the execution process 31a for executing the task assigned by the driver process 21 and transmitting the result of the execution to the driver process 21. [ Here, the execution process 31a, which receives a task from the driver process 21, performs a plurality of tasks. At this time, the execution process 31a generates RDD, which is data necessary for processing the task, May be stored in the memory 32a or a high-speed storage medium (e.g., SSD) 33a.

RDD is an abbreviation of Resilient Distributed DataSet. It means a data set that is created once by a Java object and can not be modified if it is stored in memory. Such an RDD is a job that is repeated in machine learning, Hadoop, Jobs are designed to look at performance improvements due to caching.

On the other hand, after the RDD is created, the execution process 31a can perform all the tasks using the RDD and then transmit the results of the task execution to the driver process 21 of the master 20. [

Hereinafter, a spark-based memory management system according to an embodiment of the present invention based on the spark cluster system 100 shown in FIG. 1 will be described in detail.

FIG. 2 is a diagram illustrating a schematic configuration of a spark-based memory management system according to an embodiment of the present invention.

2, a spark-based memory management system 200 according to one embodiment of the present invention may include a network 10, a master 20 and a plurality of worker nodes 30a, 30b, ..., 30x. have. At this time, the network 10, the master 20, and the plurality of worker nodes 30a, 30b, ..., 30x shown in FIG. 2 correspond to the network 10, the master 20, Based memory management system 200 according to one embodiment of the present invention shown in Figure 2, the master 20 may be the same as the driver manager 22 (30a, 30b, ..., 30x) , And the worker node 30a may further include a walker manager 34a and an analysis tool 35a.

2, only one worker node 30a among a plurality of worker nodes 30a, 30b, ..., and 30x will be described for convenience of description, and one The contents described with respect to the worker node 30a of the worker node 30a may be similarly applied to the other worker nodes 30b, ..., 30x. That is, each of the plurality of worker nodes 30a, 30b, ..., 30x may include respective walker managers and analysis tools.

First, the overall concept of the spark-based memory management system 200 according to an embodiment of the present invention will be briefly described. The driver manager 22 receives information changed by the control of the walker manager 34a from the walker manager 34a And can update the received changed information to the driver process 21. [ This driver manager 22 can communicate with the walker manager 34a via the network 10. [

The walker manager 34a may be included in the worker node 30a with an execution process 31a storing the RDD and processing the task and the walker manager 34a may use the system analysis tool or analysis tool 35a It is possible to check whether a bottleneck phenomenon occurs in the spark environment every predetermined period by the user. In this paper, bottleneck factors in the spark environment are exemplified by the occurrence of full garbage collection (Full GC) and shuffle spill. In this case, a Java analysis tool can be used as the analysis tool 35a in order to check whether full garbage collection has occurred. In order to check the occurrence of the shuffle spill, Spark Web UI and computer resource analysis Tools are available.

More specifically, there are two major factors that affect performance degradation when a job is performed in a spark environment. The first factor is the occurrence of a Full GC (or Major GC) in the Java Garbage Collection to clean up old objects. When full GC occurs, the spark becomes a bottleneck, a framework for large data distribution processing, because it uses a lot of computer resources. In order to solve this bottleneck, in the present invention, when it is judged that full GC is generated, a high-speed storage medium can be used to secure a sufficient space of memory, and more specifically, some RDDs stored in memory The memory space can be ensured by moving and storing.

Also, a second factor affecting performance degradation in a spark environment is the occurrence of a shuffle spill. In the spark environment, if there is insufficient memory used for shuffling while performing a job, a shuffle spill will occur where the data to be spilled on the disk is stored on the disk. At this time, when the shuffle data stored in the memory is moved to the disk, a serialization for making the Java object occurs. This is because as the amount of data to be shuffled increases, the CPU usage increases, A bottleneck will occur in the CPU to be used. In order to solve such a bottleneck, in the present invention, when it is determined that a shuffle spill has occurred, some RDDs stored in the memory are moved to a high-speed storage medium and stored, thereby securing a memory space. The required memory can be increased. The description will be described later in more detail.

When the bottleneck factor is identified, the worker manager 34a can solve the bottleneck phenomenon by exchanging information (for example, information for determining whether or not the bottleneck phenomenon has occurred) with the execution process 31a have. The worker manager 34a also receives the changed information from the execution process 31 when the storage location or the memory setting of the RDD is changed through the execution process 31 and transmits the received change information to the driver manager 22 Lt; / RTI >

In other words, when the task is assigned by the driver process 21, the worker manager 34a sends information to the system analysis tool 35a (see FIG. 5) to determine whether a bottleneck phenomenon such as a Java GC, Java heap capacity, ). ≪ / RTI > Thereafter, the worker manager 34a can obtain the RDD information and the memory information of the execution process 31a from the execution process 31a so as to control the storage location of the RDD and the memory setting of the execution process 31a . Thereafter, the worker manager 34a can change the storage location of the RDD, the memory setting, and the like by the execution process 31a in order to solve the bottleneck phenomenon. Thereafter, the driver manager 22 can receive change information (e.g., storage location change information of the RDD, change information of the memory setting, etc.) due to the resolution of the bottleneck from the walker manager 34a. At this time, the driver manager 22 can receive the change information from each of the walker managers included in the plurality of worker nodes 30a, 30b, ..., 30x. Thereafter, the driver manager 22 can update the change information received from the worker manager 34a to the driver process 21. [

In the spark-based memory management system 200 according to the embodiment of the present invention, the driver process 21 and the execution process 31a in the worker node 30a in the master 20 communicate with each other via the network 10, And the worker manager 34a in the driver manager 22 and the worker node 30a within the master 20 transmits the RDD information and the status information of the worker node 30a For example, information such as Java GC, Java heap capacity, CPU usage, etc. can be transmitted and received as information for determining the occurrence of a bottleneck. Hereinafter, the present invention will be described in more detail.

The master 20 can execute the driver process 21 that divides the job into a plurality of tasks and allocates them to the plurality of worker nodes 30a, 30b, ..., 30x. The divided tasks are transmitted to the plurality of worker nodes 30a, 30b, ..., 30x through the network 10, and in response, the worker node 30a executes the tasks assigned by the driver process 21 Process 31a can be executed.

At this time, the execution process 31a may generate the RDD required for executing the task and store the RDD in the memory 32a or the high-speed storage medium 33a. For example, the memory 32a may be SD, micro SD, etc., and the high-speed storage medium 33a may be an SSD, but is not limited thereto.

Thereafter, the execution process 31a may perform the task using the generated RDD, and may transmit the execution result of the task to the driver process 21. [ In this case, in the spark-based memory management system 200 according to the embodiment of the present invention, in order to improve the spark performance, the walker manager 34a determines whether or not a bottleneck phenomenon occurs in the spark environment, It is possible to transmit the changed information of the changed execution process 31a to the driver manager 22 of the master 20 by performing processing for eliminating the occurrence of a bottleneck and performing processing.

The walker manager 34a can determine whether a bottleneck phenomenon has occurred by checking whether at least one of a full garbage collection and a shuffle spill has occurred at a predetermined cycle. In addition, when it is determined that the bottleneck phenomenon has occurred, the worker manager 34a can eliminate bottlenecks by moving some of RDDs stored in the memory 32a to the high-speed storage medium 33a. On the other hand, the worker manager 34a may store the generated RDD in the high-speed storage medium 33a instead of the memory 32a in consideration of the size of the heap in order to prevent the occurrence of full garbage collection in advance. This can be more easily understood with reference to FIG.

FIG. 3 is a diagram illustrating a process of eliminating bottlenecks through the walker manager 34a in the spark-based memory management system 200 according to an embodiment of the present invention.

Referring to FIG. 3, in step S310, the worker manager 34a can check a state of a Java virtual machine and a shuffle spill in a predetermined period to determine whether a bottleneck has occurred. Here, the period for confirming the state of the Java virtual machine and the shuffle spill, that is, the predetermined period, can be set by user input.

The walker manager 34a periodically monitors the state of the Java virtual machine to check whether full garbage collection (full GC) has occurred, which is one of the bottleneck factors. The walker manager 34a periodically monitors the shuffle spill, It is possible to confirm the occurrence of one shuffle spill.

At this time, in order to prevent occurrence of the bottleneck caused by full garbage collection (Full GC) in advance, the walker manager 34a stores the generated RDD in the memory 32a or the high-speed storage medium 33a , The Java virtual machine status can be checked (S320) and the heap size can be monitored first. Here, the size of the heap may be the size of the space in which the RDD can be stored in the entire storage space of the memory 32a, or in other words, the capacity of the heap storing the RDD. The setting of the heap size (or the hip capacity) may be pre-set by user input.

The walker manager 34a may monitor the size of the heap and determine whether the capacity of the RDD stored in the memory 32a exceeds a predetermined heap size (or the heap capacity) (S321). If the capacity of the RDD stored in the memory 32a exceeds the predetermined heap size (S321-Y) as a result of the determination (S321-Y), the walker manager 34a causes the generated RDD to be stored in the high- 33a (S322). If the capacity of the RDD stored in the memory 32a does not exceed the predetermined heap size (S321-N) as a result of the determination, the walker manager 34a stores the generated RDD in the memory 32a (Step S323). If the RDD is not the last RDD (S324-N), the walker manager 34a performs S320 again. If the RDD is the last RDD (S324-Y) The corresponding algorithm can be terminated.

When the generated RDD is not stored in the memory 32a (S323), the memory 32a becomes insufficient while the job is being executed, so that full garbage collection (Full) GC) may occur. When full GC occurs, a large amount of computing resources such as CPU are used, which causes a bottleneck in spark clusters. Accordingly, the worker manager 34a can check the state of the Java virtual machine (S320) to determine whether full garbage collection has occurred (S325) in order to eliminate the bottleneck due to Full GC.

If it is determined in step S325 that full garbage collection does not occur (S325-N), the worker manager 34a may return to step S320. If it is determined in step S325 that full garbage collection has occurred (S325-Y), the walker manager 34a selects the first RDD among the RDDs stored in the memory 32a based on the preset RDD replacement algorithm, RDD can be identified as a part of RDD (S326).

The predetermined RDD replacement algorithm may be, for example, an LRU (Least Recently Used) algorithm, an LFU (Least Frequently Used) algorithm, a FIFO (First In, First Out) algorithm, and the like. Since each algorithm is a known technique, a detailed description will be omitted below.

Thereafter, the worker manager 34a copies the first RDD stored in the memory 32a, stores it in the high-speed storage medium 33a, and deletes the first RDD stored in the memory 32a, thereby storing the first RDD stored in the memory 32a The first RDD can be moved to the high-speed storage medium 33a (S327). At this time, the worker manager 34a can control the movement of the RDD between the memory 32a and the high-speed storage medium 33a using a spark application programming interface (API). That is, the worker manager 34a can move the first RDD stored in the memory 32a to the high-speed storage medium 33a by using the spark API.

Thereafter, the worker manager 34a may transmit the changed information (e.g., change information of the storage location of the RDD) changed by the movement of the first RDD to the driver manager 22 (S328). After step S328, the worker manager 34a may go back to step S320 to repeatedly perform the determination of whether or not the bottleneck phenomenon by Full GC is repeated at a predetermined cycle.

On the other hand, in step S310, the worker manager 34a may check the shuffle spill in a predetermined cycle (S330) to determine whether the bottleneck phenomenon has occurred or not.

At this time, the shuffle spill means that the capacity of the memory 32a is insufficient and the intermediate data generated through the shuffle is spilled on the disk. The reason that shuffle spill becomes a bottleneck in spark clusters is because the CPU to be used in the operation to serialize the RDD due to the spill. At this time, when serialization is performed, the usage amount of the CPU reaches 100%, and the execution process 31a can not process the task. When the bottleneck phenomenon occurs, the whole job execution time of the memory management system 200 is affected by the time that the bottleneck becomes the bottleneck, and the performance of the spark is deteriorated. Therefore, in order to improve the performance of the spark, the walker manager 34a may determine whether or not the shuffle spill occurs, and then perform processing for eliminating the bottleneck caused by the shuffle spill. At this time, the worker manager 34a can use the analysis tool 35a including the Spark Web UI and the computer resource analysis tool to check whether or not the shuffle spill has occurred and check the CPU usage amount .

If it is determined in step S330 that the shuffle spill has occurred at a predetermined cycle and the shuffle spill has occurred, the walker manager 34a determines that the percentage of the shuffle space in the memory 32a has reached a predetermined value (T%, for example, 80% ) (Step S331), it is possible to determine whether to increase the specific gravity of the shuffle space. Here, a predetermined value (for example, 80%) as a reference for determining whether to increase the specific area of the shuffle space is a maximum value that can be used as a shuffle space in the entire space of the memory 32a, Lt; / RTI >

More specifically, when it is determined that the shuffle spill has occurred, the walker manager 34a determines how much the current shuffle space is used in the entire space of the memory 32a (i.e., (A percentage of a percentage). For example, assume that the maximum space that can be used as the shuffle space in the memory 32a is 80%. When the space used as the current shuffle space in the memory 32a is 80% or more (S331-Y), the space used as the current shuffle space in the memory 32a is expressed as 80% , The worker manager 34a can no longer increase the shuffle space, and in this case, the corresponding algorithm can be terminated.

On the other hand, if it is determined that the space used as the current shuffle space in the memory 32a is not equal to or greater than the predetermined value (80%) (S331-N), the walker manager 34a The specific gravity can be increased. More specifically, in the case of S331-N, the worker manager 34a selects, from among the RDDs stored in the memory 32a based on the preset RDD replacement algorithm, The RDD can be identified as some RDD (S332).

The predetermined RDD replacement algorithm may be, for example, an LRU (Least Recently Used) algorithm, an LFU (Least Frequently Used) algorithm, a FIFO (First In, First Out) algorithm, and the like.

Thereafter, the worker manager 34a copies the second RDD stored in the memory 32a, stores it in the high-speed storage medium 33a, and then deletes the second RDD stored in the memory 32a, thereby storing the second RDD stored in the memory 32a The second RDD can be moved to the high-speed storage medium 33a (S333). At this time, the worker manager 34a can control the movement of the RDD between the memory 32a and the high-speed storage medium 33a using a spark application programming interface (API). That is, the worker manager 34a can move the second RDD stored in the memory 32a to the high-speed storage medium 33a by using the spark API.

The worker manager 34a can increase the weight of the shuffle space so that the shuffle spill does not occur based on the free space in the memory 32a secured by deleting the second RDD from the memory 32a (S334 ). At this time, when the specific gravity of the shuffle space is increased in step S334, the maximum increase specific gravity value of the shuffle space that can be increased at one time (that is, the maximum rate at which the second RDD can be expanded into the shuffle space at the time of deletion) Can be set.

The walker manager 34a then transmits the change information (e.g., change information of the storage location of the RDD, change information of the memory 32a setting, etc.) changed by the movement of the second RDD to the driver manager 22 S335). After step S336, the walker manager 34a can again determine whether the specific weight of the shuffle space in the memory 32a is equal to or greater than a predetermined value (S336). If the space used as the current shuffle space is equal to or greater than the predetermined value in step S336 (S336-Y), the walker manager 34a can not increase the shuffle space any more and can terminate the algorithm. On the other hand, if it is determined in step S336 that the space used as the current shuffle space is not equal to or greater than the preset value (S336-N), the walker manager 34a repeatedly performs the determination of whether or not the bottleneck phenomenon by the shuffle spill occurs, , It may return to the step for performing step S330 again.

As described above, after the worker manager 34a periodically checks whether two bottleneck factors (e.g., Full GC, shuffle spill) have occurred and resolves the respective bottleneck phenomena, the spark- The memory management system 200 of the present invention can improve the performance of the spark more effectively.

The walker manager 34a here can receive RDD information and memory information from the execution process 31a and can control the storage location of the RDD and the settings of the memory by the execution process 31a based on the received information have. At this time, when it is determined that a change has occurred in the storage location of the RDD or the setting of the memory, the worker manager 34a may receive change information of the storage location of the RDD or change information of the memory setting from the execution process 31a . Then, the worker manager 34a can transmit the received change information to the driver manager 22 as change information of the execution process 31a.

 The driver manager 22 of the master 20 can update the change information received from the worker manager 34a with respect to the driver process 21. [

The spark-based memory management system 200 according to an exemplary embodiment of the present invention can improve caching efficiency by managing caching in units of spark RDD, and can be applied to various big data distribution frameworks. In addition, the memory management system 200 can achieve high efficiency at a low cost by using SSD, which is a high-performance storage, and can improve performance in an in-memory system.

Hereinafter, the operation flow of the present invention will be briefly described based on the details described above.

4A and 4B are schematic operation flowcharts of a spark-based memory management method according to an embodiment of the present invention.

The spark-based memory management method shown in FIGS. 4A and 4B can be performed by the spark-based memory management system 200 described above. Accordingly, the contents of the spark-based memory management system 200 can be similarly applied to FIG. 4A and FIG. 4B even if omitted from the following description.

Referring to FIGS. 4A and 4B, in step S410, a driver process included in the master may be executed to divide a job into a plurality of tasks.

Next, in step S420, the execution process included in the worker node may be executed, the task assigned by the driver process may be performed, and the execution result may be transmitted to the driver process. At this time, in step S420, the execution process may generate an RDD for performing a task and store the RDD in a memory or a high-speed storage medium.

In step S420, the worker manager determines whether the predetermined hip size is exceeded at the time of storing the generated RDD in order to prevent the occurrence of the bottleneck phenomenon by full garbage collection in advance, If the preset heap size is exceeded, the execution process can be controlled such that the RDD is stored on the fast storage medium instead of the memory.

In step S420, the worker manager included in the worker node determines whether or not a bottleneck phenomenon occurs in the spark environment, performs processing for eliminating the occurrence of the bottleneck phenomenon according to the determination result, The change information of the execution process can be transmitted to the master (S421).

In step S421, the worker manager can eliminate the bottleneck phenomenon by moving some of the RDDs stored in the memory to the high-speed storage medium. At this time, the worker manager can move some RDDs to the high-speed storage medium by using the spark API.

In step S421, the worker manager can determine whether a bottleneck phenomenon has occurred by checking whether at least one of a full garbage collection and a shuffle spill has occurred at a predetermined period.

In step S421, if it is determined that full garbage collection has occurred, the walker manager identifies the first RDD to be transferred to the high-speed storage medium among the RDDs stored in the memory as a partial RDD based on the predetermined RDD replacement algorithm, The RDD can be copied and stored in the high-speed storage medium, and the first RDD stored in the memory can be deleted.

Further, in step S421, when it is determined that the shuffle spill has occurred, the walker manager can determine whether to increase the specific gravity of the shuffle space by determining whether the specific gravity of the shuffle space is less than a predetermined value in the memory.

At this time, in step S421, when it is determined that the weight of the shuffle space is less than the preset value, the walker manager selects the RDD stored in the memory based on the preset RDD replacement algorithm to increase the weight of the shuffle space The second RDD to be moved can be identified as some RDD, the second RDD can be copied and stored in the high-speed storage medium, and the second RDD stored in the memory can be deleted.

In step S421, the worker manager may increase the weight of the shuffle space by deleting the second RDD from the memory, wherein the maximum increase gravity value of the shuffle space that can be increased at one time may be set by user input.

In step S421, the worker manager can receive the RDD information and the memory information from the execution process, and control the storage location of the RDD and the memory setting by the execution process.

If it is determined in step S421 that a change has occurred in the storage location of the RDD and the setting of the memory, the walker manager receives the change information of the storage location of the RDD and the change information of the memory setting from the execution process, And can be transferred to the driver manager as change information of the execution process.

Thereafter, in step S422, the driver manager included in the master can update the change information received from the walker manager with respect to the driver process (S422).

In the above description, steps S410 through S420 may be further divided into additional steps or combined into fewer steps, according to embodiments of the present disclosure. Also, some of the steps may be omitted as necessary, and the order between the steps may be changed.

The spark-based memory management method according to one embodiment of the present invention may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and configured for the present invention or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

The above-described spark-based memory management method may also be implemented in the form of a computer program or an application executed by a computer stored in a recording medium.

It will be understood by those of ordinary skill in the art that the foregoing description of the embodiments is for illustrative purposes and that those skilled in the art can easily modify the invention without departing from the spirit or essential characteristics thereof. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

The scope of the present invention is defined by the appended claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included within the scope of the present invention.

200: Spark-based memory management system
20: Master
21: Driver Process 22: Driver Manager
30a: Worker node
31a: Execution Process
32a: memory 33a: high-speed storage medium
34a: Walker Manager 35a: Analysis Tool

Claims (21)

A memory management system in a spark environment,
A master for executing a driver process of dividing and assigning a job to a plurality of tasks;
A worker node for executing a task assigned by the driver process and executing an execution process for transmitting the execution result to the driver process,
Lt; / RTI >
Wherein the worker node determines whether or not a bottleneck phenomenon occurs in the spark environment, performs processing for eliminating the occurrence of the bottleneck phenomenon according to a determination result, and updates the change information of the changed execution process by performing the processing And a walker manager for transmitting to the master,
Wherein the master includes a driver manager for updating the change information received from the worker manager with respect to the driver process,
Wherein the execution process comprises: generating an RDD for performing the task and storing the RDD in a memory or a high-speed storage medium,
Wherein the worker manager removes the bottleneck by moving some of the RDDs stored in the memory to the fast storage medium and confirms whether at least one of full garbage collection and shuffle spill has occurred at a predetermined cycle, Wherein the memory management system determines whether a phenomenon occurs. delete delete The method according to claim 1,
The walker manager,
Identifies a first RDD to be transferred to the fast storage medium among the RDDs stored in the memory as the partial RDD based on a predetermined RDD replacement algorithm when the full garbage collection has occurred, Fast storage medium, and deletes the first RDD stored in the memory. The method according to claim 1,
The walker manager,
In order to prevent the occurrence of the bottleneck caused by the full garbage collection, it is determined whether or not the generated RDD exceeds the preset heap size. If the RDD exceeds the predetermined size, the RDD replaces the memory And controls the execution process to be stored in the fast storage medium. The method according to claim 1,
The walker manager,
And determines whether to increase the weight of the shuffle space by determining whether the shuffle space weight is less than a predetermined value in the memory when it is determined that the shuffle spill has occurred. The method according to claim 6,
The walker manager,
And a second RDD to be transferred to the high-speed storage medium among the RDDs stored in the memory based on a predetermined RDD replacement algorithm to increase the specific gravity of the shuffle space when it is determined that the specific gravity of the shuffle space is less than the preset value Identify as some RDDs, copy the second RDDs to the fast storage medium, and delete the second RDDs stored in the memory. 8. The method of claim 7,
The walker manager,
Wherein the maximum incremental gravity value of the shuffle space that can be increased at one time by increasing the gravity of the shuffle space by deleting the second RDD from the memory is configurable by user input. The method according to claim 1,
Wherein the worker manager moves the portion of the RDD to the fast storage medium using a spark API. The method according to claim 1,
The walker manager,
Receiving the information of the RDD and the information of the memory from the execution process and controlling the storage location of the RDD by the execution process and the setting of the memory,
Wherein the change information of the storage location of the RDD and the change information of the memory setting are transmitted to the driver manager as change information of the execution process when it is determined that a change occurs in the storage location of the RDD and the setting of the memory, Spark-based memory management system. A method for managing memory in a spark environment,
(a) executing a driver process included in a master to divide a job into a plurality of tasks and allocate them;
(b) executing an execution process included in the worker node, performing a task assigned by the driver process, and transmitting an execution result to the driver process,
Lt; / RTI >
The step (b)
(b1) a walker manager included in the worker node determines whether or not a bottleneck phenomenon occurs in the spark environment, performs processing for eliminating the occurrence of the bottleneck phenomenon according to a determination result, and performs the processing Transmitting change information of the changed execution process to the master; And
(b2) updating, by the driver manager included in the master, the change information received from the walker manager to the driver process,
In the step (b)
Wherein the execution process generates an RDD for performing the task and stores the RDD in a memory or a high-speed storage medium,
In the step (b1)
Wherein the worker manager removes the bottleneck by moving some of the RDDs stored in the memory to the high-speed storage medium and confirms whether at least one of the full garbage collection and the shuffle spill has occurred at a predetermined cycle, And determining whether or not a phenomenon occurs. delete delete 12. The method of claim 11,
In the step (b1)
The walker manager identifies, as the partial RDD, a first RDD to be transferred to the fast storage medium among RDDs stored in the memory based on a predetermined RDD replacement algorithm when it is determined that the full garbage collection has occurred, Storing the RDD in the fast storage medium, and deleting the first RDD stored in the memory. 12. The method of claim 11,
In the step (b)
The walker manager determines whether the generated RDD exceeds a preset heap size in order to prevent the occurrence of the bottleneck caused by the full garbage collection. If the generated RDD exceeds the predetermined size, And the RDD is stored in the fast storage medium instead of the memory. 12. The method of claim 11,
In the step (b1)
Wherein the walker manager determines whether to increase the weight of the shuffle space by determining whether the shuffle space weight is less than a predetermined value in the memory when it is determined that the shuffle spill has occurred, How to manage memory. 17. The method of claim 16,
In the step (b1)
The walker manager moves from the RDD stored in the memory to the high-speed storage medium based on a preset RDD replacement algorithm to increase the weight of the shuffle space when it is determined that the weight of the shuffle space is less than the predetermined value Identifying the second RDD as the partial RDD, copying the second RDD to the fast storage medium, and deleting the second RDD stored in the memory. 18. The method of claim 17,
In the step (b1)
Wherein the walker manager is configured to increase the weight of the shuffle space by deleting the second RDD from the memory, wherein the maximum incremental gravity value of the shuffle space that can be increased at one time is set by user input. How to manage memory. 12. The method of claim 11,
In the step (b1)
Wherein the worker manager moves the portion of the RDD to the fast storage medium using a spark API. 12. The method of claim 11,
In the step (b1)
The worker manager receives information of the RDD and information of the memory from the execution process and controls the storage location of the RDD by the execution process and the setting of the memory,
Wherein the change information of the storage location of the RDD and the change information of the memory setting are transmitted to the driver manager as change information of the execution process when it is determined that a change occurs in the storage location of the RDD and the setting of the memory, A spark-based memory management method. A computer-readable recording medium having recorded thereon a program for executing the method of any one of claims 11, 14 to 20 in a computer.

Images