KR101722643B1 - Method for managing RDD, apparatus for managing RDD and storage medium for storing program managing RDD - Google Patents

Abstract

The present invention discloses a storage medium for storing a method, an apparatus, and a program for efficiently managing a resilient distributed dataset (RDD). The RDD management method according to an embodiment of the present invention relates to a method for managing RDD for data collection, analysis, and processing, which comprises the following steps: extracting metadata from genealogy for the RDD; storing the metadata extracted in the extraction step; and updating the metadata stored when performing operation related to the genealogy for the RDD, wherein the stored metadata includes RDD object information and RDD type information.

Description

[0001] The present invention relates to an RDD management method, an RDD management apparatus, and a storage medium storing an RDD management program,

The present invention relates to a method of managing RDD (Resilient Distributed Dataset), a device for managing RDD, and a storage medium storing a program for managing RDD, and more particularly, to a RDD To a method, apparatus, and storage medium for managing RDD using metadata.

Recently, as interest in big data increases, research on data analysis technology in real time has been actively carried out. An example of a popular real-time data analysis platform is Apache Spark. This real-time data analysis platform uses a dataset called Resilient Distributed Dataset (RDD) to process, analyze and store the data streamed in real time. Here, RDD is characterized by having data in a non-variable, i.e., immutable form. Therefore, when processing the data contained in the pre-generated RDD, a new RDD including the processed data is generated. In other words, a derived RDD derived from an existing RDD is additionally generated during data processing.

On the other hand, when streaming data is processed, raw RDDs are constantly generated, and a derived RDD is generated from each generated raw RDD, thereby increasing the number of RDDs. As a result, there is a shortage of space for storing the information included in the RDD, and a considerable delay occurs in searching for desired information. In addition, there is a problem in that, when the generated RDDs are deleted, RDDs must be generated again for data analysis.

The present invention has been made in recognition of the above problems, and it is an object of the present invention to provide a storage medium storing a method, an apparatus, and a program for efficiently managing an RDD.

Specifically, the present invention provides a storage medium storing a method, apparatus, and program for managing RDD using meta data related to RDD and derived RDD in order to efficiently manage the RDD.

According to an aspect of the present invention, there is provided a method for managing RDD for data collection, analysis, and processing, the method comprising: extracting metadata from a genealogy for RDD; ; Storing the extracted metadata in the extracting step; And updating the stored metadata when performing an operation related to the genealogy for the RDD, wherein the stored metadata includes RDD object information and RDD type information.

The updating may be a step of updating the status information indicating the life cycle of the RDDs generated as a result of performing an operation related to the lineage for the RDD.

The method may further include receiving information indicating whether to store at least one RDD among the RDDs generated as a result of performing the operation, in a cache or a persistent manner; Storing the at least one RDD according to the indicating information; And further updating the stored metadata based on the indicating information and the storage location information of the stored at least one RDD.

The stored metadata further includes priority information and RDD storage location information, the priority information is set depending on the RDD type information, and the RDD storage location information can indicate location information on which RDDs are stored .

Wherein the stored metadata further comprises status information indicating a lifecycle of RDDs generated as a result of an operation related to the genealogy for the RDD, the method comprising: generating, as a result of performing an operation related to the genealogy for the RDD Detecting garbage collection for the RDDs; Updating the stored metadata by changing the status information associated with RDDs selected as targets of garbage collection among the generated RDDs; And updating the stored metadata by changing the status information and the RDD storage location information for an RDD having priority higher than a predetermined priority among RDDs selected as objects of the garbage collection, And storing the RDD whose priority information is ahead of a predetermined priority based on the changed RDD storage location information.

The detected garbage collection may be performed after storing the RDD whose priority information is ahead of the predetermined priority based on the changed RDD storage location information.

The RDD type information may indicate any one of a final RDD, a heavy RDD, a general RDD, and a raw RDD.

The priority information may indicate a high priority in the order of the final RDD, the heavy RDD, the primitive RDD, and the general RDD.

A Resilient Distributed Dataset (RDD) management apparatus according to another embodiment of the present invention manages an RDD for data collection, analysis, and processing. The apparatus manages a DAD (Directed Acyclic Graph) including a genealogy for RDD, A DAG scanning agent for extracting meta data from the genealogy for the RDD; And a metadata manager for storing metadata extracted from the genealogy for the RDD and for updating metadata stored in the operation related to the genealogy for the RDD, wherein the stored metadata includes RDD object information and RDD type information And the like.

The meta data manager may update the status information indicating the life cycle of the RDDs generated as a result of the operation related to the genealogy for the RDD.

The apparatus includes a memory for storing information indicating whether to cache or store at least one RDD among RDDs generated as a result of the computation in memory, And an RDD storage manager for storing the at least one RDD, wherein the metadata manager is further adapted to update the stored metadata based on the indicating information and the storage location information of the stored at least one RDD have.

The stored metadata further includes priority information and RDD storage location information, the priority information is set depending on the RDD type information, and the RDD storage location information can indicate location information on which RDDs are stored .

Wherein the stored metadata further includes status information indicating a life cycle of RDDs generated as a result of an operation related to the genealogy for the RDD, the apparatus comprising: Further comprising an RDD lifetime monitoring agent for detecting garbage collection for the RDDs that have been selected for garbage collection, wherein the meta data manager changes the status information associated with RDDs selected as targets of garbage collection among the generated RDDs, Wherein the meta data manager further updates the stored meta data, and the meta data manager stores the status information for the RDD having the priority higher than the predetermined priority among the RDDs selected as the target of the garbage collection, Updates the stored metadata by changing information, and updates the RD D storage manager may store the RDD in which the priority information precedes the predetermined priority based on the changed RDD storage location information.

A storage medium for storing a program for managing a Resilient Distributed Dataset (RDD) according to another embodiment of the present invention is a storage medium for storing a program for managing a Resilient Distributed Dataset (RDD) for data collection, analysis and processing , Extracting metadata from the genealogy for the RDD, storing the extracted metadata in the extracting step, and updating metadata stored in the operation related to the genealogy for the RDD, wherein the stored metadata is RDD Object information and RDD type information.

According to the present invention, it is possible to provide a storage medium storing a method, an apparatus, and a program capable of efficiently storing and managing the RDD.

According to an embodiment of the present invention, by updating the metadata associated with the RDD after the RDD operation, accurate information on the actually generated RDD can be managed.

According to another embodiment of the present invention, the cached RDD and the RDD stored as the persistent can be separately managed.

According to another embodiment of the present invention, the RDD having a high degree of importance is stored and managed in advance in a separate storage location before garbage collection for the RDD is performed, thereby preventing the corresponding RDD from being deleted from the memory.

1 is a diagram illustrating a real-time data processing process according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating an example of a process of generating an RDD.
3 is a diagram illustrating a real-time data processing process according to another embodiment of the present invention.
4 is a diagram illustrating RDD metadata according to an embodiment of the present invention.
5 is a flowchart illustrating a metadata extraction process according to an embodiment of the present invention.
6 is a flowchart illustrating an initial RDD generation process according to an embodiment of the present invention.
FIG. 7 is a flowchart illustrating a selective storage process of an RDD according to an embodiment of the present invention.
8 is a flowchart illustrating a garbage collection process according to an embodiment of the present invention.
9 is a flowchart illustrating an RDD management method according to an embodiment of the present invention.

While the present invention has been described in connection with what is presently considered to be the most practical and preferred embodiment, it is to be understood that the invention is not limited to the disclosed embodiments. Also, in certain cases, there may be a term arbitrarily selected by the applicant, and in this case, the meaning thereof will be described in the description of the corresponding invention. Therefore, it is intended that the terminology used herein should be interpreted based on the meaning of the term rather than on the name of the term, and on the content of the specification throughout the specification. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

1 is a diagram illustrating a real-time data processing process according to an embodiment of the present invention.

1, a real-time data processing system according to an embodiment of the present invention may include an information collecting unit H1100, a data generating unit H1200, a data processing unit H1300, and a storage unit H1400 . Meanwhile, although the embodiment of FIG. 1 shows a real time data processing process, the data processing system of the present invention may process real time data as well as non real time data.

The information collection unit H1100 can collect various information. The information collection unit H1100 can collect data generated from sensors such as an IoT sensor, an infrared sensor, a thermal sensor, a CCTV, and the like. Also, the information collecting unit H1100 may collect information from a database including real-time or non-real-time information such as a legacy system or an SNS. The information collecting unit H1100 can output the collected information to a data generating unit H1200 to be described later.

The data generating unit H1200 can receive data from the information collecting unit H1100 and generate data. The data generated by the data generating unit H1200 may be a real time data stream or non real time based data. Further, the data generating section H1200 can output the generated data to the data processing section H1300. According to one embodiment, the data generation unit H1200 may include at least one of TCP, ZeroMQ, Kafka, and Twitter.

The data processing unit H1300 can receive data from the data generating unit H1200. At this time, the data processing unit H1300 can receive data from the data generating unit H1200 in real time or in non-real time. In addition, the data processing unit H1300 can generate a Resilient Distributed Dataset (RDD) based on the received data. The data processing unit H1300 can process data based on a general-purpose distributed platform. According to one embodiment, the data processing unit H1300 may be part of the Apache Spark platform.

The data processing unit H1300 can generate the RDD in real time or in non-real time. Here, the RDD originally generated based on the input data may be referred to as a raw RDD.

RDD, on the other hand, is a set of data that has data in a non-variable, or immutable, form. In other words, RDD can not change some of the internal data unlike existing data sets. Due to these RDD characteristics, additional RDDs are generated during data processing and processing associated with the RDD. That is, an RDD derived from the pre-generated RDD is generated at the time of data processing. At this time, the RDD derived from the pre-generated RDD may be referred to as a derived RDD. This derived RDD is a concept that is derived from a primitive RDD and generated from a pre-existing derived RDD.

Since RDD is derived from data processing and may be generated based on the data stream, the number of RDDs may increase exponentially. Also, the increase in the number of RDDs results in insufficient storage space, and a delay in the search speed for desired information may occur. Therefore, there is a need for selective management of raw RDDs, derived RDDs, and frequently used RDDs.

On the other hand, the generated RDD has a predetermined life cycle, except that it is semi-permanently stored. An RDD that has reached its end of life may be removed from memory via an operation called garbage collection. That is, the data processing unit H1300 can perform garbage collection to remove the RDD having a shortened life span from the memory.

The life cycle of an RDD can be divided into three states: initial (I), caching (C), and garbage collection (GC). In this case, the initial generation means that the RDD is created, the caching means that the generated RDD is cached in the memory, and the disappearance may mean that the RDD is deleted from the memory and stored in the hard disk. If the generated RDD is stored semi-permanently (persist, P), the lifecycle of the RDD may be persistent (P), initial (I), cached (C) (garbage collection, GC).

On the other hand, the derived RDD is generated by an operation on the RDD. There are transformations and actions in RDD operations.

Table 1 is a table showing operations for RDD.

Derived RDDs are generated by transformations and actions. More specifically, when an action is performed, an RDD is generated by referring to a lineage for an RDD to be described later. At this time, a derived RDD is generated as many times as the number of times the action associated with the action is called, and finally an action is performed to generate a final RDD (Lazy execution). In this case, the action is performed by a worker present at an individual node of the distributed platform.

A Directed Acyclic Graph (DAG) can be generated that includes a lineage for RDD prior to operation. Here, the genealogy for RDD contains data and metadata about the derived RDD and raw RDD to be generated by the transformations and actions. Thus, the metadata and data for the raw RDD and the derived RDD can be obtained through scanning for the DAG. More specifically, the genealogy for RDD includes data on derived RDDs generated when a transformation operation and an action operation are performed, data on primitive RDDs, and relationship between derived RDDs and metadata indicating the relationship between the derived RDD and the derived RDD And the like.

The main operations of the data processing unit H1300 may include the following four operations.

1) Generation of genealogy for RDD

2) Generate RDD by performing the actual operation according to the genealogy for RDD

3) Decide whether to save the initially generated RDD as a custody

4) Remove stored RDD from memory via garbage collection

FIG. 2 is a diagram illustrating an example of a process of generating an RDD.

Referring to FIG. 2, a DAG is generated before an action is performed (S21).

Then, the DAG scheduler generates an execution plan according to the generated DAG. At this time, node grouping is performed and a stage is created (S22).

Next, the task scheduler generates a task considering narrow dependency (S23). The task scheduler launches tasks through the cluster manager.

Next, the worker residing at the individual node executes the task to generate derived RDDs from the raw RDD (RR). At this time, derived RDDs are generated according to the lineage for RDD. The resultant RDD generated by performing the action is referred to as heavy RDD (HR), and the RDD generating such a heavy RDD may be referred to as the last RDD (last RDD, LR). And, the derived RDD excluding the final RDD and the heavy RDD in the derived RDD may be referred to as a general RDD (general RDD, GR). The tasks of a generic RDD have a narrow dependency in relation to other RDDs. In contrast, the tasks of the final RDD and the heavy RDD have a wide dependency on each other.

On the other hand, the embodiment of the RDD generation process shown in FIG. 2 can be said to represent the genealogy for the RDD. FIG. 2 shows a process of generating an RDD when an action is performed. However, since the genealogy for an RDD includes metadata indicating a relationship of derived RDDs when an action is performed, 2, S23 can visually represent the genealogy for RDD.

Referring to FIG. 2, the lineage for RDD is illustrated with raw RDDs (RRs) and derived RDDs, and these derived RDDs are referred to as last RDDs (LRs), heavy RDDs , HR), and general RDD (general RDD, GR). Since RDD can be divided into a raw RDD and a derived RDD, the RDD can be divided into a raw RDD, a final RDD, a heavy RDD, and a general RDD. These primitive RDDs, final RDDs, heavy RDDs, and generic RDDs may be referred to as RDD type information. In the example shown in the drawing, RDD 5 corresponds to a heavy RDD, RDD 4 corresponds to a final RDD, RDD 2 to 3 correspond to a general RDD, and RDD 1 corresponds to a raw RDD.

The storage unit H1400 may store the RDDs received from the data processing unit. At this time, the storage unit H1400 can cache or semi persistently store the RDDs in the memory, and store the RDDs removed from the memory in the hard disk. According to one embodiment, the storage process of the storage unit H1400 may be performed under the control of the data processing unit. For example, the data processing unit may generate a control command related to RDD storage and output it to the storage unit H1400.

3 is a diagram illustrating a real-time data processing process according to another embodiment of the present invention.

3, a real-time data processing system according to another embodiment of the present invention includes an information collection unit H3100, a data generation unit H3200, a data processing unit H3300, and a storage unit H3400, And a management unit H3500. Meanwhile, although the embodiment of FIG. 3 shows a real-time data processing process, the data processing system of the present invention may process real-time data as well as non-real-time data.

The information collecting unit H3100 can collect various information. The information collecting unit H3100 can collect data generated from sensors such as an IoT sensor, an infrared sensor, a heat sensor, a CCTV, and the like. In addition, the information collecting unit H3100 may collect information from a database including real-time or non-real-time information such as a legacy system or an SNS. This is as described above.

The data generating unit H3200 can receive data from the information collecting unit H3100 and generate data. The data generated by the data generating unit H3200 may be a real time data stream or non real time based data. Further, the data generating section H3200 can output the generated data to the data processing section H3300. This is as described above.

The data processing unit H3300 can receive data from the data generating unit H3200. At this time, the data processing unit H3300 can receive data from the data generating unit H3200 in real time or in non-real time. In addition, the data processing unit H3300 can generate a Resilient Distributed Dataset (RDD) based on the received data. Meanwhile, the content related to the RDD can be directly applied to the real-time data processing process according to the embodiment of the present invention, and therefore, the repetitive description will be omitted.

The RDD management unit (H3500) can generate, store, and manage the metadata of the RDD by monitoring and managing the raw RDD and the derived RDD.

According to one embodiment, the RDD management unit H3500 may include a DAG scanning agent H3510, an RDD lifetime monitoring agent H3520, a metadata manager H3530, and an RDD storage manager H3540.

DAG  Scanning Agent

The DAG scanning agent H3510 can scan the DAG. That is, the DAG scanning agent H3510 can determine whether or not a DAG has been generated. According to one embodiment, the DAG scanning agent H3510 may monitor the task to determine whether a DAG has been generated. The DAG scanning agent (H3510) can read the genealogy for the RDD from the DAG if it finds the DAG. Further, the DAG scanning agent H3510 can extract metadata from the lineage for RDD. Here, the extracted metadata may include object information and RDD type information for each RDD. Here, the object information for the RDD may include identification information of the RDD and identification information of the DAG including the RDD. The RDD type information may be information on whether the RDD corresponds to a raw RDD or a derived RDD, and whether the RDD corresponds to the last RDD in the derived RDD, that is, the RDD associated with the action, Information corresponding to a heavy RDD generated from the RDD or a general RDD excluding the final RDD and the heavy RDD. On the other hand, the RDD type information can be used as information for setting the priority for the RDD.

According to one embodiment, the DAG scanning agent H3510 can set the priority for RDD in the following order according to the RDD type information.

<RDD priority>

Priority 1: Last RDD (last RDD)

Priority 2: Heavy RDD (heavy RDD)

Priority 3: Raw RDD (raw RDD)

Priority 4: General RDD (general RDD)

The set RDD priority can configure the metadata for the RDD. That is, the metadata extracted from the DAG and the RDD priority can be output to the metadata manager H3530, which will be described later, by configuring the metadata for the RDD. That is, the DAG scanning agent H3510 can output the metadata for the RDD to the metadata manager H3530, which will be described later. In this case, the metadata manager H3530 may receive the metadata about the RDD and store the metadata in the storage.

In addition, the DAG scanning agent H3510 can output the RDD priority to the RDD lifetime monitoring agent H3520 to be described later. The RDD lifetime monitoring agent (H3520) can receive the RDD priority, and then monitor the RDD.

RDD  life span monitoring  agent

The RDD lifetime monitoring agent (H3520) can monitor the life cycle of each RDD.

According to one embodiment, the RDD lifetime monitoring agent H3520 may receive RDD priority from the DAG scanning agent H3510 and then perform monitoring on the RDD. Since the RDD lifetime monitoring agent H3520 receives the RDD priority from the DAG scanning agent H3510, the RDD lifetime monitoring agent H3520 may receive the RDD priority, Can be performed.

The RDD lifetime monitoring agent H3520 monitors whether or not an operation (transformation, action) is performed with respect to the RDD. When the operation is actually performed, the life cycle information and the storage location information about the RDD are stored in a metadata manager H3530). At this time, the metadata manager H3530 can update life cycle information, storage location information, and the like after receiving the input. Here, since the RDD is actually generated by the operation, the life cycle information output to the metadata manager H3530 represents the initial generation state (initial, I). Here, the storage location information may include specific location information such as the address of the memory in which the RDD is to be stored.

In addition, the RDD lifetime monitoring agent (H3520) can monitor whether garbage collection is performed. When the data processing unit H3300 performs garbage collection, the RDD lifetime monitoring agent H3520 detects this and then outputs the lifecycle information of the RDD to the metadata manager H3530 to be described later. At this time, the information output to the metadata manager H3530 may include lifecycle information on RDDs selected as garbage collection among the RDDs. Here, the lifecycle information of the RDDs selected as the garbage collection may indicate the garbage collection (GC) state. In this case, the metadata manager H3530 can receive the RDD lifecycle information and update the metadata about the RDD. Also, the metadata manager H3530 may change the life cycle information and the storage location information of some RDDs selected as objects of garbage collection by using the priority information described in the meta data for the RDD. At this time, the metadata manager H3530 can change the life cycle information and the storage location information when the priority described in the metadata for the RDD is equal to or higher than the predetermined priority. For example, when the priority is 2 or more, that is, when the RDD type is the last RDD (priority 1) or the heavy RDD (priority 2), the metadata manager H3530 stores the lifecycle information and the storage Location information can be changed. As a result, the RDD may not be removed by garbage collection. Further, the meta data manager H3530 can output the changed information to the RDD storage manager H3540 to be described later. The RDD storage manager H3540 can receive the changed information and store the RDD in another location. As a result, some of the RDDs selected as objects of garbage collection may be stored in advance in another place and may not be deleted by garbage collection.

Metadata Manager

The metadata manager (H3530) receives metadata about the RDD and can store, manage and update the metadata. Here, the metadata may include a DAG ID, an RDD ID, an RDD type, a priority, a status, a storage location, and the like of the RDD. Here, the DAG ID is an identifier for identifying each DAG, the RDD ID is an identifier for identifying each RDD, and the RDD type represents four different types of RDDs. That is, the RDD type can be set to any one of a raw RDD (raw RDD), a final RDD (last RDD, LR), a heavy RDD (HR), and a general RDD (general RDD). The priority order can be determined according to the RDD type as a priority set by the DAG scanning agent H3510. According to the above-described embodiment, the final RDD is ranked first, the heavy RDD is ranked second, the primitive RDD is ranked 3, and the general RDD is ranked 4. However, this is an embodiment, and the priority of the RDD may be set in a different order. The status indicates the state of the RDD life cycle. As described above, the status may correspond to one of persist (P), initial (I), cashing (C), and garbage collection (GC). The storage location indicates the location where the RDD is stored. The storage location may be either a memory or a hard disk, depending on the accessibility. If the life cycle state is persistent, initial creation, or caching, the storage location of the RDD may be a memory, and if the life cycle state is destroyed, the storage location of the RDD may be a hard disk.

The meta data manager H3530 may receive the meta data for the RDD from the DAG scanning agent H3510 and store and update the meta data.

Also, the metadata manager H3530 can receive the storage information from the RDD storage manager H3540, which will be described later, and update the metadata. Here, the storage information may include information related to the storage location and information indicating whether the generated RDD is to be stored as a persistent store or a caching store. And, the information related to the storage location may include information indicating whether the RDD is stored in the memory, the hard disk, and information indicating a specific address.

Also, the metadata manager H3530 can receive information on the RDD corresponding to the target of garbage collection from the RDD lifetime monitoring agent (H3520). The meta data manager H3530 can receive information on the RDD corresponding to the target of garbage collection and update the status of the corresponding RDD.

The metadata manager H3530 may store the metadata in a storage unit H3400 to be described later or may store the metadata in a separate database. At this time, the stored metadata may be a processed result of the metadata extracted from the genealogy for the RDD. The metadata extracted from the genealogy for the RDD can be processed by the DAG scanning agent H3510 or the metadata manager H3530 described above.

The metadata manager H3530 can output the metadata to the RDD storage manager H3540 to be described later. The metadata manager H3530 can receive metadata from the RDD storage manager H3540, indicating whether the RDD is stored as a persistent store, Can be input.

4 is a diagram illustrating RDD metadata according to an embodiment of the present invention.

Referring to FIG. 4, the RDD metadata includes DAG_ID, RDD_ID, RDD_Type, Priority, Status, and Location. In FIG. 4, DAG_ID, RDD_ID, RDD_Type, Priority, Status, and Location are shown in the first row. The RDG_ID indicated in the first column is the above-mentioned DAG ID, the RDD_ID indicated in the second column is the RDD ID described above, the RDD_Type indicated in the third column is the RDD type described above, the priority indicated in the fourth column is the above- The Status shown in the fifth column is the above-mentioned status, and the Location shown in the sixth column can be the aforementioned storage location.

The metadata associated with the RDD shown in the second row indicates information that the DAG ID of the corresponding RDD is 1 and that the RDD ID of the corresponding RDD is 1. In addition, the metadata related to the RDD shown in the second row indicates information that the type of the RDD indicates the general RDD (GR), and therefore, the information indicating that the priority is 4th. The metadata related to the RDD shown in the second row indicates information that the status of the corresponding RDD is an initial state (initial, I), and thus the storage location of the corresponding RDD indicates information that the memory is memory.

The metadata associated with the RDD shown in the third row indicates information that the DAG ID of the corresponding RDD is 1 and that the RDD ID of the corresponding RDD is 2. In addition, the metadata related to the RDD shown in the third row indicates information indicating that the type of the RDD is the last RDD (LR), and thus the priority is rank 1. The meta data associated with the RDD shown in the third row indicates information that the lifetime state of the corresponding RDD is an initial state (initial, I), and thus the storage location of the RDD indicates information that the memory is memory.

The metadata associated with the RDD shown in the fourth row indicates information that the DAG ID of the corresponding RDD is 1 and that the RDD ID of the corresponding RDD is 3. In addition, the metadata associated with the RDD indicated in the fourth row indicates information indicating that the type of the RDD indicates the information of the heavy RDD (HR), and accordingly, the priority is the second rank. The meta data associated with the RDD shown in the fourth row indicates information that the lifetime state of the corresponding RDD is an initial state (initial, I), and therefore, the storage location of the RDD indicates information that the memory is memory.

The metadata related to the RDD shown in the fifth row indicates information that the DAG ID of the corresponding RDD is 1 and the RDD ID of the corresponding RDD is 4. In addition, the metadata associated with the RDD shown in the fourth row indicates information indicating that the type of the RDD corresponds to the original RDD (RR), and thus the information indicating that the priority is three. The meta data associated with the RDD shown in the fifth row indicates information that the lifetime state of the corresponding RDD is the initial state (initial, I), and thus the storage location of the RDD indicates information that the memory is memory.

The metadata related to the RDD shown in the sixth row indicates information that the DAG ID of the corresponding RDD is 2 and the RDD ID of the corresponding RDD is 1. [ In addition, the metadata related to the RDD shown in the fifth row indicates information indicating that the type of the RDD corresponds to the general RDD (GR), and therefore, the priority is 4th. The meta data associated with the RDD shown in the fifth row indicates that the life cycle status of the corresponding RDD indicates garbage collection (C), and thus the corresponding RDD is a hard disk-based file system or a big data echo system Based repository or the like.

The metadata related to the RDD shown in the seventh row shows information that the DAG ID of the corresponding RDD is 2 and the RDD ID of the corresponding RDD is 2. In addition, the metadata related to the RDD shown in the seventh row indicates information indicating that the type of the RDD corresponds to the general RDD (GR), and therefore, the priority is rank 4. The meta data associated with the RDD shown in the seventh row indicates information indicating that the lifetime state of the corresponding RDD is caching (C), and thus the storage location of the corresponding RDD indicates the memory.

The metadata associated with the RDD shown in the eighth row shows information that the DAG ID of the corresponding RDD is 2 and the RDD ID of the corresponding RDD is 3. In addition, the metadata related to the RDD shown in the eighth row indicates information indicating that the type of the corresponding RDD indicates the heavy RDD (HR), and accordingly, the information indicating that the priority is second rank. The meta data associated with the RDD shown in the eighth row indicates information indicating that the lifetime state of the corresponding RDD is caching (C), and thus the storage location of the corresponding RDD indicates memory.

The metadata related to the RDD shown in the ninth line shows information that the DAG ID of the corresponding RDD is 2 and the RDD ID of the corresponding RDD is 4. In addition, the metadata associated with the RDD shown in the ninth row indicates information indicating that the type of the RDD is the primitive RDD (RR), and thus the priority is the third rank. The metadata related to the RDD shown in the eighth row indicates information that the life cycle state of the corresponding RDD is persist (P), and thus the storage location of the RDD indicates information that the memory is memory.

The metadata associated with the RDD shown in the 10th row indicates the information that the DAG ID of the corresponding RDD is 2 and the RDD ID of the corresponding RDD is 5. In addition, the metadata associated with the RDD shown in the tenth row indicates information indicating that the type of the RDD corresponds to the last RDD (LR), and thus the priority is rank 1. The metadata associated with the RDD shown in the tenth row indicates information indicating that the life cycle state of the corresponding RDD is caching (C), and thus the storage location of the corresponding RDD indicates the memory.

Meanwhile, the metadata manager H3530 can store and update the metadata of the RDD received from the DAG scanning agent H3510. For example, the metadata manager H3530 may receive metadata and priority information extracted from the DAG from the DAG scanning agent H3510 and store the received metadata and priority information. As another example, the meta data manager H3530 can receive lifecycle information on the generated RDD and update it when the RDD is actually generated by performing operations (transformation, action). As another example, the metadata manager H3530 may update the metadata of the RDD by receiving the lifecycle information of the RDD before the garbage collection is performed. As another example, the metadata manager H3530 may update the RDD lifecycle information selected as the target of garbage collection by using the priority information described in the metadata about the RDD before the garbage collection is performed . As another example, the meta data manager H3530 receives information on whether the RDD initially generated from the RDD storage manager H3540, which will be described later, has a life cycle of the persiste or a life cycle of the caching, .

RDD  Storage manager

The RDD storage manager H3540 can manage the RDDs stored in the storage unit H3400.

The RDD storage manager H3540 can receive information about at least a part of the metadata from the metadata manager H3530. For example, the RDD storage manager H3540 can receive information for identifying the RDD and information about the storage location of the RDD from the metadata manager H3530. Here, the information for identifying the RDD may include the DAG ID and the RDD ID described above. That is, the RDD can be identified by combining the DAG ID and the RDD ID. The RDD storage manager (H3540) can manage the location information of RDD generated using the information for identifying the RDD and the storage location information of the RDD.

In addition, the RDD storage manager H3540 can receive information indicating whether the initially generated RDD is stored / managed as a persistent or stored / managed by caching. According to one embodiment, the RDD storage manager H3540 can receive information indicating whether the RDD initially generated from the data processing unit H3300 is stored / managed as a persistent or stored / managed by caching. At this time, the RDD storage manager H3540 can receive more information about the specific storage location from the data processing unit H3300. The RDD storage manager H3540 may receive information indicating whether the initially generated RDD is to be stored / managed as a persistent or stored / managed by caching, and may then update storage location information according to the information. It is possible to output the life cycle information of the RDD (in particular, the persistent or caching) and the storage location information of the updated RDD to the metadata manager H3530. The metadata manager H3530 can receive the above-described information from the RDD storage manager H3540 and update the metadata about the RDD. At this time, in the metadata for the RDD, the life cycle information, the storage location information, and the like of the RDD may be updated.

For example, it can be assumed that the data processing unit H3300 has decided to store the initially generated RDD #i as a persistent resource. At this time, the RDD storage manager H3540 can receive from the data processing unit H3300 information indicating that RDD #i is to be stored / managed as a persistent and information about a specific storage location. The RDD storage manager H3540 can receive the information that the RDD #i is stored in the predetermined storage location in the form of a persistent list and update the storage location information. The RDD storage manager H3540 can output the life cycle information of the RDD #i and the storage location information of the updated RDD #i to the metadata manager H3530. The metadata manager H3530 may receive the above-described information from the RDD storage manager H3540 and update the metadata of the RDD #i. Specifically, the metadata manager H3530 can update the lifecycle information on the RDD #i with the persistent update and the specific storage location information.

In addition, the RDD storage manager H3540 may store some of the RDDs selected as objects of garbage collection in another storage location before the garbage collection is performed. The RDD storage manager H3540 receives the changed information from the metadata manager H3530 and can store the RDD in another location based on the received information. At this time, the metadata manager H3530 may change the life cycle information and the storage location information of some RDDs selected as objects of garbage collection by using the priority information described in the meta data for RDD. This is as described above.

The storage unit H3400 can store the RDDs received from the data processing unit H3300. At this time, the storage unit H3400 can cache or semi-permanently store the RDDs in the memory, and selectively store the RDDs in the hard disk. According to one embodiment, the storage process of the storage unit H3400 may be performed under the control of the data processing unit H3300. For example, the data processing unit H3300 may generate a control command related to RDD storage and output it to the storage unit H3400. In addition, the storage unit H3400 may cache or semi-permanently store the RDDs in the memory according to a control command of the RDD storage manager H3540. For example, the storage unit H3400 may store some of the RDDs selected by garbage collection according to a control command of the RDD storage manager H3540 in a separate storage location or move the storage unit to another storage location have.

Hereinafter, with reference to the drawings, a description will be given of a series of processes for extracting metadata from RDD lineage, initially generating RDD, selectively storing RDD, and garbage collection of RDD. In the following drawings, the spark application and the DAG scheduler may be components of the above-described data processing unit H3300. The DAG scanning agent H3510, the RDD lifetime monitoring agent H3520, the metadata manager H3530 and the RDD storage manager H3540 may be included in the RDD management unit H3500, and the above description may be applied as it is.

5 is a flowchart illustrating a metadata extraction process according to an embodiment of the present invention.

First, the spark application creates an execution plan for data processing. The spark application outputs the generated execution plan to the DAG scheduler. The DAG scheduler receives the execution plan and generates the DAG according to the execution plan. In addition, the DAG scheduler outputs a standby command to the RDD lifetime monitoring agent (H3520). The RDD lifetime monitoring agent (H3520) receives the standby command and maintains the standby state.

The DAG scanning agent H3510 scans the DAG. If the DAG scheduler generates the DAG according to the schedule, the DAG scanning agent H3510 can detect it. The DAG scanning agent (H3510) discovers the DAG and extracts the operation related metadata from the lineage of the RDD included in the DAG. DAG scanning agent (H3510) assigns priority according to the type of RDD. The DAG scanning agent H3510 outputs priority information to the RDD lifetime monitoring agent H3520. In addition, the DAG scanning agent H3510 outputs the metadata for the RDD to the RDD lifetime monitoring agent H3520. Here, the metadata for RDD includes metadata extracted from the genealogy for priority information and RDD. The RDD lifetime monitoring agent (H3520) receives priority information and monitors whether the RDD is actually generated according to the genealogy for RDD. The metadata manager H3530 stores metadata about the RDD received from the DAG scanning agent H3510.

6 is a flowchart illustrating an initial RDD generation process according to an embodiment of the present invention.

This figure shows the process in which RDD is actually generated based on the lineage for RDD. First, the DAG scheduler outputs an operation execution instruction in a spark application according to a schedule. The spark application performs an operation by receiving an operation execution command from the DAG scheduler. That is, the spark application performs at least one or more transformations related to an action, and finally performs an action to initially create an actual RDD. The RDD lifetime monitoring agent (H3520) detects that the actual RDD is generated while monitoring whether RDD is generated. The RDD lifetime monitoring agent H3520 detects that the actual RDD has been generated and outputs information on the lifecycle of the RDDs to the meta data manager H3530. The metadata manager H3530 can update the life cycle of the generated RDD. At this time, since the RDD is in the initial generation state, the metadata manager H3530 can update the life cycle of the actually generated RDD to the initial generation state (initial, I).

FIG. 7 is a flowchart illustrating a selective storage process of an RDD according to an embodiment of the present invention.

This drawing shows a process of determining whether to store the initially generated RDD as a persistent or a caching. First, the spark application decides whether to store the initially created RDD as a persistent or a caching. The RDD storage manager H3540 receives information indicating whether the initially generated RDD is to be saved as a persistent or a caching. At this time, the RDD storage manager H3540 can receive further information about the specific storage location where the corresponding RDD is to be stored. The RDD storage manager H3540 stores the RDD as caching or persistent based on the received information. That is, the RDD storage manager H3540 stores the corresponding RDD as a caching or persistent storage location. Then, the RDD storage manager H3540 outputs information on the life cycle information of the RDD concerned and the specific storage location of the RDD to the meta data manager H3530. Here, the life cycle information may correspond to persistent or caching. The metadata manager H3530 can receive the information from the RDD storage manager H3540 and update the metadata.

8 is a flowchart illustrating a garbage collection process according to an embodiment of the present invention.

This drawing shows a process in which garbage collection is performed. First, the DAG scheduler outputs a garbage collection execution command as a spark application according to a schedule. The spark application receives the garbage collection execution command from the DAG scheduler. The spark application then performs operations for garbage collection. The RDD lifetime monitoring agent (H3520) monitors whether garbage collection is performed and outputs lifecycle information for RDD to the metadata manager (H3530) when garbage collection is detected. The RDD lifetime monitoring agent (H3520) outputs lifecycle information of the RDD selected as a target of garbage collection to the metadata manager (H3530). The metadata manager H3530 receives the RDD lifecycle information and updates the metadata about the RDD. At this time, the meta data manager H3530 can update the RDD lifecycle information of the RDD selected as the target of the garbage collection to garbage collection (GC). In addition, the metadata manager H3530 can further update the life cycle information and the storage location information of some RDDs selected as objects of garbage collection using the priority information pre-stored in the metadata. The meta data manager H3530 outputs the updated information to the RDD storage manager H3540. The RDD storage manager H3540 receives the updated information. The RDD storage manager H3540 stores the RDD based on the received information.

In one embodiment, the metadata manager H3530 may change the life cycle information and the storage location information when the priority stored in the metadata is equal to or higher than the predetermined priority. That is, when the RDD having a high priority is selected as the object of garbage collection, the metadata manager H3530 can change the life cycle information and storage location information so that the corresponding RDD is not removed. Also, the meta data manager H3530 can output the changed life cycle information and storage location information to the RDD storage manager H3540. The RDD storage manager H3540 can further store the RDD having a higher priority in another storage location based on the changed life cycle information and storage location information. According to this embodiment, the RDD having a high priority is stored in a separate storage location, so that even if garbage collection is performed, the RDD is not deleted. Further, the separately stored RDD can be identified through object identification information such as a combination of DAG ID and RDD ID. Thus, according to this embodiment, important RDDs can be maintained on the memory, and accessibility to important RDDs can be increased.

9 is a flowchart illustrating an RDD management method according to an embodiment of the present invention.

Each step of the RDD management method according to an embodiment of the present invention can be performed by the component (s) of the RDD management apparatus described above.

The method first extracts metadata from the genealogy for RDD (s9100).

Then, the method stores extracted metadata (s9200). Next, when the operation related to the lineage for the RDD is performed, the method updates the stored metadata (s9300). Here, the stored metadata includes RDD object information and RDD type information. The RDD object information may include a DAG identifier and an RDD identifier, and the RDD type information may indicate either a final RDD, a heavy RDD, a general RDD, or a primitive RDD.

The updating step may correspond to updating the status information indicating the life cycle of the RDDs generated as a result of the operation related to the lineage for the RDD. That is, when an operation for RDD is actually performed and an RDD is generated, the status information indicating that the life cycle of RDDs is an initial generation can be updated.

Optionally, the method can further perform the following steps. First, the method receives information indicating whether to store at least one RDD among the RDDs generated as a result of the computation, in a memory in a cache or persist. Then, at least one RDD is stored according to the indicated information. Next, the stored metadata is further updated based on the indicating information and the storage location information of the stored at least one RDD. That is, the method can distinguish and store the RDDs according to the information indicating whether the RDDs in the initial generation state (initial, I) are stored by caching or persistent storage, and manage the metadata for the RDDs classified and stored.

Alternatively, the stored metadata further includes priority information and RDD storage location information, the priority information is set depending on RDD type information, and the RDD storage location information may indicate location information on which RDDs are stored . At this time, the priority information may have a high priority in the order of the final RDD, the heavy RDD, the primitive RDD, and the general RDD.

Optionally, the method can further perform the following steps. First, the method detects garbage collection for RDDs generated as a result of performing an operation related to the lineage for the RDD. The method further updates the stored metadata by changing the status information related to the RDDs selected as objects of garbage collection among the generated RDDs. Next, the method further updates the stored metadata by changing the status information and the RDD storage location information of the RDD having priority higher than the predetermined priority among the RDDs selected as objects of garbage collection , And stores the RDD whose priority information is ahead of the preset priority based on the changed RDD storage location information. Next, sensed garbage collection may be performed. That is, the above method can separately store and manage RDDs having a high priority, that is, high priority RDDs selected as objects of garbage collection.

Each step of the RDD management method according to an embodiment of the present invention described above can be programmed by software, and the programmed program can be stored in a storage medium.

Each component that constitutes a module, unit, or device may be processors that execute sequential execution processes stored in memory (or storage unit). Each of the steps described in the above embodiments may be performed by hardware / processors. Each of the units described in the above embodiments may be operated as a hardware / processor. Further, the methods proposed by the present invention can be executed as codes. The code may be written to a storage medium readable by the processor and thus read by a processor provided by the apparatus.

Although the drawings have been described for the sake of convenience of explanation, it is also possible to design a new embodiment by incorporating the embodiments described in each drawing. It is also within the scope of the present invention to design a computer-readable recording medium in which a program for executing the previously described embodiments is recorded according to the needs of ordinary artisans.

The apparatus and method according to the present invention are not limited to the configuration and method of the embodiments described above as described above, but the embodiments described above may be modified so that all or some of the embodiments are selectively And may be configured in combination.

On the other hand, it is possible to implement the method proposed by the present invention as a code that can be read by a processor in a computer-readable recording medium provided in a network device. The processor-readable recording medium includes all kinds of recording apparatuses in which data that can be read by the processor is stored. Examples of the recording medium on which the processor can read are ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, and the like. In addition, the processor-readable recording medium may be distributed over network-connected computer systems so that code readable by the processor in a distributed fashion can be stored and executed.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It will be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the present invention.

In this specification, both the invention of the invention and the invention of the method are explained, and the description of both inventions can be supplemented as necessary.

It will be understood by those skilled in the art that various changes and modifications can be made therein without departing from the spirit or scope of the invention. Accordingly, it is intended that the present invention cover the modifications and variations of this invention provided they come within the scope of the appended claims and their equivalents.

In the present specification, the apparatus and method inventions are all referred to, and descriptions of both the apparatus and method inventions can be supplemented and applied to each other.

H1100, H3100: Information collecting section
H1200, H3200: Data generation unit
H1300, H3300: Data processing section
H1400, H3400: Storage unit
H3500: RDD management unit
H3510: DAG scanning agent
H3520: RDD lifetime monitoring agent
H3530: Metadata manager
H3540: RDD Storage Manager

Claims (14)

A method of managing a Resilient Distributed Dataset (RDD) for data collection, analysis and processing, the method comprising:
The processor comprising: extracting metadata from the lineage for the RDD;
Storing the extracted metadata in the extracting step; And
Updating the stored metadata when the processor performs an operation related to the genealogy for the RDD, wherein the stored metadata includes RDD object information and RDD type information. The method according to claim 1,
Wherein the updating is a step of updating status information indicating a life cycle of RDDs generated as a result of an operation related to the lineage for the RDD. 3. The method of claim 2,
Receiving, by the processor, information indicating whether to cache or store at least one RDD among RDDs generated as a result of performing the operation, in a memory;
The processor storing the at least one RDD in accordance with the indicating information; And
Further comprising: the processor further updating the stored metadata based on the indicating information and the storage location information of the stored at least one RDD. The method according to claim 1,
Wherein the stored metadata further include priority information and RDD storage location information,
Wherein the priority information is set depending on the RDD type information, and the RDD storage location information indicates location information on which RDDs are stored. 5. The method of claim 4,
Wherein the stored metadata further includes status information indicating a life cycle of RDDs generated as a result of performing an operation related to the genealogy for the RDD,
The method comprises:
The processor detecting garbage collection of RDDs generated as a result of performing an operation related to the lineage for the RDD;
The processor further updating the stored metadata by changing the status information associated with RDDs selected as targets of garbage collection among the generated RDDs; And
Wherein the processor changes the status information and the RDD storage location information for the RDD having priority higher than a predetermined priority among the RDDs selected as the objects of the garbage collection, And storing the RDD whose priority information is ahead of a predetermined priority based on the changed RDD storage location information. 6. The method of claim 5,
Wherein the garbage collection detected in the detecting step is performed after storing the RDD whose priority information is prior to the predetermined priority based on the changed RDD storage location information. The method according to claim 1,
Wherein the RDD type information indicates any one of a final RDD, a heavy RDD, a general RDD, and a raw RDD. The method according to claim 6,
Wherein the priority information indicates a high priority in the order of a final RDD, a heavy RDD, a primitive RDD, and a general RDD. 1. An apparatus for managing a Resilient Distributed Dataset (RDD) for data collection, analysis and processing,
A DAG scanning agent that scans a Directed Acyclic Graph (DAG) including a lineage for RDD and extracts metadata from the lineage for the RDD; And
A metadata manager for storing metadata extracted from the genealogy for the RDD and for updating the metadata stored in the operation related to the genealogy for the RDD, wherein the stored metadata includes RDD object information and RDD type information Wherein the RDD management device comprises: 10. The method of claim 9,
Wherein the meta data manager updates status information indicating a life cycle of RDDs generated as a result of an operation related to the genealogy for the RDD. 11. The method of claim 10,
The apparatus comprises:
Receiving information indicative of whether to cache or store at least one RDD among the RDDs generated as a result of performing the operation on a memory, An RDD storage manager for storing the RDD;
Wherein the metadata manager further updates the stored metadata based on the indicating information and the storage location information of the stored at least one RDD. 12. The method of claim 11,
Wherein the stored metadata further include priority information and RDD storage location information,
Wherein the priority information is set depending on the RDD type information, and the RDD storage location information indicates location information on which RDDs are stored. 13. The method of claim 12,
Wherein the stored metadata further includes status information indicating a life cycle of RDDs generated as a result of performing an operation related to the genealogy for the RDD,
The apparatus comprises:
And an RDD lifetime monitoring agent for detecting garbage collection of RDDs generated as a result of performing an operation related to the genealogy for the RDD,
Wherein the metadata manager further updates the stored metadata by changing the status information associated with RDDs selected as targets of garbage collection among the generated RDDs,
Wherein the meta data manager changes the status information and the RDD storage location information for the RDD having priority higher than a predetermined priority among the RDDs selected as the objects of the garbage collection, Lt; / RTI &gt;
Wherein the RDD storage manager stores the RDD whose priority information is ahead of a predetermined priority based on the changed RDD storage location information. A storage medium storing a computer-readable program for managing a Resilient Distributed Dataset (RDD) for data collection, analysis and processing,
The metadata extraction unit extracts metadata from the genealogy for the RDD, stores the metadata extracted in the extracting step, and updates metadata stored in the operation related to the genealogy for the RDD, wherein the stored metadata includes an RDD object Information and RDD type information. &Lt; RTI ID = 0.0 &gt; A &lt; / RTI &gt;

Images