KR101640231B1 - Cloud Driving Method for supporting auto-scaled Hadoop Distributed Parallel Processing System - Google Patents

Abstract

The present invention relates to a cloud driving method for supporting an auto-scaled Hadoop distributed parallel processing system. The cloud driving method for supporting the auto-scaled Hadoop distributed parallel processing system according to the present invention includes the steps of: generating an input split by loading an input file by a client; analyzing available resources of a slave node by a master node, and receiving VM resource pool information from a cloud management server to allocate a job to the slave node and a VM node; receiving the generated input split by the slave node to allocate a map task to a task tracker of the slave node and the VM node, respectively, and transmitting a generated intermediate calculation result to a reduce task; returning, when the allocated map task is finished, a resource of the corresponding VM node to the VM resource pool; executing the reduce task using the intermediate calculation result as an input by the task tracker of the slave node and the VM node, and storing result data to an HDFS of the slave node; and returning, when the allocated reduce task is finished, a resource of the corresponding VM node to the VM resource pool. According to the present invention, computing resources of data to be processed in real time is analyzed to increase and decrease the computing resources in real time, so that the computing resources are efficiently used.

Description

[0001] The present invention relates to a Hadoop Distributed Parallel Processing System (HATS)

The present invention relates to a method of driving a cloud, and more particularly, to a method and apparatus for operating a cloud by using a Hadoop system to analyze input data in real time, The present invention relates to a cloud driving method for supporting automatic distributed parallel processing Hadoop system capable of improving the efficiency of map reduction processing by automatically increasing / decreasing required computing resources according to expected processing time.

Today, various types of data, such as text messages, voice, video, and location, rather than structured data such as social media and multimedia, Big data, which is a huge amount of data, which is so large as to be difficult to store, manage and analyze in the conventional way due to the explosion of the data.

Smart phones, wearable devices, etc., the amount of data that individuals create is increasing tremendously. Through analysis of these big data, companies are able to identify not only the patterns of consumers' purchases, but also their relationships and interests, It was possible.

Hadoop was developed as a method for analyzing such big data. Hadoop is a Hadoop Distributed File System (HDFS) that is complementary to the Google File System (Google File System), an open source implementation of MapReduce, source-based framework. The core of Hadoop consists of two parts: the storage part (HDFS) and the processing part (map redess). Hadoop divides files into large blocks (for example, 64MB or 128MB), and stores these blocks in a distributed manner in the nodes in the cluster. To process the stored data, Hadoop MapReduce sends the execution code to the node storing each data, and these nodes concurrently perform data processing in parallel. This approach has the advantage of being able to process data faster because the data and execution code operate faster and more efficiently at the same node.

However, the mapping reduction used in the conventional Hadoop system is not effective when the performance of the node host server is different, and efficient multi-task scheduling is provided when performing multiple map reduction operations in one cluster at the same time can not do.

In addition, since the data analysis method used in the conventional Hadoop system processes the data in the computing resources allocated to the data analysis at the start of the map reuse, even in the situation where the amount of data increases exponentially, When the analysis starts, there is a disadvantage that the idle computing resources can not be used.

In Korean Patent Laid-Open No. 10-2012-0041907 (Prior Art 1), Map Reduce (R) is used to guarantee parallel processing of large-scale data such as genomic computation using data mining or the like. (Homomorphic encryption) to ensure end-to-end confidentiality by using the results with consistent accuracy without relying on mapper and reducer Quot; large-scale data dispersion calculation method based on map reduction and its system " Korean Patent Laid-Open Publication No. 10-2014-0080795 (Prior Art 2) discloses a technique in which a plurality of virtual machines each receive a task completion time for each of a plurality of slots from a virtualization platform, Calculates the remaining time until completion of execution of all the slots of the task and transfers the remaining time to the master node, calculates an average value of a plurality of remaining times received by the master node, and calculates a CPU resource By controlling quotas so that the task execution time of each virtual machine is the same, it is possible to improve the performance by ensuring that the entire map reduction tasks are processed in a uniform time in a virtualized cluster environment. "The load of Hadoop MapReduce in a virtualized environment Dispersion method and system "

However, the prior art document 1 is a framework in which Map Reduce used as a distributed processing technique in a cloud computing environment for storing large capacity data and parallel processing capability performs distributed processing of parallel processing of a large amount of data, Data mining, and genome computation. However, in order to solve the problem that privacy infringement may occur in the process of map / reduce / map data, the prior art document 2 When the Hadoop map redistributes tasks in clusters, the task is terminated in wave form because all tasks are not concurrently terminated on each node due to differences in data locality and configured cluster environment. In order to cope with the problem that serious performance degradation is caused due to the nature of the map reduction, The prior art documents 1 and 2 have the above-described problems. In other words, when performing multiple map reduction tasks simultaneously in one cluster, efficient multi-task scheduling can not be provided. Even if idle computing resources are present in a situation where the amount of data increases exponentially, It is not possible to use the apparatus.

Korean Patent Laid-Open No. 10-2012-0041907 (published May 23, 2012) Korean Patent Publication No. 10-2014-0080795 (published on Jul. 01, 2014)

The present invention has been made in view of the above problems, and it is an object of the present invention to provide an automatic distributed parallel processing Hadoop system that can efficiently use computing resources by analyzing computing resources of data to be processed in real time, The present invention provides a method of operating a cloud for supporting the cloud.

In order to accomplish the above object, a cloud driving method for supporting a Hadoop system for automatic distributed parallel processing according to the present invention is for supporting a Hadoop system for processing large data, including a client, a master node, a slave node, , And a cloud host system, and analyzes the workload of a job requested by the client in real time at a compile time or a run time to obtain a virtual machine (VM) resource pool the method comprising the steps of: a) calculating a spare virtual machine (VM) from a virtual machine (VM) pool and automatically decreasing or decreasing a VM node to reallocate and run a job to reduce the execution time of a job, Loading an input file to create an input split; b) Analyzing available resources of the slave node by receiving the VM resource pool information from the cloud management server, and assigning a job to the slave node and the VM node; c) Assigning a map task to the slave node and the task tracker of the VM node, respectively, and transferring the generated intermediate artifacts to the resume task; d) if the allocated map task is terminated, Returning the resources of the node to the VM resource pool through the cloud management server; e) executing the resume task by inputting the intermediate artifacts by the slave node and the task tracker of the VM node, HDFS, and f) when the allocated reuse task is terminated, the resource of the corresponding VM node Group include those characterized by comprising the step of reducing the VM pool of resources through the cloud management server.

The step of generating the input splits in step a) includes the steps of: a-1) reading all the input files stored in the HDFS of the slave node by the client client of the client and analyzing the analysis method according to the file format; (A-2) creating input splitting data specifying how to divide the input file stored in units of chunks according to the file format, and a-3) And generating input split information consisting of a pair of a key and a value that can be easily accessed by the map task in the input split data according to a format.

In addition, the step of assigning a job to the slave node and the VM node in the step b) may include: b-1) a job tracker of the master node sends a job to the slave node and the VM node, HDFS, and analyzing available resources of the slave node and calculating the number of map tasks and redessing tasks that can be allocated to the slave node; and b-2) Receiving the VM resource pool information capable of allocating a VM node and recalculating the number of usable VM nodes and a map task and a reduce task assignable to the VM node; b-3) the size of a job registered in a job queue and unassigned job is analyzed from the viewpoint of data or processing, and information of the available VM resource pool is used B) assigning the unassigned map task to a task tracker of the slave node and the VM node based on the computed information, at a compile time, And b-5) periodically checking whether map tasks have been completed. If the map resources are not completed, analyzing available resources of the slave node, and analyzing the VM resource pool information And allocating a job to the slave node and the VM node.

The task tracker of the slave node and the VM node transmits a job defined by the user to a key and a value, (C-2) executing the map task according to the input format of the VM node to generate the intermediate output, and (c-2) assigning the intermediate output of the map task, which is executed by the slave node and the VM node's partition management unit, And c-3) distributing and delivering the intermediate artifacts generated by execution of the map task to the slave node and the resume task being executed in the task tracker of the VM node, .

In addition, the step of storing the resultant data of the step e) in the HDFS of the slave node may include the steps of: (e-1) sorting and summing intermediate deliverables distributed and distributed from the slave node and the division management unit of the VM node, and -2) The task tracker of the slave node and the VM node executes a task of reducing the sum of the intermediate outputs to an input value and storing the output file in the HDFS of the slave node.

According to the present invention, computing resources of data to be processed in real time can be analyzed and computing resources can be increased or decreased in real time, so that computing resources can be efficiently used.

In addition, the cloud system can improve the performance of mapping by configuring a separate job cluster according to the map reduction task to provide effective multi-task scheduling.

1 is a diagram schematically illustrating a configuration of a cloud system employed for implementing a cloud driving method for supporting an automatic distributed parallel processing Hadoop system according to the present invention.
FIG. 2 is a diagram showing an internal configuration of each of a client, a master node, and a slave node of the cloud system shown in FIG. 1. FIG.
3 is a diagram illustrating an internal configuration of a job tracker module embedded in a master node of the cloud system shown in FIG.
FIG. 4 is a diagram illustrating an internal configuration of a task tracker module built in a slave node of the cloud system shown in FIG. 1. FIG.
FIG. 5 is a diagram illustrating internal configurations of a cloud management server and a cloud host server of the cloud system shown in FIG. 1. FIG.
6 is a diagram illustrating a flow of data according to execution of map reduction in a cloud driving method for supporting an automatic distributed parallel processing Hadoop system according to the present invention.
FIG. 7 is a flowchart illustrating an overall execution process of a cloud drive method for supporting an automatic distributed parallel processing Hadoop system according to the present invention.
FIG. 8 is a flowchart illustrating a process of generating an input split in a cloud driving method for supporting an automatic distributed parallel processing Hadoop system according to the present invention.
FIG. 9 is a flowchart illustrating a process of assigning a job to a slave node and a VM node in a cloud driving method for supporting an automatic distributed parallel processing Hadoop system according to the present invention.
10 is a flowchart illustrating a process of transferring generated intermediate artifacts to a redessing task in a cloud driving method for supporting an automatic distributed parallel processing Hadoop system according to the present invention.
11 is a flowchart illustrating a process of storing result data in an HDFS of a slave node in a cloud driving method for supporting an automatic distributed parallel processing Hadoop system according to the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Before describing the cloud driving method for supporting the Hadoop system in the automatic distributed parallel processing according to the present invention, the cloud system employed for implementing the cloud driving method of the present invention will be described first.

1 is a diagram schematically illustrating a configuration of a cloud system employed for implementing a cloud driving method for supporting an automatic distributed parallel processing Hadoop system according to the present invention.

Referring to FIG. 1, a cloud system 100 employed for implementing the cloud driving method of the present invention includes a client 110, a master node 120, a slave node 130, a cloud management server 140, And a host server 150.

The master node 120 and the slave node 130 constitute a VM resource pool 102 and the cloud management server 140 and the cloud host server 150 constitute a physical server farm 104. VM resource pool 102 is a pool of virtualization resources that provide resources to operate as virtual machines (VMs) by allocating CPUs, memory, HDD, and network resources to VM nodes 130e-130g lt; / RTI >

2, the client 110 analyzes an input file stored in the Hadoop Distributed File System (HDFS) 133 of the slave node 130 and stores the input split file and the configuration file (For example, a file containing information on which slave node is to be used among a plurality of slave nodes, etc.), a specific file (this corresponds to a kind of code for performing work, for example, "Map Reduce.jar" A job client 111 that stores the same file in the HDFS 133 of the slave node 130 and informs the job tracker 121 that it is ready to start the map re-start, And a communication interface 112 for performing data communication through the network 120 and the slave node 130 and the network communication network 160.

2, the master node 120 sends a job requested by the job client 111 to the slave node 130 and the task tracker of the virtual machine node as a map task and a resume task A master node management unit 122 for managing the directory structure of the HDFS 133 (to be described later) of the slave node 130 and a slave node 130 and a network communication network 160 And a communication interface 123 for performing data communication through the communication interface 123.

As shown in FIG. 3, the job tracker 121 manages the requested jobs as a queue to manage the slave node 130 or the VM nodes 130e to 130g (see FIG. 6) A job management unit 121a for assigning jobs to the task tracker 131 and a resource management unit 121a for analyzing the resources of the slave node 130 and the VM resource pool 102 managed by the cloud management server 140 An analysis unit 121b and a job analyzer 121c for analyzing a time for processing a job allocated to the task tracker 131. [

2, the slave node 130 manages map tasks and resume tasks assigned by the job manager 121a of the job tracker 121 of the master node 120 according to the priority order, And transmits the available resource information of the slave node as a heart beat to the resource analyzer 121b of the job tracker 121 of the master node 120 A slave node management unit 134 for managing the data stored in the HDFS 133, and a slave node management unit 134 for managing data stored in the master node And a communication interface 135 for performing data communication through the network communication network 160. [ The slave node 130 may include a plurality of slave nodes 130a to 130c.

4, the task tracker 131 includes a task manager for managing a map task and a reduce task assigned by the job manager 121a of the job tracker 121 of the master node 120 And a division management unit 131b for converting the intermediate output resulting from the execution of the map task into input data that can be processed by the reduction task.

As shown in FIG. 5, the cloud management server 140 includes a VM node connection unit 141 for connecting to and controlling the virtual machine (VM) node operating in the cloud host server 150, A resource pool management unit 143 for managing a resource pool used by the VM node, and a resource pool management unit 143 for managing the creation, execution, and deletion of the VM node operating in the master node 150, And a communication interface 144 for performing data communication through the network communication network 160. [

5, the cloud host server 150 receives the VM nodes 130e to 130g (see FIG. 6) activated according to a request from the VM node connection unit 141 of the cloud management server 140 A virtualization management unit 152 for providing all the functions necessary for virtualization through a hypervisor engine and a virtualization management unit 152 for managing the virtualization management unit 152 in response to a request from the cluster management unit 142 of the cloud management server 140 A cluster agent unit 153 for executing a request for creation, execution, and deletion of a required VM node, and a resource management unit 143 for managing a usage amount of a CPU, a memory, a disk usage, and a VM node of the cloud host server, And a communication interface 155 for performing data communication with the cloud management server 140 through the network communication network 160. [ The cloud host server 150 may include a plurality of cloud host servers 150a to 150c.

The VM nodes 130e to 130g serve as a virtual machine (VM) by receiving CPU, memory, HDD, and network resources from the cloud host server 150, and are connected to a local physical slave node 130).

Now, a cloud driving method for supporting the automatic distributed parallel processing Hadoop system according to the present invention based on the cloud system having the above-described configuration will be described.

FIG. 6 is a flowchart illustrating a data flow according to execution of map reduction in a cloud driving method for supporting an automatic distributed parallel processing Hadoop system according to the present invention. FIG. Parallel Processing It is a flowchart showing the entire execution process of the cloud drive method for supporting the Hadoop system.

6 and 7, the cloud driving method for supporting the Hadoop system for automatic distributed parallel processing according to the present invention is for supporting the Hadoop system for processing big data, and includes the client 110, Based on a cloud system having a master node 120, a slave node 130, a cloud management server 140 and a cloud host server 150, After analyzing in real time at runtime to calculate the free virtual machine (VM) from the virtual machine (VM) resource pool of the cloud system that is needed for it, And to reduce the execution time of the job.

In the cloud driving method of the present invention as described above, the client client 111 of the client 110 firstly loads the input file 601 for the map reduction processing and outputs the input split 602a to 602d (Step S710).

Here, generation of the input splits 602a to 602d as described above will be described in detail with reference to FIG.

FIG. 8 is a flowchart illustrating a process of generating an input split in a cloud driving method for supporting an automatic distributed parallel processing Hadoop system according to the present invention.

Referring to FIG. 8, first, the input file 601 stored in the HDFS 133 of the slave node 130 is read by the job client 111 of the client 110 and analyzed according to the file format And analyzes the file by classifying the method (step S711).

Then, in step S712, the input splits 602a to 602d are created in which the input file 601 stored in units of chunks is divided according to the file format.

Then, in accordance with the file format, information on the input splits 602a to 602d consisting of pairs of keys and values that can be easily accessed by the map task in the input split data is generated (step S713).

When the creation of the input splits 602a to 602d is completed, the job tracker 121 of the master node 120 transmits the slave node (slave node) 602a through the resource monitor unit 132 of the slave node 130 130 and receives information on the VM resource pool 102 from which the VM nodes 130e to 130g can be allocated from the resource pool management unit 143 of the cloud management server 140. The slave node 130 and the VM nodes 130e to 130g (step S720).

Here, such job allocation will be described in more detail with reference to FIG.

FIG. 9 is a flowchart illustrating a process of assigning a job to a slave node and a VM node in a cloud driving method for supporting an automatic distributed parallel processing Hadoop system according to the present invention.

9, the job tracker 121 of the master node 120 determines whether the chunk of the input file 601 is physically located in the HDFS 133 of the slave node 130, In step S721, the available resources of the slave node 130 are analyzed and the number of the map tasks and the redess tasks that can be allocated to the slave node 130 is calculated.

The VM node 130e to 130g receives information on the VM resource pool 102 from which the VM nodes 130e to 130g can be allocated from the resource pool management unit 143 of the cloud management server 140, And the map task and the reduce task that can be allocated to the VM nodes 130e to 130g are calculated again (step S722).

Thereafter, a job registered in the job queue of the master node 120 is analyzed for the size of a job in terms of data or processing, and the available VM resource pool 102 The job is relocated to a queue using the information of the job (step S723).

Then, the unassigned map tasks are dynamically allocated to the slave node 130 and the task tracker 131 of the VM nodes 130e to 130g, statically at compile time and dynamically at run time, based on the calculated information (Step S724).

Thereafter, it is periodically checked whether the map tasks have been completed. If the map tasks are not completed, the slave node analyzes the available resources of the slave node, and receives a VM resource pool information and allocates a job to the slave node and the VM node. The process from step S721 to step S724 is repeated (step S725).

The task tracker 131 of the slave node 130 transmits the generated input split 602a to the VM nodes 130e to 130g after completing the job assignment to the slave node 130 and the VM nodes 130e to 130g. And assigns map tasks to the slave node 130 and the task tracker 131 of the VM nodes 130e to 130g, respectively, and transfers the generated intermediate artifacts to the reduce task (step S730).

Here, the transfer of the intermediate product generated as described above to the reduce task will be described in more detail with reference to FIG.

10 is a flowchart illustrating a process of transferring generated intermediate artifacts to a redessing task in a cloud driving method for supporting an automatic distributed parallel processing Hadoop system according to the present invention.

10, the task tracker 131 of the slave node 130 and the VM nodes 130e to 130g executes a map task according to an input format of a key and a value, To generate an intermediate output (step S731).

Then the partition management unit 131b of the slave node 130 and the VM nodes 130e to 130g of the task tracker 131 exchanges the intermediate output of the map task being executed in accordance with the key value, (Step S732).

In step S733, intermediate products generated by execution of the map task are distributed to the slave node 130 and the redess task being executed by the task tracker 131 of the VM nodes 130e to 130g.

After the completion of the transfer of the generated intermediate products to the task for reduction, when the map task assigned to the task tracker 131 of the VM nodes 130e to 130g is terminated, the resource of the corresponding VM node 130e to 130g And returns to the VM resource pool 102 through the resource pool management unit 143 of the cloud management server 140 (step S740).

The task tracker 131 of the slave node 130 and the VM nodes 130e to 130g executes the resume task with the intermediate output as input and stores the resultant data in the HDFS 133 of the slave node 130 (Step S750).

Here, storing the above result data in the HDFS 133 of the slave node 130 will be described in more detail with reference to FIG.

11 is a flowchart illustrating a process of storing result data in an HDFS of a slave node in a cloud driving method for supporting an automatic distributed parallel processing Hadoop system according to the present invention.

11, the partition management unit 131b of the task tracker 131 of the slave node 130 and the VM nodes 130e to 130g sorts and sums intermediate deliverables distributed and delivered (step S751).

Then, the task tracker 131 of the slave node 130 and the VM nodes 130e to 130g executes the task of reducing the summed intermediate output to the input value to output the output files 603a and 603b to the slave node (Step S752).

After the result data is stored in the HDFS 133 of the slave node 130 and the task assigned to the task tracker 131 of the VM node is terminated, resources of the VM nodes 130e to 130g are allocated to the cloud And is returned to the VM resource pool 102 through the resource pool management unit 143 of the management server 140 (step S760).

As described above, the cloud system supporting the automatic distributed parallel processing Hadoop system according to the present invention analyzes computing resources of data to be processed in real time and increases / decreases computing resources in real time, There are advantages to use.

In addition, the cloud system supporting the automatic distributed parallel processing Hadoop system according to the present invention provides a separate multi-task scheduling by configuring a separate job cluster according to the map reduction task through the cloud system, There is an advantage that performance can be improved.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but many variations and modifications may be made without departing from the spirit and scope of the invention. Be clear to the technician. Accordingly, the true scope of protection of the present invention should be construed according to the following claims, and all technical ideas within the scope of the same should be construed as being included in the scope of the present invention.

102: VM resource pool 104: physical server farm
110: client 111: job client
112: communication interface 120: master node
121: job tracker 121a: job manager
121b: resource analysis unit 121c: job analysis unit
122: master node management unit 123: communication interface
130, 130a to 130c: slave nodes 130e to 130g: VM node
131: Task tracker 131a: Task manager
131b: division management unit 132: resource monitor unit
133: HDFS 134: Slave Node Management Unit
135: Communication interface 140: Cloud management server
141: VM node connection unit 142: Cluster management unit
143: resource pool management unit 144: communication interface
150, 150a to 150c: Cloud host server 151: VM node management unit
152: virtualization management unit 153: cluster agent unit
154: resource monitor unit 155: communication interface
160: network communication network 601: input file
602a to 602d: input splits 603a and 603b:

Claims (5)

To support the Hadoop system for large data processing, the workload of a job requested by a client based on a cloud system including a client, a master node, a slave node, a cloud management server, and a cloud host server, After analyzing in real time at runtime to calculate the free virtual machine (VM) from the virtual machine (VM) resource pool of the cloud system that is needed for it, A method of driving a cloud, the method comprising:
a) loading an input file by the client to generate an input split,
b) analyzing available resources of the slave node by the master node, receiving VM resource pool information from the cloud management server, and assigning a job to the slave node and the VM node;
c) receiving the generated input split by the slave node, assigning a map task to the slave node and a task tracker of the VM node, respectively, and delivering the generated intermediate artifacts to the reduce task,
d) returning resources of the corresponding VM node to the VM resource pool through the cloud management server when the allocated map task is terminated;
e) executing the resume task by inputting the intermediate output by the slave node and the task tracker of the VM node and storing the resultant data in the HDFS of the slave node, and
and f) returning resources of the corresponding VM node to the VM resource pool through the cloud management server when the allocated redis task is terminated. The automatic distributed parallel processing according to claim 1, Way. The method according to claim 1,
The step of generating an input split of step a)
a) reading all the input files stored in the HDFS of the slave node by a client client of the client and analyzing the files by classifying the analysis method according to the file format;
(a-2) creating input split data specifying how to divide the input file stored in units of chunks according to the file format, and
(a-3) generating input split information consisting of a pair of key and value that can be easily accessed by the map task in the input split data according to the file format Distributed Parallel Processing A method of running a cloud to support Hadoop systems. The method according to claim 1,
Wherein the step of assigning a job to the slave node and the VM node in step b)
b-1) the job tracker of the master node analyzes information of which slave node the chunk of the input file is physically located in and which of the slave nodes, analyzes the available resources of the slave node, Calculating a number of redundancy tasks;
b-2) receives VM resource pool information capable of allocating the VM node from the resource pool management unit of the cloud management server, recalculates the number of available VM nodes and a map task that can be allocated to the VM node and a reduce task , ≪ / RTI &
b-3) Analyzing a job registered in the job queue of the master node and not allocated from the viewpoint of data or processing, and analyzing the size of the available VM resource pool information Relocating a job to a queue using the queue,
b-4) assigning the map task not allocated to the slave node and the task tracker of the VM node statically at compile time and dynamically at run time, respectively, based on the calculated information, and
b-5) periodically checks whether map tasks have been completed, and analyzes the available resource of the slave node when it has not been completed, receives a VM resource pool information and allocates a job to the slave node and the VM node The method of claim 1, wherein the Hadoop system includes a plurality of processors. The method according to claim 1,
The step of delivering the generated intermediate product of step c) to a reduce task comprises:
c-1) executing the map task according to an input format of a key and a value by a task tracker of the slave node and the VM node to generate an intermediate artifact;
c-2) exchanging and merging the intermediate artifacts of each slave node and the map task being executed by the division management unit of the VM node according to a key value, and
c-3) distributing and delivering the intermediate artifacts generated by the execution of the map task to the slave node and the redundancy task executing in the task tracker of the VM node, and delivering the distributed artifacts. How to run a cloud for support. The method according to claim 1,
The step of storing the resultant data in the HDFS of the slave node includes:
e-1) sorting and summing intermediate deliverables distributed and distributed from the slave node and the division management unit of the VM node, and
and e-2) the task tracker of the slave node and the VM node executes a task of reducing the sum of the intermediate outputs to an input value and storing the output file in the HDFS of the slave node. Distributed Parallel Processing A method of running a cloud to support Hadoop systems.

Images