KR20090066010A - Apparatus for sorting and combining intermediate results of multiple map tasks on the distributed parallel processing system and method thereof - Google Patents

Abstract

A multiple map task intermediate result queue, a combined device and a method thereof in a distributed parallel processing system removing a duplicate key existing in a whole destination file are provided to reduce task processing load, the number of task output file and total size of an output file transmitted in the distributed parallel processing system. A map task performing unit generates an intermediate result by performing the allocated map task. A reduce task performance removes task transmission schedule information on a transmission schedule intermediate result list(S512). The reduce task performance sets up intermediate result path information about a residual task identifier(S522).

Description

Apparatus for Sorting and Combining Intermediate Results of Multiple Map Tasks on the Distributed Parallel Processing System and Method

The present invention relates to an apparatus and method for sorting and combining intermediate results of multiple Map tasks in a distributed parallel processing system. In particular, key / value pairs generated by multiple Map tasks at the node level of a distributed parallel job processing system supporting a MapReduce programming model. The number of Map task output files passed to the Reduce task execution node, the total size of the output files, and Reduce by combining them into one output file through duplicate key elimination and sorting of multiple output files consisting of a list of the An apparatus and method for sorting and combining intermediate results of multiple Map tasks in a distributed parallel processing system that can reduce task processing load.

The present invention is derived from the research conducted as part of the IT new growth engine core technology development project of the Ministry of Information and Communication and the Ministry of Information and Communication Research and Development. [Task Management Number: 2007-S-016-01, Title: Low-cost, large-scale global Internet service solution Development].

As the Internet develops, a lot of data is generated and distributed every day by netizens, and recently, a large amount of data has been collected and accumulated as much as possible among many companies, especially search engine companies and web portals. Extracting meaningful information as quickly as possible is becoming a competitive advantage for companies.

As a result, many companies are building large clusters at low cost and doing a lot of research on large-scale data distribution management and work-distribution parallel processing technologies. It is attracting attention as a treatment method.

The MapReduce model is a distributed parallel programming model that Google proposes to support distributed parallelism of large amounts of data stored on clusters of low-cost large nodes. One of the tasks you write is based on the Map function you write. It consists of two phases, the Map phase, and the Reduce phase, which is the main axis of the Reduce function that the user creates.

In this case, each Map and Reduce steps are replicated to multiple nodes as multiple tasks and distributed and executed in parallel.In the Map step, the operation of extracting key / value pairs from input data is basically performed. Eliminate duplicate keys in the extracted key / value pairs and apply business logic to perform operations that consist of the desired final result key / value pair.

However, the MapReduce model creates constraints in the form of key / value pairs for the input / output data model of the job and the input / output data models of the Map and Reduce tasks for efficient distributed parallel processing. There are often duplicate keys in the intermediate file that are being created.

Since duplicate keys increase network transmission costs, Google supports / uses the concept of a combiner to eliminate this. The combiner is a module that removes duplicate keys by performing the Reduce function, which is created by the user and performed during the Reduce task, in advance in the Map task execution node, and performs partial sorting to generate partial results in advance. By using it, you can reduce the processing load on the Reduce task by reducing the size of intermediate results that are passed to the Reduce task execution node.

Fortunately, recently, multi-core processors are becoming less expensive, with low-cost large clusters that are equipped with multi-core processors, which means that the computational speed is high and distributed parallelism is not a problem.

However, in low-cost large clusters of low-cost PCs with multicore processors, multiple Map tasks belonging to one job are more likely to be performed on a single node, so applying a combiner only removes duplicate keys in individual result files. The problem is that you cannot remove duplicate keys that exist in the entire result file generated by the entire Map task.

The present invention combines a single output file through duplicate key removal and sorting of multiple output files consisting of a list of key / value pairs generated by multiple Map tasks at the node level of a distributed parallel job processing system that supports the MapReduce programming model. Device to sort and combine multiple Map task intermediate results in a distributed parallel processing system that can reduce the number of Map task output files that are passed to the Reduce task execution node, the overall size of the output files, and the burden of processing Reduce tasks by passing them to the next Reduce task input. And to provide a method.

In order to achieve the above object, a multi-map task intermediate result sorting and combining method according to the present invention includes: (a) receiving split input data, assigning a user-generated Map task to one or more nodes, and distributing the Map task in parallel and parallel processing; Generating at least one intermediate result file; (b) applying a Reduce function to one or more intermediate result files generated by the respective Map tasks, and generating one or more intermediate results through duplicate key removal and sorting; (c) After the distributed parallel processing of the Map task is finished, distributed parallel processing of one or more intermediate results through one or more Reduce tasks generates one or more final results.

Here, the method may further include reading the divided portion of the entire input data before the step (a), dividing it into a size suitable for processing by the user-written Map function, and loading the same into a memory.

Step (a) may include: (a-1) extracting a user-defined key / value by performing the Map function and loading the same into a buffer in a memory; (a-2) sorting a list of key / value pairs when a certain amount of data is accumulated in the buffer; (a-3) partitioning the sorted list of key / value pairs into an intermediate result file for an individual Reduce task.

In addition, the step (b) may include (b-1) combining and removing duplicate keys of the intermediate result file; (b-2) arranging the combined intermediate result files to generate individual intermediate results, wherein in the step (c), the intermediate result generation for the whole divided input data is completed. It is desirable to be informed of.

A plurality of Map task task managers that generate a Map task executor, extract key / value pairs through distributed parallel processing on split input data, and generate an intermediate result file when the map task is allocated according to another aspect of the present invention; One or more task level combiners configured to remove intermediate keys by applying a reduce function to the intermediate result file generated by the task manager of the individual map execution node, and then sort and generate intermediate results; Comprising a node level combiner that consolidates the generated intermediate results when the entire assigned Map task is completed, removes duplicate keys and sorts them to produce the final intermediate result file, and processes the reduce task by removing duplicate keys of the final intermediate file. In the distributed parallel processing system, which is characterized by a low load, a multi-map task intermediate result sorting and combining apparatus is provided.

Here, in the distributed parallel processing system according to the present invention, a multi-map task intermediate result sorting and combining apparatus is configured to divide and load the divided input data into a size suitable for performing the Map task, the map performing task manager, and the task level combiner. And a task manager that controls and manages the entire work process performed by the node level combiner. It further includes a Reduce Perform Task Manager that generates a Reduce Task Performer to perform a Reduce task to produce the final result consisting of the desired final key / value pair.

Here, when the Map task executor completes the Map task, the map task executor notifies the map task task manager of the map task completion information including <map task identifier, intermediate result file path>, and the node level combiner provides the map task task manager. After the Map task is completed, an intermediate result is generated, but the intermediate result of the Map task which is not delivered to the Reduce task performer is inserted into the "list of intermediate results to be combined" to be managed.

Meanwhile, the Reduce task executor generates and manages a "transmission scheduled intermediate result list" from <Map task identifier and intermediate result file path> information generated from the Map task completion notification delivered by the Map task executor.

An apparatus and method for sorting and combining intermediate results of multiple Map tasks in a distributed parallel processing system according to the present invention is a list of key / value pairs generated by multiple Map tasks at a node level of a distributed parallel job processing system supporting a MapReduce programming model. By combining and eliminating duplicate keys in the configured multiple output files into one output file and passing it to the Reduce task input, the number of Map task output files passed to the Reduce task execution node, the total size of the output files, and the Reduce task processing load. There is an effect to reduce.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following embodiments are provided to those skilled in the art to fully understand the present invention, can be modified in various forms, the scope of the present invention is limited to the embodiments described below no.

1 is a block diagram illustrating a data transmission process between a Map task and a Reduce task according to an embodiment of the present invention. As shown in FIG. 1, an exemplary process is composed of m Map tasks 110_1 to 110_m and r Reduce tasks 150_1 to 150_r, respectively, to be distributed and processed in parallel, and an individual Map task (for example, 110_1) Since r intermediate result files (eg, 120_1 to 120_r) to be transmitted to r Reduce tasks 150_1 to 150_r are generated, m Map tasks 110_1 to 110_m generate a total of m × r intermediate result files.

In this case, the individual Reduce task (eg, 150_1) copies the intermediate results 120_1, 130_1, ˜ 140_1 in parallel using a limited number of copier threads.

At this time, if the number of copy threads 160 is configured as m number of map tasks, it is possible to deliver intermediate results of the map task to individual reduce tasks in one parallel copy. However, when the number of map tasks is large, the map task Since there are too many connection sessions for the transfer of intermediate results between the performing node and the reducing task performing node, it takes a lot of load to establish, maintain, and terminate the connection session. Therefore, it is preferable that the copy thread 160 generate and use only a certain number. Do.

For example, when using c copy threads 160, all m Map task intermediate results 120_1, 120_2, ˜ 140_1 transmitted to individual Reduce tasks (eg, 150_1) are sequentially transmitted by c.

Figure 2 is a graph showing the total time that is the time required to transfer the intermediate result file Map task by capacity and number according to an embodiment of the present invention to the Reduce task.

Here, the x-axis of the graph represents the capacity × number [Byte] of the Map task intermediate result file, and the y-axis represents the time [Second], and the output A of the graph represents the unit transmission time [MB / s], which is the capacity of the file transferred per second, B is the total transfer time [Second], which is the time taken to transfer the entire intermediate result file.

In FIG. 2, it can be interpreted that the larger the number of result files of the Map task and the smaller the file size, the larger the unit transfer time, that is, the data transfer performance tends to decrease. This is because, as the number of files increases, the load on processing other than the actual data transfer process such as connection establishment process, transfer preparation process, and connection termination process for data transmission increases.

From this, it can be inferred that the overall time required for distributed parallel processing can be reduced if the size of the Map task intermediate result file is maintained above a certain level and the number of files is reduced, as suggested by the present invention.

3 is a block diagram illustrating a distributed parallel processing system according to an embodiment of the present invention. As shown in FIG. 3, one job manager 311 is performed by the job management node 310 to perform control and management of the entire job processing process performed by the plurality of task managers 321 and 421, and a separate job manager 311. Task task nodes 320 and 330 may be performed to manage a Map task and a Reduce task.

The job manager 311 is performed by the job manager node 310 to control and manage the overall job processing performed by the plurality of task managers 321 and 331.

The task managers 321 and 331 are executed in a task execution node (eg, 320) to manage the Map task and the Reduce task, and one node level combiner 322, a Map task executor 324 that actually performs a plurality of Map tasks. And a Reduce task executor 323 that actually performs a number of Reduce tasks.

The Map task executor 324 or the Reduce task executor 323 is created in memory while the Map task or the Reduce task is assigned and performed, and is removed when the individual task execution is completed.

The Map task is an operation that extracts key / value pairs from the input data. The Reduce task is an operation process that removes duplicate keys in the extracted key / value pairs and uses the business logic to construct the desired final key / value pair.

Node level combiner 322 also performs the Reduce function on the Map task execution node to perform duplicate key removal, partial sorting, and partial result pre-generation to reduce the intermediate result file size passed to the Reduce task execution node, Performer 323 reduces processing load.

4 is a block diagram illustrating a process of combining intermediate results of a Map task according to an embodiment of the present invention. As illustrated in FIG. 4, the map task intermediate result combining is performed by performing a map task, task level combining, and node level combining.

First, the divided portion 411 of the entire input data managed on the global file system 410 is read, divided into pieces suitable for processing by the user-written Map function, and loaded into the memory (S410).

Subsequently, the user map function is applied to the divided input data, and the user-defined key / value is extracted and stacked in the in-memory buffer 420 (S420).

Whenever a certain amount of data is accumulated in the buffer 420, a process of sorting a list of key / value pairs is performed (S430).

Next, the sorted list of key / value pairs is divided into partial results for each Reduce task (S440), and duplicate keys of each divided partial result are removed and combined (S450).

Subsequently, the combined partial result is created as a temporary file (intermediate file) (S460). After all Map task processing for the divided part of the input data is completed, the generated partial result temporary file is read and merged (S470).

Then, the integrated partial result is rewritten to generate the final intermediate result 440 (S470).

Thereafter, using the node-level combiner 430, a plurality of map tasks are performed on one node to remove duplicate keys for a plurality of final intermediate results 440 ˜ generated to construct an integrated result file 450 ( S490).

Here, if the processing time of the entire Map task for the divided portion of the input data is more than a predetermined time, the node level combiner 430 may generate the integrated result file 450 for some Map tasks in advance and deliver the result to the Reduce task. .

5 is a flowchart illustrating a process of processing a Map task and a Reduce task according to an embodiment of the present invention centering on a node level combining process. A description with reference to FIG. 5 is as follows.

The map performing task manager 530 newly generates a map task performer 520 when the map task is assigned (S501), and the reduce performing task manager 550 generates a new reduce task performer 560 when the reduce task is assigned. (S502).

At this time, the map task executor 520 performs the assigned Map task to generate an intermediate result, and the reduce task executor 560 performs the assigned Reduce task to generate the final result.

When the Map task executor 520 completes the execution of the Map task (S503), the map task executor 520 notifies and transmits the map task completion information including <Map task identifier and intermediate result file path> to the map execution task manager 530 (S504). .

The map performing task manager 530 notifies all Reduce performing task managers 550 of the received Map task completion information through the job manager 540 (S505) (S506), and the Reduce performing task manager 550 performs all of the Reduce. The task executor 560 is notified of this (S507).

The Reduce task executor 560 inserts <Map task identifier, intermediate result file path> information notified from the Map task executor 520 into the "send intermediate result list" (S508), and then "send intermediate result list". In step S509, the information on <Map task identifier and intermediate result file path> is sequentially taken out (S509), and a request for transmitting a map task intermediate result is requested to the map performing task manager 530 (S510).

Then, the Map performing task manager 530 transmits the intermediate result to the Reduce task performer 560 (S511), and the Reduce task executor 560 completes the intermediate result transmission in the "Map to be sent" list when the intermediate results are completed. The task transmission schedule information is removed (S512).

The node level combiner 510 receives a plurality of Map task intermediate results from the Map performing task manager 530 in which the Map task is completed and generated intermediate results, but has not yet been passed to the Reduce task performer 560 (S513). It is inserted into the scheduled intermediate result list "and managed (S514).

When the execution of the Map task is completed (S514), the node level combining process is applied to the untransmitted multiple intermediate results.

At this time, the intermediate result transmitted from the Map performing task manager 530 to the Reduce task performer 560 before the node level combining process is applied is removed from the "combining scheduled intermediate result list" (S515).

Meanwhile, the intermediate result at which the node level combining process is completed (S516) and the intermediate result generated during the Map task processing are also removed from the “combination scheduled intermediate result list” (S517).

Here, the node-level join information is passed to the Reduce task executor 560 through the Map task task manager 530, the task manager 540, and the Reduce task task manager 550 in the form of <Map task identifier list, join intermediate result path>. Notification is made (S518, S519, S520, S521).

The reduce task executor 560 removes the remaining task information except for one by reflecting this in the "send scheduled intermediate result list" and sets newly received Map intermediate result path information for the remaining task identifier (S522).

In this case, the reflection on the "intermediate result list to be transmitted" is performed during the intermediate result selection process for the transmission request.

FIG. 6 is a flowchart illustrating a process of performing an operation of counting words included in input data using a multi-map task intermediate result sorting and combining apparatus in a distributed parallel processing system according to an exemplary embodiment of the present invention. A description with reference to FIG. 6 is as follows.

In this embodiment, since the input data of the word counting job is divided into two divided files, the first partition 611 is divided by the first map task 621, and the second partition 612 is the second map task ( 622).

The first Map task 621 generates a first intermediate result file 631 having a key / value of all occurrence words included in the first partition 611 and the number of occurrences of each word, respectively, and the task level combiner 641. ) Applies the Reduce function to the first intermediate result file 631 to generate the first intermediate result 651 with the duplicate key removed.

The second Map task 622 generates a second intermediate result file 632 with key / values for every occurrence word and number of occurrences of each word in the second partition 612, respectively, and task level combiner 642. ) Applies the Reduce function to the second intermediate result file 632 to produce a second intermediate result 652 that eliminates duplicate keys.

The node level combiner 660 removes duplicate keys of the first intermediate result 651 generated through the first Map task 621 and the second intermediate result 652 generated through the second Map task, and then sorts the result. Generate a final intermediate result file 670 that combines the intermediate result files of several Map tasks into one intermediate result file.

The configuration of the present invention has been described above through the preferred embodiments and the accompanying drawings. However, these are only examples and are not used to limit the scope of the present invention. Those skilled in the art will understand from this that various modifications and equivalent other embodiments are possible. The true scope of protection of the present invention should be defined by the technical spirit of the appended claims.

1 is a block diagram illustrating a data transmission process between a Map task and a Reduce task according to the present invention.

Figure 2 is a graph showing the total time that is the time required for the transfer of the Map task intermediate result file by capacity and number according to the present invention to the Reduce task.

3 is a block diagram illustrating a distributed parallel processing system in accordance with the present invention.

4 is a block diagram illustrating a process of combining intermediate results of a Map task according to the present invention;

5 is a flowchart illustrating a node task combining process of a Map task and a Reduce task process according to the present invention.

FIG. 6 is a flowchart illustrating a process of performing an operation of counting words included in input data using a multi-map task intermediate result sorting and combining apparatus in a distributed parallel processing system according to the present invention. FIG.

<Description of main parts of drawing>

510: Node level combiner 520: Map task performer

530: Map Perform Task Manager 540: Task Manager

550: Reduce Perform Task Manager 560: Reduce Task Performer

Claims (12)

(a) receiving split input data, assigning a user-written Map task to one or more nodes, and generating one or more intermediate result files by performing distributed parallel processing of the Map task; (b) applying a Reduce function to one or more intermediate result files generated by the respective Map tasks, and generating one or more intermediate results through duplicate key removal and sorting; (c) when distributed parallel processing of the Map task is completed, distributed parallel processing of one or more intermediate results through one or more Reduce tasks to generate one or more final results; Method of sorting and combining intermediate results of the multi-map task comprising a. According to claim 1, Before the step (a), Reading the divided portion of the entire input data, dividing the divided portion into sizes suitable for processing by the user-written Map function, and loading the same into memory; Method for sorting and combining intermediate results of the multiple map task further comprises. According to claim 1, wherein step (a), (a-1) extracting a user-defined key / value by performing the Map function and loading the same into a buffer in a memory; (a-2) sorting a list of key / value pairs when a certain amount of data is accumulated in the buffer; (a-3) dividing the sorted list of key / value pairs into an intermediate result file for an individual Reduce task; Method of sorting and combining intermediate results of the multi-map task comprising a. According to claim 1, wherein step (b), (b-1) combining and removing duplicate keys of the intermediate result file; (b-2) sorting the combined intermediate result files to produce individual intermediate results; Method for sorting and combining intermediate results of multiple Map task comprising a. The method of claim 1, wherein in the step (c), the intermediate result generation for the whole divided input data is completed. Determination of notification of the completion information of the individual Map task execution Method for sorting and combining intermediate results of multiple Map tasks. A plurality of Map execution task managers that generate a Map task executor, extract key / value pairs through distributed parallel processing on split input data, and generate an intermediate result file when the map task is assigned; One or more task level combiners configured to remove intermediate keys by applying a reduce function to the intermediate result file generated by the task manager of the individual map execution node, and then sort and generate intermediate results; A node level combiner that consolidates the generated intermediate results upon completion of the assigned total Map task, removes duplicate keys, and sorts to produce the final intermediate result file, Device for sorting and combining the results of multiple Map tasks in a distributed parallel processing system, characterized in that the reduced task processing load due to the elimination of duplicate keys of the final intermediate file. The method of claim 6, A memory for dividing and dividing the divided input data into a size suitable for performing the Map task; Apparatus for sorting and combining the results of multiple intermediate map tasks in a distributed parallel processing system further comprising. The method of claim 6, A task manager configured to control and manage an entire work process performed by the map performing task manager, the task level combiner, and the node level combiner; Apparatus for sorting and combining the results of multiple intermediate map tasks in a distributed parallel processing system further comprising. The method of claim 6, wherein the Map task executor, When the Map task execution is completed, notify the Map execution task manager of the Map task completion information including <Map task identifier, intermediate result file path>. An apparatus for sorting and combining intermediate results of multiple Map tasks in a distributed parallel processing system. The method of claim 6, Reduce Perform Task Manager, which creates a Reduce Task Attendant to perform a Reduce task to produce the final result consisting of the desired final key / value pair; Apparatus for sorting and combining the results of multiple intermediate map tasks in a distributed parallel processing system further comprising. The method of claim 10, wherein the node-level combiner, Inserting and managing an intermediate result of the Map task which has been generated by the completion of the Map task of the Map performing task manager but not delivered to the Reduce task performer to the "combined intermediate result list" An apparatus for sorting and combining intermediate results of multiple Map tasks in a distributed parallel processing system. The method of claim 10, wherein the reduce task executor, Arranging and managing intermediate results of multiple Map tasks in a distributed parallel processing system by generating and managing "transferred intermediate result list" from <Map task identifier and intermediate result file path> information generated from the Map task completion notification delivered by the Map task performer. Coupling device.

Images