KR102002246B1 - Method and apparatus for allocating resource for big data process - Google Patents

Abstract

The present invention relates to a method and an apparatus for distributing resources, capable of optimizing resource distribution by predicting an execution time by stochastically including failures of a Hadoop distributed file system (HDFS) and MapReduce. According to an embodiment of the present invention, a method for distributing resources for big data processing in a MapReduce and an HDFS comprises the following steps of: obtaining a failure probability (P_m) in a ″map″ step and a failure probability (P_r) in a ″reduce″ step; calculating an expectation value of the number of tasks by considering the obtained failure probability (P_m) and failure probability (P_r); predicting an execution time of data processing based on the calculated expectation value; and determining the number of slots to be allocated by using the predicted execution time.

Description

Resource distribution method and apparatus for big data processing {METHOD AND APPARATUS FOR ALLOCATING RESOURCE FOR BIG DATA PROCESS}

The present invention relates to a method and apparatus for distributing resources for processing big data, and more particularly, to a resource distribution capable of optimizing resource distribution by predicting execution time by probably including failures of the Hadoop distributed file system and MapReduce. A method and apparatus are disclosed.

Big data is a technology that is attracting attention in various fields and areas as the amount of digital information generated due to the proliferation of IoT and the spread of smart devices begins to explode. Big data system construction is to apply large amount of data efficiently to various convergence technologies and various systems such as IoT, AI, AR / VR. Related prior arts are Korean Patent Registration No. 10-1432751.

Hadoop is an open source framework for big data processing. It consists of Hadoop Distributed File System (HDFS), MapReduce data processing and resource manager Yarn.

For big data systems that can meet deadlines, Hadoop distributed file systems require resource allocation and work execution time prediction for them.

When configuring a distributed file system, it is necessary to allocate the least amount of resources to get the job done within the deadline. A resource is a map or reduce slot for each work node, which is a computing device like the CPU of each node when an individual computer is a node. It can be seen as a computing device of the provided virtual machine. . Optimization of these resources is important not only for efficient system operation but also for economic reasons. Hadoop does not support deadline requirements, so it has relied solely on user judgment. However, it is difficult for the user to predict this by considering various factors such as unexpected errors and failures.

There are three problems for the Hadoop system that are not solved by the existing prediction model. First, the existing prediction model is mainly based on the environment where failure does not occur. Second, failure probability is not included even in the case of the HP model in which failure is considered. Third, there was a problem of not considering the failure rate of MapReduce, server crash, disk drive failure rate, and failure due to large disk drive failure rate.

Therefore, it is necessary to study the technique for predicting execution time and optimizing resources in consideration of the probability of failure.

An object of the present invention is to provide a resource distribution method and apparatus for big data processing capable of resource optimization by predicting Hadoop execution time in consideration of failures occurring in the MapReduce and Hadoop distributed file system.

According to an embodiment of the present invention to achieve the above object, in the resource distribution method in the MapReduce and Hadoop Distributed File System (HDFS), the probability of failure (P _m ) and the recovery at the map stage Obtaining a probability of failure P _r in the deuce step; Calculating an expected value of the number of tasks in consideration of the obtained failure probability (P _m ) and failure probability (P _r ); Predicting execution time of data processing based on the calculated expected value; And determining the number of slots to allocate using the predicted execution time.

In order to achieve the above object, according to an embodiment of the present invention, in the resource distribution device in the MapReduce and Hadoop Distributed File System (HDFS), the probability of failure (P _m ) and the recovery at the map stage A failure probability acquisition unit for obtaining a failure probability P _r in the deuce step; An expected value calculator for calculating an expected value of the number of tasks in consideration of the obtained failure probability P _m and failure probability P _r ; An execution time estimator for predicting an execution time for data processing based on the calculated expected value; A slot number determiner configured to determine the number of slots to allocate using the estimated execution time; And a controller configured to control the failure probability obtaining unit, the expected value calculating unit, the execution time predicting unit, and the slot number determining unit.

The resource distribution method and apparatus for big data processing according to an embodiment of the present invention may optimize the resources allocated to the Hadoop system by effectively predicting execution time of failures occurring in the MapReduce and Hadoop distributed file systems.

1 is a diagram illustrating an example of a Hadoop distributed file system related to an embodiment of the present invention.
2 is a diagram illustrating an example in which a map reduce operation related to an embodiment of the present invention is performed in a cluster.
3 is a block diagram of a resource distribution apparatus according to an embodiment of the present invention.
4 is a flowchart illustrating a resource distribution method according to an embodiment of the present invention.
5 is a diagram illustrating a structure of a control message related to an embodiment of the present invention and the exchange of the same between nodes.

Hereinafter, a resource distribution method and apparatus for big data processing according to an embodiment of the present invention will be described with reference to the drawings.

As used herein, the singular forms "a", "an" and "the" include plural forms unless the context clearly indicates otherwise. In this specification, terms such as “consisting of” or “comprising” should not be construed as necessarily including all of the various components or steps described in the specification, and some of the components or some steps It should be construed that it may not be included or may further include additional components or steps.

1 is a diagram illustrating an example of a Hadoop distributed file system related to an embodiment of the present invention.

As shown, the Hadoop distributed file system operates by being divided into a NameNode 100 and a WorkNode 101, and the NameNode 100 processes a stored location of each file in a memory. The node 101 stores physical files in nodes distributed in block units. It is designed using inexpensive hardware, but in the event of a failure, adding a new worknode allows you to maintain the existing system, which is more fault tolerant than other distributed systems.

MapReduce is a programming model for data processing and divides the processing process into map and reduce steps. It is processed according to user defined map and reduce function, and has a master-slave structure in which one master node (JobTracker) manages a plurality of slave nodes (TaskTrackers).

2 is a diagram illustrating an example in which a map reduce operation related to an embodiment of the present invention is performed in a cluster.

As shown, Hadoop first splits the input data set and this data split is scheduled to one slave node (TaskTrackers) and processed by the map task (200).

Each TaskTracker has predefined slots for executing maps and reduce tasks. If there are more than one slot, they are executed in multiplewaves. Upon completion of the map task, all intermediate data <key, value> pairs are grouped using a sort-merge algorithm (201). It is then passed to the TaskTrackers reserved for the reduce task and shuffled (202). Finally, the reduce task processes intermediate data to generate a result (203).

Map reduce consists of the map phase, shuffle phase, and reduce phase. In the map phase, the input dataset is read from the Hadoop distributed file system, divided into chunks, and then Sends to Map function defined by. The Map function processes this to produce intermediate data.

The shuffle phase is performed simultaneously with the shuffling and sorting operations, sorting the intermediate values and making one or more copies. If the number of reduce tasks is smaller than the number of reduce slots, it ends with a single wave, and if it is large, it ends with multiple waves.

The reduce step processes the sorted intermediate values into a user-defined reduce function to produce the final output and write it to the Hadoop distributed file system.

Execution time prediction reflecting a failure according to an embodiment of the present invention has a different calculation depending on at what stage a failure occurs.

Hereinafter, a detailed execution time prediction method and a resource optimization method of the present invention will be described with reference to the restart portion of FIG. 2.

The upper arrow portion of FIG. 2 illustrates restart when failure occurs in each step of map reduction according to an embodiment of the present invention.

If a failure occurs in the map phase, the operation returns to the beginning of the map phase and reruns only for the already executed map task and the executed map task, and thus does not affect the reduce task (204). If a failure occurs in the reduce phase, the work must be done from the beginning of the map phase, so the failure in the reduce phase is very fatal for the entire work time (205).

The work proceeds as above, and the total execution time estimation and resource distribution can be performed as the following steps.

First, the number of MapTasks considering a failure can be examined, and the execution time of the MapPhase can be predicted using them. In addition, the number of reduce tasks can be examined and used to predict the execution time of the reduce phase. With them, we can estimate total run time and use them to perform resource allocation to each node.

Of course, the resource distribution step may be performed in order to perform the map task count check and the reduce task count check and predict the total execution time using them.

Hereinafter, it is possible to predict the final execution time reflecting the failure rate of each step according to an embodiment of the present invention, and will be described in detail with respect to the resource distribution apparatus and method that can optimize the resources to be allocated to the Hadoop system.

3 is a block diagram of a resource distribution apparatus according to an embodiment of the present invention.

As shown, the resource distribution device 300 includes a failure probability obtaining unit 310, an expected value calculating unit 320, an execution time predicting unit 330, a slot number determining unit 340, and a control unit 350. can do.

The failure probability acquisition unit 310 may acquire a failure probability P _m occurring in the map step and a failure probability P _r occurring in the reduce step. Since no failure occurs in the shuffle phase, the probability of failure in the shuffle phase is not obtained. The failure probability P _m and the failure probability P _r can be obtained empirically.

The expected value calculator 320 may calculate an expected value of the number of tasks in consideration of the acquired failure probability P _m and the failure probability P _r . For example, the expected value calculator 320 may calculate the expected value E (N _m.tf ) of the map task number and the number of shuffle tasks in consideration of the failure probability P _m and the failure probability P _r . the expected value (E (N _sh.2.tf)), Li deuce expected value of the number of tasks (E (N _r.tf)) can be calculated.

The runtime predictor 330 is the expected value of the calculated number of map tasks (E (N _m.tf)), the expected value of the shuffling task number (E (N _sh.2.tf)), and re-task deuce The total execution time can be estimated based on the expected value E (N _r.tf ).

The slot number determiner 340 may determine the optimal number of slots to allocate using the estimated total execution time.

The controller 350 may generally control the failure probability obtaining unit 310, the expected value calculating unit 320, the execution time predicting unit 330, and the slot number determining unit 340.

Hereinafter, a method of predicting execution time in consideration of a failure probability and determining a slot based thereon will be described in detail.

,

,

라고 할 때, 각 단계의 실행 시간의 합으로 모델링 할 수 있으며 다음의 수학식 1과 같이 표현될 수 있다.The final execution time is the execution time of the map reduction, shuffle phase, and reduce phase of MapReduce, respectively.

,

,

In this case, it can be modeled as the sum of execution times of each step and can be expressed as Equation 1 below.

,

,

,

이라 하고, 셔플은 first wave와 other wave의 숫자와 평균 수행 시간이 다르므로 fist wave와 other wave를 각각 다르게

,

,

,

로 놓는다. 또한 각 태스크를 수행하는 node에서의 슬롯 수를 맵 슬롯(map slot)을

, shuffler과 리듀스 슬롯(reduce slot)을

로 놓으면, 각 단계의 수행 시간과 총 수행 시간은 수학식 2 와 같이 표현될 수 있다.To find the time of each step, we need the number of tasks to be performed in the map and reduce steps, and their average execution time.

,

,

,

Since shuffles have different numbers and average execution times of first and other waves,

,

,

,

Place it. Also, map slots are used to determine the number of slots in the node that performs each task.

, shuffler and reduce slots

If set to, the execution time and total execution time of each step may be expressed as in Equation 2.

라 할 때, 수학식 2를 이용해 실패가 발생한 시각까지 수행한 태스크의 숫자를 구할 수 있다. 맵 단계에서 실패가 발생했다면, 실패 발생까지 수행하여 실패 후에 다시 수행해야 하는 태스크 숫자를

이라 할 때 이는 수학식 3과 같이 표현될 수 있다.Now, to find a formula that takes into account the failure,

In this case, using Equation 2, the number of tasks performed up to the time of failure can be obtained. If a failure occurred in the map phase, run the failure up to the number of tasks that must be performed again after the failure.

This may be expressed as Equation 3 below.

개라고 할 때 결과적으로 재수행시 한 work node에서 추가적으로 수행해야 하는 태스크(task)수 N_m.fail.m는 수학식 4와 같이 표현할 수 있다.The work node in the map phase

As a result, the number of tasks N _m.fail.m to be additionally performed in one work node when re-executing can be expressed as in Equation 4.

In Equation 4, if the parenthesis is not divided into rounding marks and there is a task remaining, the number of executions is added once more. The total number of tasks that must be performed is the sum of the number of tasks that must be performed if the original failure did not occur and the number of tasks that must be performed again because of the failure. When the original task number N _m.1, the total number N _m.tf tasks referred to need to do, it can be represented by the equation N _m.tf 5.

On the other hand, since the failure in the map phase does not affect the reduce task, the number of reduce tasks N _r.tf in consideration of the failure may be expressed by Equation 6.

는 수학식 7과 같이 표현할 수 있다.If the failure occurs in the reduce phase and the number of reduce tasks performed so far is N _r.done , the time at which the failure occurred

Can be expressed as in Equation 7.

In summary, the _reduced task number N _r.done performed until failure occurs may be expressed as in Equation 8.

개라고 할 때 리듀스에서 실패가 생겨서 재수행을 할 경우, 맵 단계와 리듀스 단계에서 한 워크 노드에서 추가적으로 수행해야 하는 태스크수 N_m.fail.r, N_r.fail는 수학식 9로 나타낼 수 있다.The outermost parenthesis in Equation 8 is rounded up, and N _r.done is taken as an integer. Like the way in the map phase,

In case of redo when redistribution occurs due to failure, the number of additional tasks N _m.fail.r and N _{r.fail to} be performed in one work node in map and reduce phase are represented by Equation 9. Can be.

In Equation (9), if the remaining values are the same as described above, the number of executions should be increased once more. The map phase is divided into a first stage task N _m.1 that is purely executed in the map phase and a second stage task executed in parallel with the shuffle task in the shuffle stage. However, no error occurs in the second step task executed in parallel with the shuffle step. Therefore, when calculating the number of tasks considering failure in the map phase, only the first stage task N _m.1 is considered. However, if a failure occurs in the reduce step, the first step task and the second step task must be repeated again. In Equation 9, N _m is a value obtained by adding up the number of first step tasks and the number of second step tasks.

Therefore, the total number of tasks that will be actually performed when a failure occurs and is re-executed can be represented by Equation 10.

4 is a flowchart illustrating a resource distribution method according to an embodiment of the present invention.

First, the failure probability acquisition unit 310 may acquire a failure probability P _m occurring in the map step and a failure probability P _r occurring in the reduce step (S410). The failure probability P _m and the failure probability P _r can be obtained empirically.

The expected value calculator 320 may calculate an expected value of the number of tasks in consideration of a failure probability (S420).

라 하면 이를 반영한 맵과 리듀스에서 수행해야 할 태스크 수의 기대값인

는 수학식 11로 표현 할 수 있다.The probability of failure in the map and reduce phases

Is the expected value of the number of tasks

Can be expressed by Equation 11.

을 수학식 12로 예측할 수 있다(S430).The execution time estimator 330 uses the equation (10) to determine the total execution time.

It can be predicted by Equation 12 (S430).

In Equation 12, since the task number is not divided by the slot number, it is necessary to perform it again.

가 데드라인 t를 넘기지 않는 조건에서

및

의 최소값을 산출하고, 산출된 최소값을 최적의 슬롯 개수로 결정할 수 있다.The slot number determiner 340 may determine the number of slots in the map step, shuffle step, and reduce step by using Equation 12 to allocate slots for each step (S440). For example, the slot number determiner 340 uses the equation (12) to determine the number of slots.

Does not cross the deadline t

And

The minimum value of may be calculated, and the calculated minimum value may be determined as the optimal number of slots.

이 최소값을 갖도록 하는 방법에 대해 설명하도록 하겠다. In the following embodiment, the Lagrange Multipliers method is used to determine the number of slots in the map and reduce.

I will explain how to have this minimum.

은 수학식 13, 목적식

은 수학식 14 로 나타낼 수 있다.Constraints

Is Equation 13, objective

May be represented by Equation 14.

Therefore, Lagrangian functoin and its derivative are solved as shown in Equation 15.

을 최소화 하는 값을 구하면,

은 아래의 수학식 16을 만족할 때 최소화 된다.With the above equations

If you find a value that minimizes

Is minimized when the following Equation 16 is satisfied.

In this way, the optimal allocation of map and reduce slots is possible, and the optimal allocation is less likely to drop over the deadline, so retransmission and retry are not performed. The result is better system efficiency and economics.

5 is a diagram illustrating a structure of a control message related to an embodiment of the present invention and the exchange of the same between nodes.

를, 발생하지 않았다면 비워둔다 (503). 이 메시지의 비트는 노드 수와 데드라인의 변화에 따른 실패(Failure) 시각 변화에 따라 변화가 가능하고, 또한 이 메시지는 단독으로 컨트롤 메시지로 사용되거나 IP layer, TCP layer, MAC layer 등의 헤더의 옵션 필드에 적용하여 각 노드와 컨트롤러에서 주고받는 것이 가능하다 (504).As shown in FIG. 5, the HALF controller 500 and the nodes may be controlled through a message exchanged between the nodes. The first bit of the message indicates whether there is an error (501), the second to fifth numbers of the node that sent the message (502), and the sixth to ninth failures.

If not, leave blank (503). The bit of this message can be changed according to the change of failure time according to the change of node number and deadline, and this message can be used alone as a control message or in the header of IP layer, TCP layer, MAC layer, etc. It is possible to send and receive from each node and controller by applying to the option field (504).

As described above, the resource distribution method and apparatus for big data processing according to an embodiment of the present invention effectively predicts execution time for failures occurring in MapReduce and Hadoop distributed file system, thereby managing resources allocated to the Hadoop system. Can be optimized

The above-described resource distribution method for big data processing may be implemented in the form of program instructions that can be executed by various computer means and recorded in a computer-readable recording medium. In this case, the computer-readable recording medium may include program instructions, data files, data structures, and the like, alone or in combination. Meanwhile, the program instructions recorded on the recording medium may be those specially designed and configured for the present invention, or may be known and available to those skilled in computer software.

Computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks such as floppy disks. Magnetic-Optical Media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like.

The recording medium may be a transmission medium such as an optical or metal wire, a waveguide, or the like including a carrier wave for transmitting a signal specifying a program command, a data structure, or the like.

In addition, program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

The resource distribution method and apparatus for the big data processing described above may not be limitedly applied to the configuration and method of the above-described embodiments, but the embodiments may be all or all of the embodiments so that various modifications may be made. Some may be optionally combined.

300: resource distribution device
310: failure probability acquisition unit
420: expected value calculation unit
330: execution time prediction unit
340: slot number determination unit
350: control unit

Claims (16)

In the resource distribution method of MapReduce and Hadoop Distributed File System (HDFS),
Obtaining a failure probability P _m at the map stage and a failure probability P _r at the reduce stage;
Calculating an expected value of the number of tasks in consideration of the obtained failure probability (P _m ) and failure probability (P _r );
Predicting execution time of data processing based on the calculated expected value; And
And determining the number of slots to allocate using the predicted execution time.
The method of claim 1, wherein calculating the expected value
Calculating an expected value of the number of map tasks in consideration of the failure probability P _m ; And
And calculating an expected value of the number of map and reduce tasks in consideration of the failure probability (P _r ).
The method of claim 2, wherein the calculating of the expected value of the number of map tasks
And calculating an expected value of the number of map tasks in consideration of four cases in which the failure occurs or does not occur in the map step and the reduce step, respectively.
The method of claim 2, wherein the calculating of the expected values of the map and the number of reduce tasks is performed.
And calculating an expected value of the number of reduce tasks in consideration of two cases where a failure occurs or does not occur in the reduce step.
The method of claim 2, wherein the calculating step of calculating an expected value of the number of tasks in consideration of the obtained failure probability P _m and the failure probability P _r is performed.
And calculating an expected value of the number of shuffle tasks in consideration of two cases where a failure occurs or does not occur in the reduce step.
The method of claim 1, wherein calculating the expected value
Calculating an expected value of the number of map tasks, an expected value of the number of shuffle tasks, and an expected value of the number of reduce tasks, the expected value of the number of map tasks, an expected value of the number of shuffle tasks, and an expected number of reduce tasks The value is calculated by Equation 1 below.
[Equation 1]

Where E (N _m.tf ) is the expected value of the number of map tasks, E (N _sh.2.tf ) is the expected value of the number of shuffle tasks, and E (N _r.tf ) is the number of reduce tasks Expected value P _m is the probability of failure in the map phase, P _r is the probability of failure in the reduce phase N _m.1 is the number of map tasks if no failure occurred in the map phase, and N _{m. fail.m} is the number of additional map tasks to perform on one work node if a failure occurs in the map phase, and N _m.fail.r , N _r.fail are the map phase and reduce if the failure occurs in the reduce phase. The number of additional tasks to be performed on one work node in a step, N _sh.2 is the number of shuffled tasks if no failures occur in the reduce phase.)
7. The method of claim 6, wherein estimating execution time is
Resource distribution method characterized in that performed using the following equation (2).
[Equation 2]

(here,

Is the total run time,

Is the average execution time of one task in the map phase,

Is the average execution time of one task in the map phase,

Is the average execution time of one task in the shuffle phase,

The average execution time of one task in the shuffle phase 2, N _sh. 1 is the number of tasks in the shuffle phase 1, and N _sh. 2 is the number of tasks in the shuffle phase 2.

Is the number of map slots,

Is the number of reduce slots.)
8. The method of claim 7, wherein determining the number of slots to allocate
The above using Equation 2

Does not cross the deadline t

And

Calculating a minimum value of the resource distribution method.
In the resource distribution device in MapReduce and Hadoop Distributed File System (HDFS),
A failure probability acquisition unit for obtaining a failure probability P _m at a map stage and a failure probability P _r at a reduce stage;
An expected value calculator for calculating an expected value of the number of tasks in consideration of the obtained failure probability P _m and failure probability P _r ;
An execution time estimator for predicting an execution time for data processing based on the calculated expected value;
A slot number determiner configured to determine the number of slots to allocate using the estimated execution time; And
And a controller for controlling the failure probability obtaining unit, the expected value calculating unit, the execution time predicting unit, and the slot number determining unit.
The method of claim 9, wherein the expected value calculator
The expected value of the number of map tasks is calculated in consideration of the failure probability P _m ,
And an expected value of the number of map and reduce tasks in consideration of the failure probability (P _r ).
The method of claim 10, wherein the expected value calculator
And calculating an expected value of the number of map tasks in consideration of four cases where failure occurs or does not occur in the map step and the reduce step, respectively.
The method of claim 10, wherein the expected value calculator
And calculating an expected value of the number of reduce tasks in consideration of two cases where a failure occurs or does not occur in the reduce step.
The method of claim 10, wherein the expected value calculator
And calculating an expected value of the number of shuffle tasks in consideration of two cases where a failure occurs or does not occur in the reduce step.
The method of claim 9, wherein the expected value calculator
The expected value of the number of map tasks, the expected value of the number of shuffle tasks, and the expected value of the number of reduce tasks are calculated. A resource distribution device, characterized in that calculated by the formula (1).
[Equation 1]

Where E (N _m.tf ) is the expected value of the number of map tasks, E (N _sh.2.tf ) is the expected value of the number of shuffle tasks, and E (N _r.tf ) is the number of reduce tasks Expected value P _m is the probability of failure in the map phase, P _r is the probability of failure in the reduce phase N _m.1 is the number of map tasks if no failure occurred in the map phase, and N _{m. fail.m} is the number of additional map tasks to perform on one work node if a failure occurs in the map phase, and N _m.fail.r , N _r.fail are the map phase and reduce if the failure occurs in the reduce phase. The number of additional tasks to be performed on one work node in a step, N _sh.2 is the number of shuffled tasks if no failures occur in the reduce phase.)
15. The method of claim 14, wherein the execution time predictor
Resource distribution apparatus characterized in that for performing the execution time prediction using the following equation (2).
[Equation 2]

(here,

Is the total run time,

Is the average execution time of one task in the map phase,

Is the average execution time of one task in the map phase,

Is the average execution time of one task in the shuffle phase,

The average execution time of one task in the shuffle phase 2, N _sh. 1 is the number of tasks in the shuffle phase 1, and N _sh. 2 is the number of tasks in the shuffle phase 2.

Is the number of map slots,

Is the number of reduce slots.)
The method of claim 15, wherein the slot number determiner
The above using Equation 2

Does not cross the deadline t

And

Calculating a minimum value of the resource distribution apparatus.

Images