CN104951372A - Method for dynamic allocation of Map/Reduce data processing platform memory resources based on prediction - Google Patents

Abstract

Provided is a method for dynamic allocation of Map/Reduce data processing platform memory resources based on prediction. The allocation method comprises the steps of initialization, task memory resource use prediction, task memory resource release, task memory resource adding and backtrack. In the method, against the characteristic of obvious fluctuation of the memory resource use amount during the Map task and Reduce task running process, according to historical records of the memory use amount during the Map task and Reduce task running process, a linear regression and t test method is adopted, task memory use rules are calculated, the memory amount needing to be used in task follow-up operation process is predicted, the memory allocation amount of a Map task and a Reduce task which are in running is dynamically added or reduced according to the predicted task memory use amount, so that the use efficiency of the Map/Reduce data processing platform memory resources is effectively improved, and the execution efficiency of Map/Reduce operation is improved.

Description

A kind of Map/Reduce data processing platform (DPP) memory source dynamic allocation method based on prediction

Technical field

The invention belongs to Distributed Calculation field, be specifically related to usage forecastings and the dynamic allocation method of memory source in Map/Reduce type mass data processing platform.

Background technology

Map/Reduce is a kind of novel parallel computational model, has been widely used in mass data processing field.Internal memory is the important computations resource supporting Map/Reduce application operation.In actual motion, a Map/Reduce application is made up of one or more Map/Reduce operation.The execution of each Map/Reduce operation comprises a Map stage and a Reduce stage usually.Wherein, Map stage and Reduce stage can be mapped as multiple Map task process and Reduce task process executed in parallel respectively.The memory source of operation platform (hereinafter referred to as " Map/Reduce platform ") in units of task needed for its operation of Map/Reduce application distribution of Map/Reduce application.

Because Map/Reduce application generally has the feature of large data processing, whether distribute sufficient memory source, become the key factor of restriction Map/Reduce application execution efficiency.At present, Map/Reduce platform adopts the method being set to guiding with user usually to the distribution of memory source, namely user initiatively initiates memory source application request before Map task and Reduce task run or in running, provide the memory source demand determined, the demand that Map/Reduce platform is specified according to user is distributed for it or adds memory source; Task is once obtain memory source and take continuing, until task run terminates, or the passive release when other operation tasks need additional memory source.

But, said method applies to actual Map/Reduce productivity platform, and there are the following problems: Map task and Reduce task often have significant undulatory property to the use amount of memory source in its operational process, and the actual consumption demand of user to the memory source of task is difficult to accurate assurance.Therefore, adopt in Map/Reduce platform and be set to user the memory allocation method that leads, cause the excessive application memory source of user to be the fact of outwardness.Meanwhile, be set in the method for guiding existing with user, its excessive memory source occupied initiatively cannot discharge and use for Map task to be scheduled and Reduce task by task.This makes to treat that delay start performs scheduler task owing to cannot obtain initial Memory Allocation, thus greatly reduces the task throughput of platform and the utilization factor of memory source.In addition, be difficult to prevent malicious user from excessively applying for memory source with the method that user is set to lead, thus cause the phenomenon of platform resource malice competition.

Summary of the invention

The inventive method has the feature of obvious undulatory property for Map and Reduce task memory source use amount in operational process, according to the internal memory use amount historical record in Map task and Reduce task run process, adopt linear regression and t method of inspection, statistics task internal memory laws of use, the amount of ram used is needed in prediction task follow-up operation, and according to task internal memory usage forecastings amount, dynamically add or reduce the Memory Allocation amount of Map task and the Reduce task run, thus effectively improve the service efficiency of Map/Reduce platform internal memory resource, promote the execution efficiency of Map/Reduce operation.

Memory source distribution method of the present invention is divided into five steps: initialization, task memory source usage forecastings, the release of task memory source, task memory source add and backtracking.In the method, the parameter that five basic is had: anticipation function matching frequency threshold value C _max, task memory source adds judgment threshold U _a, task memory source release judgment threshold U _r, internal memory additional quantity step-length τ computing time, task preemption weights ratio θ.C _maxgeneral value between 3 ~ 5, U _avalue between 0.1 ~ 0.5, U _rvalue is between 0.5 ~ 1, and τ value was at 5 ~ 10 seconds, and θ value is between 0 ~ 1.

Said method realizes on computers according to the following steps:

(1) initialization: gather operation task t from the existing resource of Map/Reduce platform and task management assembly _ij(1≤i≤m, 1≤j≤n) initialization information needed for internal memory dynamic assignment, comprises task current memory sendout RC _ij, task brings into operation moment c_i _ijwith the memory source use amount historical record set RN of task _ij.Wherein, i represents the numbering of Map/Reduce operation belonging to task, and j represents that task is doing mission number in the industry.

(2) task t is set up _ijmemory source use amount anticipation function.

2.1) task t is set _ijmemory source usage forecastings amount be function about the time, anticipation function shape as wherein, a _ijwith c _ijit is parameter to be evaluated;

2.2) operation task t in platform is made _ijmemory source use amount historical record set RU _ijbe expressed as RU _ij={ r _l| r _l=(t _l, m _l), t _l>=t _l-1, m _l>=0}, wherein, t _lbe l record period, m _lfor t _lrecord period task t _ijinternal memory use amount.RU is set _ijfor forecast sample;

2.3) x=ln (t+1) is made, y=q _ij(t), by task t _ijmemory source demand forecast functional transformation be y=b _ijx+c _ij, parameter transformation to be evaluated is b _ijand c _ij; Make C _ijfor the number of times of Function Fitting failure, C _ij← 0;

2.4) adopt linear regression method to function y=b _ijx+c _ijcarry out matching, wherein b _ijand c _ijbe regression coefficient;

2.4.1) for task t _ijhistory memory source use record set RU _ijin each record r _l, calculate the record r after a conversion _l': (x _l, y _l), x _l=ln (t _l+ 1), y _l=m _l;

2.4.2) utilize formula (1)-(6), calculate regression coefficient b _ijand c _ijestimated value with wherein, n _ijrepresent RU _ijmiddle current memory resource uses the number of record.

$\stackrel{\‾}{x}=\frac{1}{n_{\mathrm{ij}}}{\mathrm{\Σ}}_{l=1}^{n_{\mathrm{ij}}}{x}_l---\left(1\right)$

$\stackrel{\‾}{y}=\frac{1}{n_{\mathrm{ij}}}{\mathrm{\Σ}}_{l=1}^{n_{\mathrm{ij}}}{y}_l---\left(2\right)$

$S_{\mathrm{xx}}={\mathrm{\Σ}}_{l=1}^{n_{\mathrm{ij}}}{(x_l-\stackrel{\‾}{x})}^2={\mathrm{\Σ}}_{l=1}^{n_{\mathrm{ij}}}{x}_l^2-\frac{1}{n_{\mathrm{ij}}}{\left({\mathrm{\Σ}}_{l=1}^{n_{\mathrm{ij}}}{x}_l\right)}^2---\left(3\right)$

$S_{\mathrm{xy}}={\mathrm{\Σ}}_{l=1}^{n_{\mathrm{ij}}}(x_l-\stackrel{\‾}{x})(y_l-\stackrel{\‾}{y})={\mathrm{\Σ}}_{l=1}^{n_{\mathrm{ij}}}{x}_l{y}_l-\frac{1}{n_{\mathrm{ij}}}\left({\mathrm{\Σ}}_{l=1}^{n_{\mathrm{ij}}}{x}_l\right)\left({\mathrm{\Σ}}_{l=1}^{n_{\mathrm{ij}}}{y}_l\right)---\left(4\right)$

Draw empirical regression equation thus

2.5) t method of inspection is utilized, to regression equation carry out significance test.Order $S_{\mathrm{yy}}={\mathrm{\Σ}}_{l=1}^{n_{\mathrm{ij}}}{(y_l-\stackrel{\‾}{y})}^2,$ inspection whether meet.Wherein, be degree of freedom be n _ijt distribution function α/2 tantile of-2, α is level of significance;

2.6) if inspection meets, then Function Fitting success, performs step 2.10); Otherwise, perform step 2.7);

2.7) C _ij← C _ij+ 1, if C _ij> C _max, then matching failure, performs step 2.9); Otherwise, perform step 2.8); Wherein, C _maxfor anticipation function matching frequency threshold value;

2.8) forecast sample is revised; To RU _ijin all memory sources use record r _l=(t _l, m _l), r _l∈ RU _ijif meet r _l-1=(t _l-1, m _l-1), r _l-1∈ RU _ij, m _l-m _l-1< 0, then arrange RU _ij← RU _ij-{ r _l, perform step 2.4;

2.9) task t is marked _ijmemory source use amount anticipation function builds unsuccessfully, performs step (3);

2.10) task t is marked _ijmemory source use amount anticipation function successfully constructs, by task t _ijmemory source demand forecast function in optimum configurations to be estimated be

(3) calculation task t _ijmemory source additional quantity RA _ijwith burst size RD _ij.Current time is made to be c_c _ij, task t _ijcurrent memory use amount is RN _ij.

3.1) initialization RA _ij=0, RD _ij=0;

3.2) estimation tasks t _ijcomplete moment c_f _ij.Make task t _ijthe progress of process is p _ij, according to formula (7) estimation tasks t _ijcomplete moment c_f _ij:

c_f _ij＝c_i _ij+(c_c _ij-c_i _ij)/p _ij(7)

3.3) judgement task t _ijthe need of additional memory source; If then perform step 3.7); Otherwise, perform step 3.4); Wherein, U _afor memory source adds judgment threshold.

3.4) according to step (2), task t is judged _ijanticipation function whether successfully construct, if so, then perform step 3.5), otherwise, perform step 3.6);

3.5) RA is calculated according to formula (8) _ij, perform step 3.8);

${\mathrm{RA}}_{\mathrm{ij}}=\left\{\begin{array}{c}{q}_{\mathrm{ij}}(c\_c_{\mathrm{ij}}-{c\_i}_{\mathrm{ij}}+\mathrm{\&tau;})-{\mathrm{RN}}_{\mathrm{ij}},\mathrm{\&tau;}<{c\_f}_{\mathrm{ij}}-{c\_c}_{\mathrm{ij}}\\ q_{\mathrm{ij}}({c\_f}_{\mathrm{ij}}-{c\_i}_{\mathrm{ij}})-{\mathrm{RN}}_{\mathrm{ij}},\mathrm{\&tau;}\&GreaterEqual;{c\_f}_{\mathrm{ij}}-c\_c_{\mathrm{ij}}\end{array}\right.---\left(8\right)$

Wherein, τ is internal memory additional quantity predicted time step-length;

3.6) RA is calculated according to formula (9) _ij, perform step 3.8);

RA _ij＝(RC _ij-RN _ij)×1.5 (9)

3.7) RA is set _ij=0;

3.8) judgement task t _ijthe need of releasing memory resource.If then perform step 3.9); Otherwise, perform step 3.12); Wherein, U _rfor internal memory release judgment threshold.

3.9) according to step (2), task t is judged _ijanticipation function whether successfully construct, if so, then perform step 3.10), otherwise, perform step 3.11);

3.10) RM is made _ij=q _ij(c_f _ij-c_i _ij), calculate RD according to formula (10) _ij, perform step (4);

3.11) RD is calculated according to formula (11) _ij, perform step (4);

${\mathrm{RD}}_{\mathrm{ij}}=\frac{1}{5}{\mathrm{RN}}_{\mathrm{ij}}---\left(11\right)$

3.12) RD is set _ij=0;

(4) task t _ijmemory source discharges.Make t _ijrunning place node server is N _k, perform RC _ij← RC _ij-RD _ij, R _{k_free}← R _{k_free}+ RD _ij, wherein, R _{k_free}for node N _kfree memory stock number.

(5) task t _ijmemory source adds.Make task t _ijrunning place node server is N _k, R _{k_free}for node N _kfree memory stock number.

5.1) according to step (4), judge whether to meet RA _ij≤ R _{k_free}, if so, then perform R _{k_free}← R _{k_free}-RA _ij, and go to step 5.9); Otherwise, perform step 5.2);

5.2) node server N is made _kupper all task composition set TR run, to each the task t in TR _uv, the preempting priority A of this task is calculated according to formula (12) _uv,

A _uv＝θ×p _uv+(1-θ)×f _u，0≤θ≤1 (12)

Wherein, p _uvtask t _uvoperation progress, f _utask t _uvaffiliated operation J _uin number of having finished the work account for the ratio of general assignment number, θ is weighting ratio;

5.3) all preempting priorities are chosen in TR higher than task t _ijtask composition set TR ';

5.4) TP is made to be node server N _kon need the set of tasks seizing its memory source, arrange initial setting up memory source seizes total amount Pr_R _k=0;

5.5) judge whether to meet RA _ij> R _{k_free}+ Pr_R _kand if so, then step 5.6 is performed); Otherwise, perform step 5.7);

5.6) from TR ', choosing task according to task preemption priority order from high to low, might as well be t ' by selected task presentation _uv, task t ' _uvcurrent memory sendout be expressed as RC ' _uv; For task t ' each in TR ' _uv, Pr_R is set _k← Pr_R _k+ RC ' _uv, TR ' ← TR '-t ' _uv, TP ← TP ∪ t ' _uv.Perform step 5.5);

5.7) judge whether to meet RA _ij≤ R _{k_free}+ Pr_R _k, if so, then perform R _{k_free}← R _{k_free}+ Pr_R _k-RA _ij, and perform step 5.8); Otherwise, perform step 5.10);

5.8) operation is seized to each the tasks carrying memory source in set of tasks TP, namely stop the operation process of this task, task is labeled as again and treats scheduler task;

5.9) task t is marked _ijmemory source adds successfully, arranges task t _ijcurrent memory Resourse Distribute amount RC _ij← RC _ij+ RA _ij, go to step (6);

5.10) task t is marked _ijmemory source adds unsuccessfully.

(6) recall: after a memory source dynamic assignment period distances t, judge task t _ijwhether terminate, be, go to step (7), otherwise forward step (1) to; Wherein, memory source dynamic assignment period distances t refers between adjacent two subtask memory source dynamic assignment, terminates to the interval duration second time Memory Allocation from first time Memory Allocation.

(7) terminate: stop task t _ijmemory source code reassignment function.

When there is multiple operation task in platform, in each step of the inventive method, according to described each operation task of method processed in sequence, to complete the Dram Resourse Distribute to all operation tasks.

In order to realize said method, the present invention sets up a task internal memory and uses and run Schedule monitoring device on the server at each task run place of Map/Reduce platform, use information, task run progress msg for periodically obtaining task memory source, the task memory source that watch-dog obtains by the present invention uses task memory source use amount historical record set required in information structure invention step (2); And be RN using the current memory use amount that the task memory usage information obtained for the last time is required in step (3) _ij, using the Task Progress information that obtains for the last time as the current operation progress of this task.The operation progress of task and its memory source use amount information obtain by periodic task running state monitoring existing in multiplexing Map/Reduce platform mechanism.In order to realize the method, the present invention sets up task internal memory usage forecastings device in Map/Reduce platform, for using and run internal memory use amount information matching task memory source usage forecastings function that Schedule monitoring device provides according to task internal memory and calculation task needs to add or the amount of ram (step (2) and step (3)) of release.In order to realize the method, the present invention sets up task memory source dynamic allocator in Map/Reduce platform, for predicting the outcome of providing according to fallout predictor, complete adding or release (step (4) and step (5)) operation task memory source.

Accompanying drawing explanation

The deployment diagram of the Map/Reduce platform that Fig. 1 depends on for the inventive method.

Fig. 2 adopts newly-increased software module and interactive relation figure thereof in the Map/Reduce platform of the inventive method.

Fig. 3 is the overview flow chart of the inventive method.

Fig. 4 is that task memory source use amount anticipation function builds process flow diagram.

Fig. 5 be task memory source add/burst size calculate process flow diagram.

Fig. 6 is task memory source additive proccess process flow diagram.

Embodiment

Below in conjunction with the drawings and specific embodiments, the present invention is illustrated.

Memory source dynamic allocation method proposed by the invention can depend on existing Map/Reduce data processing platform (DPP) (as Hadoop platform), realizes by revising and increase newly corresponding software module.Fig. 1 is the deployment diagram of the Map/Reduce platform that this method depends on.This platform is made up of multiple computer server (platform nodes), is connected between server by network.Platform nodes is divided into two classes: comprise a management node (Master) and multiple computing node (Slave).The Map/Reduce platform that the inventive method depends on comprises four class kernel software modules: resource management module (ResourceManager), application management module (ApplicationMaster), task execution module (TaskContainer) and node administration module (NodeManager).Wherein, ResourceManager is responsible for the memory source information of all nodes in maintenance platform, executive job scheduling and carries out the original allocation of task memory source according to the memory requirements that user submits to, only in management node deploy; NodeManager is responsible for starting and the operation that ends task, and monitors this node and to take up an official post business ruuning situation and resource service condition, each computing node is all disposed a NodeManager.ApplicationMaster is responsible for Map/Reduce operation life cycle management, to each Map and Reduce task record its initial memory source demand information and task status information that operation comprises, and management and supervision is carried out to the startup, operation, end etc. of task.Each the Map/Reduce operation submitted in Map/Reduce platform and run is correspondence one independently ApplicationMaster all, and this module is in computing node deploy.TaskContainer is responsible for performing Map task or Reduce task.Each Map task or Reduce task be correspondence one independently TaskContainer all, and this module is in task run place node deploy.In above-mentioned four class software modules, ResourceManager module and NodeManager module dispose operation when Map/Reduce platform starts, and ApplicationMaster module and TaskContainer module trigger when corresponding Map/Reduce Hand up homework and Map or Reduce task run to dispose respectively and run.

Fig. 2 is in depended on Map/Reduce platform, need increase for implementing the inventive method software module and interactive relation figure thereof.Shaded block is the software module that must increase newly in existing Map/Reduce platform for realizing the inventive method, comprises Mission Monitor module (MemCollector), internal memory usage forecastings module (MemPredictor), internal memory reallocation module (MemReallocator), task status update module (TaskUpdator) and resource updates module (MemUpdator).Wherein, MemCollector is responsible for internal memory service condition and the task run progress msg of all task execution module of collecting its place node, the internal memory use amount that each task memory usage information prediction task that MemPredictor is responsible for providing according to MemCollector module is follow-up, MemReallocator is responsible for according to internal memory usage forecastings information, the Memory Allocation amount of adjustment (comprise and adding and release) each task.The submodule that above-mentioned three modules can be used as NodeManager is deployed on each computing node.TaskUpdator, as the submodule of ApplicationMaster, is deployed in computing node, is responsible for receiving task status adjustment information, and revises the status information of the corresponding task of preserving in ApplicationMaster.MemUpdator, as the submodule of ResourceManager, is deployed in management node, is responsible for collecting task memory source on computing node and distributes transition information, and revise the computing node safeguarded in ResourceManager can storage allocation resource information.In above-mentioned newly-increased module, between the submodule being under the jurisdiction of same software module, adopt the communication mode of shared variable and method call, be under the jurisdiction of the network communication mode that different software mould module cross-thread adopts remote procedure call (RPC).

For realizing the inventive method, a shared variable is set between MemCollector, MemPredictor and MemReallocator module of each computing node--operation task information list RunTasklist, the corresponding operation task of each list cell.For arbitrary operation task t _ij, the storage information format of list cell is as follows:

Wherein, task memory source uses each record in historical record set to comprise record moment and internal memory use amount information.For realizing the inventive method, MemCollector on each computing node needs memory source use amount and the operation progress msg of periodically collecting operation task on this node, and the information of collection is added in shared variable RunTasklist respective list unit, obtain for internal memory usage forecastings module MemPredictor.Specific implementation method comprises:

1) task registration task t _ijafter corresponding TaskContainer starts, the mode called with RPC, the MemCollector to place node carries out task registration.The task registration information that MemCollector sends according to TaskContainer, comprise the IP address of TaskUpdator corresponding to task number, job number, task process number, task, the list cell that establishment one is new in shared variable RunTasklist, and fill in corresponding information.

2) after task internal memory use amount and operation progress msg are collected in registration of finishing the work, the callback interface of MemCollector by embedding in NodeManager module, according to task number j and job number i, the internal memory use amount information of task process periodically (every 5 seconds) is obtained from NodeManager, then by the task internal memory use amount information of acquisition and record the moment accordingly, the task memory source use amount historical record set RU of RunTasklist respective list unit is appended to by record moment order _ijxiang Zhong, and with the task current memory sendout RC of task current memory sendout information updating RunTasklist respective list unit obtained _ijentry value.Meanwhile, MemCollector, periodically in the mode that RPC calls, obtains task run progress msg from TaskContainer, and the current operation progress p of the task of the value of acquisition being assigned to respective list unit in RunTasklist _ij.

3) task is nullified after task end of run, the mode that TaskContainer calls with RPC, and the MemCollector to place node sends task logout message, and task logout message comprises job number, task number information.MemCollector, according to job number and task number information, stops this duty cycle internal memory use amount and the collection operation running progress msg, and deletes the corresponding list cell of this task in RunTasklist.

The specific implementation method of the inventive method is described below in conjunction with Fig. 3 summary of the invention main-process stream.In this implementation method, arranging of five basic parameters is as follows: anticipation function matching frequency threshold value C _max=5, task memory source adds judgment threshold U _a=0.1, task memory source release judgment threshold U _r=0.5, internal memory additional quantity step-length τ computing time is 10 seconds, task preemption weights ratio θ=0.8.This implementation method can be divided into following steps:

(1) initialization task t _ijthe MemCollector run on the computing node of place reads task t in shared variable RunTasklist _ijinformation, according to task number j and job number i, by the callback interface embedded in NodeManager module, the current memory sendout information of task is obtained from NodeManager, and the TaskUpdator IP address corresponding according to the task in mission bit stream, in the mode that RPC calls, obtain task t from corresponding TaskUpdator _ijbring into operation the moment, above-mentioned value is assigned to task t in RunTasklist respectively _ijtask current memory sendout RC in corresponding lists unit _ijand task to bring into operation moment c_i _ij.Task memory source use amount historical record set RU _ijoperation progress p current with task _ij, then as previously mentioned, by memory source use amount and the acquisition of operation progress msg of MemCollector periodic harvest operation task.

(2) task t is set up _ijmemory source use amount anticipation function.

2.1) task t is set _ijmemory source usage forecastings amount be function about the time, anticipation function shape as wherein, a _ijwith c _ijit is parameter to be evaluated;

2.2) task t _ijthe MemPredictor run on the computing node of place reads task t in shared variable RunTasklist _ijmemory source use amount historical record set RU _ij, RU _ij={ r _l| r _l: (t _l, m _l), t _o=0, t _l>=t _l-1, 0≤l≤n _ij, wherein, t _lbe the 1st record moment, m _lfor t _lrecord moment task t _ijinternal memory use amount.RU is set _ijfor forecast sample;

2.3) x=ln (t+1) is made, y=q _ij(t), by task t _ijmemory source demand forecast functional transformation be y=b _ijx+c _ij, parameter transformation to be evaluated is b _ijand c _ij; MemPredictor arranges the number of times C of Function Fitting failure _ij=0;

2.4) MemPredictor is according to summary of the invention 2.4) in method, adopt linear regression method to carry out converting the matching of anticipation function;

2.5) MemPredictor is according to summary of the invention 2.5) described in method, adopt t method of inspection to carry out significance test to fitting function;

2.6) MemPredictor judges whether inspection meets, and if so, then Function Fitting success, performs step 2.10); Otherwise, perform step 2.7);

2.7) MemPredictor is by C _ijadd 1, judge whether matching total degree is greater than total degree threshold value C _max, then matching failure, performs step 2.9); Otherwise, perform step 2.10).Wherein, C _maxfor anticipation function matching frequency threshold value;

2.8) forecast sample is revised.MemPredictor reads task t in shared variable RunTasklist _ijmemory source use amount historical information set RU _ij, from RU _ijthe all r that satisfies condition of middle removal _l-1: (t _l-1, m _l-1), r _l-1∈ RU _ij, m _l-m _l-1the memory source use amount record r of < 0 _l: (t _l, m _l), and perform step 2.4);

2.9) MemPredictor marks task t _ijmemory source use amount anticipation function builds unsuccessfully, performs step (3);

2.10) MemPredictor marks task t _ijmemory source use amount anticipation function successfully constructs, by task t _ijmemory source demand forecast function in optimum configurations to be estimated be

(3) calculation task t _ijmemory source additional quantity RA _ijwith burst size RD _ij.Current time is made to be c_c _ij, task t _ijcurrent memory use amount is RN _ij.

3.1) MemPredictor arranges RA _ij=0, RD _ij=0;

3.2) MemPredictor reads task t from shared variable RunTasklist _ijthe task c_i initial time of running _ijwith task t _ijcurrent operation progress p _ij, and according to summary of the invention 3.2) described method estimation tasks t _jjcomplete moment c_f _ij;

3.3) MemPredictor reads task t from shared variable RunTasklist _ijcurrent memory Resourse Distribute amount RC _ij, and obtain t according to the priority in record moment _ijthe last memory source uses record, as task t _ijcurrent memory use amount RN _ij.MemPredictor judges whether meet, if so, then perform step 3.7); Otherwise, perform step 3.4).

3.4) MemPredictor is according to step (2), judges task t _ijanticipation function whether successfully construct, if so, then perform step 3.5), otherwise, perform step 3.6);

3.5) MemPredictor calculates RA according to formula (8) _ij, perform step 3.8);

${\mathrm{RA}}_{\mathrm{ij}}=\left\{\begin{array}{c}{q}_{\mathrm{ij}}(c\_c_{\mathrm{ij}}-{c\_i}_{\mathrm{ij}}+\mathrm{\&tau;})-{\mathrm{RN}}_{\mathrm{ij}},\mathrm{\&tau;}<{c\_f}_{\mathrm{ij}}-{c\_c}_{\mathrm{ij}}\\ q_{\mathrm{ij}}({c\_f}_{\mathrm{ij}}-{c\_i}_{\mathrm{ij}})-{\mathrm{RN}}_{\mathrm{ij}},\mathrm{\&tau;}\&GreaterEqual;{c\_f}_{\mathrm{ij}}-c\_c_{\mathrm{ij}}\end{array}\right.---\left(8\right)$

3.6) RA is calculated according to formula (9) _ij, perform step 3.8);

RA _ij＝(RC _ij-RN _ij)×1.5 (9)

3.7) MemPredictor arranges RA _ij=0;

3.8) MemPredictor judges whether meet, if so, then perform step 3.9); Otherwise, perform step 3.12).

3.9) MemPredictor is according to step (2), judges task t _jjanticipation function whether successfully construct, if so, then perform step 3.10), otherwise, perform step 3.11);

3.10) RM is made _ij=q _ij(c_f _ij-c_i _ij), MemPredictor calculates RD according to formula (10) _ij, perform step (4);

3.11) MemPredictor calculates RD according to formula (11) _ij, perform step (4);

${\mathrm{RD}}_{\mathrm{ij}}=\frac{1}{5}{\mathrm{RN}}_{\mathrm{ij}}---\left(11\right)$

3.12) MemPredictor arranges RD _ij=0;

(4) task t _ijmemory source discharges, and makes t _ijrunning place node server is N _k, according to step (3) execution result, to task t _ijperform following operation:

4.1) MemPredictor calls the task internal memory release nation method that the MemReallocator that is in same computing node provides, by the task number of this task, job number and internal memory burst size RD _ijpass to MemReallocator;

4.2) MemReallocator is according to the internal memory burst size information of task, the mode called with RPC, by place computing node N _kwith releasing memory amount RD _ijsend to MemUpdator;

4.3) MemUpdator is according to node number and internal memory burst size information, the node N safeguarded in amendment ResourceManager primary module _kfree memory stock number R _{k_free}, R _{k_free}← R _{k_free}+ RD _ij, then return internal memory to MemReallocator and discharge successfully/failure information;

4.4) MemReallocator is according to return message, if success, then revises task t in shared variable RunTasklist _ijcurrent memory Resourse Distribute amount RC _ij, perform RC _ij← RC _ij-RD _ij.

(5) task memory source adds.According to step (2) execution result, to task t _ijperform following operation:

5.1) the task internal memory that MemPredictor calls the MemReallocator that is in same computing node and provides adds nation method, by the task number of this task, job number and internal memory additional quantity RA _ijpass to MemReallocator;

5.2) MemReallocator is according to the internal memory additional quantity information of task, the mode called with RPC, by place computing node N _kwith additional amount of ram RA _ijsend to MemUpdator;

5.3) MemUpdator is according to internal memory supplement information, reads the node N safeguarded in ResourceManager primary module _kfree memory stock number R _{k_free}if, R _{k_free}>=RA _ij, then MemUpdator performs R _{k_free}← R _{k_free}-RA _ij, and return internal memory to MemReallocator and add success message, and perform step 5.9); Otherwise, return failed message to MemReallocator, and perform step 5.4);

5.4) MemReallocator is according to return messages, if add unsuccessfully, MemReallocator reads each operation task t in shared variable RunTasklist _uvthe current operation progress p of task _uv, and in the mode that RPC calls, account for the ratio f of general assignment number from number of already having finished the work belonging to the TaskUpdator acquisition task that this task is corresponding _u, MemReallocator is according to content 5.2 of the present invention)-5.6) described in method, calculate the preempting priority of operation task, choose the set of tasks TP needing to seize its memory source;

5.5) MemReallocator obtains all task current memory Resourse Distribute amounts in TP from shared variable RunTasklist, and summation is as node N _kpreemptible memory source amount Pr_R _k; The mode that MemReallocator calls with RPC, by task t _ijplace computing node N _k, add amount of ram RA _ijand node N _kpreemptible memory source amount Pr_R _kinformation sends to MemUpdator;

5.6) MemUpdator seizes supplement information according to internal memory, reads the node N safeguarded in ResourceManager primary module _kfree memory stock number R _{k_free}, judge R _{k_free}+ Pr_R _k>=RA _ijwhether meet, if so, then perform step 5.7), otherwise, return internal memory to MemReallocator and seize additional failed message, and perform step 5.10);

5.7) MemUpdator performs R _{k_free}← R _{k_free}+ Pr_R _k-RA _ij, and return internal memory to MemReallocator and seize additional success message;

5.8) MemReallocator seizes additional return messages according to internal memory, if add successfully, then call the method for its place NodeManager, stop the TaskContainer that in TP, all tasks are corresponding and run process, the record that operation task is corresponding is deleted in shared variable RunTasklist, and the task status that the corresponding TaskUpdator of calling task provides changes RPC method, task status is re-set as and waits to dispatch by notice TaskUpdator;

5.9) MemReallocator marks task t _ijmemory source adds successfully, task t in amendment shared variable RunTasklist _ijcurrent memory Resourse Distribute amount RC _ij, perform RC _ij← RC _ij+ RA _ij, go to step (6);

5.10) MemReallocator seizes additional return messages according to internal memory, if add unsuccessfully, emReallocator marks task t _ijmemory source adds unsuccessfully.

(6) recall.After a memory source dynamic assignment cycle (being set to 10 seconds in this programme) is terminated, task t _ijthe MemColloector run on the computing node of place searches in shared variable RunTasklist whether there is task t _ijcorresponding list cell, if exist, then performs step (1), if do not exist, then performs step (7).

(7) terminate: stop task t _ijmemory source code reassignment function.

When there is multiple operation task in platform, can in each step of above-mentioned implementation method, according to described each operation task of method processed in sequence, to complete the Dram Resourse Distribute to all operation tasks.

According to Map/Reduce platform internal memory resource optimal distribution method proposed by the invention, inventor has done relevant performance test.Test result shows, the inventive method is applicable to typical Map/Reduce application load.Adopt the Map/Reduce type mass data processing platform more existing main flow Map/Reduce type mass data processing platform of the inventive method, as Hadoop, the utilization factor of Job execution efficiency and platform internal memory resource can be promoted preferably.

The Map/Reduce type mass data processing platform Predra realized according to specific embodiments of the present invention and current the most widely used Map/Reduce type mass data processing platform Hadoop is carried out Performance comparision by performance test.Operation throughput is chosen in test, operation average turnaround time and the average memory source utilization factor of task are as performance index, and the method proposed to embody the present invention is promoting Map/Reduce Job execution efficiency and optimizing the advantage in Map/Reduce platform internal memory resource utilization.Wherein, throughput refers to the quantity fulfiled assignment in the unit interval, by operation number/hour in units of; The operation average turnaround time refers to that operation has run required averaging time from being submitted to, in seconds; The operation average latency refers to that operation is from being submitted to the required averaging time that brings into operation, in seconds; The average memory source utilization factor of task refer to single task in platform the actual amount of ram used account for the ratio average of the Memory Allocation total amount that it obtains.Performance test runs on the group system be made up of 24 computing nodes, the hardware configuration of computing node comprises: CPU, 16GB DDR3RAM, 1TB SATA hard disc of 4 Intel (R) Xeon (R) CPU E5-2660 0@2.20GHz, adopt gigabit Ethernet to interconnect between node, operating system is Centos6.5.The Prof. Du Yucang hybrid test load of simulation actual production environment is selected in test.Mixed load simulates the situation in actual production environment in four: the map task quantity that input data type and scale, homework type and mixing quantity, operation comprise and reduce task quantity, operation arrive distribution.Comprise 6 kinds of homework types in mixed load, operation arrival interval meets Poisson distribution, and arriving intensity is on average within every 10 seconds, submit an operation to.The concrete configuration of mixed load is as table 1:

Table 1 test mixing load configuration

for the test of different task memory source demand

First test arranges different memory demands for Map task and Reduce task, analyzes the performance of Pradra under different task memory requirements.In the test of this group, the average Reduce task quantity that dissimilar operation comprises is arranged as table 2.

The average Reduce number of tasks that table 2 operation comprises is arranged

In the test of this group, the memory source initial demand amount of map and reduce task is arranged as table 2.

Table 3 task memory source initial demand amount is arranged

Operation throughput test result under table 4 different task internal memory initial demand amount

The average turnaround time test result of operation under table 5 different task internal memory initial demand amount

The average memory usage test result of task under table 6 different task internal memory initial demand amount

Table 4, table 5 and table 6 are given in different task memory source demands respectively and arrange down, the test result of operation throughput in Predra and Hadoop platform, operation average turnaround time and the average memory usage of task.From test result, adopt the Predrap platform comparatively Hadoop platform operation throughput maximum lift 47.5% of the inventive method, on average improve 29%; The operation average turnaround time is maximum shortens 57.1%, shorten in average 37.4%; Task average memory usage maximum lift 170.9%, on average improves 141.2%.Because Map/Reduce platform is the original allocation that the task memory source demand proposed according to user carries out task memory source, therefore the initial Memory Allocation amount of task is memory demand.In Hadoop platform, task, once obtain initial Memory Allocation, just occupies distributed internal memory until task run terminates.Compare with Hadoop platform, the advantage of Predra platform is can for the feature of the actual use amount fluctuation of memory source in task run process, by the actual demand amount of internal memory in the means real-time assessment task run process of prediction, and carry out the dynamic assignment of memory source, thus task to be occupied but untapped memory source discharges and distributes to other and treats scheduler task, shorten and treat that scheduler task waits for the time of memory source original allocation, therefore the index characterizing Job execution efficiency at operation average turnaround time and operation throughput these two can obtain better performance.Meanwhile, Predra platform takies but the amount of ram used without reality by adopting the memory source dynamic allocation method based on prediction can reduce task, therefore improves the utilization factor of memory source.

the test of Reduce task quantity is comprised for different operations

This partial test is that Map/Reduce operation arranges different Reduce number of tasks, analyzes the performance of Pradra under the different scene of the task scale that comprises in operation.In the test of this group, the memory source initial demand amount of map and reduce task is respectively 1024MB and 2048MB, and the Reduce number of tasks that operation comprises is arranged as table 7.

The Reduce number of tasks that table 7Map/Reduce operation comprises is arranged

The different single job of table 8 comprises Reduce number of tasks and arranges lower operation throughput test result

The different single job of table 9 comprises Reduce number of tasks and arranges the average turnaround time test result of lower operation

The different single job of table 10 comprises Reduce number of tasks and arranges the average memory usage test result of lower task

Table 8, table 9 and table 10 are given in different single jobs respectively and comprise Reduce number of tasks and arrange down, the test result of operation throughput in Predra and Hadoop platform, operation average turnaround time and the average memory usage of task.From test result, adopt the Predrap platform comparatively Hadoop platform operation throughput maximum lift 48.4% of the inventive method, on average improve 35.9%; The operation average turnaround time is maximum shortens and maximumly shortens 66.9%, shorten in average 42.9%., shorten in average 37.4%; Task average memory usage maximum lift 196.8%, on average improves 165.7%.In Map/Reduce platform, the file size of map task quantity usually handled by operation that single job comprises is determined, sets without the need to user.Therefore, the Reduce task quantity that variation operation comprises can simulate the task scale difference because operation comprises, and causes the scene that platform internal memory resource contention intensity is different.Experimental result shows, under different operations comprises the setting of Reduce number of tasks, Pradra all obtains the performance being better than Hadoop, its reason remains and dynamically obtains memory source according to the actual requirements because Pradra supports that task is in operation, and the unactual memory share used can be discharged, obtain initial Memory Allocation quickly and startup optimization for other tasks.

Last it is noted that above example only in order to illustrate the present invention and and unrestricted technology described in the invention, and all do not depart from technical scheme and the improvement thereof of the spirit and scope of invention, and it all should be encompassed in the middle of right of the present invention.

Claims (7)

1. based on a Map/Reduce data processing platform (DPP) memory source dynamic allocation method for prediction, memory source distribution method is divided into five steps: initialization, task memory source usage forecastings, the release of task memory source, task memory source add and backtracking; In the method, five basic parameters are had: anticipation function matching frequency threshold value C _max, task memory source adds judgment threshold U _a, task memory source release judgment threshold U _r, internal memory additional quantity step-length τ computing time, task preemption weights ratio θ; C _maxgeneral value between 3 ~ 5, U _avalue between 0.1 ~ 0.5, U _rvalue is between 0.5 ~ 1, and τ value was at 5 ~ 10 seconds, and θ value is between 0 ~ 1;

It is characterized in that: described method realizes on computers according to the following steps,

(1) initialization: gather operation task t from the existing resource of Map/Reduce platform and task management assembly _ij(1≤i≤m, 1≤j≤n) initialization information needed for internal memory dynamic assignment, comprises task current memory sendout RC _ij, task brings into operation moment c_i _ijwith the memory source use amount historical record set RN of task _ij; Wherein, i represents Map/Reduce job number belonging to task, and j represents that task is doing mission number in the industry;

(2) task t is set up _ijmemory source use amount anticipation function;

2.1) task t is set _ijmemory source usage forecastings amount be function about the time, anticipation function shape as wherein, a _ijwith c _ijit is parameter to be evaluated;

2.2) operation task t in platform is made _ijmemory source use amount historical record set RU _ijbe expressed as RU _ij={ r _l| r _l=(t _l, m _l), t _l>=t _l-1, m _l>=0}, wherein, t _lbe l record period, m _lfor t _lrecord period task t _ijinternal memory use amount; RU is set _ijfor forecast sample;

2.3) x=ln (t+1) is made, y=q _ij(t), by task t _ijmemory source demand forecast functional transformation be y=b _ijx+c _ij, parameter transformation to be evaluated is b _ijand c _ij; Make C _ijfor the number of times of Function Fitting failure, C _ij← 0;

2.4) adopt linear regression method to function y=b _ijx+c _ijcarry out matching, wherein b _ijand c _ijbe regression coefficient;

2.4.1) for task t _ijhistory memory source use record set RU _ijin each record r _l, calculate the record r after a conversion _l': (x _l, y _l), x _l=ln (t _l+ 1), y _l=m _l;

2.4.2) utilize formula (1)-(6), calculate regression coefficient b _ijand c _ijestimated value with wherein, n _ijrepresent RU _ijmiddle current memory resource uses the number of record;

$\stackrel{\&OverBar;}{x}=\frac{1}{n_{\mathrm{ij}}}{\mathrm{\&Sigma;}}_{l=1}^{n_{\mathrm{ij}}}{x}_l---\left(1\right)$

$\stackrel{\&OverBar;}{y}=\frac{1}{n_{\mathrm{ij}}}{\mathrm{\&Sigma;}}_{l=1}^{n_{\mathrm{ij}}}{y}_l---\left(2\right)$

$S_{\mathrm{xx}}={\mathrm{\&Sigma;}}_{l=1}^{n_{\mathrm{ij}}}{(x_l-\stackrel{\&OverBar;}{x})}^2={\mathrm{\&Sigma;}}_{l=1}^{n_{\mathrm{ij}}}{x}_l^2-\frac{1}{n_{\mathrm{ij}}}{\left({\mathrm{\&Sigma;}}_{l=1}^{n_{\mathrm{ij}}}{x}_l\right)}^2---\left(3\right)$

$S_{\mathrm{xy}}={\mathrm{\&Sigma;}}_{l=1}^{n_{\mathrm{ij}}}(x_l-\stackrel{\&OverBar;}{x})(y_l-\stackrel{\&OverBar;}{y})={\mathrm{\&Sigma;}}_{l=1}^{n_{\mathrm{ij}}}{x}_l{y}_l-\frac{1}{n_{\mathrm{ij}}}\left({\mathrm{\&Sigma;}}_{l=1}^{n_{\mathrm{ij}}}{x}_l\right)\left({\mathrm{\&Sigma;}}_{l=1}^{n_{\mathrm{ij}}}{y}_l\right)---\left(4\right)$

Draw empirical regression equation thus

2.5) t method of inspection is utilized, to regression equation carry out significance test; Order $S_{\mathrm{yy}}={\mathrm{\&Sigma;}}_{l=1}^{n_{\mathrm{ij}}}{(y_l-\stackrel{\&OverBar;}{y})}^2,$ inspection whether meet; Wherein, be degree of freedom be n _ijt distribution function α/2 tantile of-2, α is level of significance;

2.6) if inspection meets, then Function Fitting success, performs step 2.10); Otherwise, perform step 2.7);

2.7) C _ij← C _ij+ 1, if C _ij> C _max, then matching failure, performs step 2.9); Otherwise, perform step 2.8); Wherein, C _maxfor anticipation function matching frequency threshold value;

2.8) forecast sample is revised; To RU _ijin all memory sources use record r _l=(t _l, m _l), r _l∈ RU _ijif meet r _l-1=(t _l-1, m _l-1), r _l-1∈ RU _ij, m _l-m _l-1< 0, then arrange RU _ij← RU _ij-{ r _l, perform step 2.4;

2.9) task t is marked _ijmemory source use amount anticipation function builds unsuccessfully, performs step (3);

2.10) task t is marked _ijmemory source use amount anticipation function successfully constructs, by task t _ijmemory source demand forecast function in optimum configurations to be estimated be

(3) calculation task t _ijmemory source additional quantity RA _ijwith burst size RD _ij; Current time is made to be c_c _ij, task t _ijcurrent memory use amount is RN _ij;

3.1) initialization RA _ij=0, RD _ij=0;

3.2) estimation tasks t _ijcomplete moment c_f _ij; Make task t _ijthe progress of process is p _ij, according to formula (7) estimation tasks t _ijcomplete moment c_f _ij:

c_f _ij＝c_i _ij+(c_c _ij-C_i _ij)/p _ij(7)

3.3) judgement task t _ijthe need of additional memory source; If then perform step 3.7); Otherwise, perform step 3.4); Wherein, U _afor memory source adds judgment threshold;

3.4) according to step (2), task t is judged _ijanticipation function whether successfully construct, if so, then perform step 3.5), otherwise, perform step 3.6);

3.5) RA is calculated according to formula (8) _ij, perform step 3.8);

${\mathrm{RA}}_{\mathrm{ij}}=\left\{\begin{array}{c}{q}_{\mathrm{ij}}({c\_c}_{\mathrm{ij}}-{c\_i}_{\mathrm{ij}}+\mathrm{\&tau;})-{\mathrm{RN}}_{\mathrm{ij}},\mathrm{\&tau;}<{c\_f}_{\mathrm{ij}}-{c\_c}_{\mathrm{ij}}\\ q_{\mathrm{ij}}({c\_f}_{\mathrm{ij}}-{c\_i}_{\mathrm{ij}})-{\mathrm{RN}}_{\mathrm{ij}},\mathrm{\&tau;}\&GreaterEqual;{c\_f}_{\mathrm{ij}}-{c\_c}_{\mathrm{ij}}\end{array}\right.---\left(8\right)$

Wherein, τ is internal memory additional quantity predicted time step-length;

3.6) RA is calculated according to formula (9) _ij, perform step 3.8);

RA _ij＝(RC _ij-RN _ij)×1.5 (9)

3.7) RA is set _ij=0;

3.8) judgement task t _ijthe need of releasing memory resource; If then perform step 3.9); Otherwise, perform step 3.12); Wherein, U _rfor internal memory release judgment threshold;

3.9) according to step (2), task t is judged _ijanticipation function whether successfully construct, if so, then perform step 3.10), otherwise, perform step 3.11);

3.10) RM is made _ij=q _ij(c_f _ij-c_i _ij), calculate RD according to formula (10) _ij, perform step (4);

3.11) RD is calculated according to formula (11) _ij, perform step (4);

${\mathrm{RD}}_{\mathrm{ij}}=\frac{1}{5}{\mathrm{RN}}_{\mathrm{ij}}---\left(11\right)$

3.12) RD is set _ij=0;

(4) task t _ijmemory source discharges; Make t _ijrunning place node server is N _k, perform RC _ij← RC _ij-RD _ij, R _{k_free}← R _{k_free}+ RD _ij, wherein, R _{k_free}for node N _kfree memory stock number;

(5) task t _ijmemory source adds; Make task t _ijrunning place node server is N _k, R _{k_free}for node N _kfree memory stock number;

5.1) according to step (4), judge whether to meet RA _ij≤ R _{k_free}, if so, then perform R _{k_free}← R _{k_free}-RA _ij, and go to step 5.9); Otherwise, perform step 5.2);

5.2) node server N is made _kupper all task composition set TR run, to each the task t in TR _uv, the preempting priority A of this task is calculated according to formula (12) _uv,

A _uv＝θ×p _uv+(1-θ)×f _u，0≤θ≤1 (12)

Wherein, p _uvtask t _uvoperation progress, f _utask t _uvaffiliated operation J _uin number of having finished the work account for the ratio of general assignment number, θ is weighting ratio;

5.3) all preempting priorities are chosen in TR higher than task t _ijtask composition set TR ';

5.4) TP is made to be node server N _kon need the set of tasks seizing its memory source, arrange initial setting up memory source seizes total amount Pr-R _k=0;

5.5) judge whether to meet RA _ij> R _{k_free}+ Pr_R _kand if so, then step 5.6 is performed); Otherwise, perform step 5.7);

5.6) from TR ', choosing task according to task preemption priority order from high to low, might as well be t ' by selected task presentation _uv, task t ' _uvcurrent memory sendout be expressed as RC ' _uv; For task t ' each in TR ' _uv, Pr_R is set _k← Pr_R _k+ RC ' _uv, TR ' ← TR '-t ' _uv, TP ← TP ∪ t ' _uv; Perform step 5.5);

5.7) judge whether to meet RA _ij≤ R _{k_free}+ Pr_R _k, if so, then perform R _{k_free}← R _{k_free}+ Pr_R _k-RA _ij, and perform step 5.8); Otherwise, perform step 5.10);

5.8) operation is seized to each the tasks carrying memory source in set of tasks TP, namely stop the operation process of this task, task is labeled as again and treats scheduler task;

5.9) task t is marked _ijmemory source adds successfully, arranges task t _ijcurrent memory Resourse Distribute amount RC _ij← RC _ij+ RA _ij, go to step (6);

5.10) task t is marked _ijmemory source adds unsuccessfully;

(6) recall: after a memory source dynamic assignment period distances t, judge task t _ijwhether terminate, be, go to step (7), otherwise forward step (1) to; Wherein, memory source dynamic assignment period distances t refers between adjacent two subtask memory source dynamic assignment, terminates to the interval duration second time Memory Allocation from first time Memory Allocation;

(7) terminate: stop task t _ijmemory source code reassignment function.

2. a kind of Map/Reduce data processing platform (DPP) memory source dynamic allocation method based on prediction according to claim 1, it is characterized in that: when there is multiple operation task in platform, in each step of this method, according to described each operation task of method processed in sequence, to complete the Dram Resourse Distribute to all operation tasks.

3. a kind of Map/Reduce data processing platform (DPP) memory source dynamic allocation method based on prediction according to claim 1, it is characterized in that: this method is set up a task internal memory and used and run Schedule monitoring device on the server at each task run place of Map/Reduce platform, use information, task run progress msg for periodically obtaining task memory source, the task memory source that watch-dog obtains by this method uses task memory source use amount historical record set required in information structure step (2); And be RN using the current memory use amount that the task memory usage information obtained for the last time is required in step (3) _ij, using the Task Progress information that obtains for the last time as the current operation progress of this task; The operation progress of task and its memory source use amount information are obtained by periodic task running state monitoring existing in multiplexing Map/Reduce platform mechanism; In order to realize the method, task internal memory usage forecastings device is set up, for using and run internal memory use amount information matching task memory source usage forecastings function that Schedule monitoring device provides according to task internal memory and calculation task needs to add or the amount of ram of release in Map/Reduce platform; In order to realize the method, in Map/Reduce platform, setting up task memory source dynamic allocator, for predicting the outcome of providing according to fallout predictor, completing adding or release operation task memory source.

4. a kind of Map/Reduce data processing platform (DPP) memory source dynamic allocation method based on prediction according to claim 1, it is characterized in that: the memory source dynamic allocation method proposed depends on existing Map/Reduce data processing platform (DPP), realize by revising and increase newly corresponding software module; This platform is made up of multiple computer server or platform nodes, is connected between server by network; Platform nodes is divided into two classes, comprises a management node and multiple computing node; The Map/Reduce platform that this method depends on comprises four class kernel software modules: resource management module and ResourceManager, application management module and ApplicationMaster, task execution module and TaskContainer and node administration module and NodeManager; Wherein, ResourceManager is responsible for the memory source information of all nodes in maintenance platform, executive job scheduling and carries out the original allocation of task memory source according to the memory requirements that user submits to, only in management node deploy; NodeManager is responsible for starting and the operation that ends task, and monitors this node and to take up an official post business ruuning situation and resource service condition, each computing node is all disposed a NodeManager; ApplicationMaster is responsible for Map/Reduce operation life cycle management, to each Map and Reduce task record its initial memory source demand information and task status information that operation comprises, and management and supervision is carried out to the startup, operation, end etc. of task; Each the Map/Reduce operation submitted in Map/Reduce platform and run is correspondence one independently ApplicationMaster all, and this module is in computing node deploy; TaskContainer is responsible for performing Map task or Reduce task; Each Map task or Reduce task be correspondence one independently TaskContainer all, and this module is in task run place node deploy; In above-mentioned four class software modules, ResourceManager module and NodeManager module dispose operation when Map/Reduce platform starts, and ApplicationMaster module and TaskContainer module trigger when corresponding Map/Reduce Hand up homework and Map or Reduce task run to dispose respectively and run;

Software module must be increased newly in existing Map/Reduce platform for realizing this method, comprising Mission Monitor module and MemCollector, internal memory usage forecastings module and MemPredictor, internal memory reallocation module and MemReallocator, task status update module and TaskUpdator and resource updates module and MemUpdator; Wherein, MemCollector is responsible for internal memory service condition and the task run progress msg of all task execution module of collecting its place node, the internal memory use amount that each task memory usage information prediction task that MemPredictor is responsible for providing according to MemCollector module is follow-up, MemReallocator is responsible for, according to internal memory usage forecastings information, adjusting the Memory Allocation amount of each task; The submodule that above-mentioned three modules can be used as NodeManager is deployed on each computing node; TaskUpdator, as the submodule of ApplicationMaster, is deployed in computing node, is responsible for receiving task status adjustment information, and revises the status information of the corresponding task of preserving in ApplicationMaster; MemUpdator, as the submodule of ResourceManager, is deployed in management node, is responsible for collecting task memory source on computing node and distributes transition information, and revise the computing node safeguarded in ResourceManager can storage allocation resource information; In above-mentioned newly-increased module, between the submodule being under the jurisdiction of same software module, adopt the communication mode of shared variable and method call, be under the jurisdiction of the network communication mode that different software mould module cross-thread adopts remote procedure call (RPC).

5. a kind of Map/Reduce data processing platform (DPP) memory source dynamic allocation method based on prediction according to claim 4, it is characterized in that: for realizing this method, a shared variable is set between MemCollector, MemPredictor and MemReallocator module of each computing node--operation task information list RunTasklist, the corresponding operation task of each list cell; For arbitrary operation task t _ij, the storage information format of list cell is as follows:

Wherein, task memory source uses each record in historical record set to comprise record moment and internal memory use amount information.

6. a kind of Map/Reduce data processing platform (DPP) memory source dynamic allocation method based on prediction according to claim 4, it is characterized in that: for realizing this method, MemCollector on each computing node needs memory source use amount and the operation progress msg of periodically collecting operation task on this node, and the information of collection is added in shared variable RunTasklist respective list unit, obtain for internal memory usage forecastings module MemPredictor; Specific implementation method comprises,

1) task registration task t _ijafter corresponding TaskContainer starts, the mode called with RPC, the MemCollector to place node carries out task registration; The task registration information that MemCollector sends according to TaskContainer, comprise the IP address of TaskUpdator corresponding to task number, job number, task process number, task, the list cell that establishment one is new in shared variable RunTasklist, and fill in corresponding information;

2) after task internal memory use amount and operation progress msg are collected in registration of finishing the work, the callback interface of MemCollector by embedding in NodeManager module, according to task number j and job number i, the internal memory use amount information of task process is periodically obtained from NodeManager, then by the task internal memory use amount information of acquisition and record the moment accordingly, the task memory source use amount historical record set RU of RunTasklist respective list unit is appended to by record moment order _ijxiang Zhong, and with the task current memory sendout RC of task current memory sendout information updating RunTasklist respective list unit obtained _ijentry value; Meanwhile, MemCollector, periodically in the mode that RPC calls, obtains task run progress msg from TaskContainer, and the current operation progress p of the task of the value of acquisition being assigned to respective list unit in RunTasklist _ij;

3) task is nullified after task end of run, the mode that TaskContainer calls with RPC, and the MemCollector to place node sends task logout message, and task logout message comprises job number, task number information; MemCollector, according to job number and task number information, stops this duty cycle internal memory use amount and the collection operation running progress msg, and deletes the corresponding list cell of this task in RunTasklist.

7. a kind of Map/Reduce data processing platform (DPP) memory source dynamic allocation method based on prediction according to claim 1, it is characterized in that: in this implementation method, arranging of five basic parameters is as follows: anticipation function matching frequency threshold value C _max=5, task memory source adds judgment threshold U _a=0.1, task memory source release judgment threshold U _r=0.5, internal memory additional quantity step-length τ computing time is 10 seconds, task preemption weights ratio θ=0.8; This implementation method can be divided into following steps:

(1) initialization task t _ijthe MemCollector run on the computing node of place reads task t in shared variable RunTasklist _ijinformation, according to task number j and job number i, by the callback interface embedded in NodeManager module, the current memory sendout information of task is obtained from NodeManager, and the TaskUpdator IP address corresponding according to the task in mission bit stream, in the mode that RPC calls, obtain task t from corresponding TaskUpdator _ijbring into operation the moment, above-mentioned value is assigned to task t in RunTasklist respectively _ijtask current memory sendout RC in corresponding lists unit _ijand task to bring into operation moment c_i _ij; Task memory source use amount historical record set RU _ijoperation progress p current with task _ij, then as previously mentioned, by memory source use amount and the acquisition of operation progress msg of MemCollector periodic harvest operation task;

(2) task t is set up _ijmemory source use amount anticipation function;

2.1) task t is set _ijmemory source usage forecastings amount be function about the time, anticipation function shape as wherein, a _ijwith c _ijit is parameter to be evaluated;

2.2) task t _ijthe MemPredictor run on the computing node of place reads task t in shared variable RunTasklist _ijmemory source use amount historical record set RU _ij, RU _ij={ r _l| r _l: (t _l, m _l), t ₀=0, t _l>=t _l-1, 0≤l≤n _ij, wherein, t _lbe the 1st record moment, m _lfor t _lrecord moment task t _ijinternal memory use amount; RU is set _ijfor forecast sample;

2.3) x=ln (t+1) is made, y=q _ij(t), by task t _ijmemory source demand forecast functional transformation be y=b _ijx+c _ij, parameter transformation to be evaluated is b _ijand c _ij; MemPredictor arranges the number of times C of Function Fitting failure _ij=0;

2.4) MemPredictor is according to summary of the invention 2.4) in method, adopt linear regression method to carry out converting the matching of anticipation function;

2.5) MemPredictor is according to summary of the invention 2.5) described in method, adopt t method of inspection to carry out significance test to fitting function;

2.6) MemPredictor judges whether inspection meets, and if so, then Function Fitting success, performs step 2.10); Otherwise, perform step 2.7);

2.7) MemPredictor is by C _ijadd 1, judge whether matching total degree is greater than total degree threshold value C _max, then matching failure, performs step 2.9); Otherwise, perform step 2.10); Wherein, C _maxfor anticipation function matching frequency threshold value;

2.8) forecast sample is revised; MemPredictor reads task t in shared variable RunTasklist _ijmemory source use amount historical information set RU _ij, from RU _ijthe all r that satisfies condition of middle removal _l-1: (t _l-1, m _l-1), r _l-1∈ RU _ij, m _l-m _l-1the memory source use amount record r of < 0 _l: (t _l, m _l), and perform step 2.4);

2.9) MemPredictor marks task t _ijmemory source use amount anticipation function builds unsuccessfully, performs step (3);

2.10) MemPredictor marks task t _ijmemory source use amount anticipation function successfully constructs, by task t _ijmemory source demand forecast function in optimum configurations to be estimated be

(3) calculation task t _ijmemory source additional quantity RA _ijwith burst size RD _ij; Current time is made to be c_c _ij, task t _ijcurrent memory use amount is RN _ij;

3.1) MemPredictor arranges RA _ij=0, RD _ij=0;

3.2) MemPredictor reads task t from shared variable RunTasklist _ijthe task c_i initial time of running _ijwith task t _ijcurrent operation progress p _ij, and according to summary of the invention 3.2) described method estimation tasks t _ijcomplete moment c_f _ij;

3.3) MemPredictor reads task t from shared variable RunTasklist _ijcurrent memory Resourse Distribute amount RC _ij, and obtain t according to the priority in record moment _ijthe last memory source uses record, as task t _ijcurrent memory use amount RN _ij; MemPredictor judges whether meet, if so, then perform step 3.7); Otherwise, perform step 3.4);

3.4) MemPredictor is according to step (2), judges task t _ijanticipation function whether successfully construct, if so, then perform step 3.5), otherwise, perform step 3.6);

3.5) MemPredictor calculates RA according to formula (8) _ij, perform step 3.8);

${\mathrm{RA}}_{\mathrm{ij}}=\left\{\begin{array}{c}{q}_{\mathrm{ij}}({c\_c}_{\mathrm{ij}}-{c\_i}_{\mathrm{ij}}+\mathrm{\&tau;})-{\mathrm{RN}}_{\mathrm{ij}},\mathrm{\&tau;}<{c\_f}_{\mathrm{ij}}-{c\_c}_{\mathrm{ij}}\\ q_{\mathrm{ij}}({c\_f}_{\mathrm{ij}}-{c\_i}_{\mathrm{ij}})-{\mathrm{RN}}_{\mathrm{ij}},\mathrm{\&tau;}\&GreaterEqual;{c\_f}_{\mathrm{ij}}-{c\_c}_{\mathrm{ij}}\end{array}\right.---\left(8\right)$

3.6) RA is calculated according to formula (9) _ij, perform step 3.8);

RA _ij＝(RC _ij-RN _ij)×1.5 (9)

3.7) MemPredictor arranges RA _ij=0;

3.8) MemPredictor judges whether meet, if so, then perform step 3.9); Otherwise, perform step 3.12);

3.9) MemPredictor is according to step (2), judges task t _ijanticipation function whether successfully construct, if so, then perform step 3.10), otherwise, perform step 3.11);

3.10) RM is made _ij=q _ij(c_f _ij-c_i _ij), MemPredictor calculates RD according to formula (10) _ij, perform step (4);

3.11) MemPredictor calculates RD according to formula (11) _ij, perform step (4);

${\mathrm{RD}}_{\mathrm{ij}}=\frac{1}{5}{\mathrm{RN}}_{\mathrm{ij}}---\left(11\right)$

3.12) MemPredictor arranges RD _ij=0;

(4) task t _ijmemory source discharges, and makes t _ijrunning place node server is N _k, according to step

(3) execution result, to task t _ijperform following operation:

4.1) MemPredictor calls the task internal memory release nation method that the MemReallocator that is in same computing node provides, by the task number of this task, job number and internal memory burst size RD _ijpass to MemReallocator;

4.2) MemReallocator is according to the internal memory burst size information of task, the mode called with RPC, by place computing node N _kwith releasing memory amount RD _ijsend to MemUpdator;

4.3) MemUpdator is according to node number and internal memory burst size information, the node N safeguarded in amendment ResourceManager primary module _kfree memory stock number R _{k_free}, R _{k_free}← R _{k_free}+ RD _ij, then return internal memory to MemReallocator and discharge successfully/failure information;

4.4) MemReallocator is according to return message, if success, then revises task t in shared variable RunTasklist _ijcurrent memory Resourse Distribute amount RC _ij, perform RC _ij← RC _ij-RD _ij;

(5) task memory source adds; According to step (2) execution result, to task t _ijperform following operation:

5.1) the task internal memory that MemPredictor calls the MemReallocator that is in same computing node and provides adds nation method, by the task number of this task, job number and internal memory additional quantity RA _ijpass to MemReallocator;

5.2) MemReallocator is according to the internal memory additional quantity information of task, the mode called with RPC, by place computing node N _kwith additional amount of ram RA _ijsend to MemUpdator;

5.3) MemUpdator is according to internal memory supplement information, reads the node N safeguarded in ResourceManager primary module _kfree memory stock number R _{k_free}if, R _{k_free}>=RA _ij, then MemUpdator performs R _{k_free}← R _{k_free}-RA _ij, and return internal memory to MemReallocator and add success message, and perform step 5.9); Otherwise, return failed message to MemReallocator, and perform step 5.4);

5.4) MemReallocator is according to return messages, if add unsuccessfully, MemReallocator reads each operation task t in shared variable RunTasklist _uvthe current operation progress p of task _uv, and in the mode that RPC calls, account for the ratio f of general assignment number from number of already having finished the work belonging to the TaskUpdator acquisition task that this task is corresponding _u, MemReallocator is according to 5.2)-5.6) described in method, calculate the preempting priority of operation task, choose the set of tasks TP needing to seize its memory source;

5.5) MemReallocator obtains all task current memory Resourse Distribute amounts in TP from shared variable RunTasklist, and summation is as node N _kpreemptible memory source amount Pr_R _k; The mode that MemReallocator calls with RPC, by task t _ijplace computing node N _k, add amount of ram RA _ijand node N _kpreemptible memory source amount Pr_R _kinformation sends to MemUpdator;

5.6) MemUpdator seizes supplement information according to internal memory, reads the node N safeguarded in ResourceManager primary module _kfree memory stock number R _{k_free}, judge R _{k_free}+ Pr_R _k>=RA _ijwhether meet, if so, then perform step 5.7), otherwise, return internal memory to MemReallocator and seize additional failed message, and perform step 5.10);

5.7) MemUpdator performs R _{k_free}← R _{k_free}+ Pr_R _k-RA _ij, and return internal memory to MemReallocator and seize additional success message;

5.8) MemReallocator seizes additional return messages according to internal memory, if add successfully, then call the method for its place NodeManager, stop the TaskContainer that in TP, all tasks are corresponding and run process, the record that operation task is corresponding is deleted in shared variable RunTasklist, and the task status that the corresponding TaskUpdator of calling task provides changes RPC method, task status is re-set as and waits to dispatch by notice TaskUpdator;

5.9) MemReallocator marks task t _ijmemory source adds successfully, task t in amendment shared variable RunTasklist _ijcurrent memory Resourse Distribute amount RC _ij, perform RC _ij← RC _ij+ RA _ij, go to step (6);

5.10) MemReallocator seizes additional return messages according to internal memory, if add unsuccessfully, emReallocator marks task t _ijmemory source adds unsuccessfully;

(6) recall; After a memory source dynamic assignment end cycle, task t _ijthe MemColloector run on the computing node of place searches in shared variable RunTasklist whether there is task t _ijcorresponding list cell, if exist, then performs step (1), if do not exist, then performs step (7);

(7) terminate: stop task t _ijmemory source code reassignment function.