CN103605662A - Distributed computation frame parameter optimizing method, device and system - Google Patents

Abstract

The invention is suitable for the IT technical field and provides a distributed computation frame parameter optimizing method, device and system. The method comprises the steps of obtaining currently-submitted parallel computation frame operation; retrieving historical distributed computation frame operation similar to the distributed computation frame operation in a distributed computation frame operation historical operation database, wherein the distributed computation frame operation historical operation database comprises execution information and configuration parameters of the distributed computation frame operation; retrieving historical distributed computation frame operation similar to the distributed computation frame operation during the similar distributed computation frame operation; performing optimizing configuration on the configuration parameters of the distributed computation frame operation according to configuration parameters of the similar historical distributed computation frame operation. By adopting the distributed computation frame parameter optimizing method, device and system, parameter configuration rationalization, automatization and self learning of Mapreduce operation can be achieved.

Description

A kind of distributed computing framework parameter optimization method, Apparatus and system

Technical field

The invention belongs to IT technical field, relate in particular to a kind of distributed computing framework parameter optimization method, Apparatus and system.

Background technology

Along with global information industry is at continuous fusion development, Internet resources and data scale are also in continuous growth, especially in fields such as internet, applications, ecommerce, data volume presents the trend of rapid growth, in order to solve these data-intensive computational problems, cloud computing is arisen at the historic moment, and Map/Reduce programming model obtains applying more and more widely as a kind of important means of simplifying large-scale data processing.MapReduce is a kind of common software framework of realizing distributed parallel calculation task that Google proposes, and it has simplified the concurrent software programming mode on the super large cluster being comprised of common computer, can be used for the parallel computation of large-scale dataset.In MapReduce distributed computing system, the parameter optimization strategy of systematic parameter is directly connected to the fairness of using resource between the utilization factor of entire system resource and each user.Therefore, the parameter optimization algorithm of systematic parameter becomes a major challenge of MapReduce systems face.

Yet the scheme of current common MapReduce parameter optimization has two kinds, details are as follows:

While 1, moving Mapreduce operation, by monitoring tools as nmon, monitor that the performance index of group system are (as CPU usage, memory usage, disk and network I/O etc.), carry out fast detecting performance bottleneck, assist slip-stick artist more targetedly performance bottleneck point to be carried out to parameter improvement and optimization.

2, by carry out in advance Mapreduce operation on a simulation cluster, the operation characteristic of simultaneously monitoring this task (comprises output file, each stage running time, process and the data volume of transmission, and each resource information of taking of the progress of work etc.), calculate the cost of each stage consumption of natural resource, then by revising resource parameters, estimate the working time of actual Mapreduce operation, until reach working time, can accept scope, thereby arrive the object of MapReduce performance optimization.

This technical scheme 1 major defect is just to have provided performance bottleneck point detecting method, does not provide concrete improvement in performance scheme, even know performance bottleneck Dian， domestic consumer does not still know how to revise parameter in a lot of situation.Even if user knows how to revise MapReduce parameter, but this scheme implementation efficiency is also very low, need to manually make repeated attempts repeatedly, just can make performance reach compared with the figure of merit, therefore cannot solve the problem of the rationalization of MapReduce parameter configuration, robotization.

These technical scheme 2 major defects be the pre-execution and the analysis that increase program be cost, and because this scheme is the tuning of carrying out for individual task, when the task type submitted to as user is complicated and changeable, all to re-start pre-execution at every turn, its versatility is reduced, therefore cannot solve the problem of MapReduce parameter configuration self study.

Summary of the invention

The object of the embodiment of the present invention is to provide distributed computing framework parameter optimization method, is intended to solve the problem of the rationalization of MapReduce parameter configuration, robotization and self study.

First aspect, a kind of method of distributed computing framework parameter optimization, comprising:

Obtain the distributed computing framework operation when submit;

In the distributed computing framework history data storehouse of setting up in advance, retrieve the historical distributed computing framework operation similar with described distributed computing framework operation, described distributed computing framework history data storehouse comprises execution information and the configuration parameter of historical distributed computing framework operation;

In similar distributed computing framework operation, retrieve the historical distributed computing framework operation similar to described distributed computing framework operation;

Configuration parameter according to similar historical distributed computing framework operation, is optimized configuration to the configuration parameter of described distributed computing framework operation.

In conjunction with first aspect, described in distributed computing framework Historical Jobs runtime database, retrieve the historical distributed computing framework operation similar with described distributed computing framework operation, comprising:

When described distributed computing framework operation does not exist the distributed computing framework operational factor of appointment, in distributed computing framework Historical Jobs runtime database, retrieve the historical distributed computing framework operation similar with described distributed computing framework operation.

In conjunction with first aspect, describedly according to the configuration parameter of similar historical distributed computing framework operation, the configuration parameter of described distributed computing framework operation is optimized to configuration, comprising:

In similar historical distributed computing framework operation, obtain the highest historical distributed computing framework operation of scoring, use the configuration parameter of the highest historical distributed computing framework operation of scoring as the configuration parameter of described distributed computing framework operation, the configuration parameter of described distributed computing framework operation is optimized to configuration.

In conjunction with first aspect, when described distributed computing framework Job execution is complete, collect described distributed computing framework Job execution information and configuration parameter;

Described distributed computing framework Job execution information is marked, and described distributed computing framework operation is saved in described distributed computing framework Historical Jobs database.

In conjunction with first aspect, described distributed computing framework Job execution information is marked, be specially:

Obtain time loss and the space consuming of described distributed computing framework job run;

According to the time loss of described distributed computing framework job run and space consuming, generate time loss cost and the space consuming cost of described distributed computing framework job run;

According to the Rating Model of the distributed computing framework operation of setting up in advance, and the time loss cost of described distributed computing framework job run and space consuming cost are marked to distributed computing framework job run.

In conjunction with first aspect, the Rating Model of the distributed computing framework operation of setting up in advance in described basis, and before the time loss cost of described distributed computing framework job run and space consuming cost mark to distributed computing framework job run, comprising:

Set up Rating Model;

Described Rating Model is

Wherein, F _jobscoring when (τ, υ) represents a job run, τ, υ represents respectively corresponding time loss cost, space consuming cost, τ during Job execution scoring, the function of υ.α, β is the weight of the scoring of time loss cost and the scoring of space consuming cost, i is the sequence number of the middle property value of time loss cost, τ _icorresponding to i property value of time loss cost, j is the sequence number of the middle property value of space consuming cost, υ _jexpression is corresponding to j property value of space consuming cost,

the weight of corresponding above-mentioned two attributes respectively, parameter alpha wherein, β is for regulating setup times to consume cost and space consuming cost which more preferably; object be in order to eliminate the different difference of each cost attribute value magnitude, by arranging carry out normalized calculating F _job(τ, υ).

In conjunction with first aspect, described described distributed computing framework operation is saved in described distributed computing framework Historical Jobs database, be specially:

Adopt tree-like storage mode, described distributed computing framework operation is saved in described distributed computing framework Historical Jobs database.

In conjunction with first aspect, in the distributed computing framework history data storehouse of setting up in advance, retrieve the historical distributed computing framework operation similar with described distributed computing framework operation, comprising:

In the distributed computing framework history data storehouse of setting up in advance, adopt search tree node mode, retrieve the historical distributed computing framework operation similar with described distributed computing framework operation.

In conjunction with first aspect, the historical distributed computing framework operation that described retrieval is similar to described distributed computing framework operation, comprising:

Whether retrieve described distributed computing framework operation equates with the determinant attribute in historical distributed computing framework operation;

Determinant attribute in described distributed computing framework operation when determinant attribute in historical distributed computing framework operation equates, represents that described historical distributed computing framework operation is the historical distributed computing framework operation similar to described distributed computing framework operation.

In conjunction with first aspect, the historical distributed computing framework operation that described retrieval is similar to described distributed computing framework operation, also comprises:

Obtain the attribute field of described distributed computing framework operation and historical distributed computing framework operation, described attribute field comprises determinant attribute set and non-key community set;

According to similarity model and the described attribute field of the distributed computing framework operation of setting up in advance, generate respectively the similarity of operation and the similarity of cluster environment information, the similarity of described operation is the similarity of described distributed computing framework operation and historical distributed computing framework operation, and the similarity of described cluster environment information is that the similarity of described distributed computing framework operation and historical distributed computing framework operation is respectively in the similarity of described cluster environment information;

According to comprehensive similarity model and the weighted strategy of the distributed computing framework operation of setting up in advance, generate the comprehensive similarity of described distributed computing framework operation and historical distributed computing framework operation.

In conjunction with first aspect, the similarity model of the distributed computing framework operation of setting up in advance in described basis and described attribute field, before generating respectively the similarity of operation and the similarity of cluster environment information, comprising:

Set up the similarity model of distributed computing framework operation,

Described similarity model is:

$\mathrm{sim}(A,B)=\left(\begin{array}{c}\underset{k\&Element;\mathrm{<figure-callout\; id="1"\; label="K"\; filenames="HDA0000399144580000022.png"\; state="\{\{state\}\}">K</figure-callout>}}{\mathrm{\&Pi;}}\left\{\begin{array}{c}1,p_{A,k}=p_{B,k}\\ 0,p_{A,k}\&NotEqual;p_{B,k}\end{array}\right.\end{array}\right)*\left(\frac{\underset{i\&Element;I}{\mathrm{\&Sigma;}}{p}_{A,i}*p_{B,i}}{\sqrt{\underset{i\&Element;I}{\mathrm{\&Sigma;}}{p}_{A,i}^2}*\sqrt{\underset{i\&Element;I}{\mathrm{\&Sigma;}}{p}_{B,i}^2}}\right)$

Wherein K is determinant attribute set, and k is the sequence number of the middle determinant attribute of determinant attribute set, and I is non-key community set, the sequence number of non-key attribute during i is non-key community set, p _a,kk the determinant attribute that represents operation A, p _a,ifor i the non-key attribute of sign operation A, p _b,kfor k the determinant attribute of sign operation B, p _b,ii the non-key attribute for sign operation B.

In conjunction with first aspect, comprehensive similarity model and the weighted strategy of the distributed computing framework operation of setting up in advance in basis, before generating the comprehensive similarity of described distributed computing framework operation and historical distributed computing framework operation, comprising:

Set up the comprehensive similarity model of distributed computing framework operation,

Described comprehensive similarity model is:

sim(A,B)＝α×sim(JobA,JobB)+β×sim(ClusterA,ClusterB)

Wherein sim (JobA, JobB) represents operation A, the similarity of B, and sim (ClusterA, ClusterB) represents operation A, B is respectively in the similarity of described cluster environment information.Sim (A, B) represents operation A, and between B, with reference to the similarity of cluster environment information, α is the first weight parameter in weighted strategy, and β is the second weight parameter in weighted strategy.

In conjunction with second aspect, a kind of MapReduce parameter optimization device, comprising:

Acquiring unit, for obtaining the distributed computing framework operation when submit;

The first retrieval unit, for the distributed computing framework history data storehouse setting up in advance, retrieve the historical distributed computing framework operation similar with described distributed computing framework operation, described distributed computing framework history data storehouse comprises execution information and the configuration parameter of historical distributed computing framework operation;

The second retrieval unit, for the distributed computing framework operation similar, retrieves the historical distributed computing framework operation similar to described distributed computing framework operation;

Dispensing unit, for according to the configuration parameter of similar historical distributed computing framework operation, is optimized configuration to the configuration parameter of described distributed computing framework operation.

In conjunction with second aspect, described the first retrieval unit, comprising:

Check subelement, for checking whether described distributed computing framework operation exists the distributed computing framework operational factor of appointment;

Carry out subelement, for when there is not the distributed computing framework operational factor of appointment in described distributed computing framework operation, described in carrying out, in distributed computing framework Historical Jobs runtime database, retrieve the step of the historical distributed computing framework operation similar with described distributed computing framework operation.

In conjunction with second aspect, described dispensing unit, also for the historical distributed computing framework operation similar, obtain the highest historical distributed computing framework operation of scoring, use the configuration parameter of the highest historical distributed computing framework operation of scoring as the configuration parameter of described distributed computing framework operation, the configuration parameter of described distributed computing framework operation is optimized to configuration.

In conjunction with second aspect, also comprise:

Collector unit, for when described distributed computing framework Job execution is complete, collects described distributed computing framework Job execution information and configuration parameter;

Scoring unit, for described distributed computing framework Job execution information is marked, and is saved in described distributed computing framework operation in described distributed computing framework Historical Jobs database.

In conjunction with second aspect, described scoring unit, comprising:

Obtain subelement, for obtaining time loss and the space consuming of described distributed computing framework job run;

Generate subelement, for according to the time loss of described distributed computing framework job run and space consuming, generate time loss cost and the space consuming cost of described distributed computing framework job run;

Scoring subelement, for according to the Rating Model of the distributed computing framework operation of setting up in advance, and the time loss cost of described distributed computing framework job run and space consuming cost are marked to distributed computing framework job run.

In conjunction with second aspect, described scoring unit, also comprises:

Set up subelement, for setting up Rating Model;

Described Rating Model is

Wherein, F _jobscoring when (τ, υ) represents a job run, τ, υ represents respectively corresponding time loss cost, space consuming cost, τ during Job execution scoring, the function of υ.α, β is the weight of the scoring of time loss cost and the scoring of space consuming cost, i is the sequence number of the middle property value of time loss cost, τ _icorresponding to i property value of time loss cost, j is the sequence number of the middle property value of space consuming cost, υ _jexpression is corresponding to j property value of space consuming cost,

the weight of corresponding above-mentioned two attributes respectively, parameter alpha wherein, β is for regulating setup times to consume cost and space consuming cost which more preferably;

object be in order to eliminate the different difference of each cost attribute value magnitude, by arranging

carry out normalized calculating F _job(τ, υ).

In conjunction with second aspect, described scoring unit, also comprises:

Preserve subelement, for adopting tree-like storage mode, described distributed computing framework operation is saved in described distributed computing framework Historical Jobs database.

In conjunction with second aspect, the first retrieval unit, comprising:

The first retrieval subelement, for the distributed computing framework history data storehouse setting up in advance, adopts search tree node mode, retrieves the historical distributed computing framework operation similar with described distributed computing framework operation.

In conjunction with second aspect, the second retrieval unit, comprising:

Whether the second retrieval subelement, equate with the determinant attribute of historical distributed computing framework operation for retrieving described distributed computing framework operation;

Similar subelement, for when the determinant attribute of described distributed computing framework operation is when determinant attribute in historical distributed computing framework operation equates, represent that described historical distributed computing framework operation is the historical distributed computing framework operation similar to described distributed computing framework operation.

In conjunction with second aspect, the second retrieval unit, also comprises:

Obtain subelement, for obtaining the attribute field of described distributed computing framework operation and historical distributed computing framework operation, described attribute field comprises determinant attribute set and non-key community set;

First generates subelement, similarity model and the described attribute field of the distributed computing framework operation of setting up in advance for basis, generate respectively the similarity of operation and the similarity of cluster environment information, the similarity of described operation is the similarity of described distributed computing framework operation and historical distributed computing framework operation, and the similarity of described cluster environment information is that the similarity of described distributed computing framework operation and historical distributed computing framework operation is respectively in the similarity of described cluster environment information;

Second generates subelement, for according to comprehensive similarity model and the weighted strategy of the distributed computing framework operation of setting up in advance, generates the comprehensive similarity of described distributed computing framework operation and historical distributed computing framework operation.

In conjunction with second aspect, first generates subelement, also comprises:

First sets up subelement, for setting up the similarity model of distributed computing framework operation,

Described similarity model is:

$\mathrm{sim}(A,B)=\left(\begin{array}{c}\underset{k\&Element;\mathrm{<figure-callout\; id="1"\; label="K"\; filenames="HDA0000399144580000022.png"\; state="\{\{state\}\}">K</figure-callout>}}{\mathrm{\&Pi;}}\left\{\begin{array}{c}1,p_{A,k}=p_{B,k}\\ 0,p_{A,k}\&NotEqual;p_{B,k}\end{array}\right.\end{array}\right)*\left(\frac{\underset{i\&Element;I}{\mathrm{\&Sigma;}}{p}_{A,i}*p_{B,i}}{\sqrt{\underset{i\&Element;I}{\mathrm{\&Sigma;}}{p}_{A,\mathrm{<figure-callout\; id="2"\; label="i"\; filenames="HDA0000399144580000022.png"\; state="\{\{state\}\}">i</figure-callout>}}^2}*\sqrt{\underset{i\&Element;I}{\mathrm{\&Sigma;}}{p}_{B,i}^2}}\right)$

Wherein K is determinant attribute set, and k is the sequence number of the middle determinant attribute of determinant attribute set, and I is non-key community set, the sequence number of non-key attribute during i is non-key community set, p _a,kk the determinant attribute that represents operation A, p _a,ifor i the non-key attribute of sign operation A, p _b,kfor k the determinant attribute of sign operation B, p _b,ii the non-key attribute for sign operation B.

In conjunction with second aspect, second generates subelement, also comprises:

Second sets up subelement, for setting up the comprehensive similarity model of distributed computing framework operation,

Described comprehensive similarity model is:

sim(A,B)＝α×sim(JobA,JobB)+β×sim(ClusterA,ClusterB)

Wherein sim (JobA, JobB) represents operation A, the similarity of B, and sim (ClusterA, ClusterB) represents operation A, B is respectively in the similarity of described cluster environment information.Sim (A, B) represents operation A, and between B, with reference to the similarity of cluster environment information, α is the first weight parameter in weighted strategy, and β is the second weight parameter in weighted strategy.

The third aspect, the client and the management of computing node that comprise the distributed computing framework operation of above-mentioned parameter optimization device, submission, wherein, between the client of described distributed computing framework operation and described management of computing node, by described parameter optimization device, connect.

In the present embodiment, according to the configuration parameter of similar historical distributed computing framework operation, configuration parameter to described distributed computing framework operation is optimized configuration, thereby avoided parameter configuration unreasonable, cause the situation of the significant wastage of cluster computational resource, avoided user to want manually the configuration parameter of MapReduce operation to be adjusted simultaneously, tuning mode efficiency is low, and configuration parameter is only applicable to current task, the situation that does not possess versatility, make when user submits a Mapreduce operation to, can to parameter, be optimized configuration automatically, the Mapreduce operation having moved before can learning, and do not need all again to optimize MapReduce parameter at every turn, realizing the configuration of Mapreduce job parameter rationalizes, robotization and self study.

Accompanying drawing explanation

Fig. 1 is the realization flow figure of the method for a kind of distributed computing framework parameter optimization of providing of the embodiment of the present invention;

Fig. 2 is the implementing procedure figure of the preservation distributed computing framework operation that provides of the embodiment of the present invention;

Fig. 3 is the preferably sample figure of the tree-like distributed storage Computational frame operation that provides of the embodiment of the present invention;

Fig. 4 is the preferably implementing procedure figure of the distributed computing framework parameter optimization that provides of the present embodiment;

Fig. 5 is the structural representation of a kind of parameter optimization device of providing of the embodiment of the present invention;

Fig. 6 is preferably network architecture diagram of the distributed computing framework Parameter Optimization System that provides of the present embodiment;

Fig. 7 is the structural representation of a kind of parameter optimization device of providing of the embodiment of the present invention.

Embodiment

In order to make object of the present invention, technical scheme and advantage clearer, below in conjunction with drawings and Examples, the present invention is further elaborated.Should be appreciated that specific embodiment described herein, only in order to explain the present invention, is not intended to limit the present invention.

Fig. 1 is the realization flow figure of the method for a kind of distributed computing framework parameter optimization of providing of the embodiment of the present invention;

In step S101, obtain the distributed computing framework operation when submit;

In the present embodiment, distributed computing framework includes but not limited to parallel computation framework Mapreduce.

In step S102, in the distributed computing framework history data storehouse of setting up in advance, retrieve the historical distributed computing framework operation similar with described distributed computing framework operation, described distributed computing framework history data storehouse comprises execution information and the configuration parameter of historical distributed computing framework operation;

In the present embodiment, by depositing in the distributed computing framework history data storehouse of distributed computing framework operation, according to the similar sign of distributed computing framework operation, retrieve the distributed computing framework operation similar with distributed computing framework operation, retrieve the implementation process of the distributed computing framework operation similar with distributed computing framework operation, at following embodiment, describe, at this, do not repeat.

In the present embodiment, when described distributed computing framework operation exists the distributed computing framework operational factor of appointment, according to the distributed computing framework operational factor of appointment, carry out described distributed computing framework operation;

In step S103, in similar distributed computing framework operation, retrieve the historical distributed computing framework operation similar to described distributed computing framework operation;

In the present embodiment, whether the operation of retrieval distributed computing framework equates with the determinant attribute in historical distributed computing framework operation, determinant attribute in distributed computing framework operation is when determinant attribute in historical distributed computing framework operation equates, when namely the numerical value of both determinant attributes is consistent, represent that this history distributed computing framework operation is the historical distributed computing framework operation similar to distributed computing framework operation.

In the present embodiment, when retrieving the distributed computing framework operation similar with distributed computing framework operation, in similar distributed computing framework operation, by the similarity model of setting up in advance, retrieve the distributed computing framework operation similar to this distributed computing framework operation, the implementation process of retrieving the distributed computing framework operation similar to this distributed computing framework operation, describes at following embodiment, at this, does not repeat.

In step S104, the configuration parameter according to similar historical distributed computing framework operation, is optimized configuration to the configuration parameter of described distributed computing framework operation.

In the present embodiment, according to the configuration parameter of similar historical distributed computing framework operation, configuration parameter to described distributed computing framework operation is optimized configuration, particularly, when retrieving the distributed computing framework operation similar to this distributed computing framework operation, in similar historical distributed computing framework operation, obtain the highest historical distributed computing framework operation of scoring, use the configuration parameter of the highest historical distributed computing framework operation of scoring as the configuration parameter of described distributed computing framework operation, configuration parameter to described distributed computing framework operation is optimized configuration, and carry out this distributed computing framework operation, this distributed computing framework history data storehouse comprises execution information and the configuration parameter of historical distributed computing framework operation.

In the present embodiment, by analysis distribution formula Computational frame job run historical information, find a series of distributed computing framework operations similar to current distributed computing framework operation, from distributed computing framework history library, find the configuration parameter of the highest distributed computing framework operation of a set of scoring, configuration reference as current distributed computing framework job parameter, thereby avoided parameter configuration unreasonable, cause the situation of the significant wastage of cluster computational resource, avoided user to want manually the configuration parameter of distributed computing framework operation to be adjusted simultaneously, tuning mode efficiency is low, and configuration parameter is only applicable to current task, the situation that does not possess versatility, make when user submits a distributed computing framework operation to, can to parameter, be optimized configuration automatically, the distributed computing framework operation having moved before can learning, and do not need all again to optimize distributed computing framework parameter at every turn, realizing the configuration of distributed computing framework job parameter rationalizes, robotization and self study.

As a preferred embodiment of the present invention, in the distributed computing framework history data storehouse of setting up in advance, retrieve the historical distributed computing framework operation similar with described distributed computing framework operation, comprising:

Check whether described distributed computing framework operation exists the distributed computing framework operational factor of appointment;

When there is not the distributed computing framework operational factor of appointment in described distributed computing framework operation, described in carrying out, in distributed computing framework Historical Jobs runtime database, retrieve the step of the historical distributed computing framework operation similar with described distributed computing framework operation.

In the present embodiment, when submitting task to, whether the ginseng list file in the operation of system acquisition distributed computing framework, there is the distributed computing framework operational factor of appointment by the inspection of ginseng list file.

As a preferred embodiment of the present invention, according to the configuration parameter of similar historical distributed computing framework operation, configuration parameter to described distributed computing framework operation is optimized configuration, also can be in similar historical distributed computing framework operation, obtaining marks is greater than a plurality of distributed computing framework operations of pre-set threshold value, according to the similarity degree of a plurality of distributed computing framework operations, the configuration parameter of a plurality of distributed computing framework operations of weighted calculation, the configuration parameter that use weighted calculation obtains is as the configuration parameter of described distributed computing framework operation, configuration parameter to described distributed computing framework operation is optimized configuration.For example, for new operation Job_A, its parameter setting can be based on following K operation B the most similar _icalculate:

P _A＝Σ _i＝1,2,...Ksim(A,B _i)*P _i

P wherein _afor some configuration parameters of new operation Job_A, K is the number of jobs that the scoring selected is greater than certain threshold value, B _irepresent i operation in these operations.Generally speaking new job parameter can be calculated and produce according to Similarity-Weighted based on TopK the highest operation of scoring, owing to combining a plurality of distributed computing framework job parameter configurations, can make like this optimization model of new work industry more stable.

With reference to figure 2, Fig. 2 is the implementing procedure figure of the preservation distributed computing framework operation that provides of the embodiment of the present invention, and details are as follows:

S201, when described distributed computing framework Job execution is complete, collects described distributed computing framework Job execution information and configuration parameter;

S202, marks to described distributed computing framework Job execution information, and described distributed computing framework operation is saved in described distributed computing framework Historical Jobs database.

As a preferred embodiment of the present invention, collect described distributed computing framework Job execution information and configuration parameter, comprising:

Collect current cluster environment information and the operation configuration parameter information of described distributed computing framework operation.

In the present embodiment, cluster environment information includes but not limited to computing node number, memory amount, cpu total amount, dfs block size, dfs number of copies, the network bandwidth, disk I/O.

In the present embodiment, operation configuration parameter includes but not limited to Mapper class, Reducer class, corresponding byte number, input-output file form, respective path, split information.

In the present embodiment, collect current cluster environment information and the operation configuration parameter information of described distributed computing framework operation, the similarity that is mainly used in the operation of subsequent distribution formula Computational frame is calculated.

As a preferred embodiment of the present invention, the described execution information to distributed computing framework operation is marked, and comprising:

Obtain time loss and the space consuming of described distributed computing framework job run;

According to the time loss of described distributed computing framework job run and space consuming, generate time loss cost and the space consuming cost of described distributed computing framework job run;

According to the Rating Model of the distributed computing framework operation of setting up in advance, and the time loss cost of described distributed computing framework job run and space consuming cost are marked to distributed computing framework job run.

In the present embodiment, the time loss that time loss is cost attribute.

Reference table 1, table 1 is the preferably time loss cost attribute table that the embodiment of the present invention provides, it has comprised the cost attribute name of part consumption computing time.

Table-1 time loss cost attribute table

Note: in concrete enforcement, can be not limited to above-mentioned attribute field.

In the present embodiment, according to the time loss of distributed computing framework job run, generate the time loss cost of distributed computing framework job run.Can pass through, each phases-time consumes, and the data volume of input and output is calculated acquisition.For example, HdfsReadCost in time loss cost attribute table, reading of the every byte HDFS of its time loss cost is consuming time, can read the T.T. that quantity reads divided by byte by byte, generates HdfsReadCost time loss cost (reading of every byte HDFS is consuming time).

In the present embodiment, the space consuming that space consuming is cost attribute.

Reference table 2, table 2 is preferably space consuming cost attribute tables that the embodiment of the present invention provides, it has comprised the cost attribute name that part computer memory consumes.

Table-2 space consuming cost attribute tables

Note: in concrete enforcement, can be not limited to above-mentioned attribute field.

In the present embodiment, according to the space consuming of distributed computing framework job run, generate the space consuming cost of distributed computing framework job run.Can consume by each stage space, and the data volume of input and output is calculated acquisition.

For example, cost attribute name MapAvgMemBytes, its space consuming cost be average every byte of Map stage to internal memory resource consumption, by average memory consumption divided by input the number that records generate, specific formula for calculation is as follows:

$\mathrm{MapAvgMemBytes}=\frac{\mathrm{MapAvgMem}}{\mathrm{MapInputBytes}}$

Wherein MapAvgMem is average memory consumption, the number that records that MapInputBytes is input,

MapAvgMem and MapInputBytes can directly obtain by the Metrics of distributed computing framework.In addition, cost attribute name MapCPUCostPerBytes, its space consuming cost be Map each byte of stage average CPU consume, the number that records by MAP stage CPU internal memory wastage in bulk or weight divided by input generates, specific formula for calculation is as follows:

$\mathrm{MapCPUCostPerBytes}=\frac{\mathrm{MapCPUCost}}{\mathrm{MapInputBytes}}$

Wherein MapCPUCost and MapInputBytes field can directly be obtained by the Metrics of distributed computing framework.

As a preferred embodiment of the present invention, the Rating Model of the distributed computing framework operation of setting up in advance in described basis, and before the time loss cost of described distributed computing framework job run and space consuming cost mark to distributed computing framework job run, comprising:

Set up Rating Model;

Described Rating Model is

Wherein, F _jobscoring when (τ, υ) represents a job run, τ, υ represents respectively corresponding time loss cost, space consuming cost, τ during Job execution scoring, the function of υ.α, β is the weight of the scoring of time loss cost and the scoring of space consuming cost, i is the sequence number of the middle property value of time loss cost, τ _icorresponding to i property value of time loss cost, j is the sequence number of the middle property value of space consuming cost, υ _jexpression is corresponding to j property value of space consuming cost, the weight of corresponding above-mentioned two attributes respectively, parameter alpha wherein, β is for regulating setup times to consume cost and space consuming cost which more preferably;

object be in order to eliminate the different difference of each cost attribute value magnitude, by arranging

carry out normalized calculating F _job(τ, υ).

As a preferred embodiment of the present invention, described distributed computing framework operation is saved in described distributed computing framework Historical Jobs database, be specially:

Adopt tree-like storage mode, described distributed computing framework operation is saved in described distributed computing framework Historical Jobs database.

In the present embodiment, adopt tree-like storage mode, with tree root to the path representation of leaf node the class title of corresponding operation, described distributed computing framework operation is saved in described distributed computing framework Historical Jobs database.

With reference to figure 3, Fig. 3 is the preferably sample figure of the tree-like distributed storage Computational frame operation that provides of the embodiment of the present invention.

In the present embodiment, in the present embodiment, (as the dotted line frame in figure) in same leaf node left in the operation of identical class name in, so that follow-up when searching for similar distributed computing framework operation, can go out similar distributed computing framework operation by fast search.

As a preferred embodiment of the present invention, in the distributed computing framework history data storehouse of setting up in advance, retrieve the historical distributed computing framework operation similar with described distributed computing framework operation, comprising:

In the distributed computing framework history data storehouse of setting up in advance, adopt search tree node mode, retrieval adopts the distributed computing framework operation similar with described distributed computing framework operation of tree-like storage.

In the present embodiment, adopt search tree node mode, retrieval adopts the distributed computing framework operation similar with described distributed computing framework operation of tree-like storage.For example, with reference to figure 5, operation for com.huawei.pagerank.PageRank-1, by root node root->com->huawei-GreatT.Gre aT.GTpagerank, search for totally 3 times and can obtain the distributed computing framework operation similar with described distributed computing framework operation.

As a preferred embodiment of the present invention, the historical distributed computing framework operation that described retrieval is similar to described distributed computing framework operation, comprising:

Obtain the attribute field of described distributed computing framework operation and historical distributed computing framework operation, described attribute field comprises determinant attribute set and non-key community set;

According to similarity model and the described attribute field of the distributed computing framework operation of setting up in advance, generate respectively the similarity of operation and the similarity of cluster environment information, the similarity of described operation is the similarity of described distributed computing framework operation and historical distributed computing framework operation, and the similarity of described cluster environment information is that the similarity of described distributed computing framework operation and historical distributed computing framework operation is respectively in the similarity of described cluster environment information;

According to comprehensive similarity model and the weighted strategy of the distributed computing framework operation of setting up in advance, generate the comprehensive similarity of described distributed computing framework operation and historical distributed computing framework operation.

In the present embodiment, reference table 3, table 3 is preferably attribute field tables that the embodiment of the present invention provides, it comprises the field of using when part distributed computing framework operation similarity is calculated.

Table-3Job task attribution table

Note: in concrete enforcement, can be not limited to above-mentioned attribute field.

In the present embodiment, reference table 4, table 4 is attribute field tables of the better cluster environment that provides of the embodiment of the present invention, it comprises the field of using when part distributed computing framework operation similarity is calculated.

Table-4 cluster environment parameter similarities

Note: in concrete enforcement, can be not limited to above-mentioned attribute field.

As a preferred embodiment of the present invention, the similarity model of the distributed computing framework operation of setting up in advance in described basis and described attribute field, before generating respectively the similarity of operation and the similarity of cluster environment information, comprising:

Set up the similarity model of distributed computing framework operation,

Described similarity model is:

$\mathrm{sim}(A,B)=\left(\begin{array}{c}\underset{k\&Element;\mathrm{<figure-callout\; id="1"\; label="K"\; filenames="HDA0000399144580000022.png"\; state="\{\{state\}\}">K</figure-callout>}}{\mathrm{\&Pi;}}\left\{\begin{array}{c}1,p_{A,k}=p_{B,k}\\ 0,p_{A,k}\&NotEqual;p_{B,k}\end{array}\right.\end{array}\right)*\left(\frac{\underset{i\&Element;I}{\mathrm{\&Sigma;}}{p}_{A,i}*p_{B,i}}{\sqrt{\underset{i\&Element;I}{\mathrm{\&Sigma;}}{p}_{A,\mathrm{<figure-callout\; id="2"\; label="i"\; filenames="HDA0000399144580000022.png"\; state="\{\{state\}\}">i</figure-callout>}}^2}*\sqrt{\underset{i\&Element;I}{\mathrm{\&Sigma;}}{p}_{B,i}^2}}\right)$

Wherein K is determinant attribute set, and k is the sequence number of the middle determinant attribute of determinant attribute set, and I is non-key community set, the sequence number of non-key attribute during i is non-key community set, p _a,kk the determinant attribute that represents operation A, p _a,ifor i the non-key attribute of sign operation A, p _b,kfor k the determinant attribute of sign operation B, p _b,ii the non-key attribute for sign operation B.

Need to describe, for determinant attribute, be based on absolute phase isotype, once exist unequally in determinant attribute, directly returning to similarity is 0, and namely distributed computing framework operation and historical distributed computing framework operation are dissimilar.

As a preferred embodiment of the present invention, comprehensive similarity model and the weighted strategy of the distributed computing framework operation of setting up in advance in basis, before generating the comprehensive similarity of described distributed computing framework operation and historical distributed computing framework operation, comprising:

Set up the comprehensive similarity model of distributed computing framework operation,

Described comprehensive similarity model is:

sim(A,B)＝α×sim(JobA,JobB)+β×sim(ClusterA,ClusterB)

Wherein sim (JobA, JobB) represents operation A, the similarity of B, and sim (ClusterA, ClusterB) represents operation A, B is respectively in the similarity of described cluster environment information.Sim (A, B) represents operation A, and between B, with reference to the similarity of cluster environment information, α is the first weight parameter in weighted strategy, and β is the second weight parameter in weighted strategy.

In the present embodiment, the similarity between operation obtains based on above-mentioned formula, and parameter declaration is consistent with description above.

$\mathrm{sim}(\mathrm{JobA},\mathrm{JobB})=\left(\begin{array}{c}\underset{k\&Element;\mathrm{<figure-callout\; id="1"\; label="K"\; filenames="HDA0000399144580000022.png"\; state="\{\{state\}\}">K</figure-callout>}}{\mathrm{\&Pi;}}\left\{\begin{array}{c}1,p_{A,k}=p_{B,k}\\ 0,p_{A,k}\&NotEqual;p_{B,k}\end{array}\right.\end{array}\right)*\left(\frac{\underset{i\&Element;I}{\mathrm{\&Sigma;}}{p}_{A,i}*p_{B,i}}{\sqrt{\underset{i\&Element;I}{\mathrm{\&Sigma;}}{p}_{A,\mathrm{<figure-callout\; id="2"\; label="i"\; filenames="HDA0000399144580000022.png"\; state="\{\{state\}\}">i</figure-callout>}}^2}*\sqrt{\underset{i\&Element;I}{\mathrm{\&Sigma;}}{p}_{B,i}^2}}\right)$

For the similarity between cluster, suppose that cluster parameter is not as determinant attribute, its similarity is calculated and can be reduced to so:

$\mathrm{sim}(\mathrm{ClusterA},\mathrm{ClusterB})=\frac{\underset{i\&Element;I}{\mathrm{\&Sigma;}}{p}_{A,i}*p_{B,i}}{\sqrt{\underset{i\&Element;I}{\mathrm{\&Sigma;}}{p}_{A,\mathrm{<figure-callout\; id="2"\; label="i"\; filenames="HDA0000399144580000022.png"\; state="\{\{state\}\}">i</figure-callout>}}^2}*\sqrt{\underset{i\&Element;I}{\mathrm{\&Sigma;}}{p}_{B,\mathrm{<figure-callout\; id="2"\; label="i"\; filenames="HDA0000399144580000022.png"\; state="\{\{state\}\}">i</figure-callout>}}^2}}$

Wherein I is cluster parameter sets, p _a,ifor i the property value of sign cluster A, p _b,ifor j the property value of sign operation B, it is a simple computation model based on cosine similarity.

As a preferred embodiment of the present invention, Fig. 4 is the preferably implementing procedure figure of the distributed computing framework parameter optimization that provides of the present embodiment, and details are as follows:

S401, obtains parallel computation framework (distributed computing framework, the distributed computing framework) operation when submit;

S402, judges the whether self-defined distributed computing framework parameter of described distributed computing framework operation user, is S403, otherwise carries out S404;

S403, submits distributed computing framework operation to according to user-specified parameters;

S404 retrieves similar operation from history run storehouse;

Whether S405, there is operation of the same type, is to carry out S405, otherwise carries out S406;

S406, Uses Defaults and submits distributed computing framework operation to;

S407, selects a distributed computing framework operation that scoring is the highest;

S408, submits distributed computing framework operation to according to optimized parameter;

S409, distributed computing framework job run finishes, and collects distributed computing framework configuration and running log;

S409, to current distributed computing framework operation scoring, puts into database by result.

With reference to figure 5, Fig. 5 is the structural representation of a kind of parameter optimization device of providing of the embodiment of the present invention, for convenience of explanation, only shows the part relevant to the present embodiment, and details are as follows:

Acquiring unit 51, for obtaining the distributed computing framework operation when submit;

The first retrieval unit 52, for the distributed computing framework history data storehouse setting up in advance, retrieve the historical distributed computing framework operation similar with described distributed computing framework operation, described distributed computing framework history data storehouse comprises execution information and the configuration parameter of historical distributed computing framework operation;

The second retrieval unit 53, for the distributed computing framework operation similar, retrieves the historical distributed computing framework operation similar to described distributed computing framework operation;

Dispensing unit 54, for according to the configuration parameter of similar historical distributed computing framework operation, is optimized configuration to the configuration parameter of described distributed computing framework operation.

Further, in this device, described the first retrieval unit, comprising:

Check subelement, for checking whether described distributed computing framework operation exists the distributed computing framework operational factor of appointment;

Carry out subelement, for when there is not the distributed computing framework operational factor of appointment in described distributed computing framework operation, described in carrying out, in distributed computing framework Historical Jobs runtime database, retrieve the step of the historical distributed computing framework operation similar with described distributed computing framework operation.

Further, in this device, described dispensing unit, also for the historical distributed computing framework operation similar, obtain the highest historical distributed computing framework operation of scoring, use the configuration parameter of the highest historical distributed computing framework operation of scoring as the configuration parameter of described distributed computing framework operation, the configuration parameter of described distributed computing framework operation is optimized to configuration.

Further, in this device, also comprise:

Collector unit, for when described distributed computing framework Job execution is complete, collects described distributed computing framework Job execution information and configuration parameter;

Scoring unit, for described distributed computing framework Job execution information is marked, and is saved in described distributed computing framework operation in described distributed computing framework Historical Jobs database.

Further, in this device, described scoring unit, comprising:

Obtain subelement, for obtaining time loss and the space consuming of described distributed computing framework job run;

Generate subelement, for according to the time loss of described distributed computing framework job run and space consuming, generate time loss cost and the space consuming cost of described distributed computing framework job run;

Scoring subelement, for according to the Rating Model of the distributed computing framework operation of setting up in advance, and the time loss cost of described distributed computing framework job run and space consuming cost are marked to distributed computing framework job run.

Further, in this device, described scoring unit, also comprises:

Set up subelement, for setting up Rating Model;

Described Rating Model is

Wherein, F _jobscoring when (τ, υ) represents a job run, τ, υ represents respectively corresponding time loss cost, space consuming cost, τ during Job execution scoring, the function of υ.α, β is the weight of the scoring of time loss cost and the scoring of space consuming cost, i is the sequence number of the middle property value of time loss cost, τ _icorresponding to i property value of time loss cost, j is the sequence number of the middle property value of space consuming cost, υ _jexpression is corresponding to j property value of space consuming cost,

the weight of corresponding above-mentioned two attributes respectively, parameter alpha wherein, β is for regulating setup times to consume cost and space consuming cost which more preferably;

object be in order to eliminate the different difference of each cost attribute value magnitude, by arranging

carry out normalized calculating F _job(τ, υ).

Further, in this device, described scoring unit, also comprises:

Preserve subelement, for adopting tree-like storage mode, described distributed computing framework operation is saved in described distributed computing framework Historical Jobs database.

Further, in this device, the first retrieval unit, comprising:

The first retrieval subelement, for the distributed computing framework history data storehouse setting up in advance, adopts search tree node mode, retrieves the historical distributed computing framework operation similar with described distributed computing framework operation.

Further, in this device, the second retrieval unit, comprising:

Whether the second retrieval subelement, equate with the determinant attribute of historical distributed computing framework operation for retrieving described distributed computing framework operation;

Similar subelement, for when the determinant attribute of described distributed computing framework operation is when determinant attribute in historical distributed computing framework operation equates, represent that described historical distributed computing framework operation is the historical distributed computing framework operation similar to described distributed computing framework operation.

Further, in this device, the second retrieval unit, also comprises:

Obtain subelement, for obtaining the attribute field of described distributed computing framework operation and historical distributed computing framework operation, described attribute field comprises determinant attribute set and non-key community set;

First generates subelement, similarity model and the described attribute field of the distributed computing framework operation of setting up in advance for basis, generate respectively the similarity of operation and the similarity of cluster environment information, the similarity of described operation is the similarity of described distributed computing framework operation and historical distributed computing framework operation, and the similarity of described cluster environment information is that the similarity of described distributed computing framework operation and historical distributed computing framework operation is respectively in the similarity of described cluster environment information;

Second generates subelement, for according to comprehensive similarity model and the weighted strategy of the distributed computing framework operation of setting up in advance, generates the comprehensive similarity of described distributed computing framework operation and historical distributed computing framework operation.

Further, in this device, first generates subelement, also comprises:

First sets up subelement, for setting up the similarity model of distributed computing framework operation,

Described similarity model is:

$\mathrm{sim}(A,B)=\left(\begin{array}{c}\underset{k\&Element;\mathrm{<figure-callout\; id="1"\; label="K"\; filenames="HDA0000399144580000022.png"\; state="\{\{state\}\}">K</figure-callout>}}{\mathrm{\&Pi;}}\left\{\begin{array}{c}1,p_{A,k}=p_{B,k}\\ 0,p_{A,k}\&NotEqual;p_{B,k}\end{array}\right.\end{array}\right)*\left(\frac{\underset{i\&Element;I}{\mathrm{\&Sigma;}}{p}_{A,i}*p_{B,i}}{\sqrt{\underset{i\&Element;I}{\mathrm{\&Sigma;}}{p}_{A,\mathrm{<figure-callout\; id="2"\; label="i"\; filenames="HDA0000399144580000022.png"\; state="\{\{state\}\}">i</figure-callout>}}^2}*\sqrt{\underset{i\&Element;I}{\mathrm{\&Sigma;}}{p}_{B,i}^2}}\right)$

Wherein K is determinant attribute set, and k is the sequence number of the middle determinant attribute of determinant attribute set, and I is non-key community set, the sequence number of non-key attribute during i is non-key community set, p _a,kk the determinant attribute that represents operation A, p _a,ifor i the non-key attribute of sign operation A, p _b,kfor k the determinant attribute of sign operation B, p _b,ii the non-key attribute for sign operation B.

Further, in this device, second generates subelement, also comprises:

Second sets up subelement, for setting up the comprehensive similarity model of distributed computing framework operation,

Described comprehensive similarity model is:

sim(A,B)＝α×sim(JobA,JobB)+β×sim(ClusterA,ClusterB)

Wherein sim (JobA, JobB) represents operation A, the similarity of B, and sim (ClusterA, ClusterB) represents operation A, B is respectively in the similarity of described cluster environment information.Sim (A, B) represents operation A, and between B, with reference to the similarity of cluster environment information, α is the first weight parameter in weighted strategy, and β is the second weight parameter in weighted strategy.

As a preferred embodiment of the present invention, a kind of distributed computing framework Parameter Optimization System, the client of the distributed computing framework operation of parameter optimization device, submission and management of computing node, wherein, between the client of described distributed computing framework operation and described management of computing node, by described parameter optimization device, connect.

With reference to figure 6, Fig. 6 is preferably network architecture diagram of the distributed computing framework Parameter Optimization System that provides of the present embodiment.

Wherein, parameter automatic optimization configuration module is submitted between client and distributed computing framework Master in distributed computing framework operation.When user submits task to by distributed computing framework client, parameter optimization configuration module can carry out to this task the parameter configuration of a series of robotizations, then the management of computing node M aster that is submitted to distributed computing framework is upper, then by Master, this distributed computing framework operation is distributed on administered a plurality of Worker nodes and executed the task.

Wherein, distributed computing framework historic task Runtime Library has been deposited the distributed computing framework job information of having moved on cluster, and its Storage Format can be in this locality, HDFS or database, while retrieving similar task for parameter optimization configuration module.

With reference to figure 7, Fig. 7 is the structural representation of a kind of parameter optimization device of providing of the embodiment of the present invention, and the specific embodiment of the invention does not limit the specific implementation of described parameter optimization device, and described parameter optimization device 700, comprising:

Processor (English: processor) 701, communication interface (English: Communications Interface) 702, storer (English: memory) 103, bus 704.

Processor 701, communication interface 702, storer 703 completes mutual communication by bus 704.

Communication interface 702, for communicating with other communication facilitiess;

Processor 701, for executive routine.

Particularly, program can comprise program code, and described program code comprises computer-managed instruction.

Processor 701 may be a central processing unit (English: central processing unit, abbreviation: CPU.

Storer 703, for storage program.Its Program, for obtaining parallel computation framework (distributed computing framework, the distributed computing framework) operation when submit, checks whether described distributed computing framework operation exists the distributed computing framework operational factor of appointment;

For when there is not the distributed computing framework operational factor of appointment in described distributed computing framework operation, in the distributed computing framework history data storehouse of setting up in advance, retrieve the historical distributed computing framework operation similar with described distributed computing framework operation, described distributed computing framework history data storehouse comprises execution information and the configuration parameter of historical distributed computing framework operation; For in similar distributed computing framework operation, retrieve the historical distributed computing framework operation similar to described distributed computing framework operation; For according to the configuration parameter of similar historical distributed computing framework operation, the configuration parameter of described distributed computing framework operation is optimized to configuration.

The foregoing is only preferred embodiment of the present invention, not in order to limit the present invention, all any modifications of doing within the spirit and principles in the present invention, be equal to and replace and improvement etc., within all should being included in protection scope of the present invention.

Claims (25)

1. a method for distributed computing framework parameter optimization, is characterized in that, comprising:

Obtain the distributed computing framework operation when submit;

In distributed computing framework Historical Jobs runtime database, retrieve the historical distributed computing framework operation similar with described distributed computing framework operation, described distributed computing framework Historical Jobs runtime database comprises execution information and the configuration parameter of historical distributed computing framework operation;

In similar distributed computing framework operation, retrieve the historical distributed computing framework operation similar to described distributed computing framework operation;

Configuration parameter according to similar historical distributed computing framework operation, is optimized configuration to the configuration parameter of described distributed computing framework operation.

2. the method for claim 1, is characterized in that, described in distributed computing framework Historical Jobs runtime database, and before retrieving the historical distributed computing framework operation similar with described distributed computing framework operation, described method also comprises:

Check whether described distributed computing framework operation exists the distributed computing framework operational factor of appointment;

When there is not the distributed computing framework operational factor of appointment in described distributed computing framework operation, described in carrying out, in distributed computing framework Historical Jobs runtime database, retrieve the step of the historical distributed computing framework operation similar with described distributed computing framework operation.

3. the method for claim 1, is characterized in that, describedly according to the configuration parameter of similar historical distributed computing framework operation, the configuration parameter of described distributed computing framework operation is optimized to configuration, comprising:

In similar historical distributed computing framework operation, obtain the highest historical distributed computing framework operation of scoring, use the configuration parameter of the highest historical distributed computing framework operation of scoring as the configuration parameter of described distributed computing framework operation, the configuration parameter of described distributed computing framework operation is optimized to configuration; Or

In similar historical distributed computing framework operation, obtaining marks is greater than a plurality of distributed computing framework operations of pre-set threshold value, according to the similarity degree of a plurality of distributed computing framework operations, the configuration parameter of a plurality of distributed computing framework operations of weighted calculation, the configuration parameter that use weighted calculation obtains, as the configuration parameter of described distributed computing framework operation, is optimized configuration to the configuration parameter of described distributed computing framework operation.

4. the method for claim 1, is characterized in that, also comprises:

When described distributed computing framework Job execution is complete, collect described distributed computing framework Job execution information and configuration parameter;

Described distributed computing framework Job execution information is marked, and described distributed computing framework operation is saved in described distributed computing framework Historical Jobs database.

5. method as claimed in claim 4, is characterized in that, described distributed computing framework Job execution information is marked, and is specially:

Obtain time loss and the space consuming of described distributed computing framework job run;

According to the time loss of described distributed computing framework job run and space consuming, generate time loss cost and the space consuming cost of described distributed computing framework job run;

According to the Rating Model of the distributed computing framework operation of setting up in advance, and the time loss cost of described distributed computing framework job run and space consuming cost are marked to distributed computing framework job run.

6. method as claimed in claim 4, it is characterized in that, the Rating Model of the distributed computing framework operation of setting up in advance in described basis, and before the time loss cost of described distributed computing framework job run and space consuming cost mark to distributed computing framework job run, comprising:

Set up Rating Model;

Described Rating Model is

Wherein, F _jobscoring when (τ, υ) represents a job run, τ, υ represents respectively corresponding time loss cost, space consuming cost, τ during Job execution scoring, the function of υ.α, β is the weight of the scoring of time loss cost and the scoring of space consuming cost, i is the sequence number of the middle property value of time loss cost, τ _icorresponding to i property value of time loss cost, j is the sequence number of the middle property value of space consuming cost, υ _jexpression is corresponding to j property value of space consuming cost, the weight of corresponding above-mentioned two attributes respectively, parameter alpha wherein, β is for regulating setup times to consume cost and space consuming cost which more preferably;

object be in order to eliminate the different difference of each cost attribute value magnitude, by arranging carry out normalized calculating F _job(τ, υ).

7. method as claimed in claim 4, is characterized in that, described described distributed computing framework operation is saved in described distributed computing framework Historical Jobs database, is specially:

Adopt tree-like storage mode, described distributed computing framework operation is saved in described distributed computing framework Historical Jobs database.

8. the method for claim 1, is characterized in that, in the distributed computing framework history data storehouse of setting up in advance, retrieves the historical distributed computing framework operation similar with described distributed computing framework operation, comprising:

In the distributed computing framework history data storehouse of setting up in advance, adopt search tree node mode, retrieve the historical distributed computing framework operation similar with described distributed computing framework operation.

9. the method for claim 1, is characterized in that, the historical distributed computing framework operation that described retrieval is similar to described distributed computing framework operation, comprising:

Whether retrieve described distributed computing framework operation equates with the determinant attribute in historical distributed computing framework operation;

Determinant attribute in described distributed computing framework operation when determinant attribute in historical distributed computing framework operation equates, represents that described historical distributed computing framework operation is the historical distributed computing framework operation similar to described distributed computing framework operation.

10. method as claimed in claim 9, is characterized in that, the historical distributed computing framework operation that described retrieval is similar to described distributed computing framework operation, also comprises:

Obtain the attribute field of described distributed computing framework operation and historical distributed computing framework operation, described attribute field comprises determinant attribute set and non-key community set;

According to similarity model and the described attribute field of the distributed computing framework operation of setting up in advance, generate respectively the similarity of operation and the similarity of cluster environment information, the similarity of described operation is the similarity of described distributed computing framework operation and historical distributed computing framework operation, and the similarity of described cluster environment information is that the similarity of described distributed computing framework operation and historical distributed computing framework operation is respectively in the similarity of described cluster environment information;

According to comprehensive similarity model and the weighted strategy of the distributed computing framework operation of setting up in advance, generate the comprehensive similarity of described distributed computing framework operation and historical distributed computing framework operation.

11. methods as claimed in claim 10, is characterized in that, the similarity model of the distributed computing framework operation of setting up in advance in described basis and described attribute field, before generating respectively the similarity of operation and the similarity of cluster environment information, comprising:

Set up the similarity model of distributed computing framework operation,

Described similarity model is:

$\mathrm{sim}(A,B)=\left(\begin{array}{c}\underset{k\&Element;K}{\mathrm{\&Pi;}}\left\{\begin{array}{c}1,p_{A,k}=p_{B,k}\\ 0,p_{A,k}\&NotEqual;p_{B,k}\end{array}\right.\end{array}\right)*\left(\frac{\underset{i\&Element;I}{\mathrm{\&Sigma;}}{p}_{A,i}*p_{B,i}}{\sqrt{\underset{i\&Element;I}{\mathrm{\&Sigma;}}{p}_{A,i}^2}*\sqrt{\underset{i\&Element;I}{\mathrm{\&Sigma;}}{p}_{B,i}^2}}\right)$

Wherein K is determinant attribute set, and k is the sequence number of the middle determinant attribute of determinant attribute set, and I is non-key community set, the sequence number of non-key attribute during i is non-key community set, p _a,kk the determinant attribute that represents operation A, p _a,ifor i the non-key attribute of sign operation A, p _b,kfor k the determinant attribute of sign operation B, p _b,ii the non-key attribute for sign operation B.

12. methods as claimed in claim 10, it is characterized in that, comprehensive similarity model and the weighted strategy of the distributed computing framework operation of setting up in advance in basis, before generating the comprehensive similarity of described distributed computing framework operation and historical distributed computing framework operation, comprising:

Set up the comprehensive similarity model of distributed computing framework operation,

Described comprehensive similarity model is:

sim(A,B)＝α×sim(JobA,JobB)+β×sim(ClusterA,ClusterB)

Sim (JobA wherein, JobB) represent operation A, the similarity of B, sim (ClusterA, ClusterB) represents operation A, B is respectively in the similarity of described cluster environment information, sim (A, B) represents operation A, between B with reference to the similarity of cluster environment information, α is the first weight parameter in weighted strategy, and β is the second weight parameter in weighted strategy.

13. 1 kinds of distributed computing framework parameter optimization devices, is characterized in that, comprising:

Acquiring unit, for obtaining the distributed computing framework operation when submit;

The first retrieval unit, in the distributed computing framework history data storehouse of setting up in advance, retrieve the historical distributed computing framework operation similar with described distributed computing framework operation, described distributed computing framework history data storehouse comprises execution information and the configuration parameter of historical distributed computing framework operation;

The second retrieval unit, for the distributed computing framework operation similar, retrieves the historical distributed computing framework operation similar to described distributed computing framework operation;

Dispensing unit, for according to the configuration parameter of similar historical distributed computing framework operation, is optimized configuration to the configuration parameter of described distributed computing framework operation.

14. parameter optimization devices as claimed in claim 13, is characterized in that, described the first retrieval unit, comprising:

Check subelement, for checking whether described distributed computing framework operation exists the distributed computing framework operational factor of appointment;

Carry out subelement, for when there is not the distributed computing framework operational factor of appointment in described distributed computing framework operation, described in carrying out, in distributed computing framework Historical Jobs runtime database, retrieve the step of the historical distributed computing framework operation similar with described distributed computing framework operation.

15. parameter optimization devices as claimed in claim 13, it is characterized in that, described dispensing unit, also for the historical distributed computing framework operation similar, obtain the highest historical distributed computing framework operation of scoring, use the configuration parameter of the highest historical distributed computing framework operation of scoring as the configuration parameter of described distributed computing framework operation, to the configuration parameter of described distributed computing framework operation be optimized configuration or

In similar historical distributed computing framework operation, obtaining marks is greater than a plurality of distributed computing framework operations of pre-set threshold value, according to the similarity degree of a plurality of distributed computing framework operations, the configuration parameter of a plurality of distributed computing framework operations of weighted calculation, the configuration parameter that use weighted calculation obtains, as the configuration parameter of described distributed computing framework operation, is optimized configuration to the configuration parameter of described distributed computing framework operation.

16. parameter optimization devices as claimed in claim 13, is characterized in that, also comprise:

Collector unit, for when described distributed computing framework Job execution is complete, collects described distributed computing framework Job execution information and configuration parameter;

Scoring unit, for described distributed computing framework Job execution information is marked, and is saved in described distributed computing framework operation in described distributed computing framework Historical Jobs database.

17. parameter optimization devices as claimed in claim 16, is characterized in that, described scoring unit, comprising:

Obtain subelement, for obtaining time loss and the space consuming of described distributed computing framework job run;

Generate subelement, for according to the time loss of described distributed computing framework job run and space consuming, generate time loss cost and the space consuming cost of described distributed computing framework job run;

Scoring subelement, for according to the Rating Model of the distributed computing framework operation of setting up in advance, and the time loss cost of described distributed computing framework job run and space consuming cost are marked to distributed computing framework job run.

18. parameter optimization devices as claimed in claim 16, is characterized in that, described scoring unit, also comprises:

Set up subelement, for setting up Rating Model;

Described Rating Model is

Wherein, F _jobscoring when (τ, υ) represents a job run, τ, υ represents respectively corresponding time loss cost, space consuming cost, τ during Job execution scoring, the function of υ.α, β is the weight of the scoring of time loss cost and the scoring of space consuming cost, i is the sequence number of the middle property value of time loss cost, τ _icorresponding to i property value of time loss cost, j is the sequence number of the middle property value of space consuming cost, υ _jexpression is corresponding to j property value of space consuming cost,

the weight of corresponding above-mentioned two attributes respectively, parameter alpha wherein, β is for regulating setup times to consume cost and space consuming cost which more preferably;

object be in order to eliminate the different difference of each cost attribute value magnitude, by arranging carry out normalized calculating F _job(τ, υ).

19. parameter optimization devices as claimed in claim 16, is characterized in that, described scoring unit, also comprises:

Preserve subelement, for adopting tree-like storage mode, described distributed computing framework operation is saved in described distributed computing framework Historical Jobs database.

20. parameter optimization devices as claimed in claim 13, is characterized in that, described the first retrieval unit, comprising:

The first retrieval subelement, for the distributed computing framework history data storehouse setting up in advance, adopts search tree node mode, retrieves the historical distributed computing framework operation similar with described distributed computing framework operation.

21. parameter optimization devices as claimed in claim 13, is characterized in that, described the second retrieval unit, comprising:

Whether the second retrieval subelement, equate with the determinant attribute of historical distributed computing framework operation for retrieving described distributed computing framework operation;

Similar subelement, for when the determinant attribute of described distributed computing framework operation is when determinant attribute in historical distributed computing framework operation equates, represent that described historical distributed computing framework operation is the historical distributed computing framework operation similar to described distributed computing framework operation.

22. parameter optimization devices as claimed in claim 21, is characterized in that, described the second retrieval unit, also comprises:

Obtain subelement, for obtaining the attribute field of described distributed computing framework operation and historical distributed computing framework operation, described attribute field comprises determinant attribute set and non-key community set;

First generates subelement, similarity model and the described attribute field of the distributed computing framework operation of setting up in advance for basis, generate respectively the similarity of operation and the similarity of cluster environment information, the similarity of described operation is the similarity of described distributed computing framework operation and historical distributed computing framework operation, and the similarity of described cluster environment information is that the similarity of described distributed computing framework operation and historical distributed computing framework operation is respectively in the similarity of described cluster environment information;

Second generates subelement, for according to comprehensive similarity model and the weighted strategy of the distributed computing framework operation of setting up in advance, generates the comprehensive similarity of described distributed computing framework operation and historical distributed computing framework operation.

23. parameter optimization devices as claimed in claim 22, is characterized in that, described first generates subelement, also comprises:

First sets up subelement, for setting up the similarity model of distributed computing framework operation,

Described similarity model is:

$\mathrm{sim}(A,B)=\left(\begin{array}{c}\underset{k\&Element;K}{\mathrm{\&Pi;}}\left\{\begin{array}{c}1,p_{A,k}=p_{B,k}\\ 0,p_{A,k}\&NotEqual;p_{B,k}\end{array}\right.\end{array}\right)*\left(\frac{\underset{i\&Element;I}{\mathrm{\&Sigma;}}{p}_{A,i}*p_{B,i}}{\sqrt{\underset{i\&Element;I}{\mathrm{\&Sigma;}}{p}_{A,i}^2}*\sqrt{\underset{i\&Element;I}{\mathrm{\&Sigma;}}{p}_{B,i}^2}}\right)$

Wherein K is determinant attribute set, and k is the sequence number of the middle determinant attribute of determinant attribute set, and I is non-key community set, the sequence number of non-key attribute during i is non-key community set, p _a,kk the determinant attribute that represents operation A, p _a,ifor i the non-key attribute of sign operation A, p _b,kfor k the determinant attribute of sign operation B, p _b,ii the non-key attribute for sign operation B.

24. parameter optimization devices as claimed in claim 22, is characterized in that, described second generates subelement, also comprises:

Second sets up subelement, for setting up the comprehensive similarity model of distributed computing framework operation,

Described comprehensive similarity model is:

sim(A,B)＝α×sim(JobA,JobB)+β×sim(ClusterA,ClusterB)

Wherein sim (JobA, JobB) represents operation A, the similarity of B, and sim (ClusterA, ClusterB) represents operation A, B is respectively in the similarity of described cluster environment information.Sim (A, B) represents operation A, and between B, with reference to the similarity of cluster environment information, α is the first weight parameter in weighted strategy, and β is the second weight parameter in weighted strategy.

25. 1 kinds of distributed computing framework Parameter Optimization Systems, it is characterized in that, the client and the management of computing node that comprise the distributed computing framework operation of parameter optimization device described in claim 13 to 24 any one, submission, wherein, between the client of described distributed computing framework operation and described management of computing node, by described parameter optimization device, connect.

Images