CN106021495B - A kind of task parameters optimization method of distributed iterative computing system - Google Patents

Abstract

The invention relates to a task parameter optimization method in a distributed iterative computing system, belonging to the technical field of distributed data processing. This method first collects the running data of the historical tasks in the distributed iterative computing system, and builds the historical database; when optimizing the task parameters, it filters the significantly irrelevant running data in the historical database according to the constraints; The operating data in the database and the operating data after the first filtering are calculated for the similarity of the directed acyclic graph, and the operating data whose similarity is lower than a certain threshold is filtered twice; finally, the results after the two filtering are calculated and sorted , and use the task parameters corresponding to the sorted running data as the task parameter optimization results. The invention can automatically optimize the task parameters of the distributed iterative computing system, is a plug-and-play self-adaptive optimization method, and can significantly reduce the threshold for users to use the distributed iterative computing system.

Description

A Method of Task Parameter Optimization for Distributed Iterative Computing System

technical field

The invention belongs to the technical field of distributed data processing, in particular to a task parameter optimization method in a distributed iterative computing system.

Background technique

Using distributed iterative computing systems to process large-scale data sets has become the main practice of data processing. Compared with traditional stand-alone data processing solutions, distributed iterative computing systems that are now popular and widely used, such as Apache Spark, use multiple machines to divide data, thereby greatly increasing the scale of data processing. Moreover, multiple machines participate in the data processing process, which increases the parallel number of data processing and speeds up the processing speed of large-scale data.

Despite the above advantages, the normal operation of a distributed iterative computing system task requires reasonable task parameters. Unreasonable task parameters will reduce the processing speed of the task in the distributed iterative computing system. Reasonable task parameters can increase the parallelism of task data processing in the distributed iterative computing system, reduce network transmission overhead and reduce scheduling time overhead, thus speeding up task processing. There are dozens of task parameters involved in the distributed iterative computing system, and there are intricate relationships among the task parameters. The configuration of task parameters brings additional overhead to developers, and task parameters determined manually may not necessarily achieve good operating performance.

There are many task parameters in the distributed iterative computing system and it is difficult to configure them well. One of the problems that arises from this is whether to optimize the task parameters in the distributed iterative computing system. At present, the optimization of task parameters in distributed iterative computing systems mainly relies on the experience of engineers to make decisions. But this optimization method is too subjective, experienced engineers can often get better task parameters, but inexperienced engineers can't get better task parameters.

Contents of the invention

The purpose of the present invention is to propose a method for optimizing task parameters of a distributed iterative computing system in view of the problem that there are many task parameters in the existing distributed iterative computing system and it is not easy to configure well. The invention can automatically optimize the task parameters of the distributed iterative computing system, is a plug-and-play self-adaptive optimization method, and can significantly reduce the threshold for users to use the distributed iterative computing system.

The task parameter optimization method in the distributed iterative computing system proposed by the present invention firstly collects the operation data of the historical tasks in the distributed iterative computing system, and builds a historical database; when optimizing the task parameters, the historical database is significantly irrelevant according to the constraints Filter the operation data once; then perform DAG similarity calculation on the operation data in the historical database corresponding to the task to be optimized and the operation data after the first filter, and perform secondary calculation on the operation data whose similarity is lower than a certain threshold. Finally, the results after the two filters are calculated and sorted, and the task parameters corresponding to the sorted running data are used as the task parameter optimization results. The method specifically includes the following steps:

(1) Obtain the operation data of each historical task from the distributed iterative computing system, save the operation data of each historical task in the historical database, and each item of data in the historical database represents the operational data of a historical task;

(2) according to user request, carry out task parameter optimization to the task in the distributed iterative computing system, suppose the running data of the historical task identical with this task that finds out from historical database be J _src ;

(3) Find out from the historical database the historical task operation data that meets all hardware resource constraints to form the data set S _hardware ;

(4) Find out the relative difference between the total size of the input data and the total size of the input data of J _src obtained in the step (2) from all the operating data of _Shardware obtained in step (3) is smaller than the set input data size difference The operating data of the threshold constitutes the data set S _datasize ;

(5) in all operating data of the S _datasize that step (4) obtains, find out directed acyclic graph and the directed acyclic graph of J _src in scale similar operating data to form data set S _dag ;

(6) the similarity between the directed acyclic graph of each operation data and the directed acyclic graph of J _src in the S _dag that calculation step (5) obtains, and set the similarity threshold;

(7) Traversing the calculation results of step (6), discarding the operating data whose similarity between DAG in S _dag and DAG in J _src is lower than the set similarity threshold, and setting The data set is S _sim ;

(8) to each operation data in the S _sim that step (7) obtains according to the formula The calculation results are sorted from high to low, and only the running data of the first n items in the sorted calculation results are kept, n is a positive integer; where time _dst represents the running time of J _dst ; the data composed of the sorted results is set The set is S _rank ;

(9) the task parameter of each piece of operating data in the S _rank that step (8) obtains is displayed to the user on the graphic display interface, and the task parameter optimization process ends;

(10) When the user requests to optimize the task of the distributed iterative computing system again, return to step (2).

The optimization method of task parameter in the distributed iterative computing system that the present invention proposes, its characteristic and beneficial effect are:

1. The method of the present invention enables the computer to undertake the work of optimizing task parameters in the distributed iterative computing system, reducing the workload of the user when using the distributed iterative computing system. In the case that the user is not familiar with the distributed iterative computing system, it can provide the user with more effective task parameters, reducing the pressure of using the distributed iterative computing system.

2. This method combines the empirical rules of system optimization and the optimization method based on similarity search, which improves the reliability and usability of task parameter optimization.

3. The method of the present invention can adapt to changes in the system and be changed, and is an adaptive tuning method. During the operation of the distributed iterative computing system, this method will continuously collect the operating data generated by the system, so that the amount of data in the historical database becomes larger and larger, and the types of tasks that can be covered are more and more. The increase in the amount of data will make the task parameter optimization results better as the system runs.

4. The method of the present invention does not need to modify the original distributed iterative computing system, and belongs to the plug-and-play method.

Description of drawings

Fig. 1 is an overall flowchart of the task parameter optimization method in the distributed iterative computing system proposed by the present invention.

detailed description

The present invention proposes a task parameter optimization method in a distributed iterative computing system, which will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

The present invention proposes a method for optimizing task parameters in a distributed iterative computing system. The overall process is shown in Figure 1. This method first collects the operating data of historical tasks in the distributed iterative computing system and builds a historical database; when optimizing task parameters, According to the constraint conditions, the significantly irrelevant operating data in the historical database is filtered once; then the similarity calculation of the directed acyclic graph is performed between the operating data in the historical database corresponding to the task to be optimized and the filtered operating data, and the similarity The operating data whose degree is lower than a certain threshold is filtered twice; finally, the results after the two filtering are calculated and sorted, and the task parameters corresponding to the sorted operating data are used as the task parameter optimization results. The method specifically includes the following steps:

(1) Obtain the running data of each historical task from the distributed iterative computing system. The running data of a historical task includes task parameters, hardware resource information (total memory, the number of CPU cores that can be run, and the number of machine nodes), and the total number of input data. Size and the corresponding directed acyclic graph (during the execution of the task, the task is divided into multiple subtasks, and the directed acyclic graph is used to reflect the dependencies between the subtasks; the nodes on the directed acyclic graph represent Subtasks, the edges on the directed acyclic graph represent the sequence relationship between the subtasks, and the labels on the nodes represent the specific names of the subtasks); then save the running data of each historical task in the historical database, and the history Each item of data in the database represents the running data of a historical task;

(2) According to the user's request, the task in the distributed iterative computing system is optimized for task parameters, and the operating data of finding out the same historical task as the task from the historical database is J _src ;

(3) Find out from the historical database the historical task operation data that meets all hardware resource constraints to form a data set S _hardware ; the constraints include: the relative difference in value between the overall memory of the operating data and the overall memory of J _src is smaller than the set memory The difference threshold (the threshold set in this embodiment is 30%); the relative difference in value between the number of executable CPU cores of the running data and the number of executable CPU cores of J _src is less than the set core number difference threshold (set in this embodiment) The threshold of the machine node number is 30%); the relative difference in value between the number of machine nodes of the running data and the number of machine nodes of J _src is less than the set machine node number difference threshold (the threshold set in this embodiment is 30%);

(4) Find out the relative difference between the total size of the input data and the total size of the input data of J _src obtained in step (2) in value (in megabytes) from all the operating data of the _Shardware obtained in step (3) is smaller than the set value The operating data of the determined input data size difference threshold, form the data set S _datasize (the threshold set in this embodiment is 30%);

(5) Find out the directed acyclic graph and the directed acyclic graph of J _src in all operating data of S _datasize that step (4) obtains and form the data collection S _dag of operating data similar in scale; Two directed The similar scale of the acyclic graph includes the following two conditions: first, the relative difference in value of the number of nodes on the two directed acyclic graphs is smaller than the set difference threshold of the number of directed acyclic graph nodes (set in this embodiment The threshold is 30%); second, the relative difference in value of the number of edges on the two directed acyclic graphs is smaller than the set directed acyclic graph edge number difference threshold (the threshold set in this embodiment is 30% );

(6) Calculate the similarity between the directed acyclic graph of each item of operating data in S _dag obtained in step (5) and the directed acyclic graph of J _src , and set the similarity threshold; the specific calculation process steps are as follows:

(6-1) Let J _dst be the running data of any item in S _dag , calculate the similarity between the directed acyclic graph of J _dst and the directed acyclic graph of J _src , and define the directed acyclic graph of J _src as G _src = (N _src , E _src , L _src ), where N _src represents the node set in the directed acyclic graph G _src , E _src represents the edge set in the directed acyclic graph G _src , and L _src represents the directed acyclic graph G src A collection of labels on each node in the ring graph G _src ; define the directed acyclic graph of J _dst as G _dst = (N _dst , E _dst , L _dst ), where N _dst represents the directed acyclic graph G The set of nodes in _dst , E _dst represents the set of edges in the directed acyclic graph G _dst , and L _dst represents the set formed by the labels on each node in the directed acyclic graph G _dst ;

(6-2) The similarity between the DAGs of J _dst and J _src is defined by the following formula:

In the formula, sim(G _src ,G _dst ) represents the similarity between the DAG of J _dst and the DAG of J _src , and the value range is [0,1];

skipN(G _src ,G _dst ) represents the sum of the number of nodes added or deleted by G _src and G _dst in the process of making G _src and G _dst equal;

skipE(G _src ,G _dst ) represents the sum of the number of edges added or deleted by G _src and G _dst in the process of making G _src and G _dst equal;

n _src and n _dst respectively represent any node in the directed acyclic graph G _src and any node in the directed acyclic graph G _dst ;

l _src and l _dst represent the labels corresponding to nodes n _src and n _dst respectively;

edit(l _src , l _dst ) represents the edit distance between the two labels l _src and l _dst , that is, the minimum number of editing operations required in the process of converting the label l _src into the label l _dst (allowed editing operations include: converting l Replace a character in _src with another character; delete a character in l _src ; add a character to l _src );

|l _src | and |l _dst | represent the string lengths of label l _src and label l _dst respectively;

(6-3) Repeat steps (6-1) to (6-2), calculate the similarity between the directed acyclic graph of each operation data in S _dag and the directed acyclic graph of J _src ;

(7) Traversing the calculation result of step (6), discarding the DAG similarity between DAG and J _src in S _dag is lower than the operating data of the set similarity threshold (the threshold set in this embodiment is 0.3), let the data set composed of remaining operating data be S _sim ;

(8) to each operation data in the S _sim that step (7) obtains according to the formula The calculation results are sorted from high to low, and only the running data of the first n items in the sorted calculation results are retained, and n is a positive integer (the number of specific retained results is determined according to the actual situation, and this embodiment retains the calculation of the first 10 items after sorting result); in the formula, time _dst represents the running time of J _dst ; this formula takes two factors into consideration, one is the similarity between J _dst and J _src on the directed acyclic graph, and the other is the history corresponding to J _dst The running time of the task; sorting under this evaluation index, the higher the similarity on the directed acyclic graph, or the shorter the running time of the historical task corresponding to J _dst , the higher the running data in the sorting result; set the sorting The data set composed of the obtained results is S _rank ;

(9) the task parameter of each piece of operating data in the S _rank that step (8) obtains is displayed to the user on the graphic display interface, and the task parameter optimization process ends;

(10) When the user requests to optimize the task of the distributed iterative computing system again, return to step (2).

Claims (5)

1. a kind of method that task parameters optimize in distributed iterative computing system, it is characterised in that this method is gathered point first The service data of historic task in cloth iterative calculation system, builds historical data base；When carrying out task parameters optimization, according to about Beam condition is once filtered to notable incoherent service data in historical data base；Then it is corresponding to task to be optimized to go through Service data in history database carries out the Similarity Measure of directed acyclic graph with the service data after once filtering, and to similar Degree carries out secondary filter less than the service data of certain threshold value；Finally the result after filtering twice is sorted by calculating, and will The task parameters corresponding to service data after sequence are used as task parameters optimum results.

2. the method as described in claim 1, it is characterised in that this method specifically includes following steps：

(1) service data of each historic task is obtained from distributed iterative computing system, by the operation of each historic task Data are saved in historical data base, and each item data represents the service data of a historic task in historical data base；

(2) asked according to user, task parameters optimization is carried out to the task in distributed iterative computing system, if from historical data Being found out in storehouse is J with the historic task of task identical service data_src；

(3) the historic task service data composition data set for meeting all hardware resource constraint is found out from historical data base S_hardware；

(4) S obtained in step (3)_hardwareAll service datas in find out what input data total size and step (2) were obtained J_srcInput data total size numerically relative different be less than setting input data difference in size threshold value service data group Into data acquisition system S_datasize；

(5) S obtained in step (4)_datasizeAll service datas all directed acyclic graphs in find out and J_srcOriented nothing The service data composition data set S of ring figure directed acyclic graph close in scale_dag；

(6) S that calculation procedure (5) is obtained_dagIn each service data directed acyclic graph and J_srcDirected acyclic graph it is similar Degree, and set similarity threshold；

(7) result of calculation of traversal step (6), abandons S_dagMiddle directed acyclic graph and J_srcDirected acyclic graph similarity less than setting The service data of fixed similarity threshold, if the data acquisition system of remaining service data composition is S_sim；

(8) S obtained to step (7)_simIn each service data according to formulaResult of calculation

It is ranked up from high to low, and only retains service data first n in result of calculation after sequence, n is positive integer；In formula, Define J_srcDirected acyclic graph be G_src；If S_dagAny one of service data be J_dst, define J_dstDirected acyclic graph be G_dst, time_dstRepresent J_dstRun time；sim(G_src,G_dst) represent J_dstDirected acyclic graph and J_srcDirected acyclic graph phase Like degree；If the data acquisition system that acquired results are constituted after sequence is S_rank；

(9) S for obtaining step (8)_rankIn the task parameters of each service data be shown to use on pattern display interface Family, task parameters Optimizing Flow terminates；

(10) when user asks to optimize the task of distributed iterative computing system again, step (2) is returned to.

3. method as claimed in claim 2, it is characterised in that S is calculated in step (6)_dagIn each service data oriented nothing Ring figure and J_srcDirected acyclic graph similarity, specific calculation procedure is as follows：

(6-1) sets S_dagAny one of service data be J_dst, calculate J_dstDirected acyclic graph and J_srcDirected acyclic graph it is similar Degree, defines J_srcDirected acyclic graph be G_src=(N_src,E_src,L_src), wherein N_srcRepresent directed acyclic graph G_srcIn set of node Close, E_srcRepresent directed acyclic graph G_srcIn line set, L_srcRepresent directed acyclic graph G_srcIn label institute structure on each node Into set；Define J_dstDirected acyclic graph be G_dst=(N_dst,E_dst,L_dst), wherein N_dstRepresent directed acyclic graph G_dstIn Node set, E_dstRepresent directed acyclic graph G_dstIn line set, L_dstRepresent directed acyclic graph G_dstIn mark on each node The constituted set of label；

(6-2)J_dstWith J_srcDirected acyclic graph between similarity defined by equation below：

<mrow> <mi>s</mi> <mi>i</mi> <mi>m</mi> <mrow> <mo>(</mo> <mrow> <msub> <mi>G</mi> <mrow> <mi>s</mi> <mi>r</mi> <mi>c</mi> </mrow> </msub> <mo>,</mo> <msub> <mi>G</mi> <mrow> <mi>d</mi> <mi>s</mi> <mi>t</mi> </mrow> </msub> </mrow> <mo>)</mo> </mrow> <mo>=</mo> <mi>s</mi> <mi>k</mi> <mi>i</mi> <mi>p</mi> <mi>N</mi> <mrow> <mo>(</mo> <mrow> <msub> <mi>G</mi> <mrow> <mi>s</mi> <mi>r</mi> <mi>c</mi> </mrow> </msub> <mo>,</mo> <msub> <mi>G</mi> <mrow> <mi>d</mi> <mi>s</mi> <mi>t</mi> </mrow> </msub> </mrow> <mo>)</mo> </mrow> <mo>+</mo> <mi>s</mi> <mi>k</mi> <mi>i</mi> <mi>p</mi> <mi>E</mi> <mrow> <mo>(</mo> <mrow> <msub> <mi>G</mi> <mrow> <mi>s</mi> <mi>r</mi> <mi>c</mi> </mrow> </msub> <mo>,</mo> <msub> <mi>G</mi> <mrow> <mi>d</mi> <mi>s</mi> <mi>t</mi> </mrow> </msub> </mrow> <mo>)</mo> </mrow> <mo>+</mo> <mn>2</mn> <munder> <munder> <mi>&amp;Sigma;</mi> <mrow> <msub> <mi>n</mi> <mrow> <mi>s</mi> <mi>r</mi> <mi>c</mi> </mrow> </msub> <mo>&amp;Element;</mo> <msub> <mi>N</mi> <mrow> <mi>s</mi> <mi>r</mi> <mi>c</mi> </mrow> </msub> <mo>,</mo> </mrow> </munder> <mrow> <msub> <mi>n</mi> <mrow> <mi>d</mi> <mi>s</mi> <mi>t</mi> </mrow> </msub> <mo>&amp;Element;</mo> <msub> <mi>N</mi> <mrow> <mi>d</mi> <mi>s</mi> <mi>t</mi> </mrow> </msub> </mrow> </munder> <mfrac> <mrow> <mi>e</mi> <mi>d</mi> <mi>i</mi> <mi>t</mi> <mrow> <mo>(</mo> <mrow> <msub> <mi>l</mi> <mrow> <mi>s</mi> <mi>r</mi> <mi>c</mi> </mrow> </msub> <mo>,</mo> <msub> <mi>l</mi> <mrow> <mi>d</mi> <mi>s</mi> <mi>t</mi> </mrow> </msub> </mrow> <mo>)</mo> </mrow> </mrow> <mrow> <mi>max</mi> <mrow> <mo>(</mo> <mrow> <mo>|</mo> <msub> <mi>l</mi> <mrow> <mi>s</mi> <mi>r</mi> <mi>c</mi> </mrow> </msub> <mo>|</mo> <mo>,</mo> <mo>|</mo> <msub> <mi>l</mi> <mrow> <mi>d</mi> <mi>s</mi> <mi>t</mi> </mrow> </msub> <mo>|</mo> </mrow> <mo>)</mo> </mrow> </mrow> </mfrac> </mrow>

In formula, sim (G_src,G_dst) represent J_dstDirected acyclic graph and J_srcDirected acyclic graph similarity, span is [0,1]；

skipN(G_src,G_dst) represent make G_srcAnd G_dstIn equal process, G_srcWith G_dstThe number for the node for increasing or deleting respectively Mesh sum；

skipE(G_src,G_dst) represent make G_srcAnd G_dstDuring equal, G_srcWith G_dstRespectively increase or delete side number it With；

n_srcAnd n_dstRepresent respectively in directed acyclic graph G_srcIn any one node and directed acyclic graph G_dstIn any one section Point；

l_srcAnd l_dstNode n is represented respectively_srcAnd n_dstCorresponding label；

edit(l_src,l_dst) represent l_srcAnd l_dstEditing distance on two labels, i.e., by label l_srcIt is converted into label l_dstCross Minimum edit operation number of times needed for journey；

|l_src| and | l_dst| label l is represented respectively_srcWith label l_dstString length；

(6-3) repeat step (6-1) to (6-2), calculating obtains S_dagIn each service data directed acyclic graph and J_srcHave To the similarity of acyclic figure.

4. method as claimed in claim 2, it is characterised in that step (3) described hardware resource constraints, including：Historical data Service data overall memory and J in storehouse_srcOverall memory numerically relative different be less than setting internal memory discrepancy threshold；Fortune Row data can run CPU core number and J_srcRun CPU core number numerically relative different be less than setting check figure difference threshold Value；Service data machine nodes and J_srcMachine nodes numerically relative different be less than setting machine nodes it is poor Different threshold value.

5. method as claimed in claim 2, it is characterised in that the scale of step (5) described two directed acyclic graphs is close, bag Include following two aspects condition：First, numerically relative different is less than having for setting to the interstitial content on two directed acyclic graphs To acyclic node of graph number discrepancy threshold；Second, numerically relative different is less than and set for side number on two directed acyclic graphs Fixed directed acyclic graph side number discrepancy threshold.