WO2022111125A1

WO2022111125A1 - Random-forest-based automatic optimization method for graphic data processing framework

Info

Publication number: WO2022111125A1
Application number: PCT/CN2021/124378
Authority: WO
Inventors: 陈超; 辛锦瀚; 杨永魁; 王峥; 喻之斌; 郭伟钰; 刘江佾
Original assignee: 深圳先进技术研究院
Priority date: 2020-11-27
Filing date: 2021-10-18
Publication date: 2022-06-02
Also published as: CN114565001A

Abstract

Disclosed is a random-forest-based automatic optimization method for a graphic data processing framework. The method comprises: constructing a training data set, wherein each piece of sample data in the training data set represents a correspondence between a configuration parameter combination of a graphic data processing framework, the size of an input data set and a program running time; training, on the basis of the training data set, a random forest model that includes a plurality of decision-making trees, and taking the trained random forest model as a performance prediction model for predicting, in combination with the size of the input data set, a corresponding program running time for different configuration parameter combinations; and in a search space of configuration parameters, by using the performance prediction model, predicting, for different sizes of input data sets, the performance level of different configuration parameters generated by a genetic algorithm, so as to obtain an optimal configuration parameter. By means of the present invention, the size of an input data set can be sensed, and deep and high-performance automatic optimization of configuration parameters is realized.

Description

An Automatic Tuning Method of Graph Data Processing Framework Based on Random Forest

technical field

The invention relates to the technical field of big data processing, and more particularly, to an automatic tuning method for a graph data processing framework based on random forest.

Background technique

With the development of the Internet industry and technology, in the field of big data, the scale and importance of graph data processing are also increasing. Take the Spark GraphX framework as an example, which is an embedded graph processing framework built on Apache Spark using a distributed data flow system. Spark GraphX provides a familiar configurable graph abstraction, sufficient to represent existing graph structures, and can be implemented using some basic data flow operators (such as join, map, and group). At the same time, SparkGraphX uses distributed connection optimization and materialized view maintenance to rebuild specific graph optimizations, and utilizes a distributed data flow framework to provide low-cost graph processing fault tolerance.

The performance of Spark GraphX is mainly affected by configuration parameters, and unreasonable configuration will seriously reduce the performance of the framework. Spark officially recommends a set of default configuration parameters. However, in actual graph data processing tasks, the default configuration parameters cannot be adapted according to changes in computing resources and workloads, which limits the performance of Spark GraphX and wastes A lot of computing resources. Spark GraphX has a large number of configuration parameters, and there is mutual influence between different parameters, so manual parameter adjustment is difficult and costly. Therefore, the automatic tuning method of Spark GraphX configuration parameters has great research significance.

The existing Spark GraphX framework optimization method only implements a series of system optimizations for the limitations imposed by graph parallel abstraction and sparse graph structure. The optimization objects mainly include graph features and graph systems. Based on the classic technologies of traditional database systems, indexing is carried out. , optimizations for incremental view maintenance and joins, and optimizations for standard dataflow operators in Spark to achieve performance parity with dedicated graph processing systems. However, the existing Spark GraphX optimization method only optimizes the characteristics of the graph data itself and the internal implementation of the Spark GraphX system, but does not consider the impact of the runtime configuration parameters and the size of the input data set on the performance of Spark GraphX, and the optimization effect is poor; In addition, the machine learning algorithm used by the existing Spark GraphX optimization method has poor performance and cannot be applied to the current Spark GraphX parameter tuning optimization scenario.

SUMMARY OF THE INVENTION

The purpose of the present invention is to overcome the above-mentioned defects of the prior art, and to provide an automatic tuning method for graph data processing framework based on random forest, which can be applied to the configuration parameter optimization of graph processing frameworks such as Spark GraphX.

The technical solution of the present invention is to provide an automatic tuning method for graph data processing framework based on random forest, and the method includes the following steps:

Build a training data set, each sample data of the training data set represents the configuration parameter combination of the graph data processing framework, the corresponding relationship between the size of the input data set and the running time of the program;

A random forest model including multiple decision trees is trained based on the training data set, wherein the training set of each decision tree is generated by guiding and focusing on the training data set, and the trained random forest model is used as a performance prediction model. For predicting the corresponding program running time for different parameter configuration combinations combined with the size of the input data set;

In the search space of configuration parameters, the performance prediction model is used to predict the performance of different configuration parameters generated by the genetic algorithm for different input data set sizes, and then the optimal configuration parameters are obtained.

Compared with the prior art, the present invention has the advantage that, in a heterogeneous machine cluster, the configuration parameters of the graph processing framework are used as the optimization object to realize automatic parameter adjustment and optimization, and can automatically perceive the size of the data set, and finally find the running program. best configuration. Aiming at the characteristics of the parameter adjustment optimization of the graph processing framework, the invention selects the random forest algorithm (RF) and combines the genetic algorithm (GA) to automatically perceive the scale of the input data set, so as to realize the in-depth and efficient parameter adjustment optimization.

Other features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention.

1 is a flowchart of an automatic tuning method for a graph data processing framework based on random forest according to an embodiment of the present invention;

2 is a schematic process diagram of an automatic tuning method for a graph data processing framework based on random forests according to an embodiment of the present invention;

Fig. 3 is the effect comparison diagram of the prior art and an embodiment of the present invention;

FIG. 4 is an optimization effect diagram of accelerated operation of Spark GraphX program according to an embodiment of the present invention.

Detailed ways

Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the invention unless specifically stated otherwise.

The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification.

In all examples shown and discussed herein, any specific values should be construed as illustrative only and not limiting. Accordingly, other instances of the exemplary embodiment may have different values.

It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further discussion in subsequent figures.

The present invention can be applied to various types of big data processing frameworks, such as Spark GraphX, PowerGraph, TinkerPop, and the like. For ease of understanding, this article will take the Spark GraphX framework as an example.

Combined with Figure 1 and Figure 2, the provided automatic tuning method of the graph data processing framework based on random forest includes the following steps:

Step S110, constructing a training data set, wherein each piece of sample data represents the configuration parameter combination of the graph data processing framework, the corresponding relationship between the size of the input data set and the running time of the program.

This step is the data collection part, including a parameter generator, which automatically generates parameters for each Spark GraphX program to run the program to be optimized, and automatically collects the program running time after each run, corresponding to the configuration parameters used and the size of the data set. Combined to obtain a piece of sample data, after running the Spark GraphX program multiple times, a sample set, or training data set, is finally obtained.

Specifically, the parameter generator (Conf Generator) first selects parameters that significantly affect the performance of Spark GraphX; then, according to the selected parameters, it automatically generates and assigns parameters for the operation of the program to be optimized; The parameters automatically run the program to be optimized. After each program runs, the configuration parameters used by the SparkGraphX program and the size of the input data set at runtime are collected, combined with the Spark GraphX program running time, as a piece of sample data in the training data set. In this way, a training dataset consisting of multiple sample data will be obtained after multiple runs.

Step S120, using the training data set to train a random forest model including multiple decision trees as a performance prediction model.

This step uses the training data set generated in the data collection stage to model based on the machine learning algorithm, and the purpose is to build a performance prediction model that can reflect the impact of different configuration parameters and different input data set sizes on program execution performance.

Preferably, the random forest algorithm is used for modeling, and the performance prediction model is obtained according to the following steps:

Step S121, use the guided aggregation algorithm from the training data set, take out m samples, carry out n _tree (the number of decision trees in the random forest algorithm) sampling in total, and generate n _tree training sets or training subsets according to these samples .

Step S122, use these training subsets to train into n _tree decision tree models;

Step S123, wherein for a single decision tree model, first randomly select a subset containing k Spark GraphX attributes (such as Spark GraphX parameters and data set size) from the attribute combination of the node, and then divide each time according to information gain or The Gini index selects the best attributes from the subset to split.

In step S124, each decision tree is split according to this rule until all training subsets of the node belong to the same class.

In step S125, a random forest model is formed from the finally generated decision trees, and the output result of the random forest model can be voted by the multiple tree classifiers to determine the final classification result or the final prediction result can be determined by the mean of the predicted values of the multiple trees.

The trained random forest model is the performance prediction model or simply the performance model, which can be used to predict the running time of the Spark GraphX program with different configuration parameters combined with the size of the input data set.

The above step S124 can be understood as stopping the splitting after reaching the target accuracy. With the execution of the random forest algorithm, the execution time changes less and less, and the model accuracy becomes more and more accurate. By increasing the number of decision trees, it helps to solve the "overfitting" problem. The above-mentioned n _tree , m, and k are integers greater than or equal to 2, and can be appropriately set according to actual precision or execution speed requirements.

Step S130: In the search space of configuration parameters, the performance prediction model is used to predict the performance of different configuration parameters generated by the genetic algorithm for different input data set sizes, thereby obtaining optimal configuration parameters.

In this step, the genetic algorithm is used to perform an iterative search based on the performance prediction model, and the optimal configuration parameters are finally screened.

In the search stage, the performance prediction model is used to predict the performance of different configuration parameters generated by the genetic algorithm in Spark GraphX under different input data set sizes, so as to avoid the actual running of the program, achieve efficient search, and finally obtain the optimal Configure parameters and use them directly in Spark GraphX programs, thereby improving Spark GraphX performance.

Specifically, the search process for the optimal configuration includes:

Step S131 , randomly input a set of configuration parameters in the search space of configuration parameters, and calculate the initialized individual fitness A standard through the performance prediction model.

For example, the program execution time output by the performance prediction model is used as the individual fitness criterion in combination with the actual input dataset size.

Step S132, randomly select n groups of configuration parameters (for example, n is greater than 1/5 of the number of training sets) from the search space of configuration parameters as the initialization population P, and perform random crossover operations on each individual in P and a mutation rate of 0.02. mutation operation.

For example, the mutation rate can be set to other values based on the configuration parameter search space size or requirements for execution speed.

Step S133: Use the performance prediction model to calculate the fitness of the population P and its descendants, and screen out individuals whose fitness is higher than A to form a new population P', and use the fitness A' of the individual with the highest fitness as the new fitness Standard A'.

Step S134, repeating S132 and S133 until no better individual can be generated, then the current optimal individual is the optimal configuration parameter searched.

The invention automatically collects the running data of the program to be optimized, and uses the combination of random forest algorithm modeling and genetic algorithm to realize high-performance automatic parameter adjustment optimization. The crossover mutation feature of genetic algorithm is used to avoid the search falling into local optimum, while ensuring excellent search performance.

In order to further verify the effect of the present invention, experimental verification was carried out. Based on the Spark GraphX test programs officially provided by Spark, namely PageRank (PR), NWeight (NW), Connected Component (CC) and Triangle Counting (TC), the configuration parameters of the Spark GraphX framework are automatically optimized.

In the experiment, the decision tree algorithm (DT) and the support vector machine algorithm (SVM), two of the most commonly used algorithms in the existing Spark GraphX optimization technology, were selected, and the performance was compared with the random forest algorithm (RF) used in the present invention; and , directly using the optimization method of the present invention to conduct experiments on the optimization effect of Spark GraphX under different input data sets.

Fig. 3 is the decision tree algorithm (DT) and the support vector machine algorithm (SVM) commonly used in the prior art and the random forest algorithm (RF) used in the present invention, for the modeling effect comparison of two different Spark GraphX programs selected . It can be clearly seen from the experimental results in the figure that the modeling accuracy of the RF algorithm of the present invention (right histogram) under different programs is higher than that of the other three algorithms. And the modeling accuracy of the RF algorithm is 26.1% higher than that of the DT algorithm on average, and 10.6% higher than that of the SVM algorithm. Therefore, the modeling method used in the present invention is more excellent.

Figure 4 shows the optimization of the accelerated operation of Spark GraphX programs. Since the optimization method of the present invention automatically configures reasonable parameters for different programs with different input data set sizes, compared with the official default configuration (the left histogram), this The invented optimization method (the histogram on the right) significantly improves the running speed of Spark GraphX, with an average increase of 2.0 times and a maximum increase of 2.8 times.

The experimental results show that the optimization method of the present invention realizes the automatic parameter adjustment and optimization of Spark GraphX, and the optimization performance is better than that of the prior art, and the corresponding optimal configuration can be found according to the size of different input data sets. Compared with the official default configuration, the data processing speed of Spark GraphX is significantly improved.

In summary, the present invention proposes a method that can perceive the size of the input data set, and automatically adjust parameters based on the configuration parameters of Spark GraphX, thereby realizing the automatic optimization of deep-level high performance of Spark GraphX. Compared with the prior art, the effect of the present invention is mainly reflected in the following aspects:

1) The existing Spark GraphX automatic optimization method is only for graph features and graph systems, and is implemented by optimizing internal code and graph query algorithms, but it does not go deep into optimizing the configuration parameters at runtime, and the configuration parameters will It directly affects the performance of Spark GraphX to a greater extent, so the existing optimization methods are not deep enough. The present invention performs automatic parameter tuning based on the configuration parameters of Spark GraphX, and realizes the deep-level parameter tuning optimization of Spark GraphX.

2) The existing Spark GraphX optimization method does not consider the impact of the size of the input dataset on performance, but since Spark GraphX uses Spark as the underlying computing framework, it is very sensitive to the size of the input dataset, so the size of the input dataset cannot be ignored. The optimization method proposed by the invention can automatically perceive the size of the input data set, and realizes the high-performance automatic parameter tuning optimization of Spark GraphX by combining the random forest algorithm and the genetic algorithm.

3) The machine learning algorithm used by the existing Spark GraphX optimization method has poor performance and does not meet the requirements of Spark GraphX parameter automatic tuning. The invention combines random forest algorithm and genetic algorithm, and proposes a method more suitable for automatic parameter adjustment and optimization of Spark GraphX.

The present invention may be a system, method and/or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the present invention.

A computer-readable storage medium may be a tangible device that can hold and store instructions for use by the instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer readable storage media include: portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM) or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically coded devices, such as printers with instructions stored thereon Hole cards or raised structures in grooves, and any suitable combination of the above. Computer-readable storage media, as used herein, are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (eg, light pulses through fiber optic cables), or through electrical wires transmitted electrical signals.

The computer readable program instructions described herein may be downloaded to various computing/processing devices from a computer readable storage medium, or to an external computer or external storage device over a network such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer-readable program instructions from a network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .

The computer program instructions for carrying out the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or instructions in one or more programming languages. Source or object code, written in any combination, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as the "C" language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider through the Internet connect). In some embodiments, custom electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), can be personalized by utilizing state information of computer readable program instructions. Computer readable program instructions are executed to implement various aspects of the present invention.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer or other programmable data processing apparatus to produce a machine that causes the instructions when executed by the processor of the computer or other programmable data processing apparatus , resulting in means for implementing the functions/acts specified in one or more blocks of the flowchart and/or block diagrams. These computer readable program instructions can also be stored in a computer readable storage medium, these instructions cause a computer, programmable data processing apparatus and/or other equipment to operate in a specific manner, so that the computer readable medium on which the instructions are stored includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.

Computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other equipment to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other equipment to produce a computer-implemented process , thereby causing instructions executing on a computer, other programmable data processing apparatus, or other device to implement the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more functions for implementing the specified logical function(s) executable instructions. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or actions , or can be implemented in a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation in hardware, implementation in software, and implementation in a combination of software and hardware are all equivalent.

Various embodiments of the present invention have been described above, and the foregoing descriptions are exemplary, not exhaustive, and not limiting of the disclosed embodiments. Numerous modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the various embodiments, the practical application or technical improvement in the marketplace, or to enable others of ordinary skill in the art to understand the various embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

An automatic tuning method for graph data processing framework based on random forest, including the following steps:

Build a training data set, each sample data of the training data set represents the configuration parameter combination of the graph data processing framework, the corresponding relationship between the size of the input data set and the running time of the program;

A random forest model including multiple decision trees is trained based on the training data set, wherein the training set of each decision tree is generated by guiding and focusing on the training data set, and the trained random forest model is used as a performance prediction model. For predicting the corresponding program running time for different parameter configuration combinations combined with the size of the input data set;

In the search space of configuration parameters, the performance prediction model is used to predict the performance of different configuration parameters generated by the genetic algorithm for different input data set sizes, and then the optimal configuration parameters are obtained.
The method of claim 1, wherein training a random forest model comprising a plurality of decision trees based on the training data set comprises:

Using the guided aggregation algorithm from the training data set, take out m samples, carry out n trees sampling in total, and generate n trees training sets according to these samples, and n trees correspond to the number of decision trees included in the random forest model;

Use the training set to train n tree decision trees, wherein for a single decision tree, first randomly select a subset containing k graph data processing framework attributes from the attribute combination of the node, and then divide each time according to the information gain or The Gini index selects the optimal attribute from the subset to split, and then generates multiple decision trees to form a random forest model.
The method according to claim 1, wherein the output result of the performance prediction model is determined by a classification vote of a plurality of decision trees or by an average value of the prediction values of the plurality of decision trees.
The method according to claim 1, wherein, in the search space of configuration parameters, the performance prediction model is used to predict the performance of different configuration parameters generated by a genetic algorithm for different input data set sizes, so as to obtain an optimal configuration Parameters include:

Randomly input a set of configuration parameters in the search space of configuration parameters and calculate the initialized individual fitness A standard through the performance prediction model, where the individual fitness is the predicted program execution time;

Randomly select n groups of configuration parameters from the search space of configuration parameters as the initialization population P, and perform random crossover and mutation operations on each individual in P;

Use the performance prediction model to calculate the fitness of the population P and its descendants, screen out individuals with a fitness higher than A to form a new population P', and use the fitness A' of the individual with the highest fitness as the new fitness standard A', find out the individual with the highest fitness through iterative operation, and this individual corresponds to the optimal configuration parameters.
The method of claim 1, wherein said constructing a training data set comprises:

The configuration parameters are automatically generated for each time the graph data processing framework program runs the program to be optimized, and the program running time is automatically collected after each operation, which is combined with the corresponding configuration parameters used and the size of the input data set as a piece of sample data.
The method of claim 1, wherein the graph data processing framework comprises Spark GraphX, Power Graph or Tinker Pop.
A computer-readable storage medium having stored thereon a computer program, wherein the program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6.
A computer device, comprising a memory and a processor, a computer program that can be run on the processor is stored in the memory, and characterized in that, when the processor executes the program, any one of claims 1 to 6 is implemented The steps of the method described in item.