CN103164709A - Method for optimizing support vector machine based on tabu search algorithm - Google Patents

Abstract

The invention relates to firstly considering the selection of kernel function and the selection of parameters when applying the support vector machine to solve specific problems. Although some achievements have been made in the theoretical research and application of the kernel function, the theory to guide the selection of support vector machine parameters has not yet been formed. The invention uses a tabu search algorithm to optimize the parameters of the support vector machine based on the radial basis kernel function. The classic tabu search algorithm is extended, and the neighborhood generation adopts the eight-grid method, which can be adjusted automatically. The convergence speed of the algorithm is improved without loss of accuracy, and all the local optimal solutions around are found by radially exploring the surrounding at the local optimum, so as to achieve the global optimum as much as possible. The algorithm is tested with the test function and the standard data set, and the results show that the improved algorithm can effectively find the global optimal solution, so that SVM has a higher classification accuracy.

Description

A Method of Optimizing Support Vector Machine Based on Tabu Search Algorithm

technical field

The invention relates to a method for optimizing a support vector machine, in particular to a method for optimizing a support vector machine based on a tabu search algorithm.

Background technique

Since the support vector machine is widely used in many fields, its importance is becoming more and more prominent, and the parameter selection directly affects the performance of the support vector machine to identify the target. How to determine the parameters of the support vector machine is an important content of the study of the support vector machine, and it has naturally become a research hotspot. At present, many intelligent algorithms are used to optimize the parameters of support vector machines, and the tabu search algorithm is getting more and more attention because of its high solution quality and efficiency and its powerful optimization ability in many combinatorial optimization neighborhoods. of favor.

The present invention uses the tabu search algorithm for the optimization of support vector machine parameters, makes some improvements to the tabu search algorithm and tests it based on a standard data set, and compares it with the support vector machine parameter selection based on the upper bound of structural risk to show that the algorithm of the present invention Have better performance.

The tabu search algorithm avoids roundabout searches by introducing a tabu list and a taboo criterion, and pardons some tabooed good states through the amnesty criterion, thereby ensuring a variety of effective searches and finally achieving global optimization. In the literature title: "SVM parameter selection based on the upper bound of structural risk", authors: Song Xiaoshan, Jiang Xiaoyu, Luo Jianhua, Wang Xi, Science and Technology Herald 2011, 29 (08), gave the algorithm of the upper bound of structural risk, proposed A SVM parameter selection method based on the upper bound of structural risk can obtain parameters that make SVM more generalizable.

However, there is still no uniform standard for the selection of support vector machine parameters, and the selection of parameters mostly relies on experience and adopts a trial and error method, which is not only time-consuming but also difficult to obtain satisfactory results.

Contents of the invention

When applying support vector machine to solve specific problems, the choice of kernel function and parameters should be considered first. Although some achievements have been made in the theoretical research and application of the kernel function, the theory to guide the parameter selection of the support vector machine has not yet been formed. The technical problem to be solved by the present invention is to propose a method for optimizing the support vector machine based on a tabu search algorithm in view of the blindness and low efficiency of selecting parameters of the existing support vector machine. Through this algorithm, the parameters of the support vector machine can be selected more specifically and efficiently, and the approximate optimal parameters in the solution space can be obtained through the tabu search, thereby improving the classification effect and global optimization performance of the support vector machine to a certain extent .

The present invention is achieved through the following technical solutions:

A method for optimizing support vector machines based on the tabu search algorithm, the original data is the classic test function Shaffer's F6.

The first step is to use the tabu search algorithm to optimize the penalty factor and kernel function parameters of the support vector machine. Specifically, it is divided into the following parts to achieve:

The first step A, given the algorithm parameters, randomly generates the initial parameters c and γ, initializes the taboo table and sets tab to be empty, respectively initializes the global optimal solution best_glo and the local optimal solution best_loc, and sets the escape state es as 0 (that is, it indicates that it is initially in a non-escape state);

The first step B is to judge whether the termination condition of the algorithm is satisfied. If so, end the algorithm and output the optimization result; otherwise, continue with the following steps;

The first step C is to calculate the number of times the current parameter is obtained, and set the neighborhood radius jump accordingly;

The first step D is to record the number of consecutive unupdated steps of the local optimal record best_loc. If the number of steps exceeds the preset threshold, it can be considered that the current solution falls into the local optimal. The algorithm starts to escape, explore and get the escape point, add the escape candidate set es_para, and set the escape state es to 1;

The first step E is to judge whether it is in the escape state (es=1), if so, the neighborhood radius jump is set to 1, and the parameters are successively taken from the parameters in the escape candidate set, otherwise, continue the following steps;

In the first step F, based on the neighborhood radius jump, a certain number of neighborhood solutions are generated according to certain rules as candidate solutions;

In the first step G, the adaptation value of each neighborhood solution is obtained through SVM calculation, that is, the corresponding classification accuracy rate is generated by the solution, and the solutions are sorted according to the accuracy rate from large to small;

The first step H is to make judgments on the elements in the candidate set in turn: check whether the correct rate corresponding to this parameter is greater than the local historical optimal record, if it is greater, replace the local historical optimal record, update the taboo table, and use this solution as the next step The starting point of the search; otherwise, proceed to the following steps;

The first step I is to judge whether the parameter is in the taboo list, if not, add the taboo list, and use this parameter as the starting point of the next search; otherwise, judge the next solution; if it is not obtained greater than the historical optimal record or not in The solution in the tabu table, using the best parameters as the starting point for the next search;

In the first step J, turn to step (1.B).

The second step is to use the obtained approximate optimal parameters to train to obtain the optimal support vector machine model, and to conduct experiments on the test function based on this, to verify the classification performance of the support vector machine and the ability to search for the global optimal solution, using the tabu search The algorithm-optimized support vector machine model processes the optimization function and classification samples.

The beneficial effect of the present invention is that: the present invention uses the tabu search algorithm to optimize the parameters of the support vector machine, introduces the eight-grid method to expand the structure of the neighborhood solution, and improves the escape mechanism of the traditional tabu search. Therefore, the classification model of the support vector machine is optimized, and the blindness and inaccuracy of parameter selection are reduced. After the improvement of our method, the optimized support vector machine has a certain improvement in the ability to solve the global optimal solution and the classification effect; moreover, using the tabu search algorithm to select the parameters of the support vector machine avoids the traditional support vector machine when selecting parameters. Time-consuming and labor-intensive defect, can accurately obtain approximate optimal parameters at one time. The invention uses a tabu search algorithm to optimize the parameters of the support vector machine based on the radial basis kernel function. The classic tabu search algorithm is extended, and the neighborhood generation adopts the eight-grid method, which can be adjusted automatically. The convergence speed of the algorithm is improved without loss of accuracy, and all the local optimal solutions around are found by radially exploring the surrounding at the local optimum, so as to achieve the global optimum as much as possible. The algorithm is tested with the test function and the standard data set, and the results show that the improved algorithm can effectively find the global optimal solution, so that SVM has a higher classification accuracy.

Description of drawings

Fig. 1 is a structural diagram of the neighborhood solution of the tabu search algorithm in the present invention.

Fig. 2 is the function Shaffer's F6 that adopts when solving function optimal solution of the present invention.

Fig. 3 is a trend diagram of the accuracy rate when the present invention is used for classical data set classification.

Fig. 4 is a flowchart of the method steps of the present invention.

Detailed ways

The present invention is further described below in conjunction with embodiment. The scope of the present invention is not limited by these examples, and the scope of the present invention is set forth in the claims.

As shown in Figure 4:

Start, then set the parameters, initialize the record, es=0;

Calculate the parameter frequency, set jump;

Calculate the number of unupdated steps of the local optimal record, judge whether to escape, if yes, explore the escape point, add the escape candidate set, es=1, if not, then, if es=1, let jump=1, the parameters are taken in turn Parameters in the escape candidate set, es=0;

Generate a candidate set and calculate the correct rate;

The elements of the candidate set are sorted from the largest to the smallest according to the correct rate; judge in turn whether the correct rate is greater than the local optimum, if not, it is necessary to judge whether all the candidate set elements have been judged, if not, it is necessary to re-judge whether the correct rate is greater than the local optimum, If yes, the solution is used as the starting point for the next step;

Update local and global optimal records;

Add the taboo table, the first solution is used as the starting point for the next step, further judge whether the termination condition is met, if yes, introduce, if not, return to the beginning.

Specifically, the parameter optimization design of the tabu search algorithm is as follows:

The first step A, given the algorithm parameters, randomly generates the initial parameters c and γ, initializes the taboo table and sets tab to be empty, respectively initializes the global optimal solution best_glo and the local optimal solution best_loc, and sets the escape state es as 0 (that is, it indicates that it is initially in a non-escape state); according to the possible value range of the penalty factor c and the radial basis function g, the integers c∈[0.1, 100], g∈[0.1, 1000] are selected. The variation ranges of penalty factor c and kernel function parameter γ are [0, 100] and [0, 1000] respectively.

The first step B is to judge whether the termination condition of the algorithm is satisfied. If yes, end the algorithm and output the optimization result; otherwise, continue with the following steps. The termination rule setting of tabu search optimization algorithm among the present invention is as follows:

(1) Set a maximum number of iterations, for example, 500 generations. When the running times of the algorithm reach this value, regardless of the current search state, the algorithm will be terminated and the best solution and state so far will be returned. (2) Set the maximum taboo frequency value of a single object. In order to avoid roundabout search, if a certain state occurs during the operation of the algorithm, and the taboo frequency of the corresponding function fitness value exceeds a preset value (15 times), the algorithm is terminated and the result is returned.

The first step C is to calculate the number of times the current parameter is obtained, and set the neighborhood radius jump accordingly;

The first step D is to record the number of consecutive unupdated steps of the local optimal record best_loc. If the number of steps exceeds the preset threshold, it can be considered that the current solution falls into the local optimal. The algorithm starts to escape, explore and get the escape point, add the escape candidate set es_para, and set the escape state es to 1;

The first step E is to judge whether it is in the escape state (es=1), if so, the neighborhood radius jump is set to 1, and the parameters are successively taken from the parameters in the escape candidate set, otherwise, continue the following steps;

In the first step F, based on the neighborhood radius jump, a certain number of neighborhood solutions are generated according to certain rules as candidate solutions;

In the first step G, the adaptation value of each neighborhood solution is obtained through SVM calculation, that is, the corresponding classification accuracy rate is generated by the solution, and the solutions are sorted according to the accuracy rate from large to small;

The first step H is to make judgments on the elements in the candidate set in turn: check whether the correct rate corresponding to this parameter is greater than the local historical optimal record, if it is greater, replace the local historical optimal record, update the taboo table, and use this solution as the next step The starting point of the search; otherwise, proceed to the following steps;

The first step I is to judge whether the parameter is in the taboo list, if not, add the taboo list, and use this parameter as the starting point of the next search; otherwise, judge the next solution; if it is not obtained greater than the historical optimal record or not in The solution in the tabu table, using the best parameters as the starting point for the next search;

In the first step J, turn to step (1.B).

The second step is to use the support vector machine model optimized by the tabu search algorithm to process the optimization function and classification samples, which are divided into the following parts:

The second step A is to verify the global optimization ability of the proposed algorithm based on the test of classical functions;

Shaffer’s F6 function was proposed by J.D.Shaffer et al., the expression is

$\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0.5</span>}<span\; class="notranslate">\; -</span>\frac{{\mathrm{<span\; class="notranslate">\; sin</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}\sqrt{{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 0.5</span>}}{{<span\; class="notranslate">\; [</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 0.001</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ]</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}$

Among them, the value range of the independent variable is [-100, 100]. Figure 2 is the graph of Shaffer’s F6 function in the [-10, 10] interval, it has infinite local maximum points, only one (0, 0) is the global maximum, and the maximum value is 1. There is a circular ridge around the maximum value of this function, and their values are all 0.990283. At the same time, because this function has a strong oscillation property, a wide range of variable intervals, and a large search range, it is easy to fall into a local maximum point, and Once you're in it's hard to get out.

Use the Shaffer's F6 function to test the tabu search algorithm proposed in this paper, set the number of calculation steps as 100 steps, parameter c∈(-10, 10), step size c_step=0.01, γ∈(-10, 10), step size γ_step=0.01, test 10 times, the test results are shown in Table 4-1.

Table 1 Shaffer’s F6 function test results

Take any one test result as shown in Figure 3. The figure shows the trend of the objective function as the number of operation steps increases. From the graph analysis, the algorithm can converge quickly, and then find a better point through the escape process, which shows that the performance of the algorithm is still relatively good.

The second step B, the test based on the data set. A large number of experiments have been done on the classic data sets of support vector machines, and performance comparisons have been made with similar methods.

The data set information is shown in Table 2, each data set is tested 10 times, and the experimental results are listed in Table 3.

Table 2 Dataset information

data set training set size Test set size data dimension heart 100 170 13 Australian 200 490 14 German 600 400 twenty four Vehicle 400 445 18

Table 3 Experimental results

The exemplary embodiments of the present application are described above with reference to the accompanying drawings. It should be understood by those skilled in the art that the above-mentioned embodiments are only examples for the purpose of illustration, and are not used for limitation. Any modifications, equivalent replacements, etc. All should be included in the protection scope of this application.

Claims (5)

1. A method based on tabu search algorithm optimization support vector machine, original data is that test function Shaffer's F6 is characterized in that, comprises the following steps: (1), use the tabu search algorithm to optimize the penalty factor and kernel function parameters of the support vector machine; (2) Use the obtained approximate optimal parameters to train the optimal support vector machine model, and based on this, conduct experiments on the test function to verify the classification performance of the support vector machine and the ability to search for the global optimal solution. 2. The method for optimizing a support vector machine based on a tabu search algorithm according to claim 1, wherein said step (1) comprises the following steps: (1.A), the algorithm parameters are given, the initial parameters c and γ are randomly generated, and the taboo table tab is set to empty, The global optimal record best_glo, the local optimal record best_loc are initialized, and the escape state es is set to 0, that is, the non-escape state; (1.B), determine whether the algorithm termination condition is satisfied; if so, end the algorithm and output the optimization result; otherwise, continue the following steps; (1.C), calculate the number of times the current parameter is fetched, and set the neighborhood radius jump accordingly; (1.D), record the number of consecutive unupdated steps of the local optimal record best_loc, if the local record has not been updated after a certain number of steps, it is considered that the current solution is trapped in the local optimal, start to escape, explore the escape point, and join the escape candidate set es_para, set the escape state es to 1; (1.E), determine whether it is in escape state, that is, es=1, if so, set the neighborhood radius jump to 1, and take the parameters in the escape candidate set in turn, otherwise, continue the following steps; (1.F), based on the neighborhood radius jump, generate all its neighborhood solutions as candidate solutions; (1.G), the classification accuracy rate corresponding to the neighborhood solution is obtained through SVM calculation, and the solutions are sorted according to the accuracy rate from large to small; (1.H), make judgments on the elements in the candidate set in turn: check whether the correct rate corresponding to this parameter is greater than the local historical optimal record, if it is greater, replace the local historical optimal record, update the taboo table, and use the solution as the following The starting point of one-step search; otherwise, continue to the following steps; (1.I), judge whether the parameter is in the taboo list, if not, add it to the taboo list, and use this parameter as the starting point for the next search; otherwise, judge the next solution; if no record greater than the historical optimal record or For solutions that are not in the tabu list, the best parameters are used as the starting point for the next search; (1.J), go to step (1.B). 3. The method for optimizing a support vector machine based on a tabu search algorithm according to claim 1, wherein the step (2) further uses the support vector machine model optimized by the tabu search algorithm to treat optimization functions and classification samples For processing, it is divided into the following parts to achieve: (2.A), based on the test of classical functions, verify the global optimization ability of the proposed algorithm; Shaffer's F6 function is proposed by J.D.Shaffer et al., the expression is $\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0.5</span>}<span\; class="notranslate">\; -</span>\frac{{\mathrm{<span\; class="notranslate">\; sin</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}\sqrt{{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 0.5</span>}}{{<span\; class="notranslate">\; [</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 0.001</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ]</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}$ Among them, the value range of the independent variable is, [-100,100]; use Shaffer's F6 function to test the proposed tabu search algorithm, set the number of calculation steps to 100 steps, the parameter c∈(-10,10), and the length c_step =0.01, γ∈(-10,10), step size = test N times; (2.B), data set-based test: Experiment on the classic data set of support vector machine, and compare the performance with the same type of method. 4. the method for optimizing support vector machine based on tabu search algorithm according to claim 3, is characterized in that, described step (2) is that Shaffer's F6 function is in [-10,10] interval, and it has infinite local For the maximum point, only one (0,0) is the global maximum, and the maximum value is 1. There is a circle ridge around the maximum value of this function, and their values are all 0.990283. 5. The method for optimizing a support vector machine based on a tabu search algorithm according to claim 3, characterized in that, in the step (2), the test is performed N times, and N=10 times can be taken.

Images