CN104200087A - Parameter optimization and feature tuning method and system for machine learning - Google Patents

Abstract

The invention provides a parameter optimization and feature tuning method for machine learning. The method includes the following steps that a plurality of parameter sets are generated randomly; iterative optimization based on EnKF is conducted on the parameter sets; performance evaluation is conducted on the optimized parameter sets, a set pool and a supplement parameter set are obtained according to evaluation results, and the performance of the parameter sets in the set pool is higher than the performance of the parameter sets in the supplement parameter set; iterative optimization based on the EnKF and performance evaluation are conducted on the parameter sets in the set pool and the parameter sets in the supplement parameter set again, and accordingly an optimal parameter set is obtained. By the adoption of the method, the computational efficiency for processing the computed results of parameter optimization can be improved, and universality is high. The invention further provides a parameter optimization and feature tuning system for machine learning.

Description

For the parameter optimization of machine learning and the method and system of feature tuning

Technical field

The present invention relates to the parameter optimization technical field of machine learning, particularly the method and system of a kind of parameter optimization for machine learning and feature tuning.

Background technology

For general machine learning algorithm, the performance of model depends primarily on its parameter configuration.The model that adopts different parameter combinations to generate often has very large performance difference.Parameter optimization is a stochastic optimization problems, and its randomness is mainly reflected in: the training data that generation model is used and test data have comprised limited sample, cannot reflect entirety, and the joint distribution function of parameter space based on unknown.The basic definition of this problem is as follows: a given training dataset X _t, wherein X _tdata distribution G based on unknown, the target of parameter optimization is to look for the parameter combinations θ of a machine learning algorithm F, at X _ton set up a model f, make f in lower (or minimizing) performance evaluating value that maximizes of given performance evaluating criterion g ().Basic problem is expressed as follows,

${\mathrm{\θ}}_{\mathrm{opt}}\≈\underset{\mathrm{\θ}}{\arg \min }\frac{1}{n}\underset{x\∈X_V}{\mathrm{\Σ}}g(x,F(\mathrm{\θ},X_T)).$

But, directly calculate being contemplated to be of G very difficult.Therefore, general mode is that optimal model f is at given verification msg collection x _von expectation average, in this process, can adopt Model Selection technology in order to ensure Generalization Capability, such as, cross validation.For unknown parameter space, search out best parameter group θ _optbe from parameter space, to select limited parameter combinations to train generation model, then carry out Performance Evaluation, final output performance is evaluated and tested optimum parameter combinations.

For machine learning, parameter optimization is a challenge all the time, because there is the parameter combinations of magnanimity in the higher-dimension serial number space that has multiple parameters to form.Grid search is the most the most frequently used a kind of parameter optimization technology.The parameter of machine learning forms a mesh space, the given one group of feasible value of each parameter, and each combination represents a grid lattice point.Grid search adopts the mode of limit to test each given parameter combinations.Grid search technology realizes simple, and versatility is good, guarantees to find globally optimal solution in given parameter space.Its shortcoming is, adopts the way of search of limit to have scaling concern (calculated amount is index scale with parameter scale or search granularity and rises), makes computing cost large.

Between more than ten years, many optimisation techniques are developed the parameter optimization process for accelerating machine learning in the past.These roughly can be divided into two large classes: a class is numeric type optimization method, and another kind of is evolution type optimized algorithm.Numeric type parameter optimization technology, such as Gradient Descent.These class methods are obtained corresponding information and are determined the direction of search and the step-size in search of next iteration step by numerical evaluation, search is to have the specific direction therefore can Fast Convergent.Than other technologies, numeric type parameter optimization technology has very high search efficiency.But these class methods are easily absorbed in local optimum, its effect is extremely limited by initial point.Secondly,, for the optimizing problem of quantity of parameters, this class methods speed of convergence can obviously reduce.In addition, most of numeric type method is all embedded, need to carry out the derivation of equation and program realization targetedly to former algorithm, and versatility is poor.Evolution type parameter optimization technology, has comprised genetic algorithm, simulated annealing, and particle cluster algorithm, etc.Than numeric type method, these class methods can be avoided local optimum effectively, and guarantee to find the parameter combinations of approximate global optimum.Due to, evolutional algorithm mostly is random proximity search, there is no clear and definite direction and step-length, and speed of convergence is slow, length consuming time.

In machine learning field, another basic problem is feature tuning, has comprised that feature strengthens and feature selecting.A pre-treatment step of model training is that sample characteristics is carried out to convergent-divergent, the characteristic of sample is normalized to the hydraulic performance decline that causes model within the scope of unified codomain for avoiding factor value difference, otherwise can be according to the importance of feature, weights (zoom factor) are adjusted accordingly, object is for improving model performance, is referred to as feature and strengthens.Feature selecting, for removing redundancy and incoherent feature, finds a character subset for setting up model, thereby can reduce dimension and reduce the training time of model, even can strengthen the generalization of model.Feature selecting has comprised three types of technology: filtering type, and packaging type, and embedded.Filtering type technology, according to certain statistic, assesses and selects most important feature to feature, calculates simply still and lacks modelling verification, and precision is poor.Packaging type technology, the mode of employing iteration is selected a character subset at every turn, and generation model carries out Performance Evaluation, and the character subset accuracy obtaining is high, but computing cost is large.Embedded technology is carried out feature selecting in model training process, this need to change algorithm, and versatility is not strong.Except above-mentioned three classes, by feature zoom factor is carried out to tuning, can reach equally feature selecting.In the process of feature tuning, need to carry out equally parameter optimization and find optimal parameter could determine final model performance.Therefore, can consider that two processes are merged into a process carries out tuning simultaneously.

The zoom factor of feature is completed to feature tuning as a class parameter, and together with the parameter of machine learning algorithm, carry out optimizing meeting and cause a large amount of parameters.But, still lack quick, accurate, general, can be effective to Machine Learning Parameter optimizing technology, the especially optimizing in higher-dimension continuous parameter space at present.

Summary of the invention

The present invention is intended to solve at least to a certain extent one of technical matters in above-mentioned correlation technique.

For this reason, one object of the present invention is to propose a kind of method of parameter optimization for machine learning and feature tuning, and the method can promote result of calculation and the counting yield of processing parameter optimizing, and highly versatile.

Another object of the present invention is to provide the system of a kind of parameter optimization for machine learning and feature tuning.

To achieve these goals, the embodiment of first aspect present invention has proposed a kind of method of parameter optimization for machine learning and feature tuning, comprises the following steps: generate at random multiple parameter sets; Respectively described multiple parameter sets are carried out to the iteration optimization based on EnKF; Respectively the multiple parameter sets after optimizing are carried out to Performance Evaluation, and obtain gathering pond and supplementary parameter set according to assessment result, wherein, the performance of the parameter sets in described set pond is higher than the performance of the parameter sets in described supplementary parameter set; Parameter sets in parameter sets in pair set pond and described supplementary parameter set is carried out iteration optimization and the Performance Evaluation based on EnKF again, to obtain optimized parameter set.

According to the method for the parameter optimization for machine learning of the embodiment of the present invention and feature tuning, can be efficiently in the continuous parameter space of higher-dimension, find optimum solution; And simultaneously processing parameter optimizing of the method and two problems of feature tuning, can promote accuracy and the counting yield of the result of calculation of processing parameter optimizing; In addition, the method is carried out tuning using the zoom factor for the treatment of characteristic as parameter to feature; And the method highly versatile, is applicable to the various algorithms of machine learning.

In addition, the method for the parameter optimization for machine learning according to the above embodiment of the present invention and feature tuning can also have following additional technical characterictic:

In some instances, described described multiple parameter sets are carried out to the iteration optimization based on EnKF, specifically comprise: each parameter sets is trained with generation model on predetermined training dataset by machine learning algorithm; On predetermined verification msg collection, described model is carried out to Performance Evaluation; By EnKF algorithm, described multiple parameter sets are upgraded.

In some instances, the multiple parameter sets of described random generation, specifically comprise the following steps: generate at random a parameter vector θ ∈ R ^{m × 1}, wherein each parameter generates random value in predetermined parameter area; One group of normalized vector { ρ of random generation _i| ρ _i∈ R ^{m × 1}, i=1 ..., N}, carries out orthogonalization to guarantee disturbance linear independence to it; Generate parameter sets disturbance, specifically comprise:

A′＝(F _ar ₁ρ ₁，F _ar ₂ρ ₂，...，F _ar _Nρ _N)∈R ^m×N，r _i～N(0，S _p)，

Wherein, the disturbance of A ' expression parametric geometry, ρ _irepresent the random perturbation vector generating, variable r _irepresent arbitrary width, and Gaussian distributed, variance S _pconfigurable, matrix F _a=(f ₁e ₁, f ₂e ₂..., f _ne _n), e _irepresentation unit vector, f _ia configurable zoom variables, for adjusting perturbation amplitude; By parameter vector θ, add respectively disturbance set A ' in each group perturbation vector ε _i∈ A ', obtains one group of parameter θ _i=θ+ε _i, symbiosis becomes N group parameter to form parameter sets A; Repeat above-mentioned several step, to generate N _eindividual parameter sets.

In some instances, also comprise: described training dataset and verification msg collection are carried out to feature convergent-divergent; By the parameter input machine learning algorithm in parameter sets, concentrate and train with generation model at training data; By described model, each sample is predicted, obtained the estimated value of described model; Obtain the model of all parameters, and carry out Performance Evaluation, specifically comprise:

Wherein, the predicted value that adopts each group parameter generation model to obtain that HA has represented set-inclusion, n represents the number of verification msg collection sample.

In some instances, also comprise: by following formula, described parameter sets is upgraded:

A ^a＝A ^f+A′(HA′) ^T(HA′(HA′) ^T+γγ ^T) ^-1(D-HA)，

Wherein, A represents parameter sets, A ^frepresent current parameter sets, A ^arepresent the parameter sets of upgrading, the disturbance of A ' expression parameter sets, D represents observation set, and γ represents observation set disturbance, and HA represents model prediction results set, the set disturbance of HA ' expression HA.

In some instances, described respectively to optimize after multiple parameter sets carry out Performance Evaluation, and obtain gathering pond and supplementary parameter set according to assessment result, further comprise: according to assessment result, the multiple parameter sets after described optimization are divided three classes, establish the performance number that score (A) represents parameter sets; If score (A) >=thresh1, judges that this parameter sets performance is fine, and this parameter sets is kept in set pond; If score (A)≤thresh2, judges this parameter sets poor-performing, and abandons this parameter sets; If thresh2 < score (A) < is thresh1, judges that the performance of this parameter sets is general, and parameter sets general performance is merged at random between two, to generate supplementary parameter set.

In some instances, also comprise: in the general parameter sets of performance, choose a pair of parameter sets, generate new parameter sets by the merge algorithm based on EnKF, specifically comprise: suppose that a pair of parameter sets of choosing is A _i, A _j, calculate respectively Q by following formula _ijand Q _ji:

Calculate respectively A _ijand A _ji, wherein:

$A_{\mathrm{ij}}^m={\stackrel{\&OverBar;}{A}}_i+A_i^{\&prime;}{Q}_{\mathrm{ij}},$

$A_{\mathrm{ji}}^m={\stackrel{\&OverBar;}{A}}_j+A_j^{\&prime;}{Q}_{\mathrm{ji}};$

For Q _ijand Q _jicarry out respectively UR decomposition; From matrix and the middle N row with maximum pivot of selecting respectively, according to it in parameter sets and in select corresponding parameter vector and form last parameter sets A ^m; If parameter sets list to be combined is empty, the new parameter sets of random generation, otherwise select at random another to merge parameter sets.

The embodiment of second aspect present invention provides the system of a kind of parameter optimization for machine learning and feature tuning, comprising: generation module, and described generation module is used for generating multiple parameter sets; Optimize module, described optimization module is for carrying out the iteration optimization based on EnKF to described multiple parameter sets respectively; Evaluation module, described evaluation module is for carrying out Performance Evaluation to the multiple parameter sets after optimizing respectively, and obtain gathering pond and supplementary parameter set according to assessment result, wherein, the performance of the parameter sets in described set pond is higher than the performance of the parameter sets in described supplementary parameter set; Acquisition module, described acquisition module carries out iteration optimization and the Performance Evaluation based on EnKF again for the parameter sets in parameter sets and the described supplementary parameter set in pair set pond, to obtain optimized parameter set.

According to the system of the parameter optimization for machine learning of the embodiment of the present invention and feature tuning, can be efficiently in the continuous parameter space of higher-dimension, find optimum solution; And the simultaneously processing parameter optimizing of this system and two problems of feature tuning, can promote accuracy and the counting yield of the result of calculation of processing parameter optimizing; In addition, this system is carried out tuning using the zoom factor for the treatment of characteristic as parameter to feature; And this system highly versatile, is applicable to the various algorithms of machine learning.

In addition, the system of the parameter optimization for machine learning according to the above embodiment of the present invention and feature tuning can also have following additional technical characterictic:

In some instances, described optimization module is for training with generation model each parameter sets on predetermined training dataset by machine learning algorithm, and on predetermined verification msg collection, described model is carried out to Performance Evaluation, and by EnKF algorithm, described multiple parameter sets are upgraded.

In some instances, described generation module generates multiple parameter sets, specifically comprises: generate at random a parameter vector θ ∈ R ^{m × 1}, wherein each parameter generates random value in predetermined parameter area; One group of normalized vector { ρ of random generation _i| ρ _i∈ R ^{m × 1}, i=1 ..., N}, carries out orthogonalization to guarantee disturbance linear independence to it; Generate parameter sets disturbance, specifically comprise:

A′＝(F _ar ₁ρ ₁，F _ar ₂ρ ₂，...，F _ar _Nρ _N)∈R ^m×N，r _i～N(0，S _p)，

Wherein, the disturbance of A ' expression parametric geometry, ρ _irepresent the random perturbation vector generating, variable r _irepresent arbitrary width, and Gaussian distributed, variance S _pconfigurable, matrix F _a=(f ₁e ₁, f ₂e ₂..., f _ne _n), e _irepresentation unit vector, f _ia configurable zoom variables, for adjusting perturbation amplitude; By parameter vector θ, add respectively disturbance set A ' in each group perturbation vector ε _i∈ A ', obtains one group of parameter θ _i=θ+ε _i, symbiosis becomes N group parameter to form parameter sets A; Repeat above-mentioned several step, to generate N _eindividual parameter sets.

In some instances, described optimization module is also for carrying out feature convergent-divergent to described training dataset and verification msg collection; By the parameter input machine learning algorithm in parameter sets, concentrate and train with generation model at training data; By described model, each sample is predicted, obtained the estimated value of described model; Obtain the model of all parameters, and carry out Performance Evaluation, specifically comprise:

Wherein, the predicted value that adopts each group parameter generation model to obtain that HA has represented set-inclusion, n represents the number of verification msg collection sample.

In some instances, by following formula, described parameter sets is upgraded:

A ^a＝A ^f+A′(HA′) ^T(HA′(HA′) ^T+γγ ^T) ^-1(D-HA)，

Wherein, A represents parameter sets, A ^frepresent current parameter sets, A ^arepresent the parameter sets of upgrading, the disturbance of A ' expression parameter sets, D represents observation set, and γ represents observation set disturbance, and HA represents model prediction results set, the set disturbance of HA ' expression HA.

In some instances, described evaluation module is for being divided three classes the multiple parameter sets after described optimization according to assessment result, if score (A) represents the performance number of parameter sets, and in the time of score (A) >=thresh1, judge that this parameter sets performance is fine, and this parameter sets is kept in set pond, and in the time of score (A)≤thresh2, judge this parameter sets poor-performing, and abandon this parameter sets, and in the time of thresh2 < score (A) < thresh1, the performance of this parameter sets is general, and parameter sets general performance is merged at random between two, to generate supplementary parameter set.

In some instances, described evaluation module also, for choosing a pair of parameter sets in the general parameter sets of performance, generates new parameter sets by the merge algorithm based on EnKF, specifically comprises: suppose that a pair of parameter sets of choosing is A _i, A _j, calculate respectively Q by following formula _ijand Q _ji:

Calculate respectively A _ijand A _ji, wherein:

$A_{\mathrm{ij}}^m={\stackrel{\&OverBar;}{A}}_i+A_i^{\&prime;}{Q}_{\mathrm{ij}},$

$A_{\mathrm{ji}}^m={\stackrel{\&OverBar;}{A}}_j+A_j^{\&prime;}{Q}_{\mathrm{ji}};$

For Q _ijand Q _jicarry out respectively UR decomposition; From matrix and the middle N row with maximum pivot of selecting respectively, according to it in parameter sets and in select corresponding parameter vector and form last parameter sets A ^m; If parameter sets list to be combined is empty, the new parameter sets of random generation, otherwise select at random another to merge parameter sets.

Additional aspect of the present invention and advantage in the following description part provide, and part will become obviously from the following description, or recognize by practice of the present invention.

Brief description of the drawings

Above-mentioned and/or additional aspect of the present invention and advantage accompanying drawing below combination is understood becoming the description of embodiment obviously and easily, wherein:

Fig. 1 is the process flow diagram for the parameter optimization of machine learning and the method for feature tuning according to an embodiment of the invention;

Fig. 2 is the principle schematic for the parameter optimization of machine learning and the method for feature tuning in accordance with another embodiment of the present invention;

Fig. 3 is the process flow diagram single set being upgraded according to one embodiment of the invention;

Fig. 4 is the operational flow diagram of gathering according to an embodiment of the invention evolution;

Fig. 5 is the operational flow diagram that merges according to an embodiment of the invention search;

Fig. 6 is the process flow diagram for the parameter optimization of machine learning and the method for feature tuning in accordance with another embodiment of the present invention; And

Fig. 7 is the structured flowchart for the parameter optimization of machine learning and the system of feature tuning according to an embodiment of the invention.

Embodiment

Describe embodiments of the invention below in detail, the example of described embodiment is shown in the drawings, and wherein same or similar label represents same or similar element or has the element of identical or similar functions from start to finish.Be exemplary below by the embodiment being described with reference to the drawings, only for explaining the present invention, and can not be interpreted as limitation of the present invention.

Describe according to the method and system of the parameter optimization for machine learning of the embodiment of the present invention and feature tuning below in conjunction with accompanying drawing.

Fig. 1 is the process flow diagram for the parameter optimization of machine learning and the method for feature tuning according to an embodiment of the invention.Fig. 6 is the process flow diagram for the parameter optimization of machine learning and the method for feature tuning in accordance with another embodiment of the present invention.Shown in Fig. 1 and Fig. 6, the method comprises the following steps:

Step S101, generates multiple parameter sets at random.

If feature strengthens function and do not start, parameter sets has comprised the parameter of machine learning algorithm; If feature tuning function on, parameter sets has comprised parameter and the feature zoom factor of machine learning algorithm.Suppose that parameter sets A represents, wherein set member θ _irepresent one group of parameter (in EnKF algorithm as a group system state), A is the state matrix of a m × N, and wherein, m represents number of parameters, and N represents set member's number.In some instances, suppose N _erepresent the number of parameter sets, the random concrete steps that generate of parameter sets comprise:

Step 1-1: generate at random a parameter vector wherein each parameter generates random value in predetermined parameter area.

Step 1-2: generate at random one group of normalized vector { ρ _i| ρ _i∈ R ^{m × 1}, i=1 ..., N}, carries out orthogonalization to guarantee disturbance linear independence to it.

Step 1-3: generate parameter sets disturbance, specifically comprise:

A′＝(F _ar ₁ρ ₁，F _ar ₂ρ ₂，...，F _ar _Nρ _N)∈R ^m×N，r _i～N(0，S _p)，

Wherein, the disturbance of A ' expression parametric geometry, ρ _irepresent the random perturbation vector generating, variable r _irepresent arbitrary width, and Gaussian distributed, variance S _pconfigurable, matrix F _a=(f ₁e ₁, f ₂e ₂..., f _ne _n), e _irepresentation unit vector, f _ia configurable zoom variables, for adjusting perturbation amplitude.

Step 1-4: by parameter vector θ, add respectively disturbance set A ' in each group perturbation vector ε _i∈ A ', obtains one group of parameter θ _i=θ+ε _i, symbiosis becomes N group parameter to form parameter sets A.

Step 1-5: repeat above-mentioned steps, repeated execution of steps 1-1 is to step 1-4, to generate N _eindividual parameter sets.

Step S102, carries out the iteration optimization based on EnKF to multiple parameter sets respectively.In some instances, specifically comprise: each parameter sets is trained with generation model on predetermined training dataset by machine learning algorithm, and on predetermined verification msg collection, model is carried out to Performance Evaluation, finally by EnKF algorithm, multiple parameter sets are upgraded.More specifically, as shown in Figure 3, this step comprises the following steps:

Step 2-1: training dataset (is for example denoted as to X _t) and verification msg collection (be for example denoted as X _v) carry out feature convergent-divergent.

If the function of feature tuning does not start, the characteristic unification of each sample is normalized in the codomain of appointment, these normalized training datas and verification msg will be used until whole optimizing finishes always.If feature tuning function on, supposes zoom factor δ _j∈ θ _iif do not carry out feature selecting, according to δ _jeach feature is carried out to the convergent-divergent on numerical value; If carry out feature selecting, first zoom factor is normalized to [0,1], by formula δ _i/ δ _max, then select the zoom factor that is greater than given codomain to carry out the convergent-divergent on numerical value to corresponding feature, only have the characteristic after convergent-divergent train and verify, all δ _iparticipating in EnKF calculates.If feature tuning function does not start, step 2-1 only carries out first, otherwise this step is carried out at every turn.

Step 2-2: by θ _iin parameter be input to machine learning algorithm, and at training dataset X _tin train generation model, and, conventionally adopt Model Selection technology to guarantee Generalization Capability.

Step 2-3: by model, each sample is predicted, obtained the estimated value of model.More specifically, adopt θ _ithe model generating is at verification msg collection X _vupper, each sample is predicted, export a rank value, i.e. the estimated value of model output, supposes H θ _irepresent to comprise the vector predicting the outcome.

Step 2-4: obtain the model of all parameters, and carry out Performance Evaluation.Specifically, to each group parameter θ _i∈ A, repeating step 2-1 is to step 2-3, and generation model also carries out Performance Evaluation, suppose that HA has represented set-inclusion and adopted each to organize predicted value that parameter generation model obtains (in EnKF algorithm, its expression state is to the mapping of observation), HA can be expressed as:

Wherein, n represents the number of verification msg collection sample.

Further, for each parameter sets A, repeat above-mentioned steps and train assessment, obtain corresponding HA.

Further, in some instances, after step S102, also can comprise the following steps:

Step 3: generate observation set, suppose that D represents observation set, D is expressed as:

Wherein, represent i group observation vector.Observation represents given sample prior probability, if the prior probability of sample is not provided, gives an initial observation value, supposes d ₀represent the initial observation that observation vector has comprised sample, the concrete steps that observation set generates comprise:

Step 3-1: generate at random one group of normalized vector, if sample size n is few, random vector is carried out to orthogonalization, otherwise do not carry out.

Step 3-2: generate random observation disturbance, suppose that γ represents observation set disturbance:

Wherein, represent the random perturbation vector generating in step 3-1, variable v _irepresent arbitrary width, and Gaussian distributed, wherein variance S _oconfigurable.

Step 3-3: by initial observation vector d ₀, add respectively each group perturbation vector v _iβ _i∈ γ, calculates an observation vector d _i=d ₀+ v _iβ _i, symbiosis becomes the observation of N group to form observation set D.

Step 3-4: repeating step 3-1, to step 3-3, generates N _eindividual observation disturbance set and observation set.

In step S102, by following formula, parameter sets is upgraded:

A ^a＝A ^f+A′(HA′) ^T(HA′(HA′) ^T+γγ ^T) ^-1(D-HA)，

Wherein, A represents parameter sets, A ^frepresent current parameter sets, A ^arepresent the parameter sets of upgrading, the disturbance of A ' expression parameter sets, D represents observation set, and γ represents observation set disturbance, and HA represents model prediction results set, the set disturbance of HA ' expression HA.Further, HA ' calculates by following formula:

$\stackrel{\&OverBar;}{\mathrm{HA}}={\mathrm{HAM}}_N,$

${\mathrm{HA}}^{\&prime;}=\mathrm{HA}-\stackrel{\&OverBar;}{\mathrm{HA}},$

Wherein, represent ensemble average value, in each element be in the computation process of EnKF, due to (HA ' (HA ') ^t+ γ γ ^t) ^-1computing cost excessive, therefore adopt householder conversion to carry out UR and decompose to optimize calculating.Suppose represent HA ', X (i, j) represent an element in X, X (:, j) represent row wherein, make τ represent the border of a relative residual error, make 1 > > τ > 0, for each the row i=1 in X ..., N, carries out following steps:

Step 4-1: calculate the residual norm of remaining columns, computing formula is as follows:

ResNorm(k)＝||X(i：n，k)|| ₂?k＝i，...，N。

Step 4-2: if $\frac{\mathrm{\&tau;}}{1-\mathrm{\&tau;}}{\mathrm{\&Sigma;}}_{l=1}^{i-1}\left|X(l,l)\right|>(N-i+1)\mathrm{ResNorm}\left(\hat{k}\right),$ (wherein, represent the row number of maximum residul difference norm), p=i-1 so, circulation finishes, and goes to step 4-7; Otherwise p=i, exchanges two row

Step 4-3: initialization vector wherein ${\mathrm{\&omega;}}_i\left(k\right)=\left\{\begin{array}{cc}0& k<i\\ X(k,i)& k\&GreaterEqual;i\end{array}\right..$

Step 4-4: calculate $\mathrm{Norm}=\mathrm{ResNorm}\left(\hat{k}\right),$ And

If X (i, i) > 0, ω so _i=ω _i+ Norm × e _i, upgrade X (i), meet following condition:

$X(i,k)=\left\{\begin{array}{cc}X(i,k)& k<i\\ -\mathrm{Norm}& k=i\\ 0& k>i\end{array}\right.$

Otherwise, ω _i=ω _i-Norm × e _i, upgrade X (i), meet following condition:

$X(i,k)=\left\{\begin{array}{cc}X(i,k)& k<i\\ +\mathrm{Norm}& k=i\\ 0& k>i\end{array}\right..$

Step 4-5: calculate ω _i=ω _i/ || ω _i|| ₂, to k=i+1:N, carry out following renewal and calculate:

X(i：n，k)＝X(i：n，k)-2ω _i(i：n)(ω _i(i：n) ^TX(i：n，k))。

Step 4-6:i=i+1, if i > is N, circulation finishes, and goes to step 4-7; Otherwise, start to carry out from step 4-1.

Step 4-7: set up matrix s is defined as S=X (:, 1:p).

After the UR of step 4-7 decomposes, obtain following estimation formulas by above-mentioned steps 4-1:

${({\mathrm{HA}}^{\&prime;}{\left({\mathrm{HA}}^{\&prime;}\right)}^T+{\mathrm{\&gamma;\&gamma;}}^T)}^{-1}\&ap;U\left(\begin{array}{cc}{\left({\hat{S}\hat{S}}^T\right)}^{-1}& 0\\ 0& 0\end{array}\right)U^T,$

Wherein, U represents orthogonal matrix, represent the row of non-zero entry in upper triangular matrix Cover matrix S, matrix row by p row diagonal element absolute value maximum form.After decomposing, matrix U is made up of p householder conversion, is expressed as U=H (ω ₁) H (ω ₂) ... H (ω _p), wherein householder conversion H (ω _i) be defined as H (ω _i)=I-2 ω _iω _i ^t.So, the EnKF computing formula of optimization is converted into:

$A^a=A^f+A^{\&prime;}{\left({\mathrm{HA}}^{\&prime;}\right)}^{T}U\left(\begin{array}{cc}{\left({\hat{S}\hat{S}}^T\right)}^{-1}& 0\\ 0& 0\end{array}\right)U^T(D-\mathrm{HA}),$

After UR has decomposed, for the consideration of counting yield, in above-mentioned formula, residue is calculated strictly and is carried out from right to left, finally obtains the parameter sets A upgrading ^a.

Step 5: reach the bull wheel number of setting if EnKF upgrades wheel number, go to step S102, the parameter sets A obtaining with renewal ^atraining pattern and carry out Performance Evaluation, then goes to step 6 described later; Otherwise, go to step S102 and carry out the renewal calculating of EnKF next time.

Step S103, carries out Performance Evaluation to the multiple parameter sets after optimizing respectively, and obtains gathering pond and supplementary parameter set according to assessment result, and wherein, the performance of the parameter sets in set pond is higher than the performance of the parameter sets in supplementary parameter set.As shown in Figure 4, specifically comprise the following steps:

Step 6: upgrade in computation process at each EnKF, preserve parameter sets A and corresponding HA in all EnKF renewal processes, it is carried out to unified Performance Evaluation, export parameter sets optimum in this process, the calculation task of each parameter sets is exported one group of A and HA.In addition, all Performance Evaluation results will go on record.

Step 7: if the wheel number that set is evolved reaches the bull wheel number of setting, go to step 14 described later; Otherwise, go to aftermentioned step 8, proceed set and evolve.

Step 8: collect the parameter sets of each calculation task output and predict the outcome, gather assessment, and according to assessment result, the multiple parameter sets after optimizing are divided three classes, particularly, adopt two threshold values that parameter sets is divided into three classes, suppose that score (A) represents the performance number of parameter sets, assessment mode is as follows:

If score (A)>=thresh1, judges that this parameter sets performance is fine, and this parameter sets is kept in set pond, if the quantity in set pond is N _e, go to aftermentioned step 12; Otherwise, continue to carry out this step.

If score (A)≤thresh2, judges this parameter sets poor-performing, and abandons this parameter sets.

If thresh2 < score (A) < is thresh1, judges that the performance of this parameter sets is general, and parameter sets general performance is merged at random between two, to generate supplementary parameter set.After having assessed all parameter sets, if there is the parameter sets that need to merge, go to aftermentioned step 9; Otherwise, go to aftermentioned step 10.

Step 9: the parameter sets general to performance, choose a pair of parameter sets, then adopt the merge algorithm based on EnKF to generate new parameter sets A ^m, specifically comprise the following steps:

Step 9-1: suppose that a pair of parameter sets of choosing is A _i, A _j, calculate respectively Q by following formula _ijand Q _ji:

Wherein, above-mentioned calculating has adopted the optimization in step 4 to calculate.

Step 9-2: calculate respectively A _ijand A _ji, be specially;

$A_{\mathrm{ij}}^m={\stackrel{\&OverBar;}{A}}_i+A_i^{\&prime;}{Q}_{\mathrm{ij}},$

$A_{\mathrm{ji}}^m={\stackrel{\&OverBar;}{A}}_j+A_j^{\&prime;}{Q}_{\mathrm{ji}}.$

Step 9-3: to Q _ijand Q _jicarry out respectively UR decomposition.Be specially:

$Q_{\mathrm{ij}}~{\tilde{U}}_{\mathrm{ij}}{\tilde{S}}_{\mathrm{ij}}{\tilde{V}}_{\mathrm{ij}},$

Wherein, represent a upper triangular matrix, wherein matrix column according to the absolute value of pivot (diagonal element) by arranging from big to small, represent permutation matrix, for row being exchanged in the time selecting pivot.

Step 9-4: from matrix and select respectively the N row with maximum pivot (diagonal element), according to it in parameter sets and in select corresponding parameter vector and form last parameter sets A ^m.

Step 9-5: if parameter sets list to be combined is empty, go to aftermentioned step 10, i.e. the new parameter sets of random generation, otherwise select at random another to merge parameter sets, repeated execution of steps 9-1 is to step 9-5.

Step S104, the parameter sets in the parameter sets in pair set pond and supplementary parameter set is carried out iteration optimization and the Performance Evaluation based on EnKF again, to obtain optimized parameter set.As shown in Figure 4, specifically comprise the following steps:

Step 10: if having parameter sets to be dropped or merge, perform step that S101 is random generates new parameter sets.

Step 11: the set of carrying out a new round through the parameter sets merging or generate is at random evolved, and go to step S102.

Step 12: after set phylogenetic scale finishes, merge search, as shown in Figure 2.If search iteration step is initial merging, the parameter sets number in set pond is 1, goes to aftermentioned step 15; Otherwise, from set pond, choose a pair of parameter sets and merge the new parameter sets of generation, and perform step 9-1 to step 9-4.

Step 13: each newly-generated parameter sets is carried out to EnKF renewal, go to step S102.

Step 14: after EnKF upgrades, the parameter sets of collecting this process performance optimum, then gathers assessment, joins the parameter sets of score (A) >=thresh1 in set pond.Then, go to step 12 and carry out the fusion search of next iteration step.

Step 15: find out one group of best parameter of performance evaluating from record, as optimized parameter set output.

In sum, method of the present invention has adopted EnKF technology, can be used in parameter optimization and the feature tuning two aspect problems of machine learning, using feature zoom factor as a class parameter, come processing feature enhancing and feature selecting by zoom factor being carried out to optimizing, and can together carry out parameter optimization with the parameter of machine learning algorithm.As shown in Figure 2, the method has mainly comprised that set is evolved and fusion two stages of search.

More specifically, method of the present invention is based on EnKF technology, and EnKF technology can be estimated the nonlinear problem that comprises a large amount of stochastic variables, has set up a framework based on EnKF technology simultaneously, and has adopted multiple optimisation technique.Specifically, first, the method has adopted EnKF technology, regards the parameter optimization of machine learning and the unification of feature tuning problem as the state using parameter as system is estimated for a nonlinear system; Secondly, the method has been set up a framework based on EnKF, adopt many integrated technologies to be easily absorbed in the problem of local optimum for numerical optimization, adopt set evolution technology, assess through the parameter sets after EnKF calculating multiple, retention property is good, abandon poor performance, merge performance general, be mainly used in improving hunting zone and search efficiency, after set evolution finishes, adopt and merge search technique, merge the good parameter sets of performance, further search for, guarantee to find approximate optimal solution in higher dimensional space; Then, because the data volume of machine learning is large, make in EnKF computation process, the calculating of part matrix computing and storage overhead are very large, have adopted efficient UR decomposition technique for this reason, increase the operation efficiency of EnKF, thereby have strengthened practicality.

According to the method for the parameter optimization for machine learning of the embodiment of the present invention and feature tuning, can be efficiently in the continuous parameter space of higher-dimension, find optimum solution; And simultaneously processing parameter optimizing of the method and two problems of feature tuning, can promote accuracy and the counting yield of the result of calculation of processing parameter optimizing; In addition, the method is carried out tuning using the zoom factor for the treatment of characteristic as parameter to feature; And the method highly versatile, is applicable to the various algorithms of machine learning.

Further embodiment of the present invention also provides the system of a kind of parameter optimization for machine learning and feature tuning.

Fig. 7 is the structured flowchart for the parameter optimization of machine learning and the system of feature tuning according to an embodiment of the invention.As shown in Figure 7, for the parameter optimization of machine learning and the system 700 of feature tuning, comprising according to an embodiment of the invention: generation module 710, optimization module 720, evaluation module 730 and acquisition module 740.

Wherein, generation module 710 is for generating multiple parameter sets.

If feature strengthens function and do not start, parameter sets has comprised the parameter of machine learning algorithm; If feature tuning function on, parameter sets has comprised parameter and the feature zoom factor of machine learning algorithm.Suppose that parameter sets A represents, wherein set member θ _irepresent one group of parameter (in EnKF algorithm as a group system state), A is the state matrix of a m × N, and wherein, m represents number of parameters, and N represents set member's number.In some instances, suppose N _erepresent the number of parameter sets, the random concrete steps that generate parameter sets of generation module 710 comprise:

Step 1-1: generate at random a parameter vector wherein each parameter generates random value in predetermined parameter area.

Step 1-2: generate at random one group of normalized vector { ρ _i| ρ _i∈ R ^{m × 1}, i=1 ..., N}, carries out orthogonalization to guarantee disturbance linear independence to it.

Step 1-3: generate parameter sets disturbance, specifically comprise:

A′＝(F _ar ₁ρ ₁，F _ar ₂ρ ₂，...，F _ar _Nρ _N)∈R ^m×N，r _i～N(0，S _p)，

Wherein, the disturbance of A ' expression parametric geometry, ρ _irepresent the random perturbation vector generating, variable r _irepresent arbitrary width, and Gaussian distributed, variance S _pconfigurable, matrix F _a=(f ₁e ₁, f ₂e ₂..., f _ne _n), e _irepresentation unit vector, f _ia configurable zoom variables, for adjusting perturbation amplitude.

Step 1-4: by parameter vector θ, add respectively disturbance set A ' in each group perturbation vector ε _i∈ A ', obtains one group of parameter θ _i=θ+ε _i, symbiosis becomes N group parameter to form parameter sets A.

Step 1-5: repeat above-mentioned steps, repeated execution of steps 1-1 is to step 1-4, to generate N _eindividual parameter sets.

Optimize module 720 for respectively multiple parameter sets being carried out to the iteration optimization based on EnKF.In some instances, specifically comprise: having module 720 is trained with generation model each parameter sets on predetermined training dataset by machine learning algorithm, and on predetermined verification msg collection, model is carried out to Performance Evaluation, finally by EnKF algorithm, multiple parameter sets are upgraded.More specifically, as shown in Figure 3, this this process can comprise the following steps:

Step 2-1: training dataset (is for example denoted as to X _t) and verification msg collection (be for example denoted as X _v) carry out feature convergent-divergent.

If the function of feature tuning does not start, the characteristic unification of each sample is normalized in the codomain of appointment, these normalized training datas and verification msg will be used until whole optimizing finishes always.If feature tuning function on, supposes zoom factor δ _j∈ θ _iif do not carry out feature selecting, according to δ _jeach feature is carried out to the convergent-divergent on numerical value; If carry out feature selecting, first zoom factor is normalized to [0,1], by formula δ _i/ δ _max, then select the zoom factor that is greater than given codomain to carry out the convergent-divergent on numerical value to corresponding feature, only have the characteristic after convergent-divergent train and verify, all δ _iparticipating in EnKF calculates.If feature tuning function does not start, step 2-1 only carries out first, otherwise this step is carried out at every turn.

Step 2-2: by θ _iin parameter be input to machine learning algorithm, and at training dataset X _tin train generation model, and, conventionally adopt Model Selection technology to guarantee Generalization Capability.

Step 2-3: by model, each sample is predicted, obtained the estimated value of model.More specifically, adopt θ _ithe model generating is at verification msg collection X _vupper, each sample is predicted, export a rank value, i.e. the estimated value of model output, supposes H θ _irepresent to comprise the vector predicting the outcome.

Step 2-4: obtain the model of all parameters, and carry out Performance Evaluation.Specifically, to each group ginseng θ _i∈ A number, repeating step 2-1 is to step 2-3, and generation model also carries out Performance Evaluation, suppose that HA has represented set-inclusion and adopted each to organize predicted value that parameter generation model obtains (in EnKF algorithm, its expression state is to the mapping of observation), HA can be expressed as:

Wherein, n represents the number of verification msg collection sample.

Further, for each parameter sets A, repeat above-mentioned steps and train assessment, obtain corresponding HA.

Further, in some instances, after said process, also can comprise the following steps:

Step 3: generate observation set, suppose that D represents observation set, D is expressed as:

Wherein, represent i group observation vector.Observation represents given sample prior probability, if the prior probability of sample is not provided, gives an initial observation value, supposes d ₀represent the initial observation that observation vector has comprised sample, the concrete steps that observation set generates comprise:

Step 3-1: generate at random one group of normalized vector, if sample size n is few, random vector is carried out to orthogonalization, otherwise do not carry out.

Step 3-2: generate random observation disturbance, suppose that γ represents observation set disturbance:

Wherein, represent the random perturbation vector generating in step 3-1, variable v _irepresent arbitrary width, and Gaussian distributed, wherein variance S _oconfigurable.

Step 3-3: by initial observation vector d ₀, add respectively each group perturbation vector v _iβ _i∈ γ, calculates an observation vector d _i=d ₀+ v _iβ _i, symbiosis becomes the observation of N group to form observation set D.

Step 3-4: repeating step 3-1, to step 3-3, generates N _eindividual observation disturbance set and observation set.

Wherein, in this example, optimize module 720 and by following formula, parameter sets upgraded:

A ^a＝A ^f+A′(HA′) ^T(HA′(HA′) ^T+γγ ^T) ^-1(D-HA)，

Wherein, A represents parameter sets, A ^frepresent current parameter sets, A ^arepresent the parameter sets of upgrading, the disturbance of A ' expression parameter sets, D represents observation set, and γ represents observation set disturbance, and HA represents model prediction results set, the set disturbance of HA ' expression HA.。Further, HA ' calculates by following formula:

$\stackrel{\&OverBar;}{\mathrm{HA}}={\mathrm{HAM}}_N,$

${\mathrm{HA}}^{\&prime;}=\mathrm{HA}-\stackrel{\&OverBar;}{\mathrm{HA}},$

Wherein, represent ensemble average value, in each element be in the computation process of EnKF, due to (HA ' (HA ') ^t+ γ γ ^t) ^-1computing cost excessive, therefore adopt householder conversion to carry out UR and decompose to optimize calculating.Suppose represent HA ', X (i, j) represent an element in X, X (:, j) represent row wherein, make τ represent the border of a relative residual error, make 1 > > τ > 0, for each the row i=1 in X ..., N, carries out following steps:

Step 4-1: calculate the residual norm of remaining columns, computing formula is as follows:

ResNorm(k)＝||X(i：n，k)|| ₂?k＝i，...，N。

Step 4-2: if $\frac{\mathrm{\&tau;}}{1-\mathrm{\&tau;}}{\mathrm{\&Sigma;}}_{l=1}^{i-1}\left|X(l,l)\right|>(N-i+1)\mathrm{ResNorm}\left(\hat{k}\right),$ (wherein, represent the row number of maximum residul difference norm), p=i-1 so, circulation finishes, and goes to aftermentioned step 4-7; Otherwise p=i, exchanges two row $X(:,\hat{k})\&LeftRightArrow;X(:,i).$

Step 4-3: initialization vector wherein ${\mathrm{\&omega;}}_i\left(k\right)=\left\{\begin{array}{cc}0& k<i\\ X(k,i)& k\&GreaterEqual;i\end{array}\right..$

Step 4-4: calculate $\mathrm{Norm}=\mathrm{ResNorm}\left(\hat{k}\right),$ And

If X (i, i) > 0, ω so _i=ω _i+ Norm × e _i, upgrade X (i), meet following condition:

$X(i,k)=\left\{\begin{array}{cc}X(i,k)& k<i\\ -\mathrm{Norm}& k=i\\ 0& k>i\end{array}\right.$

Otherwise ω _i=ω _i-Norm × e _i,, upgrade X (i), meet following condition:

$X(i,k)=\left\{\begin{array}{cc}X(i,k)& k<i\\ +\mathrm{Norm}& k=i\\ 0& k>i\end{array}\right..$

Step 4-5: calculate ω _i=ω _i/ || ω _i|| ₂, to k=i+1:N, carry out following renewal and calculate:

X(i：n，k)＝X(i：n，k)-2ω _i(i：n)(ω _i(i：n) ^TX(i：n，k))。

Step 4-6:i=i+1, if i > is N, circulation finishes, and goes to aftermentioned step 4-7; Otherwise, start to carry out from step 4-1.

Step 4-7: set up matrix s is defined as S=X (:, 1:p).

After the UR of step 4-7 decomposes, obtain following estimation formulas by above-mentioned steps 4-1:

${({\mathrm{HA}}^{\&prime;}{\left({\mathrm{HA}}^{\&prime;}\right)}^T+{\mathrm{\&gamma;\&gamma;}}^T)}^{-1}\&ap;U\left(\begin{array}{cc}{\left({\hat{S}\hat{S}}^T\right)}^{-1}& 0\\ 0& 0\end{array}\right)U^T,$

Wherein, U represents orthogonal matrix, represent the row of non-zero entry in upper triangular matrix Cover matrix S, matrix row by p row diagonal element absolute value maximum form.After decomposing, matrix U is made up of p householder conversion, is expressed as U=H (ω ₁) H (ω ₂) ... H (ω _p), wherein householder conversion H (ω _i) be defined as H (ω _i)=I-2 ω _iω _i ^t.So, the EnKF computing formula of optimization is converted into:

$A^a=A^f+A^{\&prime;}{\left({\mathrm{HA}}^{\&prime;}\right)}^{T}U\left(\begin{array}{cc}{\left({\hat{S}\hat{S}}^T\right)}^{-1}& 0\\ 0& 0\end{array}\right)U^T(D-\mathrm{HA}),$

After UR has decomposed, for the consideration of counting yield, in above-mentioned formula, residue is calculated strictly and is carried out from right to left, finally obtains the parameter sets A upgrading ^a.

Step 5: reach the bull wheel number of setting if EnKF upgrades wheel number, go to step S102, the parameter sets A obtaining with renewal ^atraining pattern and carry out Performance Evaluation, then goes to step 6 described later; Otherwise, carry out the renewal of EnKF next time and calculate.

Evaluation module 730, for respectively the multiple parameter sets after optimizing being carried out to Performance Evaluation, and obtains gathering pond and supplementary parameter set according to assessment result, and wherein, the performance of the parameter sets in set pond is higher than the performance of the parameter sets in supplementary parameter set.In some instances, this process can be summarized as following steps:

Step 6: upgrade in computation process at each EnKF, preserve parameter sets A and corresponding HA in all EnKF renewal processes, it is carried out to unified Performance Evaluation, export parameter sets optimum in this process, the calculation task of each parameter sets is exported one group of A and HA.In addition, all Performance Evaluation results will go on record.

Step 7: if the wheel number that set is evolved reaches the bull wheel number of setting, go to step 14 described later; Otherwise, go to step 8 described later, proceed set and evolve.

Step 8: collect the parameter sets of each calculation task output and predict the outcome, gather assessment, and according to assessment result, the multiple parameter sets after optimizing are divided three classes, particularly, adopt two threshold values that parameter sets is divided into three classes, suppose that score (A) represents the performance number of parameter sets, assessment mode is as follows:

If score (A)>=thresh1, judges that this parameter sets performance is fine, and this parameter sets is kept in set pond, if the quantity in set pond is N _e, go to aftermentioned step 12; Otherwise, continue to carry out this step.

If score (A)≤thresh2, judges this parameter sets poor-performing, and abandons this parameter sets.

If thresh2 < score (A) < is thresh1, judges that the performance of this parameter sets is general, and parameter sets general performance is merged at random between two, to generate supplementary parameter set.After having assessed all parameter sets, if there is the parameter sets that need to merge, go to aftermentioned step 9; Otherwise, go to aftermentioned step 10.

Step 9: the parameter sets general to performance, choose a pair of parameter sets, then adopt the merge algorithm based on EnKF to generate new parameter sets A ^m, specifically comprise the following steps:

Step 9-1: suppose that a pair of parameter sets of choosing is A _i, A _j, calculate respectively Q by following formula _ijand Q _ji:

Wherein, above-mentioned calculating has adopted the optimization in step 4 to calculate.

Step 9-2: calculate respectively A _ijand A _ji, be specially;

$A_{\mathrm{ij}}^m={\stackrel{\&OverBar;}{A}}_i+A_i^{\&prime;}{Q}_{\mathrm{ij}},$

$A_{\mathrm{ji}}^m={\stackrel{\&OverBar;}{A}}_j+A_j^{\&prime;}{Q}_{\mathrm{ji}}.$

Step 9-3: to Q _ijand Q _jicarry out respectively UR decomposition.Be specially:

$Q_{\mathrm{ij}}~{\tilde{U}}_{\mathrm{ij}}{\tilde{S}}_{\mathrm{ij}}{\tilde{V}}_{\mathrm{ij}},$

Wherein, represent a upper triangular matrix, wherein matrix column according to the absolute value of pivot (diagonal element) by arranging from big to small, represent permutation matrix, for row being exchanged in the time selecting pivot.

Step 9-4: from matrix and select respectively the N row with maximum pivot (diagonal element), according to it in parameter sets and in select corresponding parameter vector and form last parameter sets A ^m.

Step 9-5: if parameter sets list to be combined is empty, go to aftermentioned step 10, i.e. the new parameter sets of random generation, otherwise select at random another to merge parameter sets, repeated execution of steps 9-1 is to step 9-5.

Acquisition module 740 carries out iteration optimization and the Performance Evaluation based on EnKF again for the parameter sets in parameter sets and the supplementary parameter set in pair set pond, to obtain optimized parameter set.In some instances, this process can be summarized as following steps:

Step 10: if having the parameter sets to be dropped or merge, generate new parameter sets at random.

Step 11: the set of carrying out a new round through the parameter sets merging or generate is at random evolved.

Step 12: after set phylogenetic scale finishes, merge search, as shown in Figure 2.If search iteration step is initial merging, the parameter sets number in set pond is 1, goes to aftermentioned step 15; Otherwise, from set pond, choose a pair of parameter sets and merge the new parameter sets of generation, and perform step 9-1 to step 9-4.

Step 13: each newly-generated parameter sets is carried out to EnKF renewal.

Step 14: after EnKF upgrades, the parameter sets of collecting this process performance optimum, then gathers assessment, joins the parameter sets of score (A) >=thresh1 in set pond.Then, go to step 12 and carry out the fusion search of next iteration step.

Step 15: find out one group of best parameter of performance evaluating from record, as optimized parameter set output.

In sum, system of the present invention has adopted EnKF technology, can be used in parameter optimization and the feature tuning two aspect problems of machine learning, using feature zoom factor as a class parameter, come processing feature enhancing and feature selecting by zoom factor being carried out to optimizing, and can together carry out parameter optimization with the parameter of machine learning algorithm.As shown in Figure 2, when this system is moved, mainly comprised that set is evolved and fusion two stages of search.

More specifically, system of the present invention is based on EnKF technology, and EnKF technology can be estimated the nonlinear problem that comprises a large amount of stochastic variables, has set up a framework based on EnKF technology simultaneously, and has adopted multiple optimisation technique.Specifically, first, this system has adopted EnKF technology, regards the parameter optimization of machine learning and the unification of feature tuning problem as the state using parameter as system is estimated for a nonlinear system; Secondly, this system made a framework based on EnKF, adopt many integrated technologies to be easily absorbed in the problem of local optimum for numerical optimization, adopt set evolution technology, assess through the parameter sets after EnKF calculating multiple, retention property is good, abandon poor performance, merge performance general, be mainly used in improving hunting zone and search efficiency, after set evolution finishes, adopt and merge search technique, merge the good parameter sets of performance, further search for, guarantee to find approximate optimal solution in higher dimensional space; Then, because the data volume of machine learning is large, make in EnKF computation process, the calculating of part matrix computing and storage overhead are very large, have adopted efficient UR decomposition technique for this reason, increase the operation efficiency of EnKF, thereby have strengthened practicality.

According to the system of the parameter optimization for machine learning of the embodiment of the present invention and feature tuning, can be efficiently in the continuous parameter space of higher-dimension, find optimum solution; And the simultaneously processing parameter optimizing of this system and two problems of feature tuning, can promote accuracy and the counting yield of the result of calculation of processing parameter optimizing; In addition, this system is carried out tuning using the zoom factor for the treatment of characteristic as parameter to feature; And this system highly versatile, is applicable to the various algorithms of machine learning.

In description of the invention, it will be appreciated that, term " " center ", " longitudinally ", " laterally ", " length ", " width ", " thickness ", " on ", D score, " front ", " afterwards ", " left side ", " right side ", " vertically ", " level ", " top ", " end " " interior ", " outward ", " clockwise ", " counterclockwise ", " axially ", " radially ", orientation or the position relationship of instructions such as " circumferentially " are based on orientation shown in the drawings or position relationship, only the present invention for convenience of description and simplified characterization, instead of device or the element of instruction or hint indication must have specific orientation, with specific orientation structure and operation, therefore can not be interpreted as limitation of the present invention.

In addition, term " first ", " second " be only for describing object, and can not be interpreted as instruction or hint relative importance or the implicit quantity that indicates indicated technical characterictic.Thus, at least one this feature can be expressed or impliedly be comprised to the feature that is limited with " first ", " second ".In description of the invention, the implication of " multiple " is at least two, for example two, and three etc., unless otherwise expressly limited specifically.

In the present invention, unless otherwise clearly defined and limited, the terms such as term " installation ", " being connected ", " connection ", " fixing " should be interpreted broadly, and for example, can be to be fixedly connected with, and can be also to removably connect, or integral; Can be mechanical connection, can be also electrical connection; Can be to be directly connected, also can indirectly be connected by intermediary, can be the connection of two element internals or the interaction relationship of two elements, unless separately there is clear and definite restriction.For the ordinary skill in the art, can understand as the case may be above-mentioned term concrete meaning in the present invention.

In the present invention, unless otherwise clearly defined and limited, First Characteristic Second Characteristic " on " or D score can be that the first and second features directly contact, or the first and second features are by intermediary indirect contact.And, First Characteristic Second Characteristic " on ", " top " and " above " but First Characteristic directly over Second Characteristic or oblique upper, or only represent that First Characteristic level height is higher than Second Characteristic.First Characteristic Second Characteristic " under ", " below " and " below " can be First Characteristic under Second Characteristic or tiltedly, or only represent that First Characteristic level height is less than Second Characteristic.

In the description of this instructions, the description of reference term " embodiment ", " some embodiment ", " example ", " concrete example " or " some examples " etc. means to be contained at least one embodiment of the present invention or example in conjunction with specific features, structure, material or the feature of this embodiment or example description.In this manual, to the schematic statement of above-mentioned term not must for be identical embodiment or example.And, specific features, structure, material or the feature of description can one or more embodiment in office or example in suitable mode combination.In addition,, not conflicting in the situation that, those skilled in the art can carry out combination and combination by the feature of the different embodiment that describe in this instructions or example and different embodiment or example.

Although illustrated and described embodiments of the invention above, be understandable that, above-described embodiment is exemplary, can not be interpreted as limitation of the present invention, and those of ordinary skill in the art can change above-described embodiment within the scope of the invention, amendment, replacement and modification.

Claims (14)

1. for the parameter optimization of machine learning and a method for feature tuning, it is characterized in that, comprise the following steps:

The multiple parameter sets of random generation;

Respectively described multiple parameter sets are carried out to the iteration optimization based on EnKF;

Respectively the multiple parameter sets after optimizing are carried out to Performance Evaluation, and obtain gathering pond and supplementary parameter set according to assessment result, wherein, the performance of the parameter sets in described set pond is higher than the performance of the parameter sets in described supplementary parameter set;

Parameter sets in parameter sets in pair set pond and described supplementary parameter set is carried out iteration optimization and the Performance Evaluation based on EnKF again, to obtain optimized parameter set.

2. the method for the parameter optimization for machine learning according to claim 1 and feature tuning, is characterized in that, describedly respectively described multiple parameter sets is carried out to the iteration optimization based on EnKF, specifically comprises:

Each parameter sets is trained with generation model on predetermined training dataset by machine learning algorithm;

On predetermined verification msg collection, described model is carried out to Performance Evaluation;

By EnKF algorithm, described multiple parameter sets are upgraded.

3. the method for the parameter optimization for machine learning according to claim 1 and feature tuning, is characterized in that, the multiple parameter sets of described random generation, specifically comprise the following steps:

A parameter vector θ ∈ R of random generation ^{m × 1}, wherein each parameter generates random value in predetermined parameter area;

One group of normalized vector { ρ of random generation _i| ρ _i∈ R ^{m × 1}, i=1 ..., N}, carries out orthogonalization to guarantee disturbance linear independence to it;

Generate parameter sets disturbance, specifically comprise:

A′＝(F _ar ₁ρ ₁，F _ar ₂ρ ₂，...，F _ar _Nρ _N)∈R ^m×N，r _i～N(0，S _p)，

Wherein, the disturbance of A ' expression parametric geometry, ρ _irepresent the random perturbation vector generating, variable r _irepresent arbitrary width, and Gaussian distributed, variance S _pconfigurable, matrix F _a=(f ₁e ₁, f ₂e ₂..., f _ne _n), e _irepresentation unit vector, f _ia configurable zoom variables, for adjusting perturbation amplitude;

By parameter vector θ, add respectively disturbance set A ' in each group perturbation vector ε _i∈ A ', obtains one group of parameter θ _i=θ+ε _i, symbiosis becomes N group parameter to form parameter sets A;

Repeat above-mentioned several step, to generate N _eindividual parameter sets.

4. the method for the parameter optimization for machine learning according to claim 2 and feature tuning, is characterized in that, also comprises:

Described training dataset and verification msg collection are carried out to feature convergent-divergent;

By the parameter input machine learning algorithm in parameter sets, concentrate and train with generation model at training data;

By described model, each sample is predicted, obtained the estimated value of described model;

Obtain the model of all parameters, and carry out Performance Evaluation, specifically comprise:

Wherein, the predicted value that adopts each group parameter generation model to obtain that HA has represented set-inclusion, n represents the number of verification msg collection sample.

5. the method for the parameter optimization for machine learning according to claim 2 and feature tuning, is characterized in that, also comprises: by following formula, described parameter sets is upgraded:

A ^a＝A ^f+A′(HA′) ^T(HA′(HA′) ^T+γγ ^T) ^-1(D-HA)，

Wherein, A represents parameter sets, A ^frepresent current parameter sets, A ^arepresent the parameter sets of upgrading, the disturbance of A ' expression parameter sets, D represents observation set, and γ represents observation set disturbance, and HA represents model prediction results set, the set disturbance of HA ' expression HA.

6. the method for the parameter optimization for machine learning according to claim 1 and feature tuning, it is characterized in that, describedly respectively the multiple parameter sets after optimizing are carried out to Performance Evaluation, and obtain gathering pond and supplementary parameter set according to assessment result, further comprise:

According to assessment result, the multiple parameter sets after described optimization are divided three classes, establish the performance number that score (A) represents parameter sets;

If score (A) >=thresh1, judges that this parameter sets performance is fine, and this parameter sets is kept in set pond;

If score (A)≤thresh2, judges this parameter sets poor-performing, and abandons this parameter sets;

If thresh2 < score (A) < is thresh1, judges that the performance of this parameter sets is general, and parameter sets general performance is merged at random between two, to generate supplementary parameter set.

7. the method for the parameter optimization for machine learning according to claim 6 and feature tuning, is characterized in that, also comprises:

In the general parameter sets of performance, choose a pair of parameter sets, generate new parameter sets by the merge algorithm based on EnKF, specifically comprise:

Suppose that a pair of parameter sets of choosing is A _i, A _j, calculate respectively Q by following formula _ijand Q _ji:

Calculate respectively A _ijand A _ji, wherein:

$A_{\mathrm{ij}}^m={\stackrel{\&OverBar;}{A}}_i+A_i^{\&prime;}{Q}_{\mathrm{ij}},$

$A_{\mathrm{ji}}^m={\stackrel{\&OverBar;}{A}}_j+A_j^{\&prime;}{Q}_{\mathrm{ji}};$

For Q _ijand Q _jicarry out respectively UR decomposition;

From matrix and the middle N row with maximum pivot of selecting respectively, according to it in parameter sets and in select corresponding parameter vector and form last parameter sets A ^m;

If parameter sets list to be combined is empty, the new parameter sets of random generation, otherwise select at random another to merge parameter sets.

8. for the parameter optimization of machine learning and a system for feature tuning, it is characterized in that, comprising:

Generation module, described generation module is used for generating multiple parameter sets;

Optimize module, described optimization module is for carrying out the iteration optimization based on EnKF to described multiple parameter sets respectively;

Evaluation module, described evaluation module is for carrying out Performance Evaluation to the multiple parameter sets after optimizing respectively, and obtain gathering pond and supplementary parameter set according to assessment result, wherein, the performance of the parameter sets in described set pond is higher than the performance of the parameter sets in described supplementary parameter set;

Acquisition module, described acquisition module carries out iteration optimization and the Performance Evaluation based on EnKF again for the parameter sets in parameter sets and the described supplementary parameter set in pair set pond, to obtain optimized parameter set.

9. the system of the parameter optimization for machine learning according to claim 8 and feature tuning, it is characterized in that, described optimization module is for training with generation model each parameter sets on predetermined training dataset by machine learning algorithm, and on predetermined verification msg collection, described model is carried out to Performance Evaluation, and by EnKF algorithm, described multiple parameter sets are upgraded.

10. the system of the parameter optimization for machine learning according to claim 8 and feature tuning, is characterized in that, described generation module generates multiple parameter sets, specifically comprises:

A parameter vector θ ∈ R of random generation ^{m × 1}, wherein each parameter generates random value in predetermined parameter area;

One group of normalized vector { ρ of random generation _i| ρ _i∈ R ^{m × 1}, i=1 ..., N}, carries out orthogonalization to guarantee disturbance linear independence to it;

Generate parameter sets disturbance, specifically comprise:

A′＝(F _ar ₁ρ ₁，F _ar ₂ρ ₂，...，F _ar _Nρ _N)∈R ^m×N，r _i～N(0，S _p)，

Wherein, the disturbance of A ' expression parametric geometry, ρ _irepresent the random perturbation vector generating, variable r _irepresent arbitrary width, and Gaussian distributed, variance S _pconfigurable, matrix F _a=(f ₁e ₁, f ₂e ₂..., f _ne _n), e _irepresentation unit vector, f _ia configurable zoom variables, for adjusting perturbation amplitude;

By parameter vector θ, add respectively disturbance set A ' in each group perturbation vector ε _i∈ A ', obtains one group of parameter θ _i=θ+ε _i, symbiosis becomes N group parameter to form parameter sets A;

Repeat above-mentioned several step, to generate N _eindividual parameter sets.

The system of 11. parameter optimizations for machine learning according to claim 9 and feature tuning, is characterized in that, described optimization module is also for carrying out feature convergent-divergent to described training dataset and verification msg collection; By the parameter input machine learning algorithm in parameter sets, concentrate and train with generation model at training data; By described model, each sample is predicted, obtained the estimated value of described model; Obtain the model of all parameters, and carry out Performance Evaluation, specifically comprise:

Wherein, the predicted value that adopts each group parameter generation model to obtain that HA has represented set-inclusion, n represents the number of verification msg collection sample.

The system of 12. parameter optimizations for machine learning according to claim 9 and feature tuning, is characterized in that, by following formula, described parameter sets is upgraded:

A ^a＝A ^f+A′(HA′) ^T(HA′(HA′) ^T+γγ ^T) ^-1(D-HA)，

Wherein, A represents parameter sets, A ^frepresent current parameter sets, A ^arepresent the parameter sets of upgrading, the disturbance of A ' expression parameter sets, D represents observation set, and γ represents observation set disturbance, and HA represents model prediction results set, the set disturbance of HA ' expression HA.

The system of 13. parameter optimizations for machine learning according to claim 8 and feature tuning, it is characterized in that, described evaluation module is for being divided three classes the multiple parameter sets after described optimization according to assessment result, if score (A) represents the performance number of parameter sets, and in the time of score (A) >=thresh1, judge that this parameter sets performance is fine, and this parameter sets is kept in set pond, and in the time of score (A)≤thresh2, judge this parameter sets poor-performing, and abandon this parameter sets, and in the time of thresh2 < score (A) < thresh1, the performance of this parameter sets is general, and parameter sets general performance is merged at random between two, to generate supplementary parameter set.

The system of 14. parameter optimizations for machine learning according to claim 13 and feature tuning, it is characterized in that, described evaluation module also, for choosing a pair of parameter sets in the general parameter sets of performance, generates new parameter sets by the merge algorithm based on EnKF, specifically comprises:

Suppose that a pair of parameter sets of choosing is A _i, A _j, calculate respectively Q by following formula _ijand Q _ji:

Calculate respectively A _ijand A _ji, wherein:

$A_{\mathrm{ij}}^m={\stackrel{\&OverBar;}{A}}_i+A_i^{\&prime;}{Q}_{\mathrm{ij}},$

$A_{\mathrm{ji}}^m={\stackrel{\&OverBar;}{A}}_j+A_j^{\&prime;}{Q}_{\mathrm{ji}};$

For Q _ijand Q _jicarry out respectively UR decomposition;

From matrix and the middle N row with maximum pivot of selecting respectively, according to it in parameter sets and in select corresponding parameter vector and form last parameter sets A ^m;

If parameter sets list to be combined is empty, the new parameter sets of random generation, otherwise select at random another to merge parameter sets.