CN106874998A - A kind of step matrix disassembling method certainly based on Pareto optimization - Google Patents

Abstract

The invention provides a kind of step matrix disassembling method certainly based on Pareto optimization, to determine that the weight of sample is changed into being obtained using Pareto optimization technology the weight of sample using the Learning Step of monotonic increase mode in the existing study from step, and sample weights are represented into scope [0 during Pareto optimization, 1] it is changed to [1, 1] difficult sample weights is there is rational distribution and then ensure diversity, finally selecting compromise best Knee points on optimum results PF faces can accelerate matrix decomposition process as matrix element weights, the present invention more rationally more meets cognitive science in a word.

Description

A kind of step matrix disassembling method certainly based on Pareto optimization

Technical field

The invention belongs to machine learning techniques field, and in particular to a kind of step matrix decomposition side certainly based on Pareto optimization Method.

Background technology

In machine learning field, because the problem to be solved becomes increasingly complex, the optimization object function right and wrong of many tasks Convex, this is easily ensnared into local best points in optimization process.It is inspired and often taken in learning process in humans and animals First learn simple knowledge and then gradually learn the knowledge of hardly possible to reach learning objective, Y.Bengio, J.Louradour, One kind is proposed in R.Collobert, and J.Weston is in document " Curriculum learning " (ICML, 2009) to be based on The learning strategy course learning (CL) of machine learning, its core concept be exactly define a training sample by training difficulty gradually Incremental " course ", the course for then allowing model training to define, experiment proves that CL is beneficial to accelerate convergence and obtains more preferable Training result and more preferable model generalization.But in practical application, because amount of training data is big and difficulty is indefinite, artificially Definition course is unreasonable and lacks the feedback of modelling effect during model training, in order to solve the above problems, M.Kumar, B.Packer and D.Koller are in document " Self-paced learning for latent variable models " One kind is proposed in (NIPS, pp.1189-1197,2010) " Self-paced learning (SPL) " can from step learning method Oneself determine that the complexity of training sample is expressed as sample weights during model training with by learner, by a step-length Parameter is incremented by gradually to increase the number of samples and its difficulty of model training, and from step learning method, iteratively more new samples are weighed Gradually be added to sample from the easier to the more advanced in model training and then model parameter is updated by weight, and its other party is surpassed in many applications Method.

Matrix decomposition underlying issue and is widely used in machine learning and computation vision, it is intended to decompose m × n's Data matrix Y is two less matrix U ∈ R^m×r,V∈R^n×r(r ＜ ＜ min (m, n)) causes UV^T≈ Y, can be by solving Optimization problem is realized：Wherein Ω is the index of non-missing data, l () table Show loss function, R (U, V) is regular terms, conventional loss function is as follows with regular terms：

With

But due to the nonconvex property of above-mentioned optimization problem, traditional matrix disassembling method is easily trapped into bad local minimum Especially in the case where there is abnormal data and missing data, in order to solve the above problems, Q.Zhao, D.Y.Meng, L.Jiang, Q.Xie, Z.B.Xu and A.Hauptman are in document " Self-paced learning for matrix Matrix decomposition will be applied to from step study in factorization " (AAAI, 2015) to propose from step matrix disassembling method, with reference to New optimization problem is obtained from step learning strategy and matrix decomposition target：

Wherein w_ijThe weight (penalty values are bigger, and explanation is more difficult, and weight will be smaller) of representing matrix element, f (w_ij,k) It is that k is the step parameter of monotonic increase from step function.Experiment is proved can be from noisy data effectively from step matrix disassembling method Approach truth value matrix.

Although can effectively mitigate local minimum problem from step matrix disassembling method, many problems are still present Unreasonable part, it is not meet to recognize to control the difficulty of matrix element using a step parameter for monotonic increase from step study first Can science, the knowledge quantity and difficulty that people's next step in the cognitive process to be learnt be according to previous study situation depending on；Its The complexity of matrix element represents that distribution is unreasonable in the secondary study from step, i.e. the scope of weight is [0,1], wherein difficult Matrix element be 0 without degree of difficulty distribution.

And Pareto optimization is a kind of Multipurpose Optimal Method without formulating decision rule, have wide range of applications, it is different In general single object optimization, Pareto optimization is intended to find the compromise and Pareto having had to all targets and arranges other solutions Pareto optimality disaggregation and Pareto leading surface, Knee points are exactly the optimal turning point of Pareto leading surface, are had to different target Best is compromise.In addition Pareto optimization has many successful Applications, such as C.Qian, and Y.Yu and Z.-H.Zhou are in document Subset is carried out using Pareto optimization in " Subset selection by pareto optimization " (NIPS, 2015) Select and in document " Pareto ensemble pruning " (AAAI, pp.2935-2941,2015) by Pareto application In integrated pruning.

The content of the invention

The purpose of the present invention is to overcome the problem for existing from step study from step matrix disassembling method in the prior art, effectively Mitigate matrix decomposition bad local minimum problem is absorbed in due to non convex objective function, especially in the presence of abnormal data and lack In the case of losing data.

To achieve these goals, the invention provides a kind of step matrix disassembling method certainly based on Pareto optimization, bag Include following steps：

Step 1) input matrix Y ∈ R to be decomposed^m×n, Population Size N_pAnd step information, do not lack in the matrix Y to be decomposed The matrix index collection for losing data is combined into Ω, and the step information includes initial step length k₀With step size increments μ；

Step 2) decompose matrix Y to be decomposed and obtain initial matrix U₀,V₀, calculating matrix element loss value, then according to weight Method of salary distribution f (w_ij；K) it is N with randomized generation Population Size_pInitial weight population P₀And calculate corresponding to population at individual Two target function values, nowAnd iteration ends number of times G is set_max, its In, U₀∈R^m×r,V₀∈R^r×n, r ＜ ＜ min (m, n), k represents step-length, by initial step length k₀Increase μ successively to obtain, w_ijRepresent square The weight of the i-th row jth column element of battle array,S-th population at individual is represented,RepresentCorresponding two Target function value, s={ 1 ..., N_p}；

Step 3) to P₀Carry out non-dominated ranking operation；

Step 4) iterations gen=0 is set, make P_gen=P₀Represent the gen times population of iteration；

Step 5) to population P_gen`Perform selection operation and select N_p/ 2 parent individualities constitute parent sub-group；

Step 6) parent sub-group is carried out cross and variation genetic manipulation generation Population Size be N_pProgeny population Q_gen；

Step 7) combination population P_genWith progeny population Q_genFor scale is 2N_pPopulation R_gen:R_gen=P_gen∪Q_gen；

Step 8) to population R_genPerform selection operation and select N_pIndividual population at individual constitutes a new generation colony P_gen+1:

Step 9) judge iterations gen whether less than termination iterations G_maxIf being less than G_max, iterations adds 1, returns To step 5) start a new round circulation, otherwise stop circulation, obtain population PF faces, find out the Knee points on PF facesI.e. one group element weights, wherein the value by weight less than 0 is entered as 0 again；

Step 10) by Optimization SolutionObtain last split-matrix U,V。

The w_ijThe weight of ∈ [- 1,1] representing matrix the i-th row jth column element,

Two target function value computing formula corresponding to the calculating population at individual are as follows：

Step 3) in P₀Carry out non-dominated ranking operation detailed process as follows：Population at individual allocation level is given, and is calculated The crowding distance of each population at individual, now Wherein rank^s, dis^sThe grade and crowding distance of s-th individuality of population are represented respectively.

Step 5) to population P_gen`Perform selection operation and select N_p/ 2 parent individualities constitute the specific method of parent sub-group It is as follows：Two population at individual are selected at random compares their grade and crowding distance, the individual reservation of lower grade, if grade phase Same then crowding distance individual reservation high, until selecting N_p/ 2 individual composition populations of parent；

Step 8) to population R_genPerform selection operation and select N_pIndividual population at individual composition population P of new generation_gen+1Specific side Method is as follows：The low population at individual of non-dominant grade to population P of new generation is selected successively_gen+1In, when selected same grade Number of individuals is more than N_pWhen, then crowding distance individuality high to population P of new generation is selected successively_gen+1In, final composition scale is N_p Population P of new generation_gen+1。

The beneficial effects of the invention are as follows：

1st, the step matrix disassembling method certainly based on Pareto optimization that the present invention is provided, will be weighed using Pareto optimization element Weight population, it is no longer necessary to which the step parameter of monotonic increase determines element weights, and Pareto optimization can be with adjust automatically object function Solution path get the solution for making object function optimal, in addition the genetic manipulation during Pareto optimization can make from step learn The autonomous regularized learning algorithm paces of habit process；

2nd, it is the larger rational weight distribution of Elemental partition of loss that sample weights scope is changed to [- 1,1] by this method Scope, is beneficial to simple sample and complex samples and more abundant sample weights is generated during Pareto optimization, finally from Can select and object function is rolled over as the matrix element weights for solving optimization problem from Knee solutions in Pareto optimization result The best weight of inner feelings, more conforms to cognitive science based on Pareto optimization from step matrix disassembling method in a word.

It is described in further details below in conjunction with accompanying drawing.

Brief description of the drawings

Fig. 1 is flow chart of the invention；

Fig. 2 is population P₀Schematic diagram；

Fig. 3 is Pareto leading surface schematic diagram；

Fig. 4 is target function value f₁；

Fig. 5 is target function value f₂；

Fig. 6 is matrix decomposition experimental result.

Specific embodiment

Embodiment 1：

Present embodiments provide it is a kind of it is as shown in Figure 1 based on Pareto optimization from step matrix disassembling method, including with Lower step：

Step 1) input matrix Y ∈ R to be decomposed^m×n, Population Size N_pAnd step information, do not lack in the matrix Y to be decomposed The matrix index collection for losing data is combined into Ω, and the step information includes initial step length k₀With step size increments μ；

Step 2) decompose matrix Y to be decomposed and obtain initial matrix U₀,V₀, calculating matrix element loss value, then according to weight Method of salary distribution f (w_ij；K) it is N with randomized generation Population Size_pInitial weight population P₀And calculate corresponding to population at individual Two target function values, nowAnd iteration ends number of times G is set_max, its In, U₀∈R^m×r,V₀∈R^r×n, r ＜ ＜ min (m, n), k represents step-length, by initial step length k₀Increase μ successively to obtain, w_ijRepresent square The weight of the i-th row jth column element of battle array,S-th population at individual is represented,RepresentCorresponding two Target function value, s={ 1 ..., N_p}；

Step 3) to P₀Carry out non-dominated ranking operation；

Step 4) iterations gen=0 is set, make P_gen=P₀Represent the gen times population of iteration；

Step 5) to population P_gen`Perform selection operation and select N_p/ 2 parent individualities constitute parent sub-group；

Step 6) parent sub-group is carried out cross and variation genetic manipulation generation Population Size be N_pProgeny population Q_gen；

Step 7) combination population P_genWith progeny population Q_genFor scale is 2N_pPopulation R_gen:R_gen=P_gen∪Q_gen；

Step 8) to population R_genPerform selection operation and select N_pIndividual population at individual constitutes a new generation colony P_gen+1:

Step 9) judge iterations gen whether less than termination iterations G_maxIf being less than G_max, iterations adds 1, returns To step 5) start a new round circulation, otherwise stop circulation, obtain population PF faces, find out the Knee points on PF facesI.e. one group element weights, wherein the value by weight less than 0 is entered as 0 again；

Step 10) by Optimization SolutionObtain last split-matrix U,V。

The step matrix disassembling method certainly based on Pareto optimization that the present invention is provided, will be using Pareto optimization element weights Population, it is no longer necessary to which the step parameter of monotonic increase determines element weights, Pareto optimization can be with adjust automatically object function Solution path gets the solution for making object function optimal, and the genetic manipulation during Pareto optimization can make to learn from step in addition The autonomous regularized learning algorithm paces of process.

Embodiment 2：

On the basis of embodiment 1, a kind of step matrix disassembling method certainly based on Pareto optimization is present embodiments provided, Comprise the following steps：

Step 1) input matrix Y, Population Size N_pAnd step information：

The element Y of random generator matrix Y_ijGaussian distributed N (0,1), then wherein 40% elects missing data, 20% as Data are added in Gaussian noise~N (0,0.01) disturbances that uniformly distributed noise disturbance and 20% data are added on [- 20,20]； Step information includes initial step length k and step size increments μ；

Step 2) decompose Y obtain initial matrix U₀,V₀, counting loss simultaneously generates initial weight population P₀And iteration end is set Only number of times G_max：Population P₀Illustrate map Fig. 2；

Initial matrix U is obtained using traditional matrix disassembling method₀,V₀, by matrix U₀,V₀Substitute into loss function formula (LS Or LAD loss functions) calculating elements penalty values, then according to element loss value and initial step length k generation element weightsAs Population at individual, repeatedly makes step-length increase the population at individual of μ generations 30%, the population at individual composition rule of random generation remaining 70% Mould is N_pInitial population P₀And calculate the addition population next two columns of the target function value corresponding to individuality：Fig. 4 and Fig. 5 are respectively mesh Offer of tender numerical value f₁、f₂；

At this momentWhereinRepresent s-th kind Group is individual,RepresentTwo corresponding target function values, s={ 1 ..., N_p}

Step 3) to population P₀Carry out non-dominated ranking operation：

If the fitness of an individual is below another individual fitness value shows that the individuality arranges another individuality, compare successively Fitness value compared with population at individual gives individual distribution non-dominant grade (individuality of lower grade explanation domination is more), and suitable according to individuality Angle value is answered to calculate the crowding distance of each population at individual, now population: Wherein rank^s,dis^sThe grade and crowding distance of s-th individuality of population are represented respectively；

Step 4) set iterations gen=1 make P_gen=P₀Represent the gen times population of iteration；

Step 5) to population P_genPerform selection operation and select parent sub-group：

Two population at individual are selected at random compares their grade and crowding distance, the individual reservation of lower grade, if waiting The identical then crowding distance of level individual reservation high, until selecting N_p/ 2 individual composition populations of parent；

Step 6) parent sub-group is carried out cross and variation genetic manipulation generation scale be N_pProgeny population Q_gen:

Cross and variation probability, index are set, and random selection parent individuality is simulated binary system and intersects and multinomial variation Generation new individual composition progeny population Q_gen；

Step 7) combination population P_genWith progeny population Q_genFor scale is 2N_pPopulation R_gen:

Directly merge contemporary population P_genWith progeny population Q_genComposition population R_gen：R_gen=P_gen∪Q_gen；

Step 8) to population R_genPerform selection operation and select colony P of new generation_gen+1；

The low population at individual of non-dominant grade to population P of new generation is selected successively_gen+1In, when selected same grade Number of individuals is more than N_pWhen, then crowding distance individuality high to population P of new generation is selected successively_gen+1In, final composition scale is N_p Population P of new generation_gen+1；

Step 9) judge iterations gen whether less than termination iterations G_maxIf being less than G_max, then gen=gen+1, Return to step 5) start new circulation, otherwise stop circulation, the PF faces of population are obtained, find out the Knee points on PF facesI.e. one group sample weights, wherein the value by weight less than 0 is entered as 0：

The Knee points of leading surface are distributed using the Pareto for selecting based on Angle Method or other method population, by what is selected Sample weights carry out again sample of the assignment only from weight more than 0 and carry out the renewal of next step model parameter, and weight is less than into 0 Value be entered as 0；Fig. 3 is Pareto leading surface schematic diagram；

Step 10) by optimization object functionObtain final output Matrix U, V.Fig. 6 is matrix decomposition experimental result.

It is the larger rational weight distribution model of Elemental partition of loss that sample weights scope is changed to [- 1,1] by this method Enclose, be beneficial to simple sample and complex samples and more abundant sample weights are generated during Pareto optimization, finally from handkerchief Can be selected compromise to object function as the matrix element weights for solving optimization problem from Knee solutions in tired support optimum results Best weight, more conforms to cognitive science based on Pareto optimization from step matrix disassembling method in a word.

The method and structure that various embodiments above is not described in detail belongs to the common knowledge of the industry, does not chat one by one here State.

It is exemplified as above be only to of the invention for example, do not constitute the limitation to protection scope of the present invention, it is all It is that design same or analogous with the present invention is belonged within protection scope of the present invention.

Claims (6)

1. It is 1. a kind of that matrix disassembling method is walked based on Pareto optimization certainly, it is characterised in that：Comprise the following steps：

   Step 1) input matrix Y ∈ R to be decomposed^m×n, Population Size N_pAnd step information, non-missing number in the matrix Y to be decomposed According to matrix index collection be combined into Ω, the step information includes initial step length k₀With step size increments μ；

   Step 2) decompose matrix Y to be decomposed and obtain initial matrix U₀,V₀, calculating matrix element loss value, then according to weight distribution Mode f (w_ij；K) it is N with randomized generation Population Size_pInitial weight population P₀And calculate two corresponding to population at individual Target function value, nowAnd iteration ends number of times G is set_max, wherein, U₀∈ R^m×r,V₀∈R^r×n, r ＜ ＜ min (m, n), k represents step-length, by initial step length k₀Increase μ successively to obtain, w_ijThe row of representing matrix i-th The weight of jth column element,Represent s-th population at individual, f₁ ^s,RepresentTwo corresponding target letters Numerical value, s={ 1 ..., N_p}；

   Step 3) to P₀Carry out non-dominated ranking operation；

   Step 4) iterations gen=0 is set, make P_gen=P₀Represent the gen times population of iteration；

   Step 5) to population P_gen`Perform selection operation and select N_p/ 2 parent individualities constitute parent sub-group；

   Step 6) parent sub-group is carried out cross and variation genetic manipulation generation Population Size be N_pProgeny population Q_gen；

   Step 7) combination population P_genWith progeny population Q_genFor scale is 2N_pPopulation R_gen:R_gen=P_gen∪Q_gen；

   Step 8) to population R_genPerform selection operation and select N_pIndividual population at individual constitutes a new generation colony P_gen+1:

   Step 9) judge iterations gen whether less than termination iterations G_maxIf being less than G_max, iterations adds 1, returns to step The rapid circulation for 5) starting a new round, otherwise stops circulation, obtains the PF faces of population, finds out the Knee points on PF facesI.e. one group element weights, wherein the value by weight less than 0 is entered as 0 again；

   Step 10) by Optimization SolutionObtain last split-matrix U, V.
2. 2. it is according to claim 1 it is a kind of based on Pareto optimization from step matrix disassembling method, it is characterised in that：It is described w_ijThe weight of ∈ [- 1,1] representing matrix the i-th row jth column element,
3. 3. it is according to claim 1 it is a kind of based on Pareto optimization from step matrix disassembling method, it is characterised in that：It is described The two target function value computing formula calculated corresponding to population at individual are as follows：

   $\left\{\begin{array}{c}{f}_1=\underset{(i,j)\∈\mathrm{\Ω}}{\Σ}{w}_{ij}l(Y_{ij},{\[U_0{V}_0^T\]}_{ij})\\ f_2=\underset{(i,j)\∈\mathrm{\Ω}}{\Σ}f(w_{ij};k)\end{array}\right..$
4. 4. it is according to claim 1 it is a kind of based on Pareto optimization from step matrix disassembling method, it is characterised in that step 3) to P in₀Carry out non-dominated ranking operation detailed process as follows：Population at individual allocation level is given, and calculates each population at individual Crowding distance, nowWherein rank^s, dis^s The grade and crowding distance of s-th individuality of population are represented respectively.
5. 5. it is according to claim 1 it is a kind of based on Pareto optimization from step matrix disassembling method, it is characterised in that step 5) to population P_gen`Perform selection operation and select N_pThe specific method of/2 parent individuality composition parent sub-groups is as follows：Random choosing Go out two population at individual and compare their grade and crowding distance, the individual reservation of lower grade, if grade is identical it is crowded away from Retain from individuality high, until selecting N_p/ 2 individual composition populations of parent.
6. 6. it is according to claim 1 it is a kind of based on Pareto optimization from step matrix disassembling method, it is characterised in that step 8) to population R_genPerform selection operation and select N_pIndividual population at individual composition population P of new generation_gen+1Specific method it is as follows：Successively Select the low population at individual of non-dominant grade to population P of new generation_gen+1In, when the number of individuals of selected same grade is more than N_p When, then crowding distance individuality high to population P of new generation is selected successively_gen+1In, final composition scale is N_pPopulation of new generation P_gen+1。