CN106446951A - Singular value selection-based integrated learning device - Google Patents

Abstract

The invention discloses a singular value selection-based integrated learning device. The realization process of the integrated learning device includes the following steps that: normalization preprocessing is performed on training sample sets; sampling with replacement is performed through adopting a Bootstrap random sampling method, so that M new sample sets can be generated; partial singular value decomposition (SVD) is performed on each sample in the M new sample sets, and the corresponding singular values as well as left and right singular vectors of each sample are obtained; k singular values as well as left and right singular vectors corresponding to the k singular values are randomly extracted each time, so that a 2D SVM (support vector machine) base learning device can be obtained, and the M new sample sets are trained, so that M 2D SVM (support vector machine) base learning devices can be obtained; and the base learning devices are combined according to a relative majority voting criterion, so that the integrated learning device can be obtained, and the obtained learning device is adopted to classify and identify samples to be classified. With the singular value selection-based integrated learning device of the invention adopted, problems such as large computation amount, curse of dimensionality, structure information loss of data and inherent correlation damage of the data which are caused by the application of an existing classifier to stretching matrix object data into high-dimension vectors can be solved.

Description

Integrated learning device based on singular value selection

Technical Field

The invention relates to the technical field of machine learning and image processing, in particular to a classification method taking tensor data of two dimensions or more as input samples, which can be used for target detection, pattern recognition and behavior recognition.

Background

With the rapid development of the internet and computer technology, the information volume of human beings in dozens of years is comparable to the sum of the information volume of all the past times of human beings. The continuous development of data brings great changes to the work, life and thinking of human beings, and the development of data is mainly reflected in two aspects: firstly, the scale of data is getting bigger and bigger; secondly, the structure of the data is more and more complex, compared with the traditional paper text information, various information formats such as web pages, black and white images, color images, medical images, satellite remote sensing images, videos and the like cannot be represented by structures such as simple vectors and the like, and more dimensions are needed to represent the characteristics of the data object, so that the information quantity such as data dimensions and the like is increased. Therefore, it can be said that "big data" is a subject word of the information age.

Different classification algorithms may yield different classification performance, but none of them can yield good results for all applications. Regarding the design of classifiers, various classification methods have been proposed so far by researchers in data mining, machine learning, statistics, pattern recognition and neurobiology, such as expert systems, association rules, decision trees, bayesian classifiers, support vector machines, neural networks, genetic algorithms, etc., and these methods have been applied to different fields and contribute to the development of research industry.

Although the proposed classification method has achieved a certain success in some fields, in most of the mentioned learning methods, data is generally expressed by using a vector mode, and in order to enable a learning algorithm based on the vector mode to learn about two-dimensional or more tensor data, it is generally necessary to firstly vectorize and expand the data in the tensor mode, and then learn by using a conventional learning algorithm. Taking a black-and-white image as an example, simply stretching the image and converting the image into a vector mode for processing, and ignoring data inherent structure information such as relative positions between pixels in an original image, the spatial-temporal structure of the original data can be damaged, and related information between data structures can be lost. If the size of the original data is large, processing into a vector mode can cause dimension increase, possibly causing dimension disaster or small sample high dimension problems, so that the obtained classifier has poor effect.

Therefore, in order to solve the above problems, it is necessary to provide a 2D SVM ensemble learning method, which can improve the precision of the classifier by using the advantages of the ensemble learning without breaking the space-time structure of the original data.

Disclosure of Invention

In order to overcome the defects in the prior art, the invention provides the integrated learner based on singular value selection, which improves the diversity among the base classifiers and obtains the integrated effect with strong generalization capability by randomly selecting partial singular values of the samples.

The technical scheme adopted by the invention for solving the technical problems is as follows: an ensemble learner based on singular value selection comprising the steps of:

step one, carrying out normalization pretreatment on a training sample set;

step two, sampling is carried out on the training sample set subjected to normalization pretreatment in a replacement mode by adopting a Bootstrap random sampling method, and M new sample sets are generated;

performing partial SVD on each sample in the M new sample sets to obtain a singular value and left and right singular vectors corresponding to each sample;

randomly extracting k singular values and corresponding left and right singular vectors thereof each time to generate a 2D SVM base learner, and respectively training M new sample sets to obtain M2D SVM base classifiers;

and step five, merging the base classifiers according to the relative majority voting criteria to obtain an ensemble learner, and carrying out classification and identification on the samples to be classified by using the obtained ensemble learner.

Compared with the prior art, the invention has the following positive effects:

(1) the invention solves the problems of huge computation amount and dimensional disaster caused by stretching matrix object (such as images, EEG and the like) data into high-dimensional vectors, data structure information loss, internal correlation destruction and the like of the existing classifier.

(2) According to the method, through partial singular value decomposition of the sample, a certain number of singular values and singular vectors are randomly selected from the obtained singular values and singular vectors, and the sample is compressed and denoised to a certain degree.

(3) The invention constructs a base classifier with larger diversity through singular value selection, thereby generating integration with strong generalization capability.

Drawings

The invention will now be described, by way of example, with reference to the accompanying drawings, in which:

FIG. 1 is a schematic flow chart of the present invention.

Detailed Description

An ensemble learner based on singular value selection, as shown in fig. 1, comprising the steps of:

step one, training sample setIs subjected to normalized pretreatment to obtain

For training sample setThe normalization preprocessing method adopts 0-1 normalization, which is a linear transformation of the original sample data to make the result fall to [0, 1%]Interval, the transfer function is as follows:

wherein, X_i,X_i'∈R^p×qIs the ith sample, y_i∈Y,Y＝{C₁,C₂,…,C_NIs sample X_i,X_i' corresponding class label, sample X can be seen_i,X_i' is represented in the form of a two-dimensional matrix; max (X)_i) Representing the taking of training samples X_iMaximum value of middle element, min (X)_i) Representing the taking of training samples X_iMinimum value of middle element, repmat { min (X)_i)}∈R^p×qRepresents a sample minimum matrix with elements in the matrix all min (X)_i) (ii) a Finally, all the preprocessed training samples X are used_i' and label y thereof_iConstructing a preprocessed training sample set

Step two, sampling is carried out on the normalized training sample set in a back-to-back mode by adopting a Bootstrap random sampling method, and M new sample sets are finally generated

And uniformly and randomly sampling the training sample set in a put-back mode to obtain a new sample set with the same size as the original sample set. Since there is a back-to-back uniform sampling, the probability that a sample is not selected at one time can be expressed as,when n → ∞ p ≈ 0.368, therefore, each base learner uses only about 63.2% of the samples in the initial training set, and the remaining about 36.8% of the samples can be used as a validation set to perform an "out-of-package estimation" (OOB) of the generalization performance of the learner, which has been proven to be unbiased estimation, so that no further cross-validation or separate test set is required in the integrated learning algorithm to obtain an unbiased estimation of the test set error.

Performing partial SVD on each sample in the sample set to obtain a singular value and left and right singular vectors corresponding to each sample:

(1) first, to sample X_iPerforming SVD full decomposition to obtain SVD full decomposition productThe formula is as follows: x_i＝UΣV^TWherein X is_i∈R^p×qIs a two-dimensional matrix, U ∈ R^p×pIs X_iOf left singular vectors of sigma ∈ R^p×qIs X_iOf singular values of, V^T∈R^q×qIs X_iA matrix composed of right singular vectors of (a);

(2) in most cases, the larger partial singular values of the matrix can well represent the basic information of the matrix, and the first r large singular values (i.e. the first r larger singular values) are used to approximately describe the sample X_iThus, the matrix is compressed to some extent, and the partial singular value decomposition is as follows:

wherein sigma_ip，μ_ip，v_ipIs X_iAnd its corresponding left and right singular vectors.

Step four, randomly extracting k singular values and corresponding left and right singular vectors thereof each time to generate a 2D SVM base learner:

4.1 for the two-class problem

Given a training data setWherein X_i∈R^p×qIs the ith input sample, y_i∈ { -1,1} is sample X_iCorresponding class label, one sees the input sample X_iIs represented in the form of a matrix.

The 4.1.12D SVM support vector machine is defined as follows:

s.t.y_i(<W,X_i>+b)≥1-ξ_i,i＝1,…,n (4)

ξ_i≥0,i＝1,…,n (5)

wherein, W is the direction of the classification hyperplane determined by the normal matrix, and b is the displacement item.

4.1.2 Lagrangian functions of equations (3) - (5) are obtained by the Lagrangian multiplier method as follows:

α therein_i≥0，β_iA value of > 0 is the Lagrangian multiplier.

Let L (W, b, α) pair W, b, ξ_iThe partial derivative of (c) is zero:

C＝α_i+β_i,i＝1,…,n (9)

the dual problems of formulae (3) to (5) can be obtained by substituting formulae (7) to (9) for formula (4) as follows:

0≤α_i≤C,i＝1,…,n (12)

wherein,<X_i,X_j>is X_iAnd X_jThe inner product of (d).

4.1.3 when sample X is input_iIn the form of vectors, then the optimization models (3) - (5) degenerate to the standard support vector machine. If we use the original form of the input sample to compute<X_i,X_j>Then the optimal solutions of (3) - (5) are the same as the solution of the linear support vector machine. Because of the "dimension disaster" and small sample problem, the support vector machine cannot effectively handle the matrix sample problem, and the optimization models (3) - (5) also suffer from the same problem. Specifically, the dual form of the optimization models (3) - (5) depends only on the inner product between sample data, while that in (10)<X_i,X_j>Inner product operations do not make good use of the structure information of the sample data.

Considering that the SVD decomposition of the matrix can better embody the structural information and the intrinsic correlation of the matrix data, the SVD decomposition of the matrix is used to replace the original matrix input, thereby improving the calculation of the inner product of the matrix. On one hand, the recognition capability of the learning machine can be improved; on the other hand, the learning speed of the learning machine can be accelerated.

4.1.4 according to the three steps, carrying out partial SVD decomposition on the samples to obtain r singular values and corresponding left and right singular vectors of each sample, and randomly selecting k singular values and corresponding left and right singular vectors from the r singular valuesAndthen matrix X_iThe inner product of Xj is calculated as follows:

substituting (13) into (10) to obtain:

0≤α_i≤C,i＝1,…,n (16)

as shown in (7), the weight matrix W for classifying the hyperplane can be expressed as a linear combination of training samples in a two-dimensional space, the optimization models (14) - (16) are 2D-SVM, and the 2D-SVM can be shown as the extension of a linear support vector machine on the two-dimensional matrix, so that the optimization models (14) - (16) can be solved by using an SMO algorithm.

The base learner 2D SVM classifier f (x) classification decision function is:

wherein sigma_ip、σ_q、u_ip、u_q、v_ipAnd v_qAre each X_iAnd the singular values of X and the corresponding left and right singular vectors.

4.2 Multi-Classification problem for 2D SVM

The "one-to-one" (OvO) strategy is used, as follows:

given data sety_i∈{C₁,C₂,…,C_NOvO pairs these N classes two by two, resulting in N (N-1)/2 binary classification tasks, e.g., OvO would be classified as discriminating class C_iAnd C_jTraining a classifier that measures D_tC in (1)_iClass sample as a good example，C_jClass examples are used as counter examples. In the testing stage, a new sample will be submitted to all classifiers at the same time, so that N (N-1)/2 classification results will be obtained, and the final result can be generated by voting: i.e. the most predicted classes are taken as the final classification result.

The above is a description of a 2D SVM, similar to the definition and derivation description of a Support Vector Machine (SVM).

<X_i,X_j>The processing data dimensions are different, the SVM processes the inner product of vector samples, and the 2d SVM proposed by the present invention can process the inner product of matrix (such as a matrix composed of picture pixels) samples.

However, the two matrixes cannot be directly subjected to inner product, so before the inner product is carried out, the matrixes are subjected to SVD decomposition (a part of singular values of the front R is randomly selected, a similar random forest is a base classifier with good diversity), and then the original matrixes are replaced by the selected singular values and singular vectors to carry out the inner product, so that 1, structural damage caused by matrix vector drawing (for example, the upper and lower relation of two pixels in a picture is originally, and the position relation is damaged after the matrix drawing) can be avoided, and 2, the operation speed of the inner product can be accelerated.

Finally, M new sample sets are obtained based on the Bootstrap random sampling method in the second stepThe training method is adopted for each sample set, and M2D SVM-based classifiers { h) are finally obtained₁,h₂,…,h_M}。

Step five, merging the base classifiers according to the relative majority voting criterion to obtain an ensemble learner:

in this step, the base classifiers are merged to obtain a stronger classifier-ensemble learner:

the relative majority voting criterion is adopted to merge the base classifiers, and the mathematical expression of the combination mode is as follows:

and finally, carrying out classification and identification on the sample to be classified by using the obtained ensemble learner.

Claims (6)

1. An ensemble learner based on singular value selection, comprising: the method comprises the following steps:

step one, carrying out normalization pretreatment on a training sample set;

step two, sampling is carried out on the training sample set subjected to normalization pretreatment in a replacement mode by adopting a Bootstrap random sampling method, and M new sample sets are generated;

performing partial SVD on each sample in the M new sample sets to obtain a singular value and left and right singular vectors corresponding to each sample;

randomly extracting k singular values and corresponding left and right singular vectors thereof each time to generate a 2D SVM base learner, and respectively training M new sample sets to obtain M2D SVM base classifiers;

and step five, merging the base classifiers according to the relative majority voting criteria to obtain an ensemble learner, and carrying out classification and identification on the samples to be classified by using the obtained ensemble learner.

2. The singular value selection-based ensemble learner of claim 1, wherein: the method for carrying out normalization preprocessing on the training sample set in the first step comprises the following steps:

s101, training sample setRespectively preprocessing each sample in the training sequence to obtain a normalized training sample X_i' and its class label y_i：

${X_i}^{\&prime;}=\frac{X_i-repmat\left\{min\left(X_i\right)\right\}}{max\left(X_i\right)-\min \left(X_i\right)},y_i=y_i$

Wherein, max (X)_i) Representing training sample X_iMaximum value of (1), min (X)_i) Representing training sample X_iMinimum value of (1), repmat { min (X)_i)}∈R^p×qRepresents a sample minimum matrix with elements in min (X)_i)；

S102, training sample X after all preprocessing is used_i' and its class label y_iConstructing a preprocessed training sample set

3. The singular value selection-based ensemble learner of claim 1, wherein: step three, the method for performing partial SVD decomposition on each sample in the M new sample sets is as follows:

s301, sample X_iThe SVD of (1) is in the form of: x_i＝UΣV^TWherein X is_i∈R^p×qIs a two-dimensional matrix, U ∈ R^p×pIs X_iOf left singular vectors of sigma ∈ R^p×qIs X_iOf singular values of, V^T∈R^q×qIs X_iA matrix composed of right singular vectors of (a);

s302, approximately describing sample X by using singular value with large front r_iThe partial singular value decomposition is in the form:

wherein sigma_ip，μ_ip，v_ipIs X_iAnd its corresponding left and right singular vectors.

4. The singular value selection-based ensemble learner of claim 3, wherein: fourthly, the generation method of the 2D SVM base learner comprises the following steps:

s401, for the two classification tasks, a training data set is givenWherein X_i∈R^p×qIs the ith input sample, y_i∈ { -1,1} is sample X_iA corresponding class label;

s402, the 2D SVM support vector machine is defined as follows:

$\left\{\begin{array}{c}\underset{W,b,{\mathrm{\&xi;}}_i}{min}J(W,b,{\mathrm{\&xi;}}_i)=\frac{1}{2}\left|\right|W|{|}_F^2+C\underset{i=1}{\overset{n}{\&Sigma;}}{\mathrm{\&xi;}}_i\\ \begin{array}{cc}s.t.& y_i(<W,X_i>+b)\&GreaterEqual;1-{\mathrm{\&xi;}}_i,i=1,...,n\end{array}\\ {\mathrm{\&xi;}}_i\&GreaterEqual;0,i=1,...,n\end{array}\right.$

wherein, W is a normal matrix, and b is a displacement term;

the dual problem of the 2D SVM support vector machine obtained by the Lagrange multiplier method is as follows:

$\left\{\begin{array}{c}\underset{\mathrm{\&alpha;}}{max}\underset{i=1}{\overset{n}{\&Sigma;}}{\mathrm{\&alpha;}}_i-\frac{1}{2}\underset{i,j}{\overset{n}{\&Sigma;}}{\mathrm{\&alpha;}}_i{\mathrm{\&alpha;}}_j{y}_i{y}_j<X_i,X_j>\\ \begin{array}{cc}s.t.& \underset{i=1}{\overset{n}{\&Sigma;}}{\mathrm{\&alpha;}}_i{y}_i=0\end{array}\\ 0\&le;{\mathrm{\&alpha;}}_i\&le;C,i=1,...,n\end{array}\right.---\left(1\right)$

wherein,<X_i,X_j>is X_i∈R^p×qAnd X_j∈R^p×qInner product of (C) ═ α_i+β_iI is 1, …, n, wherein α_i≥0，β_iLagrange multiplier is greater than or equal to 0;

s403, randomly selecting k singular values and corresponding left and right singular vectors from r singular values and corresponding left and right singular vectors of each sample, wherein the k singular values and the corresponding left and right singular vectors are respectivelyAndthen matrix X_iAnd X_jThe inner product of (d) is calculated as follows:

substituting formula (2) into formula (1) to obtain the final form of the 2D SVM as follows:

$\left\{\begin{array}{c}\underset{\mathrm{\&alpha;}}{\max }\underset{i=1}{\overset{n}{\&Sigma;}}{\mathrm{\&alpha;}}_i-\frac{1}{2}\underset{i,j}{\overset{n}{\&Sigma;}}\underset{p=1}{\overset{k}{\&Sigma;}}\underset{q=1}{\overset{k}{\&Sigma;}}{\mathrm{\&alpha;}}_i{\mathrm{\&alpha;}}_j{y}_i{y}_j{\mathrm{\&lambda;}}_{ip}{\mathrm{\&lambda;}}_{jp}<u_{ip},u_{jq}><v_{ip},v_{jq}>\\ \underset{i=1}{\overset{n}{\&Sigma;}}{\mathrm{\&alpha;}}_i{y}_i=0\\ 0\&le;{\mathrm{\&alpha;}}_i\&le;C,i=1,\mathrm{...},n\end{array}\right.$

the base learner 2D SVM classifier f (x) classification decision function is:

$h\left(X\right)=f\left(X\right)=sign(\underset{i,j}{\overset{n}{\&Sigma;}}\underset{p=1}{\overset{k}{\&Sigma;}}\underset{q=1}{\overset{k}{\&Sigma;}}{\mathrm{\&alpha;}}_i{y}_i{\mathrm{\&sigma;}}_{ip}{\mathrm{\&sigma;}}_q<u_{ip},u_{iq}><v_{ip},v_q>+b)$

wherein σ_ip、σ_q、u_ip、u_q、v_ipAnd v_qAre each X_iAnd the singular value of X and the corresponding left singular vector and right singular vector;

s404, carrying out classification decision on the 2D SVM multi-classification task by adopting a one-to-one strategy.

5. The singular value selection-based ensemble learner of claim 4, wherein: in step S404, the method for performing classification decision by using one-to-one strategy includes: given data setX_i∈R^p×q,y_i∈{C₁,C₂,…,C_NPair N classes two by two, resulting in N (N-1)/2 bi-categorical tasks.

6. The singular value selection-based ensemble learner of claim 4, wherein: step five, the method for merging the base classifiers according to the relative majority voting criterion comprises the following steps:

$y\left(X\right)=H\left(X\right)={\mathrm{argmax}}_{y\&Element;Y}\underset{t=1}{\overset{M}{\&Sigma;}}l(y=h_t(X\left)\right).$