CN106446951A - Singular value selection-based integrated learning device - Google Patents

Abstract

The invention discloses a singular value selection-based integrated learning device. The realization process of the integrated learning device includes the following steps that: normalization preprocessing is performed on training sample sets; sampling with replacement is performed through adopting a Bootstrap random sampling method, so that M new sample sets can be generated; partial singular value decomposition (SVD) is performed on each sample in the M new sample sets, and the corresponding singular values as well as left and right singular vectors of each sample are obtained; k singular values as well as left and right singular vectors corresponding to the k singular values are randomly extracted each time, so that a 2D SVM (support vector machine) base learning device can be obtained, and the M new sample sets are trained, so that M 2D SVM (support vector machine) base learning devices can be obtained; and the base learning devices are combined according to a relative majority voting criterion, so that the integrated learning device can be obtained, and the obtained learning device is adopted to classify and identify samples to be classified. With the singular value selection-based integrated learning device of the invention adopted, problems such as large computation amount, curse of dimensionality, structure information loss of data and inherent correlation damage of the data which are caused by the application of an existing classifier to stretching matrix object data into high-dimension vectors can be solved.

Description

A kind of integrated study device based on singular value selection

Technical field

The present invention relates to machine learning and technical field of image processing, particularly to two-dimentional and above tensor data it is The sorting technique of input sample, can be used for target detection, pattern recognition, Activity recognition.

Background technology

With the Internet and computer technology fast development, the quantity of information that the mankind faced within short decades is comparable to the mankind The summation of all ages quantity of information in the past.Work, life and thinking that the continuous development of data gives people class bring major transformation, The development of data is mainly reflected in two aspects：One be data scale increasing；Two is that the structure of data becomes increasingly complex, Compared to traditional paper Word message, webpage, black white image, coloured image, medical image, satellite remote sensing images, video etc. Various information formats can not need more dimensions to go to represent the spy of data object with the simply representation such as vector Levy, thus resulted in the increase of the quantity of information such as data dimension.It can thus be stated that " big data " is the descriptor of information age.

Different sorting algorithms is likely to be obtained different classification performances, but does not have a kind of sorting algorithm can be to all of application All obtain good result.With regard to the design of grader, up to the present, various sorting techniques by data mining, Machine learning, the researcher of statistics, pattern recognition and neurobiology aspect propose, such as specialist system, correlation rule, decision-making Tree, Bayes classifier, support vector machine, neutral net, genetic algorithm etc., these methods have been applied to different necks Domain, is that the development of scientific research cause is made that contribution.

Although the sorting technique having been proposed that obtains certain success in some fields, but in the most mathematics mentioned In learning method, data is typically represented using vector pattern, so that can be to two dimension based on the learning algorithm under vector pattern And above tensor data is learnt, it usually needs first the data of tensor pattern is carried out vectorization expansion, then again with passing The learning algorithm of system is learnt., stretch simply by image taking black white image as a example, be converted into vector pattern and carry out Process, ignore the intrinsic structural information of the data such as relative position between pixel in original image, can destroy initial data when Hollow structure, loses the relevant information between data structure.If initial data is larger, being processed as vector pattern can lead to Dimension increases it is possible to " dimension disaster " or " the high dimension of small sample " problem can be led to, thus making the grader effect obtaining not Good.

Therefore, for problem above it is necessary to provide a kind of 2D SVM integrated study, can not break legacy data when Hollow structure, and the advantage of integrated study can be utilized, the precision of lifting grader.

Content of the invention

In order to overcome the disadvantages mentioned above of prior art, the invention provides a kind of integrated study based on singular value selection Device, by randomly selecting to the part singular value of sample, thus improve the multiformity between each base grader, obtains general The strong integrated result of change ability.

The technical solution adopted for the present invention to solve the technical problems is：A kind of integrated study based on singular value selection Device, comprises the steps：

Step one, pretreatment is normalized to training sample set；

Step 2, from the pretreated training sample of normalization concentrate put back to using Bootstrap arbitrary sampling method Ground sampling, produces M new sample set；

Each sample in step 3, the sample set new to M carries out the decomposition of part SVD, obtains each sample corresponding Singular value and left and right singular vector；

Step 4, randomly draw k singular value and its corresponding left and right singular vector every time, generate the study of 2D SVM base Device, is trained to M new sample set respectively, obtains M 2D SVM base grader；

Step 5, criterion of being voted according to relative majority merge base grader and obtain integrated study device, with integrated obtaining Habit device is treated classification samples and is carried out Classification and Identification.

Compared with prior art, the positive effect of the present invention is：

(1) present invention solves existing grader and is stretched as high dimension vector to matrix object (such as image, EEG etc.) data The problems such as huge and dimension disaster and data the structural information of the operand bringing is lost with inherent related damage.

(2) present invention passes through the part singular value decomposition of sample, randomly selects in the singular values and singular vectors obtaining A number of singular values and singular vectors, have carried out compression noise reduction to sample to a certain extent.

(3) present invention selects to construct the larger base grader of multiformity by singular value, thus creating generalization ability Strong is integrated.

Brief description

Examples of the present invention will be described by way of reference to the accompanying drawings, wherein：

Fig. 1 is the schematic flow sheet of the present invention.

Specific embodiment

A kind of integrated study device based on singular value selection, as shown in figure 1, comprise the following steps：

Step one, to training sample setIt is normalized pretreatment to obtain

To training sample setThe method being normalized pretreatment adopts 0-1 standardization, is to original sample The linear transformation of notebook data, [0,1] is interval so that result is fallen, and transfer function is as follows：

Wherein, X_i,X_i'∈R^p×qIt is i-th sample, y_i∈ Y, Y={ C₁,C₂,…,C_NIt is sample X_i,X_i' corresponding class Label is it can be seen that sample X_i,X_i' it is to be represented in the form of two-dimensional matrix；max(X_i) represent take training sample X_iMiddle element Maximum, min (X_i) represent take training sample X_iThe minima of middle element, repmat { min (X_i)}∈R^p×qRepresent sample Little value matrix, the element in matrix is all min (X_i)；Finally with all pretreated training sample X_i' and its label y_iConstitute Pretreated training sample set

Step 2, the training sample after normalization are concentrated and are sampled with putting back to using Bootstrap arbitrary sampling method, Final M new sample set of generation

Training sample set is carried out putting back to uniformly random sampling, obtains an equal amount of new samples with former sample collection Collection.Due to being to put back to ground uniform sampling, so sample does not once have selected probability can be expressed as, As n → ∞, p ≈ 0.368, therefore, each base learner only employs the sample that initial training concentrates about 63.2%, Ke Yili Sample with remaining about 36.8% is made checking collection and to be the Generalization Capability of learner is carried out " bag is outer to be estimated " (OOB), this by Through proving unbiased esti-mator, so not needing to carry out cross validation or single test set again in Ensemble Learning Algorithms To obtain the unbiased esti-mator of test set error.

Step 3, the decomposition of part SVD is carried out to each sample in sample set, obtain the corresponding singular value of each sample and Left and right singular vector：

(1) first to sample X_iCarry out SVD entirely to decompose, decomposed form is：X_i=U Σ V^T, wherein X_i∈R^p×qIt is a two dimension Matrix, U ∈ R^p×pIt is X_iLeft singular vector composition matrix, Σ ∈ R^p×qIt is X_iSingular value composition diagonal matrix, V^T ∈R^q×qIt is X_iRight singular vector composition matrix；

(2) in most cases, the larger part singular value of matrix just can be very good the essential information of representing matrix, uses The big singular value of front r (i.e. front r larger singular value) carrys out approximate description sample X_i, thus matrix has been carried out to a certain degree On Information Compression, part singular value decomposition form is as follows：

Wherein σ_ip, μ_ip, v_ipFor X_iP-th singular value and its corresponding left and right singular vector.

Step 4, randomly draw k singular value and its corresponding left and right singular vector every time, generate the study of 2D SVM base Device：

4.1 for two classification problems

A given training datasetWherein X_i∈R^p×qIt is i-th input sample, y_i∈{-1,1} It is sample X_iCorresponding class label is it can be seen that input sample X_iRepresent with a matrix type.

4.1.1 2D SVM support vector machine are defined as follows：

s.t.y_i(<W,X_i>+b)≥1-ξ_i, i=1 ..., n (4)

ξ_i>=0, i=1 ..., n (5)

Wherein, W determines the direction of Optimal Separating Hyperplane for method matrix, and b is displacement item.

4.1.2 by method of Lagrange multipliers can get formula (3)-(5) Lagrangian as follows：

Wherein α_i>=0, β_i>=0 is Lagrange multiplier.

Make L (W, b, α, β, ξ) to W, b, ξ_iPartial derivative be zero can to obtain：

C=α_i+β_i, i=1 ..., n (9)

By formula (7)-(9) substitute into formula (4) can get formula (3)-(5) dual problem as follows：

0≤α_i≤ C, i=1 ..., n (12)

Wherein,<X_i,X_j>It is X_iWith X_jInner product.

4.1.3 working as input sample X_iWhen being vectorial form, then Optimized model (3)-(5) deteriorate to the supporting vector of standard Machine.If we to be calculated using the primitive form of input sample<X_i,X_j>, then the optimal solution of (3)-(5) and linear support vector As the solution of machine.Due to " dimension disaster " and small sample problem, support vector machine can not effective processing array sample problem, then Optimized model (3)-(5) also can run into same problem.Specifically, the dual form of Optimized model (3)-(5) only relies upon Inner product between sample data, and in (10)<X_i,X_j>Inner product operation does not utilize the structural information of sample data well.

SVD in view of matrix decomposes structural information and the inherence dependency that can preferably embody matrix data, utilizes The SVD of matrix decomposes to replace original matrix to input, thus improving the calculating of matrix inner products. advantage of this is that：On the one hand The identification ability of learning machine can be improved；On the other hand the pace of learning of learning machine can be accelerated.

4.1.4 decompose the r singular value and accordingly obtaining each sample according to sample is carried out in step 3 with part SVD Left and right singular vector, therefrom randomly selects k singular value and its corresponding left and right singular vector, respectivelyWithThen matrix X_iInner product meter with Xj Calculate as follows：

(13) are substituted in (10), obtains：

0≤α_i≤ C, i=1 ..., n (16)

Training sample can be expressed as by the weight matrix W that (7) can be seen that Optimal Separating Hyperplane linear on two-dimensional space Combination, Optimized model (14)-(16) are 2D-SVM, and 2D-SVM is it is seen that expansion on two-dimensional matrix for the linear SVM Optimized model (14)-(16) therefore can be solved by exhibition with SMO algorithm.

Base learner 2D SVM classifier f (X) categorised decision function is：

Wherein σ_ip、σ_q、u_ip、u_q、v_ipAnd v_qIt is respectively X_iUnusual with the singular value of X and corresponding left singular vector and the right side Vector.

4.2 for 2D SVM many classification problems

Using the strategy of " one-to-one " (OvO), specific as follows：

Data-oriented collectiony_i∈{C₁,C₂,…,C_N, this N number of classification is joined by OvO two-by-two Right, thus producing N (N-1)/2 two classification task, such as OvO will be for distinguishing classification C_iAnd C_jOne grader of training, this classification Device is D_tIn C_iClass sample is as positive example, C_jClass sample is as counter-example.In test phase, new samples will be simultaneously presented to be owned Grader, then will obtain N (N-1)/2 classification results, and final result can be produced by ballot：I.e. predicted most Classification is as final classification result.

More than the explanation to 2D SVM, the definition of similar support vector machine (SVM) and derive explanation.

<X_i,X_j>Here processing data dimension is different, and what SVM was processed is the inner product of vectorial sample, and 2D proposed by the present invention SVM can be with the inner product of processing array (as the matrix of picture pixels composition) sample.

Directly but inner product can not be carried out to two matrixes, so before carrying out inner product, first SVD decomposition is carried out to matrix (a part of singular value big to front R randomly chooses, and similar random forest is the base grader good in order to obtain multiformity), Then replace with selected singular values and singular vectors original matrix to carry out inner product, so, matrix 1. can be avoided to pull into vector The structural deterioration causing is (such as：In picture, two pixels are originally upper and lower relation, and after pulling into vector, position relationship destroys) 2. permissible Accelerate the arithmetic speed of inner product.

Finally, M new sample set is obtained based on the Bootstrap arbitrary sampling method in step 2To each Sample set adopts above training method, finally gives M 2D SVM base grader { h₁,h₂,…,h_M}.

Step 5, criterion of being voted according to relative majority merge base grader and obtain integrated study device：

In this step, merge base grader and obtain a higher grader -- integrated study device：

Base grader is merged using relative majority ballot criterion, the mathematical expression of compound mode is as follows：

Finally, treat classification samples with the integrated study device obtaining and carry out Classification and Identification.

Claims (6)

1. a kind of integrated study device based on singular value selection it is characterised in that：Comprise the steps：

Step one, pretreatment is normalized to training sample set；

Step 2, from the pretreated training sample of normalization concentrate taken out with putting back to using Bootstrap arbitrary sampling method Sample, produces M new sample set；

Each sample in step 3, the sample set new to M carries out the decomposition of part SVD, obtains each sample unusual accordingly Value and left and right singular vector；

Step 4, randomly draw k singular value and its corresponding left and right singular vector every time, generate 2D SVM base learner, point Other new sample set individual to M is trained, and obtains M 2D SVM base grader；

Step 5, criterion of being voted according to relative majority merge base grader and obtain integrated study device, with the integrated study device obtaining Treat classification samples and carry out Classification and Identification.

2. a kind of integrated study device based on singular value selection according to claim 1 it is characterised in that：Described in step Training sample set is normalized pretreatment method as follows：

S101, to training sample setIn each sample carry out pretreatment respectively, obtain the training after normalization Sample X_i' and its class label y_i：

${X_i}^{\&prime;}=\frac{X_i-repmat\left\{min\left(X_i\right)\right\}}{max\left(X_i\right)-\min \left(X_i\right)},y_i=y_i$

Wherein, max (X_i) represent training sample X_iIn maximum, min (X_i) represent training sample X_iIn minima, repmat {min(X_i)}∈R^p×qRepresent sample minimum matrix, the element in matrix is min (X_i)；

S102, with all pretreated training sample X_i' and its class label y_iConstitute pretreated training sample set

3. a kind of integrated study device based on singular value selection according to claim 1 it is characterised in that：Described in step 3 The method that each sample in the sample set new to M carries out the decomposition of part SVD is as follows：

S301, sample X_iSVD decomposed form be：X_i=U Σ V^T, wherein X_i∈R^p×qIt is a two-dimentional matrix, U ∈ R^p×pIt is X_iLeft singular vector composition matrix, Σ ∈ R^p×qIt is X_iSingular value composition diagonal matrix, V^T∈R^q×qIt is X_iThe right side unusual The matrix of vector composition；

S302, with the big singular value of front r come approximate description sample X_i, part singular value decomposition form is as follows：

Wherein σ_ip, μ_ip, v_ipFor X_iP-th singular value and its corresponding left and right singular vector.

4. a kind of integrated study device based on singular value selection according to claim 3 it is characterised in that：Described in step 4 The generation method of 2D SVM base learner is as follows：

S401, for two classification task, give a training datasetWherein X_i∈R^p×qIt is i-th input Sample, y_i∈ { -1,1 } is sample X_iCorresponding class label；

S402,2D SVM support vector machine are defined as follows：

$\left\{\begin{array}{c}\underset{W,b,{\mathrm{\&xi;}}_i}{min}J(W,b,{\mathrm{\&xi;}}_i)=\frac{1}{2}\left|\right|W|{|}_F^2+C\underset{i=1}{\overset{n}{\&Sigma;}}{\mathrm{\&xi;}}_i\\ \begin{array}{cc}s.t.& y_i(<W,X_i>+b)\&GreaterEqual;1-{\mathrm{\&xi;}}_i,i=1,...,n\end{array}\\ {\mathrm{\&xi;}}_i\&GreaterEqual;0,i=1,...,n\end{array}\right.$

Wherein, W is method matrix, and b is displacement item；

By the dual problem that method of Lagrange multipliers obtains 2D SVM support vector machine it is：

$\left\{\begin{array}{c}\underset{\mathrm{\&alpha;}}{max}\underset{i=1}{\overset{n}{\&Sigma;}}{\mathrm{\&alpha;}}_i-\frac{1}{2}\underset{i,j}{\overset{n}{\&Sigma;}}{\mathrm{\&alpha;}}_i{\mathrm{\&alpha;}}_j{y}_i{y}_j<X_i,X_j>\\ \begin{array}{cc}s.t.& \underset{i=1}{\overset{n}{\&Sigma;}}{\mathrm{\&alpha;}}_i{y}_i=0\end{array}\\ 0\&le;{\mathrm{\&alpha;}}_i\&le;C,i=1,...,n\end{array}\right.---\left(1\right)$

Wherein,<X_i,X_j>It is X_i∈R^p×qWith X_j∈R^p×qInner product, C=α_i+β_i, i=1 ..., n, in formula：α_i>=0, β_i>=0 is Lagrange multiplier；

K singular value and its corresponding is randomly selected S403, the r singular value from each sample and corresponding left and right singular vector Left and right singular vector, respectivelyWith Then matrix X_iAnd X_jInner product be calculated as follows：

Formula (2) is substituted in formula (1), the final form obtaining 2D SVM is as follows：

$\left\{\begin{array}{c}\underset{\mathrm{\&alpha;}}{\max }\underset{i=1}{\overset{n}{\&Sigma;}}{\mathrm{\&alpha;}}_i-\frac{1}{2}\underset{i,j}{\overset{n}{\&Sigma;}}\underset{p=1}{\overset{k}{\&Sigma;}}\underset{q=1}{\overset{k}{\&Sigma;}}{\mathrm{\&alpha;}}_i{\mathrm{\&alpha;}}_j{y}_i{y}_j{\mathrm{\&lambda;}}_{ip}{\mathrm{\&lambda;}}_{jp}<u_{ip},u_{jq}><v_{ip},v_{jq}>\\ \underset{i=1}{\overset{n}{\&Sigma;}}{\mathrm{\&alpha;}}_i{y}_i=0\\ 0\&le;{\mathrm{\&alpha;}}_i\&le;C,i=1,\mathrm{...},n\end{array}\right.$

Base learner 2D SVM classifier f (X) categorised decision function is：

$h\left(X\right)=f\left(X\right)=sign(\underset{i,j}{\overset{n}{\&Sigma;}}\underset{p=1}{\overset{k}{\&Sigma;}}\underset{q=1}{\overset{k}{\&Sigma;}}{\mathrm{\&alpha;}}_i{y}_i{\mathrm{\&sigma;}}_{ip}{\mathrm{\&sigma;}}_q<u_{ip},u_{iq}><v_{ip},v_q>+b)$

Wherein, σ_ip、σ_q、u_ip、u_q、v_ipAnd v_qIt is respectively X_iWith the singular value of X and corresponding left singular vector and right unusual to Amount；

S404, for 2D SVM, many classification task carry out categorised decision using man-to-man strategy.

5. a kind of integrated study device based on singular value selection according to claim 4 it is characterised in that：Step S404 institute The method using man-to-man strategy carries out categorised decision of stating is：Data-oriented collectionX_i∈R^p×q,y_i∈{C₁, C₂,…,C_N, N number of classification is matched two-by-two, thus producing N (N-1)/2 two classification task.

6. a kind of integrated study device based on singular value selection according to claim 4 it is characterised in that：Described in step 5 According to the method that relative majority ballot criterion merges base grader it is：

$y\left(X\right)=H\left(X\right)={\mathrm{argmax}}_{y\&Element;Y}\underset{t=1}{\overset{M}{\&Sigma;}}l(y=h_t(X\left)\right).$