CN104463211A - Support vector data description method based on maximum distance between centers of spheres - Google Patents

Abstract

Provided is a support vector data description method based on the maximum distance between the centers of spheres. Objectives with maximized between-class distances are added to an objective optimization function for support vector data description, so that an objective function which makes the radius of each suprasphere minimum and meanwhile makes the distances between different supraspheres maximum is obtained. Between-class information constraints are introduced to multi-class problems on the basis of the support vector data description method for the first time, the suprasphere with the minimum radius is used for surrounding the same class of samples on this basis, and the supraspheres keep away from one another as far as possible. The support vector data description method can solve the problems that classes are unbalanced and recognition dead zones exist in traditional multi-class methods, and the effectiveness of the support vector data description method applied to the multi-class problems is proved through an open set face recognition system with a rejection function. Compared with the traditional methods, the support vector data description method has the advantages of being high in robustness, good in classification effect, and the like. The support vector data description method can be used for solving supervised learning multi-class problems of small sample sets.

Description

Based on the support vector description method of maximum centre of sphere distance

Technical field

The present invention relates to a kind of support vector description method.The distance particularly related between the minimum and suprasphere of a kind of quadratic sum with all radius of hyperspheres is objective function to the maximum, in nuclear space, find a suprasphere constrains in suprasphere by of a sort sample, and constrained in outside suprasphere by the sample of other classes, and make the support vector description method based on maximum centre of sphere distance that is separated as much as possible between set up suprasphere.

Background technology

Along with the fast development of infotech, the multi-medium data such as image and video emerges in multitude, and becomes one of important channel of people's obtaining information.How effectively classifying to the information obtained is the significant challenge in machine learning field.Support vector machine (Support Vector Machine, SVM) be a kind of popular sorting technique, proposed by people such as Vapnik at first, all obtain a very large progress in its theoretical research and algorithm realization etc. in recent years, become the strong means overcoming the problem such as " dimension disaster " and " crossing study ".Its main thought finds a lineoid, can two class data points correctly be separated as much as possible, makes two class data point distance classification faces separately farthest simultaneously.Along with the proposition of support vector machine and relevant support region technology and perfect, theoretical foundation and the implementation framework of two classification problems are formed all.

But many sorting techniques are still very unripe.At present both direction is mainly contained to polytypic research: indirectly solve and directly solve.The polytypic method of indirect solution is that many classification problems are converted into two classification problems, namely forms a multi classifier with multiple two classification device.These class methods mainly contain following two kinds: one-to-many (One-Vs-All, OVA) method, one to one (One-Vs-One, OVO) method.OVA is the very simple many sorting techniques of one, is for each class builds a two classification device, for the classification of N number of classification, then will constructs N number of two classification device.Concerning the two classification device of i-th class, the formation of its training sample set for the sample belonging to i class be positive class, and other all samples not belonging to such are all negative class, but during the method training, positive and negative class Data distribution8 is uneven, causes nicety of grading to reduce.OVO method distinguishes between two multi-class data, for any two classes build Optimal Separating Hyperplane.For N class data set, then need to construct N (N-1)/2 two classification device, this method not only calculated amount is huge, and only sets up the sorter between two between classification, ignore the information with other classifications, and all there is the problem identifying blind area in the many sorting techniques of OVA and OVO.

Recent years, many researchers attempt to solve many classification problems by designing the SVM directly solving many classification problems, process Various types of data simultaneously and consider all kinds of between related information.In these class methods, foremost be adopt the method for support vector description (Support Vector Data Description, SVDD) to utilize K suprasphere to be described K class data simultaneously, and each suprasphere comprises of a sort sample data.The basic thought of SVDD is that all samples are mapped to feature space, then calculate in feature space and comprise this minimal hyper-sphere border organizing data to obtain the distributed areas of data, thus these group data are described, be mainly used to carry out single class classification and remove noise spot or singular point.Not searching lineoid with SVM unlike, SVDD but be described by calculating the distribution range of minimal hyper-sphere border to data comprising same class sample.The data being usually located at suprasphere inside are classified as target class, and the data being positioned at suprasphere border are called support vector, and outside suprasphere is then non-target sample.

Due to SVDD can be used alone to each class sample, obtain the suprasphere of each classification sample, and in this, as classification boundaries, therefore SVDD can expand to multi classifier to process many classification problems easily.Such as: the people such as Zhu utilize SVDD to classify to multi-class problem, propose a kind of Based on Sphere Structure SVMs method, the method constructs to each class training sample Solve problems the minimum sphere that comprises such sample, then judges which kind of test sample book belongs to according to test sample book from the distance of each centre of sphere.The people such as Lee propose a kind of region description support vector sorting technique solving multi-class problem based on Bayesian decision criterion, the method, first to each class training sample Solve problems structure-individual minimal hyper-sphere comprising such sample, then utilizes Bayesian formula to calculate posterior probability to judge which kind of test specimens should belong to.The discriminant function of the people such as Lei to the method that the people such as Zhu propose is modified, and carries out just using arest neighbors method to differentiate in indefinite region when classification judges indefinite from each centre of sphere distance when utilizing test sample book.The people such as Hao propose the ball-type support vector machine that solves many classification problems, other samples constrain in outside suprasphere to each class formation suprasphere to make such sample constrain in suprasphere so that the quadratic sum of all suprasphere radiuses is minimum for objective function by the method, then judge which kind of test sample book belongs to according to test sample book from the distance of each centre of sphere.Liu etc. propose a kind of SVDD Multiclass Classification based on nuclear space relative density, first this algorithm determines by SVDD the minimal hyper-sphere surrounding every class data, then calculate and be arranged in the relative density of each sample in minimal hyper-sphere overlapping region between its similar sample, last with the average of Different categories of samples relative density for standard, the sample to be tested in overlapping region is classified.Wang etc. propose structuring oneclass classification (Structured One-Class Classification) algorithm, on the basis considering Data distribution8, the multiple super ellipsoids of one class target data is described, to obtain the more effective description of target data.

In addition, current recognition technology is mostly refuse the closed set identification of knowledge for nothing, namely test sample book one matches with the sample in tranining database surely, but this situation does not meet the truth of real world applications, and opener recognition technology eliminates the hypothesis of " test sample book one matches with the sample of tranining database surely " in closed set identification, can refuse foreign peoples's sample unmatched with object library identity to know, more meet the truth in real world applications.

Summary of the invention

Technical matters to be solved by this invention is, there is provided a kind of to seek one for each class and comprise such target sample all or nearly all and the minimum optimum suprasphere of volume, and make the distance between suprasphere maximum, thus realize effective classification of multiple classification, solve the support vector description method based on maximum centre of sphere distance of data nonbalance problem between many classification problems and classification.

The technical solution adopted in the present invention is: a kind of support vector description method based on maximum centre of sphere distance, that maximized between class distance target is joined in the objective optimization function of support vector description, obtain, under the target making each suprasphere radius minimum, making the objective function that the distance between different suprasphere is maximum simultaneously.

The foundation of described objective function, first establishes for data space in a known training dataset, wherein T is the number of class, t _mbe the sample number of m class, obtain objective function:

$\begin{array}{cc}\min & \underset{m}{\mathrm{\Σ}}{R}_m^2-K\underset{m,n}{\mathrm{\Σ}}{d}_m^2+C\underset{i}{\mathrm{\Σ}}\underset{m}{\mathrm{\Σ}}{\mathrm{\ξ}}_i^m\\ s.t.& {\left|\right|\mathrm{\φ}(x_i^m-c_m)\left|\right|}^2-R_m\≤{\mathrm{\ξ}}_i^m,{\mathrm{\ξ}}_i^m\≥0,\∀i,m\end{array}---\left(1\right)$

Wherein: R _mbe the radius of m class, c _mthe centre of sphere of m class, d _mnbe the centre of sphere of m class and the distance of the n-th class centre of sphere, m, n ∈ 1 ..., T}, K are the parameter regulating radius and separation spacing, K>=0, be i-th sample of m class, C is punishment parameter, is used for controlling minimum encircle sphere radius compromise with of wrong point degree.

Support vector description method based on maximum centre of sphere distance of the present invention, under the prerequisite of supervision message utilizing training sample, fully the border of each class is portrayed, and the distance between the suprasphere of adjustment foundation, inhomogeneous sample is separated as much as possible.Make use of the information in class and between class fully.The present invention introduces between class information constrained in many classification problems first on the basis of the method for support vector description, and same class sample surrounds by the suprasphere utilizing radius minimum on this basis, and to make between suprasphere as much as possible away from.The present invention can avoid problem in traditional many sorting techniques, the problem includes: class imbalance problem and identification blind zone problem, demonstrates the validity of this invention for many classification problems by there being the opener face identification system refusing to know.The present invention, compared with classic method, has the advantage such as strong robustness, good classification effect.May be used for the many classification problems of supervised learning solving small sample set.

Accompanying drawing explanation

Fig. 1 is the schematic diagram that test sample book only falls into a certain suprasphere;

Fig. 2 is the schematic diagram that test sample book falls into multiple suprasphere;

Fig. 3 is the schematic diagram that test sample book drops on outside all supraspheres;

Fig. 4 is the process flow diagram that method of the present invention is applied to opener recognition of face.

Embodiment

Below in conjunction with embodiment and accompanying drawing, the support vector description method based on maximum centre of sphere distance of the present invention is described in detail.

Support vector description method based on maximum centre of sphere distance of the present invention, that maximized between class distance target is joined in the objective optimization function of support vector description, obtain, under the target making each suprasphere radius minimum, making the objective function that the distance between different suprasphere is maximum simultaneously.

The foundation of described objective function, first establishes for data space in a known training dataset, wherein T is the number of class, t _mbe the sample number of m class, obtain objective function:

$\begin{array}{cc}\min & \underset{m}{\mathrm{\Σ}}{R}_m^2-K\underset{m,n}{\mathrm{\Σ}}{d}_m^2+C\underset{i}{\mathrm{\Σ}}\underset{m}{\mathrm{\Σ}}{\mathrm{\ξ}}_i^m\\ s.t.& {\left|\right|\mathrm{\φ}(x_i^m-c_m)\left|\right|}^2-R_m\≤{\mathrm{\ξ}}_i^m,{\mathrm{\ξ}}_i^m\≥0,\∀i,m\end{array}---\left(1\right)$

Wherein: R _mbe the radius of m class, c _mthe centre of sphere of m class, d _mnbe the centre of sphere of m class and the distance of the n-th class centre of sphere, m, n ∈ 1 ..., T}, K are the parameter regulating radius and separation spacing, K>=0, be i-th sample of m class, C is punishment parameter, is used for controlling minimum encircle sphere radius compromise with of wrong point degree.

Objective function of the present invention can be obtained by the saddle point solving Lagrangian function:

Introduce Lagrange multiplier α _i>=0, β _i>=0, corresponding Lagrangian function is:

$L=\underset{m}{\mathrm{\Σ}}{R}_m^2-K\underset{\mathrm{mn}}{\mathrm{\Σ}}{d}_{\mathrm{mn}}^2+C\underset{i,m}{\mathrm{\Σ}}{\mathrm{\ξ}}_i^m-\underset{i,m}{\mathrm{\Σ}}{\mathrm{\α}}_i^m({\mathrm{\ξ}}_i^m+R_m^2-{\left|\right|\mathrm{\φ}\left(x_i^m\right)-c_m\left|\right|}^2)-\underset{i,m}{\mathrm{\Σ}}{\mathrm{\β}}_i^m{\mathrm{\ξ}}_i^m---\left(2\right)$

Due to the complicacy calculated, generally directly do not solve, but solve its dual problem according to Lagrange duality theory, institute in the hope of L about R _m, c _m, local derviation, and make it equal zero:

$\frac{\∂L}{\∂R_m}=2R_m-2R_m\underset{i}{\mathrm{\Σ}}{\mathrm{\α}}_i^m=0\⇒\underset{i}{\mathrm{\Σ}}{\mathrm{\α}}_i^m=1---\left(3\right)$

$\frac{\∂L}{\∂{\mathrm{\ξ}}_i^m}=C-{\mathrm{\α}}_i^m-{\mathrm{\β}}_i^m=0\⇒C={\mathrm{\α}}_i^m+{\mathrm{\β}}_i^m---\left(4\right)$

$\frac{\∂L}{\∂c_m}=2K\underset{n}{\mathrm{\Σ}}(c_n-c_m)-2\underset{i}{\mathrm{\Σ}}{\mathrm{\α}}_i^m(\mathrm{\φ}\left(x_i^m\right)-c_m)=0---\left(5\right)$

$\⇒\underset{m}{\mathrm{\Σ}}{c}_m=\underset{i,m}{\mathrm{\Σ}}{\mathrm{\α}}_i^m\mathrm{\φ}\left(x_i^m\right)$

$c_m=\frac{1}{\mathrm{TK}-1}(K\underset{i,m}{\mathrm{\Σ}}{\mathrm{\α}}_i^m\mathrm{\φ}\left(x_i^m\right)-\underset{i}{\mathrm{\Σ}}{\mathrm{\α}}_i^m\mathrm{\φ}\left(x_i^m\right))---\left(6\right)$

Be updated in (2) by (3)-(6), can obtain its dual problem is:

$\begin{array}{cc}\max & \underset{i,m}{\mathrm{\Σ}}{\mathrm{\α}}_i^{m}K(x_i^m,x_i^m)-\frac{K}{\mathrm{TK}-1}\underset{i,j,m,n=1}{\mathrm{\Σ}}{\mathrm{\α}}_i^m,{\mathrm{\α}}_j^{n}K({\mathrm{\α}}_i^m,{\mathrm{\α}}_j^n)+\frac{1}{\mathrm{TK}-1}\underset{i,j,m}{\mathrm{\Σ}}{\mathrm{\α}}_i^m{\mathrm{\α}}_j^{m}K(x_i^m,x_j^m)\\ s.t.& \underset{i}{\mathrm{\Σ}}{\mathrm{\α}}_i^m=\mathrm{1,0}\≤{\mathrm{\α}}_i^m\≤C\end{array}---\left(7\right)$

Utilize the thought of kernel function, K (x _i, x _j)=φ (x _i) φ (x _j), " " represents inner product, utilizes radial basis (RBF) kernel function in the present invention, that is: K (x _i, x _j)=exp (-q||x _i-x _j|| ²).

Above objective function is the convex programming problem of linear restriction, utilizes Novel Algorithm to solve, obtains the centre of sphere of each class can pass through formula (6) and obtain, and for the centre of sphere of each suprasphere, is known by KKT (Karush-Kuhn-Tucker) condition:

$\begin{array}{c}{\mathrm{\α}}_i^m({\left|\right|\mathrm{\φ}\left(x_i\right)-c_m\left|\right|}^2-R_m^2-{\mathrm{\ξ}}_i^m)=0,{\mathrm{\β}}_i^m{\mathrm{\ξ}}_i^m=0\\ {\left|\right|\mathrm{\φ}\left(x_i\right)-c_m\left|\right|}^2\≤R_m^2+{\mathrm{\ξ}}_i^m,C-{\mathrm{\α}}_i^m-{\mathrm{\β}}_i^m=0,{\mathrm{\ξ}}_i^m\≥0.\end{array}$

When $0<{\mathrm{\&alpha;}}_i^m<C$ Time, ${\left|\right|\mathrm{\&phi;}\left(x_i\right)-c_m\left|\right|}^2-R_m^2-{\mathrm{\&xi;}}_i^m=0,{\mathrm{\&beta;}}_i^m>0,{\mathrm{\&xi;}}_i^m=0,$

Therefore for corresponding x _ihave:

$R_m^2={\left|\right|\mathrm{\&phi;}\left(x_i\right)-c_m\left|\right|}^2$

Wherein corresponding x _iit is support vector.

Test sample book x to the distance of the centre of sphere of m suprasphere is:

$\begin{array}{c}{d}_m^2={\left|\right|\mathrm{\&phi;}\left(x\right)-c_m\left|\right|}^2\\ =K(x,x)-\frac{2K}{\mathrm{TK}-1}\underset{i,m}{\mathrm{\&Sigma;}}{\mathrm{\&alpha;}}_i^{m}K(x_i^m,x)+\frac{2}{\mathrm{TK}-1}\underset{i}{\mathrm{\&Sigma;}}{\mathrm{\&alpha;}}_i^{m}K(x_i^m,x)+c_m^2\end{array}$

Criterion for test sample book is, judges whether test sample book falls among suprasphere, if fall into suprasphere, then judge that test sample book belongs to target class, otherwise test sample book belongs to non-target class.So discriminant function is:

$f\left(x\right)=\mathrm{sign}(R_m^2-d_m^2).$

As f (x) >0, test sample book x falls into m suprasphere, otherwise drops on outside suprasphere.

For many classification problems, the relation between test sample book and the suprasphere set up has three kinds of relations:

(1) test sample book only falls into a certain suprasphere, and as shown in Figure 1, in figure, x represents test sample book, and A represents suprasphere.

(2) test sample book falls into multiple suprasphere, and as shown in Figure 2, in figure, x represents test sample book, and B, C represent suprasphere.

(3) test sample book drops on outside all supraspheres, and as shown in Figure 3, in figure, x represents test sample book, and D, E, F represent suprasphere.

In closed set identification, for situation about occurring in relation (2), (3), utilize k nearest neighbor to determine the ownership of this sample class in the present invention.And for opener identification, for situation about occurring in relation (2), utilize k nearest neighbor to determine the ownership of this test sample book classification, and for test sample book in relation (3), then determine that it is foreign peoples's sample, model will be refused.

With traditional " one-to-many ", " one to one " many sorting techniques unlike, the distance of the present invention with the quadratic sum of all radius of hyperspheres between minimum and suprasphere is objective function to the maximum, of a sort sample surrounds by the hypersphere utilizing radius minimum, make distance between suprasphere farthest simultaneously, be that " disposable " sets up multiple suprasphere, instead of design multiple two sorters and realize polytypic object.The present invention takes into full account the relation between classification, avoids the problem of class imbalance; Model sets up suprasphere for target simultaneously, effectively can solve the problem identifying blind area.

Illustrate that the present invention is having the application refusing the opener recognition of face known below in conjunction with Fig. 4.It should be noted that, the present invention not only can be applied in opener recognition of face, also can be applied in the opener identification of other biological feature.

(1) Image semantic classification and feature extraction

First align to facial image, the pretreatment operation such as unitary of illumination, then extract the feature of facial image;

(2) eigentransformation

In order to be described sample data better, eigentransformation be carried out to the raw data feature of facial image extracted, the method based on core, the method based on subspace, method etc. based on manifold learning can be adopted;

(3) utilize method establishment of the present invention to have to refuse the opener face classification model known

Utilize many disaggregated models of the present invention by suprasphere encirclement minimum for each class face sample radius, and make the spacing of different face classes maximum, reach the object that different classes is separated as far as possible;

(4) face sample to be measured is tested

Face sample to be measured carried out to pre-service and extracts feature, the high dimensional feature of extraction is converted into the low dimensional feature with discriminant information after Feature Conversion, if face sample to be measured is fallen in certain suprasphere, then this sample to be tested being classified as this type of; Judge its ownership according to the size of the relative density average of Different categories of samples nuclear space around test sample book when multiple suprasphere comprises test sample book simultaneously; If face sample to be measured drops on outside all supraspheres, think that this face sample to be measured does not belong to face sample in object set, and refused.

Claims (2)

1. the support vector description method based on maximum centre of sphere distance, it is characterized in that, that maximized between class distance target is joined in the objective optimization function of support vector description, obtain, under the target making each suprasphere radius minimum, making the objective function that the distance between different suprasphere is maximum simultaneously.

2. the support vector description method based on maximum centre of sphere distance according to claim 1, it is characterized in that, the foundation of described objective function, first establishes for data space in a known training dataset, wherein T is the number of class, t _mbe the sample number of m class, obtain objective function:

$\min \underset{m}{\mathrm{\&Sigma;}}{R}_m^2-K\underset{m,n}{\mathrm{\&Sigma;}}{d}_{\mathrm{mn}}^2+C\underset{i}{\mathrm{\&Sigma;}}\underset{m}{\mathrm{\&Sigma;}}{\mathrm{\&xi;}}_i^m---\left(1\right)$

$\begin{array}{cccc}s.t.& {\left|\right|\mathrm{\&phi;}(x_i^m-c_m)\left|\right|}^2-R_m\&le;{\mathrm{\&xi;}}_i^m& {\mathrm{\&xi;}}_i^m\&GreaterEqual;0& \&ForAll;i,m\end{array}$

Wherein: R _mbe the radius of m class, c _mthe centre of sphere of m class, d _mnbe the centre of sphere of m class and the distance of the n-th class centre of sphere, m, n ∈ 1 ..., T}, K are the parameter regulating radius and separation spacing, K>=0, be i-th sample of m class, C is punishment parameter, is used for controlling minimum encircle sphere radius compromise with of wrong point degree.