CN102646198A - Mode recognition method of mixed linear SVM (support vector machine) classifier with hierarchical structure - Google Patents

Abstract

The invention discloses a mode recognition method of a mixed linear SVM (support vector machine) classifier with a hierarchical structure. The iteration process of the mode recognition method mainly comprises the following steps of: firstly updating the training weight of each sample by applying the neighbor geometrical structural relation of training data and the classification error of the current H-MLSVMs classifier; and secondly training a linear SVM classifier with a weight by utilizing the samples with the weights, embedding the classifier into the current H-MLSVMs, and updating the classification error of the classifier. The mode recognition method disclosed by the invention is an effective mode recognition and classification method and is a universal method. Experimental results show that compared with other classical SVM methods, the classification method has the popularization capability which is always the same with a kernel function-based SVM method, has no shortcomings of difficulty in selection of a kernel function and parameters thereof, very large occupation of a memory and the like and particularly has the classification time complexity in the same order with the linear SVM; and furthermore, the classification method is more effective when the classification method is used for classifying new samples, can be further applied to a real-time mode recognition system, and has great application prospects.

Description

Mode identification method with mixed linear svm classifier device of hierarchical structure

Technical field

The present invention relates to computer patterns identification field, particularly relate to a kind of mode identification method with mixed linear svm classifier device of hierarchical structure.

Background technology

Classifier design is the research emphasis in the computer patterns identification field, because sorter is the basic tool as pattern identification research.Generally speaking, present main two critical problems of classifier algorithm: the popularization ability of (1) sorter; (2) new samples is carried out the spent time of branch time-like.

In general, the popularization ability of sorter is exactly the ability of sorter prediction unknown sample classification, the i.e. height of classification accuracy rate.Based on Statistical Learning Theory and the theoretical svm classifier device of structural risk minimization; Because it has advantages such as globally optimal solution and excellent popularization ability; Become the most successful in recent years sorter, be widely used in various fields such as artificial intelligence, machine learning, Neurobiology, medical science.Its initial motivation is exactly two types of set of data samples for linear separability, finds out the largest interval that can separate these two types of samples.But in practical application, many grouped datas are linear inseparable.For this reason; People have introduced the notion of kernel function; The data of luv space are mapped in higher-dimension even the infinite dimension nuclear space through nuclear; And these are mapped to data in the nuclear space and can be regarded as that linear separability or approximately linear can divide, thereby have solved for the inseparable classification of Data problem of linearity.Though successfully expanded the range of application of linear SVM based on the SVM of kernel method, and the excellent popularization ability is arranged, it still has following three big shortcomings:

1) difficulty selected of kernel function and parameter thereof: a lot of kernel functions are arranged at present, and like polynomial kernel function, batten kernel function, basic kernel function etc. radially, each kernel function has parameter separately again.Kernel function and parameters of choice thereof have very important effect for the popularization ability of svm classifier device.How to select suitable kernel function and parameter thereof to make the popularization ability of sorter reach and preferably remain a difficult problem that needs to be resolved hurrily.

2) need occupy very big internal memory during training classifier: need store an O (l when the training based on the SVM of kernel function ²) non-sparse nuclear matrix, wherein l is the number of training sample.This just makes SVM can not handle the classification problem of big data quantity.

3) the classification time complexity is high: the svm classifier device based on kernel function is carrying out the branch time-like to new samples, needs to calculate the kernel function between this new samples and all support vectors.Number linear dependence that so its complicated classification degree is a calcaneus branches holds vector.And the growth of the number of Steinwart proof support vector increases linear with the training sample number at least.Therefore, the sorter that big data quantity trains out, to the classification time of new data sample will be very slow, this just makes it can not apply in some real-time mode recognizing systems, such as pedestrian detecting system and aircraft vision navigation system.

In order to overcome the above-mentioned defective based on kernel function SVM, the method for some structural classification devices in original data space has been carried.Fu has proposed the linear svm classifier device of a mixing based on the joint distribution model of training data and label.But, because the joint distribution of data is difficult to estimate that when the DATA DISTRIBUTION of reality did not conform to the probability model of hypothesis, the popularization ability of sorter will reduce greatly.Li has proposed a piecewise linear classifier---many convex lines property perceptron based on the notion of " the protruding branch ".But because this sorter do not introduce the notion at soft interval, so it can not handle those two types of grouped datas that mutual overlapping is arranged, so just greatly limitations range of application.

Summary of the invention

The present invention provides a kind of mode identification method---H-MLSVMs with mixed linear svm classifier device of hierarchical structure, and it has the hierarchical tree structure, and each node place all corresponding the linear svm classifier device of a mixing.With respect to other classical SVM method; This sorting technique has and based on the SVM method of kernel function popularization ability about the same; Do not select difficulty, take very shortcoming such as big internal memory and do not exist kernel function and parameter thereof; Particularly have the classification time complexity with rank such as linear SVM, make this sorting technique divide time-like more effective new samples.

To achieve these goals, the technical scheme below the present invention has adopted:

Mode identification method with mixed linear svm classifier device of hierarchical structure, the sorter at its each node place all corresponding an iterative process.Iterative process mainly comprises following two steps each time:

(1) uses the neighbour's geometric relationship of training data and the error in classification of current H-MLSVMs sorter, upgrade the training weight of each sample;

(2) utilize the sample training that has weight to go out the heavy linear svm classifier device of a cum rights, this sorter is embedded among the current H-MLSVMs, and upgrades the error in classification of sorter, turn back to the first step.

Wherein, H-MLSVMs root node sorter is to obtain through the linear SVM that trains equal weight.Above-mentioned iterative process until among the H-MLSVMs number of all node place sorters reach a given integer, perhaps when the increasing degree of accuracy on the test data set is within enough little scope, stop.

Furtherly, described step (1) specifically comprises following substep:

At first, select " training centre's sample " through the neighbour's geometric relationship of training data and the error in classification of current H-MLSVMs sorter, promptly near the zone this " training centre's sample " is the most serious zone of the wrong branch of current H-MLSVMs sorter;

Secondly, according to each sample and the distance between the last step selected " training centre's sample ", calculated the training weight of each sample;

At last, in order to prevent to train weight to be partial to a certain type, in all kinds of, will train weight to do normalization and handle.

Furtherly, described step (2) specifically comprises following substep:

At first, the training sample that has weight above the utilization trains the heavy linear svm classifier device of a cum rights;

Secondly, combine " training centre's sample " to belong to the sorter at father and mother's node place of node, obtain a mixed linear svm classifier device;

Once more, with the sorter of this mixed linear svm classifier device as node place, " training centre's sample " place;

At last, upgrade the error in classification of sorter.

The invention has the beneficial effects as follows:

1, sorting algorithm proposed by the invention is a kind of method in common, and the arbitrary classification data type all is suitable for.

2, the present invention proposes a kind of self-adaptation adjustment weight strategy; Can automatically adjust the training weight of each sample according to current error in classification; The method that weight is set than classics; Owing to not only considered error in classification, and incorporated the geometric relationship of data, make this weight method to set up to noise point robust more.

3, the present invention has adopted a kind of sorter of hierarchical tree structure; Wherein each node place all exists a mixed linear svm classifier device; Its basic thought is exactly that layering ground is separated the positive sample of mistake branch from negative sample, and the negative sample that perhaps mistake is divided is separated from positive sample.Linear svm classifier device than classics has greatly improved the popularization ability to the inseparable data of linearity; Svm classifier device than classics based on kernel function; The popularization ability of this H-MLSVMs sorter can be similar to the popularization ability that reaches nuclear svm classifier device; And overcome nuclear SVM in training process kernel function and selection of parameter thereof difficulty, consume shortcoming such as internal memory; The more important thing is to make the classification time complexity that needs reduce greatly, thereby can be used in the real-time mode recognizing system.

Description of drawings

Below in conjunction with accompanying drawing and embodiment the present invention is further specified.

Fig. 1 is the training process flow diagram of H-MLSVMs sorter of the present invention;

Fig. 2 is the two-dimentional example exploded view of sorter training process of the present invention;

Fig. 3 is the process flow diagram of sorter of the present invention to the assorting process of new samples.

Embodiment

Through embodiment the present invention is carried out concrete description below; Only be used for the present invention is further specified; Can not be interpreted as the qualification to protection domain of the present invention, the technician in this field can make some nonessential improvement and adjustment to the present invention according to the content of foregoing invention.

Fig. 1 is the training process flow diagram of H-MLSVMs sorter of the present invention, and it is a kind of mode identification method with mixed linear svm classifier device of hierarchical structure.Fig. 2 is that the two-dimentional example of sorter training process of the present invention is showed.Fig. 3 is the process flow diagram of H-MLSVMs sorter to the assorting process of new samples.The hardware and the programming language of the concrete operation of method of the present invention do not limit, and can accomplish with any language, and other mode of operation repeats no more for this reason.

Embodiments of the invention adopt a Pentium 4 computing machine with 3.2G hertz central processing unit and 1G byte of memory also to work out the working routine of H-MLSVMs sorter with the Matlab language, have realized method of the present invention.

Mode identification method with mixed linear svm classifier device of hierarchical structure of the present invention mainly comprises following two steps: the renewal of the renewal of training sample weight and hierarchical structure sorter H-MLSVMs.

Before introducing concrete steps, below we introduce earlier the meaning of the symbol that will use.

S: the training sample set, promptly $S=\left\{(x_i,y_i)\right|x_i\∈R^n,y_i\∈\{-\mathrm{1,1}\}{\}}_{i=1}^l$

I: the node in the expression H-MLSVMs sorter.I=[0] represents root node, as

Its division of fruit, [0,1] and [0 ,-1] is its left child's node and right child's node just.Similarly, if node [0,1] divides again, then [0,1,1] and [0,1 ,-1] is its left child's node and right child's node just.If a node no longer divides, then this node just is called leafy node.The number of element is being represented the number of plies of node I in H-MLSVMs among the I.Shown in Fig. 2 (f), node [0,1,1,1] is at the 4th layer of H-MLSVMs;

F ^(t): in the process of training H-MLSVMs sorter, the linear SVM that the cum rights that is calculated during the t time iteration is heavy;

F _I: at the sorter (shown in Fig. 2 (f)) at node I place, wherein the leafy node place does not have sorter;

S _I: be divided into node I place sample set.Shown in Fig. 2 (f), all training samples the most all are divided into the root node place, that is:

Sorter f at the root node place _[0]With S set _[0]Be divided into S set _[0,1]And S _{[0 ,-1]}Sorter f at [0,1] node place _[0,1]Again with S _[0,1]Be divided into S _[0,1,1]And S _{[0,1 ,-1]}Similarly, the set at other node places also can obtain.We can find out that the set at leaf node place is non-intersect in twos, and their union has constituted whole training samples.If among the note leafy node I last element be I (| I|), then I (| I|) determined S set _IThe category attribute of middle sample.I (| I|)=1 (I (| I|)=-1) represents S _IMiddle sample is divided into positive type (negative type) by the H-MLSVMs sorter;

The number of all node place sorters among the M:H-MLSVMs;

F _t: the set of node place sorter among the H-MLSVMs after the t time iteration, shown in Fig. 2 (f), F ₄={ f _[0], f _[0,1], f _[0,1,1], f _{[0 ,-1] }}

$V^{\left(t\right)}:V^{\left(t\right)}=[V_1^{\left(t\right)},...,V_l^{\left(t\right)}],$ Wherein

Be illustrated in H-MLSVMs after the t time iteration

Sorter is to sample x _iPredicted value.

Red circle is represented positive sample among Fig. 2, blue fork expression negative sample.The training process and the classification process of H-MLSVMs sorter are described with reference to figure 2 below, and concrete steps are described below:

Initial H-MLSVMs: all training samples all are divided into the root node place, that is:

Training sample with equal weight trains a linear svm classifier device f ⁽¹⁾(shown in Fig. 2 (a)), and give root node place sorter, i.e. f with it _[0]=f ⁽¹⁾f _[0]With S set _[0]Be divided into S set _[0,1]And S _{[0 ,-1]}, S wherein _[0,1]={ x|f _[0](x)>=0, x ∈ S _[0], S _{[0 ,-1]}={ x|f _[0](x)＜0, x ∈ S _[0], can find out S _[0,1]In the negative sample that is divided by mistake is arranged in a large number, S _{[0 ,-1]}In the positive sample that is divided by mistake is arranged in a large number.This moment H-MLSVMs F ₁Structure such as Fig. 2 (f) shown in, F ₁Predicted value to training sample is V ⁽¹⁾=F ₁(S _[0])=f _[0](S _[0]).

Get into the cycle stage below, it mainly contains following two steps

(1) uses the neighbour's geometric relationship of training data and the error in classification of current H-MLSVMs sorter, upgrade the training weight of each sample.It mainly comprises:

A) find out " training centre's sample " (shown in Fig. 2 (b) Smalt triangle)

Neighbours, it is defined as N _ε(x _i)={ x _j|| | x _j-x _i||＜ε, y _i=y _i, (x _j, y _j) ∈ S}, the general value of ε does

B) according to the distance calculation sample weights of sample, promptly apart from " training centre's sample "

${\tilde{v}}_i=\exp \{-\frac{{\left|\right|x_i-x_c\left|\right|}^{-}}{\_2}\}$

C) train weight to be partial to a certain type in order to prevent; To train weight to do normalization processing

shown in Fig. 2 (b) in all kinds of, the size of mark is being represented the size of sample weights.

Can find out that from top computation process we are with sorter F ₁Mistake divides the sample in the most serious zone to give very big weight.

(2) utilize the sample training that has weight to go out the heavy linear svm classifier device of a cum rights, this sorter is embedded among the current H-MLSVMs, and upgrade the error in classification of sorter.It mainly comprises:

A) shown in Fig. 2 (b), heavy with cum rights

Training sample train the heavy linear svm classifier device f of cum rights ⁽²⁾(rhodo colo(u)r streak among Fig. 2 (b));

B) suppose x _c∈ S _I, S set promptly is described _IThe middle wrong sample that divides is the most serious.Make I ₁=I (1:|I|-1), I ₂=I (| I|), I then ₁Be father and mother's node of I.Datum node I ₁Place's sorter obtains a mixed linear svm classifier device.If I (| I|)=1, then this mixed linear svm classifier device does

If I (| I|)=-1, then this mixed linear svm classifier device does

In Fig. 2 (b), because I=[0,1], so mixed linear svm classifier device is min{f ⁽²⁾, f _[0];

C) the linear svm classifier device min{f that obtains above the general ⁽²⁾, f _[0]Give the node sorter f that [0,1] is located _[0,1], f _[0,1]Further with S set _[0,1]Be divided into S _[0,1,1]And S _{[0,1 ,-1]}, S wherein _[0,1,1]={ x|f _[0,1](x)>=0, x ∈ S _[0,1], S _{[0,1 ,-1]}={ x|f _[0,1](x)＜0, x ∈ S _[0,1], can find out, with f _[0,1]S set _[0,1]A middle part is by F ₁The wrong negative sample that divides is separated.The H-MLSVMs of this moment is F ₂=F ₁∪ { f _[0,1], can find out F ₂Compare F ₁Had more a sorter f of node place _[0,1]Thereby, improved classification accuracy rate.Shown in its structure such as Fig. 2 (f);

D) calculate F ₂To the predicted value of training sample, promptly Return (1).

Repeat to be similar to the process of step (1) (2), we have obtained H-MLSVMs sorter F ₃(shown in Fig. 2 (c)) and F ₄(shown in Fig. 2 (d)), its structure is respectively shown in Fig. 2 (f).Above-mentioned iterative process until among the H-MLSVMs number of all node place sorters reach a given integer, perhaps when the increasing degree of accuracy on the test data set is within enough little scope, stop.In test, we adopt first kind of iteration stopping method.Fig. 2 (e) is the classification boundaries through H-MLSVMs sorter after 25 iteration.

A sample needs H-MLSVMs to divide time-like when newly arriving; Let this new sample arrive certain leafy node from the root node of hierarchical structure sorter H-MLSVMs always; Judge its classification according to the functional value that calculates, concrete classification process is as shown in Figure 3.

Because F _MIn comprised the sorter at all node places among the H-MLSVMs, so among Fig. 2 " " indicate that x has arrived leafy node, export its functional value and get final product, if its functional value more than or equal to zero, it be positive type, otherwise is negative class." f _{[I, 1]}∈ F _M(f _{[I ,-1]}∈ F _M) " mean that x does not also arrive leafy node, need with sorter " f _{[I, 1]}(f _{[I ,-1]}) " further divide.Shown in sample x (pentagram) among Fig. 2 (c), the path that first three step is walked in its assorting process is shown in Fig. 2 (f) arrow.

Claims (3)

1. mode identification method with mixed linear svm classifier device of hierarchical structure is characterized in that may further comprise the steps:

(1) uses the neighbour's geometric relationship of training data and the error in classification of current H-MLSVMs sorter, upgrade the training weight of each sample;

(2) utilize the sample training that has weight to go out the heavy linear svm classifier device of a cum rights, this sorter is embedded among the current H-MLSVMs, and upgrades the error in classification of sorter, turn back to the first step;

Wherein, H-MLSVMs root node sorter is to obtain through the linear SVM that trains equal weight; The number of above-mentioned iterative process all node place sorters in H-MLSVMs reaches a given integer; Perhaps when the accuracy on the test data set increases within enough little scope, stop.

2. the mode identification method with mixed linear svm classifier device of hierarchical structure according to claim 1 is characterized in that described step (1) specifically comprises following substep:

At first, select " training centre's sample " through the neighbour's geometric relationship of training data and the error in classification of current H-MLSVMs sorter, promptly near the zone this " training centre's sample " is the most serious zone of the wrong branch of current H-MLSVMs sorter;

Secondly, according to each sample and the distance between the last step selected " training centre's sample ", calculated the training weight of each sample;

At last, in order to prevent to train weight to be partial to a certain type, in all kinds of, will train weight to do normalization and handle.

3. the mode identification method with mixed linear svm classifier device of hierarchical structure according to claim 1 is characterized in that described step (2) specifically comprises following substep:

At first, the training sample that has weight above the utilization trains the heavy linear svm classifier device of a cum rights;

Secondly, combine " training centre's sample " to belong to the sorter at father and mother's node place of node, obtain a mixed linear svm classifier device;

Once more, with the sorter of this mixed linear svm classifier device as node place, " training centre's sample " place;

At last, upgrade the error in classification of sorter.

Images