CN104732242A - Multi-classifier construction method and system - Google Patents

Abstract

The invention discloses a multi-classifier construction method and device. The method comprises the steps of processing a training sample set containing multiple classes of sample data into multiple two-class data sets, conducting feature selection on each two-class data set to obtain a corresponding feature index subset, combining multiple feature index subsets to obtain a feature index set, and conducting modeling on the training sample set subjected to feature selection to obtain a target multi-classifier. According to the method, a multi-class problem is broken up into multiple two-class problems, redundancy feature elimination is conducted on each two-class problem, so that each sub-classifier has the feature selection capacity (it can be simply interpreted that each feature index subset corresponds to one sub-classifier), and then feature selection can be conducted on data to be tested based on the feature selection capacity of each sub-classifier in advance during subsequent classification diagnosis (specifically, the feature index set obtained after fusion of all the feature subsets is used for feature selection). In this way, the problems of the prior art are solved, and diagnosis accuracy is improved.

Description

A kind of multi-categorizer construction method and system

Technical field

The invention belongs to many sorting techniques field of support vector machine (SVM, Support Vector Machine), particularly relate to a kind of multi-categorizer construction method and system.

Background technology

In many classification problems, some data, as DNA (Deoxyribonucleic acid, DNA (deoxyribonucleic acid)) to have dimension high for gene expression data in microarray data analysis, sample is little, the feature such as non-linear, thus when classifying to these type of data, very crucial to the feature selecting process of data.

At present, the multi-categorizer of support vector machine, for example, MSVM-RFE (the multiclassSVM-Recursive Feature Elimination that the people such as Shieh propose in article " MulticlassSVM-RFE for product form feature selection ", multiclass SVM-recursive feature is eliminated) algorithm, it is considered that the weight fusion of all sub-classifiers (weighing criteria by feature is selected as feature in the weight quadratic sum on all kinds of), and each sub-classifier self forming multi-categorizer does not possess the ability selecting feature, the accuracy rate finally causing classification to be diagnosed is lower.

Summary of the invention

In view of this, the object of the present invention is to provide a kind of multi-categorizer construction method and system, be intended to solve existing support vector machine multi-categorizer because of its each sub-classifier and self do not possess the ability selecting feature, and the problem causing classification accuracy lower.

For this reason, the present invention's openly following technical scheme:

A kind of multi-categorizer construction method, comprising:

The first training sample set comprising l class sample data is treated to l two class data acquisitions; The two class data that described two class data acquisitions comprise are: according to the two class data presetting sorting technique and to re-start described l class sample data gained after two class category division, l be greater than 1 natural number;

According to the feature selection approach preset, feature selecting is carried out to each described two class data acquisitions, obtain corresponding aspect indexing subset;

Merge each aspect indexing subset, obtain aspect indexing set;

Utilize support vector machines model to carry out modeling to the second training sample set, obtain target multi-categorizer; Described second training sample set is the sample set described first training sample set being carried out to gained after feature selecting, and the sample characteristics of the second training sample set is corresponding with the feature that described aspect indexing set comprises.

Said method, preferably, described default sorting technique is one-to-many OVA method, and described default feature selection approach is that support vector machine-recursive feature eliminates SVM-RFE method.

Said method, preferably, described first training sample set is wherein:

X _ifor sample data, x _i∈ R ^d, R is real number space;

Y _ix _iclass label, y _i∈ 1,2 ..., l}, l are the numbers of classification;

N is total number of training sample;

D is the dimension of sample.

Said method, preferably, described two class data acquisitions are $X_j={\{x_i,v_i\}}_{i=1}^N,v_i=\left\{\begin{array}{cc}+1,& y_i=j\\ -1,& y_i\≠j\end{array}\right.;$ Described aspect indexing subset is described aspect indexing set is wherein:

j＝1,…,l。

Said method, preferably, described second training sample set is wherein, x _i' for carrying out the sample data after feature selecting, x _i' ∈ R ^{| F|}.

Said method, preferably, also comprises:

Utilize described aspect indexing set to carry out feature selecting to the first test sample book, obtain the second test sample book;

Described target multi-categorizer is utilized to carry out classification diagnosis to described second test sample book.

A kind of multi-categorizer constructing system, comprising:

Processing module, for being treated to l two class data acquisitions by the first training sample set comprising l class sample data; The two class data that described two class data acquisitions comprise are: according to the two class data presetting sorting technique and to re-start described l class sample data gained after two class category division, l be greater than 1 natural number;

Fisrt feature selects module, for carrying out feature selecting according to the feature selection approach preset to each described two class data acquisitions, obtains corresponding aspect indexing subset;

Merging module, for merging each aspect indexing subset, obtaining aspect indexing set;

MBM, for utilizing support vector machines sorter to carry out modeling to the second training sample set, obtains target multi-categorizer; Described second training sample set is the sample set described first training sample set being carried out to gained after feature selecting, and the sample characteristics of the second training sample set is corresponding with the feature that described aspect indexing set comprises.

Said system, preferably, also comprises:

Second feature selects module, for utilizing described aspect indexing set to carry out feature selecting to the first test sample book, obtains the second test sample book;

Classification diagnostic module, carries out classification diagnosis for utilizing described target multi-categorizer to described second test sample book.

From above scheme, the training sample set comprising multiclass sample data is treated to multiple two class data acquisitions by the present invention; And feature selecting is carried out to each two class data acquisitions, obtain corresponding aspect indexing subset; And merge multiple aspect indexing subset and obtain an aspect indexing set; Afterwards modeling is carried out to the training sample set after feature selecting, obtain target multi-categorizer.Visible, the present invention is by being decomposed into multi-class problem multiple two class problems, and redundancy feature rejecting is carried out to each two class problems, make each sub-classifier (simply can be interpreted as the corresponding sub-classifier of each aspect indexing subset) possess feature and select ability; Thus follow-up when carrying out classification diagnosis, ability can be selected based on the feature of each sub-classifier in advance and feature is carried out to testing data select (the application specifically utilize and merge each character subset after the aspect indexing set of gained carry out feature selecting).The visible problem that present application addresses prior art, improves accuracy rate of diagnosis.

Accompanying drawing explanation

In order to be illustrated more clearly in the embodiment of the present invention or technical scheme of the prior art, be briefly described to the accompanying drawing used required in embodiment or description of the prior art below, apparently, accompanying drawing in the following describes is only embodiments of the invention, for those of ordinary skill in the art, under the prerequisite not paying creative work, other accompanying drawing can also be obtained according to the accompanying drawing provided.

Fig. 1 is a kind of process flow diagram of multi-categorizer construction method disclosed in the embodiment of the present invention one;

Fig. 2 is the another kind of process flow diagram of multi-categorizer construction method disclosed in the embodiment of the present invention two;

Fig. 3 is the classification performance comparison diagram of the present invention and MSVM-RFE method disclosed in the embodiment of the present invention two;

Fig. 4 is a kind of structural representation of multi-categorizer constructing system disclosed in the embodiment of the present invention three;

Fig. 5 is the another kind of structural representation of multi-categorizer constructing system disclosed in the embodiment of the present invention three.

Embodiment

For the purpose of quoting and know, the technical term hereinafter used, to write a Chinese character in simplified form or summary of abridging is explained as follows:

SVM model: i.e. SVM classifier, SVM model have employed structural risk minimization, by an optimal hyperlane, is separated error-free by two class samples, and makes the class interval between two classes reach maximum.

SVM-RFE:Support Vector Machine-Recursive Feature Elimination, support vector machine-recursive feature is eliminated, the method is initialized as whole gene sets the characteristic set needed, then reject the minimum gene of a ranking criteria mark at every turn, until obtain last feature set, SVM-RFE is a sequence backward selection algorithm (SBS, SequentialBackward Selection) based on the largest interval principle of SVM.

Below in conjunction with the accompanying drawing in the embodiment of the present invention, be clearly and completely described the technical scheme in the embodiment of the present invention, obviously, described embodiment is only the present invention's part embodiment, instead of whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art, not making the every other embodiment obtained under creative work prerequisite, belong to the scope of protection of the invention.

Embodiment one

The embodiment of the present invention one discloses a kind of multi-categorizer construction method, and with reference to figure 1, described method can comprise the following steps:

S101: the first training sample set comprising l class sample data is treated to l two class data acquisitions; The two class data that described two class data acquisitions comprise are: according to the two class data presetting sorting technique and to re-start described l class sample data gained after two class category division, l be greater than 1 natural number.

Described first training sample set, the training sample set namely not carrying out feature selecting can be expressed as:

$X={\{x_i,y_i\}}_{i=1}^N---\left(1\right)$

Wherein, x _ifor sample data, x _i∈ R ^d, R is real number space; y _ix _iclass label, y _i∈ 1,2 ..., l}, l are the numbers of classification; N is total number of training sample; D is the dimension of sample.

This step adopts OVA (One-Versus-All, one-to-many) method, pre-service is carried out to described first training sample set, namely particularly, a classification conduct+1 is selected from the l class sample data that described first training sample set comprises, other classifications all regard-1 as, with this understanding former data (i.e. original l class sample data) are carried out to the classification based training of+1 ,-1 liang of class classification, after successively the every class data in l class sample data have been carried out the training of l wheel as+1 classification, l two class data acquisitions can be obtained:

$X_j={\{x_i,v_i\}}_{i=1}^N,v_i=\left\{\begin{array}{cc}+1,& y_i=j\\ -1,& y_i\≠j\end{array}\right.---\left(2\right)$

Wherein, j=1 ..., l.

The application specifically explains described method with embryo data set (Cns5c) data instance in medical field.Described embryo data centralization comprises the data sample of 42 patients altogether, each sample has 989 genes, wherein, 42 samples comprise PNETs (S) on tumour (CRE) shaft-like inside and outside 10 medulloblastomas (M), the central nervous system (CNS/RTs) of 5 embryonal tumors and 5 kidneys, 8 curtains, 10 without plumule brain tumor (glioblastoma) (MG), 4 normal mankind's cerebellar tissue (N), this 5 class sample data.

It should be noted that, inside and outside the central nervous system (CNS/RTs) of embryonal tumors and kidney, shaft-like tumour all belongs to CRE herein.

Next, this example adopts the method for 50% resampling that described embryo data set is divided into following two subsets:

1) 21 training samples, specifically comprise 5 M, 5 CRE, 5 MG, 2 N and 4 S, be used for Select gene and adjustment sorter weight;

2) 21 test sample books, specifically comprise 5 M, 5 CRE, 5 MG, 2 N and 4 S, be used for testing, evaluate constructed by the performance of sorter; In order to reach good experiment effect, in described training sample and test sample book, there is not overlapping sample.

Each sample has 989 genes, each gene all can be used as a feature of sample, thus each sample has 989 features, in this example, described M is considered as the first kind, CRE is considered as Equations of The Second Kind, and MG is considered as the 3rd class, N is depending on being considered as the 4th class, S is considered as the 5th class, uses numeral 1,2 respectively ... ..5 represent, then at the training sample set be made up of described 21 training samples x _i∈ R ^d, y _i∈ 1,2 ..., and in l}, N=21, D=989, l=5.

Adopt described OVA method, after pre-service is carried out to the training sample set of described 21 training samples formation, 5 two class data acquisition X can be obtained _j, wherein $X_j={\{x_i,v_i\}}_{i=1}^N,v_i=\left\{\begin{array}{cc}+1,& y_i=j\\ -1,& y_i\≠j\end{array}\right.,$ N＝21，j＝1,…,5。

S102: according to the feature selection approach preset, feature selecting is carried out to each described two class data acquisitions, obtain corresponding aspect indexing subset.

S103: merge each aspect indexing subset, obtain aspect indexing set.

Obtaining l two class data acquisition X _j, j=1 ..., after l, this step adopts SVM-RFE (SVM-Recursive Feature Elimination, support vector machine-recursive feature is eliminated) method to X _jcarry out feature selecting, obtain characteristic of correspondence subset of indices:

$F_j\⊆\{1,...,D\},j=1,...,l---\left(3\right)$

Then, the l obtained an aspect indexing subset is merged into a final aspect indexing set, achieve the fusion of multiple features subset, after merging, the aspect indexing set of gained is:

$F={\∪}_{j=1}^l{F}_j---\left(4\right).$

In the example of above-mentioned medical data, utilize SVM-RFE method to 5 two class data acquisition X _j, j=1 ..., 5 carry out feature selecting, correspondingly can obtain 5 aspect indexing subsets j=1 ..., 5, merge described 5 aspect indexing subsets afterwards, obtain aspect indexing set

S104: utilize support vector machines model to carry out modeling to the second training sample set, obtain target multi-categorizer; Described second training sample set is the sample set described first training sample set being carried out to gained after feature selecting, and the sample characteristics of the second training sample set is corresponding with the feature that described aspect indexing set comprises.

The training sample set after feature selecting is carried out in the order of this step, and namely described second training sample set is:

$X^{\′}={\{x_i^{\′},y_i\}}_{i=1}^N---\left(5\right)$

Wherein, x _i' for carrying out the sample data after feature selecting, x _i' ∈ R ^{| F|}.

On this basis, adopt SVM model to the data acquisition after feature selecting carry out modeling, obtain the pattern function f () of target multi-categorizer.

In the example of above-mentioned medical data, after adopting SVM model to carry out modeling to the data acquisition (comprising 5 class sample datas) after feature selecting, finally can obtain the medical diagnosis system that can be carried out medical diagnosis, this system specifically by differentiating that patient data belongs to any classification in described five class classifications (M, CRE, MG, N and S), realizes medical diagnosis.

The application is by carrying out feature selecting to each two classes data acquisition, make each sub-classifier (simply can be interpreted as the corresponding sub-classifier of each aspect indexing subset, the feature that subclassification has is selected the feature that ability is feature based subset of indices and is selected ability) possess feature and selected ability, when the follow-up sorter model utilizing the application to build carries out classification diagnosis to testing data, ability can be selected based on the feature of each sub-classifier in advance and feature selecting is carried out to testing data, reject the redundancy feature in testing data, the classification diagnostic result achieved as finally obtaining high-accuracy provides support.

From above scheme, the training sample set comprising multiclass sample data is treated to multiple two class data acquisitions by the present invention; And feature selecting is carried out to each two class data acquisitions, obtain corresponding aspect indexing subset; And merge multiple aspect indexing subset and obtain an aspect indexing set; Afterwards modeling is carried out to the training sample set after feature selecting, obtain target multi-categorizer.Visible, the present invention is by being decomposed into multi-class problem multiple two class problems, and redundancy feature rejecting is carried out to each two class problems, make each sub-classifier (simply can be interpreted as the corresponding sub-classifier of each aspect indexing subset) possess feature and select ability; Thus follow-up when carrying out classification diagnosis, ability can be selected based on the feature of each sub-classifier in advance and feature is carried out to testing data select (the application specifically utilize each character subset to merge after the aspect indexing set of gained carry out feature selecting).The visible problem that present application addresses prior art, improves accuracy rate of diagnosis.

Embodiment two

In the present embodiment two, with reference to figure 2, described method can also comprise the following steps:

S105: utilize described aspect indexing set to carry out feature selecting to the first test sample book, obtain the second test sample book;

S106: utilize described target multi-categorizer to carry out classification diagnosis to described second test sample book.

At employing SVM model, modeling is carried out to the data acquisition after feature selecting, after obtaining target multi-categorizer, target multi-categorizer can be utilized to carry out classification diagnosis to testing data.

In the present embodiment, testing data is made to be x, wherein, x ∈ R ^d.

When utilizing target multi-categorizer to carry out classification diagnosis to testing data, first the aspect indexing set F utilizing embodiment one to provide, feature selecting is carried out to described testing data, namely select ability based on the feature of each sub-classifier and feature selecting is carried out to testing data, to reject the redundancy feature in testing data, be eliminated the testing data x ' of redundancy feature, x ' ∈ R ^{| F|}.

Afterwards, x ' is input in the pattern function f () of described object classifiers, obtains the classification diagnostic result of testing data: y=f (x ').

The present embodiment specifically utilizes 21 test sample books in above-mentioned medical example, carries out Performance Evaluation to constructed object classifiers.Wherein, medical data sample x ∈ R to be diagnosed ⁹⁸⁹; According to aspect indexing set j=1 ... 5, obtain x ', x ' ∈ R after the medical data sample x treating diagnosis carries out feature selecting ^{| F|}; Input afterwards in x ' to f (), obtain the last diagnostic result f (x ') of this medical data sample.

With reference to figure 3, on the basis that the medical data test sample book utilizing the present invention to 21 989 dimensions is classified, the classification situation of classification situation of the present invention and MSVM-RFE algorithm compared, wherein, the classification of MSVM-RFE algorithm is based on identical data set; Get 21 training samples at random and repeat experiment 10 times, average result as shown in Figure 3, can find: the present invention obtains faster than MSVM-RFE algorithm convergence, and when have selected homologous genes number, the present invention has better classification performance.

Table 1 gives the correlation data of the best classification performance that two kinds of methods have:

Table 1

Feature selection approach	MSVM-RFE	The present invention
			Best base is because of number	245	126
Average accuracy (%)	88.57	91.43

Improve 3 percentage points compared to MSVM-RFE method by the known classification accuracy of the present invention of table 1.

Embodiment three

The present embodiment discloses a kind of multi-categorizer constructing system, and this system is corresponding with multi-categorizer construction method disclosed in above each embodiment.

Corresponding to embodiment one, with reference to figure 4, described system comprises processing module 100, fisrt feature selects module 200, merge module 300 and MBM 400, wherein:

Processing module 100, for being treated to l two class data acquisitions by the first training sample set comprising l class sample data; The two class data that described two class data acquisitions comprise are: according to the two class data presetting sorting technique and to re-start described l class sample data gained after two class category division, l be greater than 1 natural number;

Fisrt feature selects module 200, for carrying out feature selecting according to the feature selection approach preset to each described two class data acquisitions, obtains corresponding aspect indexing subset;

Merging module 300, for merging each aspect indexing subset, obtaining aspect indexing set;

MBM 400, for utilizing support vector machines sorter to carry out modeling to the second training sample set, obtains target multi-categorizer; Described second training sample set is the sample set described first training sample set being carried out to gained after feature selecting, and the sample characteristics of the second training sample set is corresponding with the feature that described aspect indexing set comprises.

Corresponding to embodiment two, with reference to figure 5, described system also comprises:

Second feature selects module 500, for utilizing described aspect indexing set to carry out feature selecting to the first test sample book, obtains the second test sample book;

Classification diagnostic module 600, carries out classification diagnosis for utilizing described target multi-categorizer to described second test sample book.

For multi-categorizer constructing system disclosed in the embodiment of the present invention three, because it is corresponding with multi-categorizer construction method disclosed in embodiment one and embodiment two, so description is fairly simple, relevant similarity refers to the explanation of multi-categorizer construction method part in embodiment one and embodiment two, no longer describes in detail herein.

It should be noted that, each embodiment in this instructions all adopts the mode of going forward one by one to describe, and what each embodiment stressed is the difference with other embodiments, between each embodiment identical similar part mutually see.

For convenience of description, various module or unit is divided into describe respectively with function when describing above system.Certainly, the function of each unit can be realized in same or multiple software and/or hardware when implementing the application.

Finally, also it should be noted that, in this article, term " comprises ", " comprising " or its any other variant are intended to contain comprising of nonexcludability, thus make to comprise the process of a series of key element, method, article or equipment and not only comprise those key elements, but also comprise other key elements clearly do not listed, or also comprise by the intrinsic key element of this process, method, article or equipment.When not more restrictions, the key element limited by statement " comprising ... ", and be not precluded within process, method, article or the equipment comprising described key element and also there is other identical element.

The above is only the preferred embodiment of the present invention; it should be pointed out that for those skilled in the art, under the premise without departing from the principles of the invention; can also make some improvements and modifications, these improvements and modifications also should be considered as protection scope of the present invention.

Claims (8)

1. a multi-categorizer construction method, is characterized in that, comprising:

The first training sample set comprising l class sample data is treated to l two class data acquisitions; The two class data that described two class data acquisitions comprise are: according to the two class data presetting sorting technique and to re-start described l class sample data gained after two class category division, l be greater than 1 natural number;

According to the feature selection approach preset, feature selecting is carried out to each described two class data acquisitions, obtain corresponding aspect indexing subset;

Merge each aspect indexing subset, obtain aspect indexing set;

Utilize support vector machines model to carry out modeling to the second training sample set, obtain target multi-categorizer; Described second training sample set is the sample set described first training sample set being carried out to gained after feature selecting, and the sample characteristics of the second training sample set is corresponding with the feature that described aspect indexing set comprises.

2. method according to claim 1, is characterized in that, described default sorting technique is one-to-many OVA method, and described default feature selection approach is that support vector machine-recursive feature eliminates SVM-RFE method.

3. method according to claim 2, is characterized in that, described first training sample set is $X={\{x_i,y_i\}}_{i=1}^N,$ Wherein:

X _ifor sample data, x _i∈ R ^d, R is real number space;

Y _ix _iclass label, y _i∈ 1,2 ..., l}, l are the numbers of classification;

N is total number of training sample;

D is the dimension of sample.

4. method according to claim 3, is characterized in that, described two class data acquisitions are $X_j={\{x_i,v_i\}}_{i=1}^N,v_i=\left\{\begin{array}{cc}+1,& y_i=j\\ -1,& y_i\≠j\end{array}\right.;$ Described aspect indexing subset is $F_j\⊆\{1,...,D\};$ Described aspect indexing set is $F={\∪}_{j=1}^l{F}_j;$ Wherein:

j＝1,…,l。

5. method according to claim 4, is characterized in that, described second training sample set is wherein, x ' _ifor carrying out the sample data after feature selecting, x ' _i∈ R ^{| F|}.

6. the method according to claim 1-5 any one, is characterized in that, also comprises:

Utilize described aspect indexing set to carry out feature selecting to the first test sample book, obtain the second test sample book;

Described target multi-categorizer is utilized to carry out classification diagnosis to described second test sample book.

7. a multi-categorizer constructing system, is characterized in that, comprising:

Processing module, for being treated to l two class data acquisitions by the first training sample set comprising l class sample data; The two class data that described two class data acquisitions comprise are: according to the two class data presetting sorting technique and to re-start described l class sample data gained after two class category division, l be greater than 1 natural number;

Fisrt feature selects module, for carrying out feature selecting according to the feature selection approach preset to each described two class data acquisitions, obtains corresponding aspect indexing subset;

Merging module, for merging each aspect indexing subset, obtaining aspect indexing set;

MBM, for utilizing support vector machines sorter to carry out modeling to the second training sample set, obtains target multi-categorizer; Described second training sample set is the sample set described first training sample set being carried out to gained after feature selecting, and the sample characteristics of the second training sample set is corresponding with the feature that described aspect indexing set comprises.

8. system according to claim 7, is characterized in that, also comprises:

Second feature selects module, for utilizing described aspect indexing set to carry out feature selecting to the first test sample book, obtains the second test sample book;

Classification diagnostic module, carries out classification diagnosis for utilizing described target multi-categorizer to described second test sample book.