CN103500205A - Non-uniform big data classifying method - Google Patents

Abstract

The invention is a classification method for non-uniform big data. Classify categories of datasets that cannot be classified in computer memory and datasets of non-uniform categories. Firstly, the downsampling method is used to determine the sample size according to the theory, and the number of classifiers is determined according to the number of samples. Build an ensemble classifier for each category of big data. When testing an instance, let it classify the integrated classifier of each category, and take the category of the integrated classifier with the highest classification rate as the category of the test instance. The time complexity of this method for big data classification is linear, which can reduce the bias of non-uniform big data classification results, and improve the accuracy of the classifier through the design of the integrated classifier. Moreover, the present invention is easy to implement, and only involves some simple mathematical models when writing codes.

Description

Non-uniform Big Data Classification Method

technical field

The present invention relates to the fields of computer science and technology and information technology, in particular to big data, in particular to a processing method for classification of non-uniform big data.

Background technique

Big data refers to the collection of data that cannot be captured, managed and processed with conventional software tools within the existing physical conditions and allowed time. Big data has the following characteristics: Volume (large amount of data), Variety (variety of data types), Value (low value density), Velocity (fast processing speed), referred to as 4V.

At present, big data research usually includes two categories. First, big data challenges architecture. At present, the raw data capacity in the HADOOP clusters of many famous websites reaches dozens of PB, and there is redundancy, which needs to be scanned and updated every day. Then, in order to ensure that the failure of a single node or a single rack does not affect the operation, HADOOP usually adopts a 3-copy strategy. In this way, the cost of data needs to be considered in both time and space dimensions. Therefore, if an efficient large-scale small file management and large file management mechanism coexist, and support the storage, management and access of structured data, unstructured data and unstructured data at the same time, it must be considered. Second, big data knowledge discovery and the challenges of big data to mining algorithms. The first thing that needs to be covered is the scalability of the algorithm. Some classic data mining and machine learning algorithms, such as KNN density estimation, non-parametric BAYES, support vector machines, Gaussian process regression and hierarchical clustering algorithms, cannot be used in big data mining because their complexity is at least quadratic be better applied. Therefore, it is necessary to design a more efficient algorithm, that is, O(nlogn) or O(n).

Judging from the large amount of existing literature on big data mining, the research on big data learning mainly focuses on the upgrading and improvement of traditional methods in four aspects: division, clustering, retrieval, and incremental (batch, online or parallel) learning. At present, there are relatively few studies on the processing of non-uniform big data problems. Usually similar to other researches on big data knowledge discovery, the first thing to consider in big data classification is the complexity of the algorithm. Secondly, the existing classification algorithm (assuming that the distribution of different categories of data is uniform) directly applied to non-uniform large data tends to lead to bias (bias), that is, the classification results are biased towards large categories (that is, this category contains a large proportion of instances, For example, more than 90% in the second type of problem). Finally, common algorithms for non-uniform (imbalanced) data classification problems usually pursue the problem of minimizing classification errors, but ignore the problem of non-uniform misclassification costs.

However, the classification of non-uniform big data is an extremely challenging problem. There are a series of basic problems that need to be solved urgently, such as where to start, how to use big data for intelligent activities, and so on.

Contents of the invention

The invention studies the classification problem of non-uniform big data.

The purpose of the present invention is to provide a simple and effective non-uniform big data classification method. This method can solve the bias problem that is easy to occur in big data classification and the high complexity problem of big data algorithm. That is, this method achieves linear complexity non-uniform big data classification through downsampling (Downsampling) and two-class classification (one-vs-all) big data, and solves the bias problem and Improve classification accuracy and have robustness (robust), that is, noise resistance.

The concrete steps of this method are as follows:

(1) Obtain the number m _i of various types of big data instances, i=1,2,...,M;

决定，其中t_a/2表示置信度的值，可以通过t分布临界值获得，ε表示最大的允许误差。通过这样方法对每个类m_i抽样出D_i。(2) Use the Downsampling method to sample the D _i data set for each class m _i . The data size n _i in each data set is determined by

Decision, where t _a/2 represents the value of the confidence level, which can be obtained through the critical value of the t distribution, and ε represents the maximum allowable error. In this way, D _i is sampled for each class _mi .

(3) For the D _i data sets of each class m _i use the one-vs-all method (that is, all instances of the current class are positive classes, and all instances of other classes are negative classes) to establish D _i classifiers, that is, for Build a classifier per dataset.

(4) Perform ensemble learning on D _i classifiers of each class _mi . According to the ensemble learning theory, an ensemble classifier can be assembled from multiple meta-classifiers according to the ensemble principle. All meta-classifiers should be fast and independent of each other, with each classifier having an error rate no higher than 50%. Common classifiers of this type, such as nearest neighbor algorithm, decision tree method, neural network method or forest tree method (Forest tree), etc., can meet the above requirements. Integration principles generally include bagging, adaboost, selective ensemble, etc. In the present invention, the D _i classifiers obtained for each class m _i use the forward greedy ensemble method (forward greedy ensemble) to carry out ensemble learning on the classifiers.

(5) Test: Classify each instance in each class, and the class with the highest accuracy among the M results is the class of the test instance.

The goal of step (2) is to solve the algorithm complexity problem, that is, use part of the original data instead of all the data to build a classifier by down-sampling. In order to improve the classification accuracy, the strategy of repeating multiple times is adopted, that is, multiple sampling, the sample size of the sampling meets the above regulations, and the sampling frequency is determined by the user.

The specific steps of step (2) Downsampling method of the present invention are as follows:

个），样本数是需要建立元分类器的个数。在对每类生成一个样本数的过程中，首先取得当前类的样本数目。本发明把当前类当成A类，其他类统一称为非A类。接着，分析A类和非A类的数量级别。本发明记#(A),#(～A),#(R),#(T)为A类，非A类，计算机内存和理论要求的样本量的数据，如果(#(A)>>#(R))&&(#(A)>#(T))，则从A类中抽取与非A类差不多的实例；如果(#(～A)>>#(R))&&(#(～A)>#(T))，则从非A类中抽取与A类差不多的实例。A. When sampling each class m _i , the sample size of sampling shall not be less than the provisions in the above table (ie

), the number of samples is the number of meta-classifiers that need to be built. In the process of generating a sample number for each class, first obtain the sample number of the current class. In the present invention, the current class is regarded as class A, and other classes are collectively referred to as non-A class. Next, analyze the quantity level of A category and non-A category. The present invention records #(A), #(～A), #(R), #(T) as class A, non-class A, computer memory and data of the sample size required by theory, if (#(A)>>#(R))&&(#(A)>#(T)), then extract the instance from class A that is similar to non-A class; if (#(～A)>>#(R))&&(#( ～A)>#(T)), the instances similar to class A are extracted from non-A classes.

B. Repeat the above process until each class m _i samples D _i samples. For simplicity, the present invention fixes D _i as n.

C. So far, the entire data set has generated D=M*n samples.

Through step (2), the present invention obtains M*n samples, and each set of data has n classifiers. Step (3) of the present invention establishes a total of n meta-classifiers for n samples of each class m _i ;

Then step (4) of the present invention integrates the obtained n meta-classifiers to obtain an integrated classifier, that is, adopts the forward greedy ensemble method. The steps are as follows:

D. Construct the candidate classifier set CCS={C ₁ ,...,C _M } and the selected classifier set SCS={};

E. For each classifier C _i , select the classifier with the best accuracy, remove it from the CCS and add it to the SCS;

F. Add the classifier C _j in each current CCS to the SCS for verification. If the classification result exceeds the threshold specified by the user, skip to E and move C _j from the CCS to the SCS. Otherwise, skip to step (5), which indicates that the learning of the integrated classifier is completed;

G. Repeat F until CCS is an empty set,

So far, for M classes, the present invention has established M integrated classifiers C _i , i=1,...,M. Each ensemble classifier contains n meta-classifiers.

The above steps ensure that the resulting classifier is small, which makes the testing process relatively simple.

The non-uniform big data classification method implemented through the above steps has the following characteristics: first, due to the use of downsampling method in the classification process, the various instance data are as balanced as possible, which effectively avoids the problem of classification biased towards large categories; second, using Classification by sampling method makes the complexity of the entire classification algorithm linear; third, in order to avoid the reduction of classification accuracy caused by sampling, the present invention improves the classification accuracy by two methods, that is, multiple sampling methods and forward greedy comprehensive classification results method.

The invention uses a sampling method to reduce unbalanced classification, and reduces the complexity of the algorithm; uses sampling multiple times and establishes a meta-classifier for each sampling, and uses an integrated learning method to synthesize all meta-classifiers to improve classification results.

Sampling Big Data: Often it is very difficult to classify within the entirety of Big Data. Even if it is feasible, the complexity is very high. The sampling method makes the operation of classification of large data feasible, and reduces the complexity of classification to linear. This is exactly the expected result of big data mining.

Sample size and number of samples: The sample size of sampling is obtained according to the theory to ensure that the error between the results obtained after sampling and the original results is minimized. Taking multiple samples is beneficial to improve the classification performance;

The one-vs-all classification method has been proven to be a very effective method for dealing with non-uniform datasets. The present invention applies this method to the classification of non-uniform big data, which can solve the problem of non-uniform classification on the one hand, and solve the problem of high complexity of big data classification on the other hand;

Meta-classifiers make classification on large data sets faster, and ensemble learning can effectively improve the performance of meta-classifiers. Moreover, the forward greedy ensemble guarantees to improve the performance of the meta-classifier while reducing the complexity of the classifier, which is a strong guarantee for the linear complexity of big data processing.

Detailed ways

Example 1

It is given that the simulated big data instance contains two million, and the dimension of each instance is 1000 dimensions. The entire data set is divided into two categories, the first category contains 1.99 million instances, and the second category only contains 10,000 instances. This dataset is randomly generated and belongs to the imbalanced big data binary classification problem.

(1) Determine the confidence level of 99% and the maximum allowable error of 1%. So the sample size per dataset per class is 16641. According to the proportion, 10,000 instances are extracted from class A (data set containing 1.99 million instances), plus 10,000 instances of non-class A, each data set contains 20,000 instances. Common PCs can usually easily apply common meta-classifiers to classify datasets containing 20,000 instances.

(2) According to the above method, a total of 10 sub-datasets are generated in this example. Establish 10 classifiers using the nearest neighbor algorithm, and set k to 1 to 10, respectively.

(3) According to the 10 classifiers, the forward greedy ensemble method is used to integrate the 10 meta-classifiers into an integrated classifier.

(4) For a given test instance, use the integrated classifier obtained above to classify. If the classification result exceeds 50%, it is determined that the test instance belongs to class A, otherwise it belongs to non-class A.

Example 2

Given a simulated big data instance containing 20 million, the dimension of each instance is 1000 dimensions. The entire data set is divided into three categories, among which category A contains 12 million instances, category B contains 7.9 million instances, and category C contains 100,000 instances. This dataset is randomly generated and belongs to the imbalanced big data multiclass classification problem.

(1) Determine the confidence level of 95% and the maximum allowable error of 1%. The sample size for each dataset per class is 9604. It is a bit difficult for a general computer to process 300,000 data. Therefore, three classes need to be sampled.

(2) Sample 10 data for class A, and each data set contains 20,000 instances (note: the number of instances only needs to exceed 9604). More specifically, 10,000 samples are randomly drawn from class A, then 5,000 samples are randomly drawn from class B, and 5,000 samples are randomly drawn from class C. At this point, a sub-dataset containing 20,000 samples is obtained. By repeating this sampling 10 times, 10 sub-datasets of class A samples can be obtained. By analogy, 10 sub-datasets are sampled for class B and class C respectively. Ultimately, a total of 30 sub-datasets were generated during this process.

(3) For the 10 data sets of category A, adopt the one-vs-all classification method, that is, category A is one category, category B and category C are unified into one category, and 10 meta-classifiers are used to establish 10 classifiers. There are 9 meta-classifiers for the nearest neighbor algorithm, k ranges from 1 to 9, and one decision tree C5.0 classifier.

(4) The same method establishes 10 meta-classifiers for class B and 10 meta-classifiers for class C.

(5) Use the forward greedy ensemble method to integrate the 10 meta-classifiers of class A into an integrated classifier. The 10 meta-classifiers of class B and class C are also integrated, and an integrated classifier is collected for each.

(6) For a given test instance, use the three integrated classifiers obtained above to classify. If the classification result of the A ensemble classifier is 85%, the classification result of the B ensemble classifier is 89%, and the classification result of the C ensemble classifier is 90%, it is determined that the test instance belongs to the C class.

Claims (7)

1. The classification method of heterogeneous big data, comprises the following steps: (1) Obtain the number m _i of various types of big data instances, i=1,2,...,M; (2) Use the down-sampling method to sample the D _i data set for each class m _i ; (3) Build a meta-classifier for each dataset; (4) Carry out ensemble learning on D _i classifiers of each class m _i ; (5) Test: For each instance, classify in each class m _i , and the class with the highest accuracy rate among the obtained M results is the class of the test instance. 2. The method according to claim 1, the data amount n _i of each data set in the step (2) is given by Sure, Among them, t _a/2 represents the value of the confidence degree, which is obtained through the critical value of the t distribution, and e represents the maximum allowable error set. 3. according to the method for claim 1 or 2, the specific process of described step (2) is as follows: A. The current class is regarded as class A, and other classes are collectively called non-A class; then, analyze the quantitative level of class A and non-A class; record #(A),#(～A),#(R),#(T ) are the data of Class A, non-Class A, computer memory and theoretically required sample size, if (#(A)>>#(R))&&(#(A)>#(T)), then from A Instances similar to non-A class are extracted from the class; if (#(～A)>>#(R))&&(#(～A)>#(T)), then the non-A class is similar to A class instance of B. Repeat the above process until D _i is sampled for each class m _i , and D _i =n is fixed; C. The entire data set generates D=M*n sub-data sets. 4. The method according to claim 1, in step (3), the method for constructing Di meta-classifiers for D _i data sets of each class _mi is selected from the group consisting of: binary classification, nearest neighbor algorithm, decision tree method , neural network method or forest tree method. 5. The method according to claim 1 or 4, in step (3), the method of constructing Di meta-classifiers for D _i data sets of each class m _i is selected: two-class classification. 6. The method according to claim 1, in step (4), the Di meta-classifiers of each class mi are ensemble-learned by adopting the forward greedy ensemble classification result method to obtain an ensemble classifier. 7. According to the method of claim 1 or 6, in step (4), the specific process of adopting the method of forward greedy integration of classification results is as follows: D. Build candidate classifier set CCS={C ₁ ,...,C _M } and selected classifier set SCS={}; E. For each classifier C _i , select the classifier with the best accuracy, remove it from the CCS and add it to the SCS; F. Add the classifier C _j in each current CCS to the SCS for verification. If the classification result exceeds the threshold specified by the user, then skip to E, and move C _j from the CCS to the SCS, otherwise skip to step (5) ; G. Repeat F until CCS is an empty set, So far, for M classes, a total of M integrated classifiers Ci,i=1,...,M have been established, and each integrated classifier contains n meta-classifiers.