CN107301221A - A kind of data digging method of multiple features dimension heap fusion - Google Patents

Abstract

The invention discloses a data mining method for multi-feature dimension heap fusion. This data mining method first divides the feature system according to the business dimension, then uses different feature subsets to train the model and tunes it, then the output results of the sub-models and the initial features form a new feature set, and finally heaps and fuses the new feature set Train and tune. The multi-feature dimension heap fusion algorithm of the present invention is mainly used to solve the problem of single model instability and easy over-fitting, and enhance model stability and prediction ability by combining different learning models. Due to the limited ability to describe single-dimensional models, the combination of various dimensional models in the present invention can fully understand the characteristics of the entire dimension.

Description

A Data Mining Method of Multi-feature Dimension Heap Fusion

technical field

The invention relates to an integrated learning algorithm, in particular to a data mining method for multi-feature dimension heap fusion, and belongs to the technical field of artificial intelligence.

Background technique

The basic idea of the stack model (Stack) is to train multiple sub-models independently on the original data, and then use the output of the sub-models as a feature input to the combined model for training, and the prediction result of the combined model is the final result. The heap model is mainly composed of two parts. The first part trains the sub-models. The sub-models are independent of each other, and it is best to achieve the difference while ensuring the effect of the sub-models. The second part combines the sub-models to obtain the prediction results. The combined model can be linear can also be non-linear.

The deep stack network model (Deep Stack Network, DSN for short) is mainly used to accelerate the learning and optimization process of the deep neural network. At present, deep learning is valued in various industries, such as pattern recognition, natural language processing, and bioengineering. At the same time, influenced by AlphaGo, an artificial intelligence Go program launched by Google, which defeated the top human players, deep learning and artificial intelligence have been pushed to the forefront. However, the learning process of deep neural network DNN is actually not easy. DNN contains many intermediate layers and neural nodes, and the number of connections before neurons is even less. Generally, the parameter optimization of DNN is pre-trained through unsupervised learning, and then supervised global parameter adjustment according to the label data. Even so, with the increase of network complexity, the calculation time is difficult to control.

Therefore, the deep stacking network DSN has been proposed as a new multi-level network structure. Its main idea is the idea of model stacking. First, the first layer model is trained using the original data, and then the second layer is trained based on the single model output and the original data. Layer model, and then train the third layer model on the first and second layer models and the original data, and so on, until the final output. In this way, the original information can be effectively extracted from low to high levels, and the iterative convergence speed of the network can also be accelerated.

The single hidden layer in each layer of the DSN model does a nonlinear calculation and conversion work. The tanh function is used more often, and of course there are many other conversion functions. Starting from the input vector, the output vector is obtained through the nonlinear calculation of the hidden layer. From input to output, it can also be seen that it is a process of automatic feature extraction, without artificially constructing complex features. For example, in the application of image recognition, the initial input is the pixel value, and after layer-by-layer extraction, it continuously transitions to the process of edge, shape, partial image, and the entire image recognition.

To sum up, DSN continuously learns through a hierarchical structure. The upper-level model combines all the lower-level models and original input information, and then learns more advanced features. The sub-model structure of each layer is the same, and the ability to describe the model is mainly improved by continuously enriching the input information. DSN can control the complexity of the model by controlling the number of stacked layers of the model, so as to balance the model fitting ability and generalization ability.

Contents of the invention

The invention combines the basic heap model and the deep neural network, and proposes a data mining method for multi-feature dimension heap fusion.

The object of the invention is achieved through the following technical solutions:

A data mining method for multi-feature dimension heap fusion, comprising steps:

(1) Divide multiple feature dimensions: classify the feature system according to the business dimension, each feature dimension complements each other, design feature pool featurepool={f _c1 , f _c2 ,..., f _cn }, f _ci represents a single feature dimension;

(2) Sub-model training and tuning: perform sub-model training for a single feature dimension, optimize the model according to the evaluation index cross-validation, and obtain the sub-model output O _ci , there are a total of n sub-models and outputs;

(3) Diversity and standardization processing: In view of the different output formats of the sub-models, the output of the sub-models should be standardized before fusion;

(4) Stacking combination sub-model: the sub-model output O _ci is combined with the original features, and the model is fused by Stacking; the Stacking algorithm is divided into two layers, and the first layer uses different algorithms to form T weak classifiers, and at the same time generates a A new data set with the same size as the original data set, using the new data set and a new algorithm to form a classifier of the second layer; the first layer includes two steps;

The first step is to collect the output information of each basic learner into a new data set; for each data item in the original training set, the new data set represents the prediction and training of each basic learner for the class to which each data item belongs The real classification result of the data set; ensure that there will be no problematic data items in the training data set of the generated basic learner; then, use the new data set as the training data set of the new learner;

The second step is to filter the training set and learner step by step; the Stacking algorithm refers to the initial data set and the basic learner generated from the initial data set as the first-level training data set and the first-level classifier, correspondingly, the first-level classification The data set composed of the output of the classifier and the result of the real classification and the classification learning algorithm of the second stage are called the secondary training data set and the secondary classifier respectively.

To further realize the object of the present invention, preferably, the tuning model includes:

1) Sample tuning: In view of the large difference in the number of samples of the two types, formulate rules to filter some low-contribution samples and use highly reliable label data;

2) Feature tuning: use the tree model for feature selection, and the selection criteria include feature value distribution and correlation, feature information gain, feature call frequency, and the impact of feature knockout;

3) Model and parameter tuning: Train the model through samples, and test the effects of each model, and select a model with good performance for parameter tuning.

The parameter tuning adopts a method of adjusting parameters with fixed variables.

Compared with the existing algorithm, the present invention has the following significant advantages:

(1) The multi-feature dimension heap fusion algorithm can ensure the difference of each sub-model, and also ensure that the sub-models are independent of each other. Each sub-model can be trained in parallel and then combined. In this way, less time is spent in each round of iterative update process, and the model is easy to expand to large-scale data sets.

(2) The multi-feature dimension heap fusion algorithm is an innovative integrated learning method, which can be applied to different data sets and different business scenarios and can achieve good results.

(3) The multi-feature dimension heap fusion algorithm models the features separately according to the business dimension. These sub-models only have a single feature dimension description, and the performance is average. However, the combination of various dimension models can fully describe the entire dimension characteristics, and the prediction performance is better. excellent.

detailed description

In order to better understand the present invention, the present invention will be further described below in conjunction with the examples, but the protection scope of the present invention is not limited thereto.

A data mining method for multi-feature dimension heap fusion, comprising steps:

(1) Divide multiple feature dimensions: classify the feature system according to the business dimension, each feature dimension complements each other, design feature pool featurepool={f _c1 , f _c2 ,..., f _cn }, f _ci represents a single feature dimension.

(2) Sub-model training and tuning. Carry out sub-model training for a single feature dimension, tune the model according to the evaluation index cross-validation, and obtain the sub-model output O _ci , there are n sub-models and outputs in total. The model is tuned from three aspects: samples, features, models and parameters.

Sample tuning. For the classification imbalance problem, that is, there is a large difference in the number of samples between the two types. Formulate rules to filter some low-contribution samples and use high-reliability label data;

Feature tuning. Optimizing features, selection criteria include feature value distribution and correlation, feature information gain, feature call frequency, impact of feature knockout, etc.;

Model and parameter tuning. The model is trained through samples, and the effect of each model is obtained by testing. The model with better performance is selected for parameter tuning. In the case of multiple parameters, the method of adjusting parameters with fixed variables can be used.

(3) Diversity and standardization. The effect of model fusion depends on the performance of a single model and the coincidence of the output of each model. Under the premise of satisfying diversity, models with poor effects can also be fused. Affected by different model output formats, the output of a single model should be standardized before fusion (z-score, ranking-score and other standard score measurement methods). The differences between sub-models are reflected in the following aspects:

The type of basic classifier itself, that is, its composition algorithm is different;

The data is processed differently, including boosting (error rate weighted sampling method), bagging (average sampling method), cross-validation (cross-validation), hold-out test (model verification method), etc.;

Data feature processing and selection;

Output result processing;

Introduce randomness.

(4) Stacking combined sub-models. The sub-model output O _ci is combined with the original feature feature pool, and the model fusion is performed through Stacking. The Stacking algorithm is divided into two layers. The first layer uses different algorithms to form T weak classifiers, and at the same time generates a new data set with the same size as the original data set. This new data set and a new algorithm are used to form the classification of the second layer. device.

In the first step, the output information of each base learner is collected into a new dataset. For each data item in the original training set, the new data set represents each basic learner's prediction of the class to which each data item belongs and the actual classification result of the data in the training set. It should be noted that it must be ensured that there will be no problematic data items in the training data set for generating the basic learner. Then, use the new data set as the training data set for the new learner.

The second step is to filter the training set and learner step by step. The Stacking algorithm refers to the initial data set and the basic learner generated by the initial data set as the first-level training data set and the first-level classifier, correspondingly, the data set composed of the output of the first-level classifier and the result of the real classification and The classification learning algorithm in the second stage is called the secondary training data set and the secondary classifier, respectively.

Example: 2015 Ali Mobile Recommendation and Travel Prediction of Guangdong Traffic Citizens

The algorithm uses the classic precision (precision), recall (recall) and F1 value as evaluation criteria, and the final scores are sorted according to the F1 value. The specific calculation formula is as follows:

In mobile recommendation analysis, PredictionSet is a collection of predicted purchase data, and ReferenceSet is a collection of real purchase data. In travel prediction, PredictionSet is the predicted ride data set, and ReferenceSet is the real ride data set.

The number of online evaluations is once a day. In order to ensure the robustness of the model, an offline evaluation index is designed. A good offline evaluation should be able to simulate the online evaluation environment as much as possible, so that the offline evaluation results are consistent with the online evaluation results, reducing the dependence on online evaluation.

The following is a comparison of offline evaluation scores and online evaluation scores on the two data sets of travel prediction and mobile recommendation:

Table 1 Offline and online evaluation results of travel prediction

Table 2 Mobile recommendation offline and online evaluation results

Comparing the online and offline data sets of travel forecasting and mobile recommendation forecasting, it is found that offline scores and online scores are basically positively correlated, that is, if the scores in the offline evaluation improve, the scores corresponding to the online evaluation will also increase, but only slightly It's just a change. At the same time, it also shows that the performance of the model is stable on different data sets, there will be no overfitting, and the generalization ability is good.

The multi-feature dimension heap fusion algorithm (DFSE) is used as a sub-model integration algorithm. The F1 value (see Table 3) and the change curve in travel prediction and mobile recommendation prediction are as follows.

Table 3 Comparison of multi-feature dimension heap fusion algorithms

The Base Model (basic model) is constructed based on the time preference feature system and sliding window samples. The multi-feature dimension heap fusion algorithm (DFSE) summarizes the characteristics of each feature dimension model to maximize strengths and avoid weaknesses. The difference between models is the key to fusion and improvement of F1 performance.

Claims (3)

1. A data mining method for multi-feature dimension heap fusion, characterized in that it comprises steps: (1) Divide multiple feature dimensions: classify the feature system according to the business dimension, each feature dimension complements each other, design feature pool featurepool={f _c1 , f _c2 ,..., f _cn }, f _ci represents a single feature dimension; (2) Sub-model training and tuning: perform sub-model training for a single feature dimension, optimize the model according to the evaluation index cross-validation, and obtain the sub-model output O _ci , there are a total of n sub-models and outputs; (3) Diversity and standardization processing: In view of the different output formats of the sub-models, the output of the sub-models should be standardized before fusion; (4) Stacking combination sub-model: the sub-model output O _ci is combined with the original features, and the model is fused by Stacking; the Stacking algorithm is divided into two layers, and the first layer uses different algorithms to form T weak classifiers, and at the same time generates a A new data set with the same size as the original data set, using the new data set and a new algorithm to form a classifier of the second layer; the first layer includes two steps; The first step is to collect the output information of each basic learner into a new data set; for each data item in the original training set, the new data set represents the prediction and training of each basic learner for the class to which each data item belongs The real classification result of the data set; ensure that there will be no problematic data items in the training data set of the generated basic learner; then, use the new data set as the training data set of the new learner; The second step is to filter the training set and learner step by step; the Stacking algorithm refers to the initial data set and the basic learner generated from the initial data set as the first-level training data set and the first-level classifier, correspondingly, the first-level classification The data set composed of the output of the classifier and the result of the real classification and the classification learning algorithm of the second stage are called the secondary training data set and the secondary classifier respectively. 2. the data mining method of multi-feature dimension heap fusion according to claim 1, is characterized in that, described tuning model comprises: 1) Sample tuning: In view of the large difference in the number of samples of the two types, formulate rules to filter some low-contribution samples and use highly reliable label data; 2) Feature tuning: use the tree model for feature selection, and the selection criteria include feature value distribution and correlation, feature information gain, feature call frequency, and the impact of feature knockout; 3) Model and parameter tuning: Train the model through samples, and test the effects of each model, and select a model with good performance for parameter tuning. 3. The data mining method of multi-feature dimension heap fusion according to claim 2, characterized in that, said parameter tuning adopts the method of adjusting parameters with fixed variables.