CN109033148A - One kind is towards polytypic unbalanced data preprocess method, device and equipment - Google Patents

Abstract

The invention discloses a method, device and equipment for unbalanced data preprocessing oriented to multi-classification. The method includes: receiving the size of the final sample set and the unbalanced ratio of the sample set, and obtaining the ideal number of samples for each category; according to The number of ideal samples and the actual number of samples determine the minority class sample set and the majority class sample set; for the samples in the minority class sample set, calculate the number of other class samples and minority class samples in the k nearest neighbors , to classify and mark; for the samples in the minority class sample set, delete, save, copy or synthesize according to the mark of the sample to obtain the final minority class sample set; for the samples in the majority class sample set, calculate the k-nearest neighbor The number of the majority class sample and other class samples in the class is marked by classification; the samples in the majority class sample set are deleted or saved according to the mark of the sample, and the final majority class sample set is obtained; the final sample is generated set. The invention enables the final sample set to effectively improve the accuracy of the multi-classification algorithm.

Description

A multi-classification-oriented unbalanced data preprocessing method, device and equipment

technical field

The present invention relates to the field of big data processing, in particular to a multi-category-oriented unbalanced data preprocessing method, device and equipment.

Background technique

With the continuous advancement of technology, including the improvement of Internet speed, the upgrading of mobile Internet, the continuous development of hardware technology, the rapid development of data acquisition technology, storage technology and processing technology, data is growing at an unprecedented rate, and we have entered the big data era. The characteristics of big data such as huge data volume (volume), high speed (velocity), variety (variety), and data uncertainty (veracity) make traditional data analysis and mining technologies encounter unprecedented challenges when applied to the field of big data. .

Data classification is a basic algorithm in data analysis and mining, which has a wide range of applications and is also the basis of many other data analysis and mining algorithms. In big data, almost all data sets are unbalanced data. Imbalanced data means that at least one category in the data set contains fewer samples than other categories. The problem of data imbalance exists widely in the real world, especially in the field of big data applications. For example, in Internet text classification, the data of each category is unbalanced, and we often focus on small categories of data, such as sensitive information on the Internet, emerging topics, etc.; in e-commerce applications, a large number of users Both transaction data and behavior data are normal, but we often focus on fraudulent and abnormal behaviors in e-commerce. These data are submerged in a large amount of normal behavior data and are severely skewed and unbalanced data sets. Similar applications include medical diagnosis, classification of satellite remote sensing data, etc. Therefore, the classification of unbalanced big data is a key technical problem that urgently needs to be solved in national economic and social development, and has broad application prospects.

Due to the large difference in the number of different types of data samples in unbalanced big data, it is difficult for traditional classification learning algorithms to achieve good classification results. The example of unbalanced data classification is shown in Figure 1, where the circles are minority class samples. The triangle is the majority class sample, and the imbalance ratio is 3:1, that is, the majority class sample is 3 times that of the minority class sample, and in the actual large data set, the imbalance ratio is often 10000:1 or even higher. Therefore, it is necessary to preprocess the data before classification to obtain a good learning effect.

The existing unbalanced big data preprocessing methods are mainly aimed at the binary classification algorithm, that is, there are only two categories in the data set, the majority category and the minority category. In the preprocessing, the majority category is under-sampled and the minority category is over-sampled. Or carry out both at the same time to reduce the imbalance ratio of the data, thereby improving the classification effect. There is currently a lack of relevant research on the unbalanced big data preprocessing of multi-classification algorithms. Multi-classification algorithms mean that there are multiple categories in the data set, and the classification algorithm needs to learn through training to divide the data into one of the multiple categories. The current method is to simplify the multi-classification problem into a binary classification problem for processing, that is, divide multiple categories in the data set into multiple binary classification data sets, and process them two by two.

Transforming multi-classification problems into multiple binary classification problems faces the following problems:

1. The data set of a certain category is a minority class in a binary classification problem, and may be a majority class in another binary classification problem. This method cannot effectively deal with it. As shown in Figure 2, the circle sample set belongs to the minority class in the classification with the triangle sample set, and belongs to the majority class in the classification with the fork sample set.

2. A certain sample may be of different sample categories in different binary classification problems. For example, it is noise in one binary classification and needs to be deleted. It is an important boundary sample in another binary classification and needs to be retained. Use Existing methods cannot handle it effectively. As shown in Figure 2, the triangle sample in the circle is noise in the binary classification problem with the circle sample and needs to be deleted; it is an important boundary sample in the binary classification problem with the fork sample and needs to be retained.

In short, if the multi-classification problem is considered as multiple binary classification problems, the processing of samples cannot comprehensively consider the different situations in each category, and the accuracy of the multi-classification algorithm cannot be effectively improved.

Contents of the invention

In view of the above problems, the purpose of the present invention is to provide a multi-classification-oriented unbalanced data preprocessing method, device and equipment, which can adapt to the needs of different multi-classification algorithms and effectively improve the accuracy of multi-classification algorithms.

The embodiment of the present invention provides a multi-category-oriented unbalanced data preprocessing method, including the following steps:

Read the original sample set; wherein, the original sample set includes at least two categories of sample sets;

Receive the final sample set size input by the user and the imbalance ratio between each sample set to calculate the ideal number of samples for each sample set in the final sample set;

Judging whether the sample set belongs to the minority class sample set or the majority class sample set according to the ideal sample number and the actual sample number of each sample set;

For each sample in the minority class sample set, calculate the number of other class samples in the k-nearest neighbors of each sample and the number of samples belonging to the minority class sample set, so as to divide each sample into a noise sample Examples, unstable samples, boundary samples or stable samples and marked accordingly; among them, other class samples refer to samples in other sample sets except the samples in the minority class sample set;

For the examples in each minority class sample set, delete, save, copy or synthesize according to the mark of each sample, so as to obtain the final minority class sample set corresponding to each minority class sample set;

For each sample in the majority class sample set, calculate the number of samples belonging to the majority class sample set and other class samples in the k-nearest neighbors of each sample, so as to divide each sample into a noise sample sample, boundary sample or stable sample, and mark accordingly;

For each sample in the majority class sample set, delete or save according to the mark of each sample, so as to obtain the final majority class sample set corresponding to each majority class sample set;

A final sample set is generated according to the final minority class sample set and the majority class sample set, so as to realize preprocessing of unbalanced data.

Preferably, the judging whether the sample set belongs to the minority sample set or the majority sample set according to the ideal sample number and the actual sample number of each sample set is as follows:

For each sample set, if the number of ideal samples is greater than the actual number of samples, it is judged that the sample set is a minority class sample set; if the number of ideal samples is less than or equal to the actual number of samples, then It is judged that the sample set is a majority class sample set.

Preferably, for each sample in the minority class sample set, calculate the number of other class samples in the k-nearest neighbors of each sample and the number of samples belonging to the minority class sample set, so that each sample Examples are divided into noise samples, unstable samples, boundary samples or stable samples and marked accordingly, including:

When it is judged that most of the k-nearest neighbor samples of the samples in the minority class sample set are other class samples, the sample is marked as a noise sample;

When it is judged that most of the k-nearest neighbor samples of the samples in the minority class sample set are samples of other classes, the sample is marked as an unstable sample;

When it is judged that the number of other class samples of the k-nearest neighbor samples of the sample in the minority class sample set is close to the number of samples in the minority class sample set, then mark the sample as a boundary sample;

When it is judged that most of the k-nearest neighbor samples of the sample in the minority class sample set belong to the samples in the minority class sample set, mark the sample as a stable sample.

Preferably, the samples in each minority class sample set are deleted, saved, copied or synthesized according to the label of each sample to obtain the final minority class sample set, specifically:

For each example in the minority class example set:

Delete all noise samples in the minority sample set;

Add all unstable samples to the corresponding final minority sample set;

Copy each boundary sample, the number of copies is |c-1|, add the boundary sample and the copied sample to the corresponding final minority sample set; where c is the replication ratio, and c= (the ideal number of samples of the minority class sample set - the number of unstable samples)/(the actual number of samples of the minority class sample set - the number of noise samples - the number of unstable samples);

For each stable sample, synthesize a new sample with the surrounding samples, the number of synthesis is |c-1|, and add this sample and the newly synthesized sample to the corresponding final minority sample set; where, The synthesis method is to randomly select a sample xj belonging to the minority class sample set from the k-nearest neighbors of the stable sample x _i each time, and the newly synthesized sample x _i '= _xi +( _xi -x _j ) *a, a is a random number between 0 and 1;

Calculate the number d of samples belonging to the minority class sample set that still needs to be generated; where, d=the ideal sample number of the minority class sample set-the final minority class sample set that belongs to the minority class sample set The current number of samples of ;

Randomly select d stable samples, synthesize a new sample from each stable sample and surrounding samples, and add the synthesized new samples to the corresponding final minority class sample set;

A final minority class sample set corresponding to each minority class sample set is obtained.

Preferably, for each sample in the majority class sample set, calculate the number of samples and other class samples belonging to the majority class sample set in the k-nearest neighbors of each sample, so that each sample The samples are divided into noise samples, boundary samples or stable samples, and marked accordingly, specifically:

When it is judged that most of the k-nearest neighbor samples of the samples in the majority class sample set are other class samples, the sample is marked as a noise sample;

When it is judged that the number of other class samples of the k-nearest neighbor samples of the sample in the majority class sample set is close to the number of samples belonging to the majority class sample set, then mark the sample as a boundary sample;

When it is judged that most of the k-nearest neighbor samples of the samples in the majority class sample set belong to the samples in the majority class sample set, mark the sample as a stable sample.

Preferably, the samples in each majority class sample set are deleted or saved according to the mark of each sample, so as to obtain the final majority class sample set, which specifically includes:

For each example in the majority class example set:

remove noise samples;

Keep all boundary samples;

For each stable sample, perform a selective deletion operation until e stable samples are deleted; where, e=the actual number of samples in the majority class sample set - the number of noise samples - the majority class sample set The ideal number of samples;

A final majority class sample set corresponding to each majority class sample set is obtained.

Preferably, for each stable sample, performing a selective deletion operation until e stable samples are deleted is specifically:

Repeat the following steps until the number f of deleted stable samples is equal to e;

For the currently selected stable sample, calculate the distance from the stable sample to the surrounding k nearest neighbor samples belonging to the majority class sample set;

Calculate the probability of deleting the stable sample according to the distance; wherein, the smaller the distance, the greater the deletion probability;

If the deletion probability is greater than or equal to 0.5, delete the stable sample and update the number f of the deleted stable samples;

Select the next stable sample.

The embodiment of the present invention also provides a multi-classification-oriented unbalanced data preprocessing device, including:

A data reading unit, configured to read an original sample set; wherein, the original sample set includes at least two categories of sample sets;

The data receiving unit is used to receive the size of the final sample set input by the user and the imbalance ratio between each sample set, so as to calculate the ideal number of samples of each sample set in the final sample set;

A judging unit, configured to judge whether the sample set belongs to the minority sample set or the majority sample set according to the ideal sample number and the actual sample number of each sample set;

The minority class sample classification unit is used to calculate the number of other class samples in the k nearest neighbors of each sample and the number of samples belonging to the minority class sample set for each sample in the minority class sample set, so as to Divide each sample into noise sample, unstable sample, boundary sample or stable sample and mark accordingly; among them, other class samples refer to the samples other than the samples in the minority class sample set. samples in the sample set;

The minority class sample processing unit is used to delete, save, copy or synthesize the samples in each minority class sample set according to the mark of each sample, so as to obtain the final result corresponding to each minority class sample set Minority class sample set;

The majority class sample classification unit is used to calculate the number of samples and other class samples belonging to the majority class sample set in the k nearest neighbors of each sample for the samples in each majority class sample set, so as to Divide each sample into noise sample, boundary sample or stable sample, and mark accordingly;

The majority class sample processing unit is used to delete or save the samples in each majority class sample set according to the mark of each sample, so as to obtain the final majority class sample corresponding to each majority class sample set set;

The final sample set generating unit is configured to generate a final sample set according to the final minority class sample set and the majority class sample set, so as to realize preprocessing of unbalanced data.

The embodiment of the present invention also provides a multi-classification-oriented unbalanced data preprocessing device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, the processing When the computer executes the computer program, the above-mentioned multi-classification-oriented imbalanced data preprocessing method is realized.

The embodiment of the present invention implements a multi-classification-oriented unbalanced data preprocessing method, which uses oversampling and undersampling techniques in combination, so that the newly generated final sample set meets the requirements of the classification algorithm, and effectively improves the classification accuracy of unbalanced data. Specifically, the embodiment of the present invention allows the user to input the total number of samples required and the unbalanced ratio of multiple sample sets that are expected to be obtained, and the ideal number of samples for each sample set is obtained through calculation. According to the final sample set The number of samples determines whether each sample set is a majority class or a minority class, which solves the problem that a sample set may be a majority class and a minority class at the same time in different binary classification algorithms in traditional methods. When processing the samples in each sample set, all other types of sample sets are processed together, and the minority class samples are divided into noise samples, unstable samples, boundary samples and stable samples. , the majority class samples are divided into noise samples, boundary samples and stable samples to be processed separately, which solves the problem that a sample may belong to different categories in different binary classification algorithms in the traditional method, which leads to conflicts in the processing of samples . Finally, in the embodiment of the present invention, the final sample set can effectively improve the accuracy of the multi-classification algorithm by performing the most appropriate processing on each sample.

Description of drawings

In order to illustrate the technical solution of the present invention more clearly, the accompanying drawings used in the implementation will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some implementations of the present invention. As far as the skilled person is concerned, other drawings can also be obtained based on these drawings on the premise of not paying creative work.

Figure 1 is an example diagram of unbalanced data classification;

Figure 2 is another example of unbalanced data classification;

FIG. 3 is a schematic flowchart of a multi-classification-oriented unbalanced data preprocessing method provided by the first embodiment of the present invention;

Fig. 4 is a schematic diagram of the result of the multi-classification-oriented unbalanced data preprocessing device provided by the second embodiment of the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Please refer to FIG. 3 , the first embodiment of the present invention provides a multi-classification-oriented unbalanced data preprocessing method, which can be performed by a multi-classification-oriented imbalanced data preprocessing device (hereinafter referred to as the device), and at least includes the following step:

S101. Read an original sample set; wherein, the original sample set includes sample sets of at least two categories.

In this embodiment, the device first reads an original sample set, where the original sample set includes sample sets of at least two categories.

S102. Receive the size of the final sample set input by the user and the imbalance ratio between the various sample sets to calculate the ideal number of samples for each sample set in the final sample set.

In this embodiment, the device is a device with data processing capability, for example, the device may be a personal computer, notebook, tablet, server or server cluster, etc., which is not specifically limited in the present invention.

In this embodiment, the device first receives the desired final sample set size x input by the user and the ideal imbalance ratio a1:a2:...:an between each sample set (assuming that there are n sample sets ). Then calculate the ideal number of samples in the sample set of each category in the final sample set as

For example, suppose the ideal final sample set input by the user contains 20,000 samples, among which there are 4 sample sets, and the imbalance ratio between each sample set is 4:3:2:1, then each The ideal number of samples in the sample set are:

x1 _＝ 20000*4/(4+3+2+1)＝8000;

_x2 ＝20000*3/(4+3+2+1)＝6000;

x ₃ =20000*2/(4+3+2+1)=4000;

x ₄ =20000*1/(4+3+2+1)=2000.

S103. Determine whether the sample set belongs to the minority class sample set or the majority class sample set according to the ideal sample number and the actual sample number of each sample set.

In this embodiment, for each sample set x _i , the device will judge whether the sample set is a minority class sample set or a majority class sample set, and the judgment method is as follows:

For each sample set, if the number of ideal samples is greater than the number of actual samples (that is, the number in the original sample set), it is judged that the sample set is a minority class sample set; if the number of ideal samples is If the number is less than or equal to the actual number of samples, it is judged that the sample set is a majority class sample set.

For example, assuming that the sample set x ₁ has an ideal sample number of 8000 and an actual sample number of 12000, it means that the sample set x ₁ is a majority class sample set.

For another example, for the sample set x ₄ , the ideal number of samples is 2000, but the actual number of samples is 200, which means that the sample set x ₄ is a minority class sample set.

S104, for each sample in the minority class sample set, calculate the number of other class samples in the k-nearest neighbors of each sample and the number of samples belonging to the minority class sample set, so as to divide each sample into Noise samples, unstable samples, boundary samples or stable samples and marked accordingly; where, other class samples refer to samples in other sample sets except the samples in the minority class sample set.

In this embodiment, when processing each sample set, all sample sets of other categories are merged into one sample set, which is called a sample set of other classes, and the samples in it are called samples of other classes. For example, in processing the x ₁ sample set, the combination of x ₂ , x ₃ , and x ₄ is called the other class sample set. In this example:

When it is judged that most of the k-nearest neighbor samples of the samples in the minority class sample set are samples of other classes, the sample is marked as a noise sample.

In this embodiment, the value of k is input by the user, and different classification algorithms will have the best effect under different k values. The k-neighboring samples refer to: assuming that the position of a certain sample is x _i , select k neighbor nodes (neighbor samples) closest to x _i .

For example, when it is judged that in the k-nearest neighbor samples of a sample in the minority class sample set, the ratio of the number of other class samples to k is greater than the first threshold (such as 80%), then mark the sample as noise sample. Of course, the first threshold can be set according to actual needs, for example, it can be set to 75%, 85%, etc., which is not specifically set in the present invention.

When it is judged that most of the k-nearest neighbor samples of the samples in the minority class sample set are samples of other classes, the sample is marked as an unstable sample.

For example, when it is judged that in the k-nearest neighbor samples of a certain sample in the minority class sample set, the ratio of the number of other class samples to k is less than the first threshold (such as 80%) and greater than the second threshold (such as 60% ), the sample is marked as an unstable sample. Of course, the second threshold can be set according to actual needs, for example, it can be set to 55%, 65%, etc., which is not specifically set in the present invention.

When it is judged that the number of other class samples of the k-nearest neighbor samples of the sample in the minority class sample set is close to the number of samples in the minority class sample set, mark the sample as a boundary sample.

For example, when it is judged that in the k-nearest neighbor samples of a sample in the minority class sample set, the ratio of the number of other class samples to k is less than the second threshold (such as 60%) and greater than the third threshold (such as 40% ), mark the sample as a boundary sample.

When it is judged that most of the k-nearest neighbor samples of the sample in the minority class sample set belong to the samples in the minority class sample set, mark the sample as a stable sample.

For example, when it is judged that in the k-nearest neighbor samples of a sample in the minority class sample set, the ratio of the number of other class samples to k is less than the third threshold (such as 40%), mark the sample as a stable sample example.

S105. Delete, save, copy or synthesize the samples in each minority class sample set according to the label of each sample, so as to obtain a final minority class sample set corresponding to each minority class sample set.

In this embodiment, the samples in each minority class sample set are processed as follows according to their labels:

1. Delete all noise samples in the minority sample set;

2. Add all unstable samples to the final minority sample set corresponding to the minority sample set;

3. For each boundary sample, copy it, the number of copies is |c-1|, add the boundary sample and the copied sample together to the corresponding final minority class sample set; where c is the copy ratio, and c=(ideal sample number of the minority class sample set-unstable sample number)/(actual sample number of the minority class sample set-noise sample number-unstable sample number of instances);

Among them, the boundary samples are very important to the classification learning algorithm, and if they are synthesized with other samples to generate new samples, it is easy to cause sample deviation, so the copy operation is taken for the boundary samples.

For each stable sample, synthesize a new sample with the surrounding samples, the number of synthesis is |c-1|, and add this sample and the newly synthesized sample to the corresponding final minority class sample set; where, The synthesis method is to randomly select a sample xj belonging to the minority class sample set from the k-nearest neighbors of the stable sample x _i each time, and the newly synthesized sample x _i '= _xi +( _xi -x _j ) *a, a is a random number between 0 and 1.

Calculate the number d of samples belonging to the minority class sample set that still needs to be generated; where, d=the ideal sample number of the minority class sample set-the final minority class sample set that belongs to the minority class sample set The current number of samples of ;

Randomly select d stable samples, synthesize a new sample with each stable sample and surrounding samples, and add the synthesized new sample to the corresponding final minority class sample set.

Through the above processing, the samples of different classifications are processed separately, which improves the quality of the newly generated samples, thereby improving the performance of the classification learning algorithm, and ensuring that the number of samples in the final minority class sample set is equal to that of the user The preset ideal number of samples.

In this embodiment, the above processing is performed sequentially on each minority class sample set, that is, the corresponding final minority class sample set is obtained.

S106, for each sample in the majority class sample set, calculate the number of samples belonging to the majority class sample set and other class samples in the k nearest neighbors of each sample, so as to divide each sample into Noise samples, boundary samples or stable samples, and mark accordingly.

Specifically, when it is judged that most of the k-nearest neighbor samples of the samples in the majority class sample set are other class samples, the sample is marked as a noise sample;

When it is judged that the number of other class samples of the k-nearest neighbor samples of the sample in the majority class sample set is close to the number of samples belonging to the majority class sample set, then mark the sample as a boundary sample;

When it is judged that most of the k-nearest neighbor samples of the samples in the majority class sample set belong to the samples in the majority class sample set, mark the sample as a stable sample.

The proportions of the above-mentioned overwhelming majority, majority, and close to corresponding ratios can refer to the situation of the minority class samples, and the present invention will not repeat them here.

S107. For each sample in the majority class sample set, delete or save according to the label of each sample, so as to obtain a final majority class sample set.

Specifically, for each sample in the majority class sample set:

remove noise samples;

Keep all boundary samples;

The boundary samples are very important to the classification learning algorithm, so the boundary samples are retained, that is, no boundary samples are deleted.

For each stable sample, perform a selective deletion operation until e stable samples are deleted; where, e=the actual number of samples in the majority class sample set - the number of noise samples - the majority class sample set The ideal number of samples for .

In this embodiment, a selective deletion operation is adopted for stable samples, so as to ensure that the number of samples in the final majority class sample set is the ideal number of samples.

In one implementation, stable samples can be selectively deleted in the following manner:

Repeat the following steps until the number f of deleted stable samples is equal to e;

For the currently selected stable sample, calculate the distance from the stable sample to the surrounding k nearest neighbor samples belonging to the majority class sample set;

In this embodiment, the calculation method of the distance is different according to different classification objects. For example, if the classification object is a word vector, the distance can be calculated by using the vector Euclidean.

Calculate the probability of deleting the stable sample according to the distance; wherein, the smaller the distance, the greater the deletion probability;

If the deletion probability is greater than or equal to 0.5, then delete the stable sample, and update the number f of the deleted stable sample (that is, let f=f+1);

Select the next stable sample.

In the above embodiment, during the selective deletion process, samples with denser distribution are more likely to be deleted, so that the remaining samples retain the characteristics of all samples as much as possible. In this way, the separate processing of different samples improves the quality of the samples after undersampling, thereby improving the performance of the classification learning algorithm.

In this embodiment, the above processing is performed sequentially on each majority class sample set, that is, the corresponding final majority class sample set is obtained.

S108. Generate a final sample set according to the final minority class sample set and the majority class sample set, so as to realize preprocessing of unbalanced data.

The embodiment of the present invention implements a multi-classification-oriented unbalanced data preprocessing method, which uses oversampling and undersampling techniques in combination, so that the newly generated final sample set meets the requirements of the classification algorithm, and effectively improves the classification accuracy of unbalanced data. Specifically, the embodiment of the present invention allows the user to input the total number of samples required and the unbalanced ratio of multiple sample sets that are expected to be obtained, and the ideal number of samples for each sample set is obtained through calculation. According to the final sample set The number of samples determines whether each sample set is a majority class or a minority class, which solves the problem that a sample set may be a majority class and a minority class at the same time in different binary classification algorithms in traditional methods. When processing the samples in each sample set, all other types of sample sets are processed together, and the minority class samples are divided into noise samples, unstable samples, boundary samples and stable samples. , the majority class samples are divided into noise samples, boundary samples and stable samples to be processed separately, which solves the problem that a sample may belong to different categories in different binary classification algorithms in the traditional method, which leads to conflicts in the processing of samples . To sum up, the embodiment of the present invention makes the final sample set effectively improve the accuracy of the multi-classification algorithm by performing the most appropriate processing on each sample.

Referring to FIG. 4, the second embodiment of the present invention provides a preprocessing device for unbalanced data, including:

A data reading unit 10, configured to read an original sample set, wherein the original sample set includes sample sets of at least two categories;

The data receiving unit 20 is used to receive the size of the final sample set input by the user and the imbalance ratio between each sample set, so as to calculate the ideal number of samples for each sample set in the final sample set;

Judging unit 30, for judging whether the sample set belongs to the minority class sample set or the majority class sample set according to the ideal sample number and the actual sample number in each sample set;

Minority class example classifying unit 40, for the sample in each minority class sample set, calculate the number of other class samples in the k nearest neighbors of each sample and the number of samples belonging to the minority class sample set, In order to divide each sample into noise sample, unstable sample, boundary sample or stable sample and mark accordingly; among them, other class samples refer to samples other than the samples in the minority class sample set samples from other sample sets;

The minority class sample processing unit 50 is used to delete, save, copy or synthesize the samples in each minority class sample set according to the mark of each sample, so as to obtain the corresponding Final minority sample set;

The majority class sample classification unit 60 is used for calculating the number of samples and other class samples belonging to the majority class sample set in the k-nearest neighbors of each sample for the samples in each majority class sample set, To classify each sample as noise sample, boundary sample or stable sample, and mark accordingly;

The majority class sample processing unit 70 is used to delete or save the samples in each majority class sample set according to the mark of each sample, so as to obtain the final majority class sample set corresponding to each majority class sample set example set;

The final sample set generating unit 80 is configured to generate a final sample set according to the final minority class sample set and the majority class sample set, so as to realize preprocessing of unbalanced data.

Preferably, the specific threshold of the judgment unit 30 is used for:

For each sample set, if the number of ideal samples is greater than the actual number of samples, it is judged that the sample set is a minority class sample set; if the number of ideal samples is less than or equal to the actual number of samples, then It is judged that the sample set is a majority class sample set.

Preferably, the minority class example classification unit 40 is specifically used for:

When it is judged that most of the k-nearest neighbor samples of the samples in the minority class sample set are other class samples, the sample is marked as a noise sample;

When it is judged that most of the k-nearest neighbor samples of the samples in the minority class sample set are samples of other classes, the sample is marked as an unstable sample;

When it is judged that the number of other class samples of the k-nearest neighbor samples of the sample in the minority class sample set is close to the number of samples in the minority class sample set, then mark the sample as a boundary sample;

When it is judged that most of the k-nearest neighbor samples of the sample in the minority class sample set belong to the samples in the minority class sample set, mark the sample as a stable sample.

Preferably, the minority class example processing unit 50 is specifically configured to:

For each example in the minority class example set:

Delete all noise samples in the minority sample set;

Add all unstable samples to the corresponding final minority sample set;

Copy each boundary sample, the number of copies is |c-1|, add the boundary sample and the copied sample to the corresponding final minority sample set; where c is the replication ratio, and c= (the ideal number of samples of the minority class sample set - the number of unstable samples)/(the actual number of samples of the minority class sample set - the number of noise samples - the number of unstable samples);

For each stable sample, synthesize a new sample with the surrounding samples, the number of synthesis is |c-1|, and add this sample and the newly synthesized sample to the corresponding final minority sample set; where, The synthesis method is to randomly select a sample x _j belonging to the minority class sample set from the k-nearest neighbors of the stable sample x _i each time, and the newly synthesized sample x _i '= _xi +( _xi -x _j )*a, a is a random number between 0 and 1;

Calculate the number d of samples belonging to the minority class sample set that still needs to be generated; where, d=the ideal sample number of the minority class sample set-the final minority class sample set that belongs to the minority class sample set The current number of samples of ;

Randomly select d stable samples, synthesize a new sample from each stable sample and surrounding samples, and add the synthesized new samples to the corresponding final minority class sample set;

A final minority class sample set corresponding to each minority class sample set is obtained.

Preferably, the majority class example classification unit 60 is specifically used for:

When it is judged that most of the k-nearest neighbor samples of the samples in the majority class sample set are other class samples, the sample is marked as a noise sample;

When it is judged that the number of other class samples of the k-nearest neighbor samples of the sample in the majority class sample set is close to the number of samples belonging to the majority class sample set, then mark the sample as a boundary sample;

When it is judged that most of the k-nearest neighbor samples of the samples in the majority class sample set belong to the samples in the majority class sample set, mark the sample as a stable sample.

Preferably, the majority class example processing unit 70 is specifically used for:

For each example in the majority class example set:

remove noise samples;

Keep all boundary samples;

For each stable sample, perform a selective deletion operation until e stable samples are deleted; where, e=the actual number of samples in the majority class sample set - the number of noise samples - the majority class sample set The ideal number of samples;

A final majority class sample set corresponding to each majority class sample set is obtained.

Preferably, for each stable sample, performing a selective deletion operation until e stable samples are deleted is specifically:

Repeat the following steps until the number f of deleted stable samples is equal to e;

For the currently selected stable sample, calculate the distance from the stable sample to the surrounding k nearest neighbor samples belonging to the majority class sample set;

Calculate the probability of deleting the stable sample according to the distance; wherein, the smaller the distance, the greater the deletion probability;

If the deletion probability is greater than or equal to 0.5, delete the stable sample and update the number f of the deleted stable samples;

Select the next stable sample.

The third embodiment of the present invention also provides a preprocessing device for unbalanced data, including a processor, a memory, and a computer program stored in the memory and operable on the processor. The above steps are realized when the processor executes the computer program. Alternatively, when the processor executes the computer program, the functions of the modules in the foregoing device embodiments are realized.

Exemplarily, the computer program may be divided into one or more units, and the one or more units are stored in the memory and executed by the processor to implement the present invention. The one or more units may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the multi-classification-oriented unbalanced data preprocessing device.

The multi-category-oriented unbalanced data preprocessing device may be computing devices such as desktop computers, notebooks, palmtop computers, and cloud servers. The multi-classification-oriented unbalanced data preprocessing device may include, but not limited to, a processor and a memory. Those skilled in the art can understand that the schematic diagram is only an example of a multi-category-oriented unbalanced data preprocessing device, and does not constitute a limitation on the multi-classification-oriented unbalanced data preprocessing device, and may include more or Fewer components, or a combination of certain components, or different components, for example, the multi-category-oriented unbalanced data preprocessing device may also include input and output devices, network access devices, buses, and the like.

The so-called processor can be a central processing unit (Central Processing Unit, CPU), and can also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf Programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processor can be microprocessor or this processor also can be any conventional processor etc., described processor is the control center of described unbalanced data pre-processing equipment facing multi-category, utilizes various interfaces and line connection Parts of the entire multiclass-oriented unbalanced data preprocessing apparatus.

The memory can be used to store the computer programs and/or modules, and the processor realizes the oriented Various functions of multi-class unbalanced data preprocessing equipment. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, at least one application program required by a function (such as a sound playback function, an image playback function, etc.) and the like; the storage data area may store Data created based on the use of the mobile phone (such as audio data, phonebook, etc.), etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, A flash memory card (Flash Card), at least one magnetic disk storage device, flash memory device, or other volatile solid state storage devices.

Wherein, if the integrated unit of the multi-category-oriented unbalanced data preprocessing device is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention realizes all or part of the processes in the methods of the above embodiments, and can also be completed by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer When the program is executed by the processor, the steps in the above-mentioned various method embodiments can be realized. Wherein, the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM, Read-Only Memory) , random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, computer-readable media Excludes electrical carrier signals and telecommunication signals.

It should be noted that the device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physically separated. A unit can be located in one place, or it can be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that they have a communication connection, which can be specifically implemented as one or more communication buses or signal lines. It can be understood and implemented by those skilled in the art without creative effort.

The above description is a preferred embodiment of the present invention, and it should be pointed out that for those skilled in the art, without departing from the principle of the present invention, some improvements and modifications can also be made, and these improvements and modifications are also considered Be the protection scope of the present invention.

Claims (9)

1. a kind of unbalanced data preprocessing method for classification, is characterized in that, comprises the steps: Read the original sample set; wherein, the original sample set includes at least two categories of sample sets; Receive the final sample set size input by the user and the imbalance ratio between each sample set to calculate the ideal number of samples in each sample set in the final sample set; Judging whether the sample set belongs to the minority class sample set or the majority class sample set according to the ideal sample number and the actual sample number of each sample set; For each sample in the minority class sample set, calculate the number of other class samples in the k-nearest neighbors of each sample and the number of samples belonging to the minority class sample set, so as to divide each sample into a noise sample Examples, unstable samples, boundary samples or stable samples and marked accordingly; among them, other class samples refer to samples in other sample sets except the samples in the minority class sample set; For the examples in each minority class sample set, delete, save, copy or synthesize according to the mark of each sample, so as to obtain the final minority class sample set corresponding to each minority class sample set; For each sample in the majority class sample set, calculate the number of samples belonging to the majority class sample set and other class samples in the k-nearest neighbors of each sample, so as to divide each sample into a noise sample sample, boundary sample or stable sample, and mark accordingly; For each sample in the majority class sample set, delete or save according to the mark of each sample, so as to obtain the final majority class sample set corresponding to each majority class sample set; A final sample set is generated according to the final minority class sample set and the majority class sample set, so as to realize preprocessing of unbalanced data. 2. The multi-classification-oriented unbalanced data preprocessing method according to claim 1, wherein said sample set belongs to Minority class sample set or majority class sample set is as follows: For each sample set, if the number of ideal samples is greater than the actual number of samples, it is judged that the sample set is a minority class sample set; if the number of ideal samples is less than or equal to the actual number of samples, then It is judged that the sample set is a majority class sample set. 3. The multi-classification-oriented unbalanced data preprocessing method according to claim 1, wherein, for the samples in each minority class sample set, calculate other class samples in the k nearest neighbors of each sample Examples and the number of samples belonging to the minority class sample set, so as to divide each sample into noise samples, unstable samples, boundary samples or stable samples and mark them accordingly, including: When it is judged that most of the k-nearest neighbor samples of the samples in the minority class sample set are other class samples, the sample is marked as a noise sample; When it is judged that most of the k-nearest neighbor samples of the samples in the minority class sample set are samples of other classes, the sample is marked as an unstable sample; When it is judged that the number of other class samples of the k-nearest neighbor samples of the sample in the minority class sample set is close to the number of samples in the minority class sample set, then mark the sample as a boundary sample; When it is judged that most of the k-nearest neighbor samples of the sample in the minority class sample set belong to the samples in the minority class sample set, mark the sample as a stable sample. 4. the multi-classification-oriented unbalanced data preprocessing method according to claim 3, is characterized in that, the described sample in each minority class sample set, deletes, saves, Copy or synthesize to obtain the final minority class sample set corresponding to each minority class sample set, specifically: For each example in the minority class example set: Delete all noise samples in the minority sample set; Add all unstable samples to the corresponding final minority sample set; Copy each boundary sample, the number of copies is |c-1|, add the boundary sample and the copied sample to the corresponding final minority sample set; where c is the replication ratio, and c= (the ideal number of samples of the minority class sample set - the number of unstable samples)/(the actual number of samples of the minority class sample set - the number of noise samples - the number of unstable samples); For each stable sample, synthesize a new sample with the surrounding samples, the number of synthesis is |c-1|, and add this sample and the newly synthesized sample to the corresponding final minority sample set; where, The synthesis method is to randomly select a sample xj belonging to the minority class sample set from the k-nearest neighbors of the stable sample x _i each time, and the newly synthesized sample x _i '=xi+(xi-xj)*a, a is a random number between 0 and 1; Calculate the number d of samples belonging to the minority class sample set that still needs to be generated; where, d=the ideal sample number of the minority class sample set-the final minority class sample set that belongs to the minority class sample set The current number of samples of ; Randomly select d stable samples, synthesize a new sample from each stable sample and surrounding samples, and add the synthesized new samples to the corresponding final minority class sample set; A final minority class sample set corresponding to each minority class sample set is obtained. 5. the multi-classification-oriented unbalanced data preprocessing method according to claim 3, characterized in that, for the samples in each majority class sample set, calculate the k nearest neighbors belonging to the majority of each sample The number of samples in the class sample set and other class samples, so that each sample can be divided into noise samples, boundary samples or stable samples, and marked accordingly, specifically: When it is judged that most of the k-nearest neighbor samples of the samples in the majority class sample set are other class samples, the sample is marked as a noise sample; When it is judged that the number of other class samples of the k-nearest neighbor samples of the sample in the majority class sample set is close to the number of samples belonging to the majority class sample set, then mark the sample as a boundary sample; When it is judged that most of the k-nearest neighbor samples of the samples in the majority class sample set belong to the samples in the majority class sample set, mark the sample as a stable sample. 6. the multi-classification-oriented imbalanced data preprocessing method according to claim 5, characterized in that, the samples in each majority class sample set are deleted or saved according to the mark of each sample, To obtain the final majority class sample set corresponding to each majority class sample set, specifically including: For each example in the majority class example set: remove noise samples; Keep all boundary samples; For each stable sample, perform a selective deletion operation until e stable samples are deleted; where, e=the actual number of samples in the majority class sample set - the number of noise samples - the majority class sample set Ideal number of samples; According to the reserved boundary samples and the remaining stable samples, the final majority class sample set corresponding to each majority class sample set is obtained. 7. The multi-classification-oriented unbalanced data preprocessing method according to claim 6, characterized in that, for each stable example, a selective deletion operation is performed until deletion of e stable examples is specifically: Repeat the following steps until the number f of deleted stable samples is equal to e; For the currently selected stable sample, calculate the distance from the stable sample to the surrounding k nearest neighbor samples belonging to the majority class sample set; Calculate the probability of deleting the stable sample according to the distance; wherein, the smaller the distance, the greater the deletion probability; If the deletion probability is greater than or equal to 0.5, delete the stable sample and update the number f of the deleted stable samples; Select the next stable sample. 8. A multi-classification-oriented unbalanced data preprocessing device, characterized in that it comprises: A data reading unit, configured to read an original sample set; wherein, the original sample set includes at least two categories of sample sets; The data receiving unit is used to receive the size of the final sample set input by the user and the imbalance ratio between each sample set, so as to calculate the ideal number of samples of each sample set in the final sample set; A judging unit, configured to judge whether the sample set belongs to the minority sample set or the majority sample set according to the ideal sample number and the actual sample number of each sample set; The minority class sample classification unit is used to calculate the number of other class samples in the k nearest neighbors of each sample and the number of samples belonging to the minority class sample set for each sample in the minority class sample set, so as to Divide each sample into noise sample, unstable sample, boundary sample or stable sample and mark accordingly; among them, other class samples refer to the samples other than the samples in the minority class sample set. samples in the sample set; The minority class sample processing unit is used to delete, save, copy or synthesize the samples in each minority class sample set according to the mark of each sample, so as to obtain the final result corresponding to each minority class sample set Minority class sample set; The majority class sample classification unit is used to calculate the number of samples and other class samples belonging to the majority class sample set in the k-nearest neighbors of each sample for the samples in each majority class sample set, so as to Divide each sample into noise sample, boundary sample or stable sample, and mark accordingly; The majority class sample processing unit is used to delete or save the samples in each majority class sample set according to the mark of each sample, so as to obtain the final majority class sample corresponding to each majority class sample set set; The final sample set generating unit is configured to generate a final sample set according to the final minority class sample set and the majority class sample set, so as to realize preprocessing of unbalanced data. 9. A multi-classification-oriented unbalanced data preprocessing device, characterized in that it comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and the processor executes The computer program realizes the multi-classification-oriented unbalanced data preprocessing method according to any one of claims 1-7.