TWI564740B - Mutually-exclusive and collectively-exhaustive (mece) feature selection method and computer program product - Google Patents

Description

Mutual exclusion and complete set feature selection method, computer program product

The present invention relates to a feature selection method, and in particular to a mutually-exclusive and collectively-exhaustive feature selection method and computer program product.

In the field of machine learning and data exploration, large samples are often processed, and each sample may have many variables. However, too many variables can lead to the curse of dimensionality, making useful samples sparse. In order to solve this problem, the feature selection can generally be used to reduce the dimension or number of variables. For example, Principal Components Analysis (PCA) is a tool often used to reduce dimensions. A good feature selection method should be able to select fewer variables as possible, and the selected variables should be able to represent the original sample complete information, so that when using the selected variables to perform machine learning or data exploration, When the algorithm is implemented, the calculation can be reduced. Volume but achieved good accuracy. To this end, those with ordinary knowledge in the field are still constantly researching whether there is a better method of feature selection.

The invention proposes a feature selection method of mutual exclusion and complete set, which considers independence, importance and integrity.

Embodiments of the present invention provide a feature selection method for an electronic device. The feature selection method includes: obtaining a plurality of samples, wherein each sample has a plurality of variables; performing an independent variable generation step to divide the variable into a plurality of variable groups according to the number of clusters, and selecting from each variable group At least one variable is used to generate a plurality of independent variables; a plurality of important variables are selected from the independence variables; the first grouping index of the sample is calculated according to the important variables; and the first program is repeatedly executed for a preset number of times. The first procedure includes: generating a dummy variable for each sample; and combining the important variable with the dummy variable to calculate a second grouping indicator of the sample. The feature selection method further includes determining whether the first grouping indicator and the second grouping indicator pass the integrity test; and outputting the important variable when the first grouping indicator and the second grouping indicator pass the integrity test.

In some embodiments, the independence variable generation step includes: performing a Ward's method on the sample to determine the number of clusters; performing a K-means clustering algorithm on the variables according to the number of clusters to The variable is divided into a variable group; a third grouping indicator is calculated for each variable in each variable group; and a variable is selected from each variable group according to the third grouping indicator.

In some embodiments, the step of selecting an important variable from the independence variable comprises performing a stepwise selection algorithm on the independence variable to select an important variable.

In some embodiments, the step of selecting an important variable from the independence variable comprises performing an best subset algorithm on the independence variable to select an important variable.

In some embodiments, the step of generating a dummy variable comprises randomly generating a dummy variable for each sample in a uniform distribution.

In some embodiments, the step of determining whether the first grouping indicator and the second grouping indicator pass the integrity test comprises: calculating an average of the second grouping indicator; performing a statistical hypothesis verification, wherein the null hypothesis is The first grouping indicator is set to be the same as the average value; and if the null hypothesis is accepted, the first grouping indicator and the second grouping indicator are judged to pass the integrity test.

In some embodiments, the feature selection method further includes: if the first grouping indicator and the second grouping indicator fail the integrity test, determining whether the number of important variables is less than the number of variables; and if the number of important variables is less than the variable The number of clusters is increased by the number of clusters, and the independent variable generation step is re-executed according to the number of clusters after the increase.

The embodiment of the present invention also provides a computer program product, which can complete the above feature selection method after the computer loads the computer program product and executes it.

To make the above features and advantages of the present invention more apparent, The following specific embodiments are described in detail below with reference to the accompanying drawings.

100‧‧‧Electronic devices

102‧‧‧ sample

104‧‧‧variables

110‧‧‧ processor

120‧‧‧ memory

210, 220, 230‧‧‧ space

S301~S309, S401~S411‧‧‧ steps

FIG. 1 is a block diagram showing an electronic device according to an embodiment.

FIG. 2 is a schematic diagram illustrating the principle of mutual exclusion and complete set according to an embodiment.

FIG. 3 is a flow chart showing a feature selection method according to an embodiment.

4A and 4B are flowcharts illustrating a feature selection method according to an embodiment.

The terms "first", "second", "etc." used in this document are not intended to mean the order or the order, and are merely to distinguish between elements or operations described in the same technical terms.

Please refer to FIG. 1. FIG. 1 is a block diagram of an electronic device according to an embodiment. The electronic device 100 includes a processor 110 and a memory 120. The memory 120 stores a computer program product, and when the processor 110 executes the computer program product, a feature selection method is executed. In some embodiments, the electronic device 100 can be implemented as a computer of any form, such as a personal computer, a server, or an industrial computer. The present invention is not limited thereto.

The electronic device 100 obtains a plurality of samples 102, wherein each sample 102 has a plurality of variables, that is, one sample can be represented as a vector of a plurality of variables, and the electronic device 100 loses after performing the feature selection method. The selected variable 104 is selected. In some embodiments, the samples 102 are for a semiconductor process, such as from a sensor on a semiconductor production line, each sample may represent a product, and each variable may be temperature, thickness, or material concentration, and the like. Alternatively, the samples 102 may also be information about biological information, metal smelting, sales of the store, digital images, and the like. The invention does not limit the content, number, data type, etc. of these samples and variables.

Please refer to FIG. 2. FIG. 2 is a schematic diagram illustrating the principle of mutual exclusion and complete set according to an embodiment. It is assumed here that there are five variables, which are labeled as areas A to E in FIG. 2. If these variables are mutually exclusive, as shown by space 210, areas A~E do not overlap each other, but can be seen. There will be some gaps between the areas A~E. If these variables are complete sets, as shown by space 220, areas A~E will fill the entire space 220, but will overlap each other. If these variables are mutually exclusive and complete sets, as shown by space 230, areas A~E do not overlap each other, but fill the entire space. In the following embodiments, variables such as space 230 are selected, while being mutually exclusive and complete sets.

Please refer to FIG. 3. FIG. 3 is a flowchart illustrating a feature selection method according to an embodiment. In step S301, the preprocessing of the data is performed first, for example, incomplete, incorrect, inaccurate, or irrelevant variables may be identified first, and attempts to replace, modify, convert, or remove the variables are attempted. In this embodiment, step S301 includes the following sub-steps. First, the first collection of data can be used to collect data representing the real world physical conditions through sensors or measuring instruments, and convert the data into a specific format. Second, the data can be cleaned up to identify errors, noise, missing values, outliers, or inconsistent values. Data cleansing can increase the quality of the data to the loss For example, the maximum value, minimum value, average value, interpolation value, probability distribution, etc. of other samples can be calculated to estimate the missing value. Third, the integration of data can integrate different sources of information to produce meaningful and valuable information. Fourth, the conversion of data can be converted into binary, decimal, octal, interval scale, scale of scale, and so on. However, these sub-steps are merely examples, and in other embodiments, processing such as denoising, deleting extreme values, and the like may be performed. Alternatively, in some embodiments, the obtained variables may also be input to a sigmoid function or other function to change the distribution of the values. The invention does not limit the specific content of data preprocessing.

In step S302, independence considerations are made to obtain a plurality of independent variables that are independent of each other (or have low correlation). In some embodiments, all of the variables may be grouped first to generate a plurality of sets of variables, and then independent variables are selected from the group of variables (this step is also referred to as an independent variable generation step). For example, the Ward's method can be performed on all samples to determine a number of clusters. The Waldorf method first treats each sample as a group, and then merges the two groups each time according to the sum of square error (SSE) until all the groups are merged into the same group. Group. Let x _ij be the ith sample in the jth group, Means the average of the variables of all samples in the jth group, The average of the variables representing all samples, the sum of squared errors can be expressed as the following equation (1), and the total sum of square (SST) can be expressed as the following equation (2), and the R-squared value can be expressed as The following equation (3).

R ² =( SST - SSE )/ SST ...(3)

Therefore, when the Waldorf method is performed, the SSE will become larger and larger after the group is merged. We can set a threshold. When the SSE is greater than the threshold, the merge group is stopped to obtain the number of groups. After the number of groups is obtained, a K-means grouping algorithm can be performed on all the variables according to the number of clusters to divide the variables into a plurality of variable groups. However, the Waldorf method and the K-means algorithm described above are merely examples, and any conventional grouping algorithm can be used to generate a variable group. For example, the Huade method described above can also be replaced by a hierarchical grouping method such as single-linkage, complete-linkage, average-linkage, and centroid-linkage. The number of clusters. Alternatively, a fuzzy C-means algorithm, a Mixture of Gaussians model, a Hierarchical clustering, or the like may be used for grouping, and the grouping algorithm used may be supervised. Or an unsupervised clustering algorithm, the invention is not limited thereto.

It is worth noting that the above steps are to group the variables rather than group the samples. For example, a certain variable can be taken from all samples to generate a vector to represent the variable, and the distance between the variables can calculate the Euclidean distance between the vectors, but the invention does not limit how Define the distance between the variables. After obtaining a plurality of variable groups, a corresponding grouping indicator (also referred to as a third grouping indicator) is calculated for each variable in the variable group, and each group is determined according to the grouping indicator. Select at least one variable as an independent variable. In this embodiment, the variable having the best clustering index is selected from each variable group to generate an independent variable, but in other embodiments, two or more may be selected in each variable group. The variables are not limited by the present invention. In addition, the above-mentioned grouping index may be an R-squared value, an F-value, or other suitable index, and the present invention is not limited thereto.

In step S303, consideration of importance is performed, and a plurality of important variables are selected from the above-described independence variables. In some embodiments, step S303 is to perform a stepwise selection algorithm or a best subset algorithm to select important variables. In general, the step selection algorithm can be divided into forward selection and backward elimination. In the forward selection, in the unselected independence variable, if the F value of the variable is greater than a certain threshold, the variable having the largest F value is selected. When deleting later, if the F value of the variable in the selected variable is less than a certain threshold, the variable with the smallest F value will be deleted. The forward selection and the subsequent deletion may be performed several times, and the last selected variables are important variables. However, those skilled in the art can understand the step selection algorithm and will not be described in detail herein.

In addition, the best sub-set algorithm is a combination of variables arbitrarily selected by the user to produce a best-fitting regression model. The general selection principle is to satisfy the statistical characteristics of the data set with the fewest number of variables. The best sub-set algorithm uses the clustering indicator to include the R-squared value, the adjusted R-square, the mean square error (Mean-Square-Error), and Mallow's Cp. When the user When the number of variables of the selected variable combination is the same, the R square value may be used; if the number of variables is different, the adjusted R square value and Mallow's Cp may be selected as the mutual verification statistical tools. However, those skilled in the art should also understand the best sub-set algorithm, and will not be described here.

In step S303, important variables are to be selected, but in addition to the step selection algorithm and the optimal sub-set algorithm described above, other methods may be used in other embodiments. For example, the independence variable obtained in step S302 may be subjected to principal component analysis first, and the component having the largest eigenvalue (eigenvalue) may have multiple coefficients (also representing weights), which respectively correspond to all independent variables. Independent variables with the largest coefficient can be selected as important variables. Alternatively, adaptive enhancement (AbaBoost) can be performed according to the independence variable, in which each selected variable will have a weight, and we can select important variables based on these weights (for example, select important variables with larger weights) ). Alternatively, linear regression can be performed based on these independent variables. After the linear regression is completed, each variable will also have its own weight. We can also select important variables based on these weights. The spirit of step S303 consists in selecting important variables from independent variables (or low correlation), but those skilled in the art can understand that in the prior art, there are many means to achieve this goal. The invention does not limit the specific practice of step S303.

Next, in step S304, integrity considerations are made, in this embodiment both internal integrity and external integrity are considered. The important variable obtained from step S303 is a subset of all variables, and the internal integrity is to test whether these important variables have enough information to make a decision. (decision-making), and the external integrity is to test whether these important variables have enough information for all samples.

Specifically, a clustering indicator (also referred to as a first grouping indicator) may be calculated based on these important variables, for example, an R-squared value, here denoted as r _s ² . That is to say, the samples can be regrouped according to these important variables, and then the R-squared value r _s ^{2 is} calculated from the results of the clustering. Next, the first program is repeatedly executed a preset number of times, and the first program includes generating a dummy variable for each sample. In this embodiment, the aforementioned dummy variables are randomly generated according to a uniform distribution. However, in other embodiments, the dummy variable may also be generated according to other probability distributions; or, several important variables may be multiplied by a weight and then added, and the added value plus a random value In order to generate a dummy variable, the present invention is not limited thereto.

The first program further includes combining the important variable with the dummy variable, and regrouping the sample according to the combined variable to calculate the grouping index of the sample (also referred to as the second grouping indicator, the type of which must be the first grouping indicator mentioned above) The same, for example, the R square value). The preset number of times described above is, for example, but not limited to, 100 times, so a total of 100 R-squared values are generated. After performing the first program 100 times, it is calculated whether the first grouping indicator and the 100 second grouping indicators pass an integrity test. In this embodiment, the average of these second clustering indicators is calculated first, denoted as r _n ² , and then a hypothesis test is performed, and the null hypothesis of the statistical hypothesis test is set as the first The one-group index r _s ^{2 is the} same as the average value r _n ² and can be expressed as the following equation (4); and the alternative hypothesis can be set as the first group index r _s ^{2 is} greater than (or not equal to) the average value r _n ² can be expressed as the following equation (5).

H ₀ : r _n ² = r _s ² (4)

H ₁ : r _n ² > r _s ² ... (5)

Here, the above statistical hypothesis is determined as a Z test. However, those having ordinary knowledge in the art can understand the content of the Z test or other verification, and will not be described here. If the above null hypothesis is accepted (or not rejected), it means that the existing important variables are already complete, and it can be said that no matter how many dummy variables are added, the R-squared value will not change significantly, and the completeness can be judged to be passed. Sex test. Conversely, if the null hypothesis is rejected, it means that the added dummy variable may improve the integrity of the important variable, so it can be judged that the integrity test has not been passed.

In this embodiment, the statistical hypothesis check is used to determine whether the first cluster indicator and the plurality of second cluster indicators pass the integrity test. However, in other embodiments, the first cluster indicator and the average value r _n ² may also be determined. Whether the error is less than a critical value, and if so, judges the integrity test. Alternatively, the above average value can be replaced with the median. Alternatively, the mean squared difference between the first grouping indicator and the second grouping indicator may be calculated. If the mean squared difference is greater than a certain threshold value, it means that the integrity test is not passed. In addition, in the embodiment, the first grouping index and the second grouping index are R-squared values, but in other embodiments, other suitable grouping indicators, such as F-numbers, may also be used. Alternatively, the first clustering indicator and the second clustering indicator may also be the correct (error) rate of a prediction model. For example, if the above sample is a semiconductor product, a machine learning algorithm can be executed according to existing important variables to obtain a model to determine whether the semiconductor product is defective, and the prediction accuracy rate is regarded as a clustering index, and The model is retrained each time a dummy variable is added. That is to say, if the above dummy variable cannot increase the accuracy of the prediction, the pass integrity test can be judged. However, those skilled in the art can retouch or modify the teachings in accordance with the above teachings. The present invention does not limit how to determine whether the first grouping index and the second grouping indicator pass the integrity test.

If it is judged in step S304 that the integrity test is passed, in step S305, the selected important variable is output. If the integrity test is not passed, step S306 is performed to determine whether the number of selected important variables is less than the number of all variables. If the result of step S306 is YES, step S307 is performed to increase the number of clusters, and then returning to step S302, the independence variable generating step is re-executed based on the increased number of clusters. That is to say, when the second step S302 is performed, the clustering variable is not determined according to the Waldorf method, but the K-means algorithm is executed according to the increased number of clusters. As a result, the resulting variable group will increase, and the selected independence variables will increase. On the other hand, if the result of the step S306 is NO, the process proceeds to a step S308 to end the feature selection. In some embodiments, step S309 may also be performed to find unobserved variables, such as increasing the sensor on the semiconductor line, thereby taking more variables.

The invention also proposes a computer program product, which can complete the above feature selection method after the computer loads the computer program product and executes it. However, the present invention does not limit the programming language in which the computer program product is implemented.

Please refer to FIG. 4A and FIG. 4B, FIG. 4A and FIG. 4B are based on a real The embodiment illustrates a flow chart of the feature selection method. In step S401, a plurality of samples are acquired. In step S402, the variable is divided into a plurality of variable groups according to the number of clusters, and at least one variable is selected from each of the variable groups to generate a plurality of independent variables. In step S403, a plurality of important variables are selected from the independence variables. In step S404, the first grouping indicator of the sample is calculated based on the important variable. In step S405, a dummy variable is generated for each sample. In step S406, the important variable is combined with the dummy variable to calculate a second grouping index of the sample, wherein step S405 and step S406 are collectively referred to as a first program. In step S407, it is determined whether the preset number of times has been reached. If not, the process returns to step S405, and if yes, the process proceeds to step S408. In step S408, it is determined whether the first grouping indicator and the second grouping indicator pass the integrity test, and if so, the important variable is output in step S409, otherwise step S410 is performed. In step S410, it is determined whether the number of important variables is less than the number of all variables. If the flow is otherwise ended, if yes, step S412 is performed to increase the number of clusters and return to step S402. However, the steps in FIGS. 4A and 4B have been described in detail above, and will not be described again herein. It should be noted that the steps in FIG. 4A and FIG. 4B can be implemented as a plurality of codes or circuits, and the present invention is not limited thereto. In addition, the methods of FIGS. 4A and 4B can be used in conjunction with the above embodiments, or can be used alone. In other words, other steps can be added between the steps of Figures 4A and 4B.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention, and any one of ordinary skill in the art can make some changes and refinements without departing from the spirit and scope of the present invention. The scope of the invention is defined by the scope of the appended claims.

S301~S309‧‧‧Steps

Claims (8)

A mutually exclusive and complete set of feature selection methods for an electronic device, the feature selection method comprising: obtaining a plurality of samples, wherein each of the samples has a plurality of variables; performing an independent variable generation step, from each Obtaining one of the variables in the samples to generate a vector to represent the corresponding variable, to divide the variables into a plurality of variable groups according to a group number, and from each of the variable groups Selecting at least one of the variables to generate a plurality of independent variables; selecting a plurality of important variables from the independent variables; calculating a first grouping index of the samples according to the important variables; performing the first a first predetermined number of times, wherein the first program comprises: generating a dummy variable for each of the samples; and combining the important variables with the dummy variable to calculate a second grouping index of the samples; Determining whether the first grouping indicator and the second grouping indicator pass an integrity test; and when the first grouping indicator and the second grouping indicator pass the integrity test, The out some important variables. The feature selection method of claim 1, wherein the independence variable generation step comprises: performing a Ward's method on the samples to determine a number of clusters; performing a K-means grouping algorithm on the variables according to the number of clusters to divide the variables into the group of variables; for each of the variable groups Each of the variables calculates a third grouping indicator; and based on the third grouping indicators, one of the variables is selected from each of the group of variables. The feature selection method according to claim 1, wherein the step of selecting the important variables from the independence variables comprises: performing a stepwise selection algorithm on the independence variables to Select these important variables. The feature selection method according to claim 1, wherein the step of selecting the important variables from the independence variables comprises: performing a best subset calculus on the independent variables. The law selects these important variables. The feature selection method of claim 1, wherein the step of generating the dummy variable for each of the samples comprises: randomly generating the dummy for each of the samples in a uniform distribution variable. The method for selecting a feature according to claim 1, wherein the step of determining whether the first grouping indicator and the second grouping indicator pass the integrity test comprises: calculating the second grouping indicator An average; performing a statistical hypothesis test, wherein in the statistical hypothesis assay, a null hypothesis is set such that the first clustering indicator is the same as the average; and if the null hypothesis is accepted, determining The first cluster indicator and the second cluster indicator pass the integrity test. The method for selecting a feature as described in claim 1 further includes: if the first grouping indicator and the second grouping indicator fail to pass the integrity test, determining whether the number of the important variables is less than the variables And if the number of the important variables is less than the number of the variables, the number of the clusters is increased, and the independence variable generating step is re-executed according to the increased number of the clusters. A computer program product, when the computer is loaded into the computer program product and executed, the feature selection method according to any one of claims 1 to 7 can be completed.