CN101334843B - Pattern recognition characteristic extraction method and apparatus - Google Patents

Abstract

The invention discloses a method and a device for feature extraction in pattern recognition, in order to effectively avoid the subjectivity of pre-designated and selected feature numbers in previous feature extraction. The feature extraction method includes the steps of: determining discrete feature variables and class variables according to the pattern original information of the sample, and preprocessing the feature variables and class variables; setting a joint contribution threshold; determining the combination of feature variables and the class variable Joint contribution degree: obtain the combination of feature variables whose joint contribution degree is greater than or equal to the set joint contribution degree threshold. The feature extraction device includes: a numerical preprocessing module, a threshold setting module, a joint contribution degree determination module and a feature extraction module. The feature extraction method and device in pattern recognition of the present invention can be widely used in the feature extraction of discrete digital image information, fingerprint information, facial pattern information, voice information or handwritten/printed character information.

Description

Feature extraction method and device in pattern recognition

technical field

The invention relates to the field of pattern recognition, in particular to a feature extraction method and device in pattern recognition.

Background technique

A pattern is information with time and space distribution obtained by observing specific individual things; the category to which a pattern belongs or the overall pattern in the same category is called a pattern class (or class for short). "Pattern recognition" is to divide the patterns to be recognized into their respective pattern categories on the basis of some certain measurements or observations.

The research on pattern recognition mainly focuses on two aspects, that is, how to study how organisms (including people) perceive objects, and how to use computers to realize pattern recognition under a given task.

A computer pattern recognition system basically consists of three interrelated but distinct processes, namely data generation, pattern analysis, and pattern classification. Data generation is to quantify the original information of the input pattern, convert it into a vector, and become a form that can be easily processed by a computer. Pattern analysis is the processing of data, including feature selection, feature extraction, data dimensionality compression, and determination of possible categories. Pattern classification is to use the information obtained by pattern analysis to train the computer so as to formulate discrimination standards in order to classify the recognized patterns.

Among them, feature extraction in pattern analysis is very important for efficient pattern classification. Pattern classification involves various fields, such as image classification, speech recognition, biotechnology, medicine, etc. The efficiency of classification is always an important content of pattern classification research. In many practical problems, there are many characteristic variables that can be studied for pattern classification. If all the characteristic variables available for reference are taken into consideration for classification, the efficiency will be very low. Unusable in practice. Therefore, it is necessary to extract the feature variables, and use the feature subset obtained through feature extraction as the input of the objective classifier. After training the objective classifier, the feature subset is used for classification, thereby improving the efficiency of classification.

Feature extraction is based on searching for a feature subspace that minimizes the amount of information loss. The amount of information is measured by the mutual information between feature subspaces and class variables. Feature extraction methods not only consider the correlation between feature variables and class variables, Moreover, the correlation between feature variables is considered.

Feature extraction can be applied in Chinese medicine. Syndrome differentiation and treatment is the core of TCM. Syndrome differentiation is a method of understanding and diagnosing diseases using TCM theory. Syndrome is a complex of symptoms with unknown etiology and a symptom of abnormalities in the body. Symptoms in a broad sense include not only the information of the four diagnoses, but also gender, physical fitness, emotion, stress, diet, living habits and many other factors. In the process of syndrome differentiation, because there are too many symptoms and signs, it is difficult for doctors to take all observed symptoms into account. Different symptoms and signs play different roles in the process of syndrome differentiation. How to find out the set of symptoms and signs with the largest amount of information as a syndrome differentiation standard for a certain syndrome is a very important issue in the field of traditional Chinese medicine.

Feature extraction can also be applied to pattern recognition of digital images. The pattern recognition of digital images is based on the pixel gray value of the image for pattern classification. An image has a lot of pixels, such as commonly used 1280×960 pixels, 640×480 pixels, 320×240 pixels, 160×120 pixels, etc. If all pixels are used as the input of the pattern classifier in pattern classification, the efficiency will be very low. Therefore, feature extraction is also a very important research content for image pattern classification. In image feature extraction, each pixel is regarded as a feature variable, and the most useful pixel for pattern classification is selected as the input of objective classifier.

On the method of feature variable extraction. Correlation analysis is the basis for selecting feature sets with a large amount of information, and feature variables can be selected according to their correlation values with class variables.

At present, there are many statistical methods for analyzing correlation. The simplest method is the correlation coefficient method, but this method is only suitable for analyzing linear correlation problems, and many practical problems are nonlinear relations. The commonly used non-linear statistical analysis method is the logistic regression method, which requires that the characteristic variables be independent of each other, but many practical problems are difficult to meet this condition. More importantly, the regression coefficient of the logistic regression method cannot directly reflect the correlation value between the feature variable and the class variable, it must be determined by the odds ratio (OR) value, and the OR value has no actual physical meaning. Principal component analysis method and factor analysis method can also be used for correlation analysis. These two methods can only analyze the linear relationship between variables, and cannot measure any correlation between variables.

The entropy-based mutual information method can not only analyze the correlation between numerical variables (discrete variables and continuous variables), but also measure any correlation between variables. Mutual information is one of the core concepts in entropy theory. It is an important measure of the adaptability of nonlinear complex systems. Its essence is the information transfer between things and the statistical correlation between random variables. It has been applied to many fields, especially is the field of pattern recognition.

Compared with traditional methods, entropy-based mutual information has the following advantages:

1) It can measure both the linear correlation between variables and the nonlinear correlation between variables;

2) Compared with the logistic regression nonlinear analysis method, the entropy-based mutual information method has no independent conditions for the analyzed variables;

3) The entropy-based mutual information method can not only analyze the correlation between numerical variables (discrete variables and continuous variables), but also measure the correlation between hierarchical variables and symbolic variables.

The optimal feature selection method is to evaluate all feature combinations, which usually leads to a combination explosion problem, so it is very important to study effective feature extraction methods. At present, many scholars have been engaged in research in this area, and several effective feature extraction methods have been proposed to solve the combination problem. But in these methods, the number of selected features is usually pre-specified artificially, which will inevitably introduce personal subjectivity, so it is not a good censoring criterion.

Contents of the invention

One of the objects of the present invention is to provide a feature extraction method in pattern recognition, which can effectively avoid the subjectivity of pre-specifying the number of features to be selected.

In order to achieve the above object, the technical scheme adopted in the present invention is:

The feature extraction method in the pattern recognition comprises steps:

Determine the discrete feature variables and class variables according to the model original information of the sample, and preprocess the feature variables and class variables;

Set the joint contribution threshold;

Determine the joint contribution of the combination of feature variables and class variables;

A combination of feature variables whose joint contribution degree is greater than or equal to the set joint contribution degree threshold is acquired.

In the existing feature extraction methods, the number of selected features is usually pre-specified artificially, which inevitably introduces personal subjectivity. Based on this problem, the present invention proposes a new definition form of contribution degree based on mutual information, and uses the threshold value of the specified joint contribution degree instead of the specified number of features as the censoring criterion for feature extraction. According to the specified joint contribution threshold, extract the combination of feature variables whose joint contribution is greater than or equal to the set joint contribution threshold, so as to obtain a feature subspace that minimizes information loss, which can effectively avoid previous feature extraction. subjectivity in .

Another object of the present invention is to provide a feature extraction device in pattern recognition, which can effectively avoid the subjectivity of specifying the number of features to be selected in advance.

In order to achieve this goal, the technical solutions adopted are:

The feature extraction device in the pattern recognition includes:

The numerical preprocessing module is used to determine discrete feature variables and class variables according to the original information of the sample pattern, and preprocess the feature variables and class variables; determine the possible values of each feature variable, and determine the possible values of class variables value, set the feature subset, and initialize the feature subset to an empty set;

A threshold setting module, configured to set a joint contribution threshold;

A joint contribution determination module is used to determine the joint contribution of feature subsets and class variables;

The feature extraction module is configured to obtain a feature subset whose joint contribution is greater than or equal to the set joint contribution threshold according to the joint contribution.

In the existing feature extraction, the number of selected features is usually pre-specified artificially, which inevitably introduces individual subjectivity. Based on this problem, the present invention proposes a new definition form of contribution degree based on mutual information, and uses the threshold value of the joint contribution degree preset by the setting module instead of the specified number of features as the censoring criterion for feature extraction. The joint contribution degree of feature subsets and class variables is determined by the joint contribution degree determination module, and according to the threshold value of the joint contribution degree preset by the setting module, the joint contribution degree extracted by the feature extraction module is greater than or equal to the set joint contribution The feature subset of degree threshold is obtained to obtain a feature subspace that minimizes the amount of information loss, which can effectively avoid the subjectivity in previous feature extraction.

Description of drawings

Fig. 1 is the flowchart of pattern recognition method of the present invention;

Fig. 2 is a system block diagram of the pattern recognition device of the present invention;

Fig. 3 is a schematic diagram of mutual information between each symptom and syndrome in the embodiment of the present invention;

Fig. 4 is a schematic diagram of the contribution of each symptom in the embodiment of the present invention;

Fig. 5 is a schematic diagram of the joint contribution of selected symptoms in the embodiment of the present invention.

Detailed ways

In order to better understand the present invention, the present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

Feature extraction is to select the most important feature combination to minimize the amount of information loss. From a practical point of view, it can save a lot of classification processing time.

The present invention proposes a feature extraction method and device based on a new truncation criterion, which is mainly for feature extraction of discrete variables. In the feature extraction method and device, a new form of joint contribution degree based on mutual information is defined, and the threshold value of the specified joint contribution degree is used instead of the specified number of features as the censoring criterion for feature extraction, and the extracted joint contribution degree is greater than or The combination of feature variables equal to the set joint contribution threshold can obtain a feature subspace that minimizes the amount of information loss, which can effectively avoid the subjectivity in the previous feature extraction. At the same time, the calculation based on the sample size proposed by the present invention The method of joint mutual information can greatly reduce the amount of calculation.

A new contribution degree based on mutual information is defined as follows:

Definition: Let I(X _i ; Y), i=1, 2, ..., n represent the mutual information between each feature variable and class variable, I(X; Y) represents the total joint mutual information, each feature The mutual information-based contribution of variables is defined as:

r _i =I(X _i ;Y)/I(X;Y), i=1, 2, . . . , n

The joint contribution between the subset S of the feature variable set X and the class variable Y is:

r _s =I(S;Y)/I(X;Y)

Explanation: According to the nature of mutual information based on Shannon entropy, the more feature variables, the greater the mutual information with class variables. Therefore, the value range of contribution degree and joint contribution degree is between [0, 1].

The specific operation method of feature extraction based on joint contribution is introduced as follows:

Given a selected feature subset S, the algorithm selects the next feature variable from the feature set X to meet the requirement that the feature variable be added to S to generate a new feature subset S←{S,X _i } and the class variable The mutual information between them is the largest. If a feature variable is to be selected, the information provided by the feature variable should not be included in the selected feature subset S. For example, if two feature variables X _i and X _j are highly correlated, then the value of I(X _i ; X _j ) will be large, and when one of the variables is selected, the chance of the other variable being selected will be greatly reduced .

The feature extraction method in the pattern recognition of the present invention comprises the steps of: determining discrete feature variables and class variables according to the pattern original information of the sample, and preprocessing the feature variables and class variables; setting a joint contribution threshold; determining the feature variables The combination of the combination and the joint contribution degree of the class variable; obtain the combination of the feature variables whose joint contribution degree is greater than or equal to the set joint contribution degree threshold.

Referring to Fig. 1, in combination with the problem of syndrome differentiation and treatment in traditional Chinese medicine, the feature extraction method in the pattern recognition of the present invention is used to process the intermediate symptom information observed from the human body, including the following specific steps:

Step 1. Determine the discrete feature variables and class variables according to the original information of the sample pattern, and preprocess the feature variables and class variables; combine all feature variables into a feature variable set, and determine the possible value of each feature variable ;Determine the possible values of the class variable; set the feature subset, and initialize the feature subset to an empty set.

1022 clinical data of blood stasis syndrome were analyzed. There are 71 human symptoms recorded in these data. The values corresponding to these symptoms are the original information of the model. All symptoms are represented by discrete characteristic variables. Among them, some symptoms (characteristic variables) have two values. 0, 1 means that some symptoms (characteristic variables) have four values, which are represented by the values 0, 1, 2, and 3; syndromes of traditional Chinese medicine are represented by class variables, which have five values, representing five Three syndromes: Qi deficiency and blood stasis, Qi stagnation and blood stasis, Yang deficiency and blood stasis, mutual obstruction of phlegm and blood stasis, and obstruction of collaterals by blood stasis.

Step 2: Set the joint contribution threshold.

The value range of the threshold is [0, 1]. The specific value is usually determined according to actual needs. The larger the threshold, the more symptoms will be extracted. According to experience, the value range of the threshold is generally [0.9, 0.98 ]. The threshold value of the joint contribution degree in this embodiment is designated as 0.95.

Step 3. Determine the combination of symptoms and the joint contribution between symptoms, specifically including the following steps:

S300. Determine mutual information between each symptom and symptom;

S301. Determine the symptom that maximizes the mutual information between the symptom and the syndrome, remove the symptom from the symptom set, and add it to the feature subset;

S302. Determine the joint contribution degree of the feature subset and the syndrome.

Wherein, in step S300, the mutual information of each symptom and syndrome is through the formula:

$I(x_i;Y)=h\left(x_i\right)+h\left(Y\right)-h(x_i,Y)=\underset{j=1}{\overset{m_i}{\mathrm{\Σ}}}\underset{l=1}{\overset{k}{\mathrm{\Σ}}}p(a_j^i,c_l)\log \frac{p(a_j^i,c_l)}{p\left(a_j^i\right)p\left(c_l\right)}$ to be sure.

The mutual information formula of each symptom and syndrome is obtained as follows:

Assume that n feature variables are represented by a set X={X ₁ ; X ₂ ; ...; X _n }, and their probability density functions are p(x ¹ ), p(x ² ), ..., p(x ⁿ ). $x^i\∈\left\{a_j^i\right\},j=\mathrm{1,2},\&Center\; Dot;\&Center\; Dot;\&Center\; Dot;,m_i$ Denotes all possible values of the variable _Xi (symptom). The class variable (syndrome) is denoted by Y, and its probability distribution is denoted by p(y). The variable Y has k possible values y∈{c _i }, i=1, 2, ..., k, which means that all Features are mapped to k classes. The joint probability density function of Xi and _Y is represented by p( ^xi , y), and the Shannon entropy of feature variable _Xi can be expressed as:

$\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; log</span>}\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>$

The Shannon entropy of class variable Y can be expressed as:

$\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; Y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; log</span>}\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>$

The joint entropy between feature variable _Xi and class variable Y can be expressed as:

$\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; Y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; -</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\underset{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; log</span>}\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; )</span>$

Among them, _Xi can be replaced by a subset of feature variable set X, that is, the joint entropy can be extended to n feature variables. The mutual information between class variable Y and feature variable _Xi can be expressed as:

$\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ;</span>\mathrm{<span\; class="notranslate">\; Y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; Y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; Y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\underset{\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; log</span>}\frac{\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; a</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; )</span>}$

Where X _i can be replaced by a subset of feature variable set X.

Feature variables, class variables and their joint probability distributions are obtained through statistical methods, specifically:

j＝1，2，…，m_i表示特征变量X_i的值等于

的样本数，N_l，l＝1，2，…，k表示类变量Y的值等于c_l的样本数，

i＝1，2，…，n；j＝1，2，…，m_i；l＝1，2，…，k表示特征变量X_i的值等于

同时类变量Y的值等于c_j的样本数。Let n feature variables be represented by a set X={X ₁ , X ₂ ,...,X _n }, variable X _i has m _i values, namely $x^i\&Element;\left\{a_j^i\right\},j=\mathrm{1,2},\&CenterDot;\&Center\; Dot;\&Center\; Dot;,m_i$ , the class variable is denoted by Y, the variable Y has k values, that is, y∈{c _i }, i=1, 2,..., k, assuming we have N random samples T={ _xi , y _i }∈( A×C), where $x_i=(x_i^1,x_i^2,\&Center\; Dot;\&Center\; Dot;\&Center\; Dot;,x_i^{\mathrm{no}})\&Element;A=\left\{a_{j_1}^1\right\}\&times;\left\{a_{j_2}^2\right\}\&times;\&CenterDot;\&CenterDot;\&CenterDot;\&times;\left\{a_{j_{\mathrm{no}}}^{\mathrm{no}}\right\},$ j _i =1, 2, ..., m _i , i = 1, 2, ..., n, y _i ∈ C = {c _j }, j = 1, 2, ..., k.

j=1, 2,..., m _i means that the value of characteristic variable _Xi is equal to

The number of samples of , N _l , l=1, 2,..., k means the value of the class variable Y is equal to the number of samples of c _l ,

i=1, 2,..., n; j=1, 2,..., m _i ; l=1, 2,..., k means that the value of characteristic variable _Xi is equal to

At the same time the value of the class variable Y is equal to the number of samples of c _j .

At this time, the feature variables, class variables and their joint probability distribution can be obtained by statistical methods, namely $p\left(a_j^i\right)=N_j^i/N;$ p(c _l ) = N _l /N; $p(a_j^i,c_l)=N_{\mathrm{jl}}^i/N,$ i=1, 2, . . . , n; j=1, 2, . . . , m _i ; l=1, 2, . . . , k. Likewise, the joint probability distribution between the feature variable subset S and the class variable Y is also available.

By calculating the mutual information between each symptom and syndrome, it is shown in Figure 3.

There is a step between step S300 and step S301: removing symptoms whose mutual information with syndromes is smaller than a predetermined value from the symptom set.

After the mutual information of each symptom and symptom is obtained through the above mutual information calculation formula, the mutual information of some symptoms is very small, so these symptoms can be ignored, and feature extraction is performed on the retained symptom set, and this will not be correct. Classification has too much impact, which can save a lot of time for feature extraction.

In step S302, the joint contribution degree of feature subset and syndrome is obtained through the formula:

r _s =I(S;Y)/I(X;Y) to determine.

Among them, r _s represents the joint contribution;

I(S; Y) represents the joint mutual information, through the formula:

$I(S;Y)=h\left(S\right)+h\left(Y\right)-h(S,Y)=\underset{{\mathrm{<figure-callout\; id="1"\; label="j"\; filenames="S071B8156X20070723E000032.png"\; state="\{\{state\}\}">j</figure-callout>}}_1=1}{\overset{{\mathrm{<figure-callout\; id="1"\; label="m"\; filenames="S071B8156X20070723E000032.png"\; state="\{\{state\}\}">m</figure-callout>}}_1}{\mathrm{\&Sigma;}}}\&Center\; Dot;\&Center\; Dot;\&Center\; Dot;\underset{j_m=1}{\overset{m_m}{\mathrm{\&Sigma;}}}\underset{l=1}{\overset{k}{\mathrm{\&Sigma;}}}p(a_{{\mathrm{<figure-callout\; id="1"\; label="j"\; filenames="S071B8156X20070723E000032.png"\; state="\{\{state\}\}">j</figure-callout>}}_1}^1,\&Center\; Dot;\&Center\; Dot;\&Center\; Dot;,a_{j_1}^m,c_l)\log \frac{p(a_{{\mathrm{<figure-callout\; id="1"\; label="j"\; filenames="S071B8156X20070723E000032.png"\; state="\{\{state\}\}">j</figure-callout>}}_1}^1,\&Center\; Dot;\&Center\; Dot;\&Center\; Dot;,a_{j_1}^m,c_l)}{p(a_{{\mathrm{<figure-callout\; id="1"\; label="j"\; filenames="S071B8156X20070723E000032.png"\; state="\{\{state\}\}">j</figure-callout>}}_1}^1,\&Center\; Dot;\&Center\; Dot;\&Center\; Dot;,a_{j_1}^m)p\left(c_l\right)}$ to make sure;

I(X;Y) denotes the total joint mutual information.

The method for determining the total joint mutual information is introduced below.

According to the definition of the contribution degree, it is necessary to calculate the total joint mutual information between the symptom set and the syndrome. When the conventional mutual information calculation method is used for calculation, the calculation amount is very large, and when there are many symptoms, the combination explosion will occur. For example, there are 30 symptoms, each symptom has 4 values, and they are mapped to 2 categories, then it needs to calculate about 1.15× ¹⁰¹⁸ combined values, which is difficult to complete in actual operation. Through statistics, it can be found that in the case of limited samples, the probability of many combinations is 0, so the total joint mutual information can be calculated through samples without considering specific symptom combinations. The calculation method will be introduced below.

B=(B ₁ , B ₂ , . . . , B _N ) ^T is a frequency vector, representing the number of samples with equal values of characteristic variables (symptoms), and its calculation process will be described below. D=(D _ij ), i=1, 2,..., N; j=1, 2,..., k is a frequency matrix, which means that the values of characteristic variables (symptoms) are all equal, and the values of class variables (syndrome) The number of samples that are also equal, E=(E ₁ , E ₂ , ..., E _k ) ^T is a frequency vector, indicating the number of samples with equal values of class variables (syndrome). This algorithm can be realized through the following steps:

Step S3031: Assuming that the training sample T is known, initialize parameters: set all elements of vector B to 1, and set all elements of matrix D and vector E to 0.

Step S3032: The following procedure is used to obtain the frequency used in calculating the probability.

Let i=1, 2,..., N, j=i+1, i+2,..., N

If B _i =0, execute the next cycle;

otherwise

If y _i = c _l , then E _l = E _l + 1, l = 1, 2, . . . , k;

If x _i =x _j , then B _i =B _i +1, B _j =0;

If x _i =x _j and y _i =c _l , then D _il =D _il +1, l=1, 2, . . . , k. Step S3033: Calculate the total joint mutual information

$\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\frac{{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}}{\mathrm{<span\; class="notranslate">\; N</span>}}\mathrm{<span\; class="notranslate">\; log</span>}<span\; class="notranslate">\; (</span>\frac{{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}<span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; N</span>}}{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; B</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; )</span>$

Note: when D _ij ×B _i ×E _j is equal to 0, log(D _ij /B _i E _j )=0.

循环来计算联合概率，该算法与特征变量(症状)个数和每个特征变量(症状)可能的取值个数无关。Using this algorithm, it is easy to calculate the total joint mutual information I(X; Y), and when the sample size is not very large, the calculation amount can be greatly reduced. For example, when N=2000, n=30, k=2, only need

Cycle to calculate the joint probability, the algorithm has nothing to do with the number of characteristic variables (symptoms) and the number of possible values of each characteristic variable (symptom).

By calculating the total joint mutual information between 71 symptoms and syndromes in this embodiment is 1.7342.

According to the definition of mutual information-based contribution of each feature variable, it is easy to calculate the contribution of each symptom, and the individual contributions of all symptoms are shown in Figure 4.

Step 4: Obtain the combination of symptoms whose joint contribution is greater than or equal to the set joint contribution threshold, specifically including steps:

Comparing the determined joint contribution with the set joint contribution threshold,

If the determined joint contribution degree is greater than or equal to the set joint contribution degree threshold, then obtain the feature subset;

If the determined joint contribution degree is less than the set joint contribution degree threshold, then for the combination of each symptom of the symptom set and the feature subset, determine the symptom that maximizes the mutual information between the combination and the syndrome, and the symptom Remove from the symptom set and add to the feature subset; then go back to step 3 and execute down.

Through feature extraction, 9 symptoms are selected, and their joint contribution is 0.9711, and the results are shown in Figure 5. The order of selection is anxious noise and irritability, side numbness, chest tightness, insomnia, fatigue, occupation, varicose veins, dark purple tongue, and dark complexion, which means that the combined contribution of these 9 symptoms is the largest. These five symptoms contain the most information when diagnosing them.

In order to prove that the selected symptom combination has the largest amount of information, the most effective method is to use these symptoms to differentiate. Here, a multi-class support vector machine is selected for classification. The setting of the support vector machine is: penalty parameter C = 20, and the kernel function is selected as radial Basis function, the width is set to σ ² =0.1. 863 samples are used as training samples, and the remaining 159 samples are used as test samples. When all symptoms are used as the input of the support vector machine, 107 samples can be correctly classified through training, and the classification is correct The rate is 0.6729. When the symptom combination after feature extraction is used as the input of the support vector machine, 123 samples can be correctly classified, and the classification accuracy rate is 0.7736. Its correct rate is higher than that of all symptoms as input because there is noise in the entire symptom set. After feature extraction, the noise can be reduced, so the combination of symptoms after feature extraction is the combination with the largest amount of information.

In this feature extraction example, if the conventional mutual information calculation method is used for calculation, a combination explosion will occur, which cannot be realized in practice. However, according to the fast algorithm of discrete variable mutual information proposed here, this feature extraction can be done in about 2 hours. Finish.

Another example of the present invention is to use the present invention to recognize the digital characters of the real-time integrated circuit IC card.

This example is to quickly identify the card number printed on the produced IC card, so as to check whether the printed card number matches the input card number. 32 numbers are printed on each card, and these printed numbers are composed of Arabic numerals 0-9.

First, the digits printed on the IC card are collected by the image acquisition card to generate a digital image, and then the printed digits are divided into 32 digital areas by image processing method, and the size of each digital area is 8×10 pixels, and then Each number field is identified to determine its corresponding number. Six such IC cards need to be processed per second.

Apply the feature extraction method in the pattern recognition of the present invention, carry out feature extraction to each digital area, comprise the steps:

S01. Determine discrete feature variables and class variables according to the pattern original information of the sample, and preprocess the feature variables and class variables; combine all feature variables into a feature variable set, and determine the possible value of each feature variable; Determine the possible values of the class variables; set the feature subset, and initialize the feature subset to an empty set.

Here, the original information of the pattern is the gray value of the pixel in the digital image on the IC card, the feature variable is the pixel of the digital image, and the class variable is the digital value. Each feature variable (pixel) has two gray values 0 and 1, and the set of feature variables is 80 pixels. The digital area can be divided into 10 categories, that is, numbers 0-9.

S02. Set a joint contribution threshold.

The threshold of joint contribution in this embodiment is specified as 0.95

S03, determining the joint contribution degree between the combination of pixels and numbers, specifically including the following steps:

S031. Determine the mutual information between each pixel and the number;

S032. Determine the pixel point that maximizes the mutual information with the number, remove the pixel point from the pixel point set, and add it to the feature subset;

S033. Determine the joint contribution degree between the feature subset and the number.

Wherein, in step S031, the mutual information between each pixel and the number is through the above formula:

$I(x_i;Y)=h\left(x_i\right)+h\left(Y\right)-h(x_i,Y)=\underset{j=1}{\overset{m_i}{\mathrm{\&Sigma;}}}\underset{l=1}{\overset{k}{\mathrm{\&Sigma;}}}p(a_j^i,c_l)\log \frac{p(a_j^i,c_l)}{p\left(a_j^i\right)p\left(c_l\right)}$ to be sure.

There is another step between step S031 and step S032: removing pixels whose mutual information with numbers is smaller than a predetermined value from the set of pixels.

After the mutual information between each pixel and the number is obtained through the above mutual information calculation formula, the mutual information of some pixels is very small, so these pixels can be ignored, and feature extraction is performed on the retained set of pixels, and this It will not have much impact on the correct classification, which can greatly save the time of feature extraction.

In step S033, the joint contribution degree between the feature subset and the number is obtained by the formula:

r _s =I(S;Y)/I(X;Y) to determine.

Among them, r _s represents the joint contribution;

I(S; Y) represents joint mutual information;

I(X;Y) denotes the total joint mutual information.

S04. Obtain the combination of pixels whose joint contribution degree is greater than or equal to the set joint contribution degree threshold, specifically including steps:

Comparing the determined joint contribution with the set joint contribution threshold,

If the determined joint contribution degree is greater than or equal to the set joint contribution degree threshold, then obtain the feature subset;

If the determined joint contribution degree is less than the set joint contribution degree threshold, then for the combination of each pixel point of the pixel point set and the feature subset, determine the pixel point that maximizes the mutual information between the combination and the number , remove the pixel from the pixel set and add it to the feature subset, and then go back to step S033 for execution.

Through this feature extraction method, only 21 pixel points can achieve the expected recognition effect, which greatly improves the recognition efficiency of the card number printed on the IC card.

As shown in Figure 2, the present invention also provides a feature extraction device in pattern recognition, including:

Numerical preprocessing module 10 determines discrete feature variables and class variables according to the model original information of the sample, and preprocesses the feature variables and class variables;

Threshold value setting module 20, used to set the joint contribution degree threshold;

A joint contribution degree determination module 30, used to determine the joint contribution degree of the feature subset set by the numerical preprocessing module and the class variable;

The feature extraction module 40 is configured to obtain a feature subset whose joint contribution is greater than or equal to the set joint contribution threshold according to the joint contribution.

Wherein, the joint contribution degree determination module 30 includes:

A mutual information determination unit 301, configured to determine the mutual information between each feature variable and class variable;

The maximum value determination unit 303 is configured to determine the feature variable that maximizes the mutual information between the feature variable and the class variable according to the mutual information, remove the feature variable from the feature variable set, and add it to a subset of the feature variable set ; For the combination of each feature variable in the feature variable set and the feature subset, determine the feature variable that maximizes the mutual information between the combination and the class variable, remove the feature variable from the feature variable set, and add it to the feature subset;

A joint contribution degree determining unit 304, configured to determine the joint contribution degree of the feature subset and the class variable.

In order to save time for feature extraction, there is a filter unit 302 between the mutual information determination unit and the maximum value determination unit, which is used to remove feature variables whose mutual information with class variables is less than a predetermined value from the feature variable set. In this way, after the mutual information of each symptom and symptom is obtained through the above mutual information calculation formula, the mutual information of some symptoms is very small, so these symptoms can be ignored, and feature extraction is performed on the retained symptom set, and this will not It has too much impact on the correct classification, which can greatly save the time of feature extraction.

Described feature extraction module 40 comprises:

A comparing unit 401, configured to compare the determined joint contribution degree with a set joint contribution degree threshold;

The extraction unit 402 is configured to extract a feature subset whose joint contribution is greater than or equal to a set joint contribution threshold.

If the joint contribution degree determined by the comparison unit 401 is greater than or equal to the set joint contribution degree threshold, the extraction unit 402 will subset the features; if the joint contribution degree determined by the comparison unit 401 is less than the set joint contribution degree threshold, then the mutual information determination unit 301 determines the mutual information of each feature variable of the feature variable set and the combination of the feature subset and the class variable, and the maximum value determination unit 303 determines from it to maximize the mutual information of the combination and the class variable The feature variable is removed from the feature variable set and added to the feature subset; then the joint contribution determination unit 304 determines the joint contribution of the feature subset.

The value range of the joint contribution threshold set by the threshold setting module is generally [0.9, 0.98].

The feature extraction method and device in the pattern recognition of the present invention are mainly aimed at the feature extraction of discrete variables. In the feature extraction method and device, a new form of joint contribution is defined. This feature extraction method based on joint contribution can effectively avoid the subjectivity of the previous feature extraction method specifying the number of selected features in advance, and can Improve the speed of extraction, and can be widely used in the feature extraction of discrete digital image information, fingerprint information, face pattern information, voice information or handwritten/printed character information.

Claims (8)

1. A feature extraction method in pattern recognition is characterized by comprising the following steps:

determining discrete characteristic variables and class variables according to mode original information of the sample, combining all the characteristic variables into a characteristic variable set, and determining possible values of each characteristic variable; determining possible values of the class variables; setting a feature subset, and initializing the feature subset into an empty set;

setting a joint contribution threshold;

determining the joint contribution degree of the feature subset and the class variable;

acquiring a feature subset of which the joint contribution degree is greater than or equal to a set joint contribution degree threshold;

the step of determining the joint contribution degree of the feature subset and the class variable comprises the following steps:

a. determining mutual information between each characteristic variable and each class variable;

b. determining a characteristic variable which enables mutual information between the characteristic variable and the class variable to be maximum, removing the characteristic variable from the characteristic variable set, and adding the characteristic variable into the characteristic subset;

c. determining the joint contribution degree of the feature subset and the class variable;

wherein the joint contribution degree r of the feature subset and the class variable_sThe determination method specifically comprises the following steps:

r_s＝I(S；Y)/I(X；Y)

wherein I (S; Y) represents the joint mutual information of the feature subset and the class variable,

i (X; Y) represents the joint mutual information of all characteristic variables and class variables.

2. The method of extracting features in pattern recognition according to claim 1,

the mode original information is a value corresponding to human body symptoms, the characteristic variable is the human body symptoms, and the class variable is the disease type of the patient; or,

the mode original information is the gray value of pixel points in a digital image on the surface of the integrated circuit card, the characteristic variables are the pixel points of the digital image, and the class variables are digital values.

3. The method of extracting features in pattern recognition according to claim 1, further comprising, between step a and step b, the steps of: and removing the characteristic variables of which mutual information with the class variables is less than a preset value from the characteristic variable set.

4. The method for extracting features in pattern recognition according to claim 3, wherein the joint mutual information of all feature variables and class variables is obtained by sample calculation, and the specific process is as follows:

step 1:

using frequency vector B ═ B₁，B₂，…，B_N)^TN represents the total number of samples in which the values of the characteristic variables are equal;

using frequency matrix D ═ D (D)_ij) The number of samples indicating that the values of the characteristic variables are all equal and the values of the class variables are also equal, i is 1, 2, …, N; j ═ 1, 2, …, k; k represents the number of the values of the class variables;

using frequency vector E ═ E₁，E₂，…，E_k)^TThe number of samples representing equal values of the class variables;

step 2:

initializing parameters: let all the element values of vector B be 1, let all the element values of matrix D and vector E be 0;

and step 3:

frequency used in obtaining the calculated probability:

let i be 1, 2, …, N, j be i +1, i +2, …, N, y_iThe value of the class variable, x, representing the ith sample_iValue representing the i-th sample feature vector, c_lThe l-th value representing a class variable;

if B is present_iIf 0, then the next i loop is executed;

otherwise

If y is_i＝c_lThen E_l＝E_l+1，l＝1，2，…，k；

If x_i＝x_jThen B_i＝B_i+1，B_j＝0；

If x_i＝x_jAnd y_i＝c_lThen D_il＝D_il+1，l＝1，2，…，k；

And 4, step 4:

calculating the joint mutual information of all the characteristic variables and the class variables:

<math><mrow><mi>I</mi><mo>=</mo><munderover><mi>&Sigma;</mi><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>N</mi></munderover><munderover><mi>&Sigma;</mi><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mi>k</mi></munderover><mfrac><msub><mi>D</mi><mi>ij</mi></msub><mi>N</mi></mfrac><mi>log</mi><mrow><mo>(</mo><mfrac><mrow><msub><mi>D</mi><mi>ij</mi></msub><mo>/</mo><mi>N</mi></mrow><mrow><mrow><mo>(</mo><msub><mi>B</mi><mi>i</mi></msub><mo>/</mo><mi>N</mi><mo>)</mo></mrow><mrow><mo>(</mo><msub><mi>E</mi><mi>j</mi></msub><mo>/</mo><mi>N</mi><mo>)</mo></mrow></mrow></mfrac><mo>)</mo></mrow></mrow></math>

wherein, when D_ij×B_i×E_jWhen equal to 0, let log (D)_ij/B_iE_j)＝0。

5. The method according to claim 1, wherein the step of obtaining the combination of the feature variables whose joint contribution degree is greater than or equal to the set joint contribution degree threshold value includes:

comparing the determined joint contribution degree with the set joint contribution degree threshold,

if the determined joint contribution degree is larger than or equal to the set joint contribution degree threshold value, acquiring the feature subset;

and if the determined joint contribution degree is smaller than the set joint contribution degree threshold, determining the characteristic variable which maximizes the mutual information between the combination and the class variable for the combination of each characteristic variable of the characteristic variable set and the characteristic subset, removing the characteristic variable from the characteristic variable set, and adding the characteristic variable into the characteristic subset.

6. A feature extraction device in pattern recognition, characterized by comprising:

the numerical value preprocessing module is used for determining discrete characteristic variables and class variables according to the mode original information of the sample, combining all the characteristic variables into a characteristic variable set and determining the possible value of each characteristic variable; determining possible values of the class variables; setting a feature subset, and initializing the feature subset into an empty set;

the threshold setting module is used for setting a joint contribution degree threshold;

the joint contribution degree determining module is used for determining the joint contribution degree of the feature subset and the class variable;

the characteristic extraction module is used for acquiring a characteristic subset of which the joint contribution degree is greater than or equal to a set joint contribution degree threshold value according to the joint contribution degree;

wherein the joint contribution degree determination module comprises:

the mutual information determining unit is used for determining the mutual information between each characteristic variable and each class variable;

a maximum value determining unit, configured to determine, according to the mutual information, a feature variable that maximizes the mutual information between the feature variable and the class variable, remove the feature variable from the feature variable set, and add the feature variable to the subset of the feature variable set;

the joint contribution degree determining unit is used for determining the joint contribution degree of the feature subset and the class variable; wherein the joint contribution degree r of the feature subset and the class variable_sThe determination method specifically comprises the following steps:

r_s＝I(S；Y)/I(X；Y)

wherein I (S; Y) represents the joint mutual information of the feature subset and the class variable,

i (X; Y) represents the joint mutual information of all characteristic variables and class variables.

7. The apparatus according to claim 6, wherein a filter unit is further provided between the mutual information determining unit and the maximum value determining unit, for removing from the feature variable set the feature variables whose mutual information with the class variable is smaller than a predetermined value.

8. The apparatus of claim 6, wherein the feature extraction module comprises:

a comparison unit for comparing the determined joint contribution degree with a set joint contribution degree threshold;

and the extracting unit is used for extracting the feature subset of which the joint contribution degree is greater than or equal to the set joint contribution degree threshold.

Images