CN110647856B - Method for recognizing facial expressions based on theory of axiomatic fuzzy set - Google Patents

Abstract

The invention discloses an expression recognition method based on different semantics of a commonalization fuzzy set, which specifically comprises the following steps: step 1, constructing a semantic concept set for each expression according to the similarity in the classes and the difference between the classes; step 2, extracting face characteristic information and converting the face characteristic information into a semantic concept; step 3, making the semantic concepts into corresponding rule sets for distinguishing the categories of the expressions; and 4, providing an optimization criterion for selecting the most suitable semantic concept for each expression. The method can objectively convert the human face features into semantic concepts, and the fuzzy set and the logic operation thereof are determined algorithmically according to the distribution of the original data and the semantics of the fuzzy set. By considering ambiguity and randomness, research on how to convert information in the database into membership functions and fuzzy logic operations thereof is facilitated. The present application does not require the definition of membership functions and initial values because all data is obtained from the original database.

Description

Facial expression recognition method based on axiomatic fuzzy set theory

Technical Field

The invention belongs to the technical field of image classification, and particularly relates to a novel method for recognizing various expressions by using human face semantic knowledge based on a commonalization fuzzy set theory.

Background

Facial expression recognition has attracted a great deal of attention in the last decade, and the most basic expressions have been divided into six types. Many methods have been developed from various perspectives, principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA) being classical subspace methods, generally recognized as distinct expression forms of test criteria. They extract image features in the subspace and match the corresponding expressions according to the distance of different expression feature subspaces, but the illumination effect and texture features affect the final result.

With the development of neural networks, deep learning has been proposed and many achievements have been made in the field of facial expression recovery. After using the neural network, a large number of expressive features can be identified in the convolutional layer, and finally abstract features are extracted and used for identifying various expressions. In essence, these representative methods, regardless of whether they calculate global or local features, achieve their goal by calculating pixel values.

With the introduction of geometric features, face components are gradually attracting a great deal of attention in face recognition. And in particular the face coding system (FACS), provides a standard for the shape of facial components. On the basis of this standard, various expressions are described in detail with semantic concepts, including the eye, nose, mouth, and facial contours. Since subjectively defined semantic concepts are prone to errors in concept interpretation, face coding systems (FACS) are often considered as auxiliary information to provide assistance. The fuzzy method can well explain the facial expression by semantic concepts. However, this approach has the disadvantage of containing rich rules. When the expressive features increase, the fuzzy method may generate a large number of redundant rules, which may interfere with the estimation result of the expressive features.

Disclosure of Invention

Aiming at the defects in the prior art, the application provides an expression recognition method (Axiomatic Fuzzy Set) based on different semantics of a common physics and chemistry Fuzzy Set, which is called AFS for short in the following, and membership functions and initial values are not required to be defined and are obtained from an original database.

In order to achieve the purpose, the expression recognition method based on different semantics of the commonalization fuzzy set specifically comprises the following steps:

step 1, constructing a semantic concept set for each expression according to the similarity in the classes and the difference between the classes;

step 2, extracting face characteristic information and converting the face characteristic information into a semantic concept;

step 3, making the semantic concepts into corresponding rule sets for distinguishing the categories of the expressions;

and 4, providing an optimization criterion for selecting the most suitable semantic concept for each expression.

Further, in step 1, a semantic concept set is constructed according to the distribution of expression data, and the method specifically includes the following steps:

step one, A _i ＝Πm _j，k Is a complex concept composed of m _j,k E.g. the connection of the M,

is a set; the human face has S characteristics, wherein j represents the jth characteristic; if the jth feature consists of L attributes, L represents the ith attribute; if the l attribute has K different descriptions, K is the K description; then call to be->

The method is a face simple semantic, wherein M is a face simple semantic set and covers three semantics of large, medium and small;

suppose A _i ＝Πm _j，k The representation consists of two or more simple semantics m _j，k Combined semantics, the letter pi represents A _i All simple semantics m _j，k A combination relationship between "and", and A _i Two simple semantics m of any _j，h And m _j，k The described geometric forms are different, then called A _i A face complex semantic meaning;

thus, the semantic set is described as follows:

here, I is a non-empty index set; at the same time, EM ^* The equivalence relation R above is defined as follows (R represents a real number set);

to pair

There is an equivalence relation

(1)

So that>

(2)

So that>

Denoting quotient set by EM, EM = R;

wherein

Represents: optionally; />

Represents: (ii) present; i: denotes the ith, corresponding to the serial number; h: the h-th feature; j: representing a feature set; j: represents a certain feature in J; ai: representing a subset of semantic concepts; bi: representing a subset of semantic concepts; ak: representing a subset of semantic concepts; />

Means Ai contains or is equal to Bi;

step two, the ordered relation has continuity in the membership function describing the fuzzy set, X is a data set, M is a group of fuzzy terms on X, and the definition is as follows:

wherein the linear ordering relationship is represented by ">", and for M ∈ M, "x >" is defined as _m y "indicates that x belongs to m to a greater extent than y belongs to m; a. The ^＞ (x) In the face feature data set X, X is present ₁ ，x ₂ ∈X，x ₂ The degree of falling under A is less than or equal to x ₁ Degree of belonging to A; a. The ^＞ (x) Is belonged to _m∈A m, which is less than or equal to x; a. The ^＞ (x) The probability distribution of the semantics of the fuzzy set A and the observation data set X is determined;

step three, aiming at the fuzzy set xi E EM, mu _ξ :X→[0,1].{μ _ξ (x) The | ξ ∈ EM } } is called as logic operation rules of 'A' and 'V' about the human face semantic, is a group of coherent membership functions (EM, A) and needs to meet the following conditions:

1. if the semantics are alpha, beta belongs to EM, if alpha is less than or equal to beta (EM, lambda and lambda), then for any X belongs to X, there is mu _α (x)≤μ _β (x) (ii) a Alpha and beta are semantic concepts in the semantic set EM, x is any sample, and mu is membership; mu.s _α (x)≤μ _β (x) The explanation is as follows: the membership degree of any x sample in the semantic concept alpha is less than or equal to the membership degree of any x sample in the semantic concept beta;

2. for X e to X,

if/or>

All I ∈ I have μ _η (x)＝0；

3. If X, y belongs to X,

if>

Then there is mu _η (x)≤μ _η (y); if>

Then mu _η (x) =1; eta represents a complex semantic concept, mu _η (x) And mu _η (y) representing the membership of the sample x and the sample y in the description of the complex semantic eta; />

The coherent membership function is associated with the measure on X, and the application provides two measures of fuzzy sets, which can be constructed by considering the semantic meaning of fuzzy items and the probability distribution of data features;

step four, let (Ω, F, p) be a probability measure space, M be a set of fuzzy terms on X; let ρ be _γ Is a weight function of a fuzzy term gamma epsilon M; is provided with

Is a limited set of observation samples over the probability space (Ω, F, p); if any M belongs to M, x belongs to omega; m is _＞ (x)∈F _m Then, the following conclusions are reached:

A、{μ _ξ (x) The | xi ∈ EM } is a membership function of one of (EM, V), and then for any human face semantic fuzzy set

The membership function based on the semantic structure and the membership function based on the probability distribution are respectively described as follows:

wherein N is _u Is the number of human face semantics observed in the sample set X;

B. for all γ ∈ M, pp _γ (x) Continuous over Ω, X is a set of samples randomly drawn from the probability space (Ω, F, p), and for all X ∈ X, the membership function defined by equation (3) converges to the membership function defined by (4) as X approaches infinity.

Further, in the step 2, the information of the human face characteristic points is extracted according to FACS, the expression characteristics are constructed by marking through a marking method, and the expression characteristics are converted into semantic concepts and are divided into the following steps:

step one, according to FACS, the facial components are known to have different shapes among different expressions; FACS defines only facial action units and appears in the corresponding expression; however, it ignores the geometry of the face component. Therefore, on the basis of FACS, facial components are labeled using a labeling method to construct expressive features.

Step two, the mark point and the symbol l of the human face are expressed by coordinate values _i For representing the ith point in the image, and x _i And y _i Is a 1 _i The coordinate values of (a); point l _i Viewed as "l _i ＝(x _i ,y _i ) ", the expression characteristics are shown in FIG. 1 and Table 1, respectively:

TABLE 1

/>

Step three, are _i ,l _j ,···,l _k The area enclosed by the points; med (l) _i ,l _j ) Represents l _i And l _j The center of (a); for d (l) _i ,l _j )，∠(l _i ,l _j ,l _k ) The explanation of (a) is as follows:

where d (l) _i ,l _j ) Is represented by _i And l _j Angle (l) _i ,l _j ,l _k ) Represents l _i ，l _j ，l _k The combination angle of (c);

and step four, constructing 25 expression features according to the table 1, and reasonably describing facial expressions. Meanwhile, three semantic concepts of large, medium and small are established to distinguishDividing each characteristic; f. of _h For expressing the h-th expressive feature, like m _h,1 ，m _h,2 ，m _h,3 Respectively represent large, medium and small in semantic concepts.

Further, in step 3, a standard is established to distinguish different expressions, and the method comprises the following steps:

step one, simple concept selection

There are many features that are used to describe expressions, and these simple expressive features need to be changed from F = { F, based on similar expressive features ₁ ,f ₂ ,···,f _h ,···,f ₂₅ Get the feature set

Wherein the content of the first and second substances,

is a subset that represents k and x in the structure of the characteristic fs _i The field of (1); />

Is also a subset, which represents k and x in the feature set F _i The field of (1); />

Is a subset of F, including >>

And &>

A relationship characteristic between; retention f _h At f _h And F have more in common>

If->

If the value of (a) is greater, f _h The concept set F can be well represented, and the neighborhood x of k is measured by Euclidean distance _i 。

And step two, optimizing the concept of each expression, and providing a simple semantic concept set according to the algorithm 1, wherein the characteristics are not unique when the expression appears. Therefore, the applicable complex semantic concepts are changed;

step three, extracting complex semantic concepts for the selected expressions by using a threshold and a standard;

step four, because of the difference of individuals, the semantics of each individual can generate difference; a semantic description needs to be constructed for each expression.

And step five, inserting the most preferable semantic concepts among the expressions after obtaining the descriptions of various expressions.

The beneficial effect of this application does: the AFS can objectively convert the human face features into semantic concepts, and in the AFS theory, a fuzzy set (membership function) and its logical operation are algorithmically determined according to the distribution of original data and the semantics of the fuzzy set. The AFS framework facilitates the study of how to convert information in a database into membership functions and their fuzzy logic operations by considering ambiguity (subjective uncertainty) and randomness (subjective uncertainty). AFS does not require the definition of membership functions and initial values because all data is obtained from the original database.

Drawings

FIG. 1 is a representation of facial expression features of the present invention;

FIG. 2 is a diagram of semantic concept distribution within various categories in accordance with the present invention;

Detailed Description

In step 3, a simple concept selection, features describing facial expressions are obtained. Optimization criteria are then established to reveal differences between different expressions. Due to the similarity of the internal expressions, mutual features should be selected. With these differences, criteria are established to distinguish different expressions. The following two steps will be described in detail.

At the time of obtaining each sample x _i After the compact features of (2), converting the features into semantic concepts. Within the framework of AFS, there are three concepts of "big", "medium" and "small" for any feature. In fact, only one concept converted from the features is needed to represent the corresponding expression. Thus, a criterion has been established to select x _i The semantic concept of (2).

Where M is _xi Is to preserve the semantic concept set, m _h,t Denotes f _h The tth feature of the tth concept.

Represents m _h,t The neighborhood k of (c). />

Is a set of semantic concepts m _h,1 ,m _h,2 ,m _h,3 Neighborhood k, the distance between concepts still uses the euclidean distance. Then m is reserved _h,t They have the most similar neighborhood->

Is also based on>

Is also the largest, x can be well described _i Algorithm 1 describes this process. />

In step (b)Step 3. X according to algorithm 1 in the complex semantic extraction _i Proposes a simple semantic concept set

In this step, the threshold and criteria are used to be x _i Selecting a proper expression concept.

Here, the

Is a set of semantic concepts. />

Is formed by m _h,t A connected set of complex semantic concepts, δ being a threshold value, based on a criterion>

Use>

To represent x _i Is based on the degree of membership when>

If the degree of membership is greater than or equal to δ, it is reserved>

However, semantic concepts can only represent an individual x _i Many redundancy concepts still exist. Based on the similarity within a class and the difference between classes, construct

And &>

Two criteria to optimize x _i Semantic concepts ofAnd (4) collecting. />

And &>

Respectively as follows:

/>

herein, the

Is about a semantic concept>

All of the described->

Average value of the degree of membership of (c). Also, the same applies to

Is about the semantic concept->

All x of the description _i E.g. the average value of the membership of X. In addition, the method can be used for producing a composite material

Representing a semantic concept->

The distance measure of (d). And then use the sum of>

And &>

Performed>

To extract complex concepts.

According to

One set corresponding to each x can be obtained _i Is taken as the value of>

Describe x _i Its value is->

The largest among them.

In step 3 semantic concept description of each expression, each x is different due to individual differences _i Semantics of (A)

A difference is generated. A semantic description needs to be constructed for each expression. As can be seen in fig. 2, the three color representation uses three data to illustrate the conceptual distribution. The symbol "·" denotes an individual x _i The semantic concept of (2). The symbol a is the central concept for each expression. According to

Can obtain each x _i Semantic concept ∈ X>

Each class can find a central concept that can reflect the differences between different classes very well. The concept of each expression is changed using Sparse Representation (SR).Conversion of SR solution to l according to complexity ₁ To minimize the problems.

Here, T is a coefficient matrix. H is a central semantic concept which is a set of clustered concepts

And (4) generating. Fuzzy c-means (FCM) is used to ensure the correctness of H. Extracting a central semantic concept pick>

Plays an important role in most conventional expressions. Finally, use l ₁ Miniaturisation results in an optimal concept set->

By using l ₁ And minimizing to obtain a coefficient matrix of the semantic concept. From the coefficient matrix, a semantic concept can be derived, its value pairs

Is important.

In step 3, in the optimal concept extraction of various expressions, after descriptions of the various expressions are obtained, significant semantic concepts are inserted among the expressions. To extract salient concepts, semantic closeness is established

To evaluate a set of semantic concepts

Herein, the

Is->

Number of semantic concepts in->

Represents->

Describe down->

Degree of membership. Then a criterion is established>

Its resolution is similar to->

(equation 10). Finally, establish->

And &>

Combined optimization function>

Extract description C _i The best semantic concept for each expression. />

For predicting new samples not included in X in step 3Class labels, required to be in the entire input space

Represents a semantic concept set xi, where U _s Is the semantic concept term M in M _h,k An associated feature. Given the distribution of the data set, the membership functions of the semantic concept set γ can be specified from the observed data. For any semantic concept ξ ∈ EM, the range X in which it is discussed is extended to U ₁ ×U ₂ ×…×U _S . For each x = (u) ¹ ,u ² ,…,u ^s )∈U ₁ ×U ₂ ×…×U _S ，/>

At U ₁ ×U ₂ ×…×U _S The lower bound of the membership function above which defines the semantic concept set ξ is as follows:

here, the

I ∈ I is defined >>

Weighing and collecting device>

The lower bound of the membership function for ξ. With the aid of the formula (16), the fuzzy set can be expanded>

The range X discussed becomes U ₁ ×U ₂ ×…×U _S . Thus, using the fuzzy rule base, the input space can be established as U ₁ ×U ₂ ×…×U _S The fuzzy inference system of (1). Membership function->

Depending on the distribution of the training instances and the AFS fuzzy logic. When a new mode x ∈ U ₁ ×U ₂ ×…×U _S When provided with a label of unknown class, its degree of membership may be calculated>

Wherein +>

Is to C _i Fuzzy description of classes. If it is

Then x belongs to C _q And (4) class. />

Claims (3)

1. A expression recognition method based on different semantics of a common physicochemical fuzzy set is characterized by comprising the following steps:

step 1, constructing a semantic concept set for each expression according to the similarity in the classes and the difference between the classes;

step 2, extracting face characteristic information and converting the face characteristic information into a semantic concept;

step 3, making the semantic concepts into corresponding rule sets for distinguishing the categories of the expressions;

step 4, providing an optimization criterion for selecting the most suitable semantic concept for each expression;

in the step 1, a semantic concept set is constructed according to the distribution condition of the expression data, and the method specifically comprises the following steps:

step one, A _i ＝Πm _j,k Is a complex concept composed of m _j,k E.g. the connection of the M,

is a set; the human face has S characteristics, wherein j represents the jth characteristic; if the jth feature consists of L attributes, L represents the ith attribute; if the l attribute has K different descriptions, K is the K description; then call to be->

The method is a face simple semantic, wherein M is a face simple semantic set and covers three semantics of large, medium and small;

suppose A _i ＝Πm _j,k The representation consists of two or more simple semantics m _j,k Combined semantics, the letter pi represents A _i All simple semantics m _j,k A combination relationship between "and", and A _i Two simple semantics m of any _j,h And m _j,k The described geometric forms are different, then called A _i A face complex semantic meaning;

thus, the semantic set is described as follows:

here, I is a non-empty index set; at the same time, EM ^* The equivalence relation R above is defined as follows;

to pair

There is an equivalence relation

(1)

So that->

(2)

So that>

Denoting quotient set by EM, EM = R;

wherein

Represents: optionally; />

Represents: (ii) present; i: denotes the ith, corresponding to the serial number; h: the h-th feature; j: representing a feature set; j: represents a certain feature in J; ai: representing a subset of semantic concepts; bi: representing a subset of semantic concepts; ak: representing a subset of semantic concepts; />

Means Ai contains or is equal to Bi;

step two, the ordered relation has continuity in the membership function describing the fuzzy set, X is a data set, M is a group of fuzzy terms on X, and the definition is as follows:

wherein the linear ordering relationship is represented by ">", and for M ∈ M, "x >" is defined as _m y "indicates that x belongs to m to a greater extent than y belongs to m; a. The ^＞ (x) In the face feature data set X, X is present ₁ ，x ₂ ∈X，x ₂ The degree of falling under A is less than or equal to x ₁ Degree of belonging to A; a. The ^＞ (x) Is attributed to II _m∈A A series of elements of m which are less than or equal to x; a. The ^＞ (x) The semantic meaning of the fuzzy set A and the probability distribution of the observation data set X are determined;

step three, regarding the fuzzy set xi E EM, mu _ξ :X——→[0,1]·{μ _ξ (x) The | ξ ∈ EM } is called as the logical operation rules of 'inverted V' and 'V' about the human face semantic meaning, is a group of coherent membership functions (EM, inverted V, V-shaped) and needs to meet the following conditions:

1. if the semantic meaning is alpha, beta belongs to EM, if alpha is less than or equal to beta (EM, V), then for any X belongs to X, there is mu _α (x)≤μ _β (x) (ii) a Alpha and beta are semantic concepts in the semantic set EM, x is any sample, and mu is membership; mu.s _α (x)≤μ _β (x) The explanation is as follows: the membership degree of any x sample in the semantic concept alpha is less than or equal to the membership degree of any x sample in the semantic concept beta;

2. for X e to X,

if/or>

All I ∈ I have μ _η (x)＝0；

3. If there is

If>

Then there is mu _η (x)≤μ _η (y); if/or>

Then mu _η (x) =1; eta represents a complex semantic concept, mu _η (x) And mu _η (y) representing the degree of membership of the sample x and the sample y in the complex semantic eta description;

step four, setting (omega, F, P) as a probability measure space, and setting M as a group of fuzzy items on X; let ρ be _γ Is a weight function of a fuzzy term gamma epsilon M; is provided with

Is a limited set of observation samples over the probability space (Ω, F, P); if any M belongs to M, x belongs to omega; m is _＞ (x)∈F _m Then, the following conclusions are reached:

A、{μ _ξ (x) The | xi ∈ EM } is a membership function of one of (EM, V), and then for any human face semantic fuzzy set

The membership function based on the semantic structure and the membership function based on the probability distribution are respectively described as follows:

wherein N is _u Is the number of human face semantics observed in the sample set X;

B. for all γ ∈ M, P _γ (x) Is continuous over Ω and X is a set of samples randomly drawn from the probability space (Ω, F, P), and for all X ∈ X, the membership function defined by equation (3) converges to the membership function defined by (4) as X approaches infinity.

2. The method for recognizing the expressions based on different semantics of the axiomatic fuzzy set as claimed in claim 1, wherein the step 2 comprises extracting facial feature point information according to FACS, constructing the expression features by labeling with a labeling method, and converting into semantic concepts comprising the following steps:

step one, according to FACS, knowing that the shapes of facial components are different among different expressions; FACS defines only facial action units and appears in the corresponding expression;

step two, the mark point and the symbol l of the human face are expressed by coordinate values _i For representing the ith point in the image, and x _i And y _i Is a _i The coordinate values of (a); point l _i Viewed as "l _i ＝(x _i ,y _i ) ", the expression characteristics are shown in Table 1, respectively:

TABLE 1

/>

Step three, are _i ,l _j ,···,l _k The area enclosed by the points; med (l) _i ,l _j ) Represents l _i And l _j The center of (a); for d (l) _i ,l _j )，∠(l _i ,l _j ,l _k ) The explanation of (a) is as follows:

where d (l) _i ,l _j ) Is represented by _i And l _j Angle (l) _i ,l _j ,l _k ) Represents l _i ，l _j ，l _k The combined angle of (a);

step four, according to the table 1, 25 expression features are constructed, and meanwhile, three semantic concepts of large, medium and small are established to distinguish each feature; f. of _h For the h-th expressive feature, m _h,1 ，m _h,2 ，m _h,3 Respectively represent large, medium and small in semantic concepts.

3. The expression recognition method based on different semantics of the axiomatic fuzzy set according to claim 1, wherein a standard is established in step 3 to distinguish different expressions, and the method comprises the following steps:

step one, simple concept selection

From F = { F ₁ ,f ₂ ,···,f _h ,···,f ₂₅ Extracting feature set in

Wherein the content of the first and second substances,

is a subset which represents k and is at feature f _h X in the structure of _i The field of (1); />

Is also a subset, which represents k and x in the feature set F _i The field of (1); />

Is a subset of F, including >>

And &>

A relationship characteristic between; retention f _h At f _h And F have more neighborhoods in common>

If a neighborhood->

If the value of (a) is greater, f _h The concept set F can be well represented, and the neighborhood x of k is measured by Euclidean distance _i ；

Changing the applicable complex semantic concept;

step three, extracting complex semantic concepts for the selected expressions by using a threshold and a standard;

step four, because of the difference of individuals, the semantics of each individual can generate difference; a semantic description needs to be constructed for each expression;

and step five, inserting the most preferable semantic concepts among the expressions after obtaining the descriptions of various expressions.

Images