CN113095270A - Unsupervised cross-library micro-expression identification method - Google Patents

Unsupervised cross-library micro-expression identification method Download PDF

Info

Publication number
CN113095270A
CN113095270A CN202110439454.9A CN202110439454A CN113095270A CN 113095270 A CN113095270 A CN 113095270A CN 202110439454 A CN202110439454 A CN 202110439454A CN 113095270 A CN113095270 A CN 113095270A
Authority
CN
China
Prior art keywords
expression
micro
formula
matrix
data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110439454.9A
Other languages
Chinese (zh)
Other versions
CN113095270B (en
Inventor
贲晛烨
李冰
陈雷
肖瑞雪
李玉军
刘畅
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shandong University
Original Assignee
Shandong University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shandong University filed Critical Shandong University
Priority to CN202110439454.9A priority Critical patent/CN113095270B/en
Publication of CN113095270A publication Critical patent/CN113095270A/en
Application granted granted Critical
Publication of CN113095270B publication Critical patent/CN113095270B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168—Feature extraction; Face representation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/174—Facial expression recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
    • G06V10/467—Encoded features or binary features, e.g. local binary patterns [LBP]

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Multimedia (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Human Computer Interaction (AREA)
  • General Engineering & Computer Science (AREA)
  • Biomedical Technology (AREA)
  • Computational Linguistics (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Biophysics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Image Analysis (AREA)

Abstract

The invention relates to an unsupervised cross-library micro-expression recognition method, which comprises the following steps: firstly, aligning and recombining macro and micro expression data of a source domain, and selecting an optimal subset which is more closely associated with a target domain as an auxiliary set by a recombined combined data matrix through a source domain selection model. Then, the conditional distribution and the edge distribution of the source domain and the target domain are dynamically matched through an adaptive distribution alignment model. Finally, using L2,1And the norm carries out the reweighting of the auxiliary set sample, reduces the influence of an abnormal value and realizes the micro expression recognition. The invention adopts a transfer learning mode, takes the macro expression sample with great similarity to the micro expression as an auxiliary, and passes through the labeled macro expression database and the labeled macro expression databaseThe micro-expression database carries out unsupervised cross-database micro-expression recognition on another micro-expression database without any label, reduces time-consuming and labor-consuming manual labeling work of the target domain micro-expression database, and improves the micro-expression recognition effect.

Description

Unsupervised cross-library micro-expression identification method
Technical Field
The invention relates to an unsupervised cross-library micro-expression recognition method, and belongs to the technical field of pattern recognition and machine learning.
Background
As a typical non-verbal communication, facial expressions play an important role in human emotion analysis. When one tries to suppress the real facial expression, micro-expressions will ensue. Micro-expressions are unconscious, rapid changes in facial expression lasting from 0.065 seconds to 0.5 seconds, cannot be freely controlled like macro-expressions, and can often reveal the true emotion that a person wants to hide. Therefore, the micro expression recognition has great application value and development prospect in the fields of crime detection, lie detection and the like.
Currently, micro-expression recognition generally includes three methods: artificial feature based methods, deep learning based methods, and transfer learning based methods. In the micro-expression recognition algorithm based on artificial features, the artificial features are mainly classified into three types: local binary pattern based features, frequency domain based features, and optical flow based features. The feature complexity based on the local binary pattern is low, and the feature has gray scale invariance and rotation invariance. A series of frequency spectrum information such as micro expression phase and amplitude is obtained through Gabor transformation or Fourier transformation and the like based on the frequency domain characteristics, and the frequency domain characteristics of micro expressions are represented through the information. Capturing the micro motion of the face based on the characteristics of the optical flow, and extracting the motion characteristics of the micro expression in the optical flow field. Unlike the manual extraction-based feature descriptors, the deep learning method can automatically deduce the optimal feature representation, and more advanced recognition results, such as the motor unit-assisted attention convolution network AU-GACN, the dual-time scale convolution neural network DTSCN and the like, are obtained in recent years. However, deep learning requires a large number of training samples, existing micro-expression data sets are small, and labeling data is a tedious and expensive task, so that the training effect of the deep learning model is limited under the condition of insufficient data. To solve the problem of limited data sets, the use of migration learning methods from other related fields is also beginning to be appreciated. The transfer learning is a method for transferring knowledge in one field (namely, a source field) to another field (namely, a target field) so that the target field can obtain a better learning effect, and in micro expression recognition, common related fields are macro expression samples, voice information and the like.
The existing micro expression recognition method only aims at the condition that micro expression samples of a source domain and a target domain come from the same micro expression database. In practical applications, however, micro-expression recognition faces many complex scenarios (similar to the case where the source domain training set and the target domain testing set come from different micro-expression databases), for example, samples of the source domain and the target domain are taken from different lighting backgrounds (natural light and indoor lighting), camera types (high definition camera and near infrared camera), and times (day and night). Under the condition of cross-library, the difference of the characteristics of the source domain and the target domain is large, samples of the source domain are directly put into a classifier for training, and the identification effect on the samples of the target domain is very limited.
Disclosure of Invention
Aiming at the condition that the target domain micro expression database has no label, the invention provides an unsupervised cross-database micro expression recognition algorithm, so that the time-consuming and labor-consuming manual labeling work of the target domain micro expression database is reduced, and the micro expression recognition effect is improved.
Summary of the invention:
an unsupervised cross-library micro-expression recognition method comprises the following steps: feature extraction and processing, a source domain selection model and an adaptive distribution alignment model.
According to the invention, a transfer learning mode is adopted, a macro expression sample with great similarity to a micro expression is used as an auxiliary, and through a labeled macro expression database and a micro expression database, another micro expression database without any label is subjected to unsupervised cross-database micro expression recognition, so that the time-consuming and labor-consuming manual labeling work of a target domain micro expression database is reduced, and the micro expression recognition effect is improved.
The invention comprises two parts of a source domain selection model and an adaptive distribution alignment model. Firstly, macro and micro expression data of a source domain are aligned and recombined, and an optimal subset which is closely associated with a target domain is selected as an auxiliary set through a source domain selection model by a recombined combined data matrix, so that the migration effect is improved. Then, the conditional distribution and the edge distribution of the source domain and the target domain are dynamically matched through an adaptive distribution alignment model. At the same time, by L2,1And the norm carries out the reweighting of the auxiliary set sample, reduces the influence of an abnormal value and realizes the micro expression recognition.
Interpretation of terms:
1. the LBP characteristic refers to a Local Binary Pattern, is called Local Binary Pattern in English, is a texture characteristic operator, and has the obvious advantages of gray scale invariance, rotation invariance and the like.
2. The LBP-TOP feature refers to a Local Binary pattern in Three Orthogonal Planes, which is called Local Binary Patterns on Three Orthogonal Planes in English, and consists of LBP features in Three Orthogonal Planes of XY, XT and YT.
3. The multi-scale LBP-TOP feature refers to that for micro-expression video sequences, the face region is divided into four grid types of 1 × 1,2 × 2, 4 × 4 and 8 × 8, and the grid types are divided into 85 local sub-regions. Compared with single-scale or simple block LBP-TOP features, the multi-scale LBP-TOP features can better capture emotion information related to micro-expression emotion recognition.
4. The MDMO feature refers to a Main direction Mean Optical Flow feature, is called as Main Directional Mean Optical Flow in English, is an Optical Flow operator feature, and describes the motion situation of pixels by dividing 36 human face interested areas. The MDMO feature works well for small facial changes.
5. Feature Selection, also called Feature Subset Selection (FSS). The method is a process of selecting some most effective features from original features to reduce the dimensionality of a data set, is an important means for improving the performance of a learning algorithm, and is also a key data preprocessing step in pattern recognition. For a learning algorithm, good learning samples are the key to training the model.
6. Data normalization, which means to uniformly normalize the feature vector of each sample in order to eliminate the dimensional difference between different samples, and map the value of the original feature to [0,1 ] by linear transformation]The method of (1). The conversion formula of the linear transformation is
Figure BDA0003034464160000031
Wherein x ismaxIs the maximum value of the feature vector x, xminIs the minimum value of the eigenvector x;
7. data alignment, namely obtaining new features with uniform feature dimensions and dimensions through feature selection and data standardization, namely completing data alignment;
8. principal Component Analysis, which is called Principal Component Analysis (PCA) in English, is a commonly used feature selection method. PCA transforms a set of variables that may be correlated into a set of linearly uncorrelated variables by orthogonal transformation and selects fewer dimensions from them to reflect as much variable information as possible, if necessary.
9. The regenerated nuclear Hilbert Space, which is called Reproduced Kernel Hilbert Space for short RKHS, is a regenerative Hilbert Space. Hilbert space refers to a perfect inner product space, by "perfect" it is meant that the inner array limits are convergent. The reproducibility means that a symmetric positive definite kernel function K can regenerate the inner product of two functions, so that the inner product in a high-dimensional feature space can not be directly calculated, and the calculation amount is reduced.
10. The Maximum Mean difference, which is called Maximum Mean variance in english, MMD for short, is a measurement criterion for estimating the distribution difference between two sets of data sets, and is widely applied to the existing transfer learning method. The same or similar distribution of the two data sets exists if the MMD distance of the two data sets in their RKHS space is already as small as possible. In essence, MMD is used to measure the difference between the mean values of the source and target domains after kernel mapping Φ () maps the data into the regenerated kernel hilbert space.
11. L of matrix A2,1Norm regularization constraint:
Figure BDA0003034464160000032
in the formula, n is the number of rows and t is the number of columns. L of matrix A2,1The norm is the sum of 2 norms of each row vector of the matrix a.
The technical scheme of the invention is as follows:
an unsupervised cross-library micro-expression recognition method comprises the following steps:
1) preparing training samples, wherein the training samples comprise a macro expression sample from a macro expression database and a micro expression sample from a micro expression database; obtaining a macro expression sample, and extracting characteristics of the macro expression to form a macro expression data matrix
Figure BDA0003034464160000033
Wherein d is1Representing a characteristic dimension of the macro expression, N1Representing the number of the macro expression samples; obtaining a micro-expression sample, and extracting characteristics of the micro-expression to form a micro-expression data matrix
Figure BDA0003034464160000034
Wherein d is2Representing a characteristic dimension of the micro-expression, N2Representing the number of micro expression samples;
2) after the macro expression data matrix and the micro expression data matrix are subjected to data alignment through feature selection and data standardization, the macro expression data matrix and the micro expression data matrix are recombined into an original combined data matrix
Figure BDA0003034464160000035
Wherein N is the number of source domain samples, and N is N1+N2；
3) Selecting a group of samples with the minimum difference with the characteristic distribution of a target domain from an original joint data matrix to form an auxiliary set based on a source domain selection model;
4) quantitatively considering the edge distribution and the strip in each task based on the self-adaptive distribution alignment modelThe relative importance of the piece distribution, the weight of the edge distribution and the condition distribution in the migration process are adaptively adjusted and pass through L2,1The row sparse characteristic of the norm is used for finishing the reweighting of the auxiliary set samples and weakening the influence of an abnormal value; and classifying and identifying the micro-expression database samples without labels in the target domain through a nearest neighbor classifier based on Euclidean distance.
According to the invention, in the step 1), the feature extracted for the macro expression is an LBP feature, and when the LBP feature of the macro expression is extracted, the feature extracted for the micro expression is a multi-scale LBP-TOP feature and an MDMO feature by adopting a multi-scale LBP-TOP feature blocking mode in a face blocking mode of the macro expression.
Preferably, in step 2), feature selection is performed on the macro expression data matrix and the micro expression data matrix respectively through a Principal Component Analysis (PCA), the macro expression data matrix and the micro expression data matrix are unified into the same dimension d, and d is not less than d1And d is not more than d2(ii) a Respectively normalizing the macro expression data matrix and the micro expression data matrix to [0,1 ] through data standardization]In between, data alignment is completed.
Preferably, in step 2), the LBP characteristic dimension of the macro expression is 5015 dimension, the multi-scale LBP-TOP characteristic dimension of the micro expression is 15045 dimension, and the MDMO characteristic dimension of the micro expression is 72 dimension, and the LBP characteristic of the macro expression, the multi-scale LBP-TOP characteristic of the micro expression and the MDMO characteristic of the micro expression are unified into the same dimension by adopting a principal component analysis method. When the micro expression characteristic is a multi-scale LBP-TOP characteristic, the dimension d of the aligned original joint data matrix is 5015; and when the micro expression characteristic is an MDMO characteristic, the dimension d of the aligned original joint data matrix is 72.
Preferably, in step 3), based on the source domain selection model, an optimal subset that is more closely related to the target domain is selected from the original joint data matrix as an auxiliary set sample, and the implementation steps are as follows:
the mathematical description of the source domain selection-based model is as shown in formula (i):
Figure BDA0003034464160000041
in the formula (I), the compound is shown in the specification,
Figure BDA0003034464160000042
n samples representing the source domain,
Figure BDA0003034464160000043
m samples representing a target domain;
Figure BDA0003034464160000044
mapping the source domain data and the target domain data to a regenerated kernel Hilbert space by using a regenerated kernel Hilbert space and a kernel mapping operator phi (.); alpha is alphaiTwo-valued selection factor, alpha, for selecting a model for the source domain i1 denotes that sample i in the source domain is selected into the auxiliary set, α i0 denotes that sample i in the source domain is not selected into the auxiliary set;
introducing a selection parameter betai，
Figure BDA0003034464160000051
Simplifying formula (I), converting formula (I) into formula (II):
Figure BDA0003034464160000052
unfolding formula (II) by kernel function skill to obtain formula (III):
Figure BDA0003034464160000053
in the formula (III), Ko,o＝Φ(xo)TΦ(xo)，Ko,t＝Φ(xo)TΦ(xt)，1oAnd 1tA column vector of all 1's; because a part of samples in the source domain do not satisfy the condition of forming the auxiliary set, the selection factor alpha corresponding to the part of samplesiIs 0, so the selection parameter β is theoretically sparse. Sparse beta pass minimizationChange L of it1Norm | | | beta | | | non conducting phosphor1＝1TBeta is obtained, and lambda is a penalty parameter, then the formula (III) is finished to obtain the formula (IV):
Figure BDA0003034464160000054
for β, the minimization problem is a standard quadratic programming problem that can be solved by a number of efficient methods. And recovering the value of alpha after obtaining the beta, and selecting a sample from the source domain according to the alpha through a preset threshold or a preset proportion to form an auxiliary set.
Further preferably, λ is 1.
Further preferred, in particular, due to alpha i1 denotes that sample i in the source domain is selected into the auxiliary set, α i0 indicates that the sample i in the source domain is not selected into the auxiliary set, and in the present invention, the obtained α is sorted from large to small in value, and the first 70% of α is selectediAnd forming an auxiliary set by the corresponding ith source domain sample. Completing the source domain selection algorithm.
Preferably, in step 4), the alignment model based on adaptive distribution is performed
Figure BDA0003034464160000055
Is represented by formula (V):
Figure BDA0003034464160000056
in the formula (V), the compound is shown in the specification,
Figure BDA0003034464160000061
in order to be the auxiliary set, the method comprises the following steps of,
Figure BDA0003034464160000062
testing a set for the target domain;
Figure BDA0003034464160000063
after passing through the source domain selection algorithm of step 3),n is the number of samples of the auxiliary set formed after N samples of the source domain are subjected to a source domain selection algorithm;
Figure BDA0003034464160000064
the target domain data set is not provided with a label, and m is the number of target domain samples; c is in different micro expression label categories;
Figure BDA0003034464160000065
and
Figure BDA0003034464160000066
respectively representing the samples belonging to class c in the auxiliary set and the target domain test set,
Figure BDA0003034464160000067
is xsThe real label of (a) is,
Figure BDA0003034464160000068
is xtThe pseudo tag of (1);
Figure BDA0003034464160000069
and
Figure BDA00030344641600000610
respectively represent
Figure BDA00030344641600000611
And
Figure BDA00030344641600000612
the number of samples;
first term of formula (V)
Figure BDA00030344641600000613
Is the edge distribution distance, the second term
Figure BDA00030344641600000614
Is the conditional distribution distance, mu is the adaptive parameter of the edge distribution and the conditional distribution, and mu belongs to [0,1 ]](ii) a When mu is→ 1 time source domain
Figure BDA00030344641600000615
And a target domain
Figure BDA00030344641600000616
The data sets are similar, and the condition distribution needs to be adjusted; when μ → 0, the source domain
Figure BDA00030344641600000617
And a target domain
Figure BDA00030344641600000618
The data sets themselves have larger gaps, and the edge distribution needs to be adjusted more. Thus, the adaptive parameter μ can adaptively adjust the importance of each distribution, resulting in better results.
Approximating RKHS with mapping matrix A, formula (V) is converted to formula (VI):
Figure BDA00030344641600000619
using matrix techniques and regularization, equation (vi) is empirically estimated as (vii):
Figure BDA00030344641600000620
in the formula (VII), X is XsAnd xtA composed joint data matrix, X ═ Xs xt]；||.||FIs the F norm, λ is the regularization parameter of the F norm;
Figure BDA00030344641600000621
is an identity matrix, H is a central matrix,
Figure BDA00030344641600000622
Figure BDA00030344641600000623
the item is used forOn the basis of keeping the original characteristic structure, the complexity of the self-adaptive distribution alignment model is reduced; formula (vii) involves two constraints: the first constraint is used to ensure the transformed data ATX maintains the data structure of the original data, i.e. maintains the divergence of the data, for matrix X, its divergence matrix is XHXT(ii) a The second constraint is the range of the adaptive parameter μ; m0And McThe MMD matrix is constructed in a way shown in formulas (VIII) and (IX) respectively:
Figure BDA0003034464160000071
Figure BDA0003034464160000072
further preferably, in step 4), the values of the adaptive parameter μ are adapted using the global and local structure of the field
Figure BDA0003034464160000073
Determining the distance;
Figure BDA0003034464160000074
the distance is an error of a binary linear classifier (e.g., a support vector machine, a logistic regression model, etc.) for distinguishing two domains, and is defined as shown in formula (x):
Figure BDA0003034464160000075
in the formula (X), ε (h) is a distinction
Figure BDA0003034464160000076
And
Figure BDA0003034464160000077
average error of linear classifier h of (1); thus, the edge distribution dMIs/are as follows
Figure BDA0003034464160000078
The distance can be calculated directly. For distribution of conditions
Figure BDA0003034464160000079
Distance, by dcA value representing the class c is selected,
Figure BDA00030344641600000710
Figure BDA00030344641600000711
are respectively shown in
Figure BDA00030344641600000712
And
Figure BDA00030344641600000713
the sample of class c in (1); in summary, μ is estimated as formula (XI):
Figure BDA00030344641600000714
the adaptation parameter μ has to be recalculated in each iteration of the adaptive distribution alignment, since the prediction tag ysChanges may occur after each iteration, so dcWill change in each iteration and the distribution of changes in this characteristic will cause μ to change as well.
Further preferably, in step 4), since the source domain is an auxiliary set formed by selecting a model from the source domain and includes data information of a plurality of databases, L is introduced2,1The norm measures the specific contribution of each sample in the auxiliary set, and the influence of an abnormal value is reduced;
L2,1norm being L of the row vector2Sum of norm, minimizing L of matrix2,1The norm is such that L for each row of the matrix2The norm is as small as possible and as many 0 elements are present within a row as possible, i.e., as sparse as possible. Since each row of the transfer matrix A corresponds to one sample instance, essentially the row sparsity controls the mapping matrix for the auxiliary setAnd the feature selection degree of the sample promotes the self-adaptive re-weighting of the sample. Finally, the objective function of the adaptive distribution alignment model is shown in formula (xii):
Figure BDA0003034464160000081
preferably, in step 4), the target function formula (xi) of the adaptive distribution alignment model is a matrix decomposition problem with constraint, so that the target function formula (xi) of the adaptive distribution alignment model can be optimized by a lagrange multiplier-based method, and the method includes the following steps:
first, the objective function is transformed to obtain formula (XIII):
Figure BDA0003034464160000082
in the formula (XIII), U is a Lagrange multiplier; thus, the augmented Lagrangian equation is shown in equation (XIV):
Figure BDA0003034464160000083
note that μ is also updated at each iteration, so in an iteration, the substituted μ is the μ updated with formula (XI)(k)(ii) a In the (k +1) th iteration, the variable updating process is shown as the formula (XV):
Figure BDA0003034464160000084
in the formula (XV), Z(k)＝Diag(z(k))，
Figure BDA0003034464160000085
i is the ith element of the matrix z,
Figure BDA0003034464160000086
is a minimum valueAvoid the case where the denominator of the above formula is 0 during the update process (in the calculation, we use the tiny positive number eps in Matlab, eps is 2.2204e-16)，
Figure BDA0003034464160000087
Represents A(k)Row i of (1); the invention sets the number of iterations to 18, usually 15 iterations after the objective function has converged, and 18 are set for insurance purposes.
Zero is derived for A, and the optimization problem of formula (XV) is converted into the solving problem of the following formula characteristic decomposition, as shown in formula (XVI):
Figure BDA0003034464160000088
and solving the formula (XVI) to obtain the eigenvectors corresponding to the minimum K eigenvalues forming the mapping matrix A.
Preferably, in step 4), the target domain unlabeled micro-expression database samples are classified and identified by the nearest neighbor classifier based on the euclidean distance, and the specific implementation steps are as follows:
the method comprises the steps of obtaining a trained mapping matrix A by solving the linear optimization problem of an objective function, projecting auxiliary set data and test set data into a public subspace by adopting the mapping matrix A, and comparing and identifying data representation in the public subspace in a nearest neighbor mode, namely:
find a projection matrix and order
Figure BDA0003034464160000091
Representing all of the training data in the common subspace,
Figure BDA0003034464160000092
for the auxiliary set features and their corresponding labels obtained after the source domain selection algorithm,
Figure BDA0003034464160000093
for unlabeled target domain data sets, AsAnd AtCorresponding mapping momentsThe front n rows and the rear m rows of the array A;
for a given target domain micro-expression data set, any one test sample after projection is used
Figure BDA0003034464160000094
Nearest neighbor comparison based on Euclidean distance
Figure BDA0003034464160000095
Figure BDA0003034464160000096
Refer to the ith source domain auxiliary set sample
Figure BDA0003034464160000097
The micro-expression type label of the training sample closest to the micro-expression type label is assigned to the test sample to finish micro-expression identification.
The invention has the beneficial effects that:
the invention provides an unsupervised cross-library micro-expression identification method. According to the micro-expression identification method, the micro-expression identification task of the micro-expression database of the target domain is completed through the macro-expression sample which has the great similarity with the micro-expression, the source domain selection model and the self-adaptive distribution alignment model and the macro-expression database and the micro-expression database under the condition that the target domain has no label, so that the manual marking work is reduced, and the accuracy of micro-expression identification is practically improved.
Drawings
FIG. 1 is a schematic flow chart of an unsupervised cross-library micro-expression recognition method according to the present invention;
FIG. 2a1 is a sample diagram of positive expression of the CK + database in the present invention;
FIG. 2a2 is a sample diagram of negative expressions in the CK + database of the present invention;
FIG. 2a3 is a sample diagram of surprising expression of the CK + database in the present invention;
FIG. 2b1 is a sample diagram of positive expressions in the MMEW macroexpression database of the present invention;
FIG. 2b2 is a sample diagram of the MMEW macro expression database negative expressions in the present invention;
FIG. 2b3 is a sample schematic diagram of the surprising expression of the MMEW macro expression database in the present invention;
FIG. 2c1 is a sample diagram of positive expression in the MMEW micro-expression database of the present invention;
FIG. 2c2 is a sample diagram of negative expressions in the MMEW micro-expression database of the present invention;
FIG. 2c3 is a sample schematic diagram of the surprising expression of the MMEW micro-expression database in the present invention;
FIG. 2d1 is a sample diagram of positive expressions in the CASMEII database according to the present invention;
FIG. 2d2 is a sample diagram of negative expressions in the CASMEII database of the present invention;
FIG. 2d3 is a sample diagram of the surprising expression of the CASMEII database in the present invention;
FIG. 2e1 is a sample diagram of positive expressions in the SAMM database according to the present invention;
FIG. 2e2 is a sample diagram of negative expressions in a SAMM database according to the present invention;
FIG. 2e3 is a sample diagram of the surprising expression of SAMM database in the present invention;
FIG. 3a is a schematic diagram of a confusion matrix with the highest recognition rate using a multi-scale LBP-TOP feature in the MMEW micro-expression database of the present invention;
FIG. 3b is a schematic diagram of a confusion matrix with the highest recognition rate of the MMEW micro-expression database using MDMO features according to the present invention;
FIG. 3c is a schematic diagram of a highest recognition confusion matrix using multi-scale LBP-TOP features in the CASMIEII database of the present invention;
FIG. 3d is a schematic diagram of a confusion matrix with the highest recognition rate of the CASMIII database using MDMO features according to the present invention;
FIG. 3e is a schematic diagram of a highest recognition confusion matrix using a multi-scale LBP-TOP feature of the SAMM database in accordance with the present invention;
FIG. 3f is a schematic diagram of a confusion matrix with the highest recognition rate of SAMM database using MDMO feature in the present invention;
FIG. 4a is a schematic diagram of the parameter sensitivity of the regularization parameter λ on the SAMM database when the recognition method of the present invention uses the multi-scale LBP-TOP feature, the training set is the CK + database and the MMEW micro-expression database;
FIG. 4b is a schematic diagram of the parameter sensitivity of the regularization parameter λ on the SAMM database when the MDMO feature, the training set, and the MMEW micro-expression database are used in the recognition method of the present invention;
FIG. 4c is a schematic diagram of the parameter sensitivity of the subspace dimension K on the SAMM database when the multi-scale LBP-TOP feature, the training set and the MMEW micro-expression database are used in the recognition method of the present invention;
FIG. 4d is a parameter sensitivity diagram of a subspace dimension K on a SAMM database when the MDMO feature, the training set, and the MMEW micro-expression database are used in the recognition method of the present invention.
Detailed Description
The present invention will be further described by way of examples, but not limited thereto, with reference to the accompanying drawings.
Example 1
An unsupervised cross-library micro-expression recognition method is shown in fig. 1, and comprises the following steps:
1) preparing training samples, wherein the training samples comprise a macro expression sample from a macro expression database and a micro expression sample from a micro expression database; obtaining a macro expression sample, and extracting characteristics of the macro expression to form a macro expression data matrix
Figure BDA0003034464160000101
Wherein d is1Representing a characteristic dimension of the macro expression, N1Representing the number of the macro expression samples; obtaining a micro-expression sample, and extracting characteristics of the micro-expression to form a micro-expression data matrix
Figure BDA0003034464160000102
Wherein d is2Representing a characteristic dimension of the micro-expression, N2Representing the number of micro expression samples;
2) after the macro expression data matrix and the micro expression data matrix are subjected to data alignment through feature selection and data standardization, the macro expression data matrix and the micro expression data matrix are recombined intoRaw joint data matrix
Figure BDA0003034464160000103
Wherein N is the number of source domain samples, and N is N1+N2；
3) Selecting a group of samples with the minimum difference with the characteristic distribution of a target domain from an original joint data matrix to form an auxiliary set based on a source domain selection model; as a training set for subsequent cross-library micro-expression recognition; the source domain selection model fully utilizes the MMD criterion, and the MMD between the auxiliary set and the source domain and the MMD between the auxiliary set and the target domain are minimized, so that the auxiliary set not only can represent the characteristic distribution of macro and micro expression samples of the source domain, but also can be in closer contact with the target domain samples; the auxiliary set is used for subsequent cross-library micro expression recognition;
4) based on the self-adaptive distribution alignment model, the relative importance of the edge distribution and the condition distribution in each task is quantitatively considered, the weight values of the edge distribution and the condition distribution in the migration process are self-adaptively adjusted, and L is passed2,1The row sparse characteristic of the norm is used for finishing the reweighting of the auxiliary set samples and weakening the influence of an abnormal value; and classifying and identifying the micro-expression database samples without labels in the target domain through a nearest neighbor classifier based on Euclidean distance.
Example 2
The method for identifying the micro-expression of the unsupervised cross-library system in the embodiment 1 is characterized in that:
in the step 1), the feature extracted for the macro expression is an LBP feature, and when the LBP feature of the macro expression is extracted, the feature extracted for the micro expression is a multi-scale LBP-TOP feature and an MDMO feature by adopting a multi-scale LBP-TOP feature blocking mode in a macro expression face blocking mode. The block division mode of the multi-scale LBP-TOP features divides the face area into four grid types of 1 × 1,2 × 2, 4 × 4 and 8 × 8, and totally 85 local sub-areas, and LBP feature extraction is carried out on each face sub-area and the face sub-areas are cascaded.
In step 2), respectively selecting the characteristics of the macro expression data matrix and the micro expression data matrix by a Principal Component Analysis (PCA) method, and enabling the macro expression data matrix and the micro expression data matrix to be in a micro expression stateThe situation data matrixes are unified into the same dimension d, d is less than or equal to d1And d is not more than d2(ii) a Respectively normalizing the macro expression data matrix and the micro expression data matrix to [0,1 ] through data standardization]In between, data alignment is completed.
In the step 2), the LBP characteristic dimension of the macro expression is 5015 dimension, the multi-scale LBP-TOP characteristic dimension of the micro expression is 15045 dimension, the MDMO characteristic dimension of the micro expression is 72 dimension, and the LBP characteristic of the macro expression, the multi-scale LBP-TOP characteristic of the micro expression and the MDMO characteristic of the micro expression are unified into the same dimension by adopting a principal component analysis method. When the micro expression characteristic is a multi-scale LBP-TOP characteristic, the dimension d of the aligned original joint data matrix is 5015; and when the micro expression characteristic is an MDMO characteristic, the dimension d of the aligned original joint data matrix is 72.
In step 3), based on a source domain selection model, selecting an optimal subset which is more closely associated with a target domain from an original joint data matrix as an auxiliary set sample, wherein the implementation steps are as follows:
the mathematical description based on the source domain selection model is shown in formula (i):
Figure BDA0003034464160000121
in the formula (I), the compound is shown in the specification,
Figure BDA0003034464160000122
n samples representing the source domain,
Figure BDA0003034464160000123
m samples representing a target domain;
Figure BDA0003034464160000124
mapping the source domain data and the target domain data to a regenerated kernel Hilbert space by using a regenerated kernel Hilbert space and a kernel mapping operator phi (.); alpha is alphaiTwo-valued selection factor, alpha, for selecting a model for the source domain i1 denotes that sample i in the source domain is selected into the auxiliary set, α i0 denotes that sample i in the source domain is not selected into the auxiliary set; by alphaiTo makeThe MMD of the selected source domain sample and the target domain micro-expression sample in the regeneration core Hilbert space is as small as possible, and the MMD of the selected source domain sample and the target domain micro-expression sample in the regeneration core Hilbert space is as small as possible, so that the auxiliary set is more suitable for training the target domain micro-expression sample, can represent macro and micro expression characteristics of the source domain, avoids the influence of a less-ideal sample, reduces a negative migration phenomenon, and improves a migration effect.
Since formula (I) is not directly calculable, there is no closed-form solution, by introducing a selection parameter betai，
Figure BDA0003034464160000125
Simplifying formula (I), converting formula (I) into formula (II):
Figure BDA0003034464160000126
unfolding formula (II) by kernel function skill to obtain formula (III):
Figure BDA0003034464160000127
in the formula (III), Ko,o＝Φ(xo)TΦ(xo)，Ko,t＝Φ(xo)TΦ(xt)，1oAnd 1tA column vector of all 1's; because a part of samples in the source domain do not satisfy the condition of forming the auxiliary set, the selection factor alpha corresponding to the part of samplesiIs 0, so the selection parameter β is theoretically sparse. Sparse beta by minimizing its L1Norm | | | beta | | | non conducting phosphor1＝1TObtaining beta, taking lambda as a penalty parameter and taking lambda as 1, and then obtaining the formula (IV) after the formula (III) is finished:
Figure BDA0003034464160000128
for β, the minimization problem is a standard quadratic programming problem that can be solved by a number of efficient methods. And recovering the value of alpha after obtaining the beta, and selecting a sample from the source domain according to the alpha through a preset threshold or a preset proportion to form an auxiliary set. In particular, due to α i1 denotes that sample i in the source domain is selected into the auxiliary set, α i0 indicates that the sample i in the source domain is not selected into the auxiliary set, and in the present invention, the obtained α is sorted from large to small in value, and the first 70% of α is selectediAnd forming an auxiliary set by the corresponding ith source domain sample. Completing the source domain selection algorithm.
In step 4), based on self-adaptive distribution alignment model
Figure BDA0003034464160000131
Is represented by formula (V):
Figure BDA0003034464160000132
in the formula (V), the compound is shown in the specification,
Figure BDA0003034464160000133
in order to be the auxiliary set, the method comprises the following steps of,
Figure BDA0003034464160000134
testing a set for the target domain;
Figure BDA0003034464160000135
forming a data matrix of the auxiliary set after the source domain selection algorithm in the step 3), wherein N is the number of samples of the auxiliary set formed after N samples of the source domain pass through the source domain selection algorithm;
Figure BDA0003034464160000136
the target domain data set is not provided with a label, and m is the number of target domain samples; c is in different micro expression label categories;
Figure BDA0003034464160000137
and
Figure BDA0003034464160000138
respectively representing the samples belonging to class c in the auxiliary set and the target domain test set,
Figure BDA0003034464160000139
is xsThe real label of (a) is,
Figure BDA00030344641600001310
is xtThe pseudo tag of (1);
Figure BDA00030344641600001311
and
Figure BDA00030344641600001312
respectively represent
Figure BDA00030344641600001313
And
Figure BDA00030344641600001314
the number of samples;
first term of formula (V)
Figure BDA00030344641600001315
Is the edge distribution distance, the second term
Figure BDA00030344641600001316
Is the conditional distribution distance, mu is the adaptive parameter of the edge distribution and the conditional distribution, and mu belongs to [0,1 ]](ii) a When μ → 1, the source domain
Figure BDA00030344641600001317
And a target domain
Figure BDA00030344641600001318
The data sets are similar, and the condition distribution needs to be adjusted; when μ → 0, the source domain
Figure BDA00030344641600001319
And a target domain
Figure BDA00030344641600001320
The data sets themselves have larger gaps, and the edge distribution needs to be adjusted more. Thus, the adaptive parameter μ can adaptively adjust the importance of each distribution, resulting in better results.
Approximating RKHS with mapping matrix A, formula (V) is converted to formula (VI):
Figure BDA00030344641600001321
using matrix techniques and regularization, equation (vi) is empirically estimated as (vii):
Figure BDA0003034464160000141
in the formula (VII), X is XsAnd xtA composed joint data matrix, X ═ Xs xt]；||.||FIs the F norm, λ is the regularization parameter of the F norm;
Figure BDA0003034464160000142
is an identity matrix, H is a central matrix,
Figure BDA0003034464160000143
Figure BDA0003034464160000144
the term is used for reducing the complexity of the self-adaptive distribution alignment model on the basis of keeping the original characteristic structure; formula (vii) involves two constraints: the first constraint is used to ensure the transformed data ATX maintains the data structure of the original data, i.e. maintains the divergence of the data, for matrix X, its divergence matrix is XHXT(ii) a The second constraint is the range of the adaptive parameter μ; m0And McThe MMD matrix is constructed in a way shown in formulas (VIII) and (IX) respectively:
Figure BDA0003034464160000145
Figure BDA0003034464160000146
in step 4), the values of the adaptive parameter μ use the global and local structure of the domain, using
Figure BDA0003034464160000147
Determining the distance;
Figure BDA0003034464160000148
the distance is an error of a binary linear classifier (e.g., a support vector machine, a logistic regression model, etc.) for distinguishing two domains, and is defined as shown in formula (x):
Figure BDA0003034464160000149
in the formula (X), ε (h) is a distinction
Figure BDA00030344641600001410
And
Figure BDA00030344641600001411
average error of linear classifier h of (1); thus, the edge distribution dMIs/are as follows
Figure BDA00030344641600001412
The distance can be calculated directly. For distribution of conditions
Figure BDA00030344641600001413
Distance, by dcA value representing the class c is selected,
Figure BDA00030344641600001414
Figure BDA00030344641600001415
are respectively shown in
Figure BDA00030344641600001416
And
Figure BDA00030344641600001417
the sample of class c in (1); in summary, μ is estimated as formula (XI):
Figure BDA0003034464160000151
the adaptation parameter μ has to be recalculated in each iteration of the adaptive distribution alignment, since the prediction tag ysChanges may occur after each iteration, so dcWill change in each iteration and the distribution of changes in this characteristic will cause μ to change as well.
In step 4), since the source domain is an auxiliary set formed after the model is selected by the source domain and contains data information of a plurality of databases, L is introduced2,1The norm measures the specific contribution of each sample in the auxiliary set, and the influence of an abnormal value is reduced; l is2,1Norm being L of the row vector2Sum of norm, minimizing L of matrix2,1The norm is such that L for each row of the matrix2The norm is as small as possible and as many 0 elements are present within a row as possible, i.e., as sparse as possible. Since each row of the transfer matrix a corresponds to one sample instance, in essence, the row sparsity controls the degree of feature selection of the mapping matrix for the auxiliary set samples, facilitating adaptive re-weighting of the samples. Finally, the objective function of the adaptive distribution alignment model is shown in formula (xii):
Figure BDA0003034464160000152
in step 4), the target function formula (xi) of the adaptive distribution alignment model is a matrix decomposition problem with constraint, so that the target function formula (xi) of the adaptive distribution alignment model can be optimized by adopting a lagrange multiplier-based method, and the method comprises the following steps:
first, the objective function is transformed to obtain formula (XIII):
Figure BDA0003034464160000153
in the formula (XIII), U is a Lagrange multiplier; thus, the augmented Lagrangian equation is shown in equation (XIV):
Figure BDA0003034464160000154
note that μ is also updated at each iteration, so in an iteration, the substituted μ is the μ updated with formula (XI)(k)(ii) a In the (k +1) th iteration, the variable updating process is shown as the formula (XV):
Figure BDA0003034464160000155
in the formula (XV), Z(k)＝Diag(z(k))，
Figure BDA0003034464160000156
i is the ith element of the matrix z,
Figure BDA0003034464160000157
for a minimum value, avoid the case that the denominator of the above formula is 0 during the update process (in the calculation, we use the minimum positive number eps in Matlab, eps is 2.2204e-16)，Ai (k)Represents A(k)Row i of (1); the invention sets the number of iterations to 18, usually 15 iterations after the objective function has converged, and 18 are set for insurance purposes.
Zero is derived for A, and the optimization problem of formula (XV) is converted into the solving problem of the following formula characteristic decomposition, as shown in formula (XVI):
Figure BDA0003034464160000161
and solving the formula (XVI) to obtain the eigenvectors corresponding to the minimum K eigenvalues forming the mapping matrix A.
In the step 4), a nearest neighbor classifier based on Euclidean distance is used for classifying and identifying the micro expression database samples without labels in the target domain, and the specific implementation steps are as follows:
the method comprises the steps of obtaining a trained mapping matrix A by solving the linear optimization problem of an objective function, projecting auxiliary set data and test set data into a public subspace by adopting the mapping matrix A, and comparing and identifying data representation in the public subspace in a nearest neighbor mode, namely:
find a projection matrix and order
Figure BDA0003034464160000162
Representing all of the training data in the common subspace,
Figure BDA0003034464160000163
for the auxiliary set features and their corresponding labels obtained after the source domain selection algorithm,
Figure BDA0003034464160000164
for unlabeled target domain data sets, AsAnd AtCorresponding to the front n rows and the rear m rows of the mapping matrix A;
for a given target domain micro-expression data set, any one test sample after projection is used
Figure BDA0003034464160000165
Nearest neighbor comparison based on Euclidean distance
Figure BDA0003034464160000166
Figure BDA0003034464160000167
Refer to the ith source domain auxiliary set sample
Figure BDA0003034464160000168
The micro-expression type label of the training sample closest to the micro-expression type label is assigned to the test sample to finish micro-expression identification.
The embodiment relates to two macro expression databases CK + and MMEW macro expression databases (MMEW macro for short), three micro expression databases MMEW micro expression databases (MMEW micro for short), a CASMEII database and a SAMM database. Because the emotion types of the macro and micro expression databases are not uniform, and the number of micro expression samples of some classes is too small, in order to unify all database classes, the macro and micro expressions involved in the experiment are recombined and divided into three classes: positive, negative and surprise. The emotional expression with positive emotional colors such as happiness is actively reflected; negatively reflect emotional expressions with negative emotions such as fear, sadness, and aversion; in general, it is surprising that the generation of such emotions does not belong to a positive or negative category, and therefore, surprise is classified as an independent category. FIG. 2a1 is a sample diagram of positive expression of CK + database; FIG. 2a2 is a sample diagram of negative expressions in the CK + database; FIG. 2a3 is a sample diagram of a surprising expression of the CK + database; FIG. 2b1 is a sample schematic diagram of MMEW macro expression database positive expression; FIG. 2b2 is a sample diagram of negative expressions in the MMEW macro expression database; FIG. 2b3 is a sample schematic diagram of the MMEW macro expression database surprise expression; FIG. 2c1 is a sample diagram of positive expression in the MMEW micro-expression database; FIG. 2c2 is a sample diagram of negative expressions in the MMEW micro-expression database; FIG. 2c3 is a schematic diagram of a sample of surprising expression in the MMEW micro-expression database; FIG. 2d1 is a sample diagram of positive expressions in the CASMEII database; FIG. 2d2 is a sample diagram of negative expressions in the CASMEII database; FIG. 2d3 is a schematic diagram of a sample of surprising expressions in the CASMEII database; FIG. 2e1 is a sample diagram of positive expressions in SAMM database; FIG. 2e2 is a sample diagram of negative expressions in a SAMM database; FIG. 2e3 is a sample diagram of the surprising expression of SAMM database. The number of samples after each database repartition is shown in table 1.
TABLE 1
Figure BDA0003034464160000171
For theAnd the macro expression sample only adopts the climax frame as one data sample of the macro expression, the radius R of the LBP feature is 1, and the adjacent point P is 8. For the used micro-expression video sequence, firstly segmenting the micro-expression video sequence into a micro-expression picture sequence, uniformly interpolating the micro-expression picture sequence into 60 frames after image cutting and face alignment, wherein the adjacent point P of a multi-scale LBP-TOP is 8, and the radius R isxy＝4，Rxt＝1，R yt1. The macroexpression picture and the microexpression picture sequence adopted in the embodiment are uniformly cut into 231 × 231 pixels. In the experiment, linear kernel functions are used for all involved kernel functions.
According to different micro expression databases of the test set, the invention divides the experiment into three groups, namely the experiment of the MMEW micro expression database, the experiment of the CASME II database and the experiment of the SAMM database. Table 2, table 3 and table 4 respectively show the experimental results of the multi-scale LBP-TOP feature and MDMO feature and the corresponding regularization parameter λ and subspace dimension K when the test set is the MMEW micro-expression database, the CASME II database and the SAMM database. It should be noted that, when a single micro-expression database is used for migration, the source domain selection model and the adaptive distribution alignment model still need to be used, and in this case, the auxiliary set samples formed by the source domain are only micro-expression samples.
TABLE 2
Figure BDA0003034464160000172
Figure BDA0003034464160000181
TABLE 3
Figure BDA0003034464160000182
TABLE 4
Figure BDA0003034464160000183
As can be seen from tables 2 to 4, the unsupervised cross-library micro-expression recognition method provided by the invention has good recognition effect on three databases, and the highest recognition rate reaches 70.5%, thus the method has good recognition effect in unsupervised micro-expression recognition. Overall, the MDMO feature has a better recognition effect than the multi-scale LBP-TOP feature, because the MDMO feature performs feature extraction by dividing the region of interest from the perspective of optical flow, and compared with the multi-scale LBP-TOP feature performing feature description from the perspective of texture, the MDMO feature is more discriminative and can more specifically capture the micro-expression fine information. Meanwhile, the recognition rate of migration by using a single micro expression database is far lower than the result of migration by using macro and micro expression databases together, so that the macro expression which is rich in sample information and has certain commonality with the micro expression is introduced as an aid, and the method is a way for effectively improving the micro expression recognition rate.
Aiming at the obtained recognition results, the invention correspondingly provides a confusion matrix of the best result of each feature on each micro expression database, namely the bold part of tables 2, 3 and 4. Fig. 3a and 3b respectively and correspondingly show confusion matrixes of the MMEW micro expression database adopting multi-scale LBP-TOP characteristics and MDMO characteristics, fig. 3c and 3d respectively and correspondingly show the expressions of the multi-scale LBP-TOP and MDMO characteristics of the CASME II database, and fig. 3e and 3f respectively and correspondingly show the recognition effects of the two characteristics of the SAMM database.
Through observation of the confusion matrix, in the MMEW micro-expression database and the CASMEII database, the negative recognition effect is slightly higher than the positive and surprising recognition effects, and meanwhile, the positive and surprising recognition effects are closer. This is because the number of passive samples is large in the corresponding training sets of the two databases, and although the selection of the auxiliary set is performed by using the source domain selection algorithm, the number of passive samples is still different from that of the active and surprise, and therefore, the identification result of the passive category is relatively best. Meanwhile, the difference between the number of the positive samples and the number of the surprise samples is not large, and the identification results are similar. In the SAMM database, there are many passive and surprising samples and the number of samples is relatively close, so that on the confusion matrix of the SAMM database, the passive and surprising recognition effects are better, and the positive recognition effect is slightly lower. In addition, the recognition rate of the MDMO features in the three emotional categories was more uniform than the multi-scale LBP-TOP feature, indicating that the MDMO features have better category stability in the face of sample imbalances.
According to the method, the regularization parameter lambda and the subspace dimension K influence the recognition effect of the micro expression, so that for two different types of micro expression characteristics adopted in the experiment, a large number of experiments are carried out on each parameter on each database, and the parameter values corresponding to the optimal recognition rate are shown in tables 2 to 4. Here, the test set is selected as SAMM database, and the training set is a group of discussion parameter sensitivity of CK + database and MMEW micro-expression database. When a certain parameter is evaluated, only the evaluated parameter is changed within a certain range, other parameters are selected to be fixed values according to experience, namely the subspace dimension of the fixed multi-scale LBP-TOP characteristic is 300, the subspace dimension of the MDMO characteristic is 50, and the influence of the regularization parameter lambda on the recognition rate is researched; and fixing the regularization parameter lambda to be 0.01, and researching the influence of the selected subspace dimension on the identification rate. The results of the parameter sensitivity experiments are shown in fig. 4a, 4b, 4c, 4 d.
In order to evaluate the performance of the proposed unsupervised cross-library micro-expression recognition, the invention compares the MMEW micro-expression database, the CASME II database and the SAMM database with other most advanced cross-library methods, such as migration component analysis TCA (Transfer component analysis), Joint Distribution Adaptation (JDA), a singular value decomposition-based macro-micro migration model (SVD), a Joint direct push type migration regression and auxiliary set selection model (TTRM + ASSM), a deep Learning method Transfer Learning, and the like. Table 5, table 6 and table 7 list the micro-expression recognition rates of all the methods on these three databases, respectively. As can be seen from the table, no matter whether the training set is a single micro-expression database or a macro-micro-expression database or a micro-expression database, the model provided by the invention is superior to other most advanced methods in general.
TABLE 5
Figure BDA0003034464160000191
Figure BDA0003034464160000201
TABLE 6
Figure BDA0003034464160000202
TABLE 7
Figure BDA0003034464160000203

Claims (10)

1. An unsupervised cross-library micro-expression recognition method is characterized by comprising the following steps:
1) preparing a training sample, wherein the training sample comprises a macro expression sample and a micro expression sample; obtaining a macro expression sample, and extracting characteristics of the macro expression to form a macro expression data matrix
Figure FDA0003034464150000011
Wherein d is1Representing a characteristic dimension of the macro expression, N1Representing the number of the macro expression samples; obtaining a micro-expression sample, and extracting characteristics of the micro-expression to form a micro-expression data matrix
Figure FDA0003034464150000012
Wherein d is2Representing a characteristic dimension of the micro-expression, N2Representing the number of micro expression samples;
2) after the macro expression data matrix and the micro expression data matrix are subjected to data alignment through feature selection and data standardization, the macro expression data matrix and the micro expression data matrix are recombined into an original combined data matrix
Figure FDA0003034464150000013
Wherein N is the number of source domain samples, and N is N1+N2；
3) Selecting a group of samples with the minimum difference with the characteristic distribution of a target domain from an original joint data matrix to form an auxiliary set based on a source domain selection model;
4) based on the self-adaptive distribution alignment model, the relative importance of the edge distribution and the condition distribution in each task is quantitatively considered, the weight values of the edge distribution and the condition distribution in the migration process are self-adaptively adjusted, and L is passed2,1The row sparse characteristic of the norm is used for finishing the reweighting of the auxiliary set samples and weakening the influence of an abnormal value; and classifying and identifying the micro-expression database samples without labels in the target domain through a nearest neighbor classifier based on Euclidean distance.
2. The unsupervised cross-library micro expression recognition method of claim 1, wherein in step 1), the features extracted for macro expressions are LBP features, and when extracting the LBP features of macro expressions, the features extracted for micro expressions are multi-scale LBP-TOP features and MDMO features in a multi-scale LBP-TOP feature blocking manner by using the macro expression face blocking manner.
3. The method of claim 1, wherein the cross-library micro-expression recognition is performed,
in the step 2), feature selection is respectively carried out on the macro expression data matrix and the micro expression data matrix through a principal component analysis method, the macro expression data matrix and the micro expression data matrix are unified into the same dimension d, and d is not less than d1And d is not more than d2(ii) a Respectively normalizing the macro expression data matrix and the micro expression data matrix to [0,1 ] through data standardization]Completing data alignment;
in the step 2), the LBP characteristic dimension of the macro expression is 5015 dimension, the multi-scale LBP-TOP characteristic dimension of the micro expression is 15045 dimension, the MDMO characteristic dimension of the micro expression is 72 dimension, the LBP characteristic of the macro expression, the multi-scale LBP-TOP characteristic of the micro expression and the MDMO characteristic of the micro expression are unified into the same dimension by adopting a principal component analysis method, and when the micro expression characteristic is the multi-scale LBP-TOP characteristic, the dimension d of the aligned original combined data matrix is 5015; and when the micro expression characteristic is an MDMO characteristic, the dimension d of the aligned original joint data matrix is 72.
4. The unsupervised cross-library micro-expression recognition method of claim 1, wherein in step 3), based on the source domain selection model, an optimal subset which is more closely related to the target domain is selected from the original joint data matrix as an auxiliary set sample, and the implementation steps are as follows:
the mathematical description of the source domain selection-based model is as shown in formula (i):
Figure FDA0003034464150000021
in the formula (I), the compound is shown in the specification,
Figure FDA0003034464150000022
n samples representing the source domain,
Figure FDA0003034464150000023
m samples representing a target domain;
Figure FDA0003034464150000024
mapping the source domain data and the target domain data to a regenerated kernel Hilbert space by using a regenerated kernel Hilbert space and a kernel mapping operator phi (.); alpha is alphaiTwo-valued selection factor, alpha, for selecting a model for the source domaini1 denotes that sample i in the source domain is selected into the auxiliary set, αi0 denotes that sample i in the source domain is not selected into the auxiliary set;
introducing a selection parameter betai，
Figure FDA0003034464150000025
Simplifying formula (I), converting formula (I) into formula (II):
Figure FDA0003034464150000026
unfolding formula (II) by kernel function skill to obtain formula (III):
Figure FDA0003034464150000027
in the formula (III), Ko,o＝Φ(xo)TΦ(xo)，Ko,t＝Φ(xo)TΦ(xt)，1oAnd 1tA column vector of all 1's; sparse beta by minimizing its L1Norm | | | beta | | | non conducting phosphor1＝1TBeta is obtained, and lambda is a penalty parameter, then the formula (III) is finished to obtain the formula (IV):
Figure FDA0003034464150000028
and recovering the value of alpha after obtaining the beta, and selecting a sample from the source domain according to the alpha through a preset threshold or a preset proportion to form an auxiliary set.
5. The method for unsupervised cross-library micro-expression recognition according to claim 4, wherein λ is 1;
for the obtained alpha, sorting the obtained alpha according to the numerical value from large to small, and selecting the alpha of the first 70 percentiAnd forming an auxiliary set by the corresponding ith source domain sample.
6. The method for cross-library unsupervised micro expression recognition according to claim 1, wherein in step 4), the alignment model is based on adaptive distribution
Figure FDA0003034464150000031
Is represented by formula (V):
Figure FDA0003034464150000032
in the formula (V), the compound is shown in the specification,
Figure FDA0003034464150000033
in order to be the auxiliary set, the method comprises the following steps of,
Figure FDA0003034464150000034
testing a set for the target domain;
Figure FDA0003034464150000035
n is the number of samples of the auxiliary set formed after N samples of the source domain are subjected to a source domain selection algorithm;
Figure FDA0003034464150000036
the target domain data set is not provided with a label, and m is the number of target domain samples; c is in different micro expression label categories;
Figure FDA0003034464150000037
and
Figure FDA0003034464150000038
respectively representing the samples belonging to class c in the auxiliary set and the target domain test set,
Figure FDA0003034464150000039
is xsThe real label of (a) is,
Figure FDA00030344641500000310
is xtThe pseudo tag of (1);
Figure FDA00030344641500000311
and
Figure FDA00030344641500000312
respectively represent
Figure FDA00030344641500000313
And
Figure FDA00030344641500000314
the number of samples;
first term of formula (V)
Figure FDA00030344641500000315
Is the edge distribution distance, the second term
Figure FDA00030344641500000316
Is the conditional distribution distance, mu is the adaptive parameter of the edge distribution and the conditional distribution, and mu belongs to [0,1 ]](ii) a Approximating RKHS with mapping matrix A, formula (V) is converted to formula (VI):
Figure FDA00030344641500000317
using matrix techniques and regularization, equation (vi) is empirically estimated as (vii):
Figure FDA00030344641500000318
in the formula (VII), X is XsAnd xtA composed joint data matrix, X ═ Xs xt]；||·||FIs the F norm, λ is the regularization parameter of the F norm;
Figure FDA00030344641500000319
is an identity matrix, H is a central matrix,
Figure FDA00030344641500000320
Figure FDA00030344641500000321
the term is used to retain the original characteristicsOn the basis of the structure, the complexity of the self-adaptive distribution alignment model is reduced; formula (vii) involves two constraints: the first constraint is used to ensure the transformed data ATX maintains the data structure of the original data, i.e. maintains the divergence of the data, for matrix X, its divergence matrix is XHXT(ii) a The second constraint is the range of the adaptive parameter μ; m0And McThe MMD matrix is constructed in a way shown in formulas (VIII) and (IX) respectively:
Figure FDA0003034464150000041
Figure FDA0003034464150000042
7. the method for cross-library unsupervised micro-expression recognition according to claim 1, wherein in step 4), the adaptive parameter μ is obtained by using the global and local structure of the domain
Figure FDA0003034464150000043
Determining the distance;
Figure FDA0003034464150000044
the distance is the error of the binary linear classifier used to distinguish the two domains, and is defined as shown in formula (X):
Figure FDA0003034464150000045
in the formula (X), ε (h) is a distinction
Figure FDA0003034464150000046
And
Figure FDA0003034464150000047
average error of linear classifier h of (1); for distribution of conditions
Figure FDA0003034464150000048
Distance, by dcA value representing the class c is selected,
Figure FDA0003034464150000049
Figure FDA00030344641500000410
are respectively shown in
Figure FDA00030344641500000411
And
Figure FDA00030344641500000412
the sample of class c in (1); in summary, μ is estimated as formula (XI):
Figure FDA00030344641500000413
8. the method for unsupervised cross-library micro-expression recognition according to claim 1, wherein in step 4), L is introduced2,1The norm measures the specific contribution of each sample in the auxiliary set, and the influence of an abnormal value is reduced;
the objective function of the adaptive distribution alignment model is shown in formula (XII):
Figure FDA0003034464150000051
9. the unsupervised cross-library micro-expression recognition method as claimed in claim 1, wherein in step 4), the target function formula (XI) of the adaptive distribution alignment model is optimized by a method based on Lagrangian multipliers, which comprises the following steps:
first, the objective function is transformed to obtain formula (XIII):
Figure FDA0003034464150000052
in the formula (XIII), U is a Lagrange multiplier; thus, the augmented Lagrangian equation is shown in equation (XIV):
Figure FDA0003034464150000053
mu is also updated at each iteration, so in an iteration, the substituted mu is the one updated with formula (XI)(k)(ii) a In the (k +1) th iteration, the variable updating process is shown as the formula (XV):
Figure FDA0003034464150000054
in the formula (XV), Z(k)＝Diag(z(k))，
Figure FDA0003034464150000055
i is the ith element of the matrix z, theta is a minimum value,
Figure FDA0003034464150000056
represents A(k)Row i of (1); zero is derived for A, and the optimization problem of formula (XV) is converted into the solving problem of the following formula characteristic decomposition, as shown in formula (XVI):
Figure FDA0003034464150000057
and solving the formula (XVI) to obtain the eigenvectors corresponding to the minimum K eigenvalues forming the mapping matrix A.
10. The unsupervised cross-library micro expression recognition method according to any one of claims 1-9, wherein in step 4), a nearest neighbor classifier based on euclidean distance is used to classify and recognize the target domain unlabeled micro expression database sample, and the specific implementation steps are as follows:
the method comprises the steps of obtaining a trained mapping matrix A by solving the linear optimization problem of an objective function, projecting auxiliary set data and test set data into a public subspace by adopting the mapping matrix A, and comparing and identifying data representation in the public subspace in a nearest neighbor mode, namely:
find a projection matrix and order
Figure FDA0003034464150000058
Representing all of the training data in the common subspace,
Figure FDA0003034464150000059
for the auxiliary set features and their corresponding labels obtained after the source domain selection algorithm,
Figure FDA0003034464150000061
for unlabeled target domain data sets, AsAnd AtCorresponding to the front n rows and the rear m rows of the mapping matrix A;
for a given target domain micro-expression data set, any one test sample after projection is used
Figure FDA0003034464150000062
Nearest neighbor comparison based on Euclidean distance
Figure FDA0003034464150000063
Figure FDA0003034464150000064
Refer to the ith source domain auxiliary set sample
Figure FDA0003034464150000065
The micro-expression category label of (a),and assigning the micro-expression type label of the training sample closest to the training sample to the test sample to finish micro-expression identification.
CN202110439454.9A 2021-04-23 2021-04-23 Unsupervised cross-library micro-expression identification method Active CN113095270B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110439454.9A CN113095270B (en) 2021-04-23 2021-04-23 Unsupervised cross-library micro-expression identification method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110439454.9A CN113095270B (en) 2021-04-23 2021-04-23 Unsupervised cross-library micro-expression identification method

Publications (2)

Publication Number Publication Date
CN113095270A true CN113095270A (en) 2021-07-09
CN113095270B CN113095270B (en) 2022-07-12

Family

ID=76679584

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110439454.9A Active CN113095270B (en) 2021-04-23 2021-04-23 Unsupervised cross-library micro-expression identification method

Country Status (1)

Country Link
CN (1) CN113095270B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113628640A (en) * 2021-07-15 2021-11-09 河南工业大学 Cross-library speech emotion recognition method based on sample equalization and maximum mean difference

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017190645A1 (en) * 2016-05-03 2017-11-09 中兴通讯股份有限公司 Method, device and storage medium for micro-expression recognition
CN109543603A (en) * 2018-11-21 2019-03-29 山东大学 A kind of micro- expression recognition method based on macro sheet feelings knowledge migration
CN110097020A (en) * 2019-05-10 2019-08-06 山东大学 A kind of micro- expression recognition method based on joint sparse dictionary learning
CN110457975A (en) * 2018-05-07 2019-11-15 山东大学 A kind of micro- expression recognition method based on the migration of macro information knowledge
CN111444859A (en) * 2020-03-30 2020-07-24 哈尔滨工程大学 Unsupervised cross-domain facial expression recognition method
CN111582212A (en) * 2020-05-15 2020-08-25 山东大学 Multi-domain fusion micro-expression detection method based on motion unit

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017190645A1 (en) * 2016-05-03 2017-11-09 中兴通讯股份有限公司 Method, device and storage medium for micro-expression recognition
CN110457975A (en) * 2018-05-07 2019-11-15 山东大学 A kind of micro- expression recognition method based on the migration of macro information knowledge
CN109543603A (en) * 2018-11-21 2019-03-29 山东大学 A kind of micro- expression recognition method based on macro sheet feelings knowledge migration
CN110097020A (en) * 2019-05-10 2019-08-06 山东大学 A kind of micro- expression recognition method based on joint sparse dictionary learning
CN111444859A (en) * 2020-03-30 2020-07-24 哈尔滨工程大学 Unsupervised cross-domain facial expression recognition method
CN111582212A (en) * 2020-05-15 2020-08-25 山东大学 Multi-domain fusion micro-expression detection method based on motion unit

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
TIANHUAN HUANG ET AL: "A Multiview Representation Framework for Micro-Expression Recognition", 《SPECIAL SECTION ON ADVANCED DATA MINING METHODS FOR SOCIAL COMPUTING》 *
肖瑞雪: "基于宏表情迁移学习的微表情识别方法研究", 《中国优秀博硕士学位论文全文数据库(硕士)信息科技辑》 *
鄢克雨: "基于领域自适应方法的跨领域表情识别研究", 《中国优秀博硕士学位论文全文数据库(博士)信息科技辑》 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113628640A (en) * 2021-07-15 2021-11-09 河南工业大学 Cross-library speech emotion recognition method based on sample equalization and maximum mean difference

Also Published As

Publication number Publication date
CN113095270B (en) 2022-07-12

Similar Documents

Publication Publication Date Title
CN110532920B (en) Face recognition method for small-quantity data set based on FaceNet method
CN106845510B (en) Chinese traditional visual culture symbol recognition method based on depth level feature fusion
Dubes et al. Random field models in image analysis
CN111178432A (en) Weak supervision fine-grained image classification method of multi-branch neural network model
CN111738143B (en) Pedestrian re-identification method based on expectation maximization
CN110222213B (en) Image classification method based on heterogeneous tensor decomposition
CN109815357B (en) Remote sensing image retrieval method based on nonlinear dimension reduction and sparse representation
CN110889865B (en) Video target tracking method based on local weighted sparse feature selection
Zhang et al. A nonlinear neural network model of mixture of local principal component analysis: application to handwritten digits recognition
Yi et al. Facial expression recognition based on t-SNE and adaboostM2
CN112766400A (en) Semi-supervised classification integration method for high-dimensional data based on multiple data transformation spaces
Ren et al. Discriminative residual analysis for image set classification with posture and age variations
CN113095270B (en) Unsupervised cross-library micro-expression identification method
CN112966735B (en) Method for fusing supervision multi-set related features based on spectrum reconstruction
CN112465062A (en) Clustering method based on manifold learning and rank constraint
CN117093924A (en) Rotary machine variable working condition fault diagnosis method based on domain adaptation characteristics
KR102225586B1 (en) System and Merhod for Log Euclidean Metric Learning using Riemannian Submanifold Framework on Symmetric Positive Definite Manifolds
Barakbah et al. A new approach for image segmentation using Pillar-Kmeans algorithm
Elsayed et al. Hand gesture recognition based on dimensionality reduction of histogram of oriented gradients
CN111488923A (en) Enhanced anchor point image semi-supervised classification method
CN110287973A (en) A kind of image characteristic extracting method based on low-rank robust linear discriminant analysis
Behmanesh et al. Geometric multimodal learning based on local signal expansion for joint diagonalization
Bouguila et al. Mml-based approach for high-dimensional unsupervised learning using the generalized dirichlet mixture
CN114943862A (en) Two-stage image classification method based on structural analysis dictionary learning
CN111507243B (en) Human behavior recognition method based on Grassmann manifold analysis

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant