CN109711283B - Occlusion expression recognition method combining double dictionaries and error matrix - Google Patents

Abstract

The invention discloses an occluded facial expression recognition method combined with a double dictionary and an error matrix. matrix for dictionary learning to obtain intra-class related dictionaries and difference structure dictionaries; secondly, when classifying occluded images, the original sparse coding does not consider coding errors and cannot accurately describe the coding errors caused by occlusions. matrix representation, which can be separated from the feature matrix of unoccluded training images; the clear image can be recovered by subtracting the error matrix from the test sample; the clear image sample is low-rank decomposed into identity features using a dual-dictionary collaborative representation and expression features, and finally achieve classification according to the contribution of each category of expression features in the joint sparse representation. The present invention is robust to randomly occluded expression recognition.

Description

An occluded facial expression recognition method based on a combination of double dictionaries and error matrix

technical field

The present invention relates to the field of image processing, and more specifically, relates to a method for recognizing occluded facial expressions by combining a double dictionary and an error matrix.

Background technique

Facial expression recognition technology is a very popular topic in the fields of physiology, pattern recognition and computer vision, and related technologies are becoming more and more mature. In order to ensure the integrity of expression information, most studies are conducted under controlled experimental conditions. However, in practical applications, facial expression recognition needs to overcome various random problems such as illumination, occlusion, and posture. Therefore, facial expression recognition remains a challenging subject. In terms of occluded facial expression recognition, researchers have proposed many methods to reduce the impact of occlusion on expression recognition. The most commonly used methods include feature reconstruction, sub-region analysis, sparse representation, and deep learning. Xia et al. combined RPCA and saliency detection for occluded expression recognition. Firstly, saliency detection is used to locate the occluders, and then RPCA is used to reconstruct the occluded areas of the image. combined iterative closest point (ICP) and fuzzy C-Means (FCM) algorithms to reconstruct 54 fiducial points in occluded face images. This feature recovery of occluded regions based on facial contour features relies heavily on reliable face detection and tracking of facial features, and requires pre-positioning of occluded regions and precise face alignment, which is difficult to achieve in practical applications. Lin et al. used the geometric features of face point displacement to predict the AU unit under oral cavity occlusion. Gaussian Mixture Model (GMM) is used to model the distribution of gray pixels in the face area, and to detect the occluded area. Methods based on sub-region analysis all assume that occlusions occur only in a small part of the face, and this method can usually produce satisfactory performance for small-scale occlusions. However, the granularity of subdividing faces into local regions and its impact on performance is still an open problem, especially for random occlusions without fixed positions, shapes, and sizes. Meanwhile, sub-region based methods are also sensitive to noise due to inaccurate face localization, alignment and normalization. Cheng et al. proposed a deep neural network structure under partial occlusion conditions. Gabor filters are used to extract multi-scale and multi-directional Gabor magnitudes from face images, and they are fed into a three-layer Deep Boltzmann Machine (DBM) for emotion classification. Yao et al proposed to use the Wasserstein generative adversarial network to complement the occluded area of the face image, which can alleviate the impact caused by the lack of local expression information. Although deep learning techniques can automatically learn the most discriminative facial expression feature patterns from raw facial data, a separate occlusion detection or reconstruction process is usually not required. However, using deep architectures requires a large amount of training data to ensure proper feature learning, the difficulty of tuning a large number of system parameters, and the need for expensive computation. Zhu Minghan et al obtained occlusion base images at all levels by dividing the image into blocks, and used these images to construct a non-orthogonal occlusion dictionary, and then sparsely decomposed the image to be tested to obtain coefficients, and realized the expression category in the subspace of the image to be tested. judge. Liu Shuaishi et al. proposed a robust regularization encoding for randomly occluded expression recognition. By assigning different weights to each pixel of the expression image, continuous iterations make the weights converge to the threshold, and finally, the sparse representation of the image to be tested is calculated through the optimal weight matrix. This method has achieved a good recognition effect, but this method does not avoid the interference of identity features on expression classification. The biggest advantage of the sparse representation method is that it is not only robust to occlusion and damage, but also can be used to estimate the occluded or damaged parts of the face.

The traditional sparse representation methods SRC and CRC separate each dictionary atom and process it independently, without considering the relationship between each atom, and the resulting sparse representation is non-structural. At the same time, both SRC and CRC use the original training data set as the classification dictionary. When the training samples are damaged, occluded, deformed, etc., the algorithm lacks robustness and the classification effect is poor.

Contents of the invention

In order to overcome the shortcomings of the lack of robustness and poor classification effect of the classification method described in the above prior art, the present invention provides a method for recognizing occluded expressions combined with a double dictionary and an error matrix.

The present invention aims to solve the above-mentioned technical problems at least to a certain extent.

Technical scheme of the present invention is as follows:

A kind of occlusion facial expression recognition method of joint double dictionary and error matrix, described algorithm comprises the following steps:

S1: Data preprocessing, obtain a data sample set from the database, rotate the sample set to align the eye level, and cut out a rectangular area containing only positive facial expressions from the original expression image according to the distance between the two eyes , remove the redundant information of hair and neck;

S2: Occlusion simulation, the preprocessed data sample set is used to perform occlusion simulation by adding black blocks of different sizes at different positions, and the data samples after occlusion simulation are used as training samples;

S3: In the training phase, perform low-rank decomposition of several kinds of expression images in the training sample to obtain expression features and identity features, where both expression features and identity features are represented by a matrix; given a training sample X=[X ₁ , X ₂ , ..., X _c ], the decomposition function is as follows:

Perform low-rank decomposition on the set of training samples, and obtain L and S after decomposition;

L=[L ₁ , L ₂ , .., L _c ]∈R ^n×mc

S=[S ₁ , S ₂ , .., S _c ]∈R ^n×mc

In the formula, L represents the expression feature matrix, S represents the identity feature matrix, and c represents the number of categories; the expression features and identity features of the expression image are separated, and the dictionary learning is performed separately to obtain the double-dictionary mode of the intra-class related dictionary and the difference structure dictionary; The expression feature matrix is a low-rank matrix, and the identity feature matrix is a sparse matrix;

S4: Reconstruct the identity features and expression features of the test image through the double dictionary, and introduce a unified coding error to represent the noise information, and finally realize the classification according to the contribution of each category of expression features in the joint sparse representation.

Further, the specific process of step S4 is as follows:

S4.1: Build a classification method model based on double dictionaries and error collaborative representation as follows:

sty＝Ax ₁ +Bx ₂ +e

In the formula, e is expressed as follows: e=e _a +e _b +es , _{x 1} in the formula is the sparse coefficient vector of dictionary A, x ₂ is the sparse coefficient vector of dictionary B, dictionary A represents the dictionary of expression features, and B represents identity Feature dictionary, e _a represents the error of reconstructing expression features, e _b represents the error of reconstructing identity features, and e _s represents the error caused by occlusion; λ, η, β all represent weight coefficients, which can be adjusted appropriately;

S4.2: Use inaccurate Lagrangian to solve and optimize the above classification method model based on double dictionaries and error collaborative representation, the optimization function is as follows:

In the formula, φ is a vector of Lagrange multiplication, and ξ is a penalty parameter;

The coefficients and noise information of the test expression image are obtained by optimizing the function, and the information of the identity part and the occlusion part of the test image is reconstructed to remove the part irrelevant to the expression and reduce the redundant part of the data;

dy=y-Bx ₂ -e

In the formula, dy represents the clean expression feature of the test picture, Bx ₂ represents the reconstructed identity information, e represents the noise information and also includes the error information caused by occlusion;

S4.3: Calculate the category label of the test sample and distinguish the category of the test sample. The formula is as follows:

Where i represents the category, i ∈ (1, c); x ₁ is the sparse coefficient vector of the dictionary A; identity(y) outputs the label of the test sample; δ(i) is a matrix in which the i-th element is 1, and the rest Both are 0;

S4.4: According to the calculated category labels, classify the sample pictures into their categories, and calculate the recognition rate of the test set according to the classification results.

Further, the database is CK+ database and KDEF database.

Further, step S3 selects 6 facial expression images.

Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

1. The present invention improves the accuracy of sparse representation and enhances the robustness of recognition by decomposing the internal structure information features of the image.

2. By learning the dictionary on the low-rank part and the sparse part, the intra-class correlation dictionary and the difference structure dictionary are obtained, which reduces the size of the dictionary and reduces the problems of over-fitting and instability in the dictionary training process. In addition, the sparse representation of the double dictionary is more adaptable, the sparse structure is more effective, compact, and expressive.

3. In sparse coding classification, a single error matrix is defined to represent the error caused by occlusion information. At the same time, the double dictionary is used to jointly represent the test picture, which complements the facial expression characteristics of the expression image, thereby alleviating the problem caused by the lack of local expression information. coming impact.

Description of drawings

Fig. 1 is a frame diagram of the algorithm described in the present invention.

Fig. 2 is a schematic diagram of simulating random occlusion according to the present invention.

Fig. 3 is a schematic diagram of training samples according to the present invention.

Fig. 4 The recognition rate of the algorithm of the present invention and the comparison with the recognition rate of similar algorithms.

Detailed ways

The technical solutions of the present invention will be further described below in conjunction with the accompanying drawings and embodiments.

Example 1

Referring to Figure 1, a method for occlusion expression recognition combined with a double dictionary and an error matrix, the specific steps are as follows:

Step 1: Data preprocessing. In this embodiment, the CK+ expression database and the KDEF expression database are selected as sample sets. The CK+ database is composed of 6 facial expression image sequences of 210 subjects. Six expressions of anger, disgust, fear, happiness, sadness, and surprise are selected as experimental data, and 30 video sequences of different subjects are randomly selected in the database for each expression. The last one constitutes the training set, a total of 180 pictures, and then selects 6 expressions of 50 subjects to form a test set, a total of 300 pictures. The KDEF data set contains 70 people, 35 males and 35 females, and each expression has 5 angles. The experimental data selected by this method are all frontal face images. In the KDEF database, the front face images randomly select 30 different objects with 6 kinds of expression images as the training set, and the remaining 40 objects with 6 kinds of expression images as the test set. Since the expression images in the CK+ database have slightly tilted heads and emotions of different sizes, this method aligns the horizontal plane of the eyes by rotating them, and crops only frontal faces from the original expression images according to the distance between the two eyes. In the rectangular area of facial expression, redundant information such as hair and neck are removed. All the experiments in this embodiment are carried out under the condition that the training set images are not occluded and the test images are occluded. The simulated random occlusion process is shown in FIG. 2 .

Step 2: Occlusion simulation, the preprocessed data set is subjected to occlusion simulation by adding black blocks of different sizes at different positions;

Step 3: Training phase. The expression features and identity features are obtained by low-rank decomposition of the six expression images in the training samples, in which the expression features and identity features are represented by a matrix; the training sample data schematic diagram is shown in Figure 3. _Given a set of training samples X=[X ₁ , X ₂ , . The objective function looks like this:

Low-rank decomposition is performed on each type of training samples respectively, and L and S are obtained after decomposition.

L=[L ₁ , L ₂ , . . . , L _c ]∈R ^n×mc

S=[S ₁ , S ₂ , . . . , S _c ]∈R ^n×mc

In the formula, L represents the expression feature matrix, S represents the identity feature matrix, and c represents the number of categories. rank() is a function, rank(L) calculates the rank of the matrix L; γ is a constant greater than 0, used to balance the rank of the matrix and the sparsity of the noise matrix; n and m represent the size of the image; R ^n×mc is a set of real numbers, representing the size range of the matrix;

Step 4: Classification method based on double dictionary and error co-representation. For any facial expression picture, it not only contains facial expression information, identity information, but also other information such as noise and illumination. Perform dictionary learning on the low-rank matrix and sparse matrix in step 3 to obtain a double dictionary, and use the k-svd dictionary learning algorithm to perform dictionary learning on L and S to obtain a dictionary A that can capture class discriminant features and a dictionary B that reflects class changes, k -svd algorithm mainly improves dictionary update speed and reduces complexity. Since the dictionaries A and B are composed of non-occluded facial expression images, which can well represent facial expressions, it will cause a huge encoding deviation when the test image has partial occlusions. Therefore, this example introduces a unified discriminant reconstruction error constraint during sparse coding, which can effectively reduce the interference of noise and occlusion while better retaining the structural information of the original data. This method proposes a new model to represent the test image, as follows:

sty＝Ax ₁ +Bx ₂ +e

In the formula, e is expressed as follows: e=e _a +e _b +es , _{x 1} in the formula is the sparse coefficient vector of dictionary A, x ₂ is the sparse coefficient vector of dictionary B, dictionary A represents the dictionary of expression features, and B represents identity Feature dictionary, e _a represents the error of reconstructing expression features, e _b represents the error of reconstructing identity features, and e _s represents the error caused by occlusion; λ, η, β all represent weight coefficients, which can be adjusted appropriately.

This example uses inexact Lagrange to solve the above optimization problem, and the optimization function is as follows:

In the formula, φ is a vector of Lagrangian multiplication, and ξ is a penalty parameter. The specific solution method is as follows. Solution process: non-exact ALM solution

Input: A, B, y, λ, β, η,

Solution process: non-exact ALM solution

While there is no convergence do;

1. Fix other values and update e:

2. Fix other values and update x1 _;

3. Fix other values and update x ₂ ;

4. Update φ;

φ＝φ+ξ(y-Ax ₁ -Bx ₂ -e)

5. Update ξ, p=1.5 is the coefficient

ξ=min(pξ,ξ _max )

Judging whether the convergence condition ε=10 ^-6 is satisfied:

end

Output: x ₁ , x ₂ , e.

By optimizing the function, the coefficient and noise information of the test expression image can be obtained, and the information of the identity part and the occlusion part of the test picture can be reconstructed. Remove the part irrelevant to the expression and reduce the redundant part of the data.

dy=y-Bx ₂ -e

In the formula, dy represents the clean expression feature of the test picture, Bx ₂ represents the reconstructed identity information, and e represents the noise information and also includes the error information caused by occlusion.

In order to distinguish the category of the test sample, the expression information of the test sample is reconstructed in combination with the expression dictionary A. Which type of dictionary can better reconstruct the test sample, and the test sample is classified into the corresponding category. The specific formula is as follows:

Where i represents the category, i ∈ (1, c); x ₁ is the sparse coefficient vector of the dictionary A; identity(y) outputs the label of the test sample; δ(i) is a matrix in which the i-th element is 1, and the rest Both are 0. Calculate the final recognition rate of the entire test set according to the recognition results of each test sample.

In order to verify the effectiveness of this paper's joint dual dictionary and error collaborative representation classification method, this paper and the classic sparse representation classification method were tested on a randomly occluded expression dataset. The specific results are shown in Figure 4.

The same or similar reference numerals correspond to the same or similar components;

The terms describing the positional relationship in the drawings are only for illustrative purposes and cannot be interpreted as limitations on this patent;

Apparently, the above-mentioned embodiments of the present invention are only examples for clearly illustrating the present invention, rather than limiting the implementation of the present invention. For those of ordinary skill in the art, other changes or changes in different forms can be made on the basis of the above description. It is not necessary and impossible to exhaustively list all the implementation manners here. All modifications, equivalent replacements and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the claims of the present invention.

Claims (4)

1. A method for recognizing occlusion expressions by combining double dictionaries and an error matrix is characterized by comprising the following steps:

s1: data preprocessing, namely acquiring a data sample set from a database, aligning the horizontal planes of eyes by rotating the sample set, cutting out a rectangular area only containing facial expressions from an original expression image according to the distance between the two eyes, and removing redundant information of hair and neck;

s2: performing occlusion simulation, namely performing occlusion simulation by adding black blocks with different sizes at different positions to the preprocessed data sample set, wherein the data sample after occlusion simulation is used as a training sample;

s3: in the training stage, performing low-rank decomposition on a plurality of expression image training samples to obtain expression characteristics and identity characteristics, wherein the expression characteristics and the identity characteristics are expressed by adopting matrixes; given a set of training samples X = [ X = ₁ ，X ₂ ...，X _c ]The decomposition function is as follows:

respectively carrying out low-rank decomposition on each type of training sample to obtain L and S after decomposition;

L＝[L ₁ ，L ₂ ，..，L _c ]∈R ^n×mc

S＝[S ₁ ，S ₂ ，..，S _c ]∈R ^n×mc

in the formula, L represents an expression characteristic matrix, S represents an identity characteristic matrix, and c represents the number of categories; separating the expression features and the identity features of the expression image, and respectively performing dictionary learning to obtain a double-dictionary mode of an intra-class related dictionary and a difference structure dictionary; the expression characteristic matrix is a low-rank matrix, and the identity characteristic matrix is a sparse matrix;

s4: the identity features and the expression features of the test image are reconstructed through the double dictionaries, a coding error is introduced to express noise information, and finally classification is achieved according to the contribution of each category of expression features in the joint sparse representation.

2. The method for recognizing the occlusion expression by combining the double dictionaries and the error matrix according to claim 1, wherein the step S4 specifically comprises the following steps:

s4.1: a classification method model based on double dictionaries and error collaborative representation is constructed as follows:

s.t.y＝Ax ₁ +Bx ₂ +e

wherein e represents as follows: e = e _a +e _b +e _s X in the formula ₁ Sparse coefficient vector, x, of dictionary A ₂ Sparse coefficient vectors for dictionary B, dictionary A representing dictionary of expressive features, dictionary B representing dictionary of identity features, e _a Error representing the reconstructed expressive features, e _b Error representing the identity of the reconstruction, e _s Representing errors due to occlusion; λ, η, β all represent weight coefficients, which can be adjusted; y represents a test sample;

s4.2: and solving and optimizing the classification method model based on the double dictionaries and the error co-expression by using non-precise Lagrangian, wherein the optimization function is as follows:

phi in the formula is a vector of Lagrange multiplication, and xi is a penalty parameter;

obtaining the coefficient and noise information of the test expression image through an optimization function, and removing the part irrelevant to the expression by reconstructing the information of the identity part and the shielding part of the test image to reduce the redundant part of data;

dy＝y-Bx ₂ -e

dy in the formula represents the clean expression characteristic of the test picture, bx ₂ Representing the reconstructed identity information, and e representing noise information and also including error information caused by shielding;

s4.3: calculating a class label of the test sample, and judging the class of the test sample, wherein the formula is as follows:

where i represents a category, i ∈ (1,c); x is the number of ₁ Sparse coefficient vector of dictionary A; identity (y) outputs a label of the test specimen; δ (i) is a matrix in which the ith element is 1 and the rest are 0;

s4.4: classifying the sample pictures according to the calculated class labels, and calculating the recognition rate of the test set according to the classification result.

3. The method for recognizing the occlusion expressions by combining double dictionaries and error matrices as claimed in claim 1, wherein the databases are a CK + database and a KDEF database.

4. The method for recognizing the occlusion expression by combining the double dictionaries and the error matrix as claimed in claim 1, wherein 6 expression images are selected in the step S3.

Images