JP2019517693A - System and method for facial expression recognition and annotation - Google Patents

Abstract

The invention disclosed and claimed herein includes, in its aspects, systems and methods for identifying AU and emotional categories in an image. The system and method utilized a set of images including images of people's faces. The system and method analyzes the face image to determine AU and complexion due to facial blood flow fluctuations representing emotional categories. In aspects, the analysis may include AU, AU intensity, and Gabor transforms to determine emotional categories. In other aspects, the systems and methods can include AU, AU intensity, and chromatic variance analysis to determine emotional categories. In a further aspect, the analysis can include AUs, emotional categories, and deep neural networks trained to determine their strength.

Description

This invention was made with government support under Grant Nos. R01-EY-020834 and RO1-DC-014498 awarded by the National Institute of Ophthalmology and the National Institute of Hearing and Communication Disorders. Both institutions are part of the National Institutes of Health. The government has certain rights in the invention.

  This application claims the benefit of US Provisional Patent Application No. 62 / 343,994, entitled "System and Method for Facial Expression Recognition and Annotation," filed June 1, 2016. The entire application is incorporated herein by reference.

  Basic research on the theory of facial perception and emotion can affect large annotated databases of images of emotional expressions and video sequences. Some of the most useful and commonly needed annotations are action units (AU), AU intensity, and emotion categories. Small and medium-sized databases can be manually annotated by expert coders over several months, but not for large databases. For example, even if the expert coder could annotate each face image very quickly (for example 20 seconds per image), it would take 5.556 hours to code a million images, It can be converted to 694 days (8 hours) or converted to work 2.66 years without work.

  Existing algorithms do not recognize all AUs of all applications, do not specify AU strengths, or require excessive computations in space and / or time to handle large databases, or Tested only in a particular database (eg, even if multiple databases are used, training and testing is usually done separately in each database).

  The present disclosure provides computer vision and machine learning processes for recognizing action units (AUs), their strengths, and a large number (23) of basic and complex emotion categories across databases. Importantly, the illustrated process is the first to provide reliable recognition of AUs and their strengths across databases, and is performed in real time (30 images / sec). These features facilitate the automatic annotation of large-scale databases of one million expressions of "in the wild" emotional images. This is an achievement that can not be achieved with other systems.

  In addition, the image is semantically annotated with 421 emotional keywords.

  A computer vision process is presented for the recognition of AU and AU intensity in facial images. Among other things, the process can reliably recognize AU and AU strength across databases. Also described herein, several databases can be used to train the process to better recognize AU and AU intensities on independent image databases that are not used to train the classifier of the present invention. Demonstrated. Furthermore, the process is used to automatically build and annotate a large database of emotional expression images. Images are annotated with AU, AU intensity and emotion categories. The result is a database of 1 million images that can be easily queried by AU, AU intensity, emotional category and / or emotional keywords.

  In addition, the process facilitates a comprehensive computer vision process for identifying AUs from color features. To this end, color features can be successfully exploited for AU recognition, leading to better results than those obtained with the aforementioned system. That is, the functions that define the change in color as the AU changes from inactive to active or vice versa are consistent within the AU and the differences between them are different. In addition, the process reveals how facial color variations can be used to identify the presence of AUs in videos taken under a wide variety of imaging conditions.

  In addition, the color of the face is used to determine the emotion of the facial expression. As mentioned above, human emotional expressions are created by contracting one's face muscles, commonly called action units (AU). Furthermore, the surface of the face is also innervated by a large network of blood vessels. For example, anger increases blood flow to the face, resulting in a red face, but fear is associated with blood drain from the face, resulting in a pale face. These visible face colors allow interpretation of emotions in facial expression images, even in the absence of facial muscle activation. Since this color signal is independent of what the AU provides, the algorithm can detect emotion and color from the AU independently.

  Furthermore, a global local loss function for deep neural networks (DNN) is presented, which can be efficiently used for fine point detection of similar subject landmark interest points and AU and emotional categories. The derived local and global losses provide accurate local results without the need to use a patch based approach, and provide rapid and desirable convergence. The present global local loss function may be used for AU and emotion category recognition.

  In some embodiments, face recognition and annotation processes are used in clinical applications.

  In some embodiments, face recognition and annotation processes are used to detect psychopathological assessments.

  In some embodiments, the face recognition and annotation process is used to screen for post-traumatic stress disorders, such as in military installations or emergency rooms.

  In some embodiments, face recognition and annotation processes are used to teach children with learning disabilities (eg, autism spectrum disorders) to recognize facial expressions.

  In some embodiments, the face recognition and annotating process may be performed on people's responses at the sports arena for advertising, for example, for analyzing people who are watching advertising, for analyzing people who are watching movies, etc. Used for analysis.

  In some embodiments, face recognition and annotation processes are used for monitoring.

  In some embodiments, recognition of emotions, AUs and other annotations is used to improve or identify web searches, for example, the system can be used to image surprises or to show specific people with eyebrows Used to identify an image.

  In some embodiments, the face recognition and annotation process is used to monitor, evaluate or determine customer behavior at a retail store.

  In some embodiments, face recognition and annotation processes are used to organize personal photos of a person, for example by emotion or AU, to organize electronic photographs of facilities or individuals.

  In some embodiments, face recognition and annotation processes are used to monitor the patient's emotions, pain and mental status in a hospital or clinical setting, for example to determine the level of patient discomfort.

  In some embodiments, face recognition and annotation processes are used to monitor driver behavior and attention to roads and other vehicles.

  In some embodiments, face recognition and annotation processes are used to automatically select pictograms, stickers or other text message emotional components.

  In some embodiments, face recognition and annotation processes are used to improve the on-line survey, for example, to monitor the emotional response of the on-line survey participant.

  In some embodiments, face recognition and annotation processes are used in online education and tutoring.

  In some embodiments, the face recognition and annotation process is used to determine that the job seeker's fit is a particular company, eg, a company is looking for alert participants but another company is entertaining I am interested in personality. In another example, face recognition and annotation processes are used to determine an individual's ability during an interview or online video resume.

  In some embodiments, face recognition and annotation processes are used in games.

  In some embodiments, the face recognition and annotation process is used to evaluate a patient's response at a psychiatric office, clinic or hospital.

  In some embodiments, face recognition and annotation processes are used to monitor infants and children.

  In one aspect, computer-implemented methods are disclosed (eg, in real time, eg, to analyze images to determine AU and AU intensities). The method includes maintaining one or more kernel vector spaces (eg, kernel vector space) of the configuration or other shape features and shading features in memory (eg, persistent memory). Each kernel space is associated with one or several action units (AUs) and / or AU intensity values and / or emotion categories. Receive an image to be analyzed (e.g., an image of an expression from outside or from one or more databases). For each image to be received: i) morphological features of the face in the image, shape features, and shading features (e.g., the face space includes shape feature vectors of the feature features and shading feature vectors associated with face shading changes) Determining the face space data (e.g. face vector space) of ii and comparing the determined face space data of morphological features with a plurality of kernel spaces to determine the presence of AU, AU intensity and emotion categories To determine one or more AU values for the image.

  In some embodiments, the method includes processing a video stream comprising the plurality of images in real time to determine an AU value and an AU intensity value for each of the plurality of images.

  In some embodiments, face space data includes shape feature vectors of morphological features and shading feature vectors associated with shading changes of the face.

  In some embodiments, the determined face space of shape, shape and shading features is: i) a distance value between normalized landmarks of the Delaunay triangle formed from the image (eg Euclidean distance) and ii) It includes the distances, areas and angles defined by each Delaunay triangle corresponding to the normalized facial landmarks.

  In some embodiments, shading feature vectors associated with face shading changes are modeled by applying a Gabor filter to normalized landmark points determined from the face (eg, modeling shading changes due to local deformation of the skin Be determined).

  In some embodiments, shape feature vectors of morphological features are backpropagated to both local and global matches of landmark points and / or AUs and / or emotion categories projected onto the image. And includes landmark points derived using a deep neural network (eg, convolutional neural network, DNN) including global local (GL) loss functions configured in.

  In some embodiments, the method comprises, for each received image: i) determining a face space associated with color features of the face; and ii) a plurality of colors or kernel vector space faces for the determined color face space Determining one or more AU values of the image by comparison with iii) the face appears to be expressing a particular emotion, or has one or more AU active or Modifying the color of the image to have a specific intensity.

  In some embodiments, the AU value and the AU intensity value together define emotion and emotion intensity.

  In some embodiments, the image comprises a picture.

  In some embodiments, the image comprises a frame of a video sequence.

  In some embodiments, the image comprises the entire video sequence.

  In some embodiments, the method includes receiving an image of the expression in the wild (eg, the Internet). The received image is processed to determine the AU and AU intensity values of the face and the emotion category in the received image.

  In some embodiments, a method receives a first plurality of images from a first database, receives a second plurality of images from a second database, and receives a first plurality of images and a second received. Processing the plurality of images to determine, for each of the images, an AU value and an AU intensity value of the face in each respective image. The first plurality of images have a first captured configuration and the second plurality of images have a second captured configuration. The first acquisition form is different from the second acquisition form (for example, the acquisition form is the illumination method and size, image background, focal plane, capture resolution, storage compression level, capture pan for face, tilt, And yaw etc.)

  In another aspect, a computer-implemented method is disclosed (eg, for analyzing images to determine AU, AU intensity and emotional category using color changes in the image). The method identifies a change that defines a transition from inactive to active of the AU, the change being selected from the group consisting of chromaticity, hue and saturation, and luminance, and the identified chromaticity change Applying the Gabor transform to (eg, to obtain invariance to the minimum value of this change between facial expressions).

  In another aspect, a computer-implemented method for analyzing images to determine AU and AU intensity is disclosed. The method maintains in memory (eg, persistent memory) a plurality of color feature data associated with AU and / or AU intensities, receives an image to be analyzed, and for each image received, i) a face in the image Determining the morphological color feature of the image, and ii) comparing the determined morphological color feature with the plurality of trained color feature data, and determining one or more of the trained color feature data of the determined morphological color features Determining one or more AU values for the image by determining the presence at.

  In another aspect, a computer-implemented method is disclosed (e.g., to generate a repository of a plurality of face space data associated with an AU value and an AU intensity value, respectively), the repository comprises an image or video for AU and AU intensity. Used for classification of frame face data). The method includes analyzing multiple faces in an image or video frame to determine kernel spatial data for multiple AU values and AU intensity values. Each kernel space data is associated with a single AU value and a single AU intensity value, and each kernel space can be separated linearly or non-linearly from other kernel plane spaces.

  In some embodiments, analyzing multiple faces to determine carnations, kernel subclassing to generate multiple AU training sets for a predetermined number of AU intensity values, and determine multiple kernel spaces Performing an analysis, each of the plurality of kernel spaces correspond to a given AU value, an AU intensity value, an emotion category, and the intensity of the emotion.

  In some embodiments, the kernel space includes functional color space feature data of an image or video sequence.

  In some embodiments, the functional color space is generated by performing a discriminant function learning analysis on color images respectively derived from a given one of the plurality of images (eg, maximum margin functional classifier Determined).

  In another aspect, a non-transitory computer readable medium is disclosed. The computer readable medium stores instructions, which, when executed by the processor, cause the processor to perform any of the methods described above.

  In another aspect, a system is disclosed. The system comprises a processor and a computer readable medium having instructions stored thereon, the instructions, when executed by the processor, causing the processor to perform any of the methods described above.

FIG. 1 is a diagram illustrating the output of a computer vision process for automatically annotating emotional categories and all in a natural face image.

FIG. 2, including FIGS. 2A and 2B, is a diagram of Delaunay triangulation of detected facial landmarks and images.

FIG. 3 shows a hypothetical model in which a sample image with active AUs is divided into subclasses.

FIG. 4 shows an exemplary component diagram of a system that uses Gabor transforms to determine AUs and emotion categories.

FIG. 5 shows a color distribution system for detecting AUs using color features in video and / or still images.

FIG. 6 shows a color distribution system for detecting AUs using color features in video and / or still images.

FIG. 7 shows a network system for detecting AUs using deep neural networks in video and / or still images.

FIG. 8 shows an exemplary computer system.

  Real-time algorithm for automatic annotation of one million untouched facial expressions

  FIG. 1 shows a result database of facial expressions that can be easily queried (eg, sorted, organized, etc.) by AU, AU intensity, emotion category, or emotion / impact keywords. This database facilitates the design of new computer vision algorithms and basic, transitional and clinical research in social and cognitive psychology, social and cognitive neuroscience, neuromarketing, psychiatry, etc.

  The database is compiled from the output of a computer vision system that automatically annotates emotions about categories and AUs in pristine face images (ie, images that have not yet been curated in existing databases). Images cars are downloaded using various web search engines by selecting only images with emotional keywords associated with faces in WordNet or other dictionaries. FIG. 1 shows three example queries against a database. The top example is the result of two queries acquired when acquiring all images identified as happiness and fear. Also shown is the number of images in the database of pristine images annotated as either happy or fear. The third query shows all images in which AU 4 or 6 exist, and the results of searching for all the images with emotional keywords “anxiety” and “disapproval”.

  AU and strength recognition

  In some embodiments, a system for recognizing AUs can process over 30 images per second and is determined to be very accurate across a database. This system achieves high recognition accuracy between databases and can be implemented in real time. The system can facilitate classifying expressions into one of 23 basic and / or complex emotion categories. Emotional classification is given by the detected AU activation pattern. In some embodiments, the image may not belong to one of 23 categories. In this case, the image is annotated with AU without the emotion category. If there is no active AU in the image, the image is classified as a neutral expression. In addition to determining emotion and emotional intensity in the face, the illustrated process can be used to identify "not face" in the image.

  Face space for AU and strength recognition

  The system starts by defining a feature space that is used to represent AUs in the face image. Human perception of faces, particularly facial expressions, is known to include a combination of shape analysis and shading analysis. The system can define shape features that facilitate the recognition of emotional expressions. Shape features may be quadratic statistics of facial landmarks (i.e., the distance and angle between landmark points in the face image). Features can alternatively be referred to as morphological features, as they define facial features. It is understood that in the present application these terms may be used interchangeably.

Figure 2 (a) shows the normalized facial landmarks used by the proposed algorithm
An example of Several (eg, fifteen) landmarks can correspond to anatomical landmarks (eg, eye corners, mouth, eyebrows, nose tips, and jaws). Other landmarks may be pseudo landmarks that define the edge of the eyelid, mouth, eyebrow, lip, and chin lines, and the midline of the nose from the tip of the nose to the horizon provided by the center of the two eyes . The number of pseudo-landmarks that delineate each facial component (e.g., eyebrows) is constant, which provides landmark position equality to different faces or people.

FIG. 2 (b) shows the Delaunay triangulation performed by the system. In this example, the number of triangles in this configuration is 107. The image also shows the vector angles θ _a = (θ _a1 ,..., Θ _aqa ) ^T (q _a = 3), where the angle θ _a is the angle of the triangle coming out of the normalized landmark
Define

Can be a vector of landmark points in the j ^th sample image of AUi (j = 1,..., N _i ). here,

Is the 2D image coordinates of the k ^th landmark. n _i is the number of sample images in which AU _i exists. In some embodiments, facial landmarks may be obtained using computer vision algorithms. For example, if the computer vision algorithm has 66 landmarks
In, as shown in FIG. 2a, any number of landmarks (eg, 66 detected landmarks in a test image) can be used to automatically detect.

The training image can be normalized to have the same interocular distance of τ pixels. In particular,

age,

And l and r are the image coordinates of the center of the left and right eyes,
Is the vector

And define a 2 norm of τ = 300. The position of the center of each eye can be easily calculated as the geometric midpoint between landmarks that define the two corners of the eye.

The shape feature vectors of the constituent features can be defined as follows.

_{Let E be} the Euclidean distance between normalized landmarks a = 1,..., P−1, b = a + 1, p, and θ _a = (θ _a1 ,..., Θ _aqa ) ^{T be}
and
Normalized landmarks with numbers q _a starting at
The angle defined by each Delaunay triangle coming out of (the equivalence holds for boundary points without boundaries). Since each triangle in this figure can be defined by three angles, there are 107 triangles in this example, so the total number of angles in the shape feature vector is 321. Shape feature vector is
Delaunay triangulation p is the number of landmarks and t is the number of triangles. In this example, p = 66, t = 107 and the vector

It is.

The system has normalized landmark points

A Gabor filter centered on each of the can be used to model shading changes due to local deformation of the skin. When the facial muscles deform the skin of the face locally (for example, the bidirectional reflectance distribution function of the skin is defined as a function of the wrinkles of the skin, which means that light is transmitted between the epidermis and the dermis) This may change the way it moves, which may also change the level of hemoglobin.) From the point on the surface of the skin, the reflectance characteristics of the skin change and the light source becomes shorter.

Human early visual cortex cells can be modeled by the system using Gabor filters. Facial perception can use Gabor-like modeling to obtain invariance to the change in intensity as seen when expressing emotions. It can be defined as follows.

Is the wavelength (ie, cycles / pixel), α is the direction (ie, the angle of the normal vector of the sine function), φ is the phase (ie, the offset of the sine function), γ is the (spatial) aspect ratio, σ is Filter scale (standard deviation of Gaussian window).

In some embodiments, Gabor filter banks can be used with o orientation, s space scale, and r phase. In the Gabor filter example, it is set as follows.
γ = 1. Values are suitable for expressing emotional expressions. The values of o, s and r are learned using cross-validation of the training set.

I _ij is the j ^th sample image with AU i present, and as the feature vector of Gabor response at the k ^th landmark point

, * Defines the convolution of the filter g (.) With the image I _ij , λ _k is the k ^th element of the set λ defined above. The same applies to α _k and φ _k , but this is generally 1 and not to γ.

Feature vector of Gabor response on all landmark points AUi is for j ^th th sample image is active, is defined as follows.

The feature vectors define the shading information of local patches around landmarks of the face, and their dimensionality is g _ij ∈R ^{5 × p × o × s × r} .

The final feature vector defining the shape and shading change of AUi in the face space is defined as follows.

  Classification of face space for AU and strength recognition

The system can define the training set of AUi as
For j = 1, ..., n _i y _ij = 1, indicating that AU _i is present in the image, j = n _i + 1, ..., n _i + m _i for y _ij = 0, indicates that AUi is not present in the image, m _i is the number of sample images AUi is not active.

The above training set is ordered as follows. set
The strength a (i.e., a minimum intensity of the activation of the AU) Fukumumi the _{n ia} sample with active AUi in the set


Are n _ib samples with active AU i at intensity b (the second lowest intensity).

set


Is a n _ics samples with active AUi intensity c (next intensity).

set

Is an n _id sample with an active AUi at intensity d (which is the highest intensity) and n _ia + n _ib + n _ic + n _id = n _i .

AU can be activated at 5 intensities, which can be labeled a, b, c, d or e. In some embodiments, there are rare cases with intensity e, so in some embodiments, the other four intensities are sufficient. Otherwise, D _i (e) defines the fifth intensity.

The four training sets defined above are subsets of Di and can be represented as different subclasses of the set of images on which AUi is active, can use subclass based classifiers, and the system derives this process Use kernel subclass discriminant analysis (KSDA). KSDA can be used because it can reveal complex non-linear classification boundaries by optimizing the number of kernel matrices and subclasses. KSDA can optimize the classification criteria to optimally separate the classes. This criterion is formally given by Q _i (φ _i , h _i1 , h _i2 ) = Q _i1 (φ _i , h _i1 , h _i2 ) Q _i2 (φ _i , h _i1 , h _i2 ), Q _i1 (Φ _i , h _i1 , h _i2 ) is responsible for the maximization of isodispersity. The purpose of the kernel map is to find a kernel space F in which the data is linearly separable, and in some embodiments, the subclasses can be linearly separable in F, where the class distributions share the same variance It can be. Q _i2 (φ _i , h _i1 , h _i2 ) maximizes the distance between all subclass averages (ie, it is used to find a Bayesian classifier with minimal Bayesian error).

To see this recollection, recall that Bayesian classification boundaries are given at locations in the feature space where the probabilities of the two normal distributions are identical (eg p (z | N (μ ₁ , _{1 1} )) = p (z | N (μ ₂ , _{2 2} )) and N (μ _i , _{i i} ) are normal distributions with an average μ _i and a covariance matrix Σ _i . For example, the equation p (x | N (μ ₁ , _{1 1} )) = p (x | N (μ ₂ , _{2 2} )) is given a lower probability value than before. Bayesian error is reduced.

Thus, the first element of the above KSDA criteria is


Given by

Is the subclass covariance matrix in kernel space defined by the mapping function (ie, the covariance matrix of the samples in subclass l), j _i (.): R ^e → F, h _{i 1} is in the image H _i2 is the number of subclasses representing AUi, hi2 is the number of subclasses representing AUi and is not included in the image, e = 3t + p (p-1) / 2 + 5 × p × o × s × r is the number of dimensions of the feature vector in the face space defined in the section on face space.

The second element of the KSDA standard is



Where p _il = n _l / n _i is the previous to subclass l of class i (ie the class defining AUi) and n _l is the number of samples of subclass l,
Is the sample mean of subclass l of class i in kernel space defined by the mapping function j _i (.).

For example, the system can define the mapping function φi (.) Using a radial basis function (RBF) kernel,

ν _i is the variance of the RBF, j ₁ , j ₂ = 1, ..., n _i + m _i . Therefore, the KSDA based classifier of the present invention is given by the following solution.

  FIG. 3 shows the solution of the above equation to yield a model of AUi. In the above hypothetical model, the sample image with AU4 active is first divided into four subclasses, each subclass containing a sample of AU4 of the same intensity (a-e). Next, the derived KSDA based approach essentially uses the process of further subdividing each subclass into additional subclasses, such that the above normal distributions are linearly separated and the data in kernel space as far apart as possible. Find the kernel mapping to map.

To do this, the system divides the training set D _i into five subclasses. The first subclass (i.e., l = 1) comprises sample feature vectors corresponding to the image with active AUi at intensity a, ie, D _i (a) is incorporated herein by reference in its entirety S. Du, Y. Tao and AM Martinez, "Emotional Expressions in Complex Expressions," National Academy of Sciences Proceedings 111 (15): E1454-E1462, 2014. The second subclass (l = 2) contains sample subsets. Similarly, the third and fourth subclasses (l = 2, 3) each contain a sample subset. Finally, five subclasses (l = 5) are images for which AUi is not active, eg


Contains sample feature vectors corresponding to.

Thus, initially, the number of subclasses to define AUi active / inactive is 5 _{(i.e., h i1} = 4 and _h i2 = 1). In some embodiments, this number may be larger. For example, suppose that an image of intensity e is considered.

Optimization of Equation 14 provides additional subclasses. The derived approach optimizes the parameters of kernel map u _i as well as the number of subclasses h _i1 and h _i2 . In this embodiment, the first (five) subclasses can be further subdivided into subclasses. For example, if the kernel parameter _{i i} can not map non-linearly separable samples in D _i (a) into a space that can be linearly separated from other subsets, then Di (a) further comprises two subsets D _i (a) _{) = {D i (a 1} ), D i (a 2)} is split into. This division is simply given by nearest neighbor clustering. Formally, if the sample z _{i j + 1} is the closest to z _ij , then the division of Di (a) is


It is easily given by

The same applies to D _i (b), D _i (c), D _i (d), D _i (e), D _i (inactive). Thus, optimizing Equation 14 may result in multiple subclasses for modeling samples of each strength of activation or deactivation of AUi, eg, subclass 1 (l = 1) is Di (a Define the sample of) and divide it into two subclasses (and now h _i1 = 4), the two new first subclasses (the first new two subclasses) with the sample of Di (a ₁ ) It is used to define the sample of Di (a) (and h _i1 becomes 5), using the first subclass including the second subclass (l = ₂ ) in Di (a ₂ ). Subsequent subclasses define the samples to D _i (b), D _i (c), D _i (d), D _i (e), D _i (inactive), as defined above. Thus, the order of the samples given by D _i defines the sample feature vector associated with the image from which subclass 1 to AUi is active _, and h _i1 + h _i2 from subclass h _i1 +1 to which AUi represents an inactive image There is no change in This final result is illustrated using the hypothetical example of FIG.

In one example, all test images in the set of images I _test can be classified. First, I _test includes feature representations within the face space vector z _test that are calculated with respect to the face space as described above. Next, the vector is projected into kernel space and called z ^j _test . In order to determine if this image has an active AUi, the system calculates the closest average,

If j ^* h _i1 , I _test is labeled AUi active, otherwise it is not.

The classification results provide strength recognition. If the sample represented by subclass l is a subset of the samples of Di (a), then the identified intensity is a. Similarly, if the samples of subclass l are a subset of the samples of D _i (b), D _i (c), D _i (d) or D _i (e), then the intensity of the AUi of the test image I _test is b, c, d, e. Of course, j *> For h _i1, there is no AUi the image intensity no (or, it can be said that the strength is zero).

  FIG. 4 shows an exemplary block diagram of a system 400 for performing the functions described above with respect to FIGS. System 400 includes an image database component 410 having a set of images. System 400 includes a detector 420 for removing non-face images in the image database. Create a subset of the image set of the image that includes only the face. System 400 includes training database 430. Training database 430 is utilized by classifier component 440 to classify images into emotion categories. System 400 includes a tagging component 450 that tags an image with at least one AU and emotion category. System 400 can store tagged images in processed image database 460.

  Discriminant Function Learning of Color Features for Facial Action Unit Recognition

  In another aspect, the system facilitates a comprehensive computer vision process for identifying AUs using facial color features. Color features can be used to recognize AU and Au intensities. The functions that define the change in color as the AU changes from inactive to active or vice versa are consistent within the AU and among those differences. In addition, how the system can exploit facial color variations to identify the presence of AUs in videos taken under a wide variety of image conditions and outside of the image database. Reveal.

System, i ^th th sample video sequence _{V i = {I i1, ...} , I iri} to receive. r _i is the number of frames, I _ik ^{∈R 3 qw} is the vectorized k ^th color image of q × w RGB pixels. V _i is described as a sample function f _i (t).

The system uses the algorithm described herein to identify a set of physical facial landmarks on the face and obtain local facial regions. The system defines landmark points in vector format as s _ik = (s _{ik 1} , ..., s _{ik 66} ), i is a sample video index, k is a frame number, s i k _r ² R ² is l ^th , l = 1, l, ..., 66, 2D image coordinates of the landmark. For illustrative purposes, certain exemplary values (eg, 66 landmarks, 107 image patches) can be used.

The system 107 or a set as a set of image patches d _ijk D _ij = obtained in Delaunay triangulation as described above _{{d i1k, ..., d i107k} } defines a, d _ijk ∈ R ^3q _ij is A vector representing the j ^th triangular local region of q _ij RGB pixels, where i specifies sample video numbers (i = 1,..., n) and k is a frame (k =), as described above Specify 1, ..., r _i ).

In some embodiments, the size (ie, the number of pixels, q _ij ) of these local (triangular) regions not only differs between individuals, but also varies within the same person's video sequence. This is the result of the movement of landmark points on the face, the process necessary to produce facial expressions. The system defines a feature space that is invariant to the number of pixels in each of these local regions. The system calculates statistics on the color of the pixels in each local area as follows.

The system calculates the first and second (center) moments of the color of each local region,

Let d _ijk = (d _ijk1 ,..., d _ijkP ) ^T and μ _ijk , σ _ijk ∈R ³ . In some embodiments, additional moments are calculated.

The color feature vector of each local patch can be defined as

i is a sample video index (V _i ), j is a local patch number, and r _i is the number of frames of this video sequence. This feature representation defines the color contribution in patch j. In some embodiments, other proven features can be included to increase feature richness. For example, the response to the filter or the shape feature.

  Color invariant function representation

  The system can define the above calculated color information as a function which is invariant to time. That is, the functional representation is consistent regardless of where in the video sequence the AU is active.

The color function f (.) Defines the color change of the video sequence V, and the template function f _T (.) Models the color change associated with the activation of an AU (ie, from AU inactive to active). The system determines if f _T (.) Is at f (.).

In some embodiments, the system determines this by placing the template function f _T (.) At each possible position in the time domain of f _T (.). This is usually referred to as a sliding window approach, as it involves sliding the window left and right until all possible positions of f _T (.) are identified.

In another embodiment, the system derives a method using Gabor transform. The Gabor transform determines the content of the frequency and phase of the local section of the function to derive an algorithm for finding a match of f _T (.) In f (.) Without using sliding window search It is designed.

In this embodiment, without loss of generality, f (t) is one of the color descriptors, eg, the average of the red channel in the j ^th triangle of video i, or the opposite color representation ( It can be the first channel of the opponent color representation). And the Gabor transform of this function is
And

g (t) is a concave function,
It is. One possible pulse function may be defined as

L is a fixed time length. Other pulse functions can be used in other embodiments. With two equations
And

As a definition of the inner product of the period [0, L], and hence G (.,.), We can write

  <.,.> Are functional inner products. The Gabor transform described above is continuous in time and frequency in the absence of noise.

In order to calculate the color descriptor f _i1 (t) of the i ^th video, all functions have a vector of coefficients
A set of b basis functions that are
Defined in the color space spanned by The functional inner product of two color descriptors is
Can be defined by

Φ is a b × b matrix with elements _ij = (f _i (t), f _j (t)).

  In some embodiments, the models assume that the statistical color properties change smoothly over time, and that their effect on muscle activation has a maximum duration of L seconds. The basis functions that fit this description are the first few components of the real part of the Fourier series, ie the normalized cosine basis. Other basis functions can be used in other embodiments.

The cosine basis can be defined as ψ _z (t) = cos (2πzt), z = 0,. The corresponding normalized basis is defined as follows:

The normalized basis set allows Φ = Id _b . Here, Id _b represents a b × b unit matrix, not an arbitrary positive definite matrix.

The above derivation with cosine basis makes frequency space implicitly discrete. Gabor transform of color function
Becomes as follows,

Is the function computed on the interval [tL, t]
And c _i1z is the z ^th -th coefficient.

  It should be understood that although the system derived above does not include the time domain, it is possible to find and use time domain coefficients as needed.

  Function classification of action unit

  The system uses the Gabor transform derived above to define a feature space that is invariant to AU timing and duration. In the resulting space, the system uses linear or non-linear classifiers. In some embodiments, KSDA, Support Vector Machine (SVM) or Deep Multilayer Neural Network (DN) can be used as a classifier.

  Function color space

  The system includes functions that describe the mean and standard deviation of color information from different local patches, which use simultaneous modeling of multiple functions described below.

System is multidimensional function
And each function γ _z (t) is the mean or standard deviation of the color channel in a given patch. Each using the basis expansion approach
Is defined by the set of coefficients c _ie , so Γ _i (t) is given by

The inner product of multidimensional functions is redefined using a normalized Fourier cosine basis,
It becomes.

  Other bases can be used in other embodiments.

  The system uses a training set of video sequences to optimize each classifier. It is important to note that the system is invariant to the length of the video (ie the number of frames). Thus, the system does not use video alignment or clipping for recognition.

  In some embodiments, the system can be extended to identify AU strengths using the techniques described above and a multi-class classifier. The system can be trained to detect the AU and each of the 5 intensities a, b, c, d, e, and the AU is inactive (not present). The system may also be trained to identify emotional categories in the image of the expression using the same approach as described above.

  In some embodiments, the system can detect AUs and emotion categories in the video. In another embodiment, the system can identify AUs in still images. In order to identify AUs in still images, the system first learns to calculate the above defined functional color features from a single image using regression. In this embodiment, the system regresses the function h (x) = y to map the input image x to the required functional representation of the color y.

  Support vector machine

The training set is defined by {(γ ₁ (t), y ₁ ), ..., (γ _n (t), y _n )}, and γ _i (t) ∈ H ^v , H ^v is the order v Hilbert space of continuous function with bounded derivatives up to y _i ∈ {−1, 1} is a class label, +1 indicates that AU is active and −1 is inactive.

When the separate classes of samples are linearly separable, the function w (t) that maximizes the separability of the classes is given by

v is a bias, as above
Represents a functional inner product, ξ = (ξ ₁ ,..., Ξ _n ) ^T is a slack variable and c> 0 is a penalty value detected using cross validation.

Applying our derived approach to model Γ _i using normalized cosine coefficients with (28) converts (29) to

  c> 0 is the penalty value found using cross validation.

The system projects the original color space onto the first few (eg two) principal components of the data. Principal components are obtained by principal component analysis (PCA). The resulting p-dimensions are labeled φ _PCA k, k = 1,2, ..., p

  Once trained, the system can detect AU, AU intensity and emotion categories in the video in real time or faster than real time. In some embodiments, the system can detect AUs exceeding 30 frames / sec / CPU thread.

  Deep Network Approach Using Multilayer Perceptron

  In some embodiments, the system can include a deep network to identify non-linear classifiers in the color feature space.

The system can train a multilayer perceptron network (MPN) using the coefficients c _i . This deep neural network is composed of several (for example 5) blocks of connected layers with batch normalization and some linear or non-linear functional rectification, for example a rectification linear unit (ReLu). To effectively train the network, the system supersamples minority classes (AU active / AU strength) or downsamples majority classes (AU inactive) to perform data augmentation use. The system can also use class weights and weight attenuation.

  The neural network is trained using a gradient descent method. The resulting algorithm runs in> 30 frames / sec / CPU thread in real time or faster than real time.

  AU detection in still image

In order to apply the system to a still image, the system identifies the color function f _i of the image I _i . That is, the system defines the mapping h (I _i ) = f _i . Where f _i is its coefficient
Defined by In some embodiments, the coefficients can be learned from training data using non-linear regression.

The system utilizes a training set of _m videos {V ₁ ,..., V _m }. As mentioned above, V _i = {I _i1 ,..., I _iri }. The system has a length L (with L _i ), for example W _i1 = {I _i1 ,..., I _iL }, W _i2 = {I _i2 ,..., I _{i (L + 1)} },. ., W _{i (ri-L)} = {I _{i (ri-L),} ..., I _iri } consider all subsets of consecutive frames. The system calculates the color representation of all W _ik as described above. Thus, each _{W ik, k = 1, ...} , x ik = about _{r i -L (x i1k, ...} , x i107k) T is obtained. In the next (19),

i and k designate video W _ik and j, j = 1,..., 107 designate patches.

The system calculates the functional color representation f _ijk, j = 1,..., 107 of each W _ik for each patch. _{This, f ijk = (c ijk1,} ..., c ijkQ) is done using the approach detailed above to bring the _^T, c ^ijkq is, q ^th th j patch video W _ij Is the coefficient of The training set is given by the pair {x _ijk , f _ijk }. The training set is used to regress the function f _ijk = h (x _ijk ). For example, I is a test image in patch j, color representation
I assume. Regression is used to estimate the mapping from image to functional color representation, as defined above. For example, kernel ridge regression is used to estimate the q ^th coefficient of the test image as N.

Is the j ^th patch
Color feature vectors of the j ^th th patch of all training images, K the kernel matrix
It is. System is a radial basis function kernel
Can be used. In some embodiments, the parameters η and λ are selected to maximize accuracy and minimize model complexity. This is equivalent to optimizing the bias and variance trade-off. This system uses a solution to the bias distribution problem as known in the art.

As indicated above, the system can use regressors for test images not previously seen. if
If is a test image not seen before, its functional representation is
When
Easily obtained. This functional color representation can be used directly in the functional classifier derived above.

  FIG. 5 shows a color distribution system 500 for detecting AU or emotion using color distribution in video and / or still images. System 500 includes an image database component 510 having a set of video and / or images. System 500 includes a landmark component 520 that detects landmarks in image database 510. The landmark component 520 creates a subset of the set of images of the image having the defined landmarks. System 500 includes a statistics component 530 that calculates statistics in color changes in the video sequence or still images of the face. From the statistics component 530, an AU or emotion is determined for each video or image in the database component 510 as described above. The system 500 includes a tagging component 540 that tags the image with at least one AU or tags without an AU. System 500 may store tagged images in processed image database 550.

  Face color for recognizing emotions from images of facial expressions and editing face images to make them express different emotions

  In the above method, the system identifies AUs using configuration, shape, shading and color features. This is because AU defines a category of emotions, ie, a unique combination of AUs specifies a category of unique emotions. Nevertheless, the color of the face also conveys emotions. The face can express emotional information to the observer by altering the blood flow on the vascular network closest to the surface of the skin. For example, consider the redness associated with anger and the paleness in fear. These color patterns are caused by blood flow fluctuations and can occur even without muscle activation. Our system detects these color changes, which allows us to identify emotions without muscle movement (ie, whether or not AU is present in the image).

  Face area

The system calculates each face color image of p × q pixels,
Expressed as r landmark points of each facial component of the face
Expressed as two-dimensional coordinates of landmark points on the image. Here, i specifies a subject and j specifies an emotion category. In some embodiments, the system uses r as 66. These reference points define the inner contour and the outer elements of the face, such as the mouth, nose, eyes, eyebrows, eyebrows and marks of the chin. Delaunay triangulation can be used to generate local regions of triangles defined by landmark points on these faces. This triangulation results in several local regions (eg, 142 regions when using 66 landmark points). Let this number be a.

The system can define the function D = {d ₁ ,..., D _a } as a series of functions that return each pixel of the local region of _a . For example, d _k (I _ij ) is the k ^th th Delaunay triangle in the image I _ij , for example
A vector containing l pixels inside of, where
Defines the values of the three color channels of each pixel.

  Color space

The above derivation divides each face image into a series of local regions. The system can calculate the color statistics of each of these local regions in each image. Specifically, the system calculates the first moment and the second moment (i.e., the mean and the variance) of the data, defined as follows.

In other embodiments, additional moments of image color are utilized. All images I _ij are represented using the following feature vectors of color statistics:

Using the same model, the system defines the color feature vectors for each neutral plane as follows:
Here, n indicates that this feature vector corresponds to a neutral expression rather than an emotion category. The average neutral plane is
m is the number of identifiers in the training set. The color representation of emotional facial expressions is given by the deviation from this neutral face.

  Classification

The system uses linear or non-linear classifiers to classify emotion categories in the color space defined above. In some embodiments, linear discriminant analysis (LDA) is calculated in the color space defined above. In some embodiments, the color space can be defined by eigenvectors corresponding to non-zero eigenvalues of the following matrix:
Here, the following is the (normalized) covariance matrix.
The following is the class average.
The following is the identification matrix.
δ = .01 is the normalization parameter, and C is the number of classes.

  In other embodiments, the system can employ subclass discriminant analysis (SDA), KSDA, or deep neural networks.

  Multi-directional classification

  The selected classifier (e.g., LDA) is used to calculate the emotional category of C and the neutral color space (or spaces). In some embodiments, the system is trained to recognize 23 emotion categories, including basic emotions and complex emotions.

The system divides the available samples into 10 different sets S = {S ₁ ,..., S ₁₀ }. Here, each subset _St has the same number of samples. This division is performed such that the number of samples in each emotion category (including neutral) is equal in all subsets. The system repeats the following procedure using 1, ..., 10 t. All subsets except S _t is used to calculate the Σx and S _B. LDA subspace
The samples of subset S _t not used for the calculation of
Projected Feature vector of each test sample
Are assigned to the emotion category of the nearest category average given by Euclidean distance
All test samples
The classification accuracy in is given by:
Where n _t is the number of samples in S _t and y (t _j ) is an oracle function that returns the true emotion category of sample t _j ,
Is 0-1 loss,
It is 1 when it is and 0 otherwise. Thus, S _t serves as a test subset to determine a generalized color model. Since t is 1, ..., 10, the system can repeat this procedure 10 times. In each time, leaving one of the subset S _t for testing. Then, the average classification accuracy is calculated as follows.
The standard deviation of the cross-validated classification accuracy is
This process allows the system to identify the most generalized identification color features, ie, those that apply to images not included in the training set.

  In another embodiment, the system uses a two-way (one-to-all) classifier.

  Class of all pairs

The system assigns the sample of one emotion category (eg, emotion category c) to class 1 (eg, emotions under study), and every time that all other emotion category samples are assigned to class 2, the above method Repeat C times to identify the most distinguishable color feature of each emotion category. Formally, it is the following.

A linear or non-linear classifier (eg, KSDA) is used to distinguish the samples of.

  10-fold cross validation: The system uses the same 10-fold cross validation process and nearest-average classifier as above.

In some embodiments, to avoid bias due to sample imbalance in this two class problem, the system
Can be applied to downsampling. In some cases, the system
Every time I pull a random sample from
Repeat this procedure multiple times to match the number of samples at.

  Discrimination color model

When using LDA as a two-way classifier,
Is a series of discriminable vectors ordered from highest to lowest discriminant
give.
The following discriminant vectors define the contribution of each color feature when identifying emotion categories.
The system only holds v ₁ since this is the only basis vector associated with the non-zero eigenvalue λ ₁ > 0. Thus, the color model of emotion j is given by:

  Similar results can be obtained using SDA, KSDA, deep networks, and other classifiers.

  Image color correction to change the facial expression represented by the face

The neutral expression I _in can be modified by the system to appear to express emotions. These are corrected images
It can be called. Here, i identifies an image or an individual within an image, and j identifies an emotional category.
Corresponds to the following modified color feature vector:
In some embodiments, to generate these images, the system modifies the k ^th th pixel of the neutral image using the color model of emotion j as follows.
Here, I _ink is the k ^th th pixel _in the neutral image I _in .
Is the mean and standard deviation of the color of the pixel within the g ^th th Delaunay triangle.
Is the mean and standard deviation of the color of the pixel at d _g given by the new model y _ij .

  In some embodiments, the system smoothes the modified image with γ by a γ Gaussian filter with variance σ. Smoothing eliminates local shading and shape features, allows people to focus on face color, and makes emotion categories more explicit.

In some embodiments, the system modifies the image of the facial expression of the emotion to increase or decrease the appearance of the expressed emotion. In order to reduce the appearance of emotion j, the system removes the color pattern associated with emotion j and the resulting image
You can get The image is calculated as described above using the following associated feature vectors:

In order to increase the perception of emotion, the system defines a new color feature vector as
Result image
To get

  FIG. 6 shows a chromatic dispersion system 500 for detecting AU or emotion using chromatic dispersion in video and / or still images. System 600 includes an image database component 610 having a set of video and / or images. System 600 includes a landmark component 620 that detects landmarks in image database 610. The landmark component 620 generates a subset of the series of images having the defined landmarks. System 600 includes a statistics component 630 that calculates statistics on color changes in the video sequence or still images of the face. From the statistics component 630, an AU or emotion is determined for each video or image in the database component 610 as described above. System 600 includes a tagging component 640 that tags the image with at least one AU or without AU. System 600 may store tagged images in processed image database 650.

  System 600 includes a correction component 660 that can change the perceived emotion in the image. In some embodiments, after the system 600 determines the neutral face in the image, the correction component 660 corrects the tonality of the neutral face image to produce or correct the appearance of the determined representation of emotion or AU. Do. For example, the image is determined to include a neutral representation. The correction component 660 can change the color in the image to change the expression so as to perceive a predetermined expression such as happiness or sadness.

  In other embodiments, after the system 600 determines the facial emotion or AU in the image, the correction component 660 may adjust the intensity of the emotion or AU to increase or decrease the intensity of the emotion or AU to change the perception of the emotion or AU. Correct the color. For example, an image is determined to include a sad expression. The correction component 660 can change the color in the image such that the expression is perceived as less or more sad.

  Global local matched by DNN for fast and accurate detection and recognition of facial landmark points and action units

  In another aspect, the global-local loss function (DNN) for deep neural networks is not only fine-grained detection of similar target landmark points of interest (eg, facial landmark points), but also target characteristics such as AU. In the fine-grained recognition of, it can be used efficiently. The derived local + overall losses provide accurate local results without the need to use a patch based approach, and provide rapid and desirable convergence. This global-local loss function can be used for AU recognition or to detect faces and facial landmark points needed for AU and facial expression recognition.

  Global-local loss

  Derivation of global-local (GL) loss can be efficiently used in deep networks for detection and recognition in images. The system uses this loss to train the deep DNN to recognize the AU. The system detects facial landmark points using a portion of DNN. These detections are concatenated with the output of fully connected layers of other components of the network to detect AUs.

  Local fit

The system defines the image samples and corresponding output variables as a set of {(I ₁ , y ₁ ),..., (I _n , y _n )}. Here, I _i ∈ R ^{l × m} is an image of al × m pixels in the face, yi is a true (desired) output, and n is the number of samples.

In some embodiments, output variable y _i can be in various forms. For example, in the detection of a landmark point of a 2D object in an image, y _i is a vector of p of coordinates y _i = (u _i1 , v _i1 , ..., u _ip , v _ip ) ^T of the 2D image is there. (u _ij , v _ij ) ^T is the j ^th landmark point. For AU recognition, the output variable corresponds to the index vector y _i = (y _i1 ,..., Y _iq ) T. If AUj is present in image I _i then y _ij is 1, and if AU j is not present in the image, y _ij is -1.

The system identifies a vector of mapping functions f (I _i , w) = (f ₁ (I _i , w ₁ ),..., F _r (I _i , w _r )) ^T. The mapping function detects the input image Ii or converts it into an output vector yi of attributes, and w = (w ₁ ,..., W _r ) ^T is a vector of parameters of these mapping functions. In detection, r = p and
It is. Here, as estimated values of 2D image coordinates u _ij and v _ij
It is. Similarly, in AU recognition, r = q and
It is. here,
Is an estimate of (-1) with or without (1) AUj in image I _i and q is the number of AUs.

For a fixed mapping function f (I _i , w) (eg, a DNN), the system optimizes w as follows.

here,
Represents a loss function. The classical solution to this loss function is L ² -loss which is defined as:

Here, y _ij is the j ^th th element of y _i . This is y _ij ∈ R ² for face landmark point detection and y _ij ∈ {−1, +1} for AU recognition.

Without loss of generality, the system uses the f _i instead of _{f (I i, w),} f j (I i, w j) using the f _ij instead of. The functions fij are all the same, but the specified value of j may be different.

  The above derivation corresponds to a local fit. That is, (33) and (34) try to optimize the fit of each output independently and then adopt the average fit at all outputs.

The derivation approach described above has a fixed fit error
Also have some solutions. For example, the error can be evenly distributed to all outputs.
here,
Is the 2 norm of the vector. Or, most of the error is in one (or a few) of the estimates defined as:

In some embodiments, additional constraints are added to minimize the function.

  a ≧ 1. The system adds global criteria to facilitate convergence.

  Add global configuration

The system defines a set of constraints for adding global configurations that extend global descriptors. The constraint of (34) is local because it independently measures the fit of each element of y _i (eg, y _ij ). Nevertheless, the same criteria can be used to measure the fitness of a pair of points. Formally, it is defined as follows.

Here, g (x, z) is a function that calculates the similarity of two entries. h (.) scales the (unconstrained) output of the network to the appropriate numerical range. In landmark detection, h (f _ij ) = f _ij ∈ R ²

b is the norm of xz (e.g., 2 norm,
Here, x and z are 2D vectors that define the image coordinates of two landmarks.

In AU recognition, h (f _ij ) = sign (f _ij ) ∈ {-1, +1}

Here, sign (.) Returns -1 if the input number is negative, and +1 if this number is positive or zero. If AUj is present in the image I _i then x _ij is 1 and -1 if it is not present in the image. Therefore, the function h (.): R → {-1, +1}

  In some embodiments, the system takes into account the elements of each pair, namely the landmark points of each pair at detection and the global configuration of each pair of AU at recognition. That is, in detection, the system uses information on the distance between all landmark points, and in recognition it determines where the AU pairs co-exist (for example, if two are simultaneously present in the sample image Means not exist).

In some embodiments, global criteria can be extended to triplets. Formally, it is the following.

  Here, g (x, z, u) is a function that calculates the similarity between three entries.

In detection, this means that the system is b norm, eg
That can be calculated,
It means to calculate the area of the triangle defined by each triplet as follows.

  The three landmark points are not collinear.

In some embodiments, the equation can be extended to four or more points. For example, this equation can be extended to a convex quadrilateral as follows:

In the most general case, the system determines, for t landmark points, the area of the polygon envelope, ie the non-self-intersecting polygon contained by t landmark points {x _i1, ..., x _it } calculate. The polygon is given as follows.

The system calculates Delaunay triangulation of landmark points on the face. The polygon envelope is obtained by connecting a set of t landmark point lines counterclockwise. An ordered set of landmark points is defined as follows.
The domain of is given by:

Here, the suffix a of ga (.) Represents an area.

In some embodiments, the results of the above equation are obtained using Green's Theorem as known in the art.
Is the DNN
T outputs of, or true values
It can be done.

The system can calculate the global b norm g _n (.) For the general case of t landmark points as follows:

  The above derivation defines the extension of g (.) To more than two points in the detection task. From this, the above can be used to recognize AUs in the image.

The system calculates co-occurrence of three or more AUs in the image I _i . Formally,
Is a set of t AUs,
It is.

  GL-loss Ioss

The final local global (GL) loss function is given by:

Where global loss,
Is defined as follows.

g (.) is g _a (.) or g _n (.) or both in detection, g _AU (.) in recognition, and α _t uses cross validation of training set It is a normalization constant learned by

  Back propagation

To optimize w, which is a parameter of DNN, the system calculates

The partial derivative of the local loss is, of course, given by

The global loss definition uses the mapping function h (.). In some embodiments, when performing landmark detection, h (f _ij ) = f _ij and the partial derivative of the global loss has the same form as that of the local loss shown above. In another embodiment, when performing AU recognition, the system utilizes:
This function is not a derivative, but the system does
Redefine it as follows.
The partial derivative is

  Deep DNN

  The system includes a deep neural network to recognize AUs. DNN contains two parts. The first part of the DNN is used to detect multiple facial landmark points. The landmark points allow the system to calculate the GL loss as described above.

  The system can calculate normalized landmark points. The system can be coupled with the output of the first fully connected layer of the second part of the DNN to embed landmark position information into the DNN used to recognize the AU. This facilitates the detection of local shape changes that are typically observed in emotional expression. This is done with the definition of GL loss above.

  In some embodiments, the DNN comprises multiple layers. In the exemplary embodiment, nine layers are dedicated to detection of facial landmark points, and the other layers are used to recognize AUs in a series of images.

  The layers directed to the detection of facial landmark points are detailed as follows.

  Facial landmark point detection

  In the exemplary embodiment, the DNN includes three convolutional layers, two largest pool layers, and two complete connection layers. The system applies normalization, dropout, and rectified linear units (ReLU) at the end of each convolutional layer.

  The weights of these layers are optimized using back propagation, the derived GL loss. The equations for global loss and backpropagation are provided above.

  In one example, the system uses this portion of DNN to detect a total of 66 facial landmark points. One advantage of the proposed GL loss is that it can be trained efficiently with very large data sets. In some embodiments, the system includes a face landmark detector that employs data transformation to be invariant to transformation and partial occlusion.

  The facial landmark detector generates additional images by applying a two-dimensional affine transformation, ie, scale, reflection, translation and rotation, to the existing training set. In an exemplary embodiment, the scale is between 2 and 0.5, the rotation is -10 ° to 10 °, and translation and reflection may be generated randomly. In order to make the DNN more robust to partial occlusion, the system randomizes the occlusion box of d × d pixels, where d is 0.2 to 0.4 times the inner eye separation.

  AU recognition

  The second part of the DNN combines the facial appearance features with the landmark locations provided by the first part of the DNN. Specifically, at the output of the first complete connection layer of the second part of the DNN, the appearance image features are concatenated with normalized and automatically detected landmark points.

Formally, the vector of landmark points of the sample image (i = 1,..., N) of i _th is set as follows.
Here, s _ik ∈ R ² is the 2D image coordinates of the k ^th landmark, and n is the number of sample images. Therefore, s _i ∈ R ¹³² . Next, all images are normalized to have the same interocular distance of τ pixels. That is, it becomes the following.
Here, l and r are the image coordinates of the center of the left and right eyes, and || · ||
It is possible to use τ = 200.

The system normalizes landmark points as follows.
The system also multiplies the landmark points by the rotation matrix R so that the outer corners of the left and right eyes coincide with the horizon. the system,
And shift the outer corners of the left and right eyes in the image to predetermined positions (.5, 0) and (-. 5, 0), respectively.

  In one embodiment, the DNN is similar to that of GoogleNet, but there is a significant difference in that the GL loss as defined herein is used. The input of DNN can be a face image. The system resizes the filters of the first layer to fit the input and randomly initializes the weights of these filters. In order to embed the landmarks in the DNN, the number of filters in the first complete connection layer as well as the number of filters for output as the number of AUs can be changed. The system can use a single DNN to detect all AUs in the facial expression image.

  The weights of the second part of the DNN can be optimized using the back propagation method and the global loss defined above.

  In some embodiments, data enhancement can be performed by adding random noise to 2D landmark points and applying the above affine transformation.

  In some embodiments, the system can be trained to initialize wild AU recognition using a training database as described above.

  FIG. 7 shows a network system 700 for detecting AU and emotion categories using deep neural networks (DNNs) in video and / or still images. System 700 includes an image database component 710 having a series of videos and / or images. System 700 includes DNN 720 that determines AUs in the image set of image database 710. The DNN 720 includes a first portion 730 that defines landmarks in the series of images as described above. The DNN 720 includes a second portion 740 that determines AUs in the landmarks of the image set in the database component 710 as described above. System 700 includes a tagging component 750 that tags an image with at least one AU, or tags without an AU. System 700 can store the tagged images in processed image database 760.

  Exemplary computer device

  FIG. 8 illustrates an exemplary computer that can be used to configure hardware devices in an industrial automation system. In various aspects, the computer of FIG. 8 may include all or part of development workspace 100, as described herein. As used herein, a "computer" may include multiple computers. The computer may include, for example, a processor 821, a random access memory (RAM) module 822, a read only memory (ROM) module 823, storage 824, a database 825, one or more input / output (I / O) devices 826, an interface 827. As such, one or more hardware components can be included. Alternatively and / or additionally, controller 820 may include one or more software components, such as, for example, computer readable media including computer executable instructions for performing the methods associated with the illustrative embodiments. It is contemplated that one or more of the hardware components listed above can be implemented using software. For example, storage 824 may include software partitions associated with one or more other hardware components. It is understood that the components listed above are merely exemplary and are not intended to be limiting.

  The processor 821 includes one or more processors each configured to execute instructions and process data to perform one or more functions associated with the computer for indexing the image. Can. Processor 821 can be communicatively coupled to RAM 822, ROM 823, storage 824, database 825, I / O device 826, and interface 827. Processor 821 may be configured to execute a series of computer program instructions to perform various processes. Computer program instructions may be loaded into RAM 822 for execution by processor 821. As used herein, a processor refers to a physical hardware device that performs encoded functions to perform functions on inputs and generate outputs.

  RAM 822 and ROM 823 may each include one or more devices for storing information related to the operation of processor 821. For example, ROM 823 is configured to access and store information associated with controller 820, including information for identifying, initializing and monitoring the operation of one or more components and subsystems. May be included. RAM 822 may include a memory device for storing data associated with one or more operations of processor 821. For example, ROM 823 can load instructions into RAM 822 for execution by processor 821.

  Storage 824 may include any type of mass storage device configured to store information that processor 821 may need to perform processes consistent with the disclosed embodiments. . For example, storage 824 may include one or more magnetic and / or optical disk devices such as a hard drive, CD-ROM, DVD-ROM, or any other type of mass media device.

  Database 825 includes one or more software and / or hardware components that cooperate to store, organize, classify, filter, and / or arrange data used by controller 820 and / or processor 821. obtain. For example, database 825 may store hardware and / or software configuration data associated with input / output hardware devices and controllers as described herein. It is contemplated that database 825 may store other and / or different information from those listed above.

  I / O device 826 may include one or more components configured to communicate information with a user associated with controller 820. For example, the I / O device may include a console with an integrated keyboard and mouse so that the user can maintain a database of images, updates of related things, access to digital content. The I / O device 826 may also include a display that includes a graphical user interface (GUI) for outputting information on a monitor. The I / O unit 826 may also be, for example, a printer for printing information related to the controller 820, a user accessible disk drive (eg, USB port, floppy, CD-ROM, or DVD-ROM) Peripheral devices may be included. A drive or the like may be used to allow the user to input data stored on a portable media device, microphone, speaker system, or any other suitable type of interface device.

  The interface 827 may include one or more components configured to send and receive data via a communication network, such as the Internet, a local area network, a workstation peer-to-peer network, a direct link network, a wireless network, and the like. Or any other suitable communication platform. For example, interface 727 is configured to enable data communication via one or more modulators, demodulators, multiplexers, demultiplexers, network communication devices, wireless devices, antennas, modems, and communication networks. It may include any other type of device.

  Although the methods and systems are described in connection with the preferred embodiments and the specific examples, the embodiments herein are intended to be illustrative rather than restrictive in every respect. The scope is not intended to be limited to the particular embodiments.

Unless otherwise stated, it is by no means intended that any method described herein be construed as requiring that its steps be performed in a particular order.
Thus, if a method claim does not actually state the order in which the steps should be followed, or if it is not specifically stated in the claim or description that the steps should be limited to a particular order In no way is it intended that the order be inferred. This includes any implicit grounds for interpretation, which are stated in the specification, logical matters relating to the arrangement of steps and the flow of operations, grammatical organization or simple meanings derived from punctuation marks. Including the number or type of embodiment being implemented. Various publications can be referenced throughout this application. The entire disclosure of these publications is incorporated herein by reference to more fully describe the state of the art to which the methods and systems belong. It will be apparent to those skilled in the art that various modifications and variations can be made without departing from the scope or spirit. Other embodiments will be apparent to those skilled in the art from consideration of the specification and practice disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit being indicated by the following claims.

Claims (24)

A computer-implemented method for analyzing images to determine AU and AU intensity is
Maintain multiple kernel spaces of shape, shape and shading features, each kernel space can be separated non-linearly from other kernel spaces, each kernel space is one or more action units (AU), and one or more Associated with multiple AU intensity values,
Including receiving multiple images to be analyzed,
For each image to be received,
Determining face space data of face morphology, shape and shading features in the image, the face space including shape feature vectors, morphological feature vectors, and shading feature vectors associated with shading changes of the face;
In order to determine the presence of the determined face space data of form, shape and shading features, the face space data of the determined form features are compared with the plurality of kernel spaces to zero, one or more for the image Determine multiple AU values. The method according to claim 1 is
Processing a video stream comprising the plurality of images in real time to determine an AU value and an AU intensity value for each of the plurality of images. In the method according to claim 1,
The face space data includes feature vectors of the shape, morphological features, and shading feature vectors associated with the face. In the method according to claim 3,
The face space data of the morphological features determined are the distances and angle values between normalized landmarks in the Delaunay triangle formed from the image, and the respective ones corresponding to the normalized landmarks. Includes the angle defined by the Delaunay triangle. In the method according to claim 3,
The shading feature vector associated with the face shading change is:
It is determined by applying a Gabor filter to the normalized landmark points determined from the face.   The method according to claim 3, wherein the image features are landmark points derived using a deep neural network including global local (GL) loss function, AUs, AU intensities, emotion categories, and on the image. Identify those intensities derived using a deep neural network including a global local (GL) loss function configured to back propagate both the local and global fit of the projected landmark points to And image features.   The method of claim 1, wherein the AU value and the AU intensity value together define emotion and emotion intensity.   The method according to claim 1, wherein the image comprises a picture.   The method according to claim 1, wherein the image comprises a frame of a video sequence.   The method of claim 1 wherein the system uses video sequences from a controlled or non-controlled environment.   In the method according to claim 1, the system uses a black and white image or a color image. The method according to claim 1 is
Receive the image,
Processing the received image to determine AU values and AU intensity values of faces in the received image. The method according to claim 1 is
Receive a first plurality of images from a first database;
Receive a second plurality of images from a second database;
Processing the first plurality of received images and the second plurality of images to determine, for each image, an AU value and an AU intensity value of a face in each image;
The first plurality of images have a first acquisition form, the second plurality of images have a second acquisition form, and the first acquisition form is the second acquisition form. It is different from The method according to claim 1 is
Perform kernel subclass discriminant analysis (KSDA) on the face space;
Recognize AU and AU strength, emotion category, and emotion strength based on the KSDA. A computer-implemented method for analyzing an image to determine AU and AU intensities using color features in the image comprises
Identifying a change defining a transition from inactive to active of the AU, said change being selected from the group consisting of chromaticity, hue and saturation, and luminance,
Applying a Gabor transform to the identified color change to obtain invariance to the timing of this change in the expression. The method according to claim 15 is
Maintain in memory a plurality of color feature data associated with AU and / or AU intensity;
Receive the image to be analyzed,
For each image to be received,
Determine the color features of the face in the image by the action of the face muscle;
The determined morphological color feature is compared to the plurality of trained color feature data to determine the presence of the determined morphological color feature in one or more of the plurality of trained color feature data. Determining the zero, one or more AU values for the image. The method according to claim 15 is
Analyzing the plurality of faces in the image or video frame to determine kernel or face space for a plurality of AU values and AU intensity values, each kernel or face space comprising at least one AU value and at least one AU value Associated with the AU intensity values, each kernel or face space can be linearly or non-linearly separated from other kernels and face space data, said kernel or face space comprising functional color space feature data. In the method according to claim 17,
The functional color space is determined by performing a discriminant function learning analysis on color images respectively derived from a given one of the plurality of images.   3. A method, comprising: identifying a change indicative of a face color transition in a video sequence resulting from blood flow in the face from non-existence to presence. A computer-implemented method for analyzing an image to determine AU and AU intensity in a face image is:
Training a deep neural network using a set of training images having a face, said deep neural network being trained to identify AUs in said face image,
Identifying local and global losses using the deep neural network in the face image to determine AUs in the image. The method according to claim 20, wherein said deep neural network is
Detecting a plurality of landmark points using the first portion of the deep neural network, the landmark points facilitating calculation of the overall loss;
Detecting the local image change due to the second part of the deep neural network, wherein the landmark points are normalized and connected to embed positional information of the landmark points into the deep neural network .   22. The method of claim 21, wherein the deep neural network comprises a plurality of layers for identifying landmarks in the image and a second plurality of layers for recognizing AUs in the face image. Including. Determine the neutral plane in the image with color,
Modifying the color of the image of the neutral plane to produce an appearance of a determined expression of emotion or AU. Determine the facial emotion or AU in the image with color,
Modifying the color of the image to increase or decrease the intensity of the emotion or AU to change the perception of the emotion or AU.