CN106097373A

CN106097373A - A kind of smiling face's synthetic method based on branch's formula sparse component analysis model

Info

Publication number: CN106097373A
Application number: CN201610473441.2A
Authority: CN
Inventors: 王存刚; 王斌
Original assignee: Liaocheng University
Current assignee: Liaocheng University
Priority date: 2016-06-24
Filing date: 2016-06-24
Publication date: 2016-11-09
Anticipated expiration: 2036-06-24
Also published as: CN106097373B

Abstract

The present invention relates to a smiling face synthesis method based on a partial sparse component analysis model. First, the partial sparse component analysis model for human face representation is derived; then, the rules for reconstruction and projection are given based on the model; Then, use the projection rules to obtain the projection coefficients, and use the reconstruction rules to reconstruct the input face; then, repeat the above projection and reconstruction process for the reconstructed face; finally, output the reconstructed multiple face images , as the smile synthesis process of the input face. The remarkable effect of the invention is reflected in that the synthesized human face is basically reasonable and smooth, and the synthesized smile has a sense of reality.

Description

A Smile Synthesis Method Based on Partial Sparse Component Analysis Model

技术领域technical field

本发明涉及一种基于分部式稀疏成分分析模型的笑脸合成方法，可广泛应用于电影制作、虚拟社区、游戏娱乐和动画合成等领域。属于计算机视觉,模式识别和人机交互领域。The invention relates to a smiling face synthesis method based on a partial sparse component analysis model, which can be widely used in the fields of film production, virtual community, game entertainment, animation synthesis and the like. It belongs to the field of computer vision, pattern recognition and human-computer interaction.

背景技术Background technique

作为传递人类情感和精神状态的介质，人脸在社会交流中起着非常重要的信息传送和表达功能。人脸通过丰富的面部表情来向外界传递各种信息，近些年来，利用计算机技术重建和合成真实感的人脸面部表情成为计算机视觉、人机交互和计算机图形学领域研究者专注的研究热点，并且被广泛应用于广告、动画、影视和游戏等产业。从20世纪70年代以来，利用计算机自动合成人脸表情的研究不断发展，与之相应的算法也层出不穷。人脸表情合成方法主要包括：基于混合样本的人脸表情合成；直接表情迁移；基于Sketch的人脸表情合成；基于机器学习的人脸表情合成。其中，基于机器学习的方法被广泛用于人脸表情合成，取得了较多的研究成果。基于机器学习的人脸表情合成技术，一般会对给定的训练样本进行分析，提取出有效的人脸表情的变化，进而实现人脸表情的重建和合成。As a medium for conveying human emotion and mental state, human face plays a very important function of information transmission and expression in social communication. The human face transmits various information to the outside world through rich facial expressions. In recent years, the use of computer technology to reconstruct and synthesize realistic facial expressions has become a research hotspot for researchers in the fields of computer vision, human-computer interaction and computer graphics. , and are widely used in advertising, animation, film and television and games industries. Since the 1970s, the use of computers to automatically synthesize human facial expressions has continued to develop, and corresponding algorithms have emerged in an endless stream. The facial expression synthesis methods mainly include: facial expression synthesis based on mixed samples; direct expression transfer; human facial expression synthesis based on Sketch; human facial expression synthesis based on machine learning. Among them, the method based on machine learning is widely used in facial expression synthesis, and has achieved many research results. The facial expression synthesis technology based on machine learning generally analyzes the given training samples to extract effective facial expression changes, and then realizes the reconstruction and synthesis of facial expressions.

经对现有文献的检索发现，目前基于机器学习的人脸合成方法主要分为以下几类：一类是基于典型关联分析(Canonical Correlation Analysis,CCA)的方法，比如：Wei-Wen Feng和Byung-Uck Kim2008年发表在《ACM Transactions on Graphics(ACM图形学会刊)》的论文“Real-time data driven deformation using kernel canonicalcorrelation analysis(基于核典型关联分析的实时数据驱动的变形)”。这种方法把人脸的模型依据特征点分布划分成部件，利用典型关联分析技术，从训练样本中学习出特征点和部件之间的映射关系，并使用泊松变换技术计算得到人脸表情。这种方法的缺陷在于表情控制点需要事先确定下来，并且在之后的计算过程中不能更改。一类是基于独立主成分分析(Independent Component Analysis,ICA)合成人脸表情的方法，比如：Yong Cao和Petros Faloutsos于2003年发表在《Proceedings of the ACM SIGGRAPH/EurographicsSymposium on Computer Animation(ACM计算机动画制作会议论文集)》的论文“Unsupervised learning for speech motion editing(基于半监督学习的语音动画编辑)”。该方法使用独立成分分析技术提取出人脸表情的参数，并且把初始的人脸数据分为表情和语音两部分，通过对每一个成分的操作可以编辑人脸表情的变化。最后一类是基于多维可变形模型(Multidimensional Morphable Model,MMM)的方法，比如：Yao-Jen Chang和Tony Ezzat于2005年发表在《Proceedings of the ACM SIGGRAPH/EurographicsSymposium on Computer Animation(ACM计算机动画制作会议论文集)》的论文“Transferable videorealistic speech animation(可迁移的视频真实感语音动画制作)”，该论文基于多维可变形模型实现了人脸表情的迁移，把其他角色的面部表情迁移到了当前的模型当中，完成了面部表情的合成。这些方法中，人脸建模的模型过于复杂，计算复杂度太高，并且学习出的人脸部分不够灵活，缺乏鲁棒性。After searching the existing literature, it is found that the current face synthesis methods based on machine learning are mainly divided into the following categories: one is based on Canonical Correlation Analysis (CCA), such as: Wei-Wen Feng and Byung -Uck Kim's paper "Real-time data driven deformation using kernel canonical correlation analysis (real-time data driven deformation based on nuclear canonical correlation analysis)" published in "ACM Transactions on Graphics (Journal of ACM Graphics Society)" in 2008. This method divides the face model into parts according to the distribution of feature points, uses typical correlation analysis technology to learn the mapping relationship between feature points and parts from training samples, and uses Poisson transform technology to calculate facial expressions. The disadvantage of this method is that the expression control points need to be determined in advance and cannot be changed in the subsequent calculation process. One class is based on the method of independent component analysis (Independent Component Analysis, ICA) synthetic facial expression, such as: Yong Cao and Petros Faloutsos published in 2003 in "Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation (ACM computer animation production Conference Proceedings) "paper "Unsupervised learning for speech motion editing (speech animation editing based on semi-supervised learning)". This method uses independent component analysis technology to extract the parameters of facial expressions, and divides the initial facial data into two parts: expression and voice, and the changes of facial expressions can be edited by operating each component. The last category is based on the multidimensional deformable model (Multidimensional Morphable Model, MMM) method, such as: Yao-Jen Chang and Tony Ezzat published in 2005 in "Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation (ACM Computer Animation Conference Proceedings)" paper "Transferable videorealistic speech animation (transferable video realistic voice animation production)", this paper realized the transfer of facial expressions based on the multi-dimensional deformable model, and transferred the facial expressions of other characters to the current model Among them, the synthesis of facial expressions is completed. In these methods, the face modeling model is too complex, the computational complexity is too high, and the learned face part is not flexible enough and lacks robustness.

发明内容Contents of the invention

本发明的目的在于针对现有方法的不足，提出一种基于分部式稀疏成分分析模型的笑脸合成方法，可在仅有少量有标签训练样本的情况下合成真实感的人脸图像。人脸图像具有良好的可对齐性，具有一致的结构，本发明以此为基础设计了产生式模型--分部式稀疏成分分析模型，并基于该模型进行人脸表示，构建投影和重构规则，进而完成笑脸合成。由于人脸典型的结构，分部式方法在人脸表示上有很好的效果，并能捕获对应训练图像的某些语义部分，用这些部分的组合来表示图像。而稀疏成分分析学习到的基与样本更接近，因此在人脸表示上鲁棒性较强。分部式稀疏成分分析模型结合了分部式方法和稀疏成分分析的优点。The purpose of the present invention is to address the deficiencies of the existing methods, and propose a smiley face synthesis method based on a partial sparse component analysis model, which can synthesize realistic human face images with only a small number of labeled training samples. Face images have good alignability and a consistent structure. Based on this, the present invention designs a generative model--partial sparse component analysis model, and performs face representation, construction projection and reconstruction based on this model Rules, and then complete the synthesis of smiley faces. Due to the typical structure of the face, the partial method has a good effect on the representation of the face, and can capture some semantic parts corresponding to the training image, and use the combination of these parts to represent the image. The basis learned by sparse component analysis is closer to the sample, so it is more robust in face representation. The fractional sparse component analysis model combines the advantages of the fractional approach and sparse component analysis.

为实现上述目的，本发明通过下述技术方案来实现。一种基于分部式稀疏成分分析模型的笑脸合成方法，首先，导出用于人脸表示的分部式稀疏成分分析模型；接着，基于该模型给出重构和投影的规则；紧接着，利用投影规则得到投影系数，利用重构规则对输入的人脸进行重建；然后，对重建后的人脸重复上述投影和重构的过程；最后，把重建后的多幅人脸图像输出，作为输入人脸的笑容合成过程。In order to achieve the above object, the present invention is achieved through the following technical solutions. A smiling face synthesis method based on the partial sparse component analysis model. First, derive the partial sparse component analysis model for face representation; then, give the reconstruction and projection rules based on the model; then, use The projection rule is used to obtain the projection coefficient, and the input face is reconstructed using the reconstruction rule; then, the above-mentioned projection and reconstruction process is repeated for the reconstructed face; finally, multiple reconstructed face images are output as input Smile synthesis process of human face.

一种基于分部式稀疏成分分析模型的笑脸合成方法，具体步骤是，A smiling face synthesis method based on a partial sparse component analysis model, the specific steps are:

步骤一、学习构建产生式模型，用于人脸表示；Step 1. Learn to build a generative model for face representation;

该产生式模型为给定的人脸图片寻找共同的空间分割，并且为这个分割的每个部分学习一个稀疏成分分析模型；The generative model finds a common spatial segmentation for a given face image, and learns a sparse component analysis model for each part of this segmentation;

设产生式模型的概率分布为P(x,Z,R)，其计算方式为P(x,Z,R)＝P(xR,Z)P(R)P(Z)。首先，导出产生式模型的连续诱导先验，并利用PoE(product of experts)将连续诱导先验和多项式先验结合在一起构成P(R)，即：Suppose the probability distribution of the production model is P(x, Z, R), and its calculation method is P(x, Z, R) = P(xR, Z)P(R)P(Z). First, derive the continuous induction prior of the production model, and use PoE (product of experts) to combine the continuous induction prior and the polynomial prior to form P(R), namely:

$\begin{matrix} P P ((R R)) = = \underset{d d}{Π Π} P P (({r r}_{d d . .})) = = \underset{d d}{Π Π} \frac{11}{{Z Z}_{d d}} {P P}_{11} (({r r}_{d d . .})) {P P}_{22} (({r r}_{d d . .})) \\ = = \frac{11}{{Z Z}_{00}} \underset{d d,, k k}{Π Π} {α α}_{d d k k}^{{r r}_{d d k k}} \underset{d d,, k k}{Π Π} exp exp {{- - {r r}_{d d k k} ((11 - - \frac{44}{| | {D D.}_{d d} | |} {Σ Σ}_{{d d}^{' '} &Element; &Element; {D D.}_{d d}} {r r}_{{d d}^{' '} k k}))}} \end{matrix} - - - - - - ((11))$

其中，Z₀是归一化函数，用于记录人脸图像像素的选择的变量为r_d.＝(r_d1,,r_dK)^T，且r_d.～Mult(n＝1,α_d.)，D_d是变量r_dk的相邻变量集，P₁(r_d.)为多项式先验，P₂(r_d.)为连续性诱导先验；Among them, Z ₀ is a normalization function, and the variable used to record the selection of face image pixels is r _d. =(r _d1 ,,r _dK ) ^T , and r _d. ~Mult(n=1,α _d. ), D _d is the adjacent variable set of variable r _dk , P ₁ (r _d. ) is polynomial prior, P ₂ (r _d. ) is continuity induced prior;

接着，将人脸的每个部分建模成稀疏成分分析模型，并为每个人脸部分学习稀疏成分分析模型，求得P(xR,Z)和P(Z)；具体来说，对于一个人脸像素x_d，首先从K个人脸部分中选择一个部分，记为k，相应地从第k个稀疏成分分析模型中产生一个像素。得到下述模型：Next, each part of the face is modeled as a sparse component analysis model, and a sparse component analysis model is learned for each face part to obtain P(xR,Z) and P(Z); specifically, for a person Face pixel x _d , first select a part from K face parts, denoted as k, correspondingly generate a pixel from the kth sparse component analysis model. The following model is obtained:

${x x}_{d d} = = {Σ Σ}_{k k = = 11}^{K K} {r r}_{d d k k} {Σ Σ}_{m m = = 11}^{M m} {z z}_{k k m m} {w w}_{d d k k m m} + + {μ μ}_{d d} + + {ϵ ϵ}_{d d}$

r_d.～Multi(n＝1,α_d.)r _d. ~Multi(n=1,α _d. )

z_km～Lap(u＝0,b＝1)；z _km ~ Lap (u = 0, b = 1);

其中，x_d是人脸的像素，为第k个部分的过完备基，M是基的数目，μ_d为均值；为随机噪声；α_d.＝(α_d1,,α_dK)^T为多项式分布的参数；由上述模型可得：Among them, x _d is the pixel of the face, is the overcomplete base of the kth part, M is the number of bases, and μ _d is the mean value; is random noise; α _d. =(α _d1 ,,α _dK ) ^T is the parameter of multinomial distribution; from the above model, it can be obtained:

$P P ((x x | | R R,, Z Z)) = = \underset{d d}{Π Π} P P (({x x}_{d d} | | {r r}_{d d . .},, Z Z)) = = \underset{d d}{Π Π} N N (({x x}_{d d};; \underset{k k}{Σ Σ} {r r}_{d d k k} \underset{m m}{Σ Σ} {z z}_{k k m m} {w w}_{d d k k m m} + + {μ μ}_{d d},, {σ σ}_{d d}^{22})) - - - - - - ((22))$

$P P ((Z Z)) = = \underset{k k,, m m}{Π Π} \frac{11}{22 b b} exp exp ((- - \frac{| | {z z}_{k k m m} - - u u | |}{b b})) - - - - - - ((33))$

考虑到P(R)(式(1))，P(Z)(式(3))和P(x|R,Z)(式(2))，导出人脸表示的产生式模型P(x,Z,R)：Considering P(R) (Equation (1)), P(Z) (Equation (3)) and P(x|R,Z) (Equation (2)), the production model P(x ,Z,R):

$\begin{matrix} P P ((x x,, Z Z,, R R)) = = P P ((x x | | R R,, Z Z)) P P ((R R)) P P ((Z Z)) \\ = = \underset{d d}{Π Π} N N (({x x}_{d d};; \underset{k k}{Σ Σ} {r r}_{d d k k} \underset{m m}{Σ Σ} {z z}_{k k m m} {w w}_{d d k k m m} + + {μ μ}_{d d},, {σ σ}_{d d}^{22})) \frac{11}{{Z Z}_{00}} \underset{d d,, k k}{Π Π} {α α}_{d d k k}^{{r r}_{d d k k}} \underset{d d,, k k}{Π Π} exp exp {{- - {r r}_{d d k k} ((11 - - \frac{44}{| | {D D.}_{d d} | |} {Σ Σ}_{{d d}^{' '} &Element; &Element; {D D.}_{d d}} {r r}_{{d d}^{' '} k k}))}} \\ \underset{k k,, m m}{Π Π} \frac{11}{22 b b} exp exp ((- - \frac{| | {z z}_{k k m m} - - u u | |}{b b})) \end{matrix} - - - - - - ((44))$

步骤二、计算人脸笑容合成的投影系数；Step 2. Calculating the projection coefficients for the synthesis of smiles on human faces;

令学习出的分部式稀疏成分分析模型为θ；样本X^c在模型θ上的投影系数，是组合系数的均值；Let the learned partial sparse component analysis model be θ; the projection coefficient of the sample X ^c on the model θ is the combination coefficient the mean value of

系数的均值可通过下式估计：The mean of the coefficients can be estimated by:

${E E.}_{P P (({z z}_{k k m m}^{c c} | | {X x}^{c c},, θ θ))} [[{z z}_{k k m m}^{c c}]] \approx \approx \frac{11}{J J} {Σ Σ}_{i i = = 11}^{J J} {z z}_{k k m m}^{c c,, i i} - - - - - - ((55))$

其中，是从中采样的样本；in, From Samples sampled in;

是隐变量z_km的采样分布(或后验分布)，本发明采用蒙特卡罗EM算法来估计隐变量的后验分布；蒙特卡罗方法首先使用Gibbs采样方法把隐变量的样本从后验采样，r_dk的Gibbs采样分布可以表示为： is the sampling distribution (or posterior distribution) of latent variable z _km , the present invention adopts Monte Carlo EM algorithm to estimate the posterior distribution of latent variable; Monte Carlo method first uses Gibbs sampling method to the sample of latent variable from posterior sampling , the Gibbs sampling distribution of r _dk can be expressed as:

$P P (({r r}_{d d k k} | | x x,, {R R}_{- - d d k k},, Z Z)) = = \frac{P P ((x x,, Z Z | | R R)) P P (({r r}_{d d k k} | | {R R}_{- - d d k k})) P P (({R R}_{- - d d k k}))}{P P ((x x,, Z Z | | {R R}_{- - d d k k})) P P (({R R}_{- - d d k k}))} &Proportional; &Proportional; P P ((x x | | Z Z,, R R)) P P (({r r}_{d d k k} | | {R R}_{- - d d k k})) - - - - - - ((66))$

其中，R_-dk表示从R除去r_dk以后所得到的变量集合。把公式(2)和公式(3)代入上面的式子，得到：Among them, R _-dk represents the variable set obtained after removing r _dk from R. Substituting formula (2) and formula (3) into the above formula, we get:

$P P (({r r}_{d d k k} | | x x,, {R R}_{- - d d k k},, Z Z)) &Proportional; &Proportional; N N (({x x}_{d d};; \underset{k k}{Σ Σ} {r r}_{d d k k} \underset{m m}{Σ Σ} {z z}_{k k m m} {w w}_{d d k k m m} + + {μ μ}_{d d},, {σ σ}_{d d}^{22})) \cdot \cdot exp exp {{{r r}_{d d k k} ((\underset{{d d}^{' '} &Element; &Element; {D D.}_{d d}}{Σ Σ} {r r}_{{d d}^{' '} k k} - - 11))}} {α α}_{d d k k}^{{r r}_{d d k k}} - - - - - - ((77))$

z_km的采样分布可以类似地推导成如下形式：The sampling distribution of z _km can be similarly derived as follows:

$P P (({z z}_{k k m m} | | x x,, R R,, {Z Z}_{- - k k m m})) &Proportional; &Proportional; \{\begin{matrix} N N (({z z}_{k k m m};; {μ μ}_{z z k k m m}^{+ +},, {σ σ}_{z z k k m m}^{22 + +})) & {z z}_{k k m m} &GreaterEqual; &Greater Equal; 00 & (({P P}_{+ +})) \\ N N (({z z}_{k k m m};; {μ μ}_{z z k k m m}^{- -},, {σ σ}_{z z k k m m}^{22 - -})) & {z z}_{k k m m} \leq \leq 00 & (({P P}_{- -})) \end{matrix} - - - - - - ((88))$

其中， in,

${σ σ}_{{z z}_{k k m m}}^{22 + +} = = {σ σ}_{{z z}_{h h m m}}^{22 - -} = = 11 / / [[\underset{d d}{Σ Σ} {(({r r}_{d d k k} {w w}_{d d k k m m} / / {σ σ}_{d d k k}))}^{22}]],,$

由此得到的采样分布是不连续的，可以通过以下方式从上述分布采样：The resulting sampling distribution is discontinuous and can be sampled from the above distribution by:

(1)从式(8)中的分布P₊采样，并输出非负样本(z_km≥0)；(1) Sample from the distribution P ₊ in equation (8), and output non-negative samples (z _km ≥ 0);

(2)从式(8)中的分布P_采样，并输出负样本(z_km≤0)；(2) Sample from the distribution P_ in equation (8), and output negative samples (z _km ≤ 0);

(3)将这两组样本合并起来作为输出；即为按照这种方式从该分布中采样得到的样本；在用模型θ合成给定图像X^c的笑容时，首先是利用公式(5)将该人脸图片投影到模型θ上，计算投影系数 (3) Combine the two sets of samples as output; It is the sample sampled from the distribution in this way; when using the model θ to synthesize the smile of a given image X ^c , first use the formula (5) to project the face image onto the model θ, and calculate the projection coefficient

步骤三、给出人脸笑容合成的重构规则；Step 3. Give the reconstruction rules for the synthesis of human face smiles;

计算出投影系数以后，X^c的重建值是式(2)所示条件分布的均值：After calculating the projection coefficient, the reconstructed value of X ^c is the mean value of the conditional distribution shown in formula (2):

${\overset{^^}{x x}}_{d d}^{c c} = = \underset{k k}{Σ Σ} {α α}_{dk dk} \underset{m m}{Σ Σ} {E E.}_{P P (({z z}_{km km}^{c c} | | {X x}^{c c},, θ θ))} [[{z z}_{km km}^{c c}]] {w w}_{dkm dkm} + + {u u}_{d d} - - - - - - ((99))$

通过公式(9)在模型θ上重建人脸；Reconstruct the face on the model θ by formula (9);

步骤四、重复步骤二和步骤三，直至把所有的中间人脸全部输出，最终合成人脸笑容。Step 4. Repeat step 2 and step 3 until all the intermediate faces are output, and finally the smile of the human face is synthesized.

本发明提供的基于分部式稀疏成分分析模型的笑脸合成方法，其包括如下具体步骤：The smiling face synthesis method based on the partial sparse component analysis model provided by the present invention comprises the following specific steps:

(1)构建并学习产生式模型，用于人脸表示和建模。该产生式模型为给定的人脸图片寻找共同的空间分割，并且为这个分割的每个部分学习一个稀疏成分分析模型，完成人脸建模。首先，导出产生式模型的连续诱导先验，并利用PoE(product of experts)将连续诱导先验和多项式先验结合在一起构成模型总的先验P(R)，接着，将人脸的每个部分建模成稀疏成分分析模型，并为每个人脸部分(1) Build and learn a generative model for face representation and modeling. The generative model finds a common spatial segmentation for a given face image, and learns a sparse component analysis model for each part of this segment to complete face modeling. First, derive the continuous induction prior of the production model, and use PoE (product of experts) to combine the continuous induction prior and the polynomial prior to form the total prior P(R) of the model. Then, each face parts are modeled as a sparse component analysis model, and for each face part

学习稀疏成分分析模型，求得P(x|R,Z)和P(Z)。Learn the sparse component analysis model to obtain P(x|R,Z) and P(Z).

(2)计算人脸笑容合成在学习好的分部式稀疏成分分析模型上的投影系数；(2) Calculate the projection coefficient of the face smile synthesis on the learned partial sparse component analysis model;

(3)计算出投影系数以后，通过人脸笑容合成的重构规则在人脸模型上重建人脸；(3) After calculating the projection coefficient, reconstruct the face on the face model through the reconstruction rule of face smile synthesis;

(4)对重建后的人脸重复步骤(2)的投影和步骤(3)的重构过程；(4) Repeat the projection of step (2) and the reconstruction process of step (3) to the reconstructed face;

(5)最后，把重建后的多幅人脸图像输出，作为输入人脸的笑容合成过程。(5) Finally, the reconstructed multiple face images are output as the smile synthesis process of the input face.

本发明涉及一种基于分部式稀疏成分分析模型的笑脸合成方法。首先，构建学习产生式模型，用于人脸表示。该产生式模型为给定的人脸图片寻找共同的空间分割，并且为这个分割的每个部分学习一个稀疏成分分析模型，完成人脸建模。接着，计算输入样本(人脸图像)在学习好的模型上的投影系数，投影系数是一组组合系数的均值，该组合系数通过蒙特卡罗EM算法估计得出。紧接着，通过人脸笑容合成的重构规则在分部式稀疏成分分析模型上重建人脸。然后，对重建后的人脸重复上述投影和重构的过程。最后，把重建后的多幅人脸图像输出，作为输入人脸的笑容合成过程。The invention relates to a smiling face synthesis method based on a partial sparse component analysis model. First, build a learned generative model for face representation. The generative model finds a common spatial segmentation for a given face image, and learns a sparse component analysis model for each part of this segment to complete face modeling. Next, calculate the projection coefficient of the input sample (face image) on the learned model. The projection coefficient is the mean value of a group of combination coefficients estimated by the Monte Carlo EM algorithm. Then, the face is reconstructed on the partial sparse component analysis model through the reconstruction rule of face smile synthesis. Then, repeat the above process of projection and reconstruction for the reconstructed face. Finally, the reconstructed multiple face images are output as the smile synthesis process of the input face.

与现有技术相比，本发明的显著效果体现在：由于对人脸建模采用的是分部式稀疏成分分析模型，学习出的人脸部分的形状非常灵活(受益于分部式表示方法)，并且每个部分的鲁棒性较强(受益于稀疏成分分析模型建模每个部分)。值得注意的是，本发明的方法允许输入的图片不在训练集中，也就是输入图像对于训练好的模型来说是全新的图像，这种情况下，只有学习到的模型足够灵活时，才能稳定地重建输入人脸，使人脸逐步具有笑容，本发明的方法合成的人脸基本上合理，并且光滑，合成的笑容具有真实感。Compared with the prior art, the remarkable effect of the present invention is reflected in: because what adopt to face modeling is the partial sparse component analysis model, the shape of the part of the human face that learns is very flexible (benefits from the partial expression method ), and each part is more robust (benefiting from the sparse component analysis model modeling each part). It is worth noting that the method of the present invention allows the input picture not in the training set, that is, the input image is a brand new image for the trained model. In this case, only when the learned model is flexible enough can it be stably Reconstruction of the input human face makes the human face gradually have a smile. The human face synthesized by the method of the present invention is basically reasonable and smooth, and the synthesized smile has a sense of reality.

附图说明Description of drawings

图1为本发明的人脸笑容合成方法流程图。FIG. 1 is a flow chart of a method for synthesizing a smile on a human face according to the present invention.

图2为不同方法学习出的人脸部分对比，图2(1)：MCFA学习出的人脸部分；图2(2)：SSPCA学习出的部分；图2(3)：本发明学习出的部分。Fig. 2 compares the face parts learned by different methods, Fig. 2 (1): the face part learned by MCFA; Fig. 2 (2): the part learned by SSPCA; Fig. 2 (3): the part learned by the present invention part.

图3为本发明合成的人脸笑容。Fig. 3 is the human face smile synthesized by the present invention.

图4为本发明和经典方法在实施例数据库上的笑容合成对比图。Fig. 4 is a comparison diagram of smile synthesis of the present invention and the classical method on the embodiment database.

图5为本发明和经典方法在实施例数据库上的笑容合成对比图。Fig. 5 is a comparison diagram of smile synthesis of the present invention and the classical method on the embodiment database.

具体实施方式detailed description

以下结合附图和具体实施例对本发明的技术方案做更详细的阐述。以下实施例在以本发明技术方案为前提下进行实施，给出了详细的实施方式和过程，但本发明的保护范围不限于下述的实施例。The technical solutions of the present invention will be described in more detail below in conjunction with the accompanying drawings and specific embodiments. The following examples are implemented on the premise of the technical solutions of the present invention, and detailed implementation methods and processes are given, but the protection scope of the present invention is not limited to the following examples.

实施例1：一种基于分部式稀疏成分分析模型的笑脸合成方法，通过以下几个步骤具体实现：Embodiment 1: A method for synthesizing a smiley face based on a partial sparse component analysis model, which is implemented through the following steps:

步骤一、学习构建产生式模型，用于人脸表示。Step 1. Learn to build a generative model for face representation.

该产生式模型为给定的人脸图片寻找共同的空间分割，并且为这个分割的每个部分学习一个稀疏成分分析模型。设产生式模型的概率分布为P(x,Z,R)，其计算方式为P(x,Z,R)＝P(x|R,Z)P(R)P(Z)。首先，导出产生式模型的连续诱导先验，并利用PoE(product ofexperts)将连续诱导先验和多项式先验结合在一起构成P(R)，即：The generative model finds a common spatial segmentation for a given face image and learns a sparse component analysis model for each part of this segmentation. Suppose the probability distribution of the production model is P(x, Z, R), and its calculation method is P(x, Z, R) = P(x|R, Z)P(R)P(Z). First, derive the continuous induction prior of the production model, and use PoE (product of experts) to combine the continuous induction prior and the polynomial prior to form P(R), namely:

$\begin{matrix} P P ((R R)) = = \underset{d d}{Π Π} P P (({r r}_{d d . .})) = = \underset{d d}{Π Π} \frac{11}{{Z Z}_{d d}} {P P}_{11} (({r r}_{d d . .})) {P P}_{22} (({r r}_{d d . .})) \\ = = \frac{11}{{Z Z}_{00}} \underset{d d,, k k}{Π Π} {α α}_{d d k k}^{{r r}_{d d k k}} \underset{d d,, k k}{Π Π} exp exp {{- - {r r}_{d d k k} ((11 - - \frac{44}{| | {D D.}_{d d} | |} {Σ Σ}_{{d d}^{' '} &Element; &Element; {D D.}_{d d}} {r r}_{{d d}^{' '} k k}))}} \end{matrix}$

$((11))$

其中，Z₀是归一化函数，用于记录人脸图像像素的选择的变量为r_d.＝(r_d1,,r_dK)^T，且r_d.～Mult(n＝1,α_d.)，D_d是变量r_dk的相邻变量集，P₁(r_d.)为多项式先验，P₂(r_d.)为连续性诱导先验。Among them, Z ₀ is a normalization function, and the variable used to record the selection of face image pixels is r _d. =(r _d1 ,,r _dK ) ^T , and r _d. ~Mult(n=1,α _d. ), D _d is the adjacent variable set of the variable r _dk , P ₁ (r _d. ) is the polynomial prior, and P ₂ (r _d. ) is the continuity-induced prior.

接着，将人脸的每个部分建模成稀疏成分分析模型，并为每个人脸部分学习稀疏成分分析模型，求得P(x|R,Z)和P(Z)。具体来说，对于一个人脸像素x_d，首先从K个人脸部分中选择一个部分，记为k，相应地从第k个稀疏成分分析模型中产生一个像素。得到下述模型：r_d.～Multi(n＝1,α_d.)；z_km～Lap(u＝0,b＝1)。其中，x_d是人脸的像素，为第k个部分的过完备基，M是基的数目，μ_d为均值；为随机噪声；α_d.＝(α_d1,,α_dK)^T为多项式分布的参数。由上述模型可得：Next, each part of the face is modeled as a sparse component analysis model, and a sparse component analysis model is learned for each face part to obtain P(x|R,Z) and P(Z). Specifically, for a face pixel x _d , first select a part from K face parts, denoted as k, and correspondingly generate a pixel from the kth sparse component analysis model. The following model is obtained: r _d. ~Multi(n=1,α _d. ); z _km ~Lap(u=0,b=1). Among them, x _d is the pixel of the face, is the overcomplete base of the kth part, M is the number of bases, and μ _d is the mean value; is random noise; α _d. =(α _d1 ,,α _dK ) ^T is the parameter of multinomial distribution. From the above model it can be obtained:

考虑到P(R)(式(1))，P(Z)(式(3))和P(x|R,Z)(式(2))，导出本发明中人脸表示的产生式模型P(x,Z,R)：Considering P(R) (Formula (1)), P(Z) (Formula (3)) and P(x|R, Z) (Formula (2)), derive the production model of face representation in the present invention P(x,Z,R):

步骤二、计算人脸笑容合成的投影系数。Step 2: Calculating the projection coefficients for synthesis of smiles on human faces.

令学习出的分部式稀疏成分分析模型为θ。样本X^c在模型θ上的投影系数，是组合系数的均值。系数的均值可通过下式估计：Let the learned partial sparse component analysis model be θ. The projection coefficient of the sample X ^c on the model θ is the combination coefficient mean value. The mean of the coefficients can be estimated by:

其中，是从中采样的样本。的计算方式如下。in, From Samples sampled in . is calculated as follows.

是隐变量z_km的采样分布(或后验分布)，本发明采用蒙特卡罗EM算法来估计隐变量的后验分布。蒙特卡罗方法首先使用Gibbs采样方法把隐变量的样本从后验采样，r_dk的Gibbs采样分布可以表示为： is the sampling distribution (or posterior distribution) of the hidden variable z _km , and the present invention uses the Monte Carlo EM algorithm to estimate the posterior distribution of the hidden variable. The Monte Carlo method first uses the Gibbs sampling method to sample the latent variable samples from the posterior, and the Gibbs sampling distribution of r _dk can be expressed as:

$P P (({r r}_{d d k k} | | x x,, {R R}_{- - d d k k},, Z Z)) &Proportional; &Proportional; N N (({x x}_{d d};; \underset{k k}{Σ Σ} {r r}_{d d k k} \underset{m m}{Σ Σ} {z z}_{h h m m} {w w}_{d d h h m m} + + {μ μ}_{d d},, {σ σ}_{d d}^{22})) \cdot \cdot exp exp {{{r r}_{d d k k} ((\underset{{d d}^{' '} &Element; &Element; {D D.}_{d d}}{Σ Σ} {r r}_{{d d}^{' '} k k} - - 11))}} {α α}_{d d k k}^{{r r}_{d d k k}} - - - - - - ((77))$

其中， in,

${σ σ}_{{z z}_{k k m m}}^{22 + +} = = {σ σ}_{{z z}_{k k m m}}^{22 - -} = = 11 / / [[\underset{d d}{Σ Σ} {(({r r}_{d d k k} {w w}_{d d k k m m} / / {σ σ}_{d d k k}))}^{22}]],,$

由此得到的采样分布是不连续的，可以通过以下方式有效地从上述分布采样：(1)从式(8)中的分布P₊采样，并输出非负样本(z_km≥0)；(2)从式(8)中的分布P-采样，并输出负样本(z_km≤0)；(3)将这两组样本合并起来作为输出。即为按照这种方式从该分布中采样得到的样本。在用模型θ合成给定图像X^c的笑容时，首先是利用公式(5)将该人脸图片投影到模型θ上，计算投影系数 The resulting sampling distribution is discontinuous and can be efficiently sampled from the above distribution by: (1) sampling from the distribution P ₊ in Equation (8), and outputting non-negative samples (z _km ≥ 0); ( 2) Sample from the distribution P- in equation (8), and output negative samples (z _km ≤ 0); (3) Combine these two sets of samples as output. is the sample drawn from this distribution in this way. When using the model θ to synthesize the smile of a given image X ^c , first use the formula (5) to project the face image onto the model θ, and calculate the projection coefficient

步骤三、给出人脸笑容合成的重构规则。Step 3: Give the reconstruction rules for face smile synthesis.

${\overset{^^}{x x}}_{d d}^{c c} = = \underset{k k}{Σ Σ} {α α}_{d d k k} \underset{m m}{Σ Σ} {E E.}_{P P (({z z}_{k k m m}^{c c} | | {X x}^{c c},, θ θ))} [[{z z}_{k k m m}^{c c}]] {w w}_{d d k k m m} + + {u u}_{d d} - - - - - - ((99))$

通过公式(9)在模型θ上重建人脸。The face is reconstructed on the model θ by formula (9).

本发明的具体实施例为从训练样本中学习人脸笑容，然后将学习到的表情迁移到新输入的人脸。这在动画与合成中十分有用，例如用给定的人脸自动合成一段笑容视频。训练样本从面部表情视频中获取。A specific embodiment of the present invention is to learn smiles on human faces from training samples, and then transfer the learned expressions to newly input human faces. This is very useful in animation and compositing, such as automatically compositing a video of a smile from a given human face. Training samples are obtained from facial expression videos.

实施例2；一种基于分部式稀疏成分分析模型的笑脸合成方法，具体的实施步骤详述如下(使用Visual C++语言编程实现)：Embodiment 2; A kind of smiling face synthesis method based on the partial sparse component analysis model, concrete implementation steps are described in detail as follows (using Visual C++ language programming to realize):

1、学习人脸部分先验，实现人脸分部式表示。1. Learn the face part prior, and realize the partial representation of the face.

本实施例使用CBCL人脸数据库来学习人脸部分先验，并将它迁移到人脸面部表情数据库上的笑容合成实验中。人脸部分的数目设置为K＝6，基的数目设置为M＝40。M在一个较大的范围内(根据经验M∈[20,240])都可以工作得很好，并且能够学习出合理的人脸部分先验。人脸部分学习出的基如图2所示，为了便于比较，图2中还列出了另外两种方法学习出的基，这两种方法分别是多因素因子分析(Multiple Cause Factor Analysis,MCFA)和结构化稀疏主成分分析(Structured Sparse Principle Component Analysis,SSPCA)。图2中，图2(1)是MCFA学习出的人脸部分，图2(2)是SSPCA学习出的人脸部分，图2(3)是本发明的方法学习出的人脸部分。从图2可以看出，MCFA学习到的“眼睛”和“鼻子”合理但并不连续。SSPCA学习到连续且凸的部分，但这些部分将“嘴巴”至少分割成两个不同的部分。本发明学习到的部分是合理的，参见底行的“脖子”。In this embodiment, the CBCL face database is used to learn the face part prior, and it is transferred to the smile synthesis experiment on the face and facial expression database. The number of face parts is set to K=6, and the number of bases is set to M=40. M works well over a large range (empirically M ∈ [20, 240]) and is able to learn reasonable priors on face parts. The basis learned by the face part is shown in Figure 2. For comparison, Figure 2 also lists the bases learned by the other two methods. These two methods are Multiple Cause Factor Analysis (MCFA) ) and Structured Sparse Principle Component Analysis (SSPCA). Among Fig. 2, Fig. 2 (1) is the human face part that MCFA learns, and Fig. 2 (2) is the human face part that SSPCA learns, and Fig. 2 (3) is the human face part that method of the present invention learns. It can be seen from Figure 2 that the "eyes" and "nose" learned by MCFA are reasonable but not continuous. SSPCA learns continuous and convex parts, but these parts segment the "mouth" into at least two distinct parts. The part learned by the present invention is reasonable, see "neck" in the bottom row.

2、学习产生式模型—分部式稀疏成分分析模型。2. Learn the production model—partial sparse component analysis model.

训练样本从面部表情视频中获取。对于60个人中的每一个，本实施例从中性表情和笑容之间大致选择8个中间状态(每个状态对应一副人脸图像)。对于每个状态s(s＝1,...,8)，用50个不同的人脸图像学习模型θ_s，其中基的数目设置为M＝40(研究发现较小的M容易丢失人脸细节，较大的M则会增加计算复杂度)。Training samples are obtained from facial expression videos. For each of the 60 people, this embodiment roughly selects 8 intermediate states between neutral expression and smiling face (each state corresponds to a pair of face images). For each state s (s=1,...,8), use 50 different face images to learn the model θ _s , where the number of bases is set to M=40 (studies have found that smaller M is easy to lose faces details, larger M will increase the computational complexity).

3、给定一副具有中性表情的人脸x₀，笑容合成过程如下：3. Given a face x ₀ with a neutral expression, the smile synthesis process is as follows:

(1)将人脸x₀投影到第一个模型θ₁上，利用式(5)得到投影系数(投影)；(1) Project the face x ₀ onto the first model θ ₁ , and use formula (5) to get the projection coefficient (projection);

(2)通过式(9)在模型θ₁上重建人脸，定义为(重建)；(2) Reconstruct the face on the model θ ₁ through formula (9), defined as (reconstruction);

(3)将作为输入图片，重复模型θ₂上的投影步骤(1)和重建步骤(2)；(3) Will As an input image, repeat the projection step (1) and reconstruction step (2) on the model θ ₂ ;

(4)输出作为输入人脸x₀的结果。(4) output As a result of input face x ₀ .

本发明方法合成得到的笑脸如图3所示。如该图所示，合成的人脸从左到右逐步微笑。值得注意的是，实施例中所有的输入人脸都不在训练集中，即输入图像对于学习好的模型是全新的，只有学习到的模型足够灵活时，才能稳定地重建输入人脸，使人脸逐步具有笑容。从图3可以看出，本发明方法合成的人脸基本上合理并且光滑。图4和图5给出了本发明方法、MCFA、SSPCA和结构化稀疏潜在空间方法(Latent Spaces with StructuredSparsity,LSSS)合成的人脸笑容对比。从图4和图5看出，MCFA合成的人脸有可能不光滑(如图5所示)。SSPCA和LSSS合成的结果有一些模糊。本发明方法的结果受益于两个方面：人脸部分的形状很灵活；每个部分的稀疏成分分析有很强的鲁棒性。The smiling face synthesized by the method of the present invention is shown in FIG. 3 . As shown in the figure, the synthesized face smiles progressively from left to right. It is worth noting that all the input faces in the embodiment are not in the training set, that is, the input image is completely new to the learned model, and only when the learned model is flexible enough can the input face be reconstructed stably, making the face Gradually have a smile. As can be seen from Figure 3, the face synthesized by the method of the present invention is basically reasonable and smooth. Figure 4 and Figure 5 show the comparison of smiles on faces synthesized by the method of the present invention, MCFA, SSPCA and the structured sparse latent space method (Latent Spaces with StructuredSparsity, LSSS). It can be seen from Figure 4 and Figure 5 that the face synthesized by MCFA may not be smooth (as shown in Figure 5). The results of SSPCA and LSSS synthesis are somewhat blurred. The results of the method of the present invention benefit from two aspects: the shape of the face parts is flexible; the sparse component analysis of each part is very robust.

Claims

1. A smiling face synthesis method based on a partial sparse component analysis model, characterized in that, at first, the partial sparse component analysis model for human face representation is derived; then, the reconstruction and projection are given based on the model Then, use the projection rule to obtain the projection coefficient, and use the reconstruction rule to reconstruct the input face; then, repeat the above projection and reconstruction process for the reconstructed face; finally, the reconstructed multiple face The face image is output as a smile synthesis process for the input face.

2. smiling face composition method according to claim 1, is characterized in that, comprises following concrete steps:

Step 1. Learn to build a generative model for face representation;

The generative model finds a common spatial segmentation for a given face image, and learns a sparse component analysis model for each part of this segmentation;

Suppose the probability distribution of the production model is P(x,Z,R), and its calculation method is P(x,Z,R)=P(xR,Z)P(R)P(Z); first, derive the production The continuous induction prior of the model, and use PoE (product of experts) to combine the continuous induction prior and the polynomial prior to form P(R), namely:

\begin{matrix} P P ((R R)) = = \underset{d d}{Π Π} P P (({r r}_{d d . .})) = = \underset{d d}{Π Π} \frac{11}{{Z Z}_{d d}} {P P}_{11} (({r r}_{d d . .})) {P P}_{22} (({r r}_{d d . .})) \\ = = \frac{11}{{Z Z}_{00}} \underset{d d,, k k}{Π Π} {α α}_{d d k k}^{{r r}_{d d k k}} \underset{d d,, k k}{Π Π} exp exp {{- - {r r}_{d d k k} ((11 - - \frac{44}{| | {D D.}_{d d} | |} {Σ Σ}_{{d d}^{' '} &Element; &Element; {D D.}_{d d}} {r r}_{{d d}^{' '} k k}))}} \end{matrix} - - - - - - ((11))

Among them, Z ₀ is a normalization function, and the variable used to record the selection of face image pixels is r _d. =(r _d1 ,,r _dK ) ^T , and r _d. ~Mult(n=1,α _d. ), D _d is the adjacent variable set of variable r _dk , P ₁ (r _d. ) is polynomial prior, P ₂ (r _d. ) is continuity induced prior;

Next, each part of the face is modeled as a sparse component analysis model, and a sparse component analysis model is learned for each face part to obtain P(x|R, Z) and P(Z); specifically, for For a face pixel x _d , first select a part from K face parts, denoted as k, correspondingly generate a pixel from the kth sparse component analysis model; get the following model:

{x x}_{d d} = = {Σ Σ}_{k k = = 11}^{K K} {r r}_{d d k k} {Σ Σ}_{m m = = 11}^{M m} {z z}_{k k m m} {w w}_{d d k k m m} + + {μ μ}_{d d} + + {ϵ ϵ}_{d d};;

r _d. ~Multi(n=1, α _d. );

Z _km ~ Lap (u = 0, b = 1);

Among them, x _d is the pixel of the face, is the overcomplete base of the kth part, M is the number of bases, and μ _d is the mean value; is random noise; α _d. =(α _d1 ,, α _dK ) ^T is the parameter of multinomial distribution; from the above model, it can be obtained:

P P ((x x | | R R,, Z Z)) = = \underset{d d}{Π Π} P P (({x x}_{d d} | | {r r}_{d d . .},, Z Z)) = = \underset{d d}{Π Π} N N (({x x}_{d d};; \underset{k k}{Σ Σ} {r r}_{d d k k} \underset{m m}{Σ Σ} {z z}_{k k m m} {w w}_{d d k k m m} + + {μ μ}_{d d},, {σ σ}_{d d}^{22})) - - - - - - ((22))

P P ((Z Z)) = = \underset{k k,, m m}{Π Π} \frac{11}{22 b b} exp exp ((- - \frac{| | {z z}_{k k m m} - - u u | |}{b b})) - - - - - - ((33))

Considering P(R) (Equation (1)), P(Z) (Equation (3)) and P(x|R, Z) (Equation (2)), the production model P(x , Z, R):

\begin{matrix} P (x, Z, R) = P (x | R, Z) P (R) P (Z) \\ = \underset{d}{Π} N (x_{d}; \underset{k}{Σ} r_{d k} \underset{m}{Σ} z_{k m} w_{d k m} + μ_{d}, σ_{d}^{2}) \frac{1}{Z_{0}} \underset{d, k}{Π} α_{d k}^{r_{d k}} \underset{d, k}{Π} \exp {- r_{d k} (1 - \frac{4}{| {D.}_{d} |} Σ_{d^{'} &Element; {D.}_{d}} r_{d^{'} k})} \\ \underset{k, m}{Π} \frac{1}{2 b} \exp (- \frac{| z_{k m} - u |}{b}) \end{matrix} - - - (4)

Step 2. Calculating the projection coefficients for the synthesis of smiles on human faces;

Let the learned partial sparse component analysis model be θ; the projection coefficient of the sample X ^c on the model θ is the combination coefficient the mean value of

The mean of the coefficients can be estimated by:

{E E.}_{P P (({z z}_{k k m m}^{c c} | | {X x}^{c c},, θ θ))} [[{z z}_{k k m m}^{c c}]] \approx \approx \frac{11}{J J} {Σ Σ}_{i i = = 11}^{J J} {z z}_{k k m m}^{c c,, i i} - - - - - - ((55))

in, From Samples sampled in;

is the sampling distribution (or posterior distribution) of latent variable z _km , the present invention adopts Monte Carlo EM algorithm to estimate the posterior distribution of latent variable; Monte Carlo method first uses Gibbs sampling method to the sample of latent variable from posterior sampling , the Gibbs sampling distribution of r _dk can be expressed as:

P P (({r r}_{d d k k} | | x x,, {R R}_{- - d d k k},, Z Z)) = = \frac{P P ((x x,, Z Z | | R R)) P P (({r r}_{d d k k} | | {R R}_{- - d d k k})) P P (({R R}_{- - d d k k}))}{P P ((x x,, Z Z | | {R R}_{- - d d k k})) P P (({R R}_{- - d d k k}))} &Proportional; &Proportional; P P ((x x | | Z Z,, R R)) P P (({r r}_{d d k k} | | {R R}_{- - d d k k})) - - - - - - ((66))

Among them, R _-dk represents the variable set obtained after removing r _dk from R. Substituting formula (2) and formula (3) into the above formula, we get:

P P (({r r}_{d d k k} | | x x,, {R R}_{- - d d k k},, Z Z)) &Proportional; &Proportional; N N (({x x}_{d d};; \underset{k k}{Σ Σ} {r r}_{d d k k} \underset{m m}{Σ Σ} {z z}_{k k m m} {w w}_{d d k k m m} + + {μ μ}_{d d},, {σ σ}_{d d}^{22})) \cdot \cdot exp exp {{{r r}_{d d k k} ((\underset{{d d}^{' '} &Element; &Element; {D D.}_{d d}}{Σ Σ} {r r}_{{d d}^{' '} k k} - - 11))}} {α α}_{d d k k}^{{r r}_{d d k k}} - - - - - - ((77))

The sampling distribution of z _km can be similarly derived as follows:

P P (({z z}_{k k m m} | | x x,, R R,, {Z Z}_{- - k k m m})) &Proportional; &Proportional; \{\begin{matrix} N N (({z z}_{k k m m};; {μ μ}_{z z k k m m}^{+ +},, {σ σ}_{z z k k m m}^{22 + +})) & {z z}_{k k m m} &GreaterEqual; &Greater Equal; 00 & (({P P}_{+ +})) \\ N N (({z z}_{k k m m};; {μ μ}_{z z k k m m}^{- -},, {σ σ}_{z z k k m m}^{22 - -})) & {z z}_{k k m m} \leq \leq 00 & (({P P}_{- -})) \end{matrix} - - - - - - ((88))

in,

The resulting sampling distribution is discontinuous and can be sampled from the above distribution by:

(1) Sample from the distribution P ₊ in equation (8), and output non-negative samples (z _km ≥ 0);

(2) Sample from the distribution _{P_} in equation (8), and output negative samples (z _km ≤ 0);

(3) Combine the two sets of samples as output; It is the sample sampled from the distribution in this way; when using the model θ to synthesize the smile of a given image X ^c , first use the formula (5) to project the face image onto the model θ, and calculate the projection coefficient

Step 3. Give the reconstruction rules for the synthesis of smiles on human faces;

After calculating the projection coefficient, the reconstructed value of X ^c is the mean value of the conditional distribution shown in formula (2):

{\overset{^^}{x x}}_{d d}^{c c} = = \underset{k k}{Σ Σ} {α α}_{d d k k} \underset{m m}{Σ Σ} {E E.}_{P P (({z z}_{k k m m}^{c c} | | {X x}^{c c},, θ θ))} [[{z z}_{k k m m}^{c c}]] {w w}_{d d k k m m} + + {u u}_{d d} - - - - - - ((99))

Reconstruct the face on the model θ by formula (9);

Step 4. Repeat steps 2 and 3 until all the intermediate faces are output, and finally synthesize the smile on the face.