CN105069746A - Video real-time human face substitution method and system based on partial affine and color transfer technology - Google Patents

Video real-time human face substitution method and system based on partial affine and color transfer technology Download PDF

Info

Publication number
CN105069746A
CN105069746A CN 201510520746 CN201510520746A CN105069746A CN 105069746 A CN105069746 A CN 105069746A CN 201510520746 CN201510520746 CN 201510520746 CN 201510520746 A CN201510520746 A CN 201510520746A CN 105069746 A CN105069746 A CN 105069746A
Authority
CN
Grant status
Application
Patent type
Prior art keywords
face
image
target
video
human face
Prior art date
Application number
CN 201510520746
Other languages
Chinese (zh)
Other versions
CN105069746B (en )
Inventor
孙国辉
Original Assignee
杭州欣禾圣世科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06KRECOGNITION OF DATA; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
    • G06K9/00Methods or arrangements for reading or recognising printed or written characters or for recognising patterns, e.g. fingerprints
    • G06K9/00221Acquiring or recognising human faces, facial parts, facial sketches, facial expressions
    • G06K9/00228Detection; Localisation; Normalisation
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06KRECOGNITION OF DATA; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
    • G06K9/00Methods or arrangements for reading or recognising printed or written characters or for recognising patterns, e.g. fingerprints
    • G06K9/00221Acquiring or recognising human faces, facial parts, facial sketches, facial expressions
    • G06K9/00288Classification, e.g. identification
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00Geometric image transformation in the plane of the image, e.g. from bit-mapped to bit-mapped creating a different image
    • G06T3/0006Affine transformations

Abstract

The invention relates to a video real-time human face substitution method and a video real-time human face substitution system based on a partial affine and color transfer technology, which solve the defect that the real-time human face substitution cannot be achieved when compared with the prior art. The video real-time human face substitution method comprises the steps of: carrying out video acquisition to obtain target human face video through a camera, and intercepting a current frame image; detecting a human face and feature points thereof; carrying out human face substitution; carrying out human face fusion processing; checking whether the camera is turned off; finishing the human face substitution if the camera is turned off; and continuing the video acquisition and acquiring the current frame image if the camera is not turned off, and continuing the human face detection, human face substitution and human face fusion steps. The video real-time human face substitution method and the video real-time human face substitution system improve quality, speed and efficiency of human face substitution, and can be applied to real-time substitution in the video.

Description

基于局部仿射和颜色迁移技术的视频实时人脸替换方法及其系统 Based on local affine real-time video and color transfer technology to replace human face method and system

技术领域 FIELD

[0001] 本发明涉及图像处理技术领域,具体来说是基于局部仿射和颜色迀移技术的视频实时人脸替换方法及其系统。 [0001] The present invention relates to image processing technology, and in particular is based on local affine Gan live video and color replacement method and system of the face-shift technique.

背景技术 Background technique

[0002] 人脸替换技术是计算机视觉领域中一个重要的研究方向,由于人脸替换技术代替了Photoshop等软件手工进行图像编辑融合等各种弊端,因而在商业、娱乐及其一些特殊的行业有着巨大的影响。 [0002] face replacement technology is an important area of ​​research in computer vision, due to the replacement technology replaces the manual Photoshop and other software face various disadvantages image editing fusion, which has in the business, entertainment and some special industries big influence. 例如,需要特技演员演出的一些危险动作片,都是特技替身先进行拍摄,然后在后期制作中,使用人脸自动替换技术把特技演员的脸替换成需要的人物脸,来达到影片最终的目的。 For example, you need to perform some dangerous stunt action movie, are a stunt double to shoot first, and then in post-production, the use of automatic face replacement technology to replace the stunt actor's face to face person needs to achieve the ultimate purpose of the video . 现阶段很多研究人员已经提出了不同的替换策略,取得了一定的成功,但是将其应用于视频中时无法实现。 Stage many researchers have proposed different alternatives strategy, and achieved some success, but will not be achieved when applied to the video. 主要原因在于视频人脸替换技术的特殊性,其需要做到实时性、快速度和高效率,而传统的人脸替换技术存在计算量大、有色差、时间耗费高的缺点,并且替换后的效果较差,故不能用于实际生活的视频应用中。 The main reason is that video face replacement technologies particularity, it needs to be done in real-time, fast and high efficiency, but the traditional face computationally intensive alternative techniques exist, there is color, high time consumption disadvantages and to replace the less effective, it can not be used for video applications in real life.

[0003] 现有技术中的几种人脸置换方法均无法应用于视频实时处理中,如基于泊松编辑的人脸置换方法:其根据原始人脸语义信息,在人脸图像库中自动选取语义相似的目标人脸并构建其三维模型,然后根据原始人脸的姿态和光照估计信息对目标人脸进行调整,之后通过泊松图像融合技术使目标人脸与原始图像无缝融合,使得目标人脸与原始图像/视频的无缝融合,达到替换逼真自然,从而实现脸的自动替换。 [0003] Several prior art method of replacing the face are not applied in real-time video processing, such as face replacement method based on Poisson editing: primitive face based on semantic information, is automatically selected in the facial image database semantically similar target face and construct three-dimensional models and estimation information to adjust the target primitive human face the face pose and illumination, after fusion of the target facial image and the original image Poisson seamless integration, such that the target seamless integration of the human face and the original image / video, to replace the realistic nature, in order to achieve the automatic replacement of the face. 其优点是融合后的人脸缝隙不明显;其缺点是融合后的图像有色差且运算量大,导致实时性较差,不能用于视频。 The advantage is that a face gap fused obvious; drawback is fused color image and the large amount of computation, resulting in poor real-time, not for video.

[0004] 再如基于3D人脸数据库的人脸替换方法,该方法首先对下载人脸数据库进行人脸配准和离线学习,获得每个图像中人脸的姿态、光照和表情等参数;当输入1张图像时, 系统会进行人脸检测,在图像数据库中获得与输入人脸姿态、分辨率、光照相似的人脸作为人脸替换的备选;之后对候选人进行色彩调整和光照矫正以匹配输入人脸,并完成图像融合;最后设计了一个融合边缘评价函数对所得融合结果进行排序,用融合最好的结果作为最终的替换结果。 [0004] Another example of an alternative method based on human face 3D face database, the method first face database downloaded face alignment and offline learning, the image obtained for each human face pose, illumination and expression parameters; when when an image is input, the system performs face detection, face pose obtain input resolution, similar to the lighting face as the face candidate alternative in the image database; candidates after color adjustment and correction of the light to match the input face, and the image is fused; last, a fusion of the resulting fused edge evaluation function to sort the results, with the best fusion results as the final result of the replacement. 其优点是可以融合效果较好;其缺点是不能保证用指定的1张人脸替换目标人脸就能达到很好的效果;实时性较差。 The advantage is better integration; the drawback is no guarantee that a replacement with the specified target face face can achieve good results; the real poor.

[0005] 再如基于实时视频的人脸替换方法,该方法采用AAM方法检测人脸及其特征点, 而后根据特征点获取到目标人脸,源图像人脸模板;根据三角仿射原理,将源图像的人脸形状仿射成目标人脸的形状,而后调整颜色权值,使得源图像人脸的颜色与目标人脸的颜色尽量一致。 [0005] Another example of real-time video face alternative method based on the method employed and AAM method of detecting a face feature points, and then obtains a target face, face template source image feature points; affine triangulation principle, face shape of the source image to affine shape of the target face, and then adjust the color weights, such that the color of the target color in the source image the face of a human face as consistent as possible. 其优点是实时性好,没有延迟;缺点是人脸检测率低;仿射后的人脸形状与目标人脸形状有差距。 The advantage is real-time, without delay; drawback is the face detection rate is low; the shape of a human face to the target face shape after affine gap. 颜色调整不是自动变化的,只是一个经验值。 Color adjustment is not automatically change, but an empirical value.

[0006] 以上二种方法是最近较为流行的方法,其他的方法都是在其基础上做的改进,泊松编辑算法融合效果虽然比实时视频算法好,但是时间效率却无法满足视频实时性需求; 基于3D人脸数据库的方法融合效果最好,但是时间耗费也是最高的,处理一帧时间达到了9秒。 [0006] the above two methods is more recent popular methods, other methods are done on the basis of its improvements, Poisson editing algorithm fusion effect, although better than the real-time video algorithms, but time efficiency can not meet the needs of real-time video ; fusion method based on a 3D face database is best, but also the most time-consuming, the processing time to reach a 9 seconds.

[0007] 针对各种人脸替换技术存在的局限性,在现有的硬件条件下,如何设计出一种有效、快速的能够在视频应用中使用的实时人脸替换方法已经成为当今急需解决的技术问题。 [0007] For the replacement of technical limitations of various human face, in the existing hardware conditions, how to design an effective and rapid real-time person that can be used in video applications face replacement method has become the urgent problem technical problem.

发明内容 SUMMARY

[0008] 本发明的目的是为了解决现有技术中无法做到实时人脸替换的缺陷,提供一种基于局部仿射和颜色迀移技术的视频实时人脸替换方法及其系统来解决上述问题。 [0008] The object of the present invention is to solve the prior art can not achieve real-time face-replaced defect, providing real-time video face alternative method and system based on local affine Gan and color shifting techniques to solve the above problems .

[0009] 为了实现上述目的,本发明的技术方案如下: [0009] To achieve the above object, the technical solution of the present invention is as follows:

[0010] 基于局部仿射和颜色迀移技术的视频实时人脸替换方法,包括以下步骤: [0010] The method of alternative real-time video face local affine shift and color Gan Technology, comprising the steps of:

[0011] 视频采集,通过摄像头获取目标人脸视频,并截取当前帧图像; [0011] The video capture, video acquisition target face through the camera, and the interception of the current frame image;

[0012] 进行人脸及其特征点检测,对获取到的目标人脸图像,使用强分类器来检测人脸, 在已经构建的形状模型的基础上对人脸库中的样本进行训练,将目标人脸特征点和人脸库中的样本特征点进行匹配,做出相应的变换,并对检测到的目标人脸进行特征点搜索、位置标记; [0012] and human face feature point detection on the acquired face image of the target using the strong classifier to detect a human face, the face of library samples for training the shape model has been constructed on the certain facial features, and the sample feature point in the face database match, and make the appropriate transform, and the target person detected face feature point search, a position marker;

[0013] 进行人脸变换,利用仿射变换参数,将人脸库中的样本脸仿射变换到视频中的目标人脸上,进行人脸的替换; [0013] for face transformation using the affine transformation parameters, face to face sample library affine transformation to the target person in the video's face, face replacement;

[0014] 进行人脸融合处理,将变换前的目标人脸的颜色迀移到变换后的目标人脸上,采用拉普拉斯高斯金字塔进行融合处理,并对融合处理后的目标人脸图像的边缘采用高斯滤波器进行平滑处理后显示到摄像头获取到的图像中; [0014] for face integration process, the pre-conversion target face color Gan target person moves to transform the face after using Laplace Gaussian pyramid fusion process, and after the target person face image fusion the rear edge of Gaussian smoothing filter to display the camera image acquired;

[0015] 检查摄像头是否关闭,若关闭,则结束人脸替换;若未关闭,则继续进行视频采集并获取当前帧图像,继续进行人脸检测、人脸变换和人脸融合步骤。 [0015] Check whether the camera off, when closed, the end face replaced; if not closed, the video capture and continue to get the current frame image, proceed face detection, face and face fusion step transform.

[0016] 所述的进行人脸及其特征点检测包括以下步骤: [0016] and the human face feature point detection comprising the steps of:

[0017] 读取人脸库中的样本数据,构造出若干个分类器,并将若干个分类器级联成一个强分类器; [0017] The read sample data in the face database constructed out of a plurality of classifiers and classifiers to cascade a plurality of strong classifiers;

[0018] 读取当前视频帧图像,使用强分类器对当前视频帧图像进行人脸检测; [0018] reads the current video frame image, the strong classifier using the current video frame image for face detection;

[0019] 构建形状模型并进行训练; [0019] Construction and shape model training;

[0020] 构造人脸特征点,得到仿射变换参数。 [0020] The configuration of the face feature points, affine transformation parameters obtained.

[0021] 所述的进行人脸变换包括以下步骤: [0021] transforming the human face comprising the steps of:

[0022] 对目标人脸和人脸库中样本的特征点计算出每个特征点的坐标,并将相邻的任意三个点进行特征点三角化; [0022] For any three points target face and facial features library samples calculate the coordinates of each feature point, and feature points adjacent triangulation;

[0023] 利用仿射变换参数,将人脸库中三角化后的特征点分别映射到目标人脸上,变换为目标人脸的形状,其构造的仿射变换具体步骤如下: [0023] using the affine transformation parameters, face database mapped feature points of the triangular face of the target person, is converted into the shape of the target face, specifically an affine transformation step is configured as follows:

[0024] 对于特征点三角化后的目标人脸和人脸库中的样本,找出人脸库中每个三角形和对应的目标人脸的三角形的位置; [0024] For the target feature points of the triangle faces and the face database samples to identify each triangular face database and corresponding target face position of the triangle;

[0025] 保持人脸库中每个特征点的位置关系不变,将人脸库中的每个特征点三角形映射到目标人脸特征点三角形所在的位置。 [0025] maintain the face database each feature point position relationship unchanged, will each feature point of the triangle face database mapped to the target location where the facial features of a triangle.

[0026] 所述的进行人脸融合处理包括以下步骤: [0026] The fusion process of human face comprising the steps of:

[0027] 在laP空间上将变换前的目标人脸的颜色迀移到变换后的目标人脸上,其中1 表示非彩色的亮度通道,a表示彩色黄蓝通道,P表示红绿通道; [0027] In the space before conversion on laP target face Gan target color after the transforming human face, wherein 1 represents achromatic luminance channel, a represents the yellow-blue color channel, P represents the red-green channel;

[0028] 采用拉普拉斯高斯金字塔的方法对迀移后的目标人脸的不同尺度、不同分解层的图像进行融合处理; [0028] The Laplacian of Gaussian pyramid method Gan different scales after shifting the target face, the image decomposition layer is different fusion;

[0029] 对融合处理后的目标人脸图像寻找边缘区域,采用高斯滤波器进行平滑处理。 [0029] Find the edge region of the fusion target human face image, Gaussian smoothing filter.

[0030] 所述的构造出若干个分类器包括以下步骤: [0030] The configuration of a plurality of classifiers comprises the steps of:

[0031] 均匀分布人脸库中的每个样本,通过训练得到初始分类器H。 [0031] uniform distribution of each sample face database by training a classifier to obtain an initial H. ;

[0032] 分类判断,对于分类正确的,降低其分布概率;分类错误的,提高其分布概率,得到一个新的训练集S1; [0032] The classification is determined, for correct classification, reducing the probability distribution; misclassification to improve the probability distribution to obtain a new training set Sl;

[0033] 获得分类器,使用训练集5:进行训练,得到分类器H1; [0033] The classifier is obtained, using the training set 5: is trained classifier Hl;

[0034] 迭代进行分类判断和获得分类器步骤共T次,得到{氏,H2,…,HT}共T个分类器。 [0034] Analyzing and classifying iteration classification step to obtain a total time T, to give {s, H2, ..., HT} total T classifiers.

[0035] 所述的构建形状模型并进行训练包括以下步骤: [0035] Construction of the shape model and training comprising the steps of:

[0036] 搜集400个人脸训练样本,并标记出样本中的每个脸部特征点; [0036] 400 facial collect training samples, and each mark in the facial feature points in the sample;

[0037] 将训练集中特征点的坐标串成特征向量; [0037] The coordinate of the feature point in the training set strung feature vector;

[0038] 对形状特征进行归一化和对齐处理; [0038] wherein the shape and alignment normalizing process;

[0039] 对对齐后的形状特征做主成分分析处理,主成分分析的构造具体步骤如下: [0039] The shape of the alignment feature shots component analysis, principal component analysis step is configured specifically as follows:

[0040] 输入X,X= [X1X2…xjT是m元向量变量,计算X的样本矩阵,其计算公式如下: [0040] The input X, X = [X1X2 ... xjT is m-ary vector variable, the sample matrix X is calculated, which is calculated as follows:

Figure CN105069746AD00081

[0050]其中伞=[巾丨伞J是mXm的正交向量矩阵,A= diag{入丨,入2,… ,入丄A2多…多A"是对角特征值矩阵; [0050] wherein the umbrella = [J towel Shu umbrella is mXm orthogonal vector matrix, A = diag {into Shu, into 2, ..., ... over the multi-A2 Shang A "is a diagonal matrix wherein a value;

[0051] 计算正交转换矩阵P,其计算公式如下: [0051] The orthogonal transformation matrix calculating P, is calculated as follows:

[0052] P=伞T; [0052] P = T umbrella;

[0053]将正交转换矩阵关联到J,得到主成分分析F=pj,, [0053] The orthogonal transformation matrix relative to J, principal component analysis to obtain F = pj ,,

[0054] 为每个样本的脸部特征点构建局部特征。 [0054] Construction of local facial feature points wherein each sample.

[0055] 所述的构造人脸特征点包括以下步骤: [0055] The configuration of facial features comprising the steps of:

[0056] 特征点位置计算,计算目标人脸特征点的位置,并做尺度和旋转变化; [0056] The feature point position is calculated, the calculated position of the target face feature points, and make changes in rotation and scale;

[0057] 将目标人脸的特征点和人脸库中的样本的每个局部特征点进行匹配,计算出每个局部特征点对应到目标人脸上的新的位置; [0057] Each local feature points in the feature point of the target sample face and the face database by matching the local feature points calculated for each new position corresponding to the target person's face;

[0058] 迭代进行上述步骤,得到仿射变换参数。 [0058] The above steps are iterated to obtain affine transformation parameters.

[0059] 所述的利用拉普拉斯高斯金字塔的方法对迀移后的目标人脸的不同尺度、不同分解层的图像进行融合处理包括以下步骤: [0059] The use of the Laplacian of Gaussian pyramid method different scales of the target person's face after the shift Gan, images of different layers are fused decomposition process comprising the steps of:

[0060] 对于颜色迀移后的目标人脸采用高斯金字塔获取多幅不同空间层上、多尺度的下采样图像,构建出图像金字塔,构造的高斯金字塔具体步骤如下: [0060] After the target person's face color shift Gan Gaussian pyramid obtaining multiple different spatial layers, the multi-scale downsampling image, build the image pyramid, the Gaussian pyramid configuration specific steps are as follows:

[0061] 输入颜色迀移后的目标人脸图像G。 [0061] After the target face image input color shift Gan G. ,以G。 To G. 作为高斯金字塔的第0层; As a Gaussian pyramid of layer 0;

[0062] 对原始输入图像G。 [0062] The original input image G. 进行高斯低通滤波和隔行隔列的下采样,得到高斯金字塔的第一层图像G1; Gaussian low pass filtering and downsampling the interlaced every other column to obtain a first layer of the Gaussian pyramid image G1;

[0063] 对第一层图像G1进行高斯低通滤波和隔行隔列的下采样,得到高斯金字塔的第二层图像G2; [0063] The first layer image G1 downsample Gaussian low-pass filtered interlaced and every other column to obtain a second layer Gaussian pyramid image G2;

[0064]对第1-1层图像G1i进行高斯低通滤波和隔行隔列的下采样,得到高斯金字塔的第1(1彡1彡N)层图像G1, [0064] The layer image G1i 1-1 downsample Gaussian low-pass filtered interlaced and every other column to obtain a first Gaussian pyramid (1 San San 1 N) layer image G1,

[0065] [0065]

Figure CN105069746AD00091

[0067] 重复以上过程,构成最终的高斯金字塔; [0067] The process is repeated to make the final Gaussian pyramid;

[0068] 对不同的分解层的不同频带上的金字塔图像采用拉普拉斯金字塔从金字塔顶层图像中向上采样来重建出上层图像,构造的拉普拉斯金字塔具体步骤如下: [0068] The pyramid image on different frequency bands using different decomposition level Laplacian pyramid image up-sampled from the top of the pyramid to reconstruct the upper layer in the image, the Laplacian pyramid configuration specific steps are as follows:

[0069] 对得到的高斯金字塔的顶层图像Gn使用内插法得到放大图像G:,其中N为高斯金字塔的最大层数; [0069] G to obtain an enlarged image of the image interpolation top Gn obtained using Gaussian pyramid :, where N is the maximum number of layers of the Gaussian pyramid;

[0070]对第NI层图像Gni使用内插法得到放大图像J [0070] J to obtain an enlarged image of the interpolation method using the NI layer image Gni

[0071] 将高斯金字塔的第1层图像G1使用内插法得到放大图像G Zn,其计算公式如下: [0071] The inner layer of the first interpolation image using a Gaussian pyramid G1 to obtain an enlarged image G Zn, is calculated as follows:

[0072] [0072]

Figure CN105069746AD00092

[0073] 其中N为高斯金字塔的最大层数,RjP C别为高斯金字塔的第1层的行数和列 [0073] where N is the maximum number of layers of the Gaussian pyramid, RjP C Gaussian pyramid respectively the number of rows and the columns of the first layer

Figure CN105069746AD00101

普拉斯金字塔分解的第1层图像; Plath pyramid decomposition of the image layer;

[0076] 重复高斯金字塔层各层的计算,得到拉普拉斯金字塔Ii^LP1,…,LP1,…,LPn; [0076] Gaussian pyramid levels repeated calculation of each layer to obtain Laplacian pyramid Ii ^ LP1, ..., LP1, ..., LPn;

[0077] 对重建后的图像进行合并、融合处理。 [0077] The reconstructed images are combined, the fusion process.

[0078] 基于局部仿射和颜色迀移技术的视频实时人脸替换系统,包括: [0078] Real-time video face alternative system and local affine color shifting Based Gan, comprising:

[0079] 视频采集模块,用于从摄像头下获取的视频中采集每一帧的人脸图像; [0079] The video capture module configured to capture a facial image of each frame is acquired from the video camera in the lower;

[0080] 分类器构造模块,用于对获取到的视频中的图像进行人脸检测; [0080] classifier module configured, for the acquired video image for face detection;

[0081] 形状模型训练模块,用于为每个人脸特征点构建局部特征,建立出每个特征点的位置约束; [0081] The shape of the model training module for each facial feature local feature point construct, for each constraint to establish the position of the feature point;

[0082] 主成分分析模块,用于通过形状模型构造模块来对形状特征做特征提取处理; [0082] Principal component analysis module for feature extraction process to do the shape wherein the shape model configuration module;

[0083] 人脸特征点搜索模块,用于搜索人脸特征点并计算出特征点所在的位置; [0083] The facial features search module for searching the facial feature points and calculate the position of the characteristic point is located;

[0084] 人脸仿射变换模块,用于将人脸库中的样本映射到目标人脸的相应位置上; [0084] Face affine transformation module, for mapping the sample face database to a corresponding position on the target face;

[0085] 基于拉普拉斯高斯金字塔图像融合模块,用于将颜色迀移后的目标人脸进行相应的融合处理; [0085] Fusion module Laplacian of Gaussian pyramid image based on, for the shifting target color Gan face corresponding fusion;

[0086] 所述的视频采集输入模块与分类器构造模型相连,所述的分类器构造模块的输出端分别与形状模型训练模块和主成分分析模块相连,形状模型训练模块和主成分分析模块分别与人脸特征点搜索模块的输入端相连,人脸特征点搜索模块的输出端与人脸仿射变换模块相连,人脸仿射变换模块的输出端与基于拉普拉斯高斯金字塔图像融合模块相连。 [0086] The video capture module and the input is connected to a classifier constructor model, the output of the classification module is configured respectively connected with the shape model training module and principal component analysis module, the shape of the model training module, respectively, and principal component analysis module connected to the face feature point search module inputs, facial features search module and the output terminal of the affine transformation module connected to the face, the face affine transformation module output of a Laplacian of Gaussian pyramid image fusion based module connected.

[0087] 有益效果 [0087] beneficial effects

[0088] 本发明的基于局部仿射和颜色迀移技术的视频实时人脸替换方法及其系统,与现有技术相比提高了人脸替换的质量、速度和效率,能够用于视频中进行实时替换。 [0088] Based on local affine Gan live video and color replacement method and a face shift system technology, as compared with the prior art to improve the quality of face Alternatively, the speed and efficiency of the present invention can be used in the video Real-time replacement. 利用强分类器可以准确的检测人脸;利用构造的形状模型、主成分分析和人脸特征点搜索的应用可以快速的检测出人脸的特征点;利用局部仿射变换可以将人脸库中的样本准确的映射到目标人脸上;利用拉普拉斯高斯金字塔的图像融合技术,可以将颜色迀移后的目标人脸进行分解、重建,进而准确的融合处理。 Using the strong classifier can accurately detect a human face; application using the constructed shape model, principal component analysis and facial features can be searched quickly detect the face feature point; using local affine transformation can face database the sample accurately mapped to the target person's face; the use of Laplace Gaussian pyramid image fusion technology, the color-Gan can move the target face decomposition, reconstruction, and thus accurate fusion. 整个人脸替换过程是在摄像头下实时的、准确的、快速的进行的,突破了现有技术进行人脸替换时时间耗费高、无法满足视频实时性的需求,同时替换后的图像有色差、运算量大的缺陷。 The whole face replacement process in real time in the camera, accurate and fast perform, breaking the existing technology a human face to replace time-consuming high, can not meet the real-time video of the demand, while the image after the replacement of a color, the computation is defective.

附图说明 BRIEF DESCRIPTION

[0089] 图1为本发明的方法顺序图; [0089] The method of the present invention of FIG. 1 FIG sequence;

[0090] 图2为本发明的系统连接图。 [0090] The connector system of Figure 2 of the present invention.

具体实施方式 detailed description

[0091] 为使对本发明的结构特征及所达成的功效有更进一步的了解与认识,用以较佳的实施例及附图配合详细的说明,说明如下: [0091] For a better understanding and knowledge of the structural features and effects of the present invention reached to the preferred embodiment and the detailed description of the embodiments with the accompanying drawings, as follows:

[0092] 如图1所示,本发明所述的基于局部仿射和颜色迀移技术的视频实时人脸替换方法,包括以下步骤: [0092] 1, according to the present invention, real-time video face alternative method of local affine shift technique and the color Gan-based, comprising the steps of:

[0093] 第一步,视频采集,读取当前摄像头下的视频,通过摄像头获取目标人脸视频,并截取当前帧图像,为后期处理做准备。 [0093] The first step, video capture, read the video camera under the current, acquiring the target face by video camera, and the interception of the current frame image, to prepare for post-processing.

[0094] 第二步,进行人脸及其特征点检测。 [0094] The second step, and the facial feature point detection. 对获取到的目标人脸图像,使用强分类器来检测人脸,在已经构建的形状模型的基础上对人脸库中的样本进行训练,将目标人脸特征点和人脸库中的样本特征点进行匹配,做出相应的变换,并对检测到的目标人脸进行特征点搜索、位置标记。 Of the acquired target face image, using the strong classifier to detect a human face, human face library samples for training the shape model has been constructed on the sample target facial features and face library matching feature points, and make the appropriate transform, and the detected target face feature point search, location marker. 此处特征点检测是为了后期人脸替换做准备,如果人脸特征点能够检测精准,则后期实时处理的效率就会更高,否则,就会一直检测人脸及其特征点,直到特征点检测到为止,因此在这里特征点检测非常关键,关系着能否进行实时的人脸替换。 Feature point here is to detect the human face late replacement to prepare, if facial features can be detected accurately, the late real-time processing efficiency will be higher, otherwise, would have been detected and facial feature points until the feature points detected so far, so here feature detection is critical bearing on whether real-time face replacement. 其具体步骤如下: The specific steps are as follows:

[0095] (1)读取人脸库中的样本数据,构造出若干个分类器,并将若干个分类器级联成一个强分类器。 [0095] (1) read sample data in the face database constructed out of a plurality of classifiers and classifiers to cascade a plurality of strong classifiers. 构造出若干个分类器包括以下步骤: Construct a plurality of classifiers comprises the steps of:

[0096] A、均匀分布人脸库中的每个样本,人脸库为保存众多人脸信息的数据库,通过训练得到初始分类器H。 [0096] A, each sample uniform distribution face library, face database is a database of many people face information stored by the initial classifier is trained H. .

[0097] B、分类判断,对于分类正确的,降低其分布概率;分类错误的,提高其分布概率,得到一个新的训练集Sp [0097] B, the classification is determined for correct classification, reducing the probability distribution; misclassification to improve the probability distribution, a new training set to get Sp

[0098] C、获得新分类器,使用训练集5:进行训练,得到新分类器H1<3 [0098] C, to obtain a new classifier training set 5: training, the new classifier to obtain H1 <3

[0099] D、迭代进行分类判断和获得分类器步骤共T次,得到{氏,H2,…,HT}共T个分类器。 [0099] D, is determined and the classification iteration classification step to obtain a total time T, to give {s, H2, ..., HT} total T classifiers. 将T个个分类器级联成一个强分类器,并运用该强分类器来对获得的当前帧图像进行人脸检测。 All the T and cascaded into a strong classifier, and the strong classifier using face detection is performed on the current frame image is obtained.

[0100] (2)读取当前视频帧图像,使用强分类器对当前视频帧图像进行人脸检测。 [0100] (2) reads the current video frame image, the strong classifier using the current video frame image for face detection.

[0101] (3)构建形状模型并进行训练,其具体包括以下步骤: [0101] (3) constructed shape model and training, which includes the following steps:

[0102] A、搜集400个人脸训练样本,人脸训练样本数量可以进行相应的调整,样本库多则更好,但同时训练样本的时间会更多。 [0102] A, gathering 400 people face training samples, the number of people face training samples can be adjusted, as many sample libraries better, but the time while training samples will be more. 手动标记出样本中的每个脸部特征点。 Manually marked feature points of each face of the sample. 如果库里的明星脸的脸部特征点都用软件来标记,可能会出现误标记的情况,那么后期再检测人脸并替换就会出现误差。 If the facial feature points star face of library software are used to mark, there may be circumstances mislabeled, then again late detect faces and replace the error occurs. 因此样本库的采集为人为采集方式,然后人为标记,在训练时人为标记,后期进行识别目标人脸时软件才会自动标记到目标人脸上。 Therefore sample library collection is artificially acquisition mode, and then marked man, man-made mark in training, the latter conducted face when identifying target software will automatically tag to target people face. 这里的样本库是明星脸库, 在实际应用中,要先建立好库,再建立好模板。 Here is a sample library star face database, in practical application, must first establish a good library, and then create a good template.

[0103]B、将训练集中特征点的坐标串成特征向量。 [0103] B, the training set feature point coordinates strung eigenvectors.

[0104] C、对形状特征进行归一化和对齐处理,形状特征指手动标记出样本中的脸部特征点,如眼睛、颧骨等。 [0104] C, wherein the shape and alignment normalizing treatment, wherein the shape of the manually tag refers facial feature points in the sample, such as the eyes, cheekbones like.

[0105] D、对对齐后的形状特征做主成分分析处理,主成分分析的构造具体步骤如下: [0105] D, wherein the shape after alignment shots component analysis, principal component analysis step is configured specifically as follows:

[0106] a、输入x,x= [X1X2…xjT是m元向量变量,计算x的样本矩阵,其计算公式如下: [0106] a, an input x, x = [X1X2 ... xjT is m-ary vector variable, the sample matrix of x, which is calculated as follows:

Figure CN105069746AD00121

[0116] 其中巾=[巾•巾J是mXm的正交向量矩阵,A=diag{入丨,入2,… ,入丄A2多…多A"是对角特征值矩阵; [0116] wherein towel = [J towels are towels • mXm orthogonal vector matrix, A = diag {into Shu, into 2, ..., ... over the multi-A2 Shang A "is a diagonal matrix wherein a value;

[0117] f、计算正交转换矩阵P,其计算公式如下: [0117] f, calculating an orthogonal transformation matrix P, is calculated as follows:

[0118] P= (})T; [0118] P = (}) T;

[0119] g、将正交转换矩阵关联到X,得到主成分分析7 =Pl, [0119] g, associated with the orthogonal transform matrix to X, principal component analysis to obtain 7 = Pl,

[0120] h、为每个样本的脸部特征点构建局部特征。 [0120] h, constructed as a face feature local feature point of each sample.

[0121] (4)构造人脸特征点,得到仿射变换参数。 [0121] (4) configured facial feature points, affine transformation parameters obtained. 其具体步骤如下: The specific steps are as follows:

[0122] A、特征点位置计算。 [0122] A, the feature point position calculation. 计算目标人脸特征点的位置,并做简单尺度和旋转变化。 Calculates the target facial features of the location, scale and rotation and make simple changes. 由于明星脸库的脸部大小和目标人脸的大小不同,这样特征点的位置肯定就不一样,比方说, 明星脸库明星的脸大,而目标人脸是小脸,那就需要做尺度和旋转变化了。 Depending on the size of the library's star face size of the face and the face of the target, the location of such feature points is certainly not the case, say, a big star face star face database, and the target is to face little face, it would need to do scale and changes in the rotation.

[0123] B、将目标人脸的特征点和人脸库中的样本的每个局部特征点进行匹配,计算出每个局部特征点对应到目标人脸上的新的位置; [0123] B, each local feature point of the target sample feature point of the face in the face database and matching the calculated local feature points corresponding to each new position of the target person's face;

[0124] C、迭代进行上述步骤,得到仿射变换参数。 [0124] C, the above step iteration, obtained affine transformation parameters. 仿射变换参数是目标人脸的特征点与人脸库中的样本的局部特征点匹配过程中所产生的转换矩阵,比如说目标人脸上某一个点到样本上的某个点上,我们需要从一个点到另一个点,那就需要变换,比如加或者减法,或者除法等等,这是点对点的变换,那如果是所有的目标人脸的特征点对应到样本上所有特征点上,那就是矩阵了,我们称为变换矩阵。 Affine transformation parameters is the transformation matrix local feature point matching process target sample feature points of the face and in the face database generated by, for example, on a point on a target sample point to the person's face, we required from one point to another, it would need to change, such as addition or subtraction, division, or the like, which is a point transformation, that if all the target feature point corresponding to a human face to all the sample feature points, that is a matrix, which we call the transformation matrix.

[0125] 第三步,进行人脸变换。 [0125] The third step is for face transformation. 利用仿射变换参数,将人脸库中的样本脸仿射变换到视频中的目标人脸上,进行人脸的替换。 Affine transformation parameters, will face database sample face an affine transformation to the target person in the video face replacement face. 局部仿射目的是让明星脸的形状与目标脸的形状对应一致,在人脸替换过程中显得更加真实。 Local affine goal is to shape the face of the star shape corresponding to the same target face, the face in the process of replacing all the more real. 简单来说,仿射方法做的事情就是将一个图形映射到另一个位置,在映射过程中图形本身的大小,形状可能发生变化,但仍然保留着图形内部点与点之间的位置关系。 In simple terms, affine method is to do a graphical map to another location, the size of the graphic itself, the shape may change during the mapping process, but still retains the positional relationship between the graphics inside the dots. 仿射基础是三角定位方法,所以要进行仿射变换必须计算三点坐标也就是特征点三角化。 Affine triangulation method is based on, so to calculate affine transformation must i.e. three coordinates of the feature point triangle. 将整幅变换图像分为各个子区域,对各个子区域进行仿射变换, 基于局部仿射变换的图像比基于全局变换的图像的效果更接近于目标图像。 Converting the whole image is divided into various sub-regions, each sub-region of the affine transformation, the affine transformation based on local image closer to the target image based on the image than the effect of the global transformation. 其具体步骤如下: The specific steps are as follows:

[0126] (1)对目标人脸和人脸库中样本的特征点计算出每个特征点的坐标,并将相邻的任意三个点进行特征点三角化。 [0126] (1) the target face feature point and the sample face database calculate the coordinates of each feature point, and feature points of any three adjacent triangulation points.

[0127] (2)利用仿射变换参数,将人脸库中三角化后的特征点分别映射到目标人脸上,变换为目标人脸的形状,其构造的仿射变换具体步骤如下: [0127] (2) using the affine transformation parameters, the feature points of the triangular face database are mapped to the target person's face, converting the shape of the target face, specifically an affine transformation step is configured as follows:

[0128] A、对于特征点三角化后的目标人脸和人脸库中的样本,找出人脸库中每个三角形和对应的目标人脸的三角形的位置。 [0128] A, for the feature points of the target cam face and the face database samples to identify each triangular face database and corresponding target position triangular face.

[0129] B、保持人脸库中每个特征点的位置关系不变,将人脸库中的每个特征点三角形映射到目标人脸特征点三角形所在的位置。 [0129] B, the positional relationship of each feature point of the face database maintained constant, each feature point triangle face database are mapped to the target face position where the feature point triangle.

[0130] 第四步,进行人脸融合处理。 [0130] The fourth step is for face fusion. 将变换前的目标人脸的颜色迀移到变换后的目标人脸上,采用拉普拉斯高斯金字塔进行融合处理,并对融合处理后的目标人脸图像的边缘采用高斯滤波器进行平滑处理后显示到摄像头获取到的图像中。 The target face before conversion Gan target color after the transforming human face, using a Laplacian of Gaussian pyramid fusion process, and the edge of the target person's face image fusion Gaussian smoothing filter after displaying the acquired camera image. 由于明星脸肤色与现实中目标脸的肤色存在差异,为了更有效的避免融合后产生的缝隙,引入了颜色迀移方法。 Because of differences in skin color and the reality star in the face of the target color, in order to more effectively avoid the gap created after the integration, the introduction of color Gan shift method. 以上的局部仿射方法和颜色迀移方法耗时都比较小,所以将两种方法综合运用,放在视频中可以产生比现有技术更高效的方法。 Local affine above method is time consuming and color shift Gan are relatively small, so that the integrated use of two methods, may be generated on the video more efficient than prior art methods. 同时为了进一步提高人脸融合后的效果,在此利用拉普拉斯图像融合方法,将颜色迀移后的图像融合到目标人脸中,效果要远好于现有技术方法。 Meanwhile, in order to further enhance the effect of the integration of the human face, in this Laplace image fusion method, the image colors Gan shift fused to the human face, the effect is much better than the prior art methods. 其具体步骤如下: The specific steps are as follows:

[0131] (1)在laP空间上将变换前的目标人脸的颜色迀移到变换后的目标人脸上,其中1表示非彩色的亮度通道,a表示彩色黄蓝通道,0表示红绿通道。 [0131] (1) before the conversion in the face on laP target color space Gan target after the transforming human face, wherein 1 represents achromatic luminance channel, a represents the yellow-blue color channel, 0 represents a red-green aisle. Ia0空间是在LMS 空间的基础上建立起来的,由于LMS空间三个通道间有较大的相关性,给图像处理过程带来一定的困难。 Ia0 space is established on the basis of the LMS space, due to the greater correlation between the LMS space three channels, to the image processing to bring some difficulties. 针对这种情况,提出了基于Ia0颜色空间,其中,1表示非彩色的亮度通道,a表示彩色的黄蓝通道,0表示红绿通道,与其他颜色体系不同,Ia0空间更适合人类视觉感知系统。 For this situation, proposed based Ia0 color space, wherein, 1 represents an achromatic luminance channel, a represents the yellow-blue color channel, channel 0 represents red and green, a color system different from other, more appropriate spatial Ia0 the human visual perception system . 对自然场景,IOp三通道近似正交,通道间的相关性会降到最小。 Natural scenes, IOp approximately orthogonal three-channel correlation between the channels will be minimized.

[0132] (2)采用拉普拉斯高斯金字塔的方法对迀移后的目标人脸的不同尺度、不同分解层的图像进行融合处理。 [0132] (2) using the Gaussian Laplacian pyramid approach to the different scales Gan branch target face, the image decomposition layer is different fusion. 拉普拉斯金字塔是一种多尺度、多分辨率的方法。 Laplacian pyramid is a multi-scale, multi-resolution approach. 基于金字塔分解的图像融合算法的融合过程是在不同尺度、不同空间分辨率和不同分解层上分别进行的, 与简单图像融合算法相比能够获得更好的融合效果,同时能够在更广泛的场合使用。 Fusion image fusion algorithm based pyramid decomposition is carried out separately on different scales, different resolutions and different spatial decomposition level, as compared to a simple image fusion algorithm to obtain better integration effect, and is capable of broader applications use. 其包括以下步骤: Comprising the steps of:

[0133] A、对于颜色迀移后的目标人脸采用高斯金字塔获取多幅不同空间层上、多尺度的下采样图像,构建出图像金字塔。 [0133] A, after the color of the target person face Gaussian pyramid Gan shift plurality obtaining different spatial layers, the multi-scale image of the sample to construct the image pyramid. 构造的高斯金字塔具体步骤如下: Gaussian pyramid configuration specific steps are as follows:

[0134] a、输入颜色迀移后的目标人脸图像G。 [0134] a, the target input color human facial image shift Gan G. ,以G。 To G. 作为高斯金字塔的第0层; As a Gaussian pyramid of layer 0;

[0135] b、对原始输入图像G。 [0135] b, of the original input image G. 进行高斯低通滤波和隔行隔列的下采样,得到高斯金字塔的第一层图像G1; Gaussian low pass filtering and downsampling the interlaced every other column to obtain a first layer of the Gaussian pyramid image G1;

[0136] c、对第一层图像G1进行高斯低通滤波和隔行隔列的下采样,得到高斯金字塔的第二层图像G2; [0136] c, the first layer image G1 downsample Gaussian low-pass filtered interlaced and every other column to obtain a second layer of the Gaussian pyramid image G2;

[0137] d、对第1-1层图像G1i进行高斯低通滤波和隔行隔列的下采样,得到高斯金字塔的第1(1彡1彡N)层图像G1, [0137] d, the first layer image G1i 1-1 downsample Gaussian low-pass filtered interlaced and every other column to obtain a first Gaussian pyramid (1 San San 1 N) layer image G1,

[0138] [0138]

Figure CN105069746AD00141

[0139] 其中N为高斯金字塔的最大层数,RjPC 别为高斯金字塔的第1层的行数和列 [0139] where N is the maximum number of layers of the Gaussian pyramid, RjPC number of rows is not a Gaussian pyramid of the first layer and column

Figure CN105069746AD00142

[0140] e、重复以上过程,构成最终的高斯金字塔。 [0140] e, the process is repeated to form the final Gaussian pyramid.

[0141] B、对不同的分解层的不同频带上的金字塔图像采用拉普拉斯金字塔从金字塔顶层图像中向上采样来重建出上层图像,构造的拉普拉斯金字塔具体步骤如下: [0141] B, pyramid images of different frequency bands on different layers of decomposition upsampling using Laplacian pyramid images from the pyramid top upper layer to reconstruct the image, the Laplacian pyramid configuration of the specific steps are as follows:

[0142] a、对得到的高斯金字塔的顶层图像Gn使用内插法得到放大图像G:,其中N为高斯金字塔的最大层数; [0142] a, G to obtain an enlarged image of the image interpolation top Gn obtained using Gaussian pyramid :, where N is the maximum number of layers of the Gaussian pyramid;

[0143]b、对第NI层图像Gni使用内插法得到放大图像G; [0143] b, G to obtain an enlarged image of the first interpolation image NI layer Gni use;

[0144] c、将高斯金字塔的第1层图像G1使用内插法得到放大图像GZn,其计算公式如下: [0144] c, the inner layer of the first interpolation image using a Gaussian pyramid G1 enlarged image obtained GZn, is calculated as follows:

Figure CN105069746AD00143

普拉斯金字塔分解的第1层图像; Plath pyramid decomposition of the image layer;

[0149] f、重复高斯金字塔层各层的计算,得到拉普拉斯金字塔LP1^LP1,…,LPf^IiV [0149] f, double counting layers Gaussian pyramid layer, to obtain a Laplacian pyramid LP1 ^ LP1, ..., LPf ^ IiV

[0150] C、对重建后的图像进行合并、融合处理。 [0150] C, the reconstructed image merging, fusion.

[0151] (3)对融合处理后的目标人脸图像寻找边缘区域,采用高斯滤波器进行平滑处理。 [0151] (3) find the edge of the target area of ​​the human face image fusion, Gaussian smoothing filter. 通过高斯边缘滤波方法将融合的人脸放到摄像头图像帧时边缘平滑过渡。 When the edge smoothing transitions Gauss edge filtering method by fusion of human face image frame into a camera.

[0152] 第五步,检查摄像头是否关闭,若关闭,则结束人脸替换;若未关闭,则继续进行视频采集并获取当前帧图像,继续进行人脸检测、人脸变换和人脸融合步骤。 [0152] The fifth step checks whether the camera off, when closed, the end face replaced; if not closed, the video capture and continue to get the current frame image, proceed face detection, face and face fusion step transform .

[0153] 如图2所示,基于局部仿射和颜色迀移技术的视频实时人脸替换系统,包括: [0153] 2, the real-time video face alternative system and local affine color shifting Based Gan, comprising:

[0154] 视频采集模块,用于从摄像头下获取的视频中采集每一帧的人脸图像; [0154] The video capture module configured to capture a facial image of each frame is acquired from the video camera in the lower;

[0155] 分类器构造模块,用于对获取到的视频中的图像进行人脸检测; [0155] classifier module is configured, for the acquired video image for face detection;

[0156] 形状模型训练模块,用于为每个人脸特征点构建局部特征,建立出每个特征点的位置约束; [0156] shape model training module configured for each facial feature local feature point construct, for each constraint to establish the position of the feature point;

[0157] 主成分分析模块,用于通过形状模型构造模块来对形状特征做特征提取处理; [0157] Principal component analysis module for feature extraction process to do the shape wherein the shape model configuration module;

[0158] 人脸特征点搜索模块,用于搜索人脸特征点并计算出特征点所在的位置; [0158] facial features search module for searching the facial feature points and calculate the position of the characteristic point is located;

[0159] 人脸仿射变换模块,用于将人脸库中的样本映射到目标人脸的相应位置上; [0159] Face affine transformation module, for mapping the sample face database to a corresponding position on the target face;

[0160] 基于拉普拉斯高斯金字塔图像融合模块,用于将颜色迀移后的目标人脸进行相应的融合处理; [0160] Fusion module Laplacian of Gaussian pyramid image based on, for the shifting target color Gan face corresponding fusion;

[0161] 所述的视频采集输入模块与分类器构造模型相连,所述的分类器构造模块的输出端分别与形状模型训练模块和主成分分析模块相连,形状模型训练模块和主成分分析模块分别与人脸特征点搜索模块的输入端相连,人脸特征点搜索模块的输出端与人脸仿射变换模块相连,人脸仿射变换模块的输出端与基于拉普拉斯高斯金字塔图像融合模块相连。 [0161] The video capture module and the input is connected to a classifier constructor model, the output of the classification module is configured respectively connected with the shape model training module and principal component analysis module, the shape of the model training module, respectively, and principal component analysis module connected to the face feature point search module inputs, facial features search module and the output terminal of the affine transformation module connected to the face, the face affine transformation module output of a Laplacian of Gaussian pyramid image fusion based module connected.

[0162] 以上显示和描述了本发明的基本原理、主要特征和本发明的优点。 [0162] The above and described the principles of the invention, the main features and advantages of the present invention. 本行业的技术人员应该了解,本发明不受上述实施例的限制,上述实施例和说明书中描述的只是本发明的原理,在不脱离本发明精神和范围的前提下本发明还会有各种变化和改进,这些变化和改进都落入要求保护的本发明的范围内。 The industry the art will appreciate, the present invention is not limited to the above embodiment, the above-described principles and embodiments described in the specification are only embodiments of the present invention, the present invention without departing from the spirit and scope of the invention will have a variety of changes and modifications, changes and modifications within the scope of the invention claimed fall. 本发明要求的保护范围由所附的权利要求书及其等同物界定。 Scope of the invention claimed by the appended claims and equivalents thereof defined.

Claims (9)

  1. 1. 一种基于局部仿射和颜色迀移技术的视频实时人脸替换方法,其特征在于,包括以下步骤: 11) 视频采集,通过摄像头获取目标人脸视频,并截取当前帧图像; 12) 进行人脸及其特征点检测,对获取到的目标人脸图像,使用强分类器来检测人脸, 在已经构建的形状模型的基础上对人脸库中的样本进行训练,将目标人脸特征点和人脸库中的样本特征点进行匹配,做出相应的变换,并对检测到的目标人脸进行特征点搜索、位置标记; 13) 进行人脸变换,利用仿射变换参数,将人脸库中的样本脸仿射变换到视频中的目标人脸上,进行人脸的替换; 14) 进行人脸融合处理,将变换前的目标人脸的颜色迀移到变换后的目标人脸上,采用拉普拉斯高斯金字塔进行融合处理,并对融合处理后的目标人脸图像的边缘采用高斯滤波器进行平滑处理后显示到摄像头获取 An alternative method for real-time video face local affine shift technique and the color Gan-based, characterized by comprising the steps of: 11) video capture, video acquisition target face through the camera, and the interception of the current frame image; 12) its human face feature point detection on the acquired face image of the target using the strong classifier to detect a human face, the face of library samples for training the shape model has been constructed on the target face feature point and the sample feature point in the face database match, and make the appropriate transform, and the detected target face feature point search, a position marker; 13) for face transformation using the affine transformation parameters, face database affine transformation to the sample faces the target person in the video face, a human face replacement; 14) fusion human face, the front face of the target color conversion Gan converted to a target person face, using a Laplacian of Gaussian pyramid fusion process, and the edge of the target fusion human face image display using the Gaussian filter to the camera acquires smoothed 的图像中; 15) 检查摄像头是否关闭,若关闭,则结束人脸替换;若未关闭,则继续进行视频采集并获取当前帧图像,继续进行人脸检测、人脸变换和人脸融合步骤。 Image; 15) to check whether the camera off, when closed, the end face replaced; if not closed, continued video capture and acquires the current frame image, continue to face detection, face transformation and face fusion step.
  2. 2. 根据权利要求1所述的基于局部仿射和颜色迀移技术的视频实时人脸替换方法,其特征在于,所述的进行人脸及其特征点检测包括以下步骤: 21) 读取人脸库中的样本数据,构造出若干个分类器,并将若干个分类器级联成一个强分类器; 22) 读取当前视频帧图像,使用强分类器对当前视频帧图像进行人脸检测; 23) 构建形状模型并进行训练; 24) 构造人脸特征点,得到仿射变换参数。 The real-time video-based human face alternative local affine Gan and color shifting techniques method according to claim 1, wherein said human face feature point detection and comprising the steps of: 21) in human-readable face sample data library, constructed plurality of classifiers and classifiers to cascade a plurality of strong classifiers; 22) reads the current video frame image, the strong classifier using the current video frame image for face detection ; 23) constructed shape model and training; 24) configured facial feature points, affine transformation parameters obtained.
  3. 3. 根据权利要求1所述的基于局部仿射和颜色迀移技术的视频实时人脸替换方法,其特征在于,所述的进行人脸变换包括以下步骤: 31) 对目标人脸和人脸库中样本的特征点计算出每个特征点的坐标,并将相邻的任意三个点进行特征点三角化; 32) 利用仿射变换参数,将人脸库中三角化后的特征点分别映射到目标人脸上,变换为目标人脸的形状,其构造的仿射变换具体步骤如下: 321) 对于特征点三角化后的目标人脸和人脸库中的样本,找出人脸库中每个三角形和对应的目标人脸的三角形的位置; 322) 保持人脸库中每个特征点的位置关系不变,将人脸库中的每个特征点三角形映射到目标人脸特征点三角形所在的位置。 The local affine based on real-time video and color Gan face of the replacement method of the Technique to claim 1, wherein said transforming human face comprising the steps of: 31) target face and face feature-point coordinates of the sample is calculated for each feature point, and feature points of any three adjacent triangulation points; 32) using the affine transformation parameters, the feature points of the face database are triangulated is mapped to the target person's face, converting the shape of the target face, specifically an affine transformation step is configured as follows: 321) face and the target sample face database for the feature points of the triangle, face database to identify triangle position of each triangle and the corresponding target face; holding each feature point in the face database 322) the same positional relationship, each of the feature points of the triangular face database mapped to the target facial features the position of the triangle is located.
  4. 4. 根据权利要求1所述的基于局部仿射和颜色迀移技术的视频实时人脸替换方法,其特征在于,所述的进行人脸融合处理包括以下步骤: 41) 在Ια β空间上将变换前的目标人脸的颜色迀移到变换后的目标人脸上,其中1表示非彩色的亮度通道,α表示彩色黄蓝通道,β表示红绿通道; 42) 采用拉普拉斯高斯金字塔的方法对迀移后的目标人脸的不同尺度、不同分解层的图像进行融合处理; 43) 对融合处理后的目标人脸图像寻找边缘区域,采用高斯滤波器进行平滑处理。 The local affine based on real-time video and color Gan face of the replacement method of the Technique to claim 1, wherein said human face fusion process comprising the steps of: Ια β space on 41) before conversion target color Gan target face after the transforming human face, wherein 1 represents achromatic luminance channel, [alpha] represents a yellow-blue color channels, red and green channel represents beta]; 42) using a Gaussian Laplacian pyramid method for different scales of the Gan branch target face, the image decomposition layer is different fusion; 43) of the fusion target human face image to find the edge region, Gaussian smoothing filter.
  5. 5. 根据权利要求2所述的基于局部仿射和颜色迀移技术的视频实时人脸替换方法,其特征在于,所述的构造出若干个分类器包括以下步骤: 51) 均匀分布人脸库中的每个样本,通过训练得到初始分类器H。 The local affine based on real-time video and color Gan face the alternative method of shifting technique as claimed in claim 2, wherein said construct comprises a plurality of classification steps of: 51) uniformly distributed face database each sample obtained by the initial classifier training H. ; 52) 分类判断,对于分类正确的,降低其分布概率;分类错误的,提高其分布概率,得到一个新的训练集S1; 53) 获得分类器,使用训练集51进行训练,得到分类器H1; 54) 迭代进行分类判断和获得分类器步骤共T次,得到仇,H2,…,Ητ}共T个分类器。 ; 52) classification determination, for the classification correct, reduce distribution probability; misclassification, improve the probability distribution to obtain a new training set S1; 53) obtained classifier training set 51 is trained classifier H1 ; 54) is determined iteratively and the classification step were obtained classifier T times to obtain Qiu, H2, ..., Ητ} total T classifiers.
  6. 6. 根据权利要求2所述的基于局部仿射和颜色迀移技术的视频实时人脸替换方法,其特征在于,所述的构建形状模型并进行训练包括以下步骤: 61) 搜集400个人脸训练样本,并标记出样本中的每个脸部特征点; 62) 将训练集中特征点的坐标串成特征向量; 63) 对形状特征进行归一化和对齐处理; 64) 对对齐后的形状特征做主成分分析处理,主成分分析的构造具体步骤如下: 641) 输入X,X = [X1X2…xj元向量变量,计算X的样本矩阵,其计算公式如下: The local affine based on real-time video and color Gan face alternative method of shifting technique according to claim 2, wherein said shape model constructed and trained comprising the steps of: 61) to collect training facial 400 samples and mark each facial feature points in the sample; 62) the coordinates of the training set of feature points strung eigenvectors; 63) wherein the shape and alignment normalizing process; 64) of the aligned shape feature shots component analysis, principal component analysis step is configured specifically as follows: 641) input X, X = [X1X2 ... xj element vector variable, the sample matrix X is calculated, which is calculated as follows:
    Figure CN105069746AC00031
    ,其中C J_ = 1,2,…,n是变量xi,i = 1,2,…,m的离散采样; 642) 计算样本矩阵X的第i行的平均值μ i,其计算公式如下: Wherein C J_ = 1,2, ..., n is a variable xi, i = 1,2, ..., m discrete samples; mean [mu] i of the i-th row 642) calculates a sample matrix X, which is calculated as follows:
    Figure CN105069746AC00032
    643) 计算样本矩阵X的第i行的中心矩:,其计算公式如下: 643) central moment of i-th row of the matrix X samples is calculated: which is calculated as follows:
    Figure CN105069746AC00033
    644) 计算样本矩阵X的中心矩F,其计算公式如下: 644) calculate central moments sample matrix X F, which is calculated as follows:
    Figure CN105069746AC00034
    645) 计算中心距的协方差矩阵Ω,其计算公式如下: 645) calculating a covariance matrix from the center Ω, which is calculated as follows:
    Figure CN105069746AC00035
    其中φ = [(J)1(J)2"* φη]是以!!!的正交向量矩阵, Λ =CliagU1, λ2,···,入丄λ 2多…彡λ "是对角特征值矩阵; 646) 计算正交转换矩阵Ρ,其计算公式如下: P = Φτ; 647) 将正交转换矩阵关联到X ,得到主成分分析65) 为每个样本的脸部特征点构建局部特征。 Wherein φ = [(J) 1 (J) 2 "* φη] is !!! orthogonal vector matrix, Λ = CliagU1, λ2, ···, λ 2 into Shang much ... San λ" is a diagonal feature value matrix; 646) calculating an orthogonal transform matrix Ρ, is calculated as follows: P = Φτ; 647) associated with the orthogonal transform matrix to X, to give principal component analysis build local feature 65) of facial feature points of each sample .
  7. 7. 根据权利要求2所述的基于局部仿射和颜色迀移技术的视频实时人脸替换方法,其特征在于,所述的构造人脸特征点包括以下步骤: 71)特征点位置计算,计算目标人脸特征点的位置,并做尺度和旋转变化; 72) 将目标人脸的特征点和人脸库中的样本的每个局部特征点进行匹配,计算出每个局部特征点对应到目标人脸上的新的位置; 73) 迭代进行上述步骤,得到仿射变换参数。 The local affine based on real-time video and color Gan face alternative method of shifting technique according to claim 2, wherein said configuration facial features comprising the steps of: 71) calculating the position of the feature point is calculated the target position of the face feature points, and make changes in rotation and scale; 72) each of the local feature points in the feature point of the target sample face and the face database by matching the calculated local feature points corresponding to each target the new position of the person's face; 73) above step iteration, obtained affine transformation parameters.
  8. 8.根据权利要求4所述的基于局部仿射和颜色迀移技术的视频实时人脸替换方法,其特征在于,所述的利用拉普拉斯高斯金字塔的方法对迀移后的目标人脸的不同尺度、不同分解层的图像进行融合处理包括以下步骤: 81) 对于颜色迀移后的目标人脸采用高斯金字塔获取多幅不同空间层上、多尺度的下采样图像,构建出图像金字塔,构造的高斯金字塔具体步骤如下: 811) 输入颜色迀移后的目标人脸图像G。 The local affine based on real-time video and color Gan face the Technique alternative method according to claim 4, characterized in that, using the Gaussian Laplacian pyramid method of the target shift human face after Gan different scales, different image decomposition layer is fusion process comprising the steps of: 81) after the target human face color shift Gan Gaussian pyramid obtaining multiple different spatial layers, the multi-scale downsampling image, an image pyramid is constructed, Gaussian pyramid configuration of the specific steps are as follows: 811) after the target person's face image input color shift Gan G. ,以G。 To G. 作为高斯金字塔的第O层; 812) 对原始输入图像G。 O layer as the Gaussian pyramid; 812) of the original input image G. 进行高斯低通滤波和隔行隔列的下采样,得到高斯金字塔的第一层图像G1; 813) 对第一层图像G1进行高斯低通滤波和隔行隔列的下采样,得到高斯金字塔的第二层图像G2; 814) 对第1-1层图像G1 i进行高斯低通滤波和隔行隔列的下采样,得到高斯金字塔的第1(1彡1彡N)层图像G1, Gaussian low pass filtering and downsampling the interlaced every other column to obtain a first layer of the Gaussian pyramid image G1; 813) on the first layer image G1 downsample Gaussian low-pass filtered interlaced and every other column to obtain a second Gaussian pyramid layer image G2; 814) on the first layer image G1 i 1-1 downsample Gaussian low-pass filtered interlaced and every other column to obtain a first Gaussian pyramid (1 San San 1 N) layer image G1,
    Figure CN105069746AC00041
    其中N为高斯金字塔的最大层数,RjP C 别为高斯金字塔的第1层的行数和列数, i/7(/;u?)是一个二维可分离的5X5的窗口函数, Where N is the maximum number of layers of the Gaussian pyramid, RjP C respectively of the first layer to the number of rows and columns in the Gaussian pyramid, i / 7 (/; u?) Is a separable two-dimensional window function of 5X5,
    Figure CN105069746AC00042
    815) 重复以上过程,构成最终的高斯金字塔; 82) 对不同的分解层的不同频带上的金字塔图像采用拉普拉斯金字塔从金字塔顶层图像中向上采样来重建出上层图像,构造的拉普拉斯金字塔具体步骤如下: 821) 对得到的高斯金字塔的顶层图像Gn使用内插法得到放大图像G二其中N为高斯金字塔的最大层数; 822) 对第NI层图像Gn i使用内插法得到放大图像S 823) 将高斯金字塔的第1层图像G1使用内插法得到放大图像馮%其计算公式如下: 815) repeat the process, to make the final Gaussian pyramid; 82) of the pyramid images on different frequency bands using different decomposition level Laplacian pyramid images sampled from a top of the pyramid to reconstruct the upper layer upwardly image configuration Plata Gaussian pyramid following steps: 821) to obtain an enlarged image G in which two top interpolation image obtained using a Gaussian pyramid Gn N is the maximum number of layers of the Gaussian pyramid; 822) to give the first layer image Gn NI I use interpolation enlarged image S 823) the resulting enlarged image is calculated as follows% von interpolation first layer image using a Gaussian pyramid G1:
    Figure CN105069746AC00043
    其中N为高斯金字塔的最大层数,RjP C 别为高斯金字塔的第1层的行数和列数, Where N is the maximum number of layers of the Gaussian pyramid, RjP C respectively of the first layer to the number of rows and columns of the Gaussian pyramid,
    Figure CN105069746AC00051
    Figure CN105069746AC00052
    ,其中N为拉普莱斯金字塔的最大层数,1^是拉普拉斯金字塔分解的第1层图像; 826)重复高斯金字塔层各层的计算,得到拉普拉斯金字塔LP。 , Where N is the maximum number of layers Rice Laplacian pyramid, ^ 1 is the first layer image Laplacian pyramid decomposition; 826) calculates the Gaussian pyramid level layers is repeated to obtain a Laplacian pyramid LP. ,LP1,…,LP1,…,LP n; 83)对重建后的图像进行合并、融合处理。 , LP1, ..., LP1, ..., LP n; 83) of the reconstructed image merging, fusion.
  9. 9.基于局部仿射和颜色迀移技术的视频实时人脸替换系统,其特征在于:包括: 视频采集模块,用于从摄像头下获取的视频中采集每一帧的人脸图像; 分类器构造模块,用于对获取到的视频中的图像进行人脸检测; 形状模型训练模块,用于为每个人脸特征点构建局部特征,建立出每个特征点的位置约束; 主成分分析模块,用于通过形状模型构造模块来对形状特征做特征提取处理; 人脸特征点搜索模块,用于搜索人脸特征点并计算出特征点所在的位置; 人脸仿射变换模块,用于将人脸库中的样本映射到目标人脸的相应位置上; 基于拉普拉斯高斯金字塔图像融合模块,用于将颜色迀移后的目标人脸进行相应的融合处理; 所述的视频采集输入模块与分类器构造模型相连,所述的分类器构造模块的输出端分别与形状模型训练模块和主成分分析模块相连 9. affine and the local real-time video color Gan face alternative system based on shifting technology, characterized by: comprising: a video capture module configured to capture a facial image of each frame of the video acquisition from the camera; classifier constructor module, for the acquired video image for face detection; shape model training module configured for each facial feature local feature points constructed to establish the position of each feature point constraints; principal component analysis module, with to do so by the shape of the shape model configuration module wherein the feature extraction processing; facial features search module for searching the facial feature points and calculate the positions of feature points located; face affine transformation module configured to face sample library mapped to a corresponding position on the target face; Gaussian Laplacian pyramid image fusion module for the shifting target color Gan face corresponding fusion; the input video capture module classifier constructor coupled to the model, the output of the classification module are configured to a shape model training module connected to the module and principal component analysis ,形状模型训练模块和主成分分析模块分别与人脸特征点搜索模块的输入端相连,人脸特征点搜索模块的输出端与人脸仿射变换模块相连,人脸仿射变换模块的输出端与基于拉普拉斯高斯金字塔图像融合模块相连。 , The shape of the model training module, respectively, and principal component analysis module and input face feature point search module is connected to facial features search module output end face of the affine transformation module is connected, the output end face affine transformation module fusion module is connected with the Laplacian of Gaussian pyramid image.
CN 201510520746 2015-08-23 2015-08-23 Based on local affine real-time video and color transfer technology to replace human face method and system CN105069746B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN 201510520746 CN105069746B (en) 2015-08-23 2015-08-23 Based on local affine real-time video and color transfer technology to replace human face method and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN 201510520746 CN105069746B (en) 2015-08-23 2015-08-23 Based on local affine real-time video and color transfer technology to replace human face method and system

Publications (2)

Publication Number Publication Date
CN105069746A true true CN105069746A (en) 2015-11-18
CN105069746B CN105069746B (en) 2018-02-16

Family

ID=54499104

Family Applications (1)

Application Number Title Priority Date Filing Date
CN 201510520746 CN105069746B (en) 2015-08-23 2015-08-23 Based on local affine real-time video and color transfer technology to replace human face method and system

Country Status (1)

Country Link
CN (1) CN105069746B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107230181A (en) * 2017-06-05 2017-10-03 厦门美柚信息科技有限公司 Face image fusion realization method and device

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1687957A (en) * 2005-06-02 2005-10-26 上海交通大学 Man face characteristic point positioning method of combining local searching and movable appearance model
CN101944238A (en) * 2010-09-27 2011-01-12 浙江大学 Data driving face expression synthesis method based on Laplace transformation
US20130129141A1 (en) * 2010-08-20 2013-05-23 Jue Wang Methods and Apparatus for Facial Feature Replacement

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1687957A (en) * 2005-06-02 2005-10-26 上海交通大学 Man face characteristic point positioning method of combining local searching and movable appearance model
US20130129141A1 (en) * 2010-08-20 2013-05-23 Jue Wang Methods and Apparatus for Facial Feature Replacement
CN101944238A (en) * 2010-09-27 2011-01-12 浙江大学 Data driving face expression synthesis method based on Laplace transformation

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
亢洁等: "基于拉普拉斯金字塔降维的人脸识别算法", 《陕西科技大学学报》 *
宋明黎等: "基于拉普拉斯微分的脸部细节迁移", 《计算机辅助设计与图形学学报》 *
李传学: "图像/视频中自动人脸替换研究", 《中国优秀硕士学位论文全文数据库信息科技辑》 *
林源等: "基于真实感三维头重建的人脸替换", 《清华大学学报(自然科学版)》 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107230181A (en) * 2017-06-05 2017-10-03 厦门美柚信息科技有限公司 Face image fusion realization method and device
CN107230181B (en) * 2017-06-05 2018-06-29 厦门美柚信息科技有限公司 Implementation face and image fusion device

Also Published As

Publication number Publication date Type
CN105069746B (en) 2018-02-16 grant

Similar Documents

Publication Publication Date Title
Eigen et al. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture
Dosovitskiy et al. Flownet: Learning optical flow with convolutional networks
Zhang Advances in image and video segmentation
Ahmad et al. Human action recognition using shape and CLG-motion flow from multi-view image sequences
Turk Gesture recognition
Fischer et al. Flownet: Learning optical flow with convolutional networks
Caelles et al. One-shot video object segmentation
Guo et al. Fast background subtraction based on a multilayer codebook model for moving object detection
Shi et al. Automatic acquisition of high-fidelity facial performances using monocular videos
El Zehiry et al. Graph cut optimization for the Mumford-Shah model
CN103856727A (en) Multichannel real-time video splicing processing system
Varol et al. Learning from synthetic humans
Kemelmacher-Shlizerman et al. Being john malkovich
CN102214298A (en) Method for detecting and identifying airport target by using remote sensing image based on selective visual attention mechanism
Zhu et al. Complex non-rigid motion 3d reconstruction by union of subspaces
CN1924894A (en) Multiple attitude human face detection and track system and method
Maire et al. Object detection and segmentation from joint embedding of parts and pixels
Khan et al. Automatic shadow detection and removal from a single image
CN102332095A (en) Face motion tracking method, face motion tracking system and method for enhancing reality
Johnson et al. Combining discriminative appearance and segmentation cues for articulated human pose estimation
CN103530619A (en) Gesture recognition method of small quantity of training samples based on RGB-D (red, green, blue and depth) data structure
CN103116902A (en) Three-dimensional virtual human head image generation method, and method and device of human head image motion tracking
WO2012126135A1 (en) Method of augmented makeover with 3d face modeling and landmark alignment
CN101763504A (en) Human head identification method under complex scene
Saito et al. Real-time facial segmentation and performance capture from rgb input

Legal Events

Date Code Title Description
C06 Publication
C10 Entry into substantive examination
GR01