CN112101072A - Face matching method, device, equipment and medium - Google Patents
Face matching method, device, equipment and medium Download PDFInfo
- Publication number
- CN112101072A CN112101072A CN201910526561.8A CN201910526561A CN112101072A CN 112101072 A CN112101072 A CN 112101072A CN 201910526561 A CN201910526561 A CN 201910526561A CN 112101072 A CN112101072 A CN 112101072A
- Authority
- CN
- China
- Prior art keywords
- video
- similarity
- face
- user
- hairstyle
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 46
- 238000012216 screening Methods 0.000 claims abstract description 4
- 230000037308 hair color Effects 0.000 claims description 21
- 238000004364 calculation method Methods 0.000 claims description 17
- 238000004590 computer program Methods 0.000 claims description 12
- 238000012549 training Methods 0.000 claims description 7
- 230000001815 facial effect Effects 0.000 claims description 6
- 238000000605 extraction Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 description 7
- 238000004891 communication Methods 0.000 description 6
- 239000013598 vector Substances 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 239000000284 extract Substances 0.000 description 4
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000003741 hair volume Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/172—Classification, e.g. identification
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Oral & Maxillofacial Surgery (AREA)
- General Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
- Processing Or Creating Images (AREA)
Abstract
Description
技术领域technical field
本发明涉及人脸换像技术领域,尤其涉及一种人脸匹配方法、装置、设备及介质。The present invention relates to the technical field of face swapping, and in particular, to a face matching method, device, equipment and medium.
背景技术Background technique
人脸转换是计算机视觉领域中一个比较热门的应用,人脸转换一般可以用于视频合成、提供隐私服务、肖像更换或者其他有创新性的应用。Face conversion is a popular application in the field of computer vision. Face conversion can generally be used for video synthesis, providing privacy services, portrait replacement, or other innovative applications.
目前,用户在视频中换脸后与用户人脸区别过大,无法结合用户的人脸特征向用户推荐与用户人脸更接近的人脸对应的视频,造成用户体验度过低。At present, the user's face is too different from the user's face after changing his face in the video, and it is impossible to recommend a video corresponding to a face that is closer to the user's face to the user based on the user's face characteristics, resulting in a poor user experience.
发明内容SUMMARY OF THE INVENTION
本发明实施例提供了一种人脸匹配方法、装置、设备及介质,能够在视频库中筛选出与用户相似的人脸对应的视频,使得用户在换脸后的视频人物与用户人脸更接近,提高用户的体验度。Embodiments of the present invention provide a face matching method, device, equipment and medium, which can filter out videos corresponding to faces similar to users in a video library, so that the video characters after the user changes faces are more closely related to the user's face. Proximity to improve user experience.
第一方面,本发明实施例提供了一种人脸匹配方法,方法包括:In a first aspect, an embodiment of the present invention provides a face matching method, including:
获取用户的人脸图像;Get the user's face image;
从人脸图像中提取用户的人脸特征;Extract the user's face features from the face image;
根据用户的人脸特征和预存视频库中每个视频对应的视频人脸特征,计算用户的人脸与每个视频中人脸的人脸相似度;Calculate the face similarity between the user's face and the face in each video according to the user's face features and the video face features corresponding to each video in the pre-stored video library;
利用人脸图像识别用户的性别特征以及用户的发型特征;Use face images to identify the user's gender characteristics and the user's hairstyle characteristics;
计算用户的发型特征与每个视频中人物的发型特征之间的发型相似度;Calculate the hairstyle similarity between the user's hairstyle features and the hairstyle features of the characters in each video;
根据人脸相似度和发型相似度,在视频库中筛选与用户对应的目标视频,其中,目标视频中人物的性别特征与用户的性别特征一致。According to the similarity of faces and hairstyles, the target videos corresponding to the user are screened in the video library, wherein the gender characteristics of the characters in the target videos are consistent with the gender characteristics of the user.
第二方面,本发明实施例提供了一种人脸匹配装置,装置包括:In a second aspect, an embodiment of the present invention provides a face matching device, which includes:
获取模块,用于获取用户的人脸图像;The acquisition module is used to acquire the face image of the user;
提取模块,用于从人脸图像中提取用户的人脸特征;The extraction module is used to extract the user's face features from the face image;
第一计算模块,用于根据用户的人脸特征和预存视频库中每个视频对应的视频人脸特征,计算用户的人脸与每个视频中人脸的人脸相似度;The first calculation module is used to calculate the face similarity between the user's face and the face in each video according to the user's face feature and the video face feature corresponding to each video in the pre-stored video library;
识别模块,用于利用人脸图像识别用户的性别特征以及用户的发型特征;The recognition module is used to recognize the gender characteristics of the user and the hairstyle characteristics of the user by using the face image;
第二计算模块,用于计算用户的发型特征与每个视频中人物的发型特征之间的发型相似度;The second computing module is used to calculate the hairstyle similarity between the hairstyle features of the user and the hairstyle features of the characters in each video;
选择模块,用于根据人脸相似度和发型相似度,在视频库中筛选与用户对应的目标视频,其中,目标视频中人物的性别特征与用户的性别特征一致。The selection module is used for screening the target video corresponding to the user in the video library according to the similarity of the face and the similarity of the hairstyle, wherein the gender characteristics of the characters in the target video are consistent with the gender characteristics of the user.
第三方面,本发明实施例提供了一种计算机设备,包括:至少一个处理器、至少一个存储器以及存储在存储器中的计算机程序指令,当计算机程序指令被处理器执行时实现如第一方面的方法。In a third aspect, an embodiment of the present invention provides a computer device, comprising: at least one processor, at least one memory, and computer program instructions stored in the memory, when the computer program instructions are executed by the processor, the first aspect is implemented method.
第四方面,本发明实施例提供了一种计算机可读存储介质,其上存储有计算机程序指令,其特征在于,当计算机程序指令被处理器执行时实现如第一方面的方法。In a fourth aspect, embodiments of the present invention provide a computer-readable storage medium on which computer program instructions are stored, characterized in that, when the computer program instructions are executed by a processor, the method of the first aspect is implemented.
本发明实施例提供的人脸匹配方法、装置、设备及介质,通过获取用户的人脸图像;从人脸图像中提取用户的人脸特征;根据用户的人脸特征和预存视频库中每个视频对应的视频人脸特征,计算用户的人脸与每个视频中人脸的人脸相似度;利用人脸图像识别用户的性别特征以及用户的发型特征;计算用户的发型特征与每个视频中人物的发型特征之间的发型相似度;根据人脸相似度和发型相似度,在视频库中筛选与用户对应的目标视频,其中,目标视频中人物的性别特征与用户的性别特征一致。本发明实施例能够在视频库中筛选出与用户相似的人脸对应的视频,使得用户在换脸后的视频人物与用户人脸更接近,提高用户的体验度。The face matching method, device, device and medium provided by the embodiments of the present invention obtain the user's face image; extract the user's face feature from the face image; The video face features corresponding to the video, calculate the face similarity between the user's face and the face in each video; use the face image to identify the user's gender feature and the user's hairstyle feature; calculate the user's hairstyle feature and each video. The hairstyle similarity between the hairstyle features of the characters in the video library; according to the face similarity and hairstyle similarity, the target video corresponding to the user is screened in the video library, wherein the gender features of the characters in the target video are consistent with the gender features of the user. The embodiment of the present invention can filter out videos corresponding to faces similar to the user in the video library, so that the video characters of the user after changing faces are closer to the user's face, and the user experience is improved.
附图说明Description of drawings
为了更清楚地说明本发明实施例的技术方案,下面将对本发明实施例中所需要使用的附图作简单地介绍,对于本领域普通技术人员来讲,在不付出创造性劳动的前提下,还可以根据这些附图获得其他的附图。In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the accompanying drawings required in the embodiments of the present invention will be briefly introduced below. For those of ordinary skill in the art, without creative work, the Additional drawings can be obtained from these drawings.
图1示出了根据本发明一些实施例提供的一种人脸匹配方法的流程图;1 shows a flowchart of a method for face matching provided according to some embodiments of the present invention;
图2示出了根据本发明一些实施例提供的一种人脸匹配装置的结构示意图;FIG. 2 shows a schematic structural diagram of a face matching apparatus provided according to some embodiments of the present invention;
图3示出了根据本发明一些实施例提供的一种计算设备的结构示意图。FIG. 3 shows a schematic structural diagram of a computing device according to some embodiments of the present invention.
具体实施方式Detailed ways
下面将详细描述本发明的各个方面的特征和示例性实施例,为了使本发明的目的、技术方案及优点更加清楚明白,以下结合附图及实施例,对本发明进行进一步详细描述。应理解,此处所描述的具体实施例仅被配置为解释本发明,并不被配置为限定本发明。对于本领域技术人员来说,本发明可以在不需要这些具体细节中的一些细节的情况下实施。下面对实施例的描述仅仅是为了通过示出本发明的示例来提供对本发明更好的理解。The features and exemplary embodiments of various aspects of the present invention will be described in detail below. In order to make the objects, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only configured to explain the present invention, and are not configured to limit the present invention. It will be apparent to those skilled in the art that the present invention may be practiced without some of these specific details. The following description of the embodiments is only intended to provide a better understanding of the present invention by illustrating examples of the invention.
需要说明的是,在本文中,诸如第一和第二等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来,而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且,术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含,从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素,而且还包括没有明确列出的其他要素,或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下,由语句“包括……”限定的要素,并不排除在包括所述要素的过程、方法、物品或者设备中还存在另外的相同要素。It should be noted that, in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any relationship between these entities or operations. any such actual relationship or sequence exists. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element defined by the phrase "comprises" does not preclude the presence of additional identical elements in a process, method, article, or device that includes the element.
参见图1所示,本发明实施例提供了一种人脸匹配方法,该方法包括:S101-S106。Referring to FIG. 1, an embodiment of the present invention provides a face matching method, and the method includes: S101-S106.
S101:获取用户的人脸图像。S101: Obtain a face image of a user.
在具体实施的时候,用户的人脸图像可以是用户使用终端拍摄的图片,也可以是用户在录制换脸视频之前上传的一张图片。During specific implementation, the face image of the user may be a picture taken by the user using the terminal, or may be a picture uploaded by the user before recording the face-changing video.
S102:从人脸图像中提取用户的人脸特征。S102: Extract the user's face features from the face image.
在具体实施的时候,用户的人脸特征包括用户人脸五官的形状,位置。例如,可以使用三维形变模型(3D morphable model,3DMM)从用户的人脸图像中提取出特征点坐标。提取得到的人脸特征可以是一个1×P维的向量,其中,P为大于1的整数。During specific implementation, the user's facial features include the shape and position of the user's facial features. For example, a three-dimensional morphable model (3D morphable model, 3DMM) can be used to extract feature point coordinates from a user's face image. The extracted face feature may be a 1×P-dimensional vector, where P is an integer greater than 1.
S103:根据用户的人脸特征和预存视频库中每个视频对应的视频人脸特征,计算用户的人脸与每个视频中人脸的人脸相似度。S103: Calculate the face similarity between the user's face and the face in each video according to the user's face feature and the video face feature corresponding to each video in the pre-stored video library.
在具体实施的时候,视频库中的每个视频对应的视频人脸特征是预先提取出来的,将所提取出来的视频人脸特征生成视频特征向量进行保存,例如视频库中的视频总数量为M个,则视频特征向量为M×P。计算人脸特征与每个视频中人脸的人脸相似度,例如,人脸特征为1×P维的向量,M个视频人脸特征为M×P维的向量,则计算1×P×(M×P)T得到1×M维的特征向量,该特征向量表示M个人脸相似度,M个人脸相似度是与M个视频一一对应的,也即表示用户人脸分别与M个视频人脸的人脸相似度。In the specific implementation, the video face features corresponding to each video in the video library are extracted in advance, and the extracted video face features are generated into video feature vectors for storage. For example, the total number of videos in the video library is M, the video feature vector is M×P. Calculate the similarity between the face feature and the face in each video. For example, if the face feature is a 1×P-dimensional vector, and the face features of M videos are an M×P-dimensional vector, then calculate 1×P× (M×P) T obtains a 1×M-dimensional feature vector, which represents the similarity of M faces, and the similarity of M faces corresponds to M videos one-to-one, which means that the user’s face is respectively associated with M Face similarity of video faces.
在一些实施例中,可以将M个视频人脸与用户人脸的人脸相似度大于第一预设值的视频推荐给用户进行换脸。In some embodiments, the videos whose face similarity between the M video faces and the user's face is greater than the first preset value may be recommended to the user for face swapping.
S104:利用人脸图像识别用户的性别特征以及用户的发型特征。S104: Identify the gender feature of the user and the hairstyle feature of the user by using the face image.
在具体实施的时候,根据人脸图像识别用户性别特征和用户的发型特征,例如,用户的发型样式、头发长度、头发卷度、头发颜色、刘海形状均属于用户的发型特征。During specific implementation, the user's gender feature and the user's hairstyle feature are identified according to the face image. For example, the user's hairstyle style, hair length, hair curl, hair color, and bangs shape all belong to the user's hairstyle feature.
S105:计算用户的发型特征与每个视频中人物的发型特征之间的发型相似度。S105: Calculate the hairstyle similarity between the hairstyle features of the user and the hairstyle features of the characters in each video.
在具体实施的时候,计算用户的发型特征与每个视频中人物的发型特征之间的发型相似度。During specific implementation, the hairstyle similarity between the hairstyle features of the user and the hairstyle features of the characters in each video is calculated.
在一些实施例中,发型特征包括N个类型的发型特征,视频库中的每个视频中人物的发型特征都包括N个类型的发型特征,针对每一个视频,通过下述步骤计算用户发型特征与第一视频中人物的发型特征之间的发型相似度,其中,第一视频是视频库中任意一个视频:遍历N个类型中的每个类型,将每个类型作为待处理类型;计算用户的待处理类型的发型特征与第一视频中人物的待处理类型的发型特征之间的发型相似度。In some embodiments, the hairstyle features include N types of hairstyle features, the hairstyle features of characters in each video in the video library include N types of hairstyle features, and for each video, the user's hairstyle features are calculated through the following steps The hairstyle similarity with the hairstyle features of the characters in the first video, where the first video is any video in the video library: traverse each of the N types, and use each type as the type to be processed; calculate the user The hairstyle similarity between the hairstyle features of the to-be-processed type and the hairstyle features of the to-be-processed type of characters in the first video.
例如,N个发型特征包括发型样式、头发长度、头发卷度、头发颜色、刘海形状,则用户的待处理类型的发型特征与第一视频中人物的待处理类型的发型特征之间的发型相似度计算方式如下:For example, if the N hairstyle features include hairstyle style, hair length, hair curl, hair color, and bangs shape, the hairstyles of the user's to-be-processed hairstyle feature and the to-be-processed hairstyle feature of the character in the first video are similar in hairstyles The degree is calculated as follows:
计算用户的发型样式与第一视频中人物的发型样式的相似度,作为第一视频的发型样式相似度。The similarity between the hairstyle style of the user and the hairstyle style of the character in the first video is calculated as the similarity degree of the hairstyle style of the first video.
计算用户的头发长度与第一视频中人物的头发长度的相似度,作为第一视频的头发长度相似度。The similarity between the user's hair length and the hair length of the character in the first video is calculated as the hair length similarity of the first video.
计算用户的头发卷度与第一视频中人物的头发卷度的相似度,作为第一视频的头发卷度相似度。The similarity between the hair curl of the user and the hair curl of the character in the first video is calculated as the similarity of the hair curl of the first video.
计算用户的头发颜色与第一视频中人物的头发颜色的相似度,作为第一视频的头发颜色相似度。The similarity between the user's hair color and the hair color of the character in the first video is calculated as the hair color similarity of the first video.
计算用户的刘海形状与第一视频库中人物的刘海形状的相似度,作为第一视频的头发颜色相似度。The similarity between the user's bangs shape and the bangs shape of the characters in the first video library is calculated as the hair color similarity of the first video.
S106:根据人脸相似度和发型相似度,在视频库中筛选与用户对应的目标视频,其中,目标视频中人物的性别特征与用户的性别特征一致。S106: Screen target videos corresponding to the user in the video library according to the similarity of faces and hairstyles, wherein the gender characteristics of the characters in the target videos are consistent with the gender characteristics of the user.
计算得到待处理类型的发型特征之间的发型相似度后,采用下述两种方式通过发型相似度和人脸相似度在视频库中筛选与用户对应的目标视频。After calculating the hairstyle similarity between the hairstyle features of the type to be processed, the following two methods are used to filter the target video corresponding to the user in the video library through the hairstyle similarity and the face similarity.
其一:One:
在视频库选择人脸相似度大于第一预设值的视频作为第二视频集合,遍历视频库中的每个视频,将每个视频作为第一视频,若第一视频的发型样式相似度、第一视频的头发长度的相似度、第一视频的头发卷度的相似度、第一视频的头发颜色的相似度和第一视频的刘海形状的相似度均大于第二预设值,则将第一视频添加到第一视频集合中,遍历完每个视频之后,取第二视频集合和第一视频集合的交集,在交集的视频中选取目标视频,并将目标视频推荐给用户。In the video library, select a video with a face similarity greater than the first preset value as the second video set, traverse each video in the video library, and use each video as the first video, if the hairstyle style similarity of the first video, The similarity of the hair length of the first video, the similarity of the hair curl of the first video, the similarity of the hair color of the first video, and the similarity of the bangs shape of the first video are all greater than the second preset value, then the The first video is added to the first video set, after traversing each video, the intersection of the second video set and the first video set is taken, the target video is selected from the intersected videos, and the target video is recommended to the user.
其二:Second:
针对视频库中的任意一个视频,使用相似度计算模型对第一视频的人脸相似度、第一视频的发型样式的相似度、第一视频的头发长度的相似度、第一视频的头发卷度的相似度、第一视频的头发颜色的相似度和第一视频的刘海形状的相似度进行加权求和,得到用户与第一视频中人物的综合相似度,遍历每个视频,得到用户与每个视频中人物的综合相似度;在视频库中筛选综合相似度大于第三预设值的目标视频,并将目标视频推荐给用户。For any video in the video library, use the similarity calculation model to calculate the similarity of the face of the first video, the similarity of the hairstyle style of the first video, the similarity of the hair length of the first video, and the hair volume of the first video. The degree of similarity, the similarity of the hair color of the first video, and the similarity of the bangs shape of the first video are weighted and summed, and the comprehensive similarity between the user and the characters in the first video is obtained, and each video is traversed to obtain the user and the first video. Comprehensive similarity of characters in each video; screen target videos whose comprehensive similarity is greater than the third preset value in the video library, and recommend the target video to the user.
在这里,所有目标视频中人物的性别特征与用户的性别特征一致。性别特征可以通过人脸图像进行识别,根据用户的人脸特征以及发型特征识别用户的性别,也可以通过实现采集用户资料该用户对应的账号标记性别特征。Here, the gender characteristics of people in all target videos are consistent with the gender characteristics of users. The gender feature can be identified through the face image, the user's gender can be identified according to the user's face feature and hairstyle feature, or the gender feature can be marked by collecting the user data and the account corresponding to the user.
在一些实施例中,本发明实施例提供的人脸匹配方法还包括训练相似度计算模型,具体包括:In some embodiments, the face matching method provided by the embodiments of the present invention further includes training a similarity calculation model, which specifically includes:
获取样本人脸图像,使用多个待训练模型分别计算样本人脸图像与每个视频对应人物的发型特征的相似度和人脸相似度。其中,待训练模型包括多个参数,每一个参数是与人脸相似度和发型相似度一一对应,也即,人脸相似度与发型相似度对应的权重。在这里,预先设定多组参数,也即预先设定了多个待训练模型。针对每一个待训练模型,选择人脸相似度大于第一预设值且发型特征的相似度大于第二预设值的视频;使用人脸换像模型将样本人脸图像换脸至所选择视频对应人物的人脸上,得到多个目标人脸图像;其中,人脸换像模型可以是生成式对抗网络(Generative Adversarial Networks,GAN)还可以是CycleGAN。将样本人脸图像与多个目标人脸图像输入至人脸识别模型中,计算样本人脸图像与多个目标人脸图像的相似度;计算每一个待训练模型对应的多个目标人脸图像的相似度的平均值;选择最高平均值对应的待训练模型作为相似度计算模型。Obtain a sample face image, and use multiple models to be trained to calculate the similarity and face similarity between the sample face image and the hairstyle features of the characters corresponding to each video. The model to be trained includes a plurality of parameters, and each parameter corresponds to the similarity of the face and the similarity of the hairstyle, that is, the weight corresponding to the similarity of the face and the hairstyle. Here, multiple sets of parameters are preset, that is, multiple models to be trained are preset. For each model to be trained, select a video with a face similarity greater than a first preset value and a hairstyle feature similarity greater than a second preset value; use the face swap model to swap the sample face image to the selected video Corresponding to the face of the character, a plurality of target face images are obtained; wherein, the face swapping model may be a generative adversarial network (Generative Adversarial Networks, GAN) or a CycleGAN. Input the sample face image and multiple target face images into the face recognition model, calculate the similarity between the sample face image and multiple target face images; calculate the multiple target face images corresponding to each model to be trained The average of the similarity; select the model to be trained corresponding to the highest average as the similarity calculation model.
本发明实施例提供的人脸匹配方法,通过获取用户的人脸图像;从人脸图像中提取用户的人脸特征;根据用户的人脸特征和预存视频库中每个视频对应的视频人脸特征,计算用户的人脸与每个视频中人脸的人脸相似度;利用人脸图像识别用户的性别特征以及用户的发型特征;计算用户的发型特征与每个视频中人物的发型特征之间的发型相似度;根据人脸相似度和发型相似度,在视频库中筛选与用户对应的目标视频,其中,目标视频中人物的性别特征与用户的性别特征一致。本发明实施例能够在视频库中筛选出与用户相似的人脸对应的视频,使得用户在换脸后的视频人物与用户人脸更接近,提高用户的体验度。The face matching method provided by the embodiment of the present invention obtains the face image of the user; extracts the face feature of the user from the face image; Features, calculate the face similarity between the user's face and the face in each video; use the face image to identify the user's gender characteristics and the user's hairstyle characteristics; calculate the user's hairstyle characteristics and the hairstyle characteristics of the characters in each video According to the similarity of face and hairstyle, the target video corresponding to the user is screened in the video library, wherein the gender characteristics of the characters in the target video are consistent with the gender characteristics of the user. The embodiment of the present invention can filter out videos corresponding to faces similar to the user in the video library, so that the video characters of the user after changing faces are closer to the user's face, and the user experience is improved.
参见图2所示,本发明实施例提供了一种人脸匹配装置,该装置包括:Referring to FIG. 2, an embodiment of the present invention provides a face matching device, which includes:
获取模块201,用于获取用户的人脸图像;an acquisition module 201, used for acquiring a face image of a user;
提取模块202,用于从人脸图像中提取用户的人脸特征;The extraction module 202 is used for extracting the facial features of the user from the facial image;
第一计算模块203,用于根据用户的人脸特征和预存视频库中每个视频对应的视频人脸特征,计算用户的人脸与每个视频中人脸的人脸相似度;The first calculation module 203 is used to calculate the face similarity between the user's face and the face in each video according to the user's face feature and the video face feature corresponding to each video in the pre-stored video library;
识别模块204,用于利用人脸图像识别用户的性别特征以及用户的发型特征;The identification module 204 is used for identifying the gender feature of the user and the hairstyle feature of the user by using the face image;
第二计算模块205,用于计算用户的发型特征与每个视频中人物的发型特征之间的发型相似度;The second calculation module 205 is used to calculate the hairstyle similarity between the hairstyle features of the user and the hairstyle features of the characters in each video;
选择模块206,用于根据人脸相似度和发型相似度,在视频库中筛选与用户对应的目标视频,其中,目标视频中人物的性别特征与用户的性别特征一致。The selection module 206 is configured to filter the target video corresponding to the user in the video library according to the similarity of the face and the similarity of the hairstyle, wherein the gender characteristics of the characters in the target video are consistent with the gender characteristics of the user.
在一些实施例中,用户的发型特征包括N个类型的发型特征,每个视频中人物的发型特征包括N个类型的发型特征,视频库中的视频总数量为M个,M和N分别为大于1的整数;In some embodiments, the hairstyle features of the user include N types of hairstyle features, the hairstyle features of characters in each video include N types of hairstyle features, the total number of videos in the video library is M, and M and N are respectively an integer greater than 1;
第二计算模块205,用于计算用户的发型特征与每个视频中人物的发型特征之间的发型相似度,包括:The second calculation module 205 is used to calculate the hairstyle similarity between the hairstyle features of the user and the hairstyle features of the characters in each video, including:
通过以下步骤计算用户的发型特征与第一视频中人物的发型特征之间的发型相似度:The hairstyle similarity between the hairstyle features of the user and the hairstyle features of the characters in the first video is calculated by the following steps:
遍历N个类型中的每个类型,将每个类型作为待处理类型;Traverse each of the N types and treat each type as a pending type;
计算用户的待处理类型的发型特征与第一视频中人物的待处理类型的发型特征之间的发型相似度;其中,第一视频是视频库中任意的一个视频。Calculate the hairstyle similarity between the hairstyle features of the type to be processed of the user and the hairstyle features of the type to be processed of the characters in the first video; wherein the first video is any video in the video library.
在一些实施例中,N个类型的发型特征包括:发型样式、头发长度、头发卷度、头发颜色、刘海形状;In some embodiments, the N types of hairstyle features include: hairstyle style, hair length, hair curl, hair color, bang shape;
第二计算模块205,具体用于计算用户的待处理类型的发型特征与第一视频中人物的待处理类型的发型特征之间的发型相似度,包括:The second calculation module 205 is specifically configured to calculate the hairstyle similarity between the hairstyle features of the type to be processed of the user and the hairstyle features of the type to be processed of the characters in the first video, including:
计算用户的发型样式与第一视频中人物的发型样式的相似度,作为第一视频的发型样式相似度;Calculate the similarity between the user's hairstyle style and the hairstyle style of the character in the first video as the similarity of the hairstyle style of the first video;
计算用户的头发长度与第一视频中人物的头发长度的相似度,作为第一视频的头发长度相似度;Calculate the similarity between the user's hair length and the hair length of the character in the first video as the hair length similarity of the first video;
计算用户的头发卷度与第一视频中人物的头发卷度的相似度,作为第一视频的头发卷度相似度;Calculate the similarity between the hair curl of the user and the hair curl of the character in the first video as the similarity of the hair curl of the first video;
计算用户的头发颜色与第一视频中人物的头发颜色的相似度,作为第一视频的头发颜色相似度;Calculate the similarity between the user's hair color and the hair color of the character in the first video as the hair color similarity of the first video;
计算用户的刘海形状与第一视频中人物的刘海形状的相似度,作为第一视频的头发颜色相似度。The similarity between the user's bangs shape and the bangs shape of the character in the first video is calculated as the hair color similarity of the first video.
选择模块206用于根据人脸相似度和发型相似度,在视频库中筛选与用户对应的目标视频,包括:The selection module 206 is used to filter the target video corresponding to the user in the video library according to the similarity of the face and the similarity of the hairstyle, including:
在视频库中选择人脸相似度大于第一预设值的视频,形成第二视频集合;Selecting a video whose face similarity is greater than the first preset value in the video library to form a second video set;
遍历每个视频,将每个视频作为第一视频,若第一视频的发型样式相似度、第一视频的头发长度的相似度、第一视频的头发卷度的相似度、第一视频的头发颜色的相似度和第一视频的刘海形状的相似度均大于第二预设值,则将第一视频添加到第一视频集合中;Traverse each video and use each video as the first video. If the similarity of the hairstyle style of the first video, the similarity of the length of the hair of the first video, the similarity of the curl of the hair of the first video, the similarity of the hair of the first video The similarity of the color and the similarity of the bangs shape of the first video are both greater than the second preset value, then the first video is added to the first video set;
遍历完每个视频之后,取第二视频集合和第一视频集合的交集,在交集的视频中选取目标视频。After traversing each video, the intersection of the second video set and the first video set is taken, and the target video is selected from the intersected videos.
在一些实施例中,人脸相似度包括第一视频的人脸相似度,第一视频的人脸相似度是用户的人脸与第一视频中人脸的人脸相似度;In some embodiments, the face similarity includes the face similarity of the first video, and the face similarity of the first video is the face similarity between the user's face and the face in the first video;
选择模块206用于根据人脸相似度和发型相似度,在视频库中筛选与用户对应的目标视频,包括:The selection module 206 is used to filter the target video corresponding to the user in the video library according to the similarity of the face and the similarity of the hairstyle, including:
使用相似度计算模型对第一视频的人脸相似度、第一视频的发型样式的相似度、第一视频的头发长度的相似度、第一视频的头发卷度的相似度、第一视频的头发颜色的相似度和第一视频的刘海形状的相似度进行加权求和,得到用户与第一视频中人物的综合相似度;Using the similarity calculation model to calculate the similarity of the face of the first video, the similarity of the hairstyle style of the first video, the similarity of the hair length of the first video, the similarity of the hair curl of the first video, the similarity of the first video The similarity of the hair color and the similarity of the bangs shape of the first video are weighted and summed to obtain the comprehensive similarity between the user and the characters in the first video;
遍历每个视频,得到用户与每个视频中人物的综合相似度;Traverse each video to get the comprehensive similarity between the user and the characters in each video;
在视频库中筛选综合相似度大于第三预设值的目标视频。The target videos whose comprehensive similarity is greater than the third preset value are screened in the video library.
在一些实施例中,还包括:训练模块207,用于训练相似度计算模型;In some embodiments, it further includes: a training module 207 for training a similarity calculation model;
训练模块207,用于训练相似度计算模型包括:The training module 207 for training the similarity calculation model includes:
获取样本人脸图像。Get a sample face image.
使用多个待训练模型分别计算样本人脸图像与每个视频对应人物的发型特征的相似度和人脸相似度。The similarity and face similarity between the sample face image and the hairstyle feature of the person corresponding to each video are calculated respectively by using multiple models to be trained.
针对每一个待训练模型,选择人脸相似度大于第一预设值且发型特征的相似度大于第二预设值的视频。For each model to be trained, a video with a face similarity greater than a first preset value and a hairstyle feature similarity greater than a second preset value is selected.
使用人脸换像模型将样本人脸图像换脸至所选择视频对应人物的人脸上,得到多个目标人脸图像。A face swapping model is used to swap the sample face image to the face of the person corresponding to the selected video to obtain multiple target face images.
将样本人脸图像与多个目标人脸图像输入至人脸识别模型中,计算样本人脸图像与多个目标人脸图像的相似度。The sample face image and the multiple target face images are input into the face recognition model, and the similarity between the sample face image and the multiple target face images is calculated.
计算每一个待训练模型对应的多个目标人脸图像的相似度的平均值;Calculate the average value of the similarity of multiple target face images corresponding to each model to be trained;
选择最高平均值对应的待训练模型作为相似度计算模型。Select the model to be trained corresponding to the highest average value as the similarity calculation model.
在一些实施例中,还包括,推荐模块208;In some embodiments, it also includes a recommendation module 208;
推荐模块208,用于将目标视频推荐给用户。The recommendation module 208 is used for recommending the target video to the user.
另外,结合图1描述的本发明实施例的人脸匹配方法可以由计算设备来实现。图3示出了本发明实施例提供的计算设备的硬件结构示意图。In addition, the face matching method of the embodiment of the present invention described in conjunction with FIG. 1 may be implemented by a computing device. FIG. 3 shows a schematic diagram of a hardware structure of a computing device provided by an embodiment of the present invention.
计算设备可以包括处理器301以及存储有计算机程序指令的存储器302。The computing device may include a
具体地,上述处理器301可以包括中央处理器(CPU),或者特定集成电路(Application Specific Integrated Circuit,ASIC),或者可以被配置成实施本发明实施例的一个或多个集成电路。Specifically, the above-mentioned
存储器302可以包括用于数据或指令的大容量存储器。举例来说而非限制,存储器302可包括硬盘驱动器(Hard Disk Drive,HDD)、软盘驱动器、闪存、光盘、磁光盘、磁带或通用串行总线(Universal Serial Bus,USB)驱动器或者两个或更多个以上这些的组合。在合适的情况下,存储器302可包括可移除或不可移除(或固定)的介质。在合适的情况下,存储器302可在数据处理装置的内部或外部。在特定实施例中,存储器302是非易失性固态存储器。在特定实施例中,存储器302包括只读存储器(ROM)。在合适的情况下,该ROM可以是掩模编程的ROM、可编程ROM(PROM)、可擦除PROM(EPROM)、电可擦除PROM(EEPROM)、电可改写ROM(EAROM)或闪存或者两个或更多个以上这些的组合。
处理器301通过读取并执行存储器302中存储的计算机程序指令,以实现上述实施例中的任意一种人脸匹配方法。The
在一个示例中,计算设备还可包括通信接口303和总线310。其中,如图3所示,处理器301、存储器302、通信接口303通过总线310连接并完成相互间的通信。In one example, the computing device may also include a
通信接口303,主要用于实现本发明实施例中各模块、装置、单元和/或设备之间的通信。The
总线310包括硬件、软件或两者,将计算设备的部件彼此耦接在一起。举例来说而非限制,总线可包括加速图形端口(AGP)或其他图形总线、增强工业标准架构(EISA)总线、前端总线(FSB)、超传输(HT)互连、工业标准架构(ISA)总线、无限带宽互连、低引脚数(LPC)总线、存储器总线、微信道架构(MCA)总线、外围组件互连(PCI)总线、PCI-Express(PCI-X)总线、串行高级技术附件(SATA)总线、视频电子标准协会局部(VLB)总线或其他合适的总线或者两个或更多个以上这些的组合。在合适的情况下,总线310可包括一个或多个总线。尽管本发明实施例描述和示出了特定的总线,但本发明考虑任何合适的总线或互连。The
另外,结合上述实施例中的人脸匹配方法,本发明实施例可提供一种计算机可读存储介质来实现。该计算机可读存储介质上存储有计算机程序指令;该计算机程序指令被处理器执行时实现上述实施例中的任意一种人脸匹配方法。In addition, in combination with the face matching method in the foregoing embodiments, the embodiments of the present invention may be implemented by providing a computer-readable storage medium. Computer program instructions are stored on the computer-readable storage medium; when the computer program instructions are executed by the processor, any one of the face matching methods in the foregoing embodiments is implemented.
需要明确的是,本发明并不局限于上文所描述并在图中示出的特定配置和处理。为了简明起见,这里省略了对已知方法的详细描述。在上述实施例中,描述和示出了若干具体的步骤作为示例。但是,本发明的方法过程并不限于所描述和示出的具体步骤,本领域的技术人员可以在领会本发明的精神后,作出各种改变、修改和添加,或者改变步骤之间的顺序。It is to be understood that the present invention is not limited to the specific arrangements and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above-described embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the sequence of steps after comprehending the spirit of the present invention.
以上所述的结构框图中所示的功能块可以实现为硬件、软件、固件或者它们的组合。当以硬件方式实现时,其可以例如是电子电路、专用集成电路(ASIC)、适当的固件、插件、功能卡等等。当以软件方式实现时,本发明的元素是被用于执行所需任务的程序或者代码段。程序或者代码段可以存储在机器可读介质中,或者通过载波中携带的数据信号在传输介质或者通信链路上传送。“机器可读介质”可以包括能够存储或传输信息的任何介质。机器可读介质的例子包括电子电路、半导体存储器设备、ROM、闪存、可擦除ROM(EROM)、软盘、CD-ROM、光盘、硬盘、光纤介质、射频(RF)链路,等等。代码段可以经由诸如因特网、内联网等的计算机网络被下载。The functional blocks shown in the above-described structural block diagrams may be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, it may be, for example, an electronic circuit, an application specific integrated circuit (ASIC), suitable firmware, a plug-in, a function card, or the like. When implemented in software, elements of the invention are programs or code segments used to perform the required tasks. The program or code segments may be stored in a machine-readable medium or transmitted over a transmission medium or communication link by a data signal carried in a carrier wave. A "machine-readable medium" may include any medium that can store or transmit information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, and the like. The code segments may be downloaded via a computer network such as the Internet, an intranet, or the like.
还需要说明的是,本发明中提及的示例性实施例,基于一系列的步骤或者装置描述一些方法或系统。但是,本发明不局限于上述步骤的顺序,也就是说,可以按照实施例中提及的顺序执行步骤,也可以不同于实施例中的顺序,或者若干步骤同时执行。It should also be noted that the exemplary embodiments mentioned in the present invention describe some methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the above steps, that is, the steps may be performed in the order mentioned in the embodiment, or may be different from the order in the embodiment, or several steps may be performed simultaneously.
以上所述,仅为本发明的具体实施方式,所属领域的技术人员可以清楚地了解到,为了描述的方便和简洁,上述描述的系统、模块和单元的具体工作过程,可以参考前述方法实施例中的对应过程,在此不再赘述。应理解,本发明的保护范围并不局限于此,任何熟悉本技术领域的技术人员在本发明揭露的技术范围内,可轻易想到各种等效的修改或替换,这些修改或替换都应涵盖在本发明的保护范围之内。The above are only specific implementations of the present invention. Those skilled in the art can clearly understand that, for the convenience and brevity of the description, for the specific working process of the above-described systems, modules and units, reference may be made to the foregoing method embodiments. The corresponding process in , will not be repeated here. It should be understood that the protection scope of the present invention is not limited to this. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should all cover within the protection scope of the present invention.
Claims (10)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910526561.8A CN112101072B (en) | 2019-06-18 | 2019-06-18 | Face matching method, device, equipment and medium |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910526561.8A CN112101072B (en) | 2019-06-18 | 2019-06-18 | Face matching method, device, equipment and medium |
Publications (2)
Publication Number | Publication Date |
---|---|
CN112101072A true CN112101072A (en) | 2020-12-18 |
CN112101072B CN112101072B (en) | 2024-12-13 |
Family
ID=73749385
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201910526561.8A Active CN112101072B (en) | 2019-06-18 | 2019-06-18 | Face matching method, device, equipment and medium |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN112101072B (en) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113658035A (en) * | 2021-08-17 | 2021-11-16 | 北京百度网讯科技有限公司 | Face transformation method, device, equipment, storage medium and product |
CN113965802A (en) * | 2021-10-22 | 2022-01-21 | 深圳市兆驰股份有限公司 | Immersive video interaction method, device, equipment and storage medium |
CN115776597A (en) * | 2021-08-30 | 2023-03-10 | 海信集团控股股份有限公司 | A method, device and electronic equipment for generating audio and video |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR20120049761A (en) * | 2010-11-09 | 2012-05-17 | 김양웅 | Apparatus and method of searching for target based on matching possibility |
CN105005777A (en) * | 2015-07-30 | 2015-10-28 | 科大讯飞股份有限公司 | Face-based audio and video recommendation method and face-based audio and video recommendation system |
CN105069746A (en) * | 2015-08-23 | 2015-11-18 | 杭州欣禾圣世科技有限公司 | Video real-time human face substitution method and system based on partial affine and color transfer technology |
CN105868684A (en) * | 2015-12-10 | 2016-08-17 | 乐视网信息技术(北京)股份有限公司 | Video information acquisition method and apparatus |
CN106372068A (en) * | 2015-07-20 | 2017-02-01 | 中兴通讯股份有限公司 | Method and device for image search, and terminal |
CN108009521A (en) * | 2017-12-21 | 2018-05-08 | 广东欧珀移动通信有限公司 | Humanface image matching method, device, terminal and storage medium |
-
2019
- 2019-06-18 CN CN201910526561.8A patent/CN112101072B/en active Active
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR20120049761A (en) * | 2010-11-09 | 2012-05-17 | 김양웅 | Apparatus and method of searching for target based on matching possibility |
CN106372068A (en) * | 2015-07-20 | 2017-02-01 | 中兴通讯股份有限公司 | Method and device for image search, and terminal |
CN105005777A (en) * | 2015-07-30 | 2015-10-28 | 科大讯飞股份有限公司 | Face-based audio and video recommendation method and face-based audio and video recommendation system |
CN105069746A (en) * | 2015-08-23 | 2015-11-18 | 杭州欣禾圣世科技有限公司 | Video real-time human face substitution method and system based on partial affine and color transfer technology |
CN105868684A (en) * | 2015-12-10 | 2016-08-17 | 乐视网信息技术(北京)股份有限公司 | Video information acquisition method and apparatus |
CN108009521A (en) * | 2017-12-21 | 2018-05-08 | 广东欧珀移动通信有限公司 | Humanface image matching method, device, terminal and storage medium |
Non-Patent Citations (2)
Title |
---|
FLETCHER, KI: "Attention to internal face features in unfamiliar face matching", BRITISH JOURNAL OF PSYCHOLOGY, vol. 99, 1 August 2008 (2008-08-01), pages 379 - 394 * |
黄孝平;: "基于体绘制思维的人脸识别算法优化研究", 现代电子技术, no. 24, 15 December 2015 (2015-12-15) * |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113658035A (en) * | 2021-08-17 | 2021-11-16 | 北京百度网讯科技有限公司 | Face transformation method, device, equipment, storage medium and product |
CN113658035B (en) * | 2021-08-17 | 2023-08-08 | 北京百度网讯科技有限公司 | Face transformation method, device, equipment, storage medium and product |
CN115776597A (en) * | 2021-08-30 | 2023-03-10 | 海信集团控股股份有限公司 | A method, device and electronic equipment for generating audio and video |
CN113965802A (en) * | 2021-10-22 | 2022-01-21 | 深圳市兆驰股份有限公司 | Immersive video interaction method, device, equipment and storage medium |
Also Published As
Publication number | Publication date |
---|---|
CN112101072B (en) | 2024-12-13 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN110362677B (en) | Text data category identification method and device, storage medium and computer equipment | |
CN111860044B (en) | Face changing method, device, equipment and computer storage medium | |
CN112101072A (en) | Face matching method, device, equipment and medium | |
CN111860041B (en) | Face conversion model training method, device, equipment and medium | |
CN110866469B (en) | Facial five sense organs identification method, device, equipment and medium | |
CN107423306B (en) | Image retrieval method and device | |
CN104881640B (en) | A kind of method and device for obtaining vector | |
CN109117773A (en) | Image feature point detection method, terminal device and storage medium | |
CN110570383B (en) | Image processing method and device, electronic equipment and storage medium | |
CN109690556A (en) | Personage's central characteristics particular photos match rank engine | |
CN107358490A (en) | A kind of image matching method, device and electronic equipment | |
CN112101073B (en) | Facial image processing method, device, equipment and computer storage medium | |
CN113240598A (en) | Face image deblurring method, face image deblurring device, medium and equipment | |
CN108171208A (en) | Information acquisition method and device | |
CN112973122A (en) | Game role makeup method and device and electronic equipment | |
Tan et al. | Face recognition using the weighted fractal neighbor distance | |
CN109977745B (en) | Face image processing method and related device | |
CN110390344B (en) | Alternative frame updating method and device | |
KR101713189B1 (en) | Tensor Divergence Feature Extraction System based on HoG and HOF for video obejct action classification | |
CN111931148A (en) | Image processing method and device and electronic equipment | |
CN114360015A (en) | Living body detection method, living body detection device, living body detection equipment and storage medium | |
CN108491824A (en) | model generating method and device | |
CN110046340A (en) | The training method and device of textual classification model | |
CN111860045B (en) | Face changing method, device, equipment and computer storage medium | |
CN113887279A (en) | Pedestrian re-identification method with half-length being blocked and related device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant | ||
TG01 | Patent term adjustment | ||
TG01 | Patent term adjustment |