WO2019101113A1 - 一种图像融合方法及其设备、存储介质、终端 - Google Patents

一种图像融合方法及其设备、存储介质、终端 Download PDF

Info

Publication number
WO2019101113A1
WO2019101113A1 PCT/CN2018/116832 CN2018116832W WO2019101113A1 WO 2019101113 A1 WO2019101113 A1 WO 2019101113A1 CN 2018116832 W CN2018116832 W CN 2018116832W WO 2019101113 A1 WO2019101113 A1 WO 2019101113A1
Authority
WO
WIPO (PCT)
Prior art keywords
face
image data
source
skin color
face image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2018/116832
Other languages
English (en)
French (fr)
Inventor
沈珂轶
程培
钱梦仁
傅斌
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology Shenzhen Co Ltd
Original Assignee
Tencent Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology Shenzhen Co Ltd filed Critical Tencent Technology Shenzhen Co Ltd
Publication of WO2019101113A1 publication Critical patent/WO2019101113A1/zh
Priority to US16/859,331 priority Critical patent/US11037281B2/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three-dimensional [3D] modelling for computer graphics
    • G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—Three-dimensional [3D] image rendering
    • G06T15/50—Lighting effects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/90—Determination of colour characteristics
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/64—Three-dimensional [3D] objects
    • G06V20/647—Three-dimensional [3D] objects by matching two-dimensional images to three-dimensional objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161—Detection; Localisation; Normalisation
    • G06V40/165—Detection; Localisation; Normalisation using facial parts and geometric relationships
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168—Feature extraction; Face representation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Definitions

  • the present application relates to the field of computer technologies, and in particular, to an image fusion method, an apparatus, a storage medium, and a terminal.
  • terminal applications such as cameras, P-picture software or retouching software, social apps, etc. can be installed in terminal devices such as smart phones, PDAs, and tablets.
  • terminal devices such as smart phones, PDAs, and tablets.
  • the user can add special effects, decorative beautification, beauty and beauty, and change the character shape to the original picture (for example, characters, scenery, buildings, etc.) or video.
  • special effects for example, characters, scenery, buildings, etc.
  • the embodiment of the present application provides an image fusion method, a device, a storage medium, and a terminal.
  • the process of generating image data of a target face image by analyzing image data on the basis of a three-dimensional model can improve a target face image finally obtained. The authenticity of the data.
  • the first aspect of the embodiments of the present application provides an image fusion method, which is used on a terminal, where the terminal includes: a processor and a memory, and the method may include:
  • the target skin color is merged with the source skin color data of the source face image data and the material skin color data of the material face image data to generate the merged target face image data.
  • a second aspect of the embodiments of the present application provides an image fusion device, which may include: a processor and a memory, the memory storing a computer program loaded by the processor and performing the following steps:
  • the target skin color is merged with the source skin color data of the source face image data and the material skin color data of the material face image data to generate the merged target face image data.
  • a third aspect of the embodiments of the present application provides a computer storage medium storing a plurality of instructions, the instructions being adapted to be loaded by a processor and performing the following steps:
  • the target skin color is merged with the source skin color data of the source face image data and the material skin color data of the material face image data to generate the merged target face image data.
  • a fourth aspect of the embodiments of the present application provides a terminal, which may include: a processor and a memory; wherein the memory stores a computer program, the computer program being adapted to be loaded by the processor and performing the following steps:
  • the target skin color is merged with the source skin color data of the source face image data and the material skin color data of the material face image data to generate the merged target face image data.
  • FIG. 1A is a schematic diagram of an application scenario to which an image fusion device according to an embodiment of the present disclosure is applied;
  • FIGS. 1B-1H are schematic diagrams of application of an image fusion method provided by an embodiment of the present application.
  • FIG. 2 is a schematic flowchart of an image fusion method according to an embodiment of the present application.
  • FIG. 3 is a schematic diagram of a location of a face reference point according to an embodiment of the present application.
  • FIG. 4 is a schematic diagram of a three-dimensional mesh model of a human face according to an embodiment of the present application.
  • FIG. 5A and FIG. 5B are schematic diagrams of a face type provided by an embodiment of the present application.
  • FIG. 6 is a schematic flowchart diagram of another image fusion method according to an embodiment of the present disclosure.
  • FIG. 7 is a schematic flowchart diagram of another image fusion method according to an embodiment of the present disclosure.
  • FIG. 8 is a schematic flowchart diagram of another image fusion method according to an embodiment of the present disclosure.
  • FIG. 9 is a schematic diagram of an image fusion method system according to an embodiment of the present application.
  • FIG. 10 is a schematic structural diagram of an image fusion device according to an embodiment of the present disclosure.
  • FIG. 11 is a schematic structural diagram of another image fusion device according to an embodiment of the present disclosure.
  • FIG. 12 is a schematic structural diagram of a source mesh generation module according to an embodiment of the present application.
  • FIG. 13 is a schematic structural diagram of a target data generating module according to an embodiment of the present application.
  • FIG. 14 is a schematic structural diagram of a target data generating unit according to an embodiment of the present application.
  • FIG. 15 is a schematic structural diagram of a terminal according to an embodiment of the present application.
  • the process of processing the face image data by the retouching terminal application is based on the 2D face information of the user and the 2D face information of the material, and a result of generating a face similar to the user and the pixel in the pixel by a certain fusion algorithm. image.
  • the image information that can be extracted by the fused user image and the material image as a planar image reflects the effect of the real face, which will result in The final fusion effect is poor, which affects the authenticity of the final target image.
  • the image fusion device obtains the source face image data of the current image to be fused and the material configuration information of the current material to be fused, wherein the material configuration information includes the material face image data, the material skin color data, and the material face three-dimensional network.
  • the source face three-dimensional mesh is then meshed by the material face three-dimensional mesh and the source face three-dimensional mesh to generate a target human face three-dimensional mesh, and finally the source skin color of the source face image data is adopted.
  • the data and the material skin color data of the material face image data are subjected to facial skin color fusion on the target face mesh to generate the merged target face image data.
  • FIG. 1A is a schematic diagram of an application scenario applied to an image fusion device according to an embodiment of the present disclosure.
  • the application scenario includes an image fusion device 10 and a server 20.
  • An image fusion application client is installed on the image fusion device 10.
  • the server 20 is, for example, an image fusion application server that communicates with the image fusion application client over a network.
  • the image fusion application client can perform image fusion through the image fusion method of the embodiment of the present application.
  • the interface of the image fusion application client can prompt the user to select a material to be merged. After the user selects the material to be merged, the image fusion application can be local or from the image fusion device 10.
  • the material face image data of the material to be fused is acquired on the server 20.
  • the material to be fused may be a video or an image.
  • the interface of the image fusion application client may prompt the user to take a self-portrait to obtain the source face image data of the user's face, or guide the user to select a video or image from the album on the image fusion device 10, and obtain the source from the source. Face image data.
  • the client of the image fusion application fuses the source face image data with the material face image data to form target face image data, that is, the merged face image or video, and outputs the display to the user.
  • the client of the image fusion application can communicate with the server 20 to update the material face image data of the material to be fused.
  • a separate stand-alone image fusion application can also be installed on the fusion device 10.
  • the image fusion application includes a client of the image fusion application and a stand-alone image fusion application.
  • the image fusion device 10 may be a tablet computer, a smart phone, a palmtop computer, a mobile Internet device (MID), and other terminal devices having image processing functions, and may also be used in various retouching terminal applications.
  • FIG. 1B to FIG. 1H are schematic diagrams of application of an image fusion method according to an embodiment of the present application.
  • the user enters the image fusion application on the image fusion device, there is an option on the interface of the image fusion application to allow the user to select a picture or video to be merged.
  • the user can select a self-timer or obtain a picture or video to be merged locally from the image fusion device.
  • a preview interface of the picture or video selected by the user can be displayed on the interface of the image fusion application, as shown in FIG. 1B.
  • the image fusion application determines the picture or video selected by the user, and obtains the source face image data therefrom.
  • the image fusion application has a thumbnail of the material selected by the user, as shown in FIG. 1C.
  • the material map corresponding to the material prime diagram 30 is 31.
  • a result graph 32 in which the user's face is merged with the material map 31 is generated.
  • the user can select the button 33 on the interface to save the fusion result map. Since the face image of the user to be fused is a frontal image in this process, the face in the material image to be fused is a side image. Therefore, when the fusion is performed, the image fusion application obtains a 3D map of the user's face from the user-selected picture (as shown in FIG. 1D), as shown in FIG. 1E. After that, the image fusion application rotates the 3D image of the user's face into a 3D map that is consistent with the face angle in the material map, as shown by 1F. The image fusion application then pastes the texture of the user's face image on the rotated 3D image, as shown in FIG. 1G. Finally, the fusion result graph shown in Fig. 1H is generated.
  • FIG. 2 is a schematic flowchart diagram of an image fusion method according to an embodiment of the present application. As shown in FIG. 2, the method in this embodiment of the present application may include the following steps S101 to S104.
  • the image fusion device may acquire the source face image data of the current image to be fused and the material configuration information of the current material to be fused.
  • the source face image data may be face image data in a photo or video that the user currently photographs through the image fusion device or selected from the album of the image fusion device.
  • the current material to be fused may be a material model used for retouching currently selected by the user in a retouching terminal application (for example, a certain picture show, a certain P picture, a certain camera, etc.), for example, an animated character image. , star photos, etc.
  • the material configuration information may include 3D avatar information of the current material to be fused (for example, may be a file in an obj format, and the file may include a representation of the material face image data, the material skin color data, and the material face three-dimensional grid.
  • Information about the material face related data may include the orientation of the material 3D avatar in the world coordinate system (Euler angle pitch, yaw, roll), center position (final result)
  • the specified position of the image, the scale information and the matching camera information (such as using the perspective matrix to portray this information), 2D stickers and 3D stickers, and the degree of fusion of the user's face and the material face.
  • Alpha the degree of fusion can be the same for each frame or different for each frame).
  • the avatar may be an image of the entire head or only an image of the face area.
  • the size of the material face image data and the source face image data used to generate the target face image data need to correspond to the same scale, but the face concavity, fat and thin, etc. The situation may not be completely consistent. Therefore, after the source face image data and the material face image data are acquired, the size of the source face image data may be adjusted according to the size of the material face image data. Make the size of the two correspond to the same scale.
  • S102 Perform image recognition processing on the source face image data, acquire source face feature points corresponding to the source face image data, and generate a source of the source face image data according to the source face feature points.
  • Human face 3D mesh Perform image recognition processing on the source face image data, acquire source face feature points corresponding to the source face image data, and generate a source of the source face image data according to the source face feature points.
  • the image fusion device may perform image recognition processing on the source face image data, and acquire source face feature points corresponding to the source face image data.
  • the image recognition process may be to identify a user face in a photo by using a face detection technology (for example, a face detection provided by the cross-platform computer vision library OpenCV, a new visual service platform Face++, a U-shaped face detection, etc.)
  • the process of facial features positioning, the source facial feature points may be data points capable of characterizing facial features of the source facial image data (eg, facial contours, eye contours, nose, lips, etc.).
  • the image fusion device may perform image recognition processing on the source face image data (for example, the user face in the photo may be identified and facial features may be positioned to obtain a certain number of reference feature points) And acquiring a reference feature point of the source face image data, performing third-dimensional depth information extraction on the reference feature point, and acquiring a source face feature point corresponding to the reference feature point.
  • the three-dimensional depth information extraction may be based on the reference feature points described above, and the process of reflecting the feature points of the source face image data in the three-dimensional model by estimating the facial feature points of the standard three-dimensional model.
  • the reference feature point may be a reference point indicating a facial feature, for example, a facial contour, an eye contour, a nose, a lip, etc., may be 83 reference points, or may be 68 reference points as shown in FIG.
  • the specific points can be determined by the developer according to the needs.
  • the source face feature point may be a feature point capable of corresponding to the three-dimensional model of the source face image data after further deepening on the basis of the reference feature point, for example, by using three-dimensional depth information on the 68 or 83 reference points.
  • the extraction may obtain 1000 deepened source facial feature points, which may be the vertices of each triangular patch shown in FIG. 4.
  • the image fusion device may generate a source face three-dimensional mesh of the source face image data according to the source face feature point.
  • the source human face three-dimensional mesh may be a 3D face mesh model corresponding to the face in the source face image data, such as the 3D face mesh shown in FIG. 4 or only the half face similar to FIG. 3D face mesh.
  • S103 Perform mesh fusion using the material human face three-dimensional mesh and the source human face three-dimensional mesh to generate a target human face three-dimensional mesh.
  • the scales of the source face three-dimensional mesh and the material face three-dimensional network are also in the same Under the scale.
  • the image fusion device may perform mesh fusion using the material face three-dimensional mesh and the source face three-dimensional mesh to generate a target human face three-dimensional mesh.
  • the material face 3D mesh may be similar to the source face 3D mesh, and may be a 3D face mesh model corresponding to the face in the material face image data, which may be the 3D face shown in FIG. 4 .
  • the mesh or 3D face mesh based on the 3D face mesh shown in FIG. 4 changes the backward 3D face mesh according to the Euler angle in the world coordinate system.
  • the target human face three-dimensional mesh may be a 3D face mesh of the target face image data. According to the embodiment of the present application, when the material face three-dimensional mesh is the 3D face mesh shown in FIG.
  • the final projection onto the 2D image can present the positive face effect, and when the material face three-dimensional mesh is based on FIG. 4
  • the 3D face mesh shown changes the backward 3D face mesh according to the Euler angle in the world coordinate system
  • the final projection onto the 2D image can present the effect of the side face.
  • the image fusion device may calculate a target face feature point of the target face according to the source face three-dimensional mesh and the face feature point in the material face three-dimensional mesh, and then calculate according to the The obtained target face feature points generate a three-dimensional mesh of the target face.
  • a 3D source face 3D mesh ie, a user's 3D face mesh
  • 3D material face 3D mesh has 1000 depth information.
  • the material face feature points are marked in blue, and the average points of the corresponding points of the 1000 face feature points of the user and the material (average points of the same position are averaged, a total of 1000 point pairs) are marked in red, and finally generated
  • the 1000 red face feature points are the target face feature points, and the above 1000 red face feature points can form more than 1900 triangles, and the corresponding three-dimensional grid of the face depicted by the corresponding 1900 triangle faces is
  • the target face is a three-dimensional grid.
  • the image fusion device may adopt the Image Deformation Using Moving Least Squares (MLS) method, the affine transformation, the image distortion, and the like to source the source image data and the material face.
  • the facial features of the image data tend to be the facial features indicated by the above-mentioned red facial feature points, that is, the facial features indicated by the target facial features, thereby achieving the purpose of face fusion.
  • MLS Moving Least Squares
  • S104 Perform skin color fusion on the target human face three-dimensional mesh by using the source skin color data of the source face image data and the material skin color data of the material image data to generate the merged target face image data.
  • the image fusion device needs to fill the skin color on the different triangular patches in the target human face three-dimensional mesh to obtain the final target facial image data.
  • the image fusion device may use the source skin color data of the source face image data and the material skin color data of the material image data to perform facial skin color fusion on the target human face three-dimensional mesh to generate a merged Target face image data.
  • the source skin color data may be a set of source pixel points constituting the source face image data
  • the material skin color data may be a set of material pixel points constituting the material face image data.
  • the source face 3D mesh needs to be network supplemented to generate a candidate face 3D mesh.
  • the type may be the source face three-dimensional mesh and the network elements in the material face three-dimensional grid.
  • the candidate skin color data of the candidate face three-dimensional mesh may be the symmetric source face image data, that is, the candidate skin color data of the candidate face three-dimensional mesh may be considered as the source skin color data.
  • the image fusion device may calculate the target according to the fusion degree in the source pixel point, the material pixel point, and the material configuration information (which may be a fusion degree value set according to an empirical value, usually between 0-1)
  • the pixels of a feature point on a triangular patch in the 3D mesh of the source face are: UserB, UserG, and UserR
  • the pixels of the face feature point at the corresponding position on the corresponding triangular patch in the 3D mesh of the material face are: ResourceB, ResourceG, and ResourceR
  • the pixels of the feature points at the corresponding positions on the corresponding triangular patches in the target human face three-dimensional grid are:
  • TargetB TargetB, TargetG, and TargetR
  • degree of fusion in the material configuration information is: alpha
  • TargetB (1.0–alpha)*UserB+alpha*ResourceB
  • TargetG (1.0–alpha)*UserG+alpha*ResourceG
  • TargetR (1.0–alpha)*UserR+alpha*ResourceR
  • each pixel value of the target face image data can be obtained, and the target face image data is obtained.
  • the target face image data finally generated may be three-dimensional face image data or two-dimensional face image data.
  • the target face image data is two-dimensional face image data
  • the process of generating the target image data is implemented on the basis of the three-dimensional model, the target person is finally formed in consideration of real light, shadow, and the like. Face image data is more realistic.
  • the candidate skin color data of the candidate face three-dimensional mesh may include two parts, that is, the skin color data of the candidate face three-dimensional mesh matching the source face three-dimensional mesh is the source skin color Data, the skin color data of the candidate face three-dimensional mesh that does not match the source face three-dimensional mesh is the average skin color data, and the average skin color data may be the skin color data of the source skin color data after the skin color equalization processing .
  • the skin color equalization processing may be a process of removing the average value of the skin color by removing effects such as shadows caused by light or the like in the source face image data.
  • the image fusion device may calculate a target pixel point of the target face image data according to a candidate pixel point of the candidate skin color data, a material pixel point, and a degree of fusion in the material configuration information, and further fill the image according to the target pixel point.
  • the target face 3D mesh generates the target face image data, and the specific calculation process is consistent with the above calculation process, and will not be described herein.
  • the source face image data of the current image to be fused and the material configuration information of the current material to be fused are acquired, wherein the material configuration information includes the material face image data, the material skin color data, and the material face three-dimensional grid.
  • performing image recognition processing on the source face image data acquiring the source face feature points corresponding to the source face image data, and generating a source face 3D mesh of the source face image data according to the source face feature points, and then adopting
  • the material face 3D mesh and the source face 3D mesh are meshed to generate a target face 3D mesh, and finally the source skin color data of the source face image data and the material skin color data of the material face image data are used to target the face.
  • the mesh merges the face skin color to generate the merged target face image data.
  • the 3D mesh is merged into the target face 3D mesh, and the target face 3D mesh is merged to generate the target face image data. Improve the authenticity of the final target image data obtained.
  • FIG. 6 is a schematic flowchart diagram of another image fusion method according to an embodiment of the present application. As shown in FIG. 6, the method in this embodiment of the present application may include the following steps S201 to S210.
  • the image fusion device may acquire the source face image data of the current image to be fused and the material configuration information of the current material to be fused.
  • the source face image data may be face image data in a photo or video that the user currently photographed by the image fusion device or selected from the album of the image fusion device.
  • the current material to be fused may be a material model used for retouching currently selected by the user in a retouching terminal application (for example, a certain picture show, a certain P picture, a certain camera, etc.), for example, an animated character image. , star photos, etc.
  • the material configuration information may include 3D avatar information of the current material to be fused (for example, may be a file in an obj format, and the file may include a representation of the material face image data, the material skin color data, and the material face three-dimensional grid.
  • Information about the material face related data may include the orientation of the material 3D avatar in the world coordinate system (Euler angle pitch, yaw, roll), center position (final result)
  • the specified position of the image, the scale information and the matching camera information (such as using the perspective matrix to portray this information), 2D stickers and 3D stickers, and the degree of fusion of the user's face and the material face.
  • Alpha the degree of fusion can be the same for each frame or different for each frame).
  • the size of the material face image data and the source face image data used to generate the target face image data need to correspond to the same scale, but the face concavity, fat and thin, etc. The situation may not be completely consistent. Therefore, after the source face image data and the material face image data are acquired, the size of the source face image data may be adjusted according to the size of the material face image data. Make the size of the two correspond to the same scale.
  • the image fusion device may perform image recognition processing on the source face image data to acquire a reference feature point of the source face image data.
  • the image recognition process may be to identify a user face in a photo by using a face detection technology (for example, a face detection provided by the cross-platform computer vision library OpenCV, a new visual service platform Face++, a U-shaped face detection, etc.) The process of positioning the five senses.
  • the reference feature point may be a reference point indicating a facial feature, for example, a facial contour, an eye contour, a nose, a lip, etc., may be 83 reference points, or may be 68 reference points as shown in FIG.
  • the specific points can be determined by the developer according to the needs.
  • S203 Perform three-dimensional depth information extraction on the reference feature point, acquire a source face feature point corresponding to the reference feature point, and generate a source face three-dimensional mesh according to the source face feature point.
  • the image fusion device may perform three-dimensional depth information extraction on the reference feature point, acquire a source face feature point corresponding to the reference feature point, and generate a source face according to the source face feature point. 3D mesh.
  • the three-dimensional depth information extraction may be based on the reference feature points described above, and the process of reflecting the feature points of the source face image data in the three-dimensional model by estimating the facial feature points of the standard three-dimensional model.
  • the source face feature point may be a point further deepened on the basis of the reference feature point. For example, 1000 deepened source faces may be obtained by extracting the three-dimensional depth information of the 68 or 83 reference points.
  • the feature points may be the vertices of the respective triangular patches in FIG.
  • the source face 3D mesh may be a 3D face mesh model corresponding to the face in the source face image data, and may be a 3D mesh of the user face connected by the source face feature points.
  • the model such as the 3D face mesh shown in Figure 4 or a 3D face mesh with only a half face similar to Figure 4.
  • S204 Perform mesh fusion using the source human face three-dimensional mesh and the material face three-dimensional mesh to generate a target human face three-dimensional mesh.
  • the scales of the source face three-dimensional mesh and the material face three-dimensional network are also in the same Under the scale.
  • the image fusion device may perform mesh fusion using the material face three-dimensional mesh and the source face three-dimensional mesh to generate a target human face three-dimensional mesh.
  • the material face three-dimensional mesh may be similar to the source face three-dimensional mesh may be a 3D face mesh model corresponding to a face in the material face image data, which may be FIG. 4
  • the illustrated 3D face mesh or the 3D face mesh based on the 3D face mesh shown in FIG. 4 changes the backward 3D face mesh according to the Euler angle in the world coordinate system.
  • the target human face three-dimensional mesh may be a 3D face mesh of the target face image data. According to the embodiment of the present application, when the material face three-dimensional mesh is the 3D face mesh shown in FIG.
  • the final projection onto the 2D image can present the positive face effect, and when the material face three-dimensional mesh is based on FIG. 4
  • the 3D face mesh shown changes the backward 3D face mesh according to the Euler angle in the world coordinate system
  • the final projection onto the 2D image can present the effect of the side face.
  • the image fusion device may calculate a target face feature point of the target face according to the source face three-dimensional mesh and the face feature point in the material face three-dimensional mesh, and then according to The calculated target face feature points generate a three-dimensional mesh of the target face.
  • a 3D source face 3D mesh ie, a user's 3D face mesh
  • 3D material face 3D mesh has 1000 depth information.
  • the material face feature points are marked in blue, and the average points of the corresponding points of the 1000 face feature points of the user and the material (average points of the same position are averaged, a total of 1000 point pairs) are marked in red, and finally generated
  • the 1000 red face feature points are the target face feature points, and the above 1000 red face feature points can form more than 1900 triangles, and the corresponding three-dimensional grid of the face depicted by the corresponding 1900 triangle faces is
  • the target face is a three-dimensional grid.
  • the image fusion device may adopt an MLS method, an affine transformation, an image distortion, or the like to align the facial features of the source face image data and the material face image data with the red face feature point, that is, the target.
  • the facial features indicated by the face feature points achieve the purpose of face fusion.
  • S205 Perform skin color equalization processing on the source face image data, and acquire average skin color data of the source face image data.
  • the source face three-dimensional mesh needs to be network supplemented to generate a candidate face three-dimensional mesh.
  • the type may be the source face three-dimensional mesh and the network elements in the material face three-dimensional grid.
  • the candidate skin color data of the candidate face three-dimensional mesh may be the symmetric source face image data, that is, the candidate skin color data of the candidate face three-dimensional mesh may be considered as the source skin color data.
  • the last generated candidate face image data is the face image in FIG. 5A, and the skin color of the face in FIG. 5A coincides with the skin color of the face in FIG. 5B.
  • the candidate skin color data of the candidate face three-dimensional grid may include two parts, that is, the skin color data of the candidate face three-dimensional mesh matching the source face three-dimensional mesh is the source.
  • the skin color data, the skin color data of the candidate face three-dimensional mesh that does not match the source face three-dimensional mesh is the average skin color data.
  • the image fusion device may perform skin color equalization processing on the source face image data to acquire average skin color data of the source face image data.
  • the skin color equalization processing may be a process of removing the average value of the skin color by removing effects such as shadows due to light or the like in the source face image data.
  • the average skin color data may be a set of pixel points composed of an average value of the pixel skin data after the source skin color data is removed.
  • the image fusion device may perform skin color filling on the candidate face three-dimensional mesh based on the source skin color data of the source face image data and the average skin color data to generate candidate face image data.
  • the skin color data of the candidate face three-dimensional mesh matching the source human face three-dimensional mesh may be filled with the source skin color data of the source face image data, wherein the candidate face is in the three-dimensional mesh
  • the skin color data of the source face three-dimensional mesh unmatched portion may be filled with the average skin color data.
  • the candidate skin color data of the candidate face image data may include the source skin color data and the average skin color data. For example, if the last generated candidate face image data is the face image in FIG. 5A, the skin color of the right face in FIG. 5A is the skin color of the face in FIG. 5B, and the skin color of the left face in FIG. 5A is the face color in FIG. 5B. Averaged skin tone.
  • S207 Perform face skin color fusion on the target face three-dimensional mesh by using the candidate skin color data of the candidate face image data and the material skin color data of the material image data to generate the merged target face image data.
  • the image fusion device needs to fill the skin color on the different triangular patches in the target human face three-dimensional mesh to obtain the final target facial image data.
  • the image fusion device may use the candidate skin color data of the candidate face image data and the material skin color data of the material image data to perform facial skin color fusion on the target human face three-dimensional mesh to generate a merged Target face image data.
  • the candidate skin color data may be a set of candidate pixel points constituting the candidate face image data
  • the material skin color data may be a set of material pixel points constituting the material face image data.
  • the image fusion device may calculate the target pixel point based on the candidate pixel point and the material pixel point, and generate the target face image data according to the target pixel point.
  • the degree of fusion (which may be a fusion value set according to an empirical value, usually between 0-1).
  • the pixels of a feature point on a triangular patch in the three-dimensional grid of the candidate face may be set as: CandidateB, CandidateG, and CandidateR, corresponding to the corresponding triangular patches in the three-dimensional grid of the material face
  • the pixels of the face feature points at the position are: ResourceB, ResourceG, and ResourceR.
  • the pixels of the feature points at the corresponding positions on the corresponding triangular patches in the target human face are: TargetB, TargetG, and TargetR, and the fusion in the material configuration information.
  • the degree is: alpha, then:
  • TargetB (1.0–alpha)*CandidateB+alpha*ResourceB
  • TargetG (1.0–alpha)*CandidateG+alpha*ResourceG
  • TargetR (1.0–alpha)*CandidateR+alpha*ResourceR
  • each pixel value of the target face image data can be obtained, and the target face image data is obtained.
  • the authenticity of the finally obtained target face image data is increased.
  • S208 Acquire, according to source skin color data of the source face image data, a light source type corresponding to the source face image data, and perform an effect adding process on the target face image data by using a light effect corresponding to the light source type.
  • the image fusion device may acquire the light source type corresponding to the source facial image data according to the source skin color data of the source facial image data.
  • the image fusion device may obtain a region of light and average skin color and a region of dark and average skin color by comparing the skin color of the average skin color and the color of each region in the source face image data, thereby deriving the light source type, for example, A point source or a surface source.
  • the image fusion device may also collect the result map of the source face image data under different specified illumination conditions through deep learning, and then use the result map and the corresponding illumination condition as the training data of Deep Neural Networks (DNN). Train a DNN model that outputs a source type and source position for a given picture.
  • DNN Deep Neural Networks
  • the image fusion device may perform an effect adding process on the target face image data by using a lighting effect corresponding to the light source type. For example, if the corresponding light source type of the source face image data is a point light source from the left face direction, the image fusion device may add a lighting effect of the left face direction point light source in the finally obtained target face image data. .
  • the image fusion device may further paste the 2D and 3D stickers in the material configuration information on the target face image data according to their corresponding positions and levels (zOrder), for example, 3D.
  • the glasses sticker is worn on the face of the person who generated the target face image data.
  • the image fusion device may further adjust a current display position of the target facial image data based on coordinate information indicated by the material facial image data. For example, the obtained target face image data is placed at a specified position based on the coordinate information (including the Euler direction and the center point) and the material size indicated by the material face image data in the material configuration information described above.
  • the image fusion device may divide the second display area A portion of the first display area performs a face edge filling process.
  • the first display area may be a range in which the target face image data is mapped on a 2D screen
  • the second display area may be a range in which the source face image data is mapped on a 2D screen
  • the first The display area is smaller than the second display area
  • the face that can be expressed as the target face image data is smaller than the face of the source face image data.
  • the face edge filling process may be to fill a portion of the second display area except the first display area with a padding algorithm (for example, an image restoration algorithm Inpainting provided by OpenCV).
  • the final obtained target facial image data may be output and displayed.
  • the target face image data finally generated after performing step S208-step S210 may be three-dimensional face image data, or may be two-dimensional face image data.
  • the target face image data is two-dimensional face image data, since the process of generating the target image data is implemented on the basis of the three-dimensional model, considering the real light, the shadow, and the like, the final lined target Face image data is more realistic.
  • step S208 to step S210 one or more of the steps may be selected to be performed simultaneously.
  • the final output target face image data is further increased. Real effect.
  • the candidate skin color data of the candidate face image data and the material skin color data of the material image data are used to perform facial skin color fusion on the target human face three-dimensional mesh.
  • the generating the merged target face image data may include the following steps, as shown in FIG. 7:
  • the candidate skin color data may be a set of candidate pixel points constituting the candidate face image data
  • the material skin color data may be a set of material pixel points constituting the material face image data.
  • the image fusion device may acquire candidate pixel points in the candidate skin color data and material pixel points in the material skin color data.
  • the specific process of the image fusion device based on the candidate pixel points and the material pixel points and the target degree image data is obtained by using the degree of fusion.
  • the specific process of the image fusion device based on the candidate pixel points and the material pixel points and the target degree image data is obtained by using the degree of fusion.
  • the accuracy of skin color fusion of the target face image data is improved by adopting a skin color fusion process accurate to a pixel point.
  • the source face image data of the current image to be fused and the material configuration information of the current material to be fused are acquired, wherein the material configuration information includes the material face image data, the material skin color data, and the material face three-dimensional grid.
  • performing image recognition processing on the source face image data acquiring the source face feature points corresponding to the source face image data, and generating a source face 3D mesh of the source face image data according to the source face feature points, and then adopting
  • the material face 3D mesh and the source face 3D mesh are meshed to generate a target face 3D mesh, and finally the source skin color data of the source face image data and the material skin color data of the material face image data are used to target the face.
  • the mesh merges the face skin color to generate the merged target face image data.
  • the 3D mesh is merged into the target face 3D mesh, and the target face 3D mesh is merged to generate the target face image data.
  • Improve the authenticity of the final target image data obtained by analyzing the role of the average skin color data in the skin color fusion of the target face image data, and increase the authenticity of the finally obtained target face image data;
  • the face image data is added with real lighting effects, the display position of the face image data is adjusted, and the face area of the face is filled, which further increases the real effect of the final output face image data; by using skin color fusion to the pixel
  • the process improves the accuracy of skin color fusion of the target face image data.
  • the face fusion algorithm of the prior art can obtain less information about the user's face, and less user face information will affect the final matching result when performing face fusion, resulting in generation.
  • the target result image is less authentic.
  • the embodiment of the present application provides a schematic flowchart of another image fusion method. As shown in FIG. 8, the method in the embodiment of the present application may include the following steps S401 to S404.
  • the image fusion device may obtain the source face image data of the current image to be merged and the material configuration information of the current material to be fused.
  • the image fusion device may obtain the source face image data of the current image to be merged and the material configuration information of the current material to be fused.
  • S402. Perform image recognition processing on the source face image data, acquire source face feature points corresponding to the source face image data, and generate a source of the source face image data according to the source face feature points. Human face 3D mesh.
  • step S202 the process of generating the three-dimensional mesh of the source face by the image fusion device may be referred to the related description in the foregoing step S202 to step S203, and details are not described herein again.
  • the scales of the source face three-dimensional mesh and the material face three-dimensional network are also in the same Under the scale.
  • the material face three-dimensional mesh may be similar to the source face three-dimensional mesh may be a 3D face mesh model corresponding to a face in the material face image data, which may be FIG. 4
  • the illustrated 3D face mesh or the 3D face mesh based on the 3D face mesh shown in FIG. 4 changes the backward 3D face mesh according to the Euler angle in the world coordinate system.
  • the target human face three-dimensional mesh may be a 3D face mesh of the target face image data.
  • the material face three-dimensional mesh is the 3D face mesh shown in FIG. 4
  • the final projection onto the 2D image can present the positive face effect
  • the material face three-dimensional mesh is based on FIG. 4
  • the final projection onto the 2D image can present the effect of the side face.
  • the source human face three-dimensional mesh and the material face three-dimensional mesh can generate the same target person with better facial features according to the fusion algorithm only when the types are the same.
  • Face 3D mesh the type consistency may be that the source human face three-dimensional mesh is consistent with the network element in the material face three-dimensional mesh, and the mesh element may be the source human face three-dimensional mesh.
  • a mesh orientation or a grid display area of the material face three-dimensional mesh is consistent with the face display area indicated by the material face three-dimensional grid, both of which are similar to the standard front face shown in FIG. 5A or similar In the side face shown in FIG.
  • the image fusion device may first perform mesh replenishment on the source face mesh.
  • the image fusion device may perform three-dimensionality on the source face according to the symmetry of the source face image data.
  • the grid is meshed to generate a three-dimensional grid of candidate faces that is consistent with the three-dimensional mesh type of the material face.
  • the face of the person is symmetrical (ignoring subtle differences)
  • the image fusion device extracts the The source face 3D mesh generated after the source face feature point is also directed to the side face.
  • the image fusion device can supplement the source face image data as a standard face according to the principle of face symmetry (for example, The image of FIG.
  • the face 3D mesh is the candidate face 3D mesh).
  • the image fusion device is consistent with the left and right facial expressions of the standard face image data obtained by supplementing the source face image data (side face image data) according to the principle of facial symmetry, and further obtaining a candidate face
  • the left and right facial expressions of the three-dimensional grid are also consistent.
  • the material face three-dimensional grid is a standard front face with inconsistent left and right facial expressions
  • the source face image data is side face image data
  • the left and right facial expressions of the candidate face 3D mesh obtained by the source face 3D mesh are inconsistent with the left and right facial expressions of the material face 3D mesh (for example, the material face image data indicates The left eye of the material face is open, the right eye is closed, and the left eye of the user's face indicated by the source face image data containing only the left face is open, at this time, according to the source
  • the right eye of the source face 3D mesh is also opened, which is inconsistent with the right eye of the material face 3D mesh.
  • the image fusion device will not be able to adopt the candidate.
  • the face 3D mesh and the above-mentioned material face 3D mesh are mesh-fused to generate a target face 3D mesh.
  • the image fusion device can adjust the left and right facial expressions of the candidate face 3D mesh to the expression of the 3D mesh of the material face through the expression migration algorithm, so that the final candidate face 3D mesh is obtained. It is consistent with the facial expression of the three-dimensional mesh of the above-mentioned material face.
  • the image fusion device may perform mesh fusion using the candidate face three-dimensional mesh and the material face three-dimensional mesh to generate a target human face three-dimensional mesh.
  • the candidate face three-dimensional mesh is consistent with the type of the material face three-dimensional mesh, that is, the material face three-dimensional mesh and the candidate face at the same face position.
  • the 3D mesh has corresponding feature points.
  • the candidate face three-dimensional grid contains feature points of two eye corners, and the material face three-dimensional grid also contains feature points of the two eye corners.
  • the target human face three-dimensional mesh may be a 3D face mesh of the target face image data.
  • the image fusion device may calculate a target facial feature point of the target human face according to the candidate face three-dimensional mesh and the facial feature point in the material face three-dimensional mesh, and then according to The calculated target face feature points generate a target face three-dimensional mesh.
  • a 3D candidate face 3D mesh ie, a user's 3D face mesh
  • the face feature points are marked in blue, and the average point of each corresponding point of the 1000 face feature points of the user and the material (corresponding point of the same position) The average value, a total of 1000 pairs of points is marked as red, and the resulting 1000 red face feature points are the target face feature points.
  • the above 1000 red face feature points can form more than 1900 triangles, corresponding to The three-dimensional mesh of the face depicted by more than 1900 triangular patches is the target human face three-dimensional mesh.
  • the image fusion device may adopt a moving least squares image deformation method MLS, affine transformation, image distortion, and the like to align the facial features of the source face image data and the material face image data with the red color.
  • the face feature point is the facial features indicated by the target face feature points, and the purpose of face fusion is achieved.
  • S404 Perform skin color fusion on the target face mesh by using the source skin color data of the source face image data and the material skin color data of the material face image data to generate the merged target face image data.
  • the process of generating the target facial image data by the image fusion device may be referred to the related description in the foregoing steps S205 to S210, and details are not described herein again.
  • the source face image data of the current image to be fused and the material configuration information of the current material to be fused are acquired, and then the image recognition process is performed on the source face image data to obtain a source corresponding to the source face image data.
  • a face feature point and generating a source face 3D mesh of the source face image data according to the source face feature point, and then detecting that the source face 3D mesh is inconsistent with the material face 3D mesh type And replenishing the source face three-dimensional mesh according to the symmetry of the source face image data, generating a candidate face three-dimensional mesh that is consistent with the material face three-dimensional mesh type, and adopting the The candidate face three-dimensional mesh and the material face three-dimensional mesh are mesh-fused to generate a target human face three-dimensional mesh, and finally the source skin color data of the source face image data and the material skin color data of the material face image data are used.
  • the target face mesh is used to merge the face skin color to generate the merged target face image data, even if there is a face angle of the user image data and a face angle of the material image data. Positive match, but also through the establishment of a three-dimensional picture of the user, in order to calculate the user's head the whole picture, to better integration of the three-dimensional aspects of the source data and facial image fusion material according to the user the whole picture.
  • the material face image data in the foregoing method embodiment is one frame material data in the related terminal application, and the entire face fusion process is applicable to real-time processing of a single frame, for example, it can be used for real-time preview of user fusion into another material image. After the look.
  • the material needs to process multiple frames, it is also a frame-by-frame loop processing video in the multi-frame picture to obtain the final face-fused video.
  • the system is, for example, a system for retouching terminal applications.
  • the user opens the retouching terminal on the image fusion device and enters the home page of the retouching terminal application.
  • the homepage of the retouching terminal application may have a prompt to prompt the user to select the material.
  • the image fusion device can acquire a certain frame (for example, the Nth frame) of the retouching terminal application (for example, a certain picture show, a certain P picture, a certain camera, etc.) according to the material selected by the user. Material configuration information for the material. Wherein, when the acquired material has a total of M frames, the value range of N is M ⁇ N ⁇ 1, and N and M are positive integers.
  • the material configuration information may include 3D avatar information of the current material to be fused (for example, may be an obj format file, and the file may include material face image data, material skin color data, and material face 3D mesh, etc.
  • Information indicating the material face related data may include the orientation of the material 3D avatar in the world coordinate system (the Euler angle pitch, yaw, roll), the center position (final The resulting position of the image, the scale information and the matching camera information (such as using the perspective matrix to portray this information), 2D stickers and 3D stickers, and the fusion of the user's face and the material face.
  • Degree alpha the degree of fusion can be the same for each frame or different for each frame).
  • the retouching terminal application can guide the user to take a selfie or video, or select a photo or video from the photo album. This process can be done before or after the user selects the material.
  • the image fusion device may acquire source face image data in a photo or video currently captured by the user or selected from the album.
  • the size of the material face image data and the source face image data used to generate the target face image data need to correspond to the same scale, but the face concavity, fat and thin, etc. The situation may not be completely consistent. Therefore, after the source face image data and the material face image data are acquired, the size of the source face image data may be adjusted according to the size of the material face image data. Make the size of the two correspond to the same scale.
  • the image fusion device acquires the material face image data and the source face image data
  • the image may be further analyzed to generate a target human face three-dimensional mesh.
  • the target human face three-dimensional mesh may be a face three-dimensional mesh model corresponding to a face in the target face image data generated by the final face fusion, and the specific implementation process may be implemented in steps S5021-S5025. :
  • the image fusion device may perform image recognition processing on the source face image data, acquire a reference feature point of the source face image data, and then perform depth information extraction on the reference feature point to obtain the Source face feature points corresponding to the reference feature points.
  • image recognition processing on the source face image data
  • acquire a reference feature point of the source face image data and then perform depth information extraction on the reference feature point to obtain the Source face feature points corresponding to the reference feature points.
  • the image fusion device may connect the source face feature points to a source face mesh composed of a plurality of triangular patches.
  • a source face mesh composed of a plurality of triangular patches.
  • the image fusion device has acquired the material configuration information of the Nth frame material before performing the face fusion. At this time, the image fusion device can read the related data in the material configuration information. .
  • the source face three-dimensional mesh and the material face three-dimensional mesh type are inconsistent, (for example, the source face three-dimensional mesh is a side face similar to that shown in FIG. 5B, the material face
  • the three-dimensional grid is similar to the standard front face shown in FIG. 5A, and the image fusion device may first perform mesh replenishment on the source human face three-dimensional mesh.
  • the specific implementation process refer to the person according to step S204.
  • the mesh complement of face symmetry is not repeated here.
  • the material configuration information acquired in step S5023 includes a three-dimensional mesh of the material face
  • the image fusion device may perform the three-dimensional mesh of the source face and the three-dimensional mesh of the material face.
  • the grid is merged to generate a three-dimensional mesh of the target face.
  • the image fusion device needs to fill the skin color on the different triangular patches in the target human face three-dimensional mesh to obtain the final target facial image data.
  • the specific process of skin color fusion of the target face image data can be implemented by steps S5031-S5033:
  • the image fusion device may perform the skin color equalization process on the source face image data, and obtain the average skin color data of the source face image data.
  • the image fusion device may perform the skin color equalization process on the source face image data, and obtain the average skin color data of the source face image data.
  • the image fusion device may perform skin color filling on the candidate face three-dimensional mesh based on the source skin color data of the source face image data and the average skin color data, and generate candidate face image data, and then adopt the
  • the candidate skin color data of the candidate face image data and the material skin color data of the material image data are used to fuse the face color of the target face three-dimensional mesh to generate the merged target face image data, and the specific implementation process may be
  • step S206 and step S207 and details are not described herein again.
  • the post-processing may be a post-adjustment of the generated target face image data, so that the final output target face image data effect is more realistic.
  • the processing procedure described in step S5041 and step S5042 may be included.
  • the image fusion device may use the candidate skin color data of the candidate face image data and the material skin color data of the material image data to perform facial skin color fusion on the target human face three-dimensional mesh to generate a merged Target face image data. Meanwhile, the image fusion device may adjust a current display position of the target face image data based on coordinate information indicated by the material face image data.
  • step S208 and step S209 For a detailed implementation process, refer to the detailed description of step S208 and step S209, and details are not described herein again.
  • the face is filled with edges.
  • the image fusion device may divide the first display area from the first The part of the display area is subjected to face edge filling processing.
  • face edge filling processing For a specific implementation process, refer to the detailed description of step S210, and details are not described herein again.
  • the final obtained target facial image data may be output and displayed.
  • the target face image data can be saved into a picture file of a specified format and output to the user.
  • the image fusion device detects whether all the M frames in the acquired material have been processed through the foregoing steps S501-S505. If the image fusion device detects that all the M frames in the acquired material have been processed, the target face image data may be saved as a video file of a specified format and applied in the retouching terminal. The final fused video is presented on the interface. If not processed, the target face image data finally obtained in the Nth frame is written into the video file, and the foregoing steps S501-S505 are performed on the next frame N+1 frame of the N frame, according to which Loop through to get the final fused video.
  • the source face image data of the current image to be fused and the material configuration information of the current material to be fused are acquired, wherein the material configuration information includes a material face image data, a material skin color data, and a material face three-dimensional mesh.
  • the image recognition processing is performed on the source face image data, the source face feature points corresponding to the source face image data are acquired, and the source face image 3D mesh is generated according to the source face feature points, and then the material is used.
  • the face 3D mesh and the source face 3D mesh are meshed to generate the target face 3D mesh, and finally the source skin color data of the source face image data and the material skin color data of the material face image data are used to target the face network.
  • the face is blended with the face color to generate the merged target face image data.
  • the 3D mesh is merged into the target face 3D mesh, and the target face 3D mesh is merged to generate the target face image data.
  • Improve the authenticity of the final target image data obtained by analyzing the role of the average skin color data in the skin color fusion of the target face image data, and increase the authenticity of the finally obtained target face image data;
  • the face image data is added with real lighting effects, the display position of the face image data is adjusted, and the face area of the face is filled, which further increases the real effect of the final output face image data; by using skin color fusion to the pixel
  • the process improves the accuracy of skin color fusion of the target face image data.
  • FIG. 10 is a schematic structural diagram of an image fusion device according to an embodiment of the present application.
  • the image fusion device 1 of the embodiment of the present application may include: a data acquisition module 11, a source mesh generation module 12, a target mesh generation module 13, and a target data generation module 14.
  • the data acquisition module 11 is configured to acquire source face image data of the current image to be fused and material configuration information of the current material to be fused.
  • the image fusion device 1 can acquire the source face image data of the current image to be fused and the material configuration information of the current material to be fused.
  • the source face image data may be face image data in a photo or video that the user currently photographs through the image fusion device 1 or selected from the album of the image fusion device 1.
  • the current material to be fused may be a material model used for retouching currently selected by the user in a retouching terminal application (for example, a certain picture show, a certain P picture, a certain camera, etc.), for example, an animated character image. , star photos, etc.
  • the material configuration information may include 3D avatar information of the current material to be fused (for example, may be a file in an obj format, and the file may include a representation of the material face image data, the material skin color data, and the material face three-dimensional grid.
  • Information about the material face related data may include the orientation of the material 3D avatar in the world coordinate system (Euler angle pitch, yaw, roll), center position (final result)
  • the specified position of the image, the scale information and the matching camera information (such as using the perspective matrix to portray this information), 2D stickers and 3D stickers, and the degree of fusion of the user's face and the material face.
  • Alpha the degree of fusion can be the same for each frame or different for each frame).
  • the size of the material face image data and the source face image data used to generate the target face image data need to correspond to the same scale, but the face concavity, fat and thin, etc. The situation may not be completely consistent. Therefore, after the source face image data and the material face image data are acquired, the size of the source face image data may be adjusted according to the size of the material face image data. Make the size of the two correspond to the same scale.
  • the source mesh generating module 12 is configured to perform image recognition processing on the source face image data, acquire source face feature points corresponding to the source face image data, and generate the source face feature points according to the source face feature points.
  • the source face 3D mesh of the source face image data is configured to perform image recognition processing on the source face image data, acquire source face feature points corresponding to the source face image data, and generate the source face feature points according to the source face feature points.
  • the source mesh generation module 12 may perform image recognition processing on the source face image data, and acquire source face feature points corresponding to the source face image data.
  • the image recognition process may be to identify a user face in a photo by using a face detection technology (for example, a face detection provided by the cross-platform computer vision library OpenCV, a new visual service platform Face++, a U-shaped face detection, etc.)
  • the process of facial features positioning, the source facial feature points may be data points capable of characterizing facial features of the source facial image data (eg, facial contours, eye contours, nose, lips, etc.).
  • the source mesh generation module 12 may perform image recognition processing on the source face image data (for example, the user face in the photo may be identified and facial features may be positioned to obtain a certain number of benchmarks. Feature points), acquiring reference feature points of the source face image data, performing three-dimensional depth information extraction on the reference feature points, and acquiring source face feature points corresponding to the reference feature points.
  • the three-dimensional depth information extraction may be based on the reference feature points described above, and the process of reflecting the feature points of the source face image data in the three-dimensional model by estimating the facial feature points of the standard three-dimensional model.
  • the reference feature point may be a reference point indicating a facial feature, for example, a facial contour, an eye contour, a nose, a lip, etc., may be 83 reference points, or may be 68 reference points as shown in FIG.
  • the specific points can be determined by the developer according to the needs.
  • the source face feature point may be a feature point capable of corresponding to the three-dimensional model of the source face image data after further deepening on the basis of the reference feature point, for example, by using three-dimensional depth information on the 68 or 83 reference points.
  • the extraction may obtain 1000 deepened source facial feature points, which may be the vertices of each triangular patch shown in FIG. 4.
  • the source mesh generation module 12 may generate a source face three-dimensional mesh of the source face image data according to the source face feature point.
  • the source human face three-dimensional mesh may be a 3D face mesh model corresponding to the face in the source face image data, such as the 3D face mesh shown in FIG. 4 or only the half face similar to FIG. 3D face mesh.
  • a target mesh generation module 13 configured to perform mesh fusion using the material face three-dimensional mesh and the source face three-dimensional mesh to generate a target human face three-dimensional mesh
  • the scales of the source face three-dimensional mesh and the material face three-dimensional network are also in the same Under the scale.
  • the target mesh generation module 13 may perform mesh fusion using the material face three-dimensional mesh and the source face three-dimensional mesh to generate a target human face three-dimensional mesh.
  • the material face 3D mesh may be similar to the source face 3D mesh may be a 3D face mesh model corresponding to the face in the material face image data, and may be the 3D face network shown in FIG. 4 .
  • the grid is based on the 3D face mesh shown in FIG. 4 to change the backward 3D face mesh according to the Euler angle in the world coordinate system.
  • the target human face three-dimensional mesh may be a 3D face mesh of the target face image data. According to the embodiment of the present application, when the material face three-dimensional mesh is the 3D face mesh shown in FIG.
  • the final projection onto the 2D image can present the positive face effect, and when the material face three-dimensional mesh is based on FIG. 4
  • the 3D face mesh shown changes the backward 3D face mesh according to the Euler angle in the world coordinate system
  • the final projection onto the 2D image can present the effect of the side face.
  • the target mesh generation module 13 may calculate a target facial feature point of the target face according to the source face three-dimensional mesh and the face feature points in the material face three-dimensional mesh. Then, a target human face three-dimensional mesh is generated according to the calculated target facial feature points.
  • a 3D source face 3D mesh ie, a user's 3D face mesh
  • 3D material face 3D mesh has 1000 depth information.
  • the material face feature points are marked in blue, and the average points of the corresponding points of the 1000 face feature points of the user and the material (average points of the same position are averaged, a total of 1000 point pairs) are marked in red, and finally generated
  • the 1000 red face feature points are the target face feature points, and the above 1000 red face feature points can form more than 1900 triangles, and the corresponding three-dimensional grid of the face depicted by the corresponding 1900 triangle faces is
  • the target face is a three-dimensional grid.
  • the image fusion device 1 may adopt an MLS method, an affine transformation, an image distortion, or the like to converge the facial features of the source face image data and the material face image data to the red face feature point.
  • the facial features indicated by the target face feature points achieve the purpose of face fusion.
  • the target data generating module 14 is configured to perform facial skin color fusion on the target human face three-dimensional mesh by using the source skin color data of the source face image data and the material skin color data of the material image data to generate a merged target. Face image data.
  • the image fusion device 1 needs to fill the skin color on different triangular patches in the target human face three-dimensional mesh to obtain the final target facial image data.
  • the target data generating module 14 may perform facial skin color fusion on the target human face three-dimensional mesh by using the source skin color data of the source face image data and the material skin color data of the material image data to generate a fusion image.
  • Target face image data may be a set of source pixel points constituting the source face image data
  • the material skin color data may be a set of material pixel points constituting the material face image data.
  • the source face 3D mesh needs to be network supplemented to generate a candidate face 3D mesh.
  • the type may be the source face three-dimensional mesh and the network elements in the material face three-dimensional grid.
  • the candidate skin color data of the candidate face three-dimensional mesh may be the symmetric source face image data, that is, the candidate skin color data of the candidate face three-dimensional mesh may be considered as the source skin color data.
  • the target data generating module 14 may be based on the degree of fusion in the source pixel point, the material pixel point, and the material configuration information (may be a fusion value set according to an empirical value, and usually takes a value between 0-1) Calculating a target pixel point of the target face image data, and further generating the target face image data by filling the target face three-dimensional mesh according to the target pixel point.
  • the pixels of a feature point on a triangular patch in the three-dimensional grid of the source face are: UserB, UserG, and UserR
  • the pixel of the face feature point at the corresponding position on the corresponding triangular patch in the material face mesh is: ResourceB.
  • the pixels of the feature points at the corresponding positions on the corresponding triangular patches in the target human face 3D mesh are: TargetB, TargetG and TargetR, and the degree of fusion in the material configuration information is: alpha, then:
  • TargetB (1.0–alpha)*UserB+alpha*ResourceB
  • TargetG (1.0–alpha)*UserG+alpha*ResourceG
  • TargetR (1.0–alpha)*UserR+alpha*ResourceR
  • each pixel value of the target face image data can be obtained, and the target face image data is obtained.
  • the target face image data finally generated may be three-dimensional face image data or two-dimensional face image data.
  • the target face image data is two-dimensional face image data
  • the process of generating the target image data is implemented on the basis of the three-dimensional model, the target person is finally formed in consideration of real light, shadow, and the like. Face image data is more realistic.
  • the candidate skin color data of the candidate face three-dimensional mesh may include two parts, that is, the skin color data of the candidate face three-dimensional mesh matching the source face three-dimensional mesh is the source skin color Data, the skin color data of the candidate face three-dimensional mesh that does not match the source face three-dimensional mesh is the average skin color data, and the average skin color data may be the skin color data of the source skin color data after the skin color equalization processing .
  • the skin color equalization processing may be a process of removing the average value of the skin color by removing effects such as shadows caused by light or the like in the source face image data.
  • the target data generating module 14 may calculate a target pixel point of the target face image data according to the candidate pixel point of the candidate skin color data, the material pixel point, and the degree of fusion in the material configuration information, and further fill according to the target pixel point.
  • the target human face three-dimensional mesh generates target face image data, and the specific calculation process is consistent with the above calculation process, and details are not described herein again.
  • the source face image data of the current image to be fused and the material configuration information of the current material to be fused are acquired, wherein the material configuration information includes the material face image data, the material skin color data, and the material face three-dimensional grid.
  • performing image recognition processing on the source face image data acquiring the source face feature points corresponding to the source face image data, and generating a source face 3D mesh of the source face image data according to the source face feature points, and then adopting
  • the material face 3D mesh and the source face 3D mesh are meshed to generate a target face 3D mesh, and finally the source skin color data of the source face image data and the material skin color data of the material face image data are used to target the face.
  • the mesh merges the face skin color to generate the merged target face image data.
  • the 3D mesh is merged into the target face 3D mesh, and the target face 3D mesh is merged to generate the target face image data. Improve the authenticity of the final target image data obtained.
  • FIG. 11 is a schematic structural diagram of another image fusion device according to an embodiment of the present application.
  • the image fusion device 1 of the embodiment of the present application may include: a data acquisition module 11 , a source mesh generation module 12 , a target mesh generation module 13 , a target data generation module 14 , and an effect addition module 15 .
  • the data acquisition module 11 is configured to acquire source face image data of the current image to be fused and material configuration information of the current material to be fused.
  • the data acquiring unit 11 may acquire the source face image data of the current image to be fused and the material configuration information of the current material to be fused.
  • the source face image data may be face image data in a photo or video that the user currently photographs through the image fusion device or selected from the album of the image fusion device.
  • the current material to be fused may be a material model used for retouching currently selected by the user in a retouching terminal application (for example, a certain picture show, a certain P picture, a certain camera, etc.), for example, an animated character image. , star photos, etc.
  • the material configuration information may include 3D avatar information of the current material to be fused (for example, may be a file in an obj format, and the file may include a representation of the material face image data, the material skin color data, and the material face three-dimensional grid.
  • Information about the material face related data may include the orientation of the material 3D avatar in the world coordinate system (Euler angle pitch, yaw, roll), center position (final result)
  • the specified position of the image, the scale information and the matching camera information (such as using the perspective matrix to portray this information), 2D stickers and 3D stickers, and the degree of fusion of the user's face and the material face.
  • Alpha the degree of fusion can be the same for each frame or different for each frame).
  • the size of the material face image data and the source face image data used to generate the target face image data need to correspond to the same scale, but the face concavity, fat and thin, etc. The situation may not be completely consistent. Therefore, after the source face image data and the material face image data are acquired, the size of the source face image data may be adjusted according to the size of the material face image data. Make the size of the two correspond to the same scale.
  • the source mesh generating module 12 is configured to perform image recognition processing on the source face image data, acquire source face feature points corresponding to the source face image data, and generate the source face feature points according to the source face feature points.
  • the source face 3D mesh of the source face image data is configured to perform image recognition processing on the source face image data, acquire source face feature points corresponding to the source face image data, and generate the source face feature points according to the source face feature points.
  • the source mesh generation module 12 may perform image recognition processing on the source face image data, acquire source face feature points corresponding to the source face image data, and according to the source face feature A point generates a three-dimensional mesh of the source face of the source face image data.
  • the source mesh generation module 12 may include:
  • the feature point acquiring unit 121 is configured to perform image recognition processing on the source face image data, and acquire a reference feature point of the source face image data.
  • the feature point acquiring unit 121 may perform image recognition processing on the source face image data to acquire a reference feature point of the source face image data.
  • the image recognition process may be to identify a user face in a photo by using a face detection technology (for example, a face detection provided by the cross-platform computer vision library OpenCV, a new visual service platform Face++, a U-shaped face detection, etc.) The process of positioning the five senses.
  • the reference feature point may be a reference point indicating a facial feature, for example, a facial contour, an eye contour, a nose, a lip, etc., may be 83 reference points, or may be 68 reference points as shown in FIG.
  • the specific points can be determined by the developer according to the needs.
  • the source mesh generating unit 122 is configured to perform three-dimensional depth information extraction on the reference feature point, acquire a source face feature point corresponding to the reference feature point, and generate a source face three-dimensional mesh according to the source face feature point. .
  • the source mesh generating unit 122 may perform three-dimensional depth information extraction on the reference feature point, acquire a source face feature point corresponding to the reference feature point, and generate a source face three-dimensional according to the source face feature point. grid.
  • the three-dimensional depth information extraction may be based on the reference feature points described above, and the process of reflecting the feature points of the source face image data in the three-dimensional model is calculated by matching the facial feature points of the standard three-dimensional model.
  • the source face feature point may be a point further deepened on the basis of the reference feature point. For example, 1000 deepened source faces may be obtained by extracting the three-dimensional depth information of the 68 or 83 reference points.
  • the feature points may be the vertices of the respective triangular patches in FIG.
  • the source face 3D mesh may be a 3D face mesh model corresponding to the face in the source face image data, and may be a 3D mesh of the user face connected by the source face feature points.
  • the model such as the 3D face mesh shown in Figure 4 or a 3D face mesh with only a half face similar to Figure 4.
  • the target mesh generation module 13 is configured to perform mesh fusion using the material face three-dimensional mesh and the source face three-dimensional mesh to generate a target human face three-dimensional mesh.
  • the scales of the source face three-dimensional mesh and the material face three-dimensional network are also in the same Under the scale.
  • the target mesh generation module 13 may perform mesh fusion using the material face three-dimensional mesh and the source face three-dimensional mesh to generate a target human face three-dimensional mesh.
  • the material face three-dimensional mesh may be similar to the source face three-dimensional mesh may be a 3D face mesh model corresponding to a face in the material face image data, which may be FIG. 4
  • the illustrated 3D face mesh or the 3D face mesh based on the 3D face mesh shown in FIG. 4 changes the backward 3D face mesh according to the Euler angle in the world coordinate system.
  • the target human face three-dimensional mesh may be a 3D face mesh of the target face image data.
  • the final projection onto the 2D image can present the positive face effect
  • the material face three-dimensional mesh is based on FIG. 4
  • the 3D face mesh shown changes the backward 3D face mesh according to the Euler angle in the world coordinate system
  • the final projection onto the 2D image can present the effect of the side face.
  • the target mesh generation module 13 may calculate a target facial feature point of the target face according to the source face three-dimensional mesh and the face feature points in the material face three-dimensional mesh. And generating a target human face three-dimensional mesh according to the calculated target facial feature points.
  • the 3D source human face three-dimensional mesh ie, the user's 3D face mesh
  • the face feature points are marked in blue, and the average point of each corresponding point of the 1000 face feature points of the user and the material (the same position)
  • the corresponding points are averaged, a total of 1000 pairs of points are marked in red, and the resulting 1000 red face feature points are the target face feature points, and the above 1000 red face feature points can form more than 1900 triangles.
  • the three-dimensional grid of the face depicted by the corresponding 1900 triangle faces is the target face three-dimensional grid.
  • the image fusion device 1 may adopt an MLS method, an affine transformation, an image distortion, or the like to converge the facial features of the source face image data and the material face image data to the red face feature point.
  • the facial features indicated by the target face feature points achieve the purpose of face fusion.
  • the target data generating module 14 is configured to perform facial skin color fusion on the target human face three-dimensional mesh by using the source skin color data of the source face image data and the material skin color data of the material image data to generate a merged target. Face image data.
  • the target data generating module 14 may perform facial skin color fusion on the target human face three-dimensional mesh by using the source skin color data of the source face image data and the material skin color data of the material image data.
  • the merged target face image data may be performed by using the source skin color data of the source face image data and the material skin color data of the material image data.
  • FIG. 13 a schematic diagram of a structure of a target data generating module according to an embodiment of the present application.
  • the target data generating module 14 may include:
  • the skin color data acquiring unit 141 is configured to perform skin color equalization processing on the source face image data, and acquire average skin color data of the source face image data.
  • the source face 3D mesh needs to be network supplemented to generate a candidate face 3D mesh.
  • the type may be the source face three-dimensional mesh and the network elements in the material face three-dimensional grid.
  • the candidate skin color data of the candidate face three-dimensional mesh may be the symmetric source face image data, that is, the candidate skin color data of the candidate face three-dimensional mesh may be considered as the source skin color data.
  • the last generated candidate face image data is the face image in FIG. 5A, and the skin color of the face in FIG. 5A coincides with the skin color of the face in FIG. 5B.
  • the candidate skin color data of the candidate face three-dimensional mesh may include two parts, that is, the skin color data of the candidate face three-dimensional mesh matching the source face three-dimensional mesh is the source skin color Data, the skin color data of the candidate face three-dimensional mesh that does not match the source face three-dimensional mesh is the average skin color data.
  • the skin color data acquiring unit 141 may perform skin color equalization processing on the source face image data to acquire average skin color data of the source face image data.
  • the skin color equalization processing may be a process of removing the average value of the skin color by removing effects such as shadows due to light or the like in the source face image data.
  • the average skin color data may be a set of pixel points composed of an average value of the pixel skin data after the source skin color data is removed.
  • the candidate data generating unit 142 is configured to perform skin color filling on the candidate face three-dimensional mesh based on the source skin color data of the source face image data and the average skin color data to generate candidate face image data.
  • the candidate data generating unit 142 may perform skin color filling on the candidate face three-dimensional mesh based on the source skin color data of the source face image data and the average skin color data to generate candidate face image data.
  • the skin color data of the candidate face three-dimensional mesh matching the source human face three-dimensional mesh may be filled with the source skin color data of the source face image data, wherein the candidate face is in the three-dimensional mesh
  • the skin color data of the source face three-dimensional mesh unmatched portion may be filled with the average skin color data.
  • the candidate skin color data of the candidate face image data may include the source skin color data and the average skin color data, for example, the last generated candidate face image data is the face image in FIG. 5A, and the right side face in FIG. 5A
  • the skin color is the skin color of the face in FIG. 5B, and the skin color of the left face in FIG. 5A is the skin color after the skin color of the face in FIG. 5B is averaged.
  • the target data generating unit 143 is configured to perform facial skin color fusion on the target human face three-dimensional mesh by using the candidate skin color data of the candidate face image data and the material skin color data of the material image data to generate a merged target. Face image data.
  • the image fusion device 1 needs to fill the skin color on different triangular patches in the target human face three-dimensional mesh to obtain the final target facial image data.
  • the target data generating unit 143 may perform facial skin color fusion on the target human face three-dimensional mesh by using the candidate skin color data of the candidate face image data and the material skin color data of the material image data.
  • the candidate skin color data may be a set of candidate pixel points constituting the candidate face image data
  • the material skin color data may be a set of material pixel points constituting the material face image data.
  • the target data generating unit 143 may calculate a target pixel point based on the candidate pixel point and the material pixel point, and generate target pixel image data according to the target pixel point.
  • the degree of fusion (which may be a fusion value set according to an empirical value, usually between 0-1).
  • the pixels of a feature point on a triangular patch in the three-dimensional grid of the candidate face may be set as: CandidateB, CandidateG, and CandidateR, and the face features at corresponding positions on the corresponding triangular patch in the three-dimensional grid of the material face
  • the pixels of the point are: ResourceB, ResourceG, and ResourceR.
  • the pixels of the feature points at the corresponding positions on the corresponding triangular patches in the target human face are: TargetB, TargetG, and TargetR, and the degree of fusion in the material configuration information is: alpha, Then there are:
  • TargetB (1.0–alpha)*CandidateB+alpha*ResourceB
  • TargetG (1.0–alpha)*CandidateG+alpha*ResourceG
  • TargetR (1.0–alpha)*CandidateR+alpha*ResourceR
  • each pixel value of the target face image data can be obtained, and the target face image data is obtained.
  • the authenticity of the finally obtained target face image data is increased.
  • the effect adding module 15 is configured to acquire, according to source skin color data of the source face image data, a light source type corresponding to the source face image data, and use the light effect corresponding to the light source type to target the face image data. Perform effect addition processing.
  • the effect adding module 15 may acquire the light source type corresponding to the source face image data according to the source skin color data of the source face image data.
  • the effect adding module 15 may obtain a region of light and average skin color and a region of dark and average skin color by comparing the skin color of the average skin color and the color of each region in the source face image data, thereby deriving the light source type, for example, Multiple point sources or surface sources.
  • the effect adding module 15 can also collect the result map of the source face image data under different specified illumination conditions through deep learning, and then use the result map and the corresponding illumination condition as the training of Deep Neural Networks (DNN). Data, training a DNN model that outputs a source type and source position for a given picture.
  • DNN Deep Neural Networks
  • the effect adding module 15 may perform an effect adding process on the target face image data by using a lighting effect corresponding to the light source type, for example, the corresponding light source type of the source face image data is from the left side.
  • the point source of the face direction the image fusion device may add a lighting effect of the left face direction point source in the finally obtained target face image data.
  • the effect adding module 15 may further paste the 2D and 3D stickers in the material configuration information on the target face image data according to their corresponding positions and levels (zOrder), for example, The 3D glasses sticker is worn on the face of the person who generated the target face image data.
  • the position adjustment module 16 is configured to adjust a current display position of the target face image data based on the coordinate information indicated by the material face image data.
  • the position adjustment module 16 may adjust the current display position of the target face image data based on the coordinate information indicated by the material face image data, for example, according to The coordinate information (including the Euler direction and the center point) and the material size indicated by the material face image data in the material configuration information are placed at the specified position.
  • the edge filling module 17 is configured to: when the first display area of the target face image data is smaller than the second display area of the source face image data, the first display area of the second display area The part is subjected to face edge filling processing.
  • the edge filling module 17 may divide the second display area a portion of the first display area performs a face edge filling process, and the first display area may be a range in which the target face image data is mapped on a 2D screen, and the second display area may be the source face The image data is mapped on a range on the 2D screen, and the first display area is smaller than the second display area, and the face that can be expressed as the target face image data is smaller than the face of the source face image data.
  • the face edge filling process may be to fill a portion of the second display area except the first display area with a padding algorithm (for example, an image restoration algorithm Inpainting provided by OpenCV).
  • the image fusion device 1 processes the additional technical effects of the target facial image data
  • the final obtained target facial image data may be output and displayed.
  • the target face image data finally generated after the effect adding module 15, the position adjusting module 16, and the edge filling module 17 are executed may be three-dimensional face image data, or may be two-dimensional face image data.
  • the target face image data is two-dimensional face image data, since the process of generating the target image data is implemented on the basis of the three-dimensional model, considering the real light, the shadow, and the like, the final lined target Face image data is more realistic.
  • one or more of the modules may be selected to be simultaneously executed.
  • the final output target face image data is further increased. Real effect.
  • the target data generating unit shown in FIG. 14 may include:
  • the pixel point obtaining sub-unit 1431 is configured to acquire candidate pixel points in the candidate skin color data and material pixel points in the material skin color data.
  • the candidate skin color data may be a set of candidate pixel points constituting the candidate face image data
  • the material skin color data may be a set of material pixel points constituting the material face image data.
  • the pixel point obtaining sub-unit 1431 may acquire candidate pixel points in the candidate skin color data and material pixel points in the material skin color data.
  • the target data generating sub-unit 1432 is configured to calculate a target pixel point based on the candidate pixel point and the material pixel point, and generate target pixel image data according to the target pixel point.
  • the target data generating sub-unit 1432 is based on the candidate pixel point and the material pixel point, and the specific process of acquiring the target face image data by using the degree of fusion can be referred to in the description of the method embodiment, and details are not described herein again. .
  • the accuracy of the skin color fusion of the target face image data is improved by adopting the skin color fusion process accurate to the pixel points.
  • the source face image data of the current image to be fused and the material configuration information of the current material to be fused are acquired, wherein the material configuration information includes the material face image data, the material skin color data, and the material face three-dimensional mesh.
  • the image recognition processing is performed on the source face image data, the source face feature points corresponding to the source face image data are acquired, and the source face image 3D mesh is generated according to the source face feature points, and then the material is used.
  • the face 3D mesh and the source face 3D mesh are meshed to generate the target face 3D mesh, and finally the source skin color data of the source face image data and the material skin color data of the material face image data are used to target the face network.
  • the face is blended with the face color to generate the merged target face image data.
  • the 3D mesh is merged into the target face 3D mesh, and the target face 3D mesh is merged to generate the target face image data.
  • Improve the authenticity of the final target image data obtained by analyzing the role of the average skin color data in the skin color fusion of the target face image data, and increase the authenticity of the finally obtained target face image data;
  • the face image data is added with real lighting effects, the display position of the face image data is adjusted, and the face area of the face is filled, which further increases the real effect of the final output face image data; by using skin color fusion to the pixel
  • the process improves the accuracy of skin color fusion of the target face image data.
  • the face fusion algorithm of the prior art can obtain less information about the user's face, and less user face information will affect the final matching result when performing face fusion, resulting in generation.
  • the target result image is less authentic.
  • the embodiment of the present application provides a structure diagram of another image fusion device.
  • the structure diagram shown in FIG. 10 may include: a data acquisition module 11, a source mesh generation module 12, and a target mesh generation. Module 13 and target data generation module 14.
  • the data acquisition module 11 is configured to acquire source face image data of the current image to be fused and material configuration information of the current material to be fused.
  • the process of acquiring the source face image data and the material configuration information by the data obtaining module 11 can be referred to the specific description of the foregoing method embodiment, and details are not described herein again.
  • the source mesh generating module 12 is configured to perform image recognition processing on the source face image data, acquire source face feature points corresponding to the source face image data, and generate the source face feature points according to the source face feature points.
  • the source face 3D mesh of the source face image data is configured to perform image recognition processing on the source face image data, acquire source face feature points corresponding to the source face image data, and generate the source face feature points according to the source face feature points.
  • the process of generating the three-dimensional mesh of the source face by the source mesh generating module 12 can be referred to the specific description of the foregoing method embodiment, and details are not described herein again.
  • the target mesh generation module 13 is configured to: when detecting that the source face three-dimensional mesh is inconsistent with the type of the material face three-dimensional mesh, according to the symmetry of the source face image data, the source person The face 3D mesh is complemented by the mesh, and a candidate face 3D mesh is generated corresponding to the material face 3D mesh type, and the candidate face 3D mesh and the material face 3D mesh are used for the mesh.
  • the lattice fusion generates a three-dimensional mesh of the target face.
  • the target mesh generating module 13 may be based on the symmetry of the source face image data.
  • the source face three-dimensional mesh is complemented by a mesh, and a candidate face three-dimensional mesh conforming to the material face three-dimensional mesh type is generated, and the candidate face three-dimensional mesh and the material face three-dimensional mesh are adopted.
  • the mesh is meshed to generate a three-dimensional mesh of the target face.
  • the scales of the source face three-dimensional mesh and the material face three-dimensional network are also in the same Under the scale.
  • the material face three-dimensional mesh may be similar to the source face three-dimensional mesh may be a 3D face mesh model corresponding to a face in the material face image data, which may be FIG. 4
  • the illustrated 3D face mesh or the 3D face mesh based on the 3D face mesh shown in FIG. 4 changes the backward 3D face mesh according to the Euler angle in the world coordinate system.
  • the target human face three-dimensional mesh may be a 3D face mesh of the target face image data.
  • the material face 3D mesh is the 3D face mesh shown in Figure 4
  • the final projection onto the 2D image can present the effect of the face
  • the material face 3D mesh is based on the 3D face shown in Figure 4.
  • the mesh changes the backward 3D face mesh according to the Euler angle in the world coordinate system the final projection onto the 2D image can present the effect of the side face.
  • the source human face three-dimensional mesh and the material face three-dimensional mesh can generate the same target person with better facial features according to the fusion algorithm only when the types are the same.
  • Face 3D mesh The type consistency may be that the source human face three-dimensional mesh is consistent with the network element in the material face three-dimensional mesh, and the mesh element may be the source human face three-dimensional mesh and the material face The mesh orientation of the 3D mesh or the grid display area.
  • the face display area indicated by the source face three-dimensional grid is consistent with the face display area indicated by the material face three-dimensional grid, both of which are similar to the standard front face shown in FIG. 5A or similar In the side face shown in FIG.
  • the image fusion device 1 may first perform mesh supplementation on the source face mesh.
  • the target mesh generating module 13 may perform the symmetry according to the source face image data.
  • the source face 3D mesh is mesh-added to generate a candidate face 3D mesh that is consistent with the material face 3D mesh type.
  • the face of the person is symmetrical (ignoring subtle differences)
  • the source face image data is side face image data (for example, the image in FIG. 5B)
  • the image fusion device extracts the The source face 3D mesh generated after the source face feature point is also for the side face.
  • the target mesh generation module 13 can supplement the source face image data as a standard face according to the principle of face symmetry. (For example, the image of FIG.
  • FIG. 5A is obtained by supplementing the image of FIG. 5B), and then generating a candidate face three-dimensional mesh conforming to the material face three-dimensional mesh type according to the supplemented source face image data (FIG. 5A)
  • the corresponding face 3D mesh is the candidate face 3D mesh).
  • the target mesh generation module 13 obtains the same facial expressions of the standard face image data obtained by supplementing the source face image data (side face image data) according to the principle of face symmetry, and further obtains The left and right facial expressions of the candidate face's three-dimensional grid are also consistent.
  • the material face three-dimensional grid is a standard front face with inconsistent left and right facial expressions
  • the source face image data is side face image data
  • the left and right facial expressions of the candidate face 3D mesh obtained by the source face 3D mesh are inconsistent with the left and right facial expressions of the material face 3D mesh (for example, the material face image data indicates The left eye of the material face is open, the right eye is closed, and the left eye of the user's face indicated by the source face image data containing only the left face is open, at this time, according to the source
  • the right eye of the source face 3D mesh is also expanded, which is inconsistent with the right eye of the material face 3D mesh.
  • the target mesh generation module 13 cannot be adopted.
  • the candidate face 3D mesh and the above-mentioned material face 3D mesh are mesh-fused to generate a target face 3D mesh.
  • the target mesh generation module 13 can adjust the left and right facial expressions of the candidate face three-dimensional mesh to the expression of the three-dimensional mesh of the material face through the expression migration algorithm, so that the final candidate face is obtained.
  • the three-dimensional grid is consistent with the facial expression of the three-dimensional mesh of the above-mentioned material face.
  • the target mesh generation module 13 may perform mesh fusion using the candidate face three-dimensional mesh and the material face three-dimensional mesh to generate a target human face three-dimensional mesh.
  • the candidate face three-dimensional mesh is consistent with the type of the material face three-dimensional mesh, that is, the material face three-dimensional mesh and the candidate face three-dimensional mesh have corresponding positions on the same facial position.
  • the feature points for example, the feature face of the candidate face includes two feature points of the eye corners of the eye, and the material face three-dimensional mesh also contains feature points of the eye corners of the two eyes.
  • the target human face three-dimensional mesh may be a 3D face mesh of the target face image data.
  • the target mesh generation module 13 may calculate a target facial feature point of the target face according to the candidate face three-dimensional mesh and the facial feature points in the material face three-dimensional mesh. And generating a target human face three-dimensional mesh according to the calculated target facial feature points.
  • the 3D candidate face three-dimensional mesh ie, the user's 3D face mesh
  • the point is marked in green, and there are 1000 pieces of material with depth information on the 3D surface of the 3D material.
  • the face feature points are marked in blue, and the average point of each corresponding point of the 1000 face feature points of the user and the material (the same position)
  • the corresponding points are averaged, a total of 1000 pairs of points are marked in red, and the resulting 1000 red face feature points are the target face feature points, and the above 1000 red face feature points can form more than 1900 triangles.
  • the three-dimensional grid of the face depicted by the corresponding 1900 triangle faces is the target face three-dimensional grid.
  • the image fusion device 1 may adopt a moving least squares image deformation method MLS, an affine transformation, an image distortion, and the like to converge the facial features of the source face image data and the material face image data.
  • the red facial feature point is the facial features indicated by the target facial feature points, and the purpose of facial fusion is achieved.
  • the target data generating module 14 is configured to perform facial skin color fusion on the target human face mesh by using the source skin color data of the source face image data and the material skin color data of the material face image data to generate a merged Target face image data;
  • the process of generating the target facial image data by the target data generating module 14 can be referred to the specific description of the foregoing method embodiment, and details are not described herein again.
  • the embodiment of the present application further provides a computer storage medium, where the computer storage medium may store a plurality of instructions, the instructions being adapted to be loaded by a processor and executing the method steps of the embodiment shown in FIG. 2 to FIG. 9 above.
  • the computer storage medium may store a plurality of instructions, the instructions being adapted to be loaded by a processor and executing the method steps of the embodiment shown in FIG. 2 to FIG. 9 above.
  • FIG. 15 is a schematic structural diagram of a terminal according to an embodiment of the present application.
  • the terminal 1000 may include at least one processor 1001, such as a CPU, at least one network interface 1004, a user interface 1003, a memory 1005, and at least one communication bus 1002.
  • the communication bus 1002 is used to implement connection communication between these components.
  • the user interface 1003 can include a display and a keyboard.
  • the optional user interface 1003 can also include a standard wired interface and a wireless interface.
  • Network interface 1004 can include, for example, a standard wired interface, a wireless interface (such as a WI-FI interface).
  • the memory 1005 may be a high speed RAM memory or a non-volatile memory such as at least one disk memory.
  • the memory 1005 may also be, for example, at least one storage device located remotely from the aforementioned processor 1001. As shown in FIG. 15, an operating system, a network communication module, a user interface module, and a face fusion application may be included in the memory 1005 as a computer storage medium.
  • the user interface 1003 is mainly used to provide an input interface for the user to acquire data input by the user;
  • the network interface 1004 is used for data communication by the user terminal; and
  • the processor 1001 can be used to call the memory 1005.
  • the face-sharing application stored in the store and specifically do the following:
  • the target skin color is merged with the source skin color data of the source face image data and the material skin color data of the material face image data to generate the merged target face image data.
  • the processor 1001 performs image recognition processing on the source face image data, acquires a source face feature point corresponding to the source face image data, and according to the source face feature When the point generates the source face 3D mesh of the source face image data, the following operations are specifically performed:
  • Performing three-dimensional depth information extraction on the reference feature points acquiring source facial feature points corresponding to the reference feature points, and generating a source human face three-dimensional mesh according to the source facial feature points.
  • the processor 1001 performs the following operations when performing mesh fusion using the material face 3D mesh and the source face 3D mesh to generate a target face 3D mesh:
  • the source face three-dimensional mesh When detecting that the source face three-dimensional mesh is inconsistent with the type of the material face three-dimensional mesh, the source face three-dimensional mesh is mesh-added according to the symmetry of the source face image data, and generated. A candidate face three-dimensional mesh conforming to the material face mesh type, and performing mesh fusion using the candidate face three-dimensional mesh and the material face three-dimensional mesh to generate a target human face three-dimensional mesh.
  • the processor 1001 performs facial skin color fusion on the target human face three-dimensional mesh by performing source skin color data of the source face image data and material skin color data of the material image data.
  • the processor 1001 performs facial skin color fusion on the target human face three-dimensional mesh by performing source skin color data of the source face image data and material skin color data of the material image data.
  • the candidate skin color data includes the source skin color data and the average skin color data.
  • the processor 1001 performs facial skin color fusion on the target human face three-dimensional mesh by performing candidate skin color data of the candidate face image data and material skin color data of the material image data.
  • the processor 1001 performs facial skin color fusion on the target human face three-dimensional mesh by performing candidate skin color data of the candidate face image data and material skin color data of the material image data.
  • the processor 1001 is further configured to:
  • the processor 1001 is further configured to:
  • the processor 1001 is further configured to:
  • the face edge padding is performed on a portion of the second display area except the first display area deal with.
  • the source face image data of the current image to be fused and the material configuration information of the current material to be fused are acquired, wherein the material configuration information includes the material face image data, the material skin color data, and the material face three-dimensional grid.
  • performing image recognition processing on the source face image data acquiring the source face feature points corresponding to the source face image data, and generating a source face 3D mesh of the source face image data according to the source face feature points, and then adopting
  • the material face 3D mesh and the source face 3D mesh are meshed to generate a target face 3D mesh, and finally the source skin color data of the source face image data and the material skin color data of the material face image data are used to target the face.
  • the mesh merges the face skin color to generate the merged target face image data.
  • the 3D mesh is merged into the target face 3D mesh, and the target face 3D mesh is merged to generate the target face image data.
  • Improve the authenticity of the final target image data obtained by analyzing the role of the average skin color data in the skin color fusion of the target face image data, and increase the authenticity of the finally obtained target face image data;
  • the face image data is added with real lighting effects, the display position of the face image data is adjusted, and the face area of the face is filled, which further increases the real effect of the final output face image data; by using skin color fusion to the pixel
  • the process improves the accuracy of skin color fusion of the target face image data.
  • the storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • General Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • Computer Graphics (AREA)
  • Geometry (AREA)
  • Human Computer Interaction (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Computing Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Image Processing (AREA)
  • Processing Or Creating Images (AREA)

Abstract

本申请实施例公开一种图像融合方法及其设备、存储介质、终端，其中方法包括如下步骤：获取当前待融合图像的源人脸图像数据和当前待融合素材的素材配置信息，对源人脸图像数据进行图像识别处理，获取源人脸图像数据对应的源人脸特征点，并根据源人脸特征点生成源人脸图像数据的源人脸三维网格，采用素材人脸三维网格和源人脸三维网格进行网格融合生成目标人脸三维网格，采用源人脸图像数据的源肤色数据和素材人脸图像数据的素材肤色数据对目标人脸网格进行人脸肤色融合，生成融合后的目标人脸图像数据。

Description

一种图像融合方法及其设备、存储介质、终端
本申请要求于2017年11月22日提交中国专利局、申请号为201711173149.X、发明名称为“一种图像融合方法及其设备、存储介质、终端”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及计算机技术领域，尤其涉及一种图像融合方法及其设备、存储介质、终端。
背景技术
随着计算机技术的快速发展，智能手机、掌上电脑以及平板电脑等终端设备中都可以安装对图片进行加工处理的终端应用，例如，照相机、P图软件或修图软件、社交APP等。基于上述终端应用用户可以对原有图片(例如，人物、风景或者建筑物等)或者视频进行添加特效、装饰美化、美容美妆、改变人物造型等处理。通常处于爱美或者好玩的心里，人们在社交网站或者直播网站中公开自己的照片时都会选择对自己的人脸照片进行适当的美化或者修改。
发明内容
本申请实施例提供一种图像融合方法及其设备、存储介质、终端，通过分析图像数据在三维模型的基础上进行图像融合生成目标人脸图像数据的过程，可以提高最终获得的目标人脸图像数据的真实性。
本申请实施例第一方面提供了一种图像融合方法，用于终端上，所述终端包括：处理器和存储器，所述方法可包括：
获取当前待融合图像的源人脸图像数据和当前待融合素材的素材配置信息，所述素材配置信息包括素材人脸图像数据、素材肤色数据和素材人脸三维网格；
对所述源人脸图像数据进行图像识别处理，获取所述源人脸图像数据对应的源人脸特征点，并根据所述源人脸特征点生成所述源人脸图像数据的源人脸三维网格；
采用所述素材人脸三维网格和所述源人脸三维网格进行网格融合生成目标 人脸三维网格；
采用所述源人脸图像数据的源肤色数据和所述素材人脸图像数据的素材肤色数据对所述目标人脸网格进行人脸肤色融合，生成融合后的目标人脸图像数据。
本申请实施例第二方面提供了一种图像融合设备，可包括：处理器和存储器，所述存储器存储有计算机程序，所述计算机程序由所述处理器加载并执行以下步骤：
获取当前待融合图像的源人脸图像数据和当前待融合素材的素材配置信息，所述素材配置信息包括素材人脸图像数据、素材肤色数据、素材人脸特征点和素材人脸三维网格；
对所述源人脸图像数据进行图像识别处理，获取所述源人脸图像数据对应的源人脸特征点，并根据所述源人脸特征点生成所述源人脸图像数据的源人脸三维网格；
采用所述素材人脸三维网格和所述源人脸三维网格进行网格融合生成目标人脸三维网格；
采用所述源人脸图像数据的源肤色数据和所述素材人脸图像数据的素材肤色数据对所述目标人脸网格进行人脸肤色融合，生成融合后的目标人脸图像数据。
本申请实施例第三方面提供了一种计算机存储介质，所述计算机存储介质存储有多条指令，所述指令适于由处理器加载并执行以下步骤：
获取当前待融合图像的源人脸图像数据和当前待融合素材的素材配置信息，所述素材配置信息包括素材人脸图像数据、素材肤色数据和素材人脸三维网格；
对所述源人脸图像数据进行图像识别处理，获取所述源人脸图像数据对应的源人脸特征点，并根据所述源人脸特征点生成所述源人脸图像数据的源人脸三维网格；
采用所述素材人脸三维网格和所述源人脸三维网格进行网格融合生成目标人脸三维网格；
采用所述源人脸图像数据的源肤色数据和所述素材人脸图像数据的素材肤色数据对所述目标人脸网格进行人脸肤色融合，生成融合后的目标人脸图像数据。
本申请实施例第四方面提供了一种终端，可包括：处理器和存储器；其中，所述存储器存储有计算机程序，所述计算机程序适于由所述处理器加载并执行以下步骤：
获取当前待融合图像的源人脸图像数据和当前待融合素材的素材配置信息，所述素材配置信息包括素材人脸图像数据、素材肤色数据和素材人脸三维网格；
对所述源人脸图像数据进行图像识别处理，获取所述源人脸图像数据对应的源人脸特征点，并根据所述源人脸特征点生成所述源人脸图像数据的源人脸三维网格；
采用所述素材人脸三维网格和所述源人脸三维网格进行网格融合生成目标人脸三维网格；
采用所述源人脸图像数据的源肤色数据和所述素材人脸图像数据的素材肤色数据对所述目标人脸网格进行人脸肤色融合，生成融合后的目标人脸图像数据。
附图简要说明
为了更清楚地说明本申请实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1A是本申请实施例提供的图像融合设备所适用于的应用场景示意图；
图1B-1H是本申请实施例提供的图像融合方法的应用示意图；
图2是本申请实施例提供的一种图像融合方法的流程示意图；
图3是本申请实施例提供的一种人脸基准点位置示意图；
图4是本申请实施例提供的一种人脸三维网格模型示意图；
图5A和图5B是本申请实施例提供的人脸类型示意图；
图6是本申请实施例提供的另一种图像融合方法的流程示意图；
图7是本申请实施例提供的另一种图像融合方法的流程示意图；
图8是本申请实施例提供的另一种图像融合方法的流程示意图；
图9是本申请实施例提供的一种图像融合方法系统示意图；
图10是本申请实施例提供的一种图像融合设备的结构示意图；
图11是本申请实施例提供的另一种图像融合设备的结构示意图；
图12是本申请实施例提供的源网格生成模块的结构示意图；
图13是本申请实施例提供目标数据生成模块的结构示意图；
图14是本申请实施例提供目标数据生成单元的结构示意图；
图15是本申请实施例提供的一种终端的结构示意图。
实施本发明的方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
通常修图类终端应用对人脸图像数据的处理过程是根据用户的2D人脸信息和素材的2D人脸信息，通过一定的融合算法，生成一个既像用户又像素材中的人脸的结果图像。然而，在2D模型的基础上对用户图像和素材图像进行融合生成目标结果图像时，由于融合的用户图像和素材图像作为平面图像所能提取的图像信息反映真实人脸的效果欠佳，将导致最终的融合效果较差，从而影响最终获得的目标结果图像的真实性。
而如果在图像融合设备中，例如其中的修图类终端应用上，应用本申请实施例提供的图像融合方法，在修图时可以获得更接近真实的图片。例如：图像融合设备通过获取当前待融合图像的源人脸图像数据和当前待融合素材的素材配置信息，其中，所述素材配置信息包括素材人脸图像数据、素材肤色数据和素材人脸三维网格，再对所述源人脸图像数据进行图像识别处理，获取所述源人脸图像数据对应的源人脸特征点，并根据所述源人脸特征点生成所述源人脸图像数据的源人脸三维网格，然后采用所述素材人脸三维网格和所述源人脸三维网格进行网格融合生成目标人脸三维网格，最后采用所述源人脸图像数据的源肤色数据和所述素材人脸图像数据的素材肤色数据对所述目标人脸网格进行人脸肤色融合，生成融合后的目标人脸图像数据。通过分析素材人脸三维网格和源人脸三维网格在三维模型的基础上融合为目标人脸三维网格，以及对目标人脸三维网格进行肤色融合生成目标人脸图像数据的过程，提高了最终获得的目标人脸图像数据的真实性。
图1A是本申请实施例提供的图像融合设备所适用的应用场景示意图。如图1A所示，该应用场景中包括图像融合设备10、服务器20。图像融合设备10上安装有图像融合应用程序客户端。服务器20例如为图像融合应用程序服务器，与图像融合应用程序客户端通过网络进行通信。图像融合应用程序客户端可以通过本申请实施例的图像融合方法进行图像融合。用户使用图像融合应用程序客户端时，图像融合应用程序客户端的界面可以提示用户选择一个待融合的素材，在用户选择待融合的素材后，图像融合应用程序可以从图像融合设备10的本地或从服务器20上获取该待融合的素材的素材人脸图像数据。所述待融合的素材可以是视频或者图像。之后，图像融合应用程序客户端的界面可以提示用户进行自拍，以获得用户人脸的源人脸图像数据，也可以引导用户从图像融合设备10上的相册中选择一个视频或者图像，从中获得源人脸图像数据。然后，图像融合应用程序的客户端再将源人脸图像数据与素材人脸图像数据进行融合，形成目标人脸图像数据，即，融合后的人脸图像或视频，输出显示给用户。所述图像融合应用程序的客户端可以和服务器20进行通信，更新待融合的素材的素材人脸图像数据。
所述融合设备10上还可以安装独立的单机图像融合应用程序。在下文中，图像融合应用程序包括图像融合应用程序的客户端以及单机图像融合应用程序。
本申请实施例涉及的图像融合设备10可以是平板电脑、智能手机、掌上电脑以及移动互联网设备(MID)等其他具备图像处理功能的终端设备，还可以是各种修图类终端应用在进行人脸融合处理时所采用的具备计算机处理能力的应用服务器。
图1B至图1H是本申请实施例提供的图像融合方法的应用示意图。在用户进入图像融合设备上的图像融合应用程序之后，在该图像融合应用程序的界面上有让用户选择待融合的图片或视频的选项。用户通过该选项可以选择自拍或者从图像融合设备本地获得待融合的图片或视频。在图像融合应用程序的界面上可以显示出用户选择的图片或视频的预览界面，如图1B所示。用户选择预览界面中的确定选项后，图像融合应用程序确定用户选择的图片或视频，从中获得源人脸图像数据。之后，图像融合应用程序的界面中有供用户选择的素材缩略图，如图1C所示。用户点击各个素材缩略图，可以实时生成用户的人脸 与各素材图之间的融合结果。其中，例如素材素略图30对应的素材图为31。
用户选择素材缩略图30后，便会生成用户的人脸与素材图31融合后的结果图32。之后用户可以选择界面上按键33保存该融合结果图。由于在此过程中，待融合的用户的人脸图像为正面像，所要融合的素材图中的人脸为侧面像。因此，在进行融合时，图像融合应用程序在获得用户选择的图片(如图1D所示)后，从中获得用户人脸的3D图，如图1E所示。之后，图像融合应用程序将用户人脸的3D图旋转成和素材图中人脸角度一致的3D图，如1F所示。图像融合应用程序再在旋转后的3D图上贴上用户的人脸图像的纹理，如图1G所示。最后生成图1H所示的融合结果图。
下面将结合附图2-附图9，对本申请实施例提供的图像融合方法进行详细介绍。
请参见图2，为本申请实施例提供的一种图像融合方法的流程示意图。如图2所示，本申请实施例的所述方法可以包括以下步骤S101-步骤S104。
S101，获取当前待融合图像的源人脸图像数据和当前待融合素材的素材配置信息。
根据本申请实施例，图像融合设备可以获取当前待融合图像的源人脸图像数据和当前待融合素材的素材配置信息。所述源人脸图像数据可以是用户当前通过图像融合设备所拍的或者从图像融合设备的相册中选择的照片或者视频中的人脸图像数据。所述当前待融合素材可以是修图类终端应用(例如，某图秀秀、某某P图、某某相机等)中当前被用户选中的修图所用到的素材模型，例如，动漫人物形象、明星照片等。所述素材配置信息可以包含所述当前待融合素材的3D头像信息(例如，可以是一个obj格式的文件，该文件可以包括素材人脸图像数据、素材肤色数据和素材人脸三维网格等表示素材人脸相关数据的信息)、指示最终所得结果图像中人头效果的信息(该信息可以包括素材3D头像在世界坐标系下的朝向(欧拉角pitch，yaw，roll)、中心位置(最终结果图像的指定位置)、大小(Scale)信息以及与之相匹配的相机信息(例如用透视矩阵来刻画这个信息)等)、2D的贴纸和3D的贴纸以及用户人脸和素材人脸的融合程度alpha(该融合程度可以是每一帧都相同也可以是每帧都不同)。所述头像可以是整个头部的图像，也可以仅是脸部区域的图像。
根据本申请实施例，用于生成目标人脸图像数据的所述素材人脸图像数据和所述源人脸图像数据的大小需要对应相同的尺度，但二者的脸部凹度、胖瘦等情况可以不完全一致，因此，在获取到所述源人脸图像数据和所述素材人脸图像数据后，可以根据所述素材人脸图像数据的大小调整所述源人脸图像数据的大小，使二者大小对应相同的尺度。
S102，对所述源人脸图像数据进行图像识别处理，获取所述源人脸图像数据对应的源人脸特征点，并根据所述源人脸特征点生成所述源人脸图像数据的源人脸三维网格。
根据本申请实施例，所述图像融合设备可以对所述源人脸图像数据进行图像识别处理，获取所述源人脸图像数据对应的源人脸特征点。所述图像识别处理可以是采用人脸检测技术(例如，跨平台计算机视觉库OpenCV提供的人脸检测、新型视觉服务平台Face++、优图人脸检测等)对照片中的用户人脸进行识别和五官定位的过程，所述源人脸特征点可以是能够表征所述源人脸图像数据的面部特征(例如，脸部轮廓、眼睛轮廓、鼻子、嘴唇等)的数据点。
根据本申请实施例，所述图像融合设备可以对所述源人脸图像数据进行图像识别处理(例如，可以对照片中的用户人脸进行识别和五官定位，从而得到一定数量的基准特征点)，获取所述源人脸图像数据的基准特征点，再对所述基准特征点进行三维深度信息提取，获取所述基准特征点对应的源人脸特征点。所述三维深度信息提取可以是以上述基准特征点为基础，通过匹配标准三维模型的脸部特征点推算出能够反映源人脸图像数据在三维模型中的特征点的过程。所述基准特征点可以是指示面部特征的基准点，例如，脸部轮廓、眼睛轮廓、鼻子、嘴唇等点，可以是83个基准点，也可以是如图3所示的68个基准点，具体的点数可以有开发人员根据需求而定。所述源人脸特征点可以是在所述基准特征点的基础上进一步深化后能够对应源人脸图像数据三维模型的特征点，例如，通过对上述68个或83个基准点的三维深度信息提取可以获得1000个深化后的源人脸特征点，可以是图4所示的各三角面片的顶点。
进一步的，所述图像融合设备可以根据所述源人脸特征点生成所述源人脸图像数据的源人脸三维网格。所述源人脸三维网格可以是所述源人脸图像数据中的人脸对应的3D脸部网格模型，例如图4所示的3D人脸网格或者类似于图4的只有半边脸的3D人脸网格。
S103，采用所述素材人脸三维网格和所述源人脸三维网格进行网格融合生成目标人脸三维网格。
根据本申请实施例，由于所述源人脸图像数据和所述素材人脸图像数据的大小位于同一尺度下，所述源人脸三维网格和所述素材人脸三维网络的尺度也位于同一尺度下。
具体的，所述图像融合设备可以采用所述素材人脸三维网格和所述源人脸三维网格进行网格融合生成目标人脸三维网格。所述素材人脸三维网格可以类似于上述源人脸三维网格，可以是所述素材人脸图像数据中人脸对应的3D脸部网格模型，可以是图4所示的3D人脸网格或者是基于图4所示的3D人脸网格在世界坐标系中根据欧拉角改变朝向后的3D人脸网格。所述目标人脸三维网格可以是目标人脸图像数据的3D人脸网格。根据本申请实施例，当素材人脸三维网格是图4所示的3D人脸网格时，最终投影到2D图像上可以呈现正脸的效果，当素材人脸三维网格是基于图4所示的3D人脸网格在世界坐标系中根据欧拉角改变朝向后的3D人脸网格时，最终投影到2D图像上可以呈现侧脸的效果。
根据本申请实施例，所述图像融合设备可以根据所述源人脸三维网格和所述素材人脸三维网格中的人脸特征点计算目标人脸的目标人脸特征点，再根据计算所得的目标人脸特征点生成目标人脸三维网格。例如，3D源人脸三维网格(即用户的3D人脸网格)上有1000个带深度信息的源人脸特征点标记为绿色，3D素材人脸三维网格上有1000个带深度信息的素材人脸特征点标记为蓝色，用户和素材的1000人脸特征点每个相对应点的平均点(相同位置的对应点取平均，一共1000个点对)标记为红色，最终生成的1000个红色的人脸特征点即为目标人脸特征点，通过上述1000个红色人脸特征点可以组成1900多个三角形，对应的1900多个三角面片所描绘的人脸三维网格即为目标人脸三维网格。根据本申请实施例，所述图像融合设备可以采用移动最小二乘的图像变形(Image Deformation Using Moving Least Squares，MLS)法、仿射变换、图像扭曲等算法将源人脸图像数据和素材人脸图像数据的五官位置趋于上述红色的人脸特征点即目标人脸特征点所指示的五官位置，实现人脸融合的目的。
S104，采用所述源人脸图像数据的源肤色数据和所述素材图像数据的素材肤色数据对所述目标人脸三维网格进行人脸肤色融合，生成融合后的目标人脸图像数据。
根据本申请实施例，生成所述目标人脸三维网格后，所述图像融合设备需要填充所述目标人脸三维网格中不同三角面片上的肤色才能得到最终的目标人脸图像数据。
具体的，所述图像融合设备可以采用所述源人脸图像数据的源肤色数据和所述素材图像数据的素材肤色数据对所述目标人脸三维网格进行人脸肤色融合，生成融合后的目标人脸图像数据。根据本申请实施例，所述源肤色数据可以是组成所述源人脸图像数据的源像素点的集合，所述素材肤色数据可以是组成所述素材人脸图像数据的素材像素点的集合。
根据本申请实施例，若上述源人脸三维网格与素材人脸三维网格的类型不一致时，需要对所述源人脸三维网格进行网络补充，生成候选人脸三维网格，所述类型可以是所述源人脸三维网格和所述素材人脸三维网格中的网络要素。所述候选人脸三维网格的候选肤色数据可以是对称后的源人脸图像数据，即可以认为所述候选人脸三维网格的候选肤色数据就是所述源肤色数据。所述图像融合设备可以根据上述源像素点、素材像素点和所述素材配置信息中的融合程度(可以是根据经验值进行设置的融合度值，通常取值在0-1之间)计算目标人脸图像数据的目标像素点，进而根据所述目标像素点填充所述目标人脸三维网格生成目标人脸图像数据。例如，设置源人脸三维网格中某三角面片上一个特征点的像素为：UserB、UserG和UserR，素材人脸三维网格中相应的三角面片上相应位置处的脸特征点的像素为：ResourceB、ResourceG和ResourceR，目标人脸三维网格中相应的三角面片上相应位置处的特征点的像素为：
TargetB、TargetG和TargetR，素材配置信息中的融合程度为：alpha，则有：
TargetB＝(1.0–alpha)*UserB+alpha*ResourceB
TargetG＝(1.0–alpha)*UserG+alpha*ResourceG
TargetR＝(1.0–alpha)*UserR+alpha*ResourceR
从而可以得到目标人脸图像数据的每一个像素值，获得目标人脸图像数据。
根据本申请实施例，最终生成的所述目标人脸图像数据可以是三维人脸图像数据，也可以是二维人脸图像数据。当所述目标人脸图像数据是二维人脸图 像数据时，由于生成所述目标图像数据的过程是在三维模型的基础上实现的，考虑到真实光线、阴影等问题，最终形成的目标人脸图像数据效果更逼真。
根据本申请实施例，上述候选人脸三维网格的候选肤色数据可以包括两部分，即与所述源人脸三维网格相匹配部分的候选人脸三维网格的肤色数据为所述源肤色数据，与所述源人脸三维网格不匹配部分的候选人脸三维网格的肤色数据为平均肤色数据，所述平均肤色数据可以是所述源肤色数据经肤色均衡化处理后的肤色数据。根据本申请实施例，所述肤色均衡化处理可以是去掉所述源人脸图像数据中由于光线等原因造成的阴影等效果获得肤色平均值的过程。所述图像融合设备可以根据上述候选肤色数据的候选像素点、素材像素点和所述素材配置信息中的融合程度计算目标人脸图像数据的目标像素点，进而根据所述目标像素点填充所述目标人脸三维网格生成目标人脸图像数据，具体的计算过程与上述计算过程一致此处不再赘述。
在本申请实施例中，通过获取当前待融合图像的源人脸图像数据和当前待融合素材的素材配置信息，其中素材配置信息包括素材人脸图像数据、素材肤色数据和素材人脸三维网格，再对源人脸图像数据进行图像识别处理，获取源人脸图像数据对应的源人脸特征点，并根据源人脸特征点生成源人脸图像数据的源人脸三维网格，然后采用素材人脸三维网格和源人脸三维网格进行网格融合生成目标人脸三维网格，最后采用源人脸图像数据的源肤色数据和素材人脸图像数据的素材肤色数据对目标人脸网格进行人脸肤色融合，生成融合后的目标人脸图像数据。通过分析素材人脸三维网格和源人脸三维网格在三维模型的基础上融合为目标人脸三维网格，以及对目标人脸三维网格进行肤色融合生成目标人脸图像数据的过程，提高了最终获得的目标人脸图像数据的真实性。
请参见图6，为本申请实施例提供的另一种图像融合方法的流程示意图。如图6所示，本申请实施例的所述方法可以包括以下步骤S201-步骤S210。
S201，获取当前待融合图像的源人脸图像数据和当前待融合素材的素材配置信息。
根据本申请实施例，图像融合设备可以获取当前待融合图像的源人脸图像数据和当前待融合素材的素材配置信息。所述源人脸图像数据可以是用户当前通过图像融合设备所拍的或者从图像融合设备的相册中选择的照片或者视频中 的人脸图像数据。所述当前待融合素材可以是修图类终端应用(例如，某图秀秀、某某P图、某某相机等)中当前被用户选中的修图所用到的素材模型，例如，动漫人物形象、明星照片等。所述素材配置信息可以包含所述当前待融合素材的3D头像信息(例如，可以是一个obj格式的文件，该文件可以包括素材人脸图像数据、素材肤色数据和素材人脸三维网格等表示素材人脸相关数据的信息)、指示最终所得结果图像中人头效果的信息(该信息可以包括素材3D头像在世界坐标系下的朝向(欧拉角pitch，yaw，roll)、中心位置(最终结果图像的指定位置)、大小(Scale)信息以及与之相匹配的相机信息(例如用透视矩阵来刻画这个信息)等)、2D的贴纸和3D的贴纸以及用户人脸和素材人脸的融合程度alpha(该融合程度可以是每一帧都相同也可以是每帧都不同)。
根据本申请实施例，用于生成目标人脸图像数据的所述素材人脸图像数据和所述源人脸图像数据的大小需要对应相同的尺度，但二者的脸部凹度、胖瘦等情况可以不完全一致，因此，在获取到所述源人脸图像数据和所述素材人脸图像数据后，可以根据所述素材人脸图像数据的大小调整所述源人脸图像数据的大小，使二者大小对应相同的尺度。
S202，对所述源人脸图像数据进行图像识别处理，获取所述源人脸图像数据的基准特征点。
根据本申请实施例，所述图像融合设备可以对所述源人脸图像数据进行图像识别处理，获取所述源人脸图像数据的基准特征点。所述图像识别处理可以是采用人脸检测技术(例如，跨平台计算机视觉库OpenCV提供的人脸检测、新型视觉服务平台Face++、优图人脸检测等)对照片中的用户人脸进行识别和五官定位的过程。所述基准特征点可以是指示面部特征的基准点，例如，脸部轮廓、眼睛轮廓、鼻子、嘴唇等点，可以是83个基准点，也可以是如图3所示的68个基准点，具体的点数可以有开发人员根据需求而定。
S203，对所述基准特征点进行三维深度信息提取，获取所述基准特征点对应的源人脸特征点，根据所述源人脸特征点生成源人脸三维网格。
根据本申请实施例，所述图像融合设备可以对所述基准特征点进行三维深度信息提取，获取所述基准特征点对应的源人脸特征点，根据所述源人脸特征点生成源人脸三维网格。所述三维深度信息提取可以以上述基准特征点为基 础，通过匹配标准三维模型的脸部特征点推算出能够反映源人脸图像数据在三维模型中的特征点的过程。所述源人脸特征点可以是在所述基准特征点的基础上进一步深化的点，例如，通过对上述68个或83个基准点的三维深度信息提取可以获得1000个深化后的源人脸特征点，可以是图4中各三角面片的顶点。所述源人脸三维网格可以是所述源人脸图像数据中的人脸对应的3D脸部网格模型，可以是所述源人脸特征点连接而成的用户人脸的3D网格模型，例如图4所示的3D人脸网格或者类似于图4的只有半边脸的3D人脸网格。
S204，采用所述源人脸三维网格和所述素材人脸三维网格进行网格融合生成目标人脸三维网格。
根据本申请实施例，由于所述源人脸图像数据和所述素材人脸图像数据的大小位于同一尺度下，所述源人脸三维网格和所述素材人脸三维网络的尺度也位于同一尺度下。
具体的，所述图像融合设备可以采用所述素材人脸三维网格和所述源人脸三维网格进行网格融合生成目标人脸三维网格。根据本申请实施例，所述素材人脸三维网格可以类似于上述源人脸三维网格可以是所述素材人脸图像数据中人脸对应的3D脸部网格模型，可以是图4所示的3D人脸网格或者是基于图4所示的3D人脸网格在世界坐标系中根据欧拉角改变朝向后的3D人脸网格。所述目标人脸三维网格可以是目标人脸图像数据的3D人脸网格。根据本申请实施例，当素材人脸三维网格是图4所示的3D人脸网格时，最终投影到2D图像上可以呈现正脸的效果，当素材人脸三维网格是基于图4所示的3D人脸网格在世界坐标系中根据欧拉角改变朝向后的3D人脸网格时，最终投影到2D图像上可以呈现侧脸的效果。
在本申请实施例中，所述图像融合设备可以根据所述源人脸三维网格和所述素材人脸三维网格中的人脸特征点计算目标人脸的目标人脸特征点，再根据计算所得的目标人脸特征点生成目标人脸三维网格。例如，3D源人脸三维网格(即用户的3D人脸网格)上有1000个带深度信息的源人脸特征点标记为绿色，3D素材人脸三维网格上有1000个带深度信息的素材人脸特征点标记为蓝色，用户和素材的1000人脸特征点每个相对应点的平均点(相同位置的对应点取平均，一共1000个点对)标记为红色，最终生成的1000个红色的人脸特征点即为目标人脸特征点，通过上述1000个红色人脸特征点可以组成1900多 个三角形，对应的1900多个三角面片所描绘的人脸三维网格即为目标人脸三维网格。根据本申请实施例，所述图像融合设备可以采用MLS法、仿射变换、图像扭曲等算法将源人脸图像数据和素材人脸图像数据的五官位置趋于上述红色的人脸特征点即目标人脸特征点所指示的五官位置，实现人脸融合的目的。
S205，对所述源人脸图像数据进行肤色均衡化处理，获取所述源人脸图像数据的平均肤色数据。
根据本申请实施例，若上述源人脸三维网格与素材人脸三维网格的类型不一致时，需要对所述源人脸三维网格进行网络补充，生成候选人脸三维网格。所述类型可以是所述源人脸三维网格和所述素材人脸三维网格中的网络要素。所述候选人脸三维网格的候选肤色数据可以是对称后的源人脸图像数据，即可以认为所述候选人脸三维网格的候选肤色数据就是所述源肤色数据。例如，最后生成的候选人脸图像数据为图5A中的人脸图像，则图5A中人脸的肤色与图5B中人脸的肤色一致。
在本申请实施例中，上述候选人脸三维网格的候选肤色数据可以包括两部分，即与所述源人脸三维网格相匹配部分的候选人脸三维网格的肤色数据为所述源肤色数据，与所述源人脸三维网格不匹配部分的候选人脸三维网格的肤色数据为平均肤色数据。具体的，所述图像融合设备可以对所述源人脸图像数据进行肤色均衡化处理，获取所述源人脸图像数据的平均肤色数据。所述肤色均衡化处理可以是去掉所述源人脸图像数据中由于光线等原因造成的阴影等效果获得肤色平均值的过程。所述平均肤色数据可以是所述源肤色数据去掉阴影后的像素点数据的平均值组成的像素点集合。
S206，基于所述源人脸图像数据的源肤色数据和所述平均肤色数据对所述候选人脸三维网格进行肤色填充，生成候选人脸图像数据。
具体的，所述图像融合设备可以基于所述源人脸图像数据的源肤色数据和所述平均肤色数据对所述候选人脸三维网格进行肤色填充，生成候选人脸图像数据。所述候选人脸三维网格中与所述源人脸三维网格相匹配部分的肤色数据可以用所述源人脸图像数据的源肤色数据进行填充，所述候选人脸三维网格中与所述源人脸三维网格不匹配部分的肤色数据可以用所述平均肤色数据进行填充。所述候选人脸图像数据的候选肤色数据可以包括所述源肤色数据和所述平 均肤色数据。例如，最后生成的候选人脸图像数据为图5A中的人脸图像，则图5A中右边脸的肤色为图5B中人脸的肤色，图5A中左边脸的肤色为图5B中人脸肤色平均化后的肤色。
S207，采用所述候选人脸图像数据的候选肤色数据和所述素材图像数据的素材肤色数据对所述目标人脸三维网格进行人脸肤色融合，生成融合后的目标人脸图像数据。
根据本申请实施例，生成所述目标人脸三维网格后，所述图像融合设备需要填充所述目标人脸三维网格中不同三角面片上的肤色才能得到最终的目标人脸图像数据。
具体的，所述图像融合设备可以采用所述候选人脸图像数据的候选肤色数据和所述素材图像数据的素材肤色数据对所述目标人脸三维网格进行人脸肤色融合，生成融合后的目标人脸图像数据。根据本申请实施例，所述候选肤色数据可以是组成所述候选人脸图像数据的候选像素点的集合，所述素材肤色数据可以是组成所述素材人脸图像数据的素材像素点的集合。
在本申请实施例中，所述图像融合设备可以基于所述候选像素点和所述素材像素点，并采用融合程度计算目标像素点，根据所述目标像素点生成目标人脸图像数据。所述融合程度(可以是根据经验值进行设置的融合度值，通常取值在0-1之间。
在本申请实施例的具体实现方式中，可以设置候选人脸三维网格中某三角面片上一个特征点的像素为：CandidateB、CandidateG和CandidateR，素材人脸三维网格中相应的三角面片上相应位置处的脸特征点的像素为：ResourceB、ResourceG和ResourceR，目标人脸三维网格中相应的三角面片上相应位置处的特征点的像素为：TargetB、TargetG和TargetR，素材配置信息中的融合程度为：alpha，则有：
TargetB＝(1.0–alpha)*CandidateB+alpha*ResourceB
TargetG＝(1.0–alpha)*CandidateG+alpha*ResourceG
TargetR＝(1.0–alpha)*CandidateR+alpha*ResourceR
从而可以得到目标人脸图像数据的每一个像素值，获得目标人脸图像数据。
在本申请实施例中，通过分析平均肤色数据在目标人脸图像数据肤色融合中的作用，增加了最终获得的目标人脸图像数据的真实性。
S208，根据所述源人脸图像数据的源肤色数据获取所述源人脸图像数据对应的光源类型，并采用所述光源类型对应的光照效果对所述目标人脸图像数据进行效果添加处理。
根据本申请实施例，所述目标人脸图像数据生成后，所述图像融合设备可以根据所述源人脸图像数据的源肤色数据获取所述源人脸图像数据对应的光源类型。所述图像融合设备可以通过比较所述平均肤色和所述源人脸图像数据中各个区域的肤色，获取亮与平均肤色的区域和暗与平均肤色的区域，进而推算出光源类型，例如是多个点光源或者面光源。所述图像融合设备也可以通过深度学习搜集源人脸图像数据在不同指定光照情况下的结果图，然后将结果图和对应的光照情况作为深度神经((Deep Neural Networks，DNN)的训练数据，训练出一个给定图片就能输出光源类型和光源位置的DNN模型。
进一步的，所述图像融合设备可以采用所述光源类型对应的光照效果对所述目标人脸图像数据进行效果添加处理。例如，所述源人脸图像数据的对应的光源类型是来自左侧脸方向的点光源，则所述图像融合设备可以在最终获得的目标人脸图像数据中添加左脸方向点光源的光照效果。
在本申请实施例中，所述图像融合设备还可以将上述素材配置信息中的2D和3D贴纸按照其对应的位置和层级(zOrder)粘贴在所述目标人脸图像数据上，例如，将3D眼镜贴纸佩戴在生成的目标人脸图像数据的人脸上。
S209，基于所述素材人脸图像数据所指示的坐标信息调整所述目标人脸图像数据当前的显示位置。
根据本申请实施例，所述目标人脸图像数据生成后，所述图像融合设备还可以基于所述素材人脸图像数据所指示的坐标信息调整所述目标人脸图像数据当前的显示位置。例如，根据上述素材配置信息中的素材人脸图像数据所指示的坐标信息(包括，欧拉方向和中心点)和素材大小将所得的目标人脸图像数据放在指定的位置上。
S210，当所述目标人脸图像数据的第一显示区域小于所述源人脸图像数据的第二显示区域时，对所述第二显示区域中除所述第一显示区域的部分进行人脸边缘填充处理。
根据本申请实施例，当最终生成的目标人脸图像数据的第一显示区域小于所述源人脸图像数据的第二显示区域时，所述图像融合设备可以对所述第二显 示区域中除所述第一显示区域的部分进行人脸边缘填充处理。所述第一显示区域可以是所述目标人脸图像数据映射在2D屏幕上的范围，所述第二显示区域可以是所述源人脸图像数据映射在2D屏幕上的范围，所述第一显示区域小于所述第二显示区域可以表述为所述目标人脸图像数据的人脸比所述源人脸图像数据的人脸小。所述人脸边缘填充处理可以是采用填充算法(例如，OpenCV提供的图像修复算法Inpainting)将所述第二显示区域中除所述第一显示区域的部分填充满。
进一步的，所述图像融合设备对所述目标人脸图像数据的附加技术效果处理完之后，可以对最终获得的目标人脸图像数据进行输出显示。
根据本申请实施例，执行步骤S208-步骤S210后最终生成的所述目标人脸图像数据可以是三维人脸图像数据，也可以是二维人脸图像数据。当所述目标人脸图像数据是二维人脸图像数据时，由于生成所述目标图像数据的过程是在三维模型的基础上实现的，考虑到真实光线、阴影等问题，最终成线的目标人脸图像数据效果更逼真。
需要说明的是，在执行步骤S208-步骤S210时，可以选择其中的一个或多步骤同时执行。
在本申请实施例中，通过对生成的目标人脸图像数据的添加真实的光照效果、调整人脸图像数据的显示位置以及填充人脸边缘区域，进一步增加了最终输出的目标人脸图像数据的真实效果。
在本申请实施例一种具体实现方式中，所述采用所述候选人脸图像数据的候选肤色数据和所述素材图像数据的素材肤色数据对所述目标人脸三维网格进行人脸肤色融合，生成融合后的目标人脸图像数据可以包括以下几个步骤，如图7所示：
S301，获取所述候选肤色数据中的候选像素点和所述素材肤色数据中的素材像素点。
根据本申请实施例，所述候选肤色数据可以是组成上述候选人脸图像数据的候选像素点的集合，所述素材肤色数据可以是组成上述素材人脸图像数据的素材像素点的集合。所述图像融合设备可以获取所述候选肤色数据中的候选像素点和所述素材肤色数据中的素材像素点。
S302，基于所述候选像素点和所述素材像素点，并采用融合程度计算目标像素点，根据所述目标像素点生成目标人脸图像数据。
具体的，所述图像融合设备基于所述候选像素点和所述素材像素点，并采用融合程度获取目标人脸图像数据的具体过程可以参见步骤S207中的描述，此处不再赘述。
在本申请实施例中，通过采用精确到像素点的肤色融合过程，提高了目标人脸图像数据肤色融合的准确性。
在本申请实施例中，通过获取当前待融合图像的源人脸图像数据和当前待融合素材的素材配置信息，其中素材配置信息包括素材人脸图像数据、素材肤色数据和素材人脸三维网格，再对源人脸图像数据进行图像识别处理，获取源人脸图像数据对应的源人脸特征点，并根据源人脸特征点生成源人脸图像数据的源人脸三维网格，然后采用素材人脸三维网格和源人脸三维网格进行网格融合生成目标人脸三维网格，最后采用源人脸图像数据的源肤色数据和素材人脸图像数据的素材肤色数据对目标人脸网格进行人脸肤色融合，生成融合后的目标人脸图像数据。通过分析素材人脸三维网格和源人脸三维网格在三维模型的基础上融合为目标人脸三维网格，以及对目标人脸三维网格进行肤色融合生成目标人脸图像数据的过程，提高了最终获得的目标人脸图像数据的真实性；通过分析平均肤色数据在目标人脸图像数据肤色融合中的作用，增加了最终获得的目标人脸图像数据的真实性；通过对生成的目标人脸图像数据的添加真实的光照效果、调整人脸图像数据的显示位置以及填充人脸边缘区域，进一步增加了最终输出的目标人脸图像数据的真实效果；通过采用精确到像素点的肤色融合过程，提高了目标人脸图像数据肤色融合的准确性。
需要说明的是，在2D模型的基础上进行人脸融合时，通常存在用户图像数据的人脸角度与素材图像数据的人脸角度不完全匹配的情况。例如，用户图像是半边脸，素材图像是正脸或者用户人头向左转，素材人头向右转等。当存在上述情况时，现有技术中的人脸融合算法所能获取到的用户人脸的信息较少，在进行人脸融合时较少的用户人脸信息将影响最终的匹配结果，导致生成的目标结果图像的真实性较差。
为了解决上述问题，本申请实施例提供了另一种图像融合方法的流程示意图。如图8所示，本申请实施例的所述方法可以包括以下步骤S401-步骤S404。
S401，获取当前待融合图像的源人脸图像数据和当前待融合素材的素材配置信息。
具体的，图像融合设备可以获取当前待融合图像的源人脸图像数据和当前待融合素材的素材配置信息，详细的获取过程可以参见步骤S201中的相关描述此处不再赘述。
S402，对所述源人脸图像数据进行图像识别处理，获取所述源人脸图像数据对应的源人脸特征点，并根据所述源人脸特征点生成所述源人脸图像数据的源人脸三维网格。
具体的，所述图像融合设备生成所述源人脸三维网格的过程可以参见上述步骤S202-步骤S203中的相关描述，此处不再赘述。
S403，当检测到所述源人脸三维网格与所述素材人脸三维网格的类型不一致时，根据所述源人脸图像数据的对称性对所述源人脸三维网格进行网格补充，生成与所述素材人脸三维网格类型一致的候选人脸三维网格，并采用所述候选人脸三维网格和所述素材人脸三维网格进行网格融合生成目标人脸三维网格。
根据本申请实施例，由于所述源人脸图像数据和所述素材人脸图像数据的大小位于同一尺度下，所述源人脸三维网格和所述素材人脸三维网络的尺度也位于同一尺度下。
根据本申请实施例，所述素材人脸三维网格可以类似于上述源人脸三维网格可以是所述素材人脸图像数据中人脸对应的3D脸部网格模型，可以是图4所示的3D人脸网格或者是基于图4所示的3D人脸网格在世界坐标系中根据欧拉角改变朝向后的3D人脸网格。所述目标人脸三维网格可以是目标人脸图像数据的3D人脸网格。根据本申请实施例，当素材人脸三维网格是图4所示的3D人脸网格时，最终投影到2D图像上可以呈现正脸的效果，当素材人脸三维网格是基于图4所示的3D人脸网格在世界坐标系中根据欧拉角改变朝向后的3D人脸网格时，最终投影到2D图像上可以呈现侧脸的效果。
根据本申请实施例，所述源人脸三维网格和所述素材人脸三维网格只有在 类型一致时，才能根据融合算法生成与二者脸部显示区域一样且真实性较好的目标人脸三维网格。根据本申请实施例，所述类型一致可以是所述源人脸三维网格和所述素材人脸三维网格中的网络要素一致，所述网格要素可以是所述源人脸三维网格和所述素材人脸三维网格的网格朝向或者网格显示区域。例如，所述源人脸三维网格指示的脸部显示区域和所述素材人脸三维网格指示的脸部显示区域一致，二者都是类似于图5A所示的标准正脸或者类似于图5B所示的侧脸时，可以认为所述源人脸三维网格和所述素材人脸三维网格类型一致。当所述源人脸网格和所述素材人脸网格的类型不一致时(例如，源人脸三维网格为类似于图5B所示的侧脸，素材人脸三维网格为类似于图5A所示的标准正脸)，所述图像融合设备可以先对所述源人脸网格进行网格补充。
具体的，当所述源人脸三维网格与所述素材人脸三维网格的类型不一致时，所述图像融合设备可以根据所述源人脸图像数据的对称性对所述源人脸三维网格进行网格补充，生成与所述素材人脸三维网格类型一致的候选人脸三维网格。正常情况下人的脸部都是对称的(忽略细微的差别)，当所述源人脸图像数据为侧脸图像数据时(例如，图5B中的图像)，所述图像融合设备提取所述源人脸特征点后生成的源人脸三维网格也是针对侧脸的，此时，所述图像融合设备可以根据脸部对称的原则将该源人脸图像数据补充为标准正脸(例如，由图5B的图像补充后得到图5A的图像)，然后再根据补充后的源人脸图像数据生成与所述素材人脸三维网格类型一致的候选人脸三维网格(图5A对应的人脸三维网格即是候选人脸三维网格)。
根据本申请实施例，所述图像融合设备根据脸部对称原则对源人脸图像数据(侧脸图像数据)补充后得到的标准正脸图像数据的左右脸部表情一致，进一步得到的候选人脸三维网格的左右脸部表情也是一致的。
在本申请实施例中，当所述素材人脸三维网格为左右脸部表情不一致的标准正脸，而源人脸图像数据为侧脸图像数据时，根据源人脸图像数据的对称性对源人脸三维网格进行网格补充后得到的候选人脸三维网格的左右脸部表情，与素材人脸三维网格的左右脸部表情是不一致的(例如，素材人脸图像数据所指示素材人脸的左眼是张开的，右眼是闭着的，而只包含左侧脸的源人脸图像数据指示的用户只人脸的左眼是张开的，此时，根据源人脸图像数据的对称性补充后的源人脸三维网格的右眼也是张开的，与素材人脸三维网格的右眼不一 致)，此时，所述图像融合设备将无法采用该候选人脸三维网格和上述素材人脸三维网格进行网格融合生成目标人脸三维网格。对上述种情况，所述图像融合设备可以通过表情迁移算法将候选人脸三维网格的左右脸部表情调整至与素材人脸三维网格的表情一致，使最终得到的候选人脸三维网格与上述素材人脸三维网格的面部表情一致。
进一步的，所述图像融合设备可以采用所述候选人脸三维网格和所述素材人脸三维网格进行网格融合生成目标人脸三维网格。根据本申请实施例，所述候选人脸三维网格与所述素材人脸三维网格的类型是一致的，即在相同脸部位置上所述素材人脸三维网格和所述候选人脸三维网格都有相应的特征点。例如，候选人脸三维网格中包含两只眼睛眼角的特征点，素材人脸三维网格也包含两只眼睛眼角的特征点。所述目标人脸三维网格可以是目标人脸图像数据的3D人脸网格。
在本申请实施例中，所述图像融合设备可以根据所述候选人脸三维网格和所述素材人脸三维网格中的人脸特征点计算目标人脸的目标人脸特征点，再根据计算所得的目标人脸特征点生成目标人脸三维网格，例如，3D候选人脸三维网格(即用户的3D人脸网格)上有1000个带深度信息的源人脸特征点标记为绿色，3D素材人脸三维网格上有1000个带深度信息的素材人脸特征点标记为蓝色，用户和素材的1000人脸特征点每个相对应点的平均点(相同位置的对应点取平均，一共1000个点对)标记为红色，最终生成的1000个红色的人脸特征点即为目标人脸特征点，通过上述1000个红色人脸特征点可以组成1900多个三角形，对应的1900多个三角面片所描绘的人脸三维网格即为目标人脸三维网格。根据本申请实施例，所述图像融合设备可以采用移动最小二乘的图像变形法MLS、仿射变换、图像扭曲等算法将源人脸图像数据和素材人脸图像数据的五官位置趋于上述红色的人脸特征点即目标人脸特征点所指示的五官位置，实现人脸融合的目的。
S404，采用所述源人脸图像数据的源肤色数据和所述素材人脸图像数据的素材肤色数据对所述目标人脸网格进行人脸肤色融合，生成融合后的目标人脸图像数据。
具体的，所述图像融合设备生成所述目标人脸图像数据的过程可以参见上述步骤S205-步骤S210中的相关描述，此处不再赘述。
在本申请实施例中，通过获取当前待融合图像的源人脸图像数据和当前待融合素材的素材配置信息，再对源人脸图像数据进行图像识别处理，获取源人脸图像数据对应的源人脸特征点，并根据源人脸特征点生成源人脸图像数据的源人脸三维网格，然后当检测到所述源人脸三维网格与所述素材人脸三维网格的类型不一致时，根据所述源人脸图像数据的对称性对所述源人脸三维网格进行网格补充，生成与所述素材人脸三维网格类型一致的候选人脸三维网格，并采用所述候选人脸三维网格和所述素材人脸三维网格进行网格融合生成目标人脸三维网格，最后采用源人脸图像数据的源肤色数据和素材人脸图像数据的素材肤色数据对目标人脸网格进行人脸肤色融合，生成融合后的目标人脸图像数据，即使存在用户图像数据的人脸角度与素材图像数据的人脸角度不完全匹配的情况，也能通过建立用户的三维头像，从而推算出用户的头像全貌，根据用户全貌来更好地进行源人脸图像数据和融合素材的三维层面的融合。
上述方法实施例中的素材人脸图像数据是相关终端应用中的一帧素材数据，整个人脸融合过程适用于单帧的实时处理，例如可以用于相机实时预览用户融合变成别的素材形象后的样子。当素材需要处理多帧时，也是一帧一帧的循环处理视频中的多帧图片得到最终人脸融合后的视频。
下面将举例说明一个具体的人脸融合系统的实现过程，如图9所示：
S500，进入系统。
根据本申请实施例，所述系统例如为修图类终端应用的系统。用户打开图像融合设备上的修图类终端，进入该修图类终端应用的首页。
S501，获取第N帧素材人脸图像数据和当前源人脸图像数据。
具体的，该修图类终端应用的首页上可以有提示，提示用户选择素材。图像融合设备在进行人脸融合之前，可以根据用户选择的素材获取修图类终端应用(例如，某图秀秀、某某P图、某某相机等)中某一帧(例如第N帧)素材的素材配置信息。其中，在所获取的素材共有M帧的情况下，N的取值范围为M≥N≥1，N，M为正整数。所述图像融合设备可以从所述素材中的第一帧开始处理，即，此时N＝1。所述素材配置信息中可以包含所述当前待融合素材的3D头像信息(例如，可以是一个obj格式的文件，该文件可以包括素材人脸图像数据、素材肤色数据和素材人脸三维网格等表示素材人脸相关数据的信 息)、指示最终所得结果图像中人头效果的信息(该信息可以包括素材3D头像在世界坐标系下的朝向(欧拉角pitch，yaw，roll)、中心位置(最终结果图像的指定位置)、大小(Scale)信息以及与之相匹配的相机信息(例如用透视矩阵来刻画这个信息)等)、2D的贴纸和3D的贴纸以及用户人脸和素材人脸的融合程度alpha(该融合程度可以是每一帧都相同也可以是每帧都不同)。根据本申请实施例，该修图类终端应用可以引导用户拍一张自拍照或视频，或者从相册中选择一张照片或视频。该过程可以在用户选择素材之前或之后进行。所述图像融合设备可以获取用户当前所拍的或者从相册中选择的照片或者视频中的源人脸图像数据。
根据本申请实施例，用于生成目标人脸图像数据的所述素材人脸图像数据和所述源人脸图像数据的大小需要对应相同的尺度，但二者的脸部凹度、胖瘦等情况可以不完全一致，因此，在获取到所述源人脸图像数据和所述素材人脸图像数据后，可以根据所述素材人脸图像数据的大小调整所述源人脸图像数据的大小，使二者大小对应相同的尺度。
S502，生成目标人脸三维网格。
具体的，所述图像融合设备获取所述素材人脸图像数据和所述源人脸图像数据后，可以对二者进行进一步分析处理生成目标人脸三维网格。根据本申请实施例，所述目标人脸三维网格可以是最终人脸融合所生成的目标人脸图像数据中人脸对应的脸部三维网格模型，具体的实现过程可以步骤S5021-S5025实现：
S5021，获取源人脸特征点。
具体的，所述图像融合设备可以对所述源人脸图像数据进行图像识别处理，获取所述源人脸图像数据的基准特征点，然后对所述基准特征点进行深度信息提取，获取所述基准特征点对应的源人脸特征点。具体的实现过程可以参见步骤S202和步骤S203的详细描述，此处不再赘述。
S5022，生成源人脸三维网格。
具体的，所述图像融合设备可以将上述源人脸特征点连接成由许多三角面片组成的源人脸网格，具体的实现过程可以参见步骤S203的描述，此处不再赘述。
S5023，读取素材配置信息。
根据本申请实施例，所述图像融合设备在进行人脸融合之前，已经获取了第N帧素材的素材配置信息，此时，所述图像融合设备可以读取所述素材配置信息中的相关数据。
S5024，补全源人脸三维网格。
根据本申请实施例，当所述源人脸三维网格和所述素材人脸三维网格类型不一致，(例如，源人脸三维网格为类似于图5B所示的侧脸，素材人脸三维网格为类似于图5A所示的标准正脸)，所述图像融合设备可以先对所述源人脸三维网格进行网格补充，具体的实现过程可以参见步骤S204所述的根据人脸对称性进行的网格补充，此处不再赘述。
S5025，融合源人脸三维网格和素材人脸三维网格。
根据本申请实施例，步骤S5023中获取的素材配置信息中，包括素材人脸三维网格，所述图像融合设备可以将所述源人脸三维网格和所述素材人脸三维网格进行网格融合生成目标人脸三维网格，具体的融合过程可以参见步骤S204所处的详细过程，此处不再赘述。
S503，目标人脸图像数据肤色融合。
根据本申请实施例，生成所述目标人脸三维网格后，所述图像融合设备需要填充所述目标人脸三维网格中不同三角面片上的肤色才能得到最终的目标人脸图像数据。目标人脸图像数据肤色融合的具体过程可以通过步骤S5031-S5033实现：
S5031，获取平均肤色数据。
具体的，所述图像融合设备可以对所述源人脸图像数据进行肤色均衡化处理，获取所述源人脸图像数据的平均肤色数据，具体过程可以参见步骤S205的描述，此处不再赘述。
S5032，肤色融合。
具体的，所述图像融合设备可以基于所述源人脸图像数据的源肤色数据和所述平均肤色数据对所述候选人脸三维网格进行肤色填充，生成候选人脸图像数据，再采用所述候选人脸图像数据的候选肤色数据和所述素材图像数据的素材肤色数据对所述目标人脸三维网格进行人脸肤色融合，生成融合后的目标人脸图像数据，具体的实现过程可以参见步骤S206和步骤S207的详细描述，此处不再赘述。
S504，后期处理。
根据本申请实施例，所述后期处理可以是对生成的目标人脸图像数据的后期调整，使最终输出的目标人脸图像数据效果更逼真。具体的，可以包括步骤S5041和步骤S5042所描述的处理过程。
S5041，灯光渲染，位置调整。
具体的，所述图像融合设备可以采用所述候选人脸图像数据的候选肤色数据和所述素材图像数据的素材肤色数据对所述目标人脸三维网格进行人脸肤色融合，生成融合后的目标人脸图像数据。同时，所述图像融合设备可以基于所述素材人脸图像数据所指示的坐标信息调整所述目标人脸图像数据当前的显示位置。具体的实现过程可以参见步骤S208和步骤S209的详细描述，此处不再赘述。
S5042，人脸边缘填充。
具体的，当所述目标人脸图像数据的第一显示区域小于所述源人脸图像数据的第二显示区域时，所述图像融合设备可以对所述第二显示区域中除所述第一显示区域的部分进行人脸边缘填充处理。具体的实现过程可以参见步骤S210的详细描述，此处不再赘述。
S505，结果输出。
根据本申请实施例，所述图像融合设备对所述目标人脸图像数据的附加技术效果处理完之后，可以对最终获得的目标人脸图像数据进行输出显示。在进行针对图像的人脸融合的情况下，由于整个素材只有一帧，经过前述的步骤处理完之后，可以将目标人脸图像数据保存成指定格式的图片文件，并输出显示给用户。
S506，若针对视频的人脸融合，检测是否所有帧都已处理完。
根据本申请实施例，若是针对视频的人脸融合，所述图像融合设备检测所述所获取的素材中的所有M帧是否已经经过前述步骤S501-S505处理完。如果所述图像融合设备检测到所述所获取的素材中的所有M帧都已处理完，则可以将目标人脸图像数据保存为指定格式的视频文件，并在所述修图类终端应用的界面上呈现最终融合的视频。如果没有处理完，则需将该第N帧的最终获得的目标人脸图像数据写入视频文件中，并对N帧的下一帧N+1帧进行前述步骤S501-S505的处理，依此循环，得到最终融合的视频。
S507，当所有帧都已处理完，则退出系统。
在本申请实施例中，获取当前待融合图像的源人脸图像数据和当前待融合素材的素材配置信息，其中素材配置信息包括素材人脸图像数据、素材肤色数据和素材人脸三维网格，再对源人脸图像数据进行图像识别处理，获取源人脸图像数据对应的源人脸特征点，并根据源人脸特征点生成源人脸图像数据的源人脸三维网格，然后采用素材人脸三维网格和源人脸三维网格进行网格融合生成目标人脸三维网格，最后采用源人脸图像数据的源肤色数据和素材人脸图像数据的素材肤色数据对目标人脸网格进行人脸肤色融合，生成融合后的目标人脸图像数据。通过分析素材人脸三维网格和源人脸三维网格在三维模型的基础上融合为目标人脸三维网格，以及对目标人脸三维网格进行肤色融合生成目标人脸图像数据的过程，提高了最终获得的目标人脸图像数据的真实性；通过分析平均肤色数据在目标人脸图像数据肤色融合中的作用，增加了最终获得的目标人脸图像数据的真实性；通过对生成的目标人脸图像数据的添加真实的光照效果、调整人脸图像数据的显示位置以及填充人脸边缘区域，进一步增加了最终输出的目标人脸图像数据的真实效果；通过采用精确到像素点的肤色融合过程，提高了目标人脸图像数据肤色融合的准确性。
下面将结合附图10-附图14，对本申请实施例提供的图像融合设备进行详细介绍。需要说明的是，附图10-附图14所示的设备，用于执行本申请图2-图9所示实施例的方法，为了便于说明，仅示出了与本申请实施例相关的部分，具体技术细节未揭示的，请参照本申请图2-图9所示的实施例。
请参见图10，为本申请实施例提供的一种图像融合设备的结构示意图。如图10所示，本申请实施例的所述图像融合设备1可以包括：数据获取模块11、源网格生成模块12、目标网格生成模块13和目标数据生成模块14。
数据获取模块11，用于获取当前待融合图像的源人脸图像数据和当前待融合素材的素材配置信息。
具体实现中，图像融合设备1可以获取当前待融合图像的源人脸图像数据和当前待融合素材的素材配置信息。所述源人脸图像数据可以是用户当前通过图像融合设备1所拍的或者从图像融合设备1的相册中选择的照片或者视频中的人脸图像数据。所述当前待融合素材可以是修图类终端应用(例如，某图秀 秀、某某P图、某某相机等)中当前被用户选中的修图所用到的素材模型，例如，动漫人物形象、明星照片等。所述素材配置信息可以包含所述当前待融合素材的3D头像信息(例如，可以是一个obj格式的文件，该文件可以包括素材人脸图像数据、素材肤色数据和素材人脸三维网格等表示素材人脸相关数据的信息)、指示最终所得结果图像中人头效果的信息(该信息可以包括素材3D头像在世界坐标系下的朝向(欧拉角pitch，yaw，roll)、中心位置(最终结果图像的指定位置)、大小(Scale)信息以及与之相匹配的相机信息(例如用透视矩阵来刻画这个信息)等)、2D的贴纸和3D的贴纸以及用户人脸和素材人脸的融合程度alpha(该融合程度可以是每一帧都相同也可以是每帧都不同)。
根据本申请实施例，用于生成目标人脸图像数据的所述素材人脸图像数据和所述源人脸图像数据的大小需要对应相同的尺度，但二者的脸部凹度、胖瘦等情况可以不完全一致，因此，在获取到所述源人脸图像数据和所述素材人脸图像数据后，可以根据所述素材人脸图像数据的大小调整所述源人脸图像数据的大小，使二者大小对应相同的尺度。
源网格生成模块12，用于对所述源人脸图像数据进行图像识别处理，获取所述源人脸图像数据对应的源人脸特征点，并根据所述源人脸特征点生成所述源人脸图像数据的源人脸三维网格。
根据本申请实施例，源网格生成模块12可以对所述源人脸图像数据进行图像识别处理，获取所述源人脸图像数据对应的源人脸特征点。所述图像识别处理可以是采用人脸检测技术(例如，跨平台计算机视觉库OpenCV提供的人脸检测、新型视觉服务平台Face++、优图人脸检测等)对照片中的用户人脸进行识别和五官定位的过程，所述源人脸特征点可以是能够表征所述源人脸图像数据的面部特征(例如，脸部轮廓、眼睛轮廓、鼻子、嘴唇等)的数据点。
根据本申请实施例，所述源网格生成模块12可以对所述源人脸图像数据进行图像识别处理(例如，可以对照片中的用户人脸进行识别和五官定位，从而得到一定数量的基准特征点)，获取所述源人脸图像数据的基准特征点，再对所述基准特征点进行三维深度信息提取，获取所述基准特征点对应的源人脸特征点。所述三维深度信息提取可以是以上述基准特征点为基础，通过匹配标准三维模型的脸部特征点推算出能够反映源人脸图像数据在三维模型中的特征 点的过程。所述基准特征点可以是指示面部特征的基准点，例如，脸部轮廓、眼睛轮廓、鼻子、嘴唇等点，可以是83个基准点，也可以是如图3所示的68个基准点，具体的点数可以有开发人员根据需求而定。所述源人脸特征点可以是在所述基准特征点的基础上进一步深化后能够对应源人脸图像数据三维模型的特征点，例如，通过对上述68个或83个基准点的三维深度信息提取可以获得1000个深化后的源人脸特征点，可以是图4所示的各三角面片的顶点。
进一步的，所述源网格生成模块12可以根据所述源人脸特征点生成所述源人脸图像数据的源人脸三维网格。所述源人脸三维网格可以是所述源人脸图像数据中的人脸对应的3D脸部网格模型，例如图4所示的3D人脸网格或者类似于图4的只有半边脸的3D人脸网格。
目标网格生成模块13，用于采用所述素材人脸三维网格和所述源人脸三维网格进行网格融合生成目标人脸三维网格；
根据本申请实施例，由于所述源人脸图像数据和所述素材人脸图像数据的大小位于同一尺度下，所述源人脸三维网格和所述素材人脸三维网络的尺度也位于同一尺度下。
具体实现中，目标网格生成模块13可以采用所述素材人脸三维网格和所述源人脸三维网格进行网格融合生成目标人脸三维网格。所述素材人脸三维网格可以类似于上述源人脸三维网格可以是所述素材人脸图像数据中人脸对应的3D脸部网格模型，可以是图4所示的3D人脸网格或者是基于图4所示的3D人脸网格在世界坐标系中根据欧拉角改变朝向后的3D人脸网格。所述目标人脸三维网格可以是目标人脸图像数据的3D人脸网格。根据本申请实施例，当素材人脸三维网格是图4所示的3D人脸网格时，最终投影到2D图像上可以呈现正脸的效果，当素材人脸三维网格是基于图4所示的3D人脸网格在世界坐标系中根据欧拉角改变朝向后的3D人脸网格时，最终投影到2D图像上可以呈现侧脸的效果。
根据本申请实施例，所述目标网格生成模块13可以根据所述源人脸三维网格和所述素材人脸三维网格中的人脸特征点计算目标人脸的目标人脸特征点，再根据计算所得的目标人脸特征点生成目标人脸三维网格。例如，3D源人脸三维网格(即用户的3D人脸网格)上有1000个带深度信息的源人脸特征点标记为绿色，3D素材人脸三维网格上有1000个带深度信息的素材人脸特征 点标记为蓝色，用户和素材的1000人脸特征点每个相对应点的平均点(相同位置的对应点取平均，一共1000个点对)标记为红色，最终生成的1000个红色的人脸特征点即为目标人脸特征点，通过上述1000个红色人脸特征点可以组成1900多个三角形，对应的1900多个三角面片所描绘的人脸三维网格即为目标人脸三维网格。根据本申请实施例，所述图像融合设备1可以采用MLS法、仿射变换、图像扭曲等算法将源人脸图像数据和素材人脸图像数据的五官位置趋于上述红色的人脸特征点即目标人脸特征点所指示的五官位置，实现人脸融合的目的。
目标数据生成模块14，用于采用所述源人脸图像数据的源肤色数据和所述素材图像数据的素材肤色数据对所述目标人脸三维网格进行人脸肤色融合，生成融合后的目标人脸图像数据。
根据本申请实施例，生成所述目标人脸三维网格后，所述图像融合设备1需要填充所述目标人脸三维网格中不同三角面片上的肤色才能得到最终的目标人脸图像数据。
具体实现中，目标数据生成模块14可以采用所述源人脸图像数据的源肤色数据和所述素材图像数据的素材肤色数据对所述目标人脸三维网格进行人脸肤色融合，生成融合后的目标人脸图像数据。根据本申请实施例，所述源肤色数据可以是组成所述源人脸图像数据的源像素点的集合，所述素材肤色数据可以是组成所述素材人脸图像数据的素材像素点的集合。
根据本申请实施例，若上述源人脸三维网格与素材人脸三维网格的类型不一致时，需要对所述源人脸三维网格进行网络补充，生成候选人脸三维网格，所述类型可以是所述源人脸三维网格和所述素材人脸三维网格中的网络要素。所述候选人脸三维网格的候选肤色数据可以是对称后的源人脸图像数据，即可以认为所述候选人脸三维网格的候选肤色数据就是所述源肤色数据。所述目标数据生成模块14可以根据上述源像素点、素材像素点和所述素材配置信息中的融合程度(可以是根据经验值进行设置的融合度值，通常取值在0-1之间)计算目标人脸图像数据的目标像素点，进而根据所述目标像素点填充所述目标人脸三维网格生成目标人脸图像数据。例如，设置源人脸三维网格中某三角面片上一个特征点的像素为：UserB、UserG和UserR，素材人脸网格中相应的三角面片上相应位置处的脸特征点的像素为：ResourceB、ResourceG和 ResourceR，目标人脸三维网格中相应的三角面片上相应位置处的特征点的像素为：TargetB、TargetG和TargetR，素材配置信息中的融合程度为：alpha，则有：
TargetB＝(1.0–alpha)*UserB+alpha*ResourceB
TargetG＝(1.0–alpha)*UserG+alpha*ResourceG
TargetR＝(1.0–alpha)*UserR+alpha*ResourceR
从而可以得到目标人脸图像数据的每一个像素值，获得目标人脸图像数据。
根据本申请实施例，最终生成的所述目标人脸图像数据可以是三维人脸图像数据，也可以是二维人脸图像数据。当所述目标人脸图像数据是二维人脸图像数据时，由于生成所述目标图像数据的过程是在三维模型的基础上实现的，考虑到真实光线、阴影等问题，最终形成的目标人脸图像数据效果更逼真。
根据本申请实施例，上述候选人脸三维网格的候选肤色数据可以包括两部分，即与所述源人脸三维网格相匹配部分的候选人脸三维网格的肤色数据为所述源肤色数据，与所述源人脸三维网格不匹配部分的候选人脸三维网格的肤色数据为平均肤色数据，所述平均肤色数据可以是所述源肤色数据经肤色均衡化处理后的肤色数据。根据本申请实施例，所述肤色均衡化处理可以是去掉所述源人脸图像数据中由于光线等原因造成的阴影等效果获得肤色平均值的过程。所述目标数据生成模块14可以根据上述候选肤色数据的候选像素点、素材像素点和所述素材配置信息中的融合程度计算目标人脸图像数据的目标像素点，进而根据所述目标像素点填充所述目标人脸三维网格生成目标人脸图像数据，具体的计算过程与上述计算过程一致此处不再赘述。
在本申请实施例中，通过获取当前待融合图像的源人脸图像数据和当前待融合素材的素材配置信息，其中素材配置信息包括素材人脸图像数据、素材肤色数据和素材人脸三维网格，再对源人脸图像数据进行图像识别处理，获取源人脸图像数据对应的源人脸特征点，并根据源人脸特征点生成源人脸图像数据的源人脸三维网格，然后采用素材人脸三维网格和源人脸三维网格进行网格融合生成目标人脸三维网格，最后采用源人脸图像数据的源肤色数据和素材人脸图像数据的素材肤色数据对目标人脸网格进行人脸肤色融合，生成融合后的目标人脸图像数据。通过分析素材人脸三维网格和源人脸三维网格在三维模型的 基础上融合为目标人脸三维网格，以及对目标人脸三维网格进行肤色融合生成目标人脸图像数据的过程，提高了最终获得的目标人脸图像数据的真实性。
请参见图11，为本申请实施例提供的另一种图像融合设备的结构示意图。如图11所示，本申请实施例的所述图像融合设备1可以包括：数据获取模块11、源网格生成模块12、目标网格生成模块13、目标数据生成模块14、效果添加模块15、位置调整模块16和边缘填充模块17。
数据获取模块11，用于获取当前待融合图像的源人脸图像数据和当前待融合素材的素材配置信息。
具体实现中，数据获取单元11可以获取当前待融合图像的源人脸图像数据和当前待融合素材的素材配置信息。所述源人脸图像数据可以是用户当前通过图像融合设备所拍的或者从图像融合设备的相册中选择的照片或者视频中的人脸图像数据。所述当前待融合素材可以是修图类终端应用(例如，某图秀秀、某某P图、某某相机等)中当前被用户选中的修图所用到的素材模型，例如，动漫人物形象、明星照片等。所述素材配置信息可以包含所述当前待融合素材的3D头像信息(例如，可以是一个obj格式的文件，该文件可以包括素材人脸图像数据、素材肤色数据和素材人脸三维网格等表示素材人脸相关数据的信息)、指示最终所得结果图像中人头效果的信息(该信息可以包括素材3D头像在世界坐标系下的朝向(欧拉角pitch，yaw，roll)、中心位置(最终结果图像的指定位置)、大小(Scale)信息以及与之相匹配的相机信息(例如用透视矩阵来刻画这个信息)等)、2D的贴纸和3D的贴纸以及用户人脸和素材人脸的融合程度alpha(该融合程度可以是每一帧都相同也可以是每帧都不同)。
根据本申请实施例，用于生成目标人脸图像数据的所述素材人脸图像数据和所述源人脸图像数据的大小需要对应相同的尺度，但二者的脸部凹度、胖瘦等情况可以不完全一致，因此，在获取到所述源人脸图像数据和所述素材人脸图像数据后，可以根据所述素材人脸图像数据的大小调整所述源人脸图像数据的大小，使二者大小对应相同的尺度。
源网格生成模块12，用于对所述源人脸图像数据进行图像识别处理，获取所述源人脸图像数据对应的源人脸特征点，并根据所述源人脸特征点生成所述源人脸图像数据的源人脸三维网格。
根据本申请实施例，源网格生成模块12可以对所述源人脸图像数据进行图像识别处理，获取所述源人脸图像数据对应的源人脸特征点，并根据所述源人脸特征点生成所述源人脸图像数据的源人脸三维网格。
请一并参考图12为本申请实施例提供了源网格生成模块的结构示意图。如图12所示，所述源网格生成模块12可以包括：
特征点获取单元121，用于对所述源人脸图像数据进行图像识别处理，获取所述源人脸图像数据的基准特征点。
具体实现中，特征点获取单元121可以对所述源人脸图像数据进行图像识别处理，获取所述源人脸图像数据的基准特征点。所述图像识别处理可以是采用人脸检测技术(例如，跨平台计算机视觉库OpenCV提供的人脸检测、新型视觉服务平台Face++、优图人脸检测等)对照片中的用户人脸进行识别和五官定位的过程。所述基准特征点可以是指示面部特征的基准点，例如，脸部轮廓、眼睛轮廓、鼻子、嘴唇等点，可以是83个基准点，也可以是如图3所示的68个基准点，具体的点数可以有开发人员根据需求而定。
源网格生成单元122，用于对所述基准特征点进行三维深度信息提取，获取所述基准特征点对应的源人脸特征点，根据所述源人脸特征点生成源人脸三维网格。
具体实现中，源网格生成单元122可以对所述基准特征点进行三维深度信息提取，获取所述基准特征点对应的源人脸特征点，根据所述源人脸特征点生成源人脸三维网格。所述三维深度信息提取可以以上述基准特征点为基础，通过匹配标准三维模型的脸部特征点推算出能够反映源人脸图像数据在三维模型中的特征点的过程。所述源人脸特征点可以是在所述基准特征点的基础上进一步深化的点，例如，通过对上述68个或83个基准点的三维深度信息提取可以获得1000个深化后的源人脸特征点，可以是图4中各三角面片的顶点。所述源人脸三维网格可以是所述源人脸图像数据中的人脸对应的3D脸部网格模型，可以是所述源人脸特征点连接而成的用户人脸的3D网格模型，例如图4所示的3D人脸网格或者类似于图4的只有半边脸的3D人脸网格。
目标网格生成模块13，用于采用所述素材人脸三维网格和所述源人脸三维网格进行网格融合生成目标人脸三维网格。
根据本申请实施例，由于所述源人脸图像数据和所述素材人脸图像数据的大小位于同一尺度下，所述源人脸三维网格和所述素材人脸三维网络的尺度也位于同一尺度下。
具体实现中，目标网格生成模块13可以采用所述素材人脸三维网格和所述源人脸三维网格进行网格融合生成目标人脸三维网格。根据本申请实施例，所述素材人脸三维网格可以类似于上述源人脸三维网格可以是所述素材人脸图像数据中人脸对应的3D脸部网格模型，可以是图4所示的3D人脸网格或者是基于图4所示的3D人脸网格在世界坐标系中根据欧拉角改变朝向后的3D人脸网格。所述目标人脸三维网格可以是目标人脸图像数据的3D人脸网格。根据本申请实施例，当素材人脸三维网格是图4所示的3D人脸网格时，最终投影到2D图像上可以呈现正脸的效果，当素材人脸三维网格是基于图4所示的3D人脸网格在世界坐标系中根据欧拉角改变朝向后的3D人脸网格时，最终投影到2D图像上可以呈现侧脸的效果。
在本申请实施例中，所述目标网格生成模块13可以根据所述源人脸三维网格和所述素材人脸三维网格中的人脸特征点计算目标人脸的目标人脸特征点，再根据计算所得的目标人脸特征点生成目标人脸三维网格，例如，3D源人脸三维网格(即用户的3D人脸网格)上有1000个带深度信息的源人脸特征点标记为绿色，3D素材人脸三维网格上有1000个带深度信息的素材人脸特征点标记为蓝色，用户和素材的1000人脸特征点每个相对应点的平均点(相同位置的对应点取平均，一共1000个点对)标记为红色，最终生成的1000个红色的人脸特征点即为目标人脸特征点，通过上述1000个红色人脸特征点可以组成1900多个三角形，对应的1900多个三角面片所描绘的人脸三维网格即为目标人脸三维网格。根据本申请实施例，所述图像融合设备1可以采用MLS法、仿射变换、图像扭曲等算法将源人脸图像数据和素材人脸图像数据的五官位置趋于上述红色的人脸特征点即目标人脸特征点所指示的五官位置，实现人脸融合的目的。
目标数据生成模块14，用于采用所述源人脸图像数据的源肤色数据和所述素材图像数据的素材肤色数据对所述目标人脸三维网格进行人脸肤色融合，生成融合后的目标人脸图像数据。
根据本申请实施例，目标数据生成模块14可以采用所述源人脸图像数据的源肤色数据和所述素材图像数据的素材肤色数据对所述目标人脸三维网格进行人脸肤色融合，生成融合后的目标人脸图像数据。
请一并参考图13，为本申请实施例提供了目标数据生成模块的结构示意图。如图13所示，所述目标数据生成模块14可以包括：
肤色数据获取单元141，用于对所述源人脸图像数据进行肤色均衡化处理，获取所述源人脸图像数据的平均肤色数据。
根据本申请实施例，若上述源人脸三维网格与素材人脸三维网格的类型不一致时，需要对所述源人脸三维网格进行网络补充，生成候选人脸三维网格，所述类型可以是所述源人脸三维网格和所述素材人脸三维网格中的网络要素。所述候选人脸三维网格的候选肤色数据可以是对称后的源人脸图像数据，即可以认为所述候选人脸三维网格的候选肤色数据就是所述源肤色数据。例如，最后生成的候选人脸图像数据为图5A中的人脸图像，则图5A中人脸的肤色与图5B中人脸的肤色一致。
根据本申请实施例，上述候选人脸三维网格的候选肤色数据可以包括两部分，即与所述源人脸三维网格相匹配部分的候选人脸三维网格的肤色数据为所述源肤色数据，与所述源人脸三维网格不匹配部分的候选人脸三维网格的肤色数据为平均肤色数据。具体实现中，肤色数据获取单元141可以对所述源人脸图像数据进行肤色均衡化处理，获取所述源人脸图像数据的平均肤色数据。所述肤色均衡化处理可以是去掉所述源人脸图像数据中由于光线等原因造成的阴影等效果获得肤色平均值的过程。所述平均肤色数据可以是所述源肤色数据去掉阴影后的像素点数据的平均值组成的像素点集合。
候选数据生成单元142，用于基于所述源人脸图像数据的源肤色数据和所述平均肤色数据对所述候选人脸三维网格进行肤色填充，生成候选人脸图像数据。
根据本申请实施例，候选数据生成单元142可以基于所述源人脸图像数据的源肤色数据和所述平均肤色数据对所述候选人脸三维网格进行肤色填充，生 成候选人脸图像数据。所述候选人脸三维网格中与所述源人脸三维网格相匹配部分的肤色数据可以用所述源人脸图像数据的源肤色数据进行填充，所述候选人脸三维网格中与所述源人脸三维网格不匹配部分的肤色数据可以用所述平均肤色数据进行填充。所述候选人脸图像数据的候选肤色数据可以包括所述源肤色数据和所述平均肤色数据，例如，最后生成的候选人脸图像数据为图5A中的人脸图像，则图5A中右边脸的肤色为图5B中人脸的肤色，图5A中左边脸的肤色为图5B中人脸肤色平均化后的肤色。
目标数据生成单元143，用于采用所述候选人脸图像数据的候选肤色数据和所述素材图像数据的素材肤色数据对所述目标人脸三维网格进行人脸肤色融合，生成融合后的目标人脸图像数据。
根据本申请实施例，生成所述目标人脸三维网格后，所述图像融合设备1需要填充所述目标人脸三维网格中不同三角面片上的肤色才能得到最终的目标人脸图像数据。
根据本申请实施例，目标数据生成单元143可以采用所述候选人脸图像数据的候选肤色数据和所述素材图像数据的素材肤色数据对所述目标人脸三维网格进行人脸肤色融合，生成融合后的目标人脸图像数据。所述候选肤色数据可以是组成所述候选人脸图像数据的候选像素点的集合，所述素材肤色数据可以是组成所述素材人脸图像数据的素材像素点的集合。
根据本申请实施例，所述目标数据生成单元143可以基于所述候选像素点和所述素材像素点，并采用融合程度计算目标像素点，根据所述目标像素点生成目标人脸图像数据。所述融合程度(可以是根据经验值进行设置的融合度值，通常取值在0-1之间。
根据本申请实施例，可以设置候选人脸三维网格中某三角面片上一个特征点的像素为：CandidateB、CandidateG和CandidateR，素材人脸三维网格中相应的三角面片上相应位置处的脸特征点的像素为：ResourceB、ResourceG和ResourceR，目标人脸三维网格中相应的三角面片上相应位置处的特征点的像素为：TargetB、TargetG和TargetR，素材配置信息中的融合程度为：alpha，则有：
TargetB＝(1.0–alpha)*CandidateB+alpha*ResourceB
TargetG＝(1.0–alpha)*CandidateG+alpha*ResourceG
TargetR＝(1.0–alpha)*CandidateR+alpha*ResourceR
从而可以得到目标人脸图像数据的每一个像素值，获得目标人脸图像数据。
根据本申请实施例，通过分析平均肤色数据在目标人脸图像数据肤色融合中的作用，增加了最终获得的目标人脸图像数据的真实性。
效果添加模块15，用于根据所述源人脸图像数据的源肤色数据获取所述源人脸图像数据对应的光源类型，并采用所述光源类型对应的光照效果对所述目标人脸图像数据进行效果添加处理。
根据本申请实施例，所述目标人脸图像数据生成后，效果添加模块15可以根据所述源人脸图像数据的源肤色数据获取所述源人脸图像数据对应的光源类型。所述效果添加模块15可以通过比较所述平均肤色和所述源人脸图像数据中各个区域的肤色，获取亮与平均肤色的区域和暗与平均肤色的区域，进而推算出光源类型，例如是多个点光源或者面光源。所述效果添加模块15也可以通过深度学习搜集源人脸图像数据在不同指定光照情况下的结果图，然后将结果图和对应的光照情况作为深度神经网络((Deep Neural Networks，DNN)的训练数据，训练出一个给定图片就能输出光源类型和光源位置的DNN模型。
进一步的，所述效果添加模块15可以采用所述光源类型对应的光照效果对所述目标人脸图像数据进行效果添加处理，例如，所述源人脸图像数据的对应的光源类型是来自左侧脸方向的点光源，则所述图像融合设备可以在最终获得的目标人脸图像数据中添加左脸方向点光源的光照效果。
在本申请实施例中，所述效果添加模块15还可以将上述素材配置信息中的2D和3D贴纸按照其对应的位置和层级(zOrder)粘贴在所述目标人脸图像数据上，例如，将3D眼镜贴纸佩戴在生成的目标人脸图像数据的人脸上。
位置调整模块16，用于基于所述素材人脸图像数据所指示的坐标信息调整所述目标人脸图像数据当前的显示位置。
根据本申请实施例，所述目标人脸图像数据生成后，位置调整模块16可以基于所述素材人脸图像数据所指示的坐标信息调整所述目标人脸图像数据当前的显示位置，例如，根据上述素材配置信息中的素材人脸图像数据所指示的坐标信息(包括，欧拉方向和中心点)和素材大小将所得的目标人脸图像数据放在指定的位置上。
边缘填充模块17，用于当所述目标人脸图像数据的第一显示区域小于所述源人脸图像数据的第二显示区域时，对所述第二显示区域中除所述第一显示区域的部分进行人脸边缘填充处理。
根据本申请实施例，当最终生成的目标人脸图像数据的第一显示区域小于所述源人脸图像数据的第二显示区域时，边缘填充模块17可以对所述第二显示区域中除所述第一显示区域的部分进行人脸边缘填充处理，所述第一显示区域可以是所述目标人脸图像数据映射在2D屏幕上的范围，所述第二显示区域可以是所述源人脸图像数据映射在2D屏幕上的范围，所述第一显示区域小于所述第二显示区域可以表述为所述目标人脸图像数据的人脸比所述源人脸图像数据的人脸小。所述人脸边缘填充处理可以是采用填充算法(例如，OpenCV提供的图像修复算法Inpainting)将所述第二显示区域中除所述第一显示区域的部分填充满。
进一步的，所述图像融合设备1对所述目标人脸图像数据的附加技术效果处理完之后，可以对最终获得的目标人脸图像数据进行输出显示。
根据本申请实施例，在执行效果添加模块15、位置调整模块16和边缘填充模块17后最终生成的所述目标人脸图像数据可以是三维人脸图像数据，也可以是二维人脸图像数据。当所述目标人脸图像数据是二维人脸图像数据时，由于生成所述目标图像数据的过程是在三维模型的基础上实现的，考虑到真实光线、阴影等问题，最终成线的目标人脸图像数据效果更逼真。
需要说明的是，在执行效果添加模块15、位置调整模块16和边缘填充模块17时，可以选择其中的一个或多模块同时执行。
在本申请实施例中，通过对生成的目标人脸图像数据的添加真实的光照效果、调整人脸图像数据的显示位置以及填充人脸边缘区域，进一步增加了最终输出的目标人脸图像数据的真实效果。
在本申请实施例一种具体实现方式中，如图14所示所述目标数据生成单元可以包括：
像素点获取子单元1431，用于获取所述候选肤色数据中的候选像素点和所述素材肤色数据中的素材像素点。
根据本申请实施例，所述候选肤色数据可以是组成上述候选人脸图像数据的候选像素点的集合，所述素材肤色数据可以是组成上述素材人脸图像数据的 素材像素点的集合。像素点获取子单元1431可以获取所述候选肤色数据中的候选像素点和所述素材肤色数据中的素材像素点。
目标数据生成子单元1432，用于基于所述候选像素点和所述素材像素点，并采用融合程度计算目标像素点，根据所述目标像素点生成目标人脸图像数据。
具体实现中，目标数据生成子单元1432基于所述候选像素点和所述素材像素点，并采用融合程度获取目标人脸图像数据的具体过程可以参见方法实施例中的描述，此处不再赘述。
根据本申请实施例，通过采用精确到像素点的肤色融合过程，提高了目标人脸图像数据肤色融合的准确性。
根据本申请实施例，通过获取当前待融合图像的源人脸图像数据和当前待融合素材的素材配置信息，其中素材配置信息包括素材人脸图像数据、素材肤色数据和素材人脸三维网格，再对源人脸图像数据进行图像识别处理，获取源人脸图像数据对应的源人脸特征点，并根据源人脸特征点生成源人脸图像数据的源人脸三维网格，然后采用素材人脸三维网格和源人脸三维网格进行网格融合生成目标人脸三维网格，最后采用源人脸图像数据的源肤色数据和素材人脸图像数据的素材肤色数据对目标人脸网格进行人脸肤色融合，生成融合后的目标人脸图像数据。通过分析素材人脸三维网格和源人脸三维网格在三维模型的基础上融合为目标人脸三维网格，以及对目标人脸三维网格进行肤色融合生成目标人脸图像数据的过程，提高了最终获得的目标人脸图像数据的真实性；通过分析平均肤色数据在目标人脸图像数据肤色融合中的作用，增加了最终获得的目标人脸图像数据的真实性；通过对生成的目标人脸图像数据的添加真实的光照效果、调整人脸图像数据的显示位置以及填充人脸边缘区域，进一步增加了最终输出的目标人脸图像数据的真实效果；通过采用精确到像素点的肤色融合过程，提高了目标人脸图像数据肤色融合的准确性。
需要说明的是，在2D模型的基础上进行人脸融合时，通常存在用户图像数据的人脸角度与素材图像数据的人脸角度不完全匹配的情况。例如，用户图像是半边脸，素材图像是正脸或者用户人头向左转，素材人头向右转等。当存在上述情况时，现有技术中的人脸融合算法所能获取到的用户人脸的信息较 少，在进行人脸融合时较少的用户人脸信息将影响最终的匹配结果，导致生成的目标结果图像的真实性较差。
为了解决上述问题，本申请实施例提供了另一种图像融合设备的结构示意图，具体可以参见图10所示的结构示意图可以包括：数据获取模块11、源网格生成模块12、目标网格生成模块13和目标数据生成模块14。
数据获取模块11，用于获取当前待融合图像的源人脸图像数据和当前待融合素材的素材配置信息。
具体实现中，数据获取模块11获取所述源人脸图像数据和所述素材配置信息的过程可以参见上述方法实施例的具体描述，此处不再赘述。
源网格生成模块12，用于对所述源人脸图像数据进行图像识别处理，获取所述源人脸图像数据对应的源人脸特征点，并根据所述源人脸特征点生成所述源人脸图像数据的源人脸三维网格。
具体实现中，源网格生成模块12生成所述源人脸三维网格的过程可以参见上述方法实施例的具体描述，此处不再赘述。
目标网格生成模块13，用于当检测到所述源人脸三维网格与所述素材人脸三维网格的类型不一致时，根据所述源人脸图像数据的对称性对所述源人脸三维网格进行网格补充，生成与所述素材人脸三维网格类型一致的候选人脸三维网格，并采用所述候选人脸三维网格和所述素材人脸三维网格进行网格融合生成目标人脸三维网格。
具体实现中，当检测到所述源人脸三维网格与所述素材人脸三维网格的类型不一致时，所述目标网格生成模块13可以根据所述源人脸图像数据的对称性对所述源人脸三维网格进行网格补充，生成与所述素材人脸三维网格类型一致的候选人脸三维网格，并采用所述候选人脸三维网格和所述素材人脸三维网格进行网格融合生成目标人脸三维网格。
根据本申请实施例，由于所述源人脸图像数据和所述素材人脸图像数据的大小位于同一尺度下，所述源人脸三维网格和所述素材人脸三维网络的尺度也位于同一尺度下。
根据本申请实施例，所述素材人脸三维网格可以类似于上述源人脸三维网格可以是所述素材人脸图像数据中人脸对应的3D脸部网格模型，可以是图4所示的3D人脸网格或者是基于图4所示的3D人脸网格在世界坐标系中根据欧 拉角改变朝向后的3D人脸网格。所述目标人脸三维网格可以是目标人脸图像数据的3D人脸网格。当素材人脸三维网格是图4所示的3D人脸网格时，最终投影到2D图像上可以呈现正脸的效果，当素材人脸三维网格是基于图4所示的3D人脸网格在世界坐标系中根据欧拉角改变朝向后的3D人脸网格时，最终投影到2D图像上可以呈现侧脸的效果。
根据本申请实施例，所述源人脸三维网格和所述素材人脸三维网格只有在类型一致时，才能根据融合算法生成与二者脸部显示区域一样且真实性较好的目标人脸三维网格。所述类型一致可以是所述源人脸三维网格和所述素材人脸三维网格中的网络要素一致，所述网格要素可以是所述源人脸三维网格和所述素材人脸三维网格的网格朝向或者网格显示区域。例如，所述源人脸三维网格指示的脸部显示区域和所述素材人脸三维网格指示的脸部显示区域一致，二者都是类似于图5A所示的标准正脸或者类似于图5B所示的侧脸时，可以认为所述源人脸三维网格和所述素材人脸三维网格类型一致。当所述源人脸网格和所述素材人脸网格的类型不一致时(例如，源人脸三维网格为类似于图5B所示的侧脸，素材人脸三维网格为类似于图5A所示的标准正脸)，所述图像融合设备1可以先对所述源人脸网格进行网格补充。
具体实现中，当所述源人脸三维网格与所述素材人脸三维网格的类型不一致时，所述目标网格生成模块13可以根据所述源人脸图像数据的对称性对所述源人脸三维网格进行网格补充，生成与所述素材人脸三维网格类型一致的候选人脸三维网格。正常情况下人的脸部都是对称的(忽略细微的差别)，当所述源人脸图像数据为侧脸图像数据时(例如，图5B中的图像)，所述图像融合设备提取所述源人脸特征点后生成的源人脸三维网格也是针对侧脸的，此时，所述目标网格生成模块13可以根据脸部对称的原则将该源人脸图像数据补充为标准正脸(例如，由图5B的图像补充后得到图5A的图像)，然后再根据补充后的源人脸图像数据生成与所述素材人脸三维网格类型一致的候选人脸三维网格(图5A对应的人脸三维网格即是候选人脸三维网格)。
根据本申请实施例，所述目标网格生成模块13根据脸部对称原则对源人脸图像数据(侧脸图像数据)补充后得到的标准正脸图像数据的左右脸部表情一致，进一步得到的候选人脸三维网格的左右脸部表情也是一致的。
在本申请实施例中，当所述素材人脸三维网格为左右脸部表情不一致的标准正脸，而源人脸图像数据为侧脸图像数据时，根据源人脸图像数据的对称性对源人脸三维网格进行网格补充后得到的候选人脸三维网格的左右脸部表情，与素材人脸三维网格的左右脸部表情是不一致的(例如，素材人脸图像数据所指示素材人脸的左眼是张开的，右眼是闭着的，而只包含左侧脸的源人脸图像数据指示的用户只人脸的左眼是张开的，此时，根据源人脸图像数据的对称性补充后的源人脸三维网格的右眼也是张开的，与素材人脸三维网格的右眼不一致)，此时，所述目标网格生成模块13将无法采用该候选人脸三维网格和上述素材人脸三维网格进行网格融合生成目标人脸三维网格。对上述种情况，所述目标网格生成模块13可以通过表情迁移算法将候选人脸三维网格的左右脸部表情调整至与素材人脸三维网格的表情一致，使最终得到的候选人脸三维网格与上述素材人脸三维网格的面部表情一致。
进一步的，所述目标网格生成模块13可以采用所述候选人脸三维网格和所述素材人脸三维网格进行网格融合生成目标人脸三维网格。所述候选人脸三维网格与所述素材人脸三维网格的类型是一致的，即在相同脸部位置上所述素材人脸三维网格和所述候选人脸三维网格都有相应的特征点，例如，候选人脸三维网格中包含两只眼睛眼角的特征点，素材人脸三维网格也包含两只眼睛眼角的特征点。所述目标人脸三维网格可以是目标人脸图像数据的3D人脸网格。
在本申请实施例中，所述目标网格生成模块13可以根据所述候选人脸三维网格和所述素材人脸三维网格中的人脸特征点计算目标人脸的目标人脸特征点，再根据计算所得的目标人脸特征点生成目标人脸三维网格，例如，3D候选人脸三维网格(即用户的3D人脸网格)上有1000个带深度信息的源人脸特征点标记为绿色，3D素材人脸三维网格上有1000个带深度信息的素材人脸特征点标记为蓝色，用户和素材的1000人脸特征点每个相对应点的平均点(相同位置的对应点取平均，一共1000个点对)标记为红色，最终生成的1000个红色的人脸特征点即为目标人脸特征点，通过上述1000个红色人脸特征点可以组成1900多个三角形，对应的1900多个三角面片所描绘的人脸三维网格即为目标人脸三维网格。根据本申请实施例，所述图像融合设备1可以采用移动最小二乘的图像变形法MLS、仿射变换、图像扭曲等算法将源人脸图像数据和 素材人脸图像数据的五官位置趋于上述红色的人脸特征点即目标人脸特征点所指示的五官位置，实现人脸融合的目的。
目标数据生成模块14，用于采用所述源人脸图像数据的源肤色数据和所述素材人脸图像数据的素材肤色数据对所述目标人脸网格进行人脸肤色融合，生成融合后的目标人脸图像数据；
具体实现中，目标数据生成模块14生成所述目标人脸图像数据的过程可以参见上述方法实施例的具体描述，此处不再赘述。
本申请实施例还提供了一种计算机存储介质，所述计算机存储介质可以存储有多条指令，所述指令适于由处理器加载并执行如上述图2-图9所示实施例的方法步骤，具体执行过程可以参见图2-图9所示实施例的具体说明，在此不进行赘述。
请参见图15，为本申请实施例提供了一种终端的结构示意图。如图15所示，所述终端1000可以包括：至少一个处理器1001，例如CPU，至少一个网络接口1004，用户接口1003，存储器1005，至少一个通信总线1002。其中，通信总线1002用于实现这些组件之间的连接通信。其中，用户接口1003可以包括显示屏(Display)、键盘(Keyboard)，可选用户接口1003还可以包括标准的有线接口、无线接口。网络接口1004例如可以包括标准的有线接口、无线接口(如WI-FI接口)。存储器1005可以是高速RAM存储器，也可以是非不稳定的存储器(non-volatile memory)，例如至少一个磁盘存储器。存储器1005例如还可以是至少一个位于远离前述处理器1001的存储装置。如图15所示，作为一种计算机存储介质的存储器1005中可以包括操作系统、网络通信模块、用户接口模块以及人脸融合应用程序。
在图15所示的终端1000中，用户接口1003主要用于为用户提供输入的接口，获取用户输入的数据；网络接口1004用于用户终端进行数据通信；而处理器1001可以用于调用存储器1005中存储的人脸融合应用程序，并具体执行以下操作：
获取当前待融合图像的源人脸图像数据和当前待融合素材的素材配置信息，所述素材配置信息包括素材人脸图像数据、素材肤色数据和素材人脸三维网格；
对所述源人脸图像数据进行图像识别处理，获取所述源人脸图像数据对应的源人脸特征点，并根据所述源人脸特征点生成所述源人脸图像数据的源人脸三维网格；
采用所述素材人脸三维网格和所述源人脸三维网格进行网格融合生成目标人脸三维网格；
采用所述源人脸图像数据的源肤色数据和所述素材人脸图像数据的素材肤色数据对所述目标人脸网格进行人脸肤色融合，生成融合后的目标人脸图像数据。
在一个实施例中，所述处理器1001在执行对所述源人脸图像数据进行图像识别处理，获取所述源人脸图像数据对应的源人脸特征点，并根据所述源人脸特征点生成所述源人脸图像数据的源人脸三维网格时，具体执行以下操作：
对所述源人脸图像数据进行图像识别处理，获取所述源人脸图像数据的基准特征点；
对所述基准特征点进行三维深度信息提取，获取所述基准特征点对应的源人脸特征点，根据所述源人脸特征点生成源人脸三维网格。
在一个实施例中，所述处理器1001在执行采用所述素材人脸三维网格和所述源人脸三维网格进行网格融合生成目标人脸三维网格时，具体执行以下操作：
当检测所述源人脸三维网格与所述素材人脸三维网格的类型不一致时，根据所述源人脸图像数据的对称性对所述源人脸三维网格进行网格补充，生成与所述素材人脸网格类型一致的候选人脸三维网格，并采用所述候选人脸三维网格和所述素材人脸三维网格进行网格融合生成目标人脸三维网格。
在一个实施例中，所述处理器1001在执行采用所述源人脸图像数据的源肤色数据和所述素材图像数据的素材肤色数据对所述目标人脸三维网格进行人脸肤色融合，生成融合后的目标人脸图像数据时，具体执行以下操作：
对所述源人脸图像数据进行肤色均衡化处理，获取所述源人脸图像数据的平均肤色数据；
基于所述源人脸图像数据的源肤色数据和所述平均肤色数据对所述候选人脸三维网格进行肤色填充，生成候选人脸图像数据；
采用所述候选人脸图像数据的候选肤色数据和所述素材图像数据的素材肤色数据对所述目标人脸三维网格进行人脸肤色融合，生成融合后的目标人脸图像数据；
所述候选肤色数据包括所述源肤色数据和所述平均肤色数据。
在一个实施例中，所述处理器1001在执行采用所述候选人脸图像数据的候选肤色数据和所述素材图像数据的素材肤色数据对所述目标人脸三维网格进行人脸肤色融合，生成融合后的目标人脸图像数据时，具体执行以下操作：
获取所述候选肤色数据中的候选像素点和所述素材肤色数据中的素材像素点；
基于所述候选像素点和所述素材像素点，并采用融合程度计算目标像素点，根据所述目标像素点生成目标人脸图像数据。
在一个实施例中，所述处理器1001，还用于执行以下操作：
根据所述源人脸图像数据的源肤色数据获取所述源人脸图像数据对应的光源类型，并采用所述光源类型对应的光照效果对所述目标人脸图像数据进行效果添加处理。
在一个实施例中，所述处理器1001，还用于执行以下操作：
基于所述素材人脸图像数据所指示的坐标信息调整所述目标人脸图像数据当前的显示位置。
在一个实施例中，所述处理器1001，还用于执行以下操作：
当所述目标人脸图像数据的第一显示区域小于所述源人脸图像数据的第二显示区域时，对所述第二显示区域中除所述第一显示区域的部分进行人脸边缘填充处理。
在本申请实施例中，通过获取当前待融合图像的源人脸图像数据和当前待融合素材的素材配置信息，其中素材配置信息包括素材人脸图像数据、素材肤色数据和素材人脸三维网格，再对源人脸图像数据进行图像识别处理，获取源人脸图像数据对应的源人脸特征点，并根据源人脸特征点生成源人脸图像数据的源人脸三维网格，然后采用素材人脸三维网格和源人脸三维网格进行网格融合生成目标人脸三维网格，最后采用源人脸图像数据的源肤色数据和素材人脸 图像数据的素材肤色数据对目标人脸网格进行人脸肤色融合，生成融合后的目标人脸图像数据。通过分析素材人脸三维网格和源人脸三维网格在三维模型的基础上融合为目标人脸三维网格，以及对目标人脸三维网格进行肤色融合生成目标人脸图像数据的过程，提高了最终获得的目标人脸图像数据的真实性；通过分析平均肤色数据在目标人脸图像数据肤色融合中的作用，增加了最终获得的目标人脸图像数据的真实性；通过对生成的目标人脸图像数据的添加真实的光照效果、调整人脸图像数据的显示位置以及填充人脸边缘区域，进一步增加了最终输出的目标人脸图像数据的真实效果；通过采用精确到像素点的肤色融合过程，提高了目标人脸图像数据肤色融合的准确性。
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，是可以通过计算机程序来指令相关的硬件来完成，所述的程序可存储于计算机可读取存储介质中，该程序在执行时，可包括如上述各方法的实施例的流程。其中，所述的存储介质可为磁碟、光盘、只读存储记忆体(Read-Only Memory，ROM)或随机存储记忆体(Random Access Memory，RAM)等。
以上所揭露的仅为本申请的实施例而已，当然不能以此来限定本申请之权利范围，因此依本申请权利要求所作的等同变化，仍属本申请所涵盖的范围。

Claims (15)

  1. 一种图像融合方法，用于终端上，所述终端包括：处理器和存储器，其特征在于，所述方法包括：
    获取当前待融合图像的源人脸图像数据和当前待融合素材的素材配置信息，所述素材配置信息包括素材人脸图像数据、素材肤色数据和素材人脸三维网格；
    对所述源人脸图像数据进行图像识别处理，获取所述源人脸图像数据对应的源人脸特征点，并根据所述源人脸特征点生成所述源人脸图像数据的源人脸三维网格；
    采用所述素材人脸三维网格和所述源人脸三维网格进行网格融合生成目标人脸三维网格；
    采用所述源人脸图像数据的源肤色数据和所述素材人脸图像数据的素材肤色数据对所述目标人脸网格进行人脸肤色融合，生成融合后的目标人脸图像数据。
  2. 如权利要求1所述的方法，其特征在于，所述对所述源人脸图像数据进行图像识别处理，获取所述源人脸图像数据对应的源人脸特征点，并根据所述源人脸特征点生成所述源人脸图像数据的源人脸三维网格，包括：
    对所述源人脸图像数据进行图像识别处理，获取所述源人脸图像数据的基准特征点；
    对所述基准特征点进行三维深度信息提取，获取所述基准特征点对应的源人脸特征点，根据所述源人脸特征点生成源人脸三维网格。
  3. 如权利要求1所述的方法，其特征在于，所述采用所述素材人脸三维网格和所述源人脸三维网格进行网格融合生成目标人脸三维网格，包括：
    当检测到所述源人脸三维网格与所述素材人脸三维网格的类型不一致时，根据所述源人脸图像数据的对称性对所述源人脸三维网格进行网格补充，生成与所述素材人脸网格类型一致的候选人脸三维网格，并采用所述候选人脸三维网格和所述素材人脸三维网格进行网格融合生成目标人脸三维网格。
  4. 如权利要求3所述的方法，其特征在于，所述采用所述源人脸图像数据的源肤色数据和所述素材图像数据的素材肤色数据对所述目标人脸三维网格进行人脸肤色融合，生成融合后的目标人脸图像数据，包括：
    对所述源人脸图像数据进行肤色均衡化处理，获取所述源人脸图像数据的平均肤色数据；
    基于所述源人脸图像数据的源肤色数据和所述平均肤色数据对所述候选人脸三维网格进行肤色填充，生成候选人脸图像数据；
    采用所述候选人脸图像数据的候选肤色数据和所述素材图像数据的素材肤色数据对所述目标人脸三维网格进行人脸肤色融合，生成融合后的目标人脸图像数据；
    所述候选肤色数据包括所述源肤色数据和所述平均肤色数据。
  5. 如权利要求4所述的方法，其特征在于，所述采用所述候选人脸图像数据的候选肤色数据和所述素材图像数据的素材肤色数据对所述目标人脸三维网格进行人脸肤色融合，生成融合后的目标人脸图像数据，包括：
    获取所述候选肤色数据中的候选像素点和所述素材肤色数据中的素材像素点；
    基于所述候选像素点和所述素材像素点，并采用融合程度计算目标像素点，根据所述目标像素点生成目标人脸图像数据。
  6. 如权利要求1所述的方法，其特征在于，还包括：
    根据所述源人脸图像数据的源肤色数据获取所述源人脸图像数据对应的光源类型，并采用所述光源类型对应的光照效果对所述目标人脸图像数据进行效果添加处理。
  7. 如权利要求1所述的方法，其特征在于，还包括：
    基于所述素材人脸图像数据所指示的坐标信息调整所述目标人脸图像数据当前的显示位置。
  8. 如权利要求1所述的方法，其特征在于，还包括：
    当所述目标人脸图像数据的第一显示区域小于所述源人脸图像数据的第二显示区域时，对所述第二显示区域中除所述第一显示区域的部分进行人脸边缘填充处理。
  9. 一种图像融合设备，其特征在于，包括：处理器和存储器，所述存储器存储有计算机程序，所述计算机程序由所述处理器加载并执行以下步骤：
    获取当前待融合图像的源人脸图像数据和当前待融合素材的素材配置信息，所述素材配置信息包括素材人脸图像数据、素材肤色数据、素材人脸特征点和素材人脸三维网格；
    对所述源人脸图像数据进行图像识别处理，获取所述源人脸图像数据对应的源人脸特征点，并根据所述源人脸特征点生成所述源人脸图像数据的源人脸三维网格；
    采用所述素材人脸三维网格和所述源人脸三维网格进行网格融合生成目标人脸三维网格；
    采用所述源人脸图像数据的源肤色数据和所述素材人脸图像数据的素材肤色数据对所述目标人脸网格进行人脸肤色融合，生成融合后的目标人脸图像数据。
  10. 如权利要求9所述的设备，其特征在于，所述对所述源人脸图像数据进行图像识别处理，获取所述源人脸图像数据对应的源人脸特征点，并根据所述源人脸特征点生成所述源人脸图像数据的源人脸三维网格，包括：
    对所述源人脸图像数据进行图像识别处理，获取所述源人脸图像数据的基准特征点；
    对所述基准特征点进行三维深度信息提取，获取所述基准特征点对应的源人脸特征点，根据所述源人脸特征点生成源人脸三维网格。
  11. 如权利要求9所述的设备，其特征在于，所述采用所述素材人脸三维网格和所述源人脸三维网格进行网格融合生成目标人脸三维网格，包括：
    当检测到所述源人脸三维网格与所述素材人脸三维网格的类型不一致时， 根据所述源人脸图像数据的对称性对所述源人脸三维网格进行网格补充，生成与所述素材人脸三维网格类型一致的候选人脸三维网格，并采用所述候选人脸三维网格和所述素材人脸三维网格进行网格融合生成目标人脸三维网格。
  12. 如权利要求11所述的设备，其特征在于，所述采用所述源人脸图像数据的源肤色数据和所述素材图像数据的素材肤色数据对所述目标人脸三维网格进行人脸肤色融合，生成融合后的目标人脸图像数据，包括：
    对所述源人脸图像数据进行肤色均衡化处理，获取所述源人脸图像数据的平均肤色数据；
    基于所述源人脸图像数据的源肤色数据和所述平均肤色数据对所述候选人脸三维网格进行肤色填充，生成候选人脸图像数据；
    采用所述候选人脸图像数据的候选肤色数据和所述素材图像数据的素材肤色数据对所述目标人脸三维网格进行人脸肤色融合，生成融合后的目标人脸图像数据；
    所述候选肤色数据包括所述源肤色数据和所述平均肤色数据。
  13. 如权利要求12所述的设备，其特征在于，所述采用所述候选人脸图像数据的候选肤色数据和所述素材图像数据的素材肤色数据对所述目标人脸三维网格进行人脸肤色融合，生成融合后的目标人脸图像数据，包括：
    获取所述候选肤色数据中的候选像素点和所述素材肤色数据中的素材像素点；
    基于所述候选像素点和所述素材像素点，并采用融合程度计算目标像素点，根据所述目标像素点生成目标人脸图像数据。
  14. 一种计算机存储介质，其特征在于，所述计算机存储介质存储有多条指令，所述指令适于由处理器加载并执行如权利要求1～8任意一项的方法步骤。
  15. 一种终端，其特征在于，包括：处理器和存储器；其中，所述存储器存储有计算机程序，所述计算机程序适于由所述处理器加载并执行以下步骤：
    获取当前待融合图像的源人脸图像数据和当前待融合素材的素材配置信息，所述素材配置信息包括素材人脸图像数据、素材肤色数据和素材人脸三维网格；
    对所述源人脸图像数据进行图像识别处理，获取所述源人脸图像数据对应的源人脸特征点，并根据所述源人脸特征点生成所述源人脸图像数据的源人脸三维网格；
    采用所述素材人脸三维网格和所述源人脸三维网格进行网格融合生成目标人脸三维网格；
    采用所述源人脸图像数据的源肤色数据和所述素材人脸图像数据的素材肤色数据对所述目标人脸网格进行人脸肤色融合，生成融合后的目标人脸图像数据。
PCT/CN2018/116832 2017-11-22 2018-11-22 一种图像融合方法及其设备、存储介质、终端 Ceased WO2019101113A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US16/859,331 US11037281B2 (en) 2017-11-22 2020-04-27 Image fusion method and device, storage medium and terminal

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201711173149.X 2017-11-22
CN201711173149.XA CN107993216B (zh) 2017-11-22 2017-11-22 一种图像融合方法及其设备、存储介质、终端

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/859,331 Continuation US11037281B2 (en) 2017-11-22 2020-04-27 Image fusion method and device, storage medium and terminal

Publications (1)

Publication Number Publication Date
WO2019101113A1 true WO2019101113A1 (zh) 2019-05-31

Family

ID=62032635

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/116832 Ceased WO2019101113A1 (zh) 2017-11-22 2018-11-22 一种图像融合方法及其设备、存储介质、终端

Country Status (3)

Country Link
US (1) US11037281B2 (zh)
CN (1) CN107993216B (zh)
WO (1) WO2019101113A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111243093A (zh) * 2020-01-07 2020-06-05 腾讯科技（深圳）有限公司 三维人脸网格的生成方法、装置、设备及存储介质
KR20220083830A (ko) * 2020-06-01 2022-06-20 텐센트 테크놀로지(센젠) 컴퍼니 리미티드 이미지 처리 방법 및 이미지 합성 방법, 이미지 처리 장치 및 이미지 합성 장치, 그리고 저장 매체

Families Citing this family (25)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107993216B (zh) 2017-11-22 2022-12-20 腾讯科技（深圳）有限公司 一种图像融合方法及其设备、存储介质、终端
CN108682030B (zh) * 2018-05-21 2022-04-26 北京微播视界科技有限公司 脸皮替换方法、装置及计算机设备
CN110533761B (zh) * 2018-05-23 2024-01-12 华硕电脑股份有限公司 影像显示方法、电子装置及非瞬时电脑可读取记录媒体
CN108898551B (zh) * 2018-06-14 2020-07-31 北京微播视界科技有限公司 图像合并的方法和装置
CN109118579A (zh) * 2018-08-03 2019-01-01 北京微播视界科技有限公司 动态生成人脸三维模型的方法、装置、电子设备
CN109785228B (zh) * 2018-12-29 2021-03-16 广州方硅信息技术有限公司 图像处理方法、装置、存储介质和服务器
CN110619262B (zh) * 2019-04-17 2023-09-01 深圳爱莫科技有限公司 图像识别的方法及装置
CN110084775B (zh) * 2019-05-09 2021-11-26 深圳市商汤科技有限公司 图像处理方法及装置、电子设备和存储介质
CN110838084B (zh) * 2019-09-24 2023-10-17 咪咕文化科技有限公司 一种图像的风格转移方法、装置、电子设备及存储介质
JP7102554B2 (ja) * 2019-09-30 2022-07-19 ベイジン・センスタイム・テクノロジー・デベロップメント・カンパニー・リミテッド 画像処理方法、装置及び電子機器
CN110992493B (zh) * 2019-11-21 2023-10-31 北京达佳互联信息技术有限公司 图像处理方法、装置、电子设备及存储介质
CN111083373B (zh) * 2019-12-27 2021-11-16 恒信东方文化股份有限公司 一种大屏幕及其智能拍照方法
CN111210510B (zh) * 2020-01-16 2021-08-06 腾讯科技（深圳）有限公司 三维人脸模型生成方法、装置、计算机设备及存储介质
CN111507890B (zh) * 2020-04-13 2022-04-19 北京字节跳动网络技术有限公司 图像处理方法、装置、电子设备及计算机可读存储介质
CN113763517B (zh) * 2020-06-05 2024-04-12 华为技术有限公司 人脸表情编辑方法及电子设备
CN111768479B (zh) * 2020-07-29 2021-05-28 腾讯科技（深圳）有限公司 图像处理方法、装置、计算机设备以及存储介质
US11677897B2 (en) 2020-10-19 2023-06-13 Adobe Inc. Generating stylized images in real time on mobile devices
CN112257657B (zh) * 2020-11-11 2024-02-27 网易（杭州）网络有限公司 脸部图像融合方法及装置、存储介质、电子设备
WO2022192992A1 (en) * 2021-03-18 2022-09-22 Digital Domain Virtual Human (Us), Inc. Methods and systems for markerless facial motion capture
CN113222841A (zh) * 2021-05-08 2021-08-06 北京字跳网络技术有限公司 一种图像处理方法、装置、设备及介质
CN113609960B (zh) * 2021-08-03 2023-07-28 北京奇艺世纪科技有限公司 一种目标图片的人脸驱动方法及装置
CN114119923B (zh) * 2021-11-29 2022-07-19 浙江大学 三维人脸重建方法、装置以及电子设备
US20230368471A1 (en) * 2022-05-13 2023-11-16 Joseph Isaac Gratz Method and system for converting 2-d video into a 3-d rendering with enhanced functionality
US20240005581A1 (en) * 2022-06-30 2024-01-04 Vidalign Inc. Generating 3d facial models & animations using computer vision architectures
CN115358958B (zh) * 2022-08-26 2024-11-26 北京字跳网络技术有限公司 特效图生成方法、装置、设备及存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105184249A (zh) * 2015-08-28 2015-12-23 百度在线网络技术（北京）有限公司 用于人脸图像处理的方法和装置
CN107146199A (zh) * 2017-05-02 2017-09-08 厦门美图之家科技有限公司 一种人脸图像的融合方法、装置及计算设备
CN107230181A (zh) * 2017-06-05 2017-10-03 厦门美柚信息科技有限公司 实现人脸图像融合的方法及装置
CN107993216A (zh) * 2017-11-22 2018-05-04 腾讯科技（深圳）有限公司 一种图像融合方法及其设备、存储介质、终端

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8391642B1 (en) * 2008-05-12 2013-03-05 Hewlett-Packard Development Company, L.P. Method and system for creating a custom image
US20130215113A1 (en) * 2012-02-21 2013-08-22 Mixamo, Inc. Systems and methods for animating the faces of 3d characters using images of human faces
US9314692B2 (en) * 2012-09-21 2016-04-19 Luxand, Inc. Method of creating avatar from user submitted image
CN104978764B (zh) * 2014-04-10 2017-11-17 华为技术有限公司 三维人脸网格模型处理方法和设备
US9552510B2 (en) * 2015-03-18 2017-01-24 Adobe Systems Incorporated Facial expression capture for character animation
CN106156730B (zh) * 2016-06-30 2019-03-15 腾讯科技（深圳）有限公司 一种人脸图像的合成方法和装置

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105184249A (zh) * 2015-08-28 2015-12-23 百度在线网络技术（北京）有限公司 用于人脸图像处理的方法和装置
CN107146199A (zh) * 2017-05-02 2017-09-08 厦门美图之家科技有限公司 一种人脸图像的融合方法、装置及计算设备
CN107230181A (zh) * 2017-06-05 2017-10-03 厦门美柚信息科技有限公司 实现人脸图像融合的方法及装置
CN107993216A (zh) * 2017-11-22 2018-05-04 腾讯科技（深圳）有限公司 一种图像融合方法及其设备、存储介质、终端

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111243093A (zh) * 2020-01-07 2020-06-05 腾讯科技（深圳）有限公司 三维人脸网格的生成方法、装置、设备及存储介质
CN111243093B (zh) * 2020-01-07 2023-05-12 腾讯科技（深圳）有限公司 三维人脸网格的生成方法、装置、设备及存储介质
KR20220083830A (ko) * 2020-06-01 2022-06-20 텐센트 테크놀로지(센젠) 컴퍼니 리미티드 이미지 처리 방법 및 이미지 합성 방법, 이미지 처리 장치 및 이미지 합성 장치, 그리고 저장 매체
EP4030342A4 (en) * 2020-06-01 2022-12-28 Tencent Technology (Shenzhen) Company Limited IMAGE PROCESSING METHOD AND IMAGE SYNTHESIS METHOD, IMAGE PROCESSING APPARATUS AND IMAGE SYNTHESIS APPARATUS, AND STORAGE MEDIA
JP2023517121A (ja) * 2020-06-01 2023-04-21 ▲騰▼▲訊▼科技（深▲セン▼）有限公司 画像処理及び画像合成方法、装置及びコンピュータプログラム
JP7483301B2 (ja) 2020-06-01 2024-05-15 ▲騰▼▲訊▼科技（深▲セン▼）有限公司 画像処理及び画像合成方法、装置及びコンピュータプログラム
US12277747B2 (en) 2020-06-01 2025-04-15 Tencent Technology (Shenzhen) Company Limited Method and apparatus for image processing and image synthesis for obtaining a target face image, and computer-readable storage medium
KR102839409B1 (ko) 2020-06-01 2025-07-25 텐센트 테크놀로지(센젠) 컴퍼니 리미티드 이미지 처리 방법 및 이미지 합성 방법, 이미지 처리 장치 및 이미지 합성 장치, 그리고 저장 매체

Also Published As

Publication number Publication date
US20200258206A1 (en) 2020-08-13
US11037281B2 (en) 2021-06-15
CN107993216A (zh) 2018-05-04
CN107993216B (zh) 2022-12-20

Similar Documents

Publication Publication Date Title
WO2019101113A1 (zh) 一种图像融合方法及其设备、存储介质、终端
US12293465B2 (en) Method and apparatus for constructing three-dimensional face mesh, device, and storage medium
EP4283566B1 (en) Single image 3d photography with soft-layering and depth-aware inpainting
CN113301259B (zh) 引导用户捕获数字图像的计算机可读介质、系统和方法
EP4036863B1 (en) Human body model reconstruction method and reconstruction system, and storage medium
CN109903217B (zh) 图像变形方法和装置
JP7566028B2 (ja) 多様なポートレートから照明を学習すること
CN107507217B (zh) 证件照的制作方法、装置及存储介质
CN107944420B (zh) 人脸图像的光照处理方法和装置
WO2018103244A1 (zh) 直播视频处理方法、装置及电子设备
CN108537881A (zh) 一种人脸模型处理方法及其设备、存储介质
US10872457B1 (en) Facial texture map generation using single color image and depth information
CN113628327A (zh) 一种头部三维重建方法及设备
US12211189B2 (en) Systems and methods for lightweight machine learning for image illumination control
WO2018188277A1 (zh) 视线校正方法、装置、智能会议终端及存储介质
US20240412448A1 (en) Object rendering
CN107851309A (zh) 一种图像增强方法及装置
CN114581987A (zh) 图像处理方法、装置、电子设备及存储介质
CN110493512B (zh) 摄影构图方法、装置、摄影设备、电子装置及存储介质
CN115623313A (zh) 图像处理方法、图像处理装置、电子设备、存储介质
US12051150B2 (en) Computer implemented method and system for classifying an input image for new view synthesis in a 3D visual effect, and non-transitory computer readable storage medium
CN118799439A (zh) 数字人图像融合方法、装置、设备及可读存储介质
CN113240573B (zh) 局部和全局并行学习的高分辨率图像风格变换方法及系统
RU2831513C1 (ru) Способ, компьютерное устройство и считываемый компьютером носитель для получения изображения лица с добавленным освещением
Guo et al. Efficient view manipulation for cuboid-structured images

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18881841

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18881841

Country of ref document: EP

Kind code of ref document: A1