WO2024149124A1 - 一种虚拟角色人脸处理方法、装置、设备、存储介质及产品 - Google Patents

一种虚拟角色人脸处理方法、装置、设备、存储介质及产品 Download PDF

Info

Publication number
WO2024149124A1
WO2024149124A1 PCT/CN2024/070204 CN2024070204W WO2024149124A1 WO 2024149124 A1 WO2024149124 A1 WO 2024149124A1 CN 2024070204 W CN2024070204 W CN 2024070204W WO 2024149124 A1 WO2024149124 A1 WO 2024149124A1
Authority
WO
WIPO (PCT)
Prior art keywords
face
virtual character
dimensional
key point
model
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2024/070204
Other languages
English (en)
French (fr)
Inventor
杨云
吴腾
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Bigo Technology Pte Ltd
Original Assignee
Bigo Technology Pte Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Bigo Technology Pte Ltd filed Critical Bigo Technology Pte Ltd
Publication of WO2024149124A1 publication Critical patent/WO2024149124A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Definitions

  • the embodiments of the present application relate to the field of image processing technology, and in particular to a method, device, equipment, storage medium and product for processing a virtual character's face.
  • the traditional face-pinching method is to set a drag bar on the face-pinching control panel to adjust the facial features of the standard face model.
  • the user adjusts the facial features of the standard face model by dragging the bar to generate the virtual character expected by the user.
  • the face processing operation of such a virtual character is relatively complicated and cumbersome, and generally requires professional technicians to achieve satisfactory results. It is difficult for ordinary users to generate the expected virtual character, and the face processing effect of the virtual character is poor.
  • the embodiments of the present application provide a virtual character face processing method, device, equipment, storage medium and product to solve the technical problems in the related art that the virtual character face processing operation is relatively complex and cumbersome, and the virtual character face processing effect is poor, simplify the face processing operation, and improve the virtual character face processing effect.
  • an embodiment of the present application provides a method for processing a virtual character face, comprising:
  • a virtual face model is generated based on the face deformation parameters and a set standard face model, wherein the standard face model is a three-dimensional deformable face model.
  • an embodiment of the present application provides a virtual character face processing device, including an image acquisition module, a parameter acquisition module and a face generation module, wherein:
  • the image acquisition module is configured to acquire a two-dimensional expressionless face image
  • the parameter acquisition module is configured to input the two-dimensional expressionless face image into a trained deformation parameter model, analyze and process the two-dimensional expressionless face image through the deformation parameter model, and obtain face deformation parameters reflecting the facial features in the two-dimensional expressionless face image;
  • the face generation module is configured to generate a virtual face model based on the face deformation parameters and a set standard face model, wherein the standard face model is a three-dimensional deformable face model.
  • an embodiment of the present application provides a virtual character face processing device, including: a memory and one or more processors;
  • the memory is used to store one or more programs
  • the one or more processors When the one or more programs are executed by the one or more processors, the one or more processors implement the virtual character face processing method as described in the first aspect.
  • an embodiment of the present application provides a non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are used to execute the virtual character face processing method as described in the first aspect.
  • an embodiment of the present application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor of the device reads and executes the computer program from the computer-readable storage medium, so that the device performs the virtual character face processing method as described in the first aspect.
  • the embodiment of the present application obtains a two-dimensional expressionless facial image, uses a deformation parameter model to determine the facial deformation parameters of the two-dimensional expressionless facial image, and generates a virtual facial model based on the facial deformation parameters and a set standard facial model.
  • the user only needs to upload the two-dimensional expressionless facial image to adjust the three-dimensional deformable standard facial model according to the facial deformation parameters to obtain a virtual facial model that reflects the facial features in the two-dimensional expressionless facial image, thereby simplifying the virtual character face pinching operation and improving the virtual character face processing effect.
  • FIG1 is a flow chart of a method for processing a virtual character face provided in an embodiment of the present application
  • FIG2 is a schematic diagram of a registration process of a key point set provided in an embodiment of the present application.
  • FIG3 is a schematic diagram of a registration result provided in an embodiment of the present application.
  • FIG4 is a schematic diagram of the structure of a virtual character face processing device provided in an embodiment of the present application.
  • FIG5 is a schematic diagram of the structure of a virtual character face processing device provided in an embodiment of the present application.
  • the virtual character face processing method provided by the present application can be applied to the face pinching scenes of virtual characters such as live broadcasts and games.
  • the anchor when the anchor wants to create a virtual character with its own characteristics to interact with the audience, the anchor can automatically generate a virtual face model by providing a two-dimensional expressionless face image, aiming to determine the face deformation parameters of the two-dimensional expressionless face image through a deformation parameter model, and generate a virtual face model according to the face deformation parameters and the set standard face model, simplifying the face processing operation and improving the face processing effect of the virtual character.
  • the user For the traditional virtual character face processing scheme, the user generally adjusts the facial features of the standard face model through the facial feature drag bar on the face pinching control panel to generate the virtual character expected by the user, or a professional (such as a professional face pinching master) generates a virtual character through a long face pinching operation, and there is a technical problem that the face processing operation is relatively complicated and cumbersome, and the face processing effect of the virtual character is poor.
  • a virtual character face processing method of an embodiment of the present application is provided to solve the technical problems that the operation of the existing virtual character face processing scheme is relatively complicated and cumbersome, and the face processing effect of the virtual character is poor.
  • FIG1 is a flowchart of a virtual character face processing method provided in an embodiment of the present application.
  • the virtual character face processing method provided in an embodiment of the present application can be performed by a virtual character face processing device.
  • the virtual character face processing device can be implemented by hardware and/or software and integrated in the virtual character face processing equipment.
  • the virtual character face processing method includes:
  • the two-dimensional expressionless facial image provided by this solution is a frontal expressionless two-dimensional facial image provided by a user who needs to pinch the face of a virtual character.
  • the two-dimensional expressionless facial image can be obtained by photographing a camera device, or downloaded from a download address specified by the user, or obtained from a local storage location (such as a photo album).
  • a two-dimensional expressionless face image provided by a user is obtained.
  • a virtual character face processing device such as a mobile phone, computer, etc.
  • a virtual face model reflecting the facial features in the two-dimensional expressionless face image will be generated based on the two-dimensional expressionless face image provided by the user.
  • S102 Inputting the two-dimensional expressionless face image into the trained deformation parameter model, analyzing and processing the two-dimensional expressionless face image through the deformation parameter model, and obtaining face deformation parameters reflecting the facial features in the two-dimensional expressionless face image.
  • a trained deformation parameter model is pre-configured in the virtual character face processing device, and the deformation parameter model can be used to analyze and process the provided two-dimensional expressionless face image, and output the face deformation parameters reflecting the face features in the two-dimensional expressionless face image.
  • the face deformation parameters can be the shape basis vectors used to set the three-dimensional deformable face model (3DMM, 3D Morphable model).
  • the acquired two-dimensional expressionless facial image is input into the deformation parameter model.
  • the deformation parameter model analyzes and processes the two-dimensional expressionless facial image and outputs corresponding facial deformation parameters.
  • the deformation parameter model provided by the present solution can be trained based on two-dimensional expressionless face sample images and face sample deformation parameters, wherein the face sample image parameters are determined based on the registration results of the face sample three-dimensional key point set and the virtual character key point set.
  • the deformation parameter model provided by this solution can be built based on a convolutional neural network.
  • the backbone network of the deformation parameter model can be built based on a Resnet18 network or a MobileNet network.
  • the last layer of the deformation parameter model is set to a sigmoid layer to ensure that the element value of the face deformation parameter output by the deformation parameter model is between 0 and 1.
  • the face sample deformation parameters may reflect the face features of the sample user's face in the two-dimensional expressionless face sample image, and the face sample image parameters are determined based on the registration result of the face sample three-dimensional key point set of the corresponding sample user and the virtual character key point set of the sample user.
  • the virtual character key point set of the sample user is a set of three-dimensional key points of the face sample on the sample virtual character, and the sample virtual character may be obtained by a professional face pincher performing a face pinching operation based on the face features of the sample user.
  • two-dimensional expressionless face sample images and three-dimensional key point sets of face samples of multiple sample users are collected by an image acquisition device.
  • multiple two-dimensional expressionless face sample images under different shooting environments are collected.
  • different shooting environments can be different combinations of different lighting conditions, indoor and outdoor scenes, and shooting distances, so as to improve the sample diversity of two-dimensional expressionless face sample images.
  • 100 two-dimensional expressionless face sample images covering different lighting conditions, outdoor and indoor scenes, and different shooting distances are collected for each sample user.
  • each sample user corresponds to a collection of a face sample 3D key point set
  • the face sample 3D key point set records a series of point sets consisting of 3D coordinates (i.e., face sample 3D key points) on the face image of the sample user.
  • the face sample 3D key point set can be expressed as: Where m is the number of key points in the three-dimensional key point set of the face sample, which can be determined according to the acquisition accuracy of the image acquisition device. It can be understood that It can be understood as the 3D coordinates (x 1 , y 1 , z 1 ) of the 3D key point of the face sample numbered 1 in the 3D key point set of the face sample.
  • the image acquisition device shoots the face images of the sample users from the front to obtain two-dimensional expressionless face sample images.
  • the image acquisition device shoots the face images of the sample users from the front to obtain two-dimensional expressionless face sample images.
  • a virtual character key point set of each sample user is obtained. It is understandable that since the number of key points in the virtual character key point set generated by a professional face-pinching artist based on the face-pinching operation may be different from the number of key points in the three-dimensional key point set of the face sample collected by the image acquisition device, it is necessary to perform registration processing on the three-dimensional key point set of the face sample and the virtual character key point set. Assume that the virtual character key point set is represented as Where n is the number of key points of the virtual character. It needs to be explained that the three-dimensional key point set of the face sample collected by the image acquisition device is relatively sparse compared to the key point set of the virtual character. The number of key points of the three-dimensional key point set of the face sample is about hundreds or thousands, and the number of key points of the key point set of the virtual character is generally more than ten thousand, that is, n is greater than m.
  • the transformation parameter T corresponding to the registration result may include a scaling factor, a rotation matrix, and a translation vector.
  • the face sample image parameters of each sample user are determined according to the registration result.
  • the face sample image parameters can be determined based on the corresponding transformation parameters and the face sample 3D key point set.
  • the virtual face key points with the corresponding key point number set in the finally generated virtual face model can be made to overlap with the face sample 3D key points in the 3D space.
  • the deformation parameter model is trained using the collected two-dimensional expressionless face sample images and deformation parameters of the face samples of multiple sample users as training data, and the trained deformation parameter model is configured in the virtual character face processing device.
  • the deformation parameter model is trained with the two-dimensional expressionless face sample images as input and the corresponding face sample deformation parameters as output until the set loss function requirements are met.
  • This solution determines the face sample image parameters based on the registration results of the face sample three-dimensional key point set and the virtual character key point set, and trains the deformation parameter model using the collected two-dimensional expressionless face sample images and face sample deformation parameters.
  • the deformation parameter model can accurately analyze the face deformation parameters corresponding to the input two-dimensional image, thereby improving the face processing effect and face pinching efficiency of the virtual character face.
  • ⁇ pred is the deformation parameter model that analyzes and processes the two-dimensional expressionless face sample image and predicts the output face deformation parameters
  • ⁇ true is the face sample deformation parameter corresponding to the two-dimensional expressionless face sample image.
  • the deformation parameter model can be continuously adjusted by the gradient descent method until the value of the loss function is lower than the set loss function threshold. After the loss function stops converging, the deformation parameter model is saved and configured in the application corresponding to the virtual character face processing device.
  • this solution when registering the three-dimensional key point set of the face sample and the key point set of the virtual character, this solution can be based on the following registration formula:
  • s * is the scaling factor to be solved
  • R * is the rotation matrix to be solved
  • t * is the translation vector to be solved
  • s is the scaling factor of the tuning process
  • R is the rotation matrix of the tuning process
  • t is the translation vector of the tuning process
  • Pt is the set of key points of the virtual character
  • the combination of the whole scale factor s, the rotation matrix R and the translation vector t that minimizes the value on the right side of the above formula is determined by continuously adjusting the scale factor s, the rotation matrix R and the translation vector t.
  • This part can be understood as a process of transforming the key point set of the virtual character based on the transformation parameter T, where the scaling factor s is a scalar, the rotation matrix R is a 3x3 orthogonal matrix, and the translation vector t is a 3D translation vector.
  • the registration and tuning process of the three-dimensional key point set of the face sample and the key point set of the virtual character based on the registration formula can be solved based on the iterative closest point (ICP) algorithm.
  • ICP iterative closest point
  • the present solution when registering the three-dimensional key point set of a face sample and the key point set of a virtual character, the present solution may include steps S1001-S1003:
  • S1002 Based on the second number of key points, the three-dimensional key point set of the face sample and the key point set of the virtual character are aligned to obtain a second scaling factor, a second rotation matrix and a second translation vector, and the second number of key points is greater than the first number of key points.
  • S1003 Based on the second scaling factor, the second rotation matrix and the second translation vector, determine a set of key point numbers in the virtual character key point set that matches the three-dimensional key point set of the face sample.
  • the registration operation of the face sample 3D key point set and the virtual character key point set in this scheme includes two steps: coarse registration and fine registration, wherein the coarse registration is based on the first key point number, and the fine registration is based on the second key point number, and the second key point number is greater than the first key point number.
  • the coarse registration can be understood as a relatively rough registration process when the transformation between the two point clouds is completely unknown, with the purpose of providing a better transformation initial value for the fine registration.
  • the first key point quantity of face sample three-dimensional key points is determined from the face sample three-dimensional key point set, and the face sample three-dimensional key point set and the virtual character key point set are registered to obtain a rough registration result, that is, to determine the first scaling factor, the first rotation matrix and the first translation vector.
  • the rough registration process can be performed based on the registration formula provided above.
  • the three-dimensional key points of the face sample of the second number of key points are determined from the three-dimensional key point set of the face sample, and on the basis of the first scaling factor, the first rotation matrix and the first translation vector determined above, the three-dimensional key point set of the face sample and the key point set of the virtual character are registered to obtain a precise registration result, i.e., the second scaling factor, the second rotation matrix and the second translation vector.
  • the first scaling factor may be fixed (i.e., the second scaling factor determined by precise registration is the same as the first scaling factor), the first rotation matrix and the first translation vector are used as the initial values of the second rotation matrix and the second translation vector, and the second rotation matrix and the second translation vector are continuously optimized based on the registration formula to obtain the final registration result, at which time, the second scaling factor, the second rotation matrix and the second translation vector are the scaling factor, the rotation matrix and the translation vector corresponding to the registration result.
  • a key point number set that matches the face sample three-dimensional key point set in the virtual character key point set is determined. For example, each face sample three-dimensional key point in the face sample three-dimensional key point set is traversed, and the virtual character key point in the virtual character key point set that is closest to the face sample three-dimensional key point (such as Euclidean distance, Hamilton distance, etc.) is determined, and the key point numbers of these virtual character key points in the virtual character key point set are determined, and the key point number set can be obtained based on these key point numbers.
  • This scheme uses a registration method that performs coarse registration and fine registration on the face sample three-dimensional key point set and the virtual character key point set in sequence, while improving the registration efficiency, effectively improving the registration effect, obtaining a more accurate scaling factor, rotation matrix and translation vector, and determining an accurate key point number set, effectively improving the face processing effect of the virtual character.
  • the figure illustrates the relative positions of the three-dimensional key point set of the face sample and the key point set of the virtual character after the three-dimensional key point set of the face sample and the key point set of the virtual character are registered, wherein the relatively dense key points are the virtual character key points, and the relatively sparse key points are the three-dimensional key points of the face sample (for the convenience of display, the three-dimensional key points of the face sample are coarser than the virtual character key points).
  • the three-dimensional key points of the face sample can basically match the virtual character
  • the key points overlap, and the three-dimensional key points of the face sample can find the key points that correspond one-to-one and are closest to the key points of the virtual character (which can be recorded through the key point number set).
  • the face sample image parameters provided by this solution may be determined based on the following deformation parameter determination formula:
  • ⁇ * is the face sample image parameter to be solved
  • ⁇ is the face sample image parameter in the tuning process
  • s is the second scaling factor
  • R is the second rotation matrix
  • t is the second translation vector
  • Ps is the set of three-dimensional key points of the face sample.
  • the solution process of the deformation parameter determination formula is the calculation process of the least squares optimization problem, and the final face sample image parameters can be solved by gradient descent or Newton iteration method.
  • the two-dimensional expressionless face sample image and the solved face sample image parameters can be saved as a binary data file, which will be used as sample data to train the deformation parameter model.
  • This solution calculates the face sample image parameters more accurately through the deformation parameter determination formula, effectively improving the face processing effect of the virtual character.
  • S103 Generate a virtual face model based on the face deformation parameters and a set standard face model, where the standard face model is a three-dimensional deformable face model.
  • a standard face model is pre-configured in a virtual character face processing device, wherein the standard face model provided by this solution is a three-dimensional deformable face model.
  • the three-dimensional deformable face model can adjust the standard face model according to given face deformation parameters to obtain a virtual face model of the face features reflected by the given face deformation parameters.
  • the set standard facial model can be adjusted according to the facial deformation parameters to generate a virtual facial model.
  • the generation of the virtual facial model based on the facial deformation parameters and the set standard facial model can be performed by a set image processing engine (such as Unreal Engine), and the virtual facial model is rendered by the image processing engine.
  • the present solution may perform linear weighted processing on the set standard face model and the face shape vector of the standard face model based on the face deformation parameters to obtain the virtual face model.
  • the standard face model provided by this solution is a three-dimensional deformable face model, which can be obtained by a linear weighted combination of a standard face and face deformation parameters (shape basis vectors).
  • the face image and face deformation parameters are submitted to the set image processing engine, which performs linear weighted processing on the standard face model and the face shape vector of the standard face model according to the face deformation parameters to obtain and render a virtual face model, which records the user's facial features.
  • This solution obtains a virtual face model that reflects the user's facial features by performing linear weighted processing on the standard face model and the face shape vector based on the face deformation parameters.
  • the generated virtual face model is closer to the user's image, and while simplifying the face processing operation, it improves the face processing effect.
  • the user only needs to upload a two-dimensional expressionless face image to generate a virtual face model, effectively lowering the threshold for face processing.
  • the virtual face model provided by the present solution may be determined based on the following face model generation formula:
  • S model is a virtual face model
  • s i is the face shape vector of the i-th dimension in the standard face model
  • ⁇ i is the face deformation parameter of the i-th dimension
  • ⁇ is the face deformation parameter of the 106-th dimension.
  • Standard face model and the i-th dimension face shape vector si are both n ⁇ 3 vector matrices
  • n is the number of face key points of the virtual face model
  • 3 is the coordinate dimension
  • the value range of the i-th dimension face deformation parameter is 0 ⁇ 1, and its value does not represent the semantic features of the face.
  • the standard face model and the i-th dimension face shape vector si are constants
  • the i-th dimension face deformation parameter ⁇ i is the independent variable
  • the virtual face model Smodel is the dependent variable. It needs to be explained that although the virtual face model Smodel is an n ⁇ 3 matrix, it is related to the virtual character key point set The meanings of are the same, both refer to the 3D coordinate information of n facial key points.
  • This solution uses the facial model generation formula to perform linear weighted processing on the standard facial model and the facial shape vector to obtain a virtual facial model that better fits the user's facial features, effectively improving the facial processing effect of the virtual character.
  • the user only needs to upload the two-dimensional expressionless facial image to adjust the three-dimensional deformable standard facial model according to the facial deformation parameters to obtain a virtual facial model that reflects the facial features in the two-dimensional expressionless facial image, thereby simplifying the virtual character face pinching operation and improving the virtual character face processing effect.
  • FIG4 is a schematic diagram of the structure of a virtual character face processing device provided in an embodiment of the present application.
  • the virtual character face processing device includes an image acquisition module 41, a parameter acquisition module 42 and a human face processing module 43. Face generation module 43.
  • the image acquisition module 41 is configured to acquire a two-dimensional expressionless face image
  • the parameter acquisition module 42 is configured to input the two-dimensional expressionless face image into a trained deformation parameter model, analyze and process the two-dimensional expressionless face image through the deformation parameter model, and obtain face deformation parameters that reflect the facial features in the two-dimensional expressionless face image
  • the face generation module 43 is configured to generate a virtual face model based on the face deformation parameters and a set standard face model, and the standard face model is a three-dimensional deformable face model.
  • the user only needs to upload the two-dimensional expressionless face image to adjust the three-dimensional deformable standard face model according to the face deformation parameters to obtain a virtual face model that reflects the facial features in the two-dimensional expressionless face image, thereby simplifying the virtual character face pinching operation and improving the virtual character face processing effect.
  • the deformation parameter model is trained based on the two-dimensional expressionless face sample image and the face sample deformation parameters, and the face sample image parameters are determined based on the registration results of the face sample three-dimensional key point set and the virtual character key point set.
  • the registration of the three-dimensional key point set of the face sample and the key point set of the virtual character includes:
  • the three-dimensional key point set of the face sample and the key point set of the virtual character are registered to obtain a first scaling factor, a first rotation matrix and a first translation vector;
  • the three-dimensional key point set of the face sample and the key point set of the virtual character are registered to obtain a second scaling factor, a second rotation matrix and a second translation vector, wherein the second number of key points is greater than the first number of key points;
  • a set of key point numbers that matches the three-dimensional key point set of the face sample in the virtual character key point set is determined.
  • the registration of the three-dimensional key point set of the face sample and the key point set of the virtual character is performed based on the following formula:
  • s * is the scaling factor to be solved
  • R * is the rotation matrix to be solved
  • t * is the translation vector to be solved
  • s is the scaling factor of the tuning process
  • R is the rotation matrix of the tuning process
  • t is the translation vector of the tuning process
  • Pt is the set of key points of the virtual character
  • the i-th 3D key point set Ps of the face sample 3D key points of individual face samples, is the i-th virtual character key point of the virtual character key point set P t .
  • the face sample image parameters are determined based on the following formula:
  • ⁇ * is the face sample image parameter to be solved
  • ⁇ is the face sample image parameter in the tuning process
  • s is the second scaling factor
  • R is the second rotation matrix
  • t is the second translation vector
  • Ps is the three-dimensional key point set of the face sample.
  • the face generation module 43 when the face generation module 43 generates a virtual face model based on the face deformation parameters and the set standard face model, it is configured to perform linear weighted processing on the set standard face model and the face shape vector of the standard face model based on the face deformation parameters to obtain the virtual face model.
  • the virtual face model is determined based on the following formula:
  • S model is a virtual face model
  • si is the i-th dimension face shape vector in the standard face model
  • ⁇ i is the i-th dimension face deformation parameter.
  • the embodiment of the present application also provides a virtual character face processing device, which can integrate the virtual character face processing apparatus provided in the embodiment of the present application.
  • Figure 5 is a structural diagram of a virtual character face processing device provided in the embodiment of the present application.
  • the virtual character face processing device includes: an input device 53, an output device 54, a memory 52 and one or more processors 51; the memory 52 is used to store one or more programs; when one or more programs are executed by one or more processors 51, the one or more processors 51 implement the virtual character face processing method provided in the above embodiment.
  • the virtual character face processing apparatus, device and computer provided above can be used to execute the virtual character face processing method provided in any of the above embodiments, and have corresponding functions and beneficial effects.
  • the present application also provides a non-volatile storage medium storing computer executable instructions.
  • the computer executable instructions are used to execute the virtual character face processing method provided in the above embodiment when executed by the computer processor.
  • the non-volatile storage medium storing computer executable instructions provided in the embodiment of the present application whose computer executable instructions are not limited to the virtual character face processing method provided above, can also execute the related operations in the virtual character face processing method provided in any embodiment of the present application.
  • the virtual character face processing device, equipment and storage medium provided in the above embodiment can execute the virtual character face processing method provided in any embodiment of the present application.
  • the embodiments of the present application also provide a computer program product.
  • the technical solution of the present application is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product.
  • the computer program product is stored in a storage medium, including a number of instructions for enabling a computer device, a mobile terminal or a processor therein to execute all or part of the steps of the virtual character face processing method provided in each embodiment of the present application.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • General Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Evolutionary Computation (AREA)
  • Databases & Information Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Artificial Intelligence (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Human Computer Interaction (AREA)
  • Processing Or Creating Images (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

本申请实施例提供了一种虚拟角色人脸处理方法、装置、设备、存储介质及产品。本申请实施例提供的技术方案通过获取二维无表情人脸图像，利用形变参数模型确定二维无表情人脸图像的人脸形变参数，并根据人脸形变参数和设定的标准人脸模型生成虚拟人脸模型，用户只需要上传二维无表情人脸图像，即可根据人脸形变参数对三维可形变的标准人脸模型进行调整，得到反映二维无表情人脸图像中人脸特征的虚拟人脸模型，在简化虚拟角色捏脸操作的同时，提高虚拟角色人脸处理效果。

Description

一种虚拟角色人脸处理方法、装置、设备、存储介质及产品
本申请要求在2023年01月12日提交中国专利局，申请号为202310071200.5的中国专利申请的优先权，该申请的全部内容通过引用结合在本申请中。
技术领域
本申请实施例涉及图像处理技术领域，尤其涉及一种虚拟角色人脸处理方法、装置、设备、存储介质及产品。
背景技术
随着图像处理技术的发展，用户经常使用脸部特征与自己相似的虚拟角色来与其他用户进行互动，用户一般通过对标准人脸模型进行捏脸的方式生成脸部特征与自己相似的虚拟角色。
目前传统的捏脸方式是在捏脸控制面板上设置用于调整标准人脸模型的脸部特征的拖动条，用户通过拖动条调整标准人脸模型的脸部特征，从而生成用户预期的虚拟角色。但是这种虚拟角色的人脸处理操作较为复杂和繁琐，一般需要专业的技术人员才能实现满意的效果，普通用户难以生成预期的虚拟角色，虚拟角色的人脸处理效果较差。
发明内容
本申请实施例提供一种虚拟角色人脸处理方法、装置、设备、存储介质及产品，以解决相关技术中虚拟角色的人脸处理操作较为复杂和繁琐，虚拟角色的人脸处理效果较差的技术问题，简化人脸处理操作，提高虚拟角色的人脸处理效果。
在第一方面，本申请实施例提供了一种虚拟角色人脸处理方法，包括：
获取二维无表情人脸图像；
将所述二维无表情人脸图像输入至训练好的形变参数模型，通过所述形变参数模型对所述二维无表情人脸图像进行分析处理，得到反映所述二维无表情人脸图像中人脸特征的人脸形变参数；
基于所述人脸形变参数和设定的标准人脸模型生成虚拟人脸模型，所述标准人脸模型为三维可形变人脸模型。
在第二方面，本申请实施例提供了一种虚拟角色人脸处理装置，包括图像获取模块、参数获取模块和人脸生成模块，其中：
所述图像获取模块，配置为获取二维无表情人脸图像；
所述参数获取模块，配置为将所述二维无表情人脸图像输入至训练好的形变参数模型，通过所述形变参数模型对所述二维无表情人脸图像进行分析处理，得到反映所述二维无表情人脸图像中人脸特征的人脸形变参数；
所述人脸生成模块，配置为基于所述人脸形变参数和设定的标准人脸模型生成虚拟人脸模型，所述标准人脸模型为三维可形变人脸模型。
在第三方面，本申请实施例提供了一种虚拟角色人脸处理设备，包括：存储器以及一个或多个处理器；
所述存储器，用于存储一个或多个程序；
当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现如第一方面所述的虚拟角色人脸处理方法。
在第四方面，本申请实施例提供了一种存储计算机可执行指令的非易失性存储介质，所述计算机可执行指令在由计算机处理器执行时用于执行如第一方面所述的虚拟角色人脸处理方法。
在第五方面，本申请实施例提供了一种计算机程序产品，该计算机程序产品包括计算机程序，该计算机程序存储在计算机可读存储介质中，设备的至少一个处理器从计算机可读存储介质读取并执行计算机程序，使得设备执行如第一方面所述的虚拟角色人脸处理方法。
本申请实施例通过获取二维无表情人脸图像，利用形变参数模型确定二维无表情人脸图像的人脸形变参数，并根据人脸形变参数和设定的标准人脸模型生成虚拟人脸模型，用户只需要上传二维无表情人脸图像，即可根据人脸形变参数对三维可形变的标准人脸模型进行调整，得到反映二维无表情人脸图像中人脸特征的虚拟人脸模型，在简化虚拟角色捏脸操作的同时，提高虚拟角色人脸处理效果。
附图说明
图1是本申请实施例提供的一种虚拟角色人脸处理方法的流程图；
图2是本申请实施例提供的一种关键点集合的配准流程示意图；
图3是本申请实施例提供的一种配准结果示意图；
图4是本申请实施例提供的一种虚拟角色人脸处理装置的结构示意图；
图5是本申请实施例提供的一种虚拟角色人脸处理设备的结构示意图。
具体实施方式
为了使本申请的目的、技术方案和优点更加清楚，下面结合附图对本申请具体实施例作进一步的详细描述。可以理解的是，此处所描述的具体实施例仅仅用于解释本申请，而非对本申请的限定。另外还需要说明的是，为了便于描述，附图中仅示出了与本申请相关的部分而非全部内容。在更加详细地讨论示例性实施例之前应当提到的是，一些示例性实施例被描述成作为流程图描绘的处理或方法。虽然流程图将各项操作(或步骤)描述成顺序的处理，但是其中的许多操作可以被并行地、并发地或者同时实施。此外，各项操作的顺序可以被重新安排。当其操作完成时上述处理可以被终止，但是还可以具有未包括在附图中的附加步骤。上述处理可以对应于方法、函数、规程、子例程、子程序等等。
本申请提供的虚拟角色人脸处理方法可应用于直播、游戏等虚拟角色的捏脸场景，例如在直播场景中，在主播希望创建具备自身特征的虚拟角色与观众进行互动时，主播可通过提供二维无表情人脸图像自动生成虚拟人脸模型，旨在通过形变参数模型确定二维无表情人脸图像的人脸形变参数，并根据人脸形变参数和设定的标准人脸模型生成虚拟人脸模型，简化人脸处理操作，提高虚拟角色的人脸处理效果。对于传统的虚拟角色人脸处理方案，一般是由用户通过捏脸控制面板上得脸部特征拖动条调整标准人脸模型的脸部特征，以生成用户预期的虚拟角色，或者是由专业人员(例如专业的捏脸师)通过长时间的捏脸操作生成虚拟角色，存在人脸处理操作较为复杂和繁琐，虚拟角色的人脸处理效果较差的技术问题。基于此，提供本申请实施例的一种虚拟角色人脸处理方法，以解决现有虚拟角色人脸处理方案操作较为复杂和繁琐，虚拟角色的人脸处理效果较差的技术问题。
图1给出了本申请实施例提供的一种虚拟角色人脸处理方法的流程图，本申请实施例提供的虚拟角色人脸处理方法可以由虚拟角色人脸处理装置来执 行，该虚拟角色人脸处理装置可以通过硬件和/或软件的方式实现，并集成在虚拟角色人脸处理设备中。
下述以虚拟角色人脸处理装置执行虚拟角色人脸处理方法为例进行描述。参考图1，该虚拟角色人脸处理方法包括：
S101：获取二维无表情人脸图像。
本方案提供的二维无表情人脸图像为需要进行虚拟角色捏脸的用户提供的正脸无表情的二维人脸图像，二维无表情人脸图像可通过摄像装置进行拍摄得到，也可以是通过用户指定的下载地址进行下载得到，还可以是从本地存储位置(例如相册)进行获取。
示例性的，获取用户提供的二维无表情人脸图像，例如，主播在需要创建虚拟人脸模型时，可通过虚拟角色人脸处理设备(例如手机、电脑等)拍摄自身正脸无表情的人脸图像作为二维无表情人脸图像。后续将基于用户提供的二维无表情人脸图像生成反映二维无表情人脸图像中的人脸特征的虚拟人脸模型。
S102：将二维无表情人脸图像输入至训练好的形变参数模型，通过形变参数模型对二维无表情人脸图像进行分析处理，得到反映二维无表情人脸图像中人脸特征的人脸形变参数。
本方案在虚拟角色人脸处理设备中预先配置有训练好的形变参数模型，该形变参数模型可用于对提供的二维无表情人脸图像进行分析处理，并输出反映该二维无表情人脸图像中人脸特征的人脸形变参数。其中，人脸形变参数可以是用于设定三维可形变人脸模型(3DMM，3D Morphable model)的形状基向量。
示例性的，将获取到的二维无表情人脸图像输入至形变参数模型中，形变参数模型在接收到二维无表情人脸图像后，将对二维无表情人脸图像进行分析处理并输出对应的人脸形变参数。
在一个可能的实施例中，本方案提供的形变参数模型可基于二维无表情人脸样本图像和人脸样本形变参数进行训练得到，其中，人脸样本图像参数基于人脸样本三维关键点集合以及虚拟角色关键点集合的配准结果确定。
本方案提供的形变参数模型可基于卷积神经网络进行搭建得到，形变参数模型的主干网络可基于Resnet18网络或MobileNet网络进行搭建得到，同时，将形变参数模型的最后一层layer设置为sigmoid层，以保证形变参数模型输出的人脸形变参数的元素值为0～1之间。
其中，人脸样本形变参数可反映二维无表情人脸样本图像中的样本用户人脸的人脸特征，并且人脸样本图像参数基于对应样本用户的人脸样本三维关键点集合以及样本用户的虚拟角色关键点集合的配准结果确定。可选的，样本用户的虚拟角色关键点集合为样本虚拟角色上的人脸样本三维关键点的集合，样本虚拟角色可由专业捏脸师基于样本用户的人脸特征进行捏脸操作得到。
示例性的，通过图像采集设备采集多个样本用户的二维无表情人脸样本图像以及人脸样本三维关键点集合。其中，每个样本用户对应采集多个在不同拍摄环境下的二维无表情人脸样本图像。可选的，不同的拍摄环境可以是不同的光照条件、室内外场景以及拍摄距离等的不同组合，以提高二维无表情人脸样本图像的样本多样性，例如对每个样本用户采集100张涵盖不同光照条件、户外与室内场景以及不同拍摄距离的二维无表情人脸样本图像。
其中，每个样本用户对应采集一个人脸样本三维关键点集合，人脸样本三维关键点集合记录有样本用户的人脸图像上一系列由三维坐标(即人脸样本三维关键点)组成的点集，人脸样本三维关键点集合可表示为：其中m为人脸样本三维关键点集合中的关键点数量，其可根据图像采集设备的采集精度确定，可以理解的是，可理解为人脸样本三维关键点集合中编号为1的人脸样本三维关键点的3D坐标(x1,y1,z1)。
在一个实施例中，在通过图像采集设备采集多个样本用户的二维无表情人脸样本图像时，需要保证样本用户的人脸处于摆正姿态，脸部不能做出表情，并且图像采集设备正面拍摄样本用户的人脸图像得到二维无表情人脸样本图像。同时，在样本用户的二维无表情人脸样本图像时，需要确保样本用户、处于图像采集设备的最大拍摄范围内，以保证所采集的人脸三维关键点的准确性。
在一个实施例中，获取每个样本用户的虚拟角色关键点集合。可以理解的是，由于专业的捏脸师基于捏脸操作的生成的虚拟角色上的虚拟角色关键点集合的关键点数量可能与图像采集设备采集的人脸样本三维关键点集合的关键点数量不同，需要对人脸样本三维关键点集合以及虚拟角色关键点集合进行配准处理。假设虚拟角色关键点集合表示为其中n为虚拟角色关键点的数量。需要进行解释的是，图像采集设备采集的人脸样本三维关键点集合相对于虚拟角色关键点集合是较为稀疏的，人脸样本三维关键点集合的关键点数量级在百或千左右，虚拟角色关键点集合的关键点数量一般是在万以上，即n大于m。
其中，对关键点集合的配准可理解为找到一个变换参数T使得虚拟角色关键点集合Pt在变换后的T(Pt)与人脸样本三维关键点集合Ps的重合度尽可能高，并在虚拟角色关键点集合Pt里找到能与人脸样本三维关键点集合Ps一一对应的关键点编号集合indice＝{a1,a2,…,am}。此时，对应对应对应其中，配准结果对应的变换参数T可以是包括缩放因子、旋转矩阵和平移向量。
示例性的，在对每个样本用户的人脸样本三维关键点集合和虚拟角色关键点集合进行配准处理得到配准结果后，根据配准结果确定每个样本用户的人脸样本图像参数。在一个实施例中，人脸样本图像参数可基于对应的变换参数以及人脸样本三维关键点集合确定，通过改变人脸样本图像参数，可使最终生成的虚拟人脸模型中编号对应关键点编号集合的虚拟人脸关键点与人脸样本三维关键点在三维空间上重合。
在一个实施例中，利用采集到的多个样本用户的二维无表情人脸样本图像和人脸样本形变参数作为训练数据，对形变参数模型进行训练，并将训练完成的形变参数模型配置到虚拟角色人脸处理设备中。例如，以二维无表情人脸样本图像为输入，对应的人脸样本形变参数为输出对形变参数模型进行训练，直至满足设定的损失函数要求。本方案通过根据人脸样本三维关键点集合以及虚拟角色关键点集合的配准结果确定人脸样本图像参数，并利用采集到的二维无表情人脸样本图像和人脸样本形变参数训练形变参数模型，形变参数模型可准确分析出输入的二维图像对应的人脸形变参数，提高对虚拟角色捏脸的人脸处理效果以及捏脸效率。
在一个实施例中，对形变参数模型进行训练的损失函数可表示为：
loss＝||αpred-αtrue||2
其中，αpred为形变参数模型对二维无表情人脸样本图像进行分析处理，预测输出的人脸形变参数，αtrue为二维无表情人脸样本图像对应的人脸样本形变参数。可选的，可通过梯度下降法不断调整形变参数模型，直至损失函数的值低于设定的损失函数阈值，在损失函数收敛停止后保存形变参数模型，并配置到虚拟角色人脸处理设备对应的应用程序中。
在一个可能的实施例中，本方案在对人脸样本三维关键点集合以及虚拟角色关键点集合进行配准时，可基于以下配准公式进行：
其中，s*为需要求解的缩放因子，R*为需要求解的旋转矩阵，t*为需要求解的平移向量，s为调优过程的缩放因子，R为调优过程的旋转矩阵，t为调优过程的平移向量，Pt为虚拟角色关键点集合，为人脸样本三维关键点集合Ps的第i个人脸样本三维关键点，为虚拟角色关键点集合Pt的第i个虚拟角色关键。
示例性的，在对人脸样本三维关键点集合和虚拟角色关键点集合进行配准的过程中，通过不断调整缩放因子s、旋转矩阵R和平移向量t，确定使得上述公式中右边的值达到最小的整缩放因子s、旋转矩阵R和平移向量t的组合。其中，部分可理解为基于变换参数T对虚拟角色关键点集合进行变换的过程，其中缩放因子s为标量，旋转矩阵R为3x3的正交矩阵；平移向量t为3维的平移向量。可选的，基于配准公式对人脸样本三维关键点集合和虚拟角色关键点集合的配准调优过程，可基于迭代最近点(Iterative Closest Point,ICP)算法求解配准公式。本方案通过基于配准公式对人脸样本三维关键点集合和虚拟角色关键点集合进行配准，得到准确的配准结果，从而得到更准确的人脸样本图像参数，保证人脸处理效果。
在一个可能的实施例中，如图2提供的一种关键点集合的配准流程示意图所示，本方案在对人脸样本三维关键点集合以及虚拟角色关键点集合进行配准时，可以是包括步骤S1001-S1003：
S1001：基于第一关键点数量，对人脸样本三维关键点集合以及虚拟角色关键点集合进行配准处理，得到第一缩放因子、第一旋转矩阵和第一平移向量。
S1002：基于第二关键点数量，对人脸样本三维关键点集合以及虚拟角色关键点集合进行配准处理，得到第二缩放因子、第二旋转矩阵和第二平移向量，第二关键点数量大于第一关键点数量。
S1003：基于第二缩放因子、第二旋转矩阵和第二平移向量，确定在虚拟角色关键点集合中，与人脸样本三维关键点集合匹配的关键点编号集合。
本方案对人脸样本三维关键点集合和虚拟角色关键点集合的配准操作包括粗配准和精配准两步，其中粗配准基于第一关键点数量进行，精配准基于第二关键点数量进行，并且第二关键点数量大于第一关键点数量。其中，粗配准可理解为在两幅点云之间的变换完全未知的情况下进行较为粗糙的配准过程，目的是为精配准提供较好的变换初值。
示例性的，基于第一关键点数量(例如人脸识别领域里通用的68个人脸关键点)，从人脸样本三维关键点集合中确定第一关键点数量的人脸样本三维关键点，对人脸样本三维关键点集合和虚拟角色关键点集合进行配准处理，得到粗配准结果，即确定第一缩放因子、第一旋转矩阵和第一平移向量。其中，在进行粗配准处理时，可以是基于上述提供的配准公式进行。
在一个实施例中，在确定第一缩放因子、第一旋转矩阵和第一平移向量后，基于第二关键点数量(例如人脸样本三维关键点集合对应的关键点数量)，从人脸样本三维关键点集合中确定第二关键点数量的人脸样本三维关键点，在上述确定的第一缩放因子、第一旋转矩阵和第一平移向量基础上，对人脸样本三维关键点集合和虚拟角色关键点集合进行配准处理，得到精配准结果，即第二缩放因子、第二旋转矩阵和第二平移向量。可选的，在对人脸样本三维关键点集合和虚拟角色关键点集合进行精配准处理时，可以是固定第一缩放因子(即精配准确定的第二缩放因子与第一缩放因子相同)，以第一旋转矩阵和第一平移向量为第二旋转矩阵和第二平移向量的初始值，基于配准公式不断调优第二旋转矩阵和第二平移向量，得到最终的配准结果，此时，第二缩放因子、第二旋转矩阵和第二平移向量即为配准结果对应的缩放因子、旋转矩阵和平移向量。
在一个实施例中，基于上述确定的第二缩放因子、第二旋转矩阵和第二平移向量，确定在虚拟角色关键点集合中与人脸样本三维关键点集合匹配的关键点编号集合。例如，遍历人脸样本三维关键点集合中的每一个人脸样本三维关键点，确定在虚拟角色关键点集合中，与人脸样本三维关键点的距离(例如欧氏距离、汉明顿距离等)最近的虚拟角色关键点，并确定这些虚拟角色关键点在虚拟角色关键点集合中的关键点编号，基于这些关键点编号可得到关键点编号集合。本方案通过对人脸样本三维关键点集合和虚拟角色关键点集合先后进行粗配准和精配准的配准方式，在提高配准效率的同时，有效提高配准效果，得到更精确的缩放因子、旋转矩阵和平移向量，并确定准确的关键点编号集合，有效提高对虚拟角色的人脸处理效果。
如图3提供的一种配准结果示意图所示，图中示意出了在对人脸样本三维关键点集合和虚拟角色关键点集合进行配准后，人脸样本三维关键点集合和虚拟角色关键点集合的相对位置，其中较为密集的关键点为虚拟角色关键点，较为稀疏的关键点为人脸样本三维关键点(为了方便显示，人脸样本三维关键点比虚拟角色关键点粗)，经过配准后人脸样本三维关键点能基本能和虚拟角色 关键点重合在一起，并且人脸样本三维关键点能够在虚拟角色关键点中能够找到一一对应并且距离最近的关键点(可通过关键点编号集合进行记录)。
在一个可能的实施例中，本方案提供的人脸样本图像参数可基于以下形变参数确定公式进行确定：
其中，α*为需要求解的人脸样本图像参数，α为调优过程的人脸样本图像参数，s为第二缩放因子，R为第二旋转矩阵，t为第二平移向量，indice为关键点编号集合indice＝{a1,a2,…,am}，为基于人脸样本图像参数α确定的关键点编号，Ps为人脸样本三维关键点集合。可选的，形变参数确定公式的求解过程为最小二乘法优化问题的计算过程，可通过梯度下降或牛顿迭代法求解最终的人脸样本图像参数。可选的，可将二维无表情人脸样本图像和求解得到的人脸样本图像参数保存为二进制数据文件，将作为样本数据对形变参数模型进行训练。本方案通过形变参数确定公式更准确地计算人脸样本图像参数，有效提高对虚拟角色的人脸处理效果。
S103：基于人脸形变参数和设定的标准人脸模型生成虚拟人脸模型，标准人脸模型为三维可形变人脸模型。
本方案在虚拟角色人脸处理设备中预先配置有标准人脸模型，其中，本方案提供的标准人脸模型为三维可形变人脸模型，三维可形变人脸模型可根据给定的人脸形变参数对标准人脸模型进行调整，得到给定的人脸形变参数所反映的人脸特征的虚拟人脸模型。
示例性的，在确定二维无表情人脸图像对应的人脸形变参数后，可根据人脸形变参数对设定的标准人脸模型进行调整，从而生成虚拟人脸模型。其中，基于人脸形变参数和设定的标准人脸模型生成虚拟人脸模型，可通过设定的图像处理引擎(例如虚幻引擎)进行，并由图像处理引擎渲染出虚拟人脸模型。
在一个可能的实施例中，本方案在基于人脸形变参数和设定的标准人脸模型生成虚拟人脸模型时，可以是基于人脸形变参数，对设定的标准人脸模型和标准人脸模型的人脸形状向量进行线性加权处理，得到虚拟人脸模型。
其中，本方案提供的标准人脸模型为三维可形变人脸模型，三维可形变人脸模型可由标准人脸与人脸形变参数(形状基向量)的线性加权组合得到。示例性的，在确定二维无表情人脸图像对应的人脸形变参数后，可将二维无表情 人脸图像和人脸形变参数提交至设定的图像处理引擎，由图像处理引擎根据人脸形变参数对标准人脸模型和标准人脸模型的人脸形状向量进行线性加权处理得到并渲染出虚拟人脸模型，该虚拟人脸模型记录有用户的人脸特征。本方案通过基于人脸形变参数对标准人脸模型和人脸形状向量进行线性加权处理，得到反映用户的人脸特征的虚拟人脸模型，生成的虚拟人脸模型更贴近用户形象，在简化人脸处理操作的同时，提高人脸处理效果，用户仅需上传二维无表情人脸图像即可实现虚拟人脸模型的生成，有效降低人脸处理的门槛。
在一个可能的实施例中，本方案提供的虚拟人脸模型可基于以下人脸模型生成公式进行确定：
其中，Smodel为虚拟人脸模型，为标准人脸模型，si为标准人脸模型中第i维人脸形状向量，αi为第i维人脸形变参数，α为106维的人脸形变参数。标准人脸模型和第i维人脸形状向量si均为n×3的向量矩阵，n为虚拟人脸模型的人脸关键点数量，3为坐标维数，第i维人脸形变参数的取值范围为0～1，其值并代表人脸的语义特征。其中，标准人脸模型和第i维人脸形状向量si为常数，第i维人脸形变参数αi为自变量，虚拟人脸模型Smodel为因变量，需要进行解释的是，虚拟人脸模型Smodel虽然是一个n×3的矩阵，但其和虚拟角色关键点集合的含义是一样的，都是指n个人脸关键点的3维度坐标信息。本方案通过人脸模型生成公式对标准人脸模型和人脸形状向量进行线性加权处理得到更贴合用户脸部特征的虚拟人脸模型，有效提高对虚拟角色的人脸处理效果。
上述，通过获取二维无表情人脸图像，利用形变参数模型确定二维无表情人脸图像的人脸形变参数，并根据人脸形变参数和设定的标准人脸模型生成虚拟人脸模型，用户只需要上传二维无表情人脸图像，即可根据人脸形变参数对三维可形变的标准人脸模型进行调整，得到反映二维无表情人脸图像中人脸特征的虚拟人脸模型，在简化虚拟角色捏脸操作的同时，提高虚拟角色人脸处理效果。
图4是本申请实施例提供的一种虚拟角色人脸处理装置的结构示意图。参考图4，该虚拟角色人脸处理装置包括图像获取模块41、参数获取模块42和人 脸生成模块43。
其中，图像获取模块41，配置为获取二维无表情人脸图像；参数获取模块42，配置为将二维无表情人脸图像输入至训练好的形变参数模型，通过形变参数模型对二维无表情人脸图像进行分析处理，得到反映二维无表情人脸图像中人脸特征的人脸形变参数；人脸生成模块43，配置为基于人脸形变参数和设定的标准人脸模型生成虚拟人脸模型，标准人脸模型为三维可形变人脸模型。
上述，通过获取二维无表情人脸图像，利用形变参数模型确定二维无表情人脸图像的人脸形变参数，并根据人脸形变参数和设定的标准人脸模型生成虚拟人脸模型，用户只需要上传二维无表情人脸图像，即可根据人脸形变参数对三维可形变的标准人脸模型进行调整，得到反映二维无表情人脸图像中人脸特征的虚拟人脸模型，在简化虚拟角色捏脸操作的同时，提高虚拟角色人脸处理效果。
在上述实施例的基础上，形变参数模型基于二维无表情人脸样本图像和人脸样本形变参数进行训练得到，人脸样本图像参数基于人脸样本三维关键点集合以及虚拟角色关键点集合的配准结果确定。
在上述实施例的基础上，人脸样本三维关键点集合以及虚拟角色关键点集合的配准包括：
基于第一关键点数量，对人脸样本三维关键点集合以及虚拟角色关键点集合进行配准处理，得到第一缩放因子、第一旋转矩阵和第一平移向量；
基于第二关键点数量，对人脸样本三维关键点集合以及虚拟角色关键点集合进行配准处理，得到第二缩放因子、第二旋转矩阵和第二平移向量，第二关键点数量大于第一关键点数量；
基于第二缩放因子、第二旋转矩阵和第二平移向量，确定在虚拟角色关键点集合中，与人脸样本三维关键点集合匹配的关键点编号集合。
在上述实施例的基础上，人脸样本三维关键点集合以及虚拟角色关键点集合的配准基于以下公式进行：
其中，s*为需要求解的缩放因子，R*为需要求解的旋转矩阵，t*为需要求解的平移向量，s为调优过程的缩放因子，R为调优过程的旋转矩阵，t为调优过程的平移向量，Pt为虚拟角色关键点集合，为人脸样本三维关键点集合Ps的第i 个人脸样本三维关键点，为虚拟角色关键点集合Pt的第i个虚拟角色关键点。
在上述实施例的基础上，人脸样本图像参数基于以下公式进行确定：
其中，α*为需要求解的人脸样本图像参数，α为调优过程的人脸样本图像参数，s为第二缩放因子，R为第二旋转矩阵，t为第二平移向量，indice为关键点编号集合indice＝{a1,a2,…,am}，为基于人脸样本图像参数α确定的关键点编号，Ps为人脸样本三维关键点集合。
在上述实施例的基础上，人脸生成模块43在基于人脸形变参数和设定的标准人脸模型生成虚拟人脸模型时，配置为基于人脸形变参数，对设定的标准人脸模型和标准人脸模型的人脸形状向量进行线性加权处理，得到虚拟人脸模型。
在上述实施例的基础上，虚拟人脸模型基于以下公式进行确定：
其中，Smodel为虚拟人脸模型，为标准人脸模型，si为标准人脸模型中第i维人脸形状向量，αi为第i维人脸形变参数。
值得注意的是，上述虚拟角色人脸处理装置的实施例中，所包括的各个单元和模块只是按照功能逻辑进行划分的，但并不局限于上述的划分，只要能够实现相应的功能即可；另外，各功能单元的具体名称也只是为了便于相互区分，并不用于限制本申请实施例的保护范围。
本申请实施例还提供了一种虚拟角色人脸处理设备，该虚拟角色人脸处理设备可集成本申请实施例提供的虚拟角色人脸处理装置。图5是本申请实施例提供的一种虚拟角色人脸处理设备的结构示意图。参考图5，该虚拟角色人脸处理设备包括：输入装置53、输出装置54、存储器52以及一个或多个处理器51；存储器52，用于存储一个或多个程序；当一个或多个程序被一个或多个处理器51执行，使得一个或多个处理器51实现如上述实施例提供的虚拟角色人脸处理方法。上述提供的虚拟角色人脸处理装置、设备和计算机可用于执行上述任意实施例提供的虚拟角色人脸处理方法，具备相应的功能和有益效果。
本申请实施例还提供一种存储计算机可执行指令的非易失性存储介质，计 算机可执行指令在由计算机处理器执行时用于执行如上述实施例提供的虚拟角色人脸处理方法。当然，本申请实施例所提供的一种存储计算机可执行指令的非易失性存储介质，其计算机可执行指令不限于如上提供的虚拟角色人脸处理方法，还可以执行本申请任意实施例所提供的虚拟角色人脸处理方法中的相关操作。上述实施例中提供的虚拟角色人脸处理装置、设备及存储介质可执行本申请任意实施例所提供的虚拟角色人脸处理方法，未在上述实施例中详尽描述的技术细节，可参见本申请任意实施例所提供的虚拟角色人脸处理方法。
在上述实施例的基础上，本申请实施例还提供一种计算机程序产品，本申请的技术方案本质上或者说对相关技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机程序产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备、移动终端或其中的处理器执行本申请各个实施例所提供的虚拟角色人脸处理方法的全部或部分步骤。

Claims (11)

  1. 一种虚拟角色人脸处理方法，其中，包括：
    获取二维无表情人脸图像；
    将所述二维无表情人脸图像输入至训练好的形变参数模型，通过所述形变参数模型对所述二维无表情人脸图像进行分析处理，得到反映所述二维无表情人脸图像中人脸特征的人脸形变参数；
    基于所述人脸形变参数和设定的标准人脸模型生成虚拟人脸模型，所述标准人脸模型为三维可形变人脸模型。
  2. 根据权利要求1所述的虚拟角色人脸处理方法，其中，所述形变参数模型基于二维无表情人脸样本图像和人脸样本形变参数进行训练得到，所述人脸样本图像参数基于人脸样本三维关键点集合以及虚拟角色关键点集合的配准结果确定。
  3. 根据权利要求2所述的虚拟角色人脸处理方法，其中，所述人脸样本三维关键点集合以及所述虚拟角色关键点集合的配准包括：
    基于第一关键点数量，对所述人脸样本三维关键点集合以及所述虚拟角色关键点集合进行配准处理，得到第一缩放因子、第一旋转矩阵和第一平移向量；
    基于第二关键点数量，对所述人脸样本三维关键点集合以及所述虚拟角色关键点集合进行配准处理，得到第二缩放因子、第二旋转矩阵和第二平移向量，所述第二关键点数量大于所述第一关键点数量；
    基于所述第二缩放因子、所述第二旋转矩阵和所述第二平移向量，确定在所述虚拟角色关键点集合中，与所述人脸样本三维关键点集合匹配的关键点编号集合。
  4. 根据权利要求3所述的虚拟角色人脸处理方法，其中，所述人脸样本三维关键点集合以及所述虚拟角色关键点集合的配准基于以下公式进行：
    其中，s*为需要求解的缩放因子，R*为需要求解的旋转矩阵，t*为需要求解的平移向量，s为调优过程的缩放因子，R为调优过程的旋转矩阵，t为调优过程的平移向量，Pt为虚拟角色关键点集合，为人脸样本三维关键点集合Ps的第i个人脸样本三维关键点，为虚拟角色关键点集合Pt的第i个虚拟角色关键点。
  5. 根据权利要求3所述的虚拟角色人脸处理方法，其中，所述人脸样本图像参数基于以下公式进行确定：
    其中，α*为需要求解的人脸样本图像参数，α为调优过程的人脸样本图像参数，s为第二缩放因子，R为第二旋转矩阵，t为第二平移向量，indice为关键点编号集合indice＝{a1,a2,…,am}，为基于人脸样本图像参数α确定的关键点编号，Ps为人脸样本三维关键点集合。
  6. 根据权利要求1所述的虚拟角色人脸处理方法，其中，所述基于所述人脸形变参数和设定的标准人脸模型生成虚拟人脸模型，包括：
    基于所述人脸形变参数，对设定的标准人脸模型和所述标准人脸模型的人脸形状向量进行线性加权处理，得到虚拟人脸模型。
  7. 根据权利要求1所述的虚拟角色人脸处理方法，其中，所述虚拟人脸模型基于以下公式进行确定：
    其中，Smodel为虚拟人脸模型，为标准人脸模型，si为标准人脸模型中第i维人脸形状向量，αi为第i维人脸形变参数。
  8. 一种虚拟角色人脸处理装置，其中，包括图像获取模块、参数获取模块和人脸生成模块，其中：
    所述图像获取模块，配置为获取二维无表情人脸图像；
    所述参数获取模块，配置为将所述二维无表情人脸图像输入至训练好的形变参数模型，通过所述形变参数模型对所述二维无表情人脸图像进行分析处理，得到反映所述二维无表情人脸图像中人脸特征的人脸形变参数；
    所述人脸生成模块，配置为基于所述人脸形变参数和设定的标准人脸模型生成虚拟人脸模型，所述标准人脸模型为三维可形变人脸模型。
  9. 一种虚拟角色人脸处理设备，其中，包括：存储器以及一个或多个处理器；
    所述存储器，用于存储一个或多个程序；
    当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现如权利要求1-7任一项所述的虚拟角色人脸处理方法。
  10. 一种存储计算机可执行指令的非易失性存储介质，其中，所述计算机可执行指令在由计算机处理器执行时用于执行如权利要求1-7任一项所述的虚拟角色人脸处理方法。
  11. 一种计算机程序产品，包括计算机程序，其中，所述计算机程序被处理器执行时实现权利要求1-7任一项所述的虚拟角色人脸处理方法。
PCT/CN2024/070204 2023-01-12 2024-01-02 一种虚拟角色人脸处理方法、装置、设备、存储介质及产品 Ceased WO2024149124A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202310071200.5A CN115984940B (zh) 2023-01-12 2023-01-12 一种虚拟角色人脸处理方法、装置、设备、存储介质及产品
CN202310071200.5 2023-01-12

Publications (1)

Publication Number Publication Date
WO2024149124A1 true WO2024149124A1 (zh) 2024-07-18

Family

ID=85970215

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2024/070204 Ceased WO2024149124A1 (zh) 2023-01-12 2024-01-02 一种虚拟角色人脸处理方法、装置、设备、存储介质及产品

Country Status (2)

Country Link
CN (1) CN115984940B (zh)
WO (1) WO2024149124A1 (zh)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115984940B (zh) * 2023-01-12 2026-04-03 百果园技术(新加坡)有限公司 一种虚拟角色人脸处理方法、装置、设备、存储介质及产品
CN117541758B (zh) * 2023-11-28 2024-07-09 吉林动画学院 虚拟人脸配置参数生成方法、装置、设备和存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040227826A1 (en) * 2003-05-16 2004-11-18 Benq Corporation Device and method to determine exposure setting for image of scene with human-face area
CN110363858A (zh) * 2019-06-18 2019-10-22 新拓三维技术(深圳)有限公司 一种三维人脸重建方法及系统
CN110866864A (zh) * 2018-08-27 2020-03-06 阿里巴巴集团控股有限公司 人脸姿态估计/三维人脸重构方法、装置及电子设备
CN111632374A (zh) * 2020-06-01 2020-09-08 网易（杭州）网络有限公司 游戏中虚拟角色的脸部处理方法、装置及可读存储介质
WO2022147736A1 (zh) * 2021-01-07 2022-07-14 广州视源电子科技股份有限公司 虚拟图像构建方法、装置、设备及存储介质
CN115984940A (zh) * 2023-01-12 2023-04-18 百果园技术(新加坡)有限公司 一种虚拟角色人脸处理方法、装置、设备、存储介质及产品

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114926581B (zh) * 2022-04-22 2025-05-06 中国科学院软件研究所 二维到三维人脸表情迁移方法、电子装置及存储介质

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040227826A1 (en) * 2003-05-16 2004-11-18 Benq Corporation Device and method to determine exposure setting for image of scene with human-face area
CN110866864A (zh) * 2018-08-27 2020-03-06 阿里巴巴集团控股有限公司 人脸姿态估计/三维人脸重构方法、装置及电子设备
CN110363858A (zh) * 2019-06-18 2019-10-22 新拓三维技术(深圳)有限公司 一种三维人脸重建方法及系统
CN111632374A (zh) * 2020-06-01 2020-09-08 网易（杭州）网络有限公司 游戏中虚拟角色的脸部处理方法、装置及可读存储介质
WO2022147736A1 (zh) * 2021-01-07 2022-07-14 广州视源电子科技股份有限公司 虚拟图像构建方法、装置、设备及存储介质
CN115984940A (zh) * 2023-01-12 2023-04-18 百果园技术(新加坡)有限公司 一种虚拟角色人脸处理方法、装置、设备、存储介质及产品

Also Published As

Publication number Publication date
CN115984940A (zh) 2023-04-18
CN115984940B (zh) 2026-04-03

Similar Documents

Publication Publication Date Title
CN110473141B (zh) 图像处理方法、装置、存储介质及电子设备
WO2024149124A1 (zh) 一种虚拟角色人脸处理方法、装置、设备、存储介质及产品
WO2021042208A1 (en) Dynamically estimating light-source-specific parameters for digital images using a neural network
CN116310076A (zh) 基于神经辐射场的三维重建方法、装置、设备及存储介质
CN108024762A (zh) 耳形状分析方法、耳形状分析装置以及耳形状模型生成方法
JP2017531950A (ja) 撮影テンプレートデータベースを構築し、且つ撮影推薦情報を提供するための方法及び装置
CN111277759B (zh) 构图提示方法、装置、存储介质及电子设备
CN110110118A (zh) 妆容推荐方法、装置、存储介质及移动终端
CN108830892A (zh) 人脸图像处理方法、装置、电子设备及计算机可读存储介质
CN114820907B (zh) 人脸图像卡通化处理方法、装置、计算机设备和存储介质
CN118102044A (zh) 基于3d高斯泼溅的点云数据生成方法、装置、设备及介质
CN114693853A (zh) 对象渲染方法、装置、电子设备及存储介质
CN114333034A (zh) 人脸姿态估计方法、装置、电子设备及可读存储介质
WO2025092175A1 (zh) 虚拟对象生成方法、装置、计算机设备及存储介质
CN115861515A (zh) 一种三维人脸重建方法、计算机程序产品及电子设备
WO2024174836A1 (zh) 一种人脸模型构造方法、装置、设备、存储介质及产品
CN109753873A (zh) 图像处理方法及相关装置
CN109218615A (zh) 图像拍摄辅助方法、装置、终端和存储介质
CN120495598B (zh) 数据智能化处理方法及系统
CN112861944B (zh) 一种基于混合模态输入的图像检索方法及装置
CN115861393B (zh) 图像匹配方法、航天器着陆点定位方法及相关装置
KR102906550B1 (ko) 인테리어 스타일링 서비스 제공 방법 및 장치
CN118570424A (zh) 虚拟现实导游系统
CN117745850A (zh) 地图矢量化生成方法、装置和服务器
CN116959076A (zh) 面部数据采集方法、系统及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24741123

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 24741123

Country of ref document: EP

Kind code of ref document: A1