CN112102461B - Face rendering method and device, electronic equipment and storage medium - Google Patents

Face rendering method and device, electronic equipment and storage medium Download PDF

Info

Publication number
CN112102461B
CN112102461B CN202011206984.0A CN202011206984A CN112102461B CN 112102461 B CN112102461 B CN 112102461B CN 202011206984 A CN202011206984 A CN 202011206984A CN 112102461 B CN112102461 B CN 112102461B
Authority
CN
China
Prior art keywords
face
code
rendering
image
face image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202011206984.0A
Other languages
Chinese (zh)
Other versions
CN112102461A (en
Inventor
程斌
徐善
袁东东
张世豪
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing zhimeiyuansu Technology Co.,Ltd.
Original Assignee
Beijing Zhiyuan Artificial Intelligence Research Institute
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Zhiyuan Artificial Intelligence Research Institute filed Critical Beijing Zhiyuan Artificial Intelligence Research Institute
Priority to CN202011206984.0A priority Critical patent/CN112102461B/en
Publication of CN112102461A publication Critical patent/CN112102461A/en
Application granted granted Critical
Publication of CN112102461B publication Critical patent/CN112102461B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Graphics (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Processing Or Creating Images (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a face rendering method, a face rendering device, electronic equipment and a storage medium, wherein the method comprises the following steps: acquiring at least one face image code, wherein the code has the feature information of the face image; acquiring control parameters of the three-dimensional face based on the codes; and rendering the face image according to the control parameters to obtain a first face rendering image. The face rendering method can realize controllable three-dimensional face reconstruction and rendering based on the real face image, controls the attribute, and has wide application value.

Description

Face rendering method and device, electronic equipment and storage medium
Technical Field
The invention relates to the technical field of artificial intelligence, in particular to a face rendering method, a face rendering device, electronic equipment and a storage medium.
Background
With the development of artificial intelligence technology, especially image processing technology, reconstructing a face and controlling attributes (including face angle, shape, expression, skin color, illumination and the like) based on an image has a wide application value, and is used in the fields of virtual worlds, visual special effects, medical and beauty face-lifting planning and the like. However, the current face reconstruction and rendering method is performed for virtual characters, and aims to optimize face coefficients and detail information (such as skin color, wrinkles, scars and the like), so that the attribute control of a real face image input by a user cannot be realized.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a face rendering method, apparatus, electronic device and storage medium.
In a first aspect, an embodiment of the present application provides a face rendering method, where the method includes:
acquiring a code of at least one face image, wherein the code has characteristic information of the face image;
acquiring control parameters of the three-dimensional face based on the codes;
and rendering the face image according to the control parameters to obtain a first face rendering image.
In one embodiment, the code is obtained using ResNet.
In one embodiment, the activation function is used to obtain the control parameters for the multi-tier perceptron of the ReLU.
In one embodiment, the face image is an image of a real face.
In a second aspect, an embodiment of the present application provides a face rendering method, where the method includes:
acquiring a first code of a first face image and a second code of a second face image, wherein the first code has characteristic information of the first face image, and the second code has characteristic information of the second face image;
acquiring a first control parameter of the three-dimensional face based on the first code;
converting the first control parameter to a change factor;
generating a third code according to the second code and the variation factor;
acquiring a second control parameter based on the third code;
and rendering the second face image according to the second control parameter to obtain a second face rendering image.
In one embodiment, the first face image and the second face image are both images of real faces.
In a third aspect, an embodiment of the present application provides a face rendering apparatus, where the apparatus includes:
the encoder is used for acquiring at least one code of the face image, and the code has the characteristic information of the face image;
the mapping network is used for acquiring control parameters of the three-dimensional face based on the codes;
and the renderer is used for rendering the face image according to the control parameters to obtain a first face rendering image.
In a fourth aspect, an embodiment of the present application provides a face rendering apparatus, where the apparatus includes:
the encoder is used for acquiring a first code of a first face image and a second code of a second face image, wherein the first code has characteristic information of the first face image, and the second code has characteristic information of the second face image;
the mapping network is used for acquiring a first control parameter of the three-dimensional face based on the first code and acquiring a second control parameter based on a third code;
an adaptive network for converting the first control parameter to a change factor;
a fuser for generating the third code from the second code and the variation factor;
and the renderer is used for rendering the second face image according to the second control parameter to obtain a second face rendering image.
In a fifth aspect, an embodiment of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method steps described above.
In a sixth aspect, embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored, the program being executed by a processor to implement the method steps as described above.
The technical scheme provided by the embodiment of the application can have the following beneficial effects:
in the embodiment of the application, at least one face image code is obtained, and the code has the characteristic information of the face image; acquiring control parameters of the three-dimensional face based on the codes; the face image is rendered according to the control parameters to obtain a face rendering image, controllable 3D face reconstruction and rendering can be achieved based on the real face image (not limited to virtual characters), the face attribute can be controlled, and the method has wide application value. In addition, the face rendering method provided by the embodiment of the application designs the loss function, and performs training based on the input graph, the generated graph and the rendering graph without a large amount of image data sets; in addition, the face image is rendered through the control parameters of the three-dimensional face, and the obtained face rendering image is more vivid. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
Fig. 1 is an application scene diagram of a face rendering method according to an embodiment of the present application;
fig. 2 is a schematic flowchart of a face rendering method according to an embodiment of the present application;
fig. 3 is a schematic view of a face rendering process in a specific application scenario according to an embodiment of the present application;
fig. 4 is a schematic flowchart of another face rendering method according to an embodiment of the present application;
fig. 5 is a schematic view of a face rendering process in another specific application scenario provided in the embodiment of the present application;
fig. 6 is a schematic structural diagram of a face rendering apparatus according to an embodiment of the present application;
fig. 7 is a schematic structural diagram of another face rendering apparatus according to an embodiment of the present application;
FIG. 8 is a schematic diagram of an electronic device connection configuration according to an embodiment of the application;
fig. 9 is a schematic structural diagram of a terminal according to an embodiment of the present application.
Detailed Description
The following description and the drawings sufficiently illustrate specific embodiments of the invention to enable those skilled in the art to practice them.
It should be understood that the described embodiments are only some embodiments of the invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Alternative embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.
As shown in fig. 1, an application scenario diagram according to an embodiment of the present application is that a plurality of users operate clients installed on a terminal device through the terminal device such as a mobile phone, and the clients perform data communication with a background server through a network. A specific application scenario is a process of rendering at least one facial image, but is not limited to the only application scenario, and it can be understood that any scenario that can be applied to this embodiment is included, and for convenience of description, this embodiment describes, as an example, an application scenario in which at least one facial image is rendered as shown in fig. 2. As shown in fig. 1, at least one acquired face image is sent to or from one of the plurality of clients shown in fig. 1.
As shown in fig. 2, an embodiment of the present application provides a face rendering method, which specifically includes the following steps:
s202: and acquiring at least one code of the face image, wherein the code has the characteristic information of the face image.
In the embodiment of the present application, the encoding of at least one face image is obtained by encoding with an encoder E, and the face image is optionally a real face image. The encoder E is configured to extract features of the input image and output an encoding Z. The convolutional network used is ResNet-50, and the code Z output by the network is a vector of (18, 512) dimensions.
S204: acquiring control parameters of the three-dimensional face based on the codes;
in the embodiment of the application, the mapping network F generates the semantic control parameter C of the three-dimensional face based on the code Z. In one possible implementation, the mapping network F obtains control parameters using a multi-layered perceptron with a recirculation (Rectified Linear Unit) activation function. The ReLU is an activation function commonly used in artificial neural networks, and refers to a nonlinear function represented by a ramp function and a variation thereof. Derived control parameters
Figure 740171DEST_PATH_IMAGE001
Is a vector of (1, 257) dimensions, containing a three-dimensional face shape
Figure 446703DEST_PATH_IMAGE002
Skin, skin
Figure 977041DEST_PATH_IMAGE003
Facial expression
Figure 995813DEST_PATH_IMAGE004
Scene ray
Figure 142761DEST_PATH_IMAGE005
Angle, angle
Figure 905180DEST_PATH_IMAGE006
The control parameter of (1).
S206: and rendering the face image according to the control parameters to obtain a first face rendering image.
In the embodiment of the present application, the renderer R generates the first face rendering map according to the control parameter C.
Optionally, the method further comprises S207: and obtaining a face generation chart according to the codes.
In the embodiment of the present application, the generator G generates a map based on the code Z. The generator G adopts StyleGAN and adopts the pre-trained network parameters without updating and learning; the StyleGAN output is a (1024, 1024, 3) dimensional real image with a human face. The generator can output a face image with teeth, hair, and background, not just for the face portion.
In the method of the embodiment of the present application, the encoder E and the mapping network F are trained in advance. Fig. 3 is a schematic view of a face rendering process in a specific application scenario according to an embodiment of the present application;
as shown in fig. 3, an input image I is a real face image, and the input image is encoded by an encoder E to obtain an encoding Z; training the code Z through a mapping network F to obtain a control parameter C; rendering in a renderer R according to the control parameter C to obtain a rendering graph IR. On the other hand, the code Z is processed by a generator G to obtain a generated graph IG. Wherein, a diagram I is generatedGIs the same as input ISo that an encoder E that outputs a code Z corresponding to a real face can be learned. The mapping network F is a mapping network capable of outputting three-dimensional face control parameters C capable of rendering a rendering image I having the same face as the input faceRThus, rendering graph IRAnd generating a graph IGThe face has the same shape, skin, expression, scene light and angle. Thus, a mapping network F that can output 3D face control parameters C that can render a rendering map having the same face as the input can be learned.
The generator G adopts StyleGAN and pre-trained network parameters without update learning; the StyleGAN output is a (1024, 1024, 3) dimensional real image with a human face.
The mapping network F employs a multi-tier perceptron with the activation function as ReLU. In order to ensure that the generator G can generate the same generated image I as the human face of the real input I according to the code ZGThe renderer can render and generate a graph IGWith the same face, the embodiment of the application designs the countermeasure loss and the rendering loss.
In particular, the loss l is resistedgenerative: to realize an image IGThe same image as the input I, the embodiment of the present application employs the same generated countermeasure loss l as StyleGANgenerative。
To realize rendering IRThe face has the same face as the input I, and rendering loss l is designed according to the embodiment of the applicationrendingFor calculating rendering loss lrendingThe calculation formula of (2) is as follows:
Figure 86763DEST_PATH_IMAGE007
wherein M is rendering IRBinary mask, rendering map IRThe pixel value of the three-dimensional model-based rendering is 1, and the rest are 0 and IGIs a generation picture of a human face image,IRIs a rendering picture of the face image,
Figure 10857DEST_PATH_IMAGE008
Is the coordinates of key points of the human face,
Figure 645100DEST_PATH_IMAGE009
Are the weight coefficients.
The total loss corresponding to the face reconstruction network is lcode，
Calculating lcodeThe calculation formula of (2) is as follows:
Figure 194899DEST_PATH_IMAGE010
。
by adopting the encoder E and the mapping network F which are obtained by training through the loss function, a rendering graph with the same face as the input graph and the generated graph can be guaranteed to be rendered. Furthermore, by changing the encoding and/or control parameters, the properties of the face rendering map can be controlled.
At present, the existing face rendering method usually focuses on how to optimize a three-dimensional face coefficient, and focuses more on detail information, such as skin color of a target object, wrinkles of the target object, or scars of the target object. Different from the existing face image rendering method, the face rendering method provided by the embodiment of the application can controllably reconstruct and render the acquired real face image.
Fig. 4 is a schematic flow chart of another face rendering method according to the embodiment of the present application; as shown in fig. 4, an embodiment of the present application provides another face rendering method, which can extract partial attributes from a second face image to replace corresponding face attributes in a first face image, for example, to replace a face shape, five sense organs, an expression, a skin, and light, so as to finally obtain a rendering that meets the user requirements. The method comprises the following steps:
s402: and acquiring a first code of the first face image and a second code of the second face image, wherein the first code has the characteristic information of the first face image, and the second code has the characteristic information of the second face image.
In the embodiment of the present application, the first facial image and the second facial image are optionally both real facial images. The first coding of the first face image and the second coding of the second face image are obtained by coding through a coder E respectively.
S404: acquiring a first control parameter of the three-dimensional face based on a first code;
in the embodiment of the present application, the mapping network F obtains the first control parameter.
S406: converting the first control parameter into a variation factor;
in an embodiment of the application, the first control parameter is converted by the adaptive network into a change factor, the change factor comprising an offset and a scale change factor.
S408: generating a third code according to the second code and the variation factor;
in this embodiment of the application, the generation of the third code according to the second code and the variation factor is obtained by a fusion device, and optionally, the variation factor is fused with the second code by using an Adain method, so that the third code has attribute information of the first facial image and attribute information of the second facial image.
S410: acquiring a second control parameter based on the third code;
in the embodiment of the present application, the mapping network F generates a second control parameter according to the third encoding, and accordingly, the second control parameter includes a parameter representing a partial attribute (for example, a face angle) of the first face image and a parameter representing other attributes of the second face image. Part of attributes of the first face image can be selected according to requirements and can be any attributes such as expressions, skins, light, five sense organs and the like.
S412: and rendering the second face image according to the second control parameter to obtain a second face rendering image.
In the embodiment of the application, the renderer renders according to the second control parameter, and the obtained second face rendering image has a part of attributes (such as a face angle) of the first face image, and retains other attributes of the second face image.
In the embodiment of the present application, the convolutional network used by the encoder is ResNet-50, and the mapping network uses a multi-layer perceptron whose activation function is ReLU (corrected Linear Unit).
Fig. 5 is a schematic view of a face rendering process in another specific application scenario provided in the embodiment of the present application.
In FIG. 5, the input diagram includes input diagram I1Inputting the graph I2The final output image I _ des is based on the input image I2Has an input diagram I generated by the person1A new image of some attribute of the person. For example, as shown in FIG. 5, the person generating a new image I _ des has an input map I1The angle of the character of (1) retains the input diagram I2Including facial shape, facial expression, skin, light, etc.
As shown in fig. 5, the face rendering method provided in the embodiment of the present application introduces an adaptive network a, and the adaptive network a is configured to convert the generated control parameter C1 into an offset and a scale factor of the encoding Z2. The calculated offset and scale factor affect the code Z2 in Adain, generating a new code Z _ des such that the control code generated based on the code Z _ des has the input map I1The characteristics of (1). For example, as shown in FIG. 5, the final output image has an input map I1The characteristic of the face angle in (1). Fig. 5 is only an example, and attributes of the face shape, the face expression, the face skin color, the illumination and the like of different input images may also be introduced according to requirements of different application scenarios, which is not described in detail herein. The specific process is similar to the process of fig. 5.
To change the input diagram I2The angle of the human face is an input image I1The face angle of (1) is described by taking the output as I _ des as an example, which is specifically as follows:
in the embodiment of the present application, the I _ des has and outputs for realizing rendering mapGo into picture I1The same face angle and has an input image I2The embodiment of the application introduces identity loss and condition loss for training.
Loss of identity lidentity: ensuring that the rendering graph I _ des has the same value as the input graph I2As well as other face attributes. Replacing the face angle representation position of the newly generated target control parameter C _ des with the input image I2Then generates a rendered image I through the renderer R for the replaced control parameter C2R2. I.e. generating rendering I from control parameters representing attributes of the second face imageR2. Parameter replacement is optionally performed by the renderer.
Calculating identity loss lidentityThe calculation formula of (2) is as follows:
Figure 230988DEST_PATH_IMAGE011
wherein M is rendering IR2Binary mask, rendering map IR2Wherein the pixel value rendered based on the three-dimensional model is 1, and the rest are 0,
Figure 325983DEST_PATH_IMAGE012
Is a face image IXFeatures, I extracted after convolution of the VGG convolutional neural network2Is the second face image.
Loss of condition lcontrolFor ensuring rendering I _ des and having input graph I1The face angle of (1). The input image I is obtained by replacing other bits except the face angle representation bit with the newly generated target control parameter C _ des1Then generates a rendered image I through the renderer R for the replaced control parameter C1R1. That is, the rendering I is generated based on the control parameter representing the attribute of the first face imageR1. Parameter replacement is optionally performed by the renderer.
Calculating the Condition loss lcontrolThe calculation formula of (2) is as follows:
Figure 181944DEST_PATH_IMAGE013
，
wherein M is rendering IR1Binary mask, rendering map IR1Wherein the pixel value rendered based on the three-dimensional model is 1, and the rest are 0,
Figure 286166DEST_PATH_IMAGE012
Is a face image IXFeatures, I extracted after convolution of the VGG convolutional neural network1Is the first face image.
For realizing rendering graph I _ des with input graph I1The same face angle and has an input image I2The embodiment of the application introduces the total loss ltotal。
Calculating ltotalThe calculation formula of (2) is as follows:
Figure 176762DEST_PATH_IMAGE014
。
by adopting the encoder E and the mapping network F obtained by training the loss function, the rendering graph I _ des can be provided with the input graph I1The same partial attribute (such as face angle) and has an input image I2Other attributes of the face.
It should be noted that the face angle is only an example here, and the replaced partial attribute may be the input graph I1Including facial shape, facial expression, skin, light, five-organs, etc.
The following is an embodiment of a face rendering apparatus in the embodiment of the present application, which may be used to execute the embodiment of the face rendering method in the embodiment of the present application. For details that are not disclosed in the embodiment of the face rendering apparatus of the present application, please refer to the embodiment of the face rendering method of the present application.
Referring to fig. 6, a schematic structural diagram of a face rendering apparatus according to an exemplary embodiment of the present invention is shown. The face rendering device can be implemented by software, hardware or a combination of the two as all or part of the terminal. The face rendering apparatus includes an encoder 602, a mapping network 604, and a renderer 606.
Specifically, the encoder 602 is configured to obtain an encoding of at least one face image, and the encoding has feature information of the face image.
The face image is optionally a real face image. The convolutional network used by encoder E is ResNet-50, and the output code Z is a vector of (18,512) dimensions.
A mapping network 604 for obtaining control parameters of the three-dimensional face based on the coding;
in the embodiment of the present application, the mapping network F obtains the control parameters by using a multi-layer perceptron whose activation function is ReLU (corrected Linear Unit). Control parameter
Figure 708237DEST_PATH_IMAGE015
Is a vector of (1, 257) dimensions, containing a three-dimensional face shape
Figure 51494DEST_PATH_IMAGE016
Skin, skin
Figure 710139DEST_PATH_IMAGE017
Facial expression
Figure 455241DEST_PATH_IMAGE018
Scene ray
Figure 157618DEST_PATH_IMAGE019
Angle, angle
Figure 988171DEST_PATH_IMAGE020
The control parameter of (1).
And the renderer 606 is configured to render the face image according to the control parameter to obtain a first face rendering map.
Optionally, the apparatus further comprises a generator for generating the graph according to the encoding. The generator adopts StyleGAN and adopts the pre-trained network parameters without updating.
Optionally, the apparatus is trained using the following loss function:
Figure 434196DEST_PATH_IMAGE021
；
lgenerativeto combat the loss l for the same production as StyleGANgenerative；
Figure 768225DEST_PATH_IMAGE022
Wherein M is rendering IRBinary mask, rendering map IRThe pixel value of the three-dimensional model-based rendering is 1, and the rest are 0 and IGIs the generation picture and I of the face imageRIs a rendering picture of the face image,
Figure 641503DEST_PATH_IMAGE023
Is the coordinates of key points of the human face,
Figure 208620DEST_PATH_IMAGE024
Are the weight coefficients.
Fig. 7 is a schematic structural diagram of another face rendering apparatus according to an embodiment of the present application; it should be noted that the difference between the rendering apparatus shown in fig. 7 and the rendering apparatus shown in fig. 6 is that, the rendering apparatus shown in fig. 7 can extract partial attributes from the second face image to replace corresponding face attributes in the first face image, for example, to replace face shape, five sense organs, expression, skin, and light, and finally obtain a rendering map meeting the user's requirements. The face rendering device can be implemented by software, hardware or a combination of the two as all or part of the terminal. The face rendering apparatus includes an encoder 702, a mapping network 704, an adaptive network 706, a fuser 708, and a renderer 710.
Specifically, the encoder 702 is configured to obtain a first code of a first facial image and a second code of a second facial image, where the first code has feature information of the first facial image, and the second code has feature information of the second facial image;
based on the description of the first encoding and the second encoding, refer to the description of the same or similar parts in the foregoing method embodiments, and are not repeated herein.
The mapping network 704 is used for acquiring a first control parameter of the three-dimensional face based on the first code and acquiring a second control parameter based on the third code;
based on the description of the first control parameter and the second control parameter, refer to the description of the same or similar parts in the foregoing method embodiments, and are not repeated herein.
An adaptive network 706 for converting the first control parameter into a change factor;
based on the description of the variation factor, refer to the description of the same or similar parts in the foregoing method embodiments, and are not repeated herein.
A fuser 708 for generating a third code based on the second code and the variation factor;
based on the description of the third encoding, refer to the description of the same or similar parts in the foregoing method embodiments, and are not repeated herein.
And the renderer 710 is configured to render the second face image according to the second control parameter, so as to obtain a second face rendering image.
Based on the description of the second control parameter, refer to the description of the same or similar parts in the foregoing method embodiments, and are not repeated herein.
Preferably, the first face image and the second face image are both images of real faces.
Optionally, the apparatus is trained with the following loss function:
generating rendering graph I according to control parameters representing attributes of second face imageR2And calculating the identity loss:
Figure 192756DEST_PATH_IMAGE025
wherein M is rendering IR2Binary mask, rendering map IR2Wherein the pixel value rendered based on the three-dimensional model is 1, and the rest are 0,
Figure 912450DEST_PATH_IMAGE026
Is a face image IXFeatures, I extracted after convolution of the VGG convolutional neural network2Is the second face image.
Generating rendering map I from control parameters representing attributes of first face imageR1Calculating the condition loss:
Figure 956630DEST_PATH_IMAGE027
，
wherein M is rendering IR1Binary mask, rendering map IR1Wherein the pixel value rendered based on the three-dimensional model is 1, and the rest are 0,
Figure 761775DEST_PATH_IMAGE028
Is a face image IXFeatures, I extracted after convolution of the VGG convolutional neural network1Is a first face image;
calculating total loss
Figure 549602DEST_PATH_IMAGE029
。
It should be noted that, when the face rendering apparatus provided in the foregoing embodiment executes the face rendering method, only the division of the functional units is illustrated, and in practical applications, the function distribution may be completed by different functional units according to needs, that is, the internal structure of the device is divided into different functional units, so as to complete all or part of the functions described above. In addition, the face rendering device and the face rendering method provided by the above embodiments belong to the same concept, and the implementation process is detailed in the face rendering method embodiment, which is not described herein again.
By the embodiment of the application, controllable 3D face reconstruction and attribute control (including face angle, shape, expression, skin color, illumination and the like) can be realized, and the method has wide application value, such as the fields of virtual world, special effect, medical and beauty face-lifting planning and the like; the method and the device can process and control the human face based on the real human face, and are not limited to virtual characters; by the method and the device, the real high-definition 2D rendering image of the 3D face after the face attribute is changed can be generated, the face image with teeth, hair and background can be output, and the face part can be rendered. Moreover, the embodiment of the application designs the loss function, and the loss function is trained on the basis of the input graph, the generated graph and the rendering graph without a large amount of image data sets; in addition, the face image is rendered through the control parameters of the three-dimensional face, and the obtained face rendering image is more vivid.
As shown in fig. 8, the present embodiment provides an electronic device, which includes a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the method steps as described above.
The present application provides a storage medium storing computer readable instructions, on which a computer program is stored, the program being executed by a processor to implement the method steps as described above.
Referring now to FIG. 8, shown is a schematic diagram of an electronic device suitable for use in implementing embodiments of the present application. The terminal device in the embodiments of the present application may include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a vehicle terminal (e.g., a car navigation terminal), and the like, and a fixed terminal such as a digital TV, a desktop computer, and the like. The electronic device shown in fig. 8 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.
As shown in fig. 8, an electronic device may include a processing means (e.g., a central processing unit, a graphics processor, etc.) 801 that may perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 802 or a program loaded from a storage means 808 into a Random Access Memory (RAM) 803. In the RAM803, various programs and data necessary for the operation of the electronic apparatus are also stored. The processing apparatus 801, the ROM802, and the RAM803 are connected to each other by a bus 804. An input/output (I/O) interface 805 is also connected to bus 804.
Generally, the following devices may be connected to the I/O interface 805: input devices 806 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 807 including, for example, a Liquid Crystal Display (LCD), speakers, vibrators, and the like; storage 808 including, for example, magnetic tape, hard disk, etc.; and a communication device 809. The communication means 809 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. While fig. 8 illustrates an electronic device having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication means 809, or installed from the storage means 808, or installed from the ROM 802. The computer program, when executed by the processing apparatus 801, performs the above-described functions defined in the methods of the embodiments of the present application.
It should be noted that the computer readable medium in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.
Computer program code for carrying out operations for aspects of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present application may be implemented by software or hardware. Where the name of an element does not in some cases constitute a limitation on the element itself.
Please refer to fig. 9, which provides a schematic structural diagram of a terminal according to an embodiment of the present application. As shown in fig. 9, terminal 900 may include: at least one processor 901, at least one network interface 904, a user interface 903, memory 905, at least one communication bus 902.
Wherein a communication bus 902 is used to enable connective communication between these components.
The user interface 903 may include a Display screen (Display) and a Camera (Camera), and the optional user interface 903 may also include a standard wired interface and a wireless interface.
The network interface 904 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface), among others.
Processor 901 may include one or more processing cores, among other things. The processor 901 interfaces with various components throughout the electronic device 900 using various interfaces and circuitry to perform various functions of the electronic device 900 and process data by executing or executing instructions, programs, code sets, or instruction sets stored in the memory 905, as well as invoking data stored in the memory 905. Optionally, the processor 901 may be implemented in at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 901 may integrate one or a combination of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a modem, and the like. Wherein, the CPU mainly processes an operating system, a user interface, an application program and the like; the GPU is used for rendering and drawing the content required to be displayed by the display screen; the modem is used to handle wireless communications. It is understood that the modem may not be integrated into the processor 901, but may be implemented by a single chip.
The Memory 905 may include a Random Access Memory (RAM) or a Read-Only Memory (Read-Only Memory). Optionally, the memory 905 includes a non-transitory computer-readable medium. The memory 905 may be used to store instructions, programs, code, sets of codes, or sets of instructions. The memory 905 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing the above-described method embodiments, and the like; the storage data area may store data and the like referred to in the above respective method embodiments. The memory 905 may optionally be at least one memory device located remotely from the processor 901. As shown in fig. 9, the memory 905, which is a type of computer storage medium, may include therein an operating system, a network communication module, a user interface module, and a face rendering application program.
In the terminal 900 shown in fig. 9, the user interface 903 is mainly used for providing an input interface for a user to obtain data input by the user; and the processor 901 may be configured to invoke the face rendering application stored in the memory 905 and specifically perform the operations described with reference to fig. 2-5.
In the terminal 900 shown in fig. 9, the user interface 903 is mainly used for providing an input interface for a user to obtain data input by the user; and the processor 901 may be configured to invoke a face rendering application stored in the memory 905 to perform the operations described with reference to fig. 2-5.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a read-only memory or a random access memory.
The above disclosure is only for the purpose of illustrating the preferred embodiments of the present application and is not to be construed as limiting the scope of the present application, so that the present application is not limited thereto, and all equivalent variations and modifications can be made to the present application.

Claims (5)

1. A method for rendering a face, the method comprising:
acquiring a first code of a first face image and a second code of a second face image, wherein the first code has characteristic information of the first face image, and the second code has characteristic information of the second face image;
acquiring a first control parameter of the three-dimensional face based on the first code;
converting the first control parameter to a change factor, the change factor comprising an offset and a scale change factor;
generating a third code according to the second code and the change factor, so that the third code has attribute information of the first facial image and attribute information of the second facial image;
acquiring a second control parameter based on the third code;
and rendering the second face image according to the second control parameter to obtain a second face rendering image.
2. The method of claim 1, wherein the first facial image and the second facial image are both images of real faces.
3. A face rendering apparatus, comprising:
the encoder is used for acquiring a first code of a first face image and a second code of a second face image, wherein the first code has characteristic information of the first face image, and the second code has characteristic information of the second face image;
the mapping network is used for acquiring a first control parameter of the three-dimensional face based on the first code and acquiring a second control parameter based on a third code;
an adaptive network for converting the first control parameter into a change factor, the change factor comprising an offset and a scale change factor;
a fuser for generating the third code according to the second code and the variation factor, so that the third code has attribute information of the first facial image and attribute information of the second facial image;
and the renderer is used for rendering the second face image according to the second control parameter to obtain a second face rendering image.
4. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor executes the computer program to implement the method according to claim 1 or 2.
5. A computer-readable storage medium, on which a computer program is stored, characterized in that the program is executed by a processor to implement the method according to claim 1 or 2.
CN202011206984.0A 2020-11-03 2020-11-03 Face rendering method and device, electronic equipment and storage medium Active CN112102461B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011206984.0A CN112102461B (en) 2020-11-03 2020-11-03 Face rendering method and device, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011206984.0A CN112102461B (en) 2020-11-03 2020-11-03 Face rendering method and device, electronic equipment and storage medium

Publications (2)

Publication Number Publication Date
CN112102461A CN112102461A (en) 2020-12-18
CN112102461B true CN112102461B (en) 2021-04-09

Family

ID=73784433

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011206984.0A Active CN112102461B (en) 2020-11-03 2020-11-03 Face rendering method and device, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN112102461B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115984947B (en) * 2023-02-21 2023-06-27 北京百度网讯科技有限公司 Image generation method, training device, electronic equipment and storage medium

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109978989A (en) * 2019-02-26 2019-07-05 腾讯科技（深圳）有限公司 Three-dimensional face model generation method, device, computer equipment and storage medium
CN110222722A (en) * 2019-05-14 2019-09-10 华南理工大学 Interactive image stylization processing method, calculates equipment and storage medium at system
CN111354079A (en) * 2020-03-11 2020-06-30 腾讯科技（深圳）有限公司 Three-dimensional face reconstruction network training and virtual face image generation method and device
CN111783603A (en) * 2020-06-24 2020-10-16 有半岛(北京)信息科技有限公司 Training method for generating confrontation network, image face changing method and video face changing method and device

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3381017B1 (en) * 2016-10-31 2019-11-06 Google LLC Face reconstruction from a learned embedding

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109978989A (en) * 2019-02-26 2019-07-05 腾讯科技（深圳）有限公司 Three-dimensional face model generation method, device, computer equipment and storage medium
CN110222722A (en) * 2019-05-14 2019-09-10 华南理工大学 Interactive image stylization processing method, calculates equipment and storage medium at system
CN111354079A (en) * 2020-03-11 2020-06-30 腾讯科技（深圳）有限公司 Three-dimensional face reconstruction network training and virtual face image generation method and device
CN111783603A (en) * 2020-06-24 2020-10-16 有半岛(北京)信息科技有限公司 Training method for generating confrontation network, image face changing method and video face changing method and device

Also Published As

Publication number Publication date
CN112102461A (en) 2020-12-18

Similar Documents

Publication Publication Date Title
CN110766777B (en) Method and device for generating virtual image, electronic equipment and storage medium
CN111784565B (en) Image processing method, migration model training method, device, medium and equipment
KR102627802B1 (en) Training method of virtual image generation model and virtual image generation method
WO2024051445A1 (en) Image generation method and related device
CN110766776B (en) Method and device for generating expression animation
CN111784566B (en) Image processing method, migration model training method, device, medium and equipment
CN112581635B (en) Universal quick face changing method and device, electronic equipment and storage medium
CN113272870A (en) System and method for realistic real-time portrait animation
KR102602112B1 (en) Data processing method, device, and medium for generating facial images
CN111368137A (en) Video generation method and device, electronic equipment and readable storage medium
EP3980974A1 (en) Single image-based real-time body animation
JP2019204476A (en) Image creation device, image creation method, and program
WO2020077913A1 (en) Image processing method and device, and hardware device
CN111383308A (en) Method and electronic equipment for generating animation expression
CN115965840A (en) Image style migration and model training method, device, equipment and medium
CN112102461B (en) Face rendering method and device, electronic equipment and storage medium
GB2614794A (en) Synthesizing sequences of 3D geometries for movement-based performance
CN114693876A (en) Digital human generation method, device, storage medium and electronic equipment
WO2020077912A1 (en) Image processing method, device, and hardware device
KR102470866B1 (en) Retargetimg method of 3d character facial expression and neural network learning method of thereof
CN114004905A (en) Method, device and equipment for generating character style image and storage medium
CN113850890A (en) Method, device, equipment and storage medium for generating animal image
CN115953597B (en) Image processing method, device, equipment and medium
CN114418835A (en) Image processing method, apparatus, device and medium
US10319133B1 (en) Posing animation hierarchies with dynamic posing roots

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
TR01 Transfer of patent right

Effective date of registration: 20210820

Address after: 100084 b201c-1, 3rd floor, building 8, yard 1, Zhongguancun East Road, Haidian District, Beijing

Patentee after: Beijing innovation Zhiyuan Technology Co.,Ltd.

Address before: B201d-1, 3rd floor, building 8, yard 1, Zhongguancun East Road, Haidian District, Beijing 100083

Patentee before: Beijing Zhiyuan Artificial Intelligence Research Institute

TR01 Transfer of patent right
TR01 Transfer of patent right

Effective date of registration: 20211209

Address after: B201a-1, floor 3, building 8, yard 1, Zhongguancun East Road, Haidian District, Beijing 100084

Patentee after: Beijing zhimeiyuansu Technology Co.,Ltd.

Address before: 100084 b201c-1, 3rd floor, building 8, yard 1, Zhongguancun East Road, Haidian District, Beijing

Patentee before: Beijing innovation Zhiyuan Technology Co.,Ltd.

TR01 Transfer of patent right