WO2025180271A1 - 处理音频数据的方法、训练神经网络的方法和相关装置 - Google Patents

处理音频数据的方法、训练神经网络的方法和相关装置

Info

Publication number
WO2025180271A1
WO2025180271A1 PCT/CN2025/078027 CN2025078027W WO2025180271A1 WO 2025180271 A1 WO2025180271 A1 WO 2025180271A1 CN 2025078027 W CN2025078027 W CN 2025078027W WO 2025180271 A1 WO2025180271 A1 WO 2025180271A1
Authority
WO
WIPO (PCT)
Prior art keywords
target
sample
facial
expression
information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/CN2025/078027
Other languages
English (en)
French (fr)
Other versions
WO2025180271A9 (zh
Inventor
季孝忠
朱俊伟
林楚铭
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology (Shenzhen) Co Ltd
Original Assignee
Tencent Technology (Shenzhen) Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology (Shenzhen) Co Ltd filed Critical Tencent Technology (Shenzhen) Co Ltd
Publication of WO2025180271A1 publication Critical patent/WO2025180271A1/zh
Publication of WO2025180271A9 publication Critical patent/WO2025180271A9/zh
Anticipated expiration legal-status Critical
Pending legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/242—Synchronisation processes, e.g. processing of PCR [Programme Clock References]
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/04—Time compression or expansion
    • G10L21/055—Time compression or expansion for synchronising with other signals, e.g. video signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
    • H04N21/251—Learning process for intelligent management, e.g. learning user preferences for recommending movies
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/45—Management operations performed by the client for facilitating the reception of or the interaction with the content or administrating data related to the end-user or to the client device itself, e.g. learning user preferences for recommending movies, resolving scheduling conflicts
    • H04N21/466—Learning process for intelligent management, e.g. learning user preferences for recommending movies
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/45—Management operations performed by the client for facilitating the reception of or the interaction with the content or administrating data related to the end-user or to the client device itself, e.g. learning user preferences for recommending movies, resolving scheduling conflicts
    • H04N21/466—Learning process for intelligent management, e.g. learning user preferences for recommending movies
    • H04N21/4662—Learning process for intelligent management, e.g. learning user preferences for recommending movies characterized by learning algorithms
    • H04N21/4666—Learning process for intelligent management, e.g. learning user preferences for recommending movies characterized by learning algorithms using neural networks, e.g. processing the feedback provided by the user

Definitions

  • the present disclosure relates to the field of artificial intelligence services, and more specifically to processing audio data.
  • Audio-driven face generation In recent years, technologies for generating video data from audio data have rapidly developed. Audio-driven face generation, in particular, holds broad application prospects. This technology parses input audio signals to extract information such as emotion and intonation, and uses this information to control the expressions and movements of virtual characters in real time, achieving highly synchronized performance. This technology has demonstrated widespread application value in the entertainment industry, game development, human-computer interaction interfaces, and virtual reality conferencing. With continued innovations in deep learning and neural network technologies, its expressiveness and realism are continuously improving.
  • Embodiments of the present disclosure provide a method for processing audio data, a method for training a neural network, an apparatus for processing audio data, an electronic device, and a computer-readable storage medium.
  • An embodiment of the present disclosure provides a method for processing audio data, the method comprising: determining facial prior information of a virtual object based on reference video data; determining target audio features based on target audio data; determining target expression sequence information based on the facial prior information and the target audio features; and determining target video data corresponding to the target audio data based on the target expression sequence information and a reference frame in the reference video data.
  • An embodiment of the present disclosure provides a method for training a neural network, wherein the neural network includes an audio-to-expression converter and an expression-to-face renderer, and the method includes: determining sample facial prior information of a virtual object based on sample video data; determining sample target audio features based on sample audio data corresponding to the sample video data; determining a predicted value of target sample expression sequence information using the audio-to-expression converter based on the sample facial prior information and the sample target audio features; training the audio-to-expression converter using a first loss function based on the predicted value of the target sample expression sequence information; determining a sample reference frame and a sample source video frame based on the sample video data; and training the expression-to-face renderer using a second loss function based on the sample facial prior information, the sample reference frame, and the sample source video frame.
  • An embodiment of the present disclosure provides a device for processing audio data, the device comprising: a first preprocessing module, configured to determine facial prior information of a virtual object based on reference video data; a second preprocessing module, configured to determine target audio features based on target audio data; and an audio-to-expression converter module, configured to determine target expression sequence information based on the facial prior information and the target audio features; and an expression-to-facial renderer, configured to determine target video data corresponding to the target audio data based on the target expression sequence information and a reference frame in the reference video data.
  • an embodiment of the present application provides a computer device, including:
  • the processor, the communication interface and the memory complete communication with each other through the communication bus;
  • the communication interface is an interface of the communication module;
  • the memory is used to store a computer program and transmit the computer program to the processor; the processor is used to call the computer program in the memory to execute the above method.
  • an embodiment of the present application provides a computer program product including a computer program, which, when executed on a computer, enables the computer to execute the above method.
  • the disclosed embodiment can accurately identify the facial features of the virtual object by determining the facial prior information of the virtual object in the reference video data, and thus can determine the target expression sequence information that conforms to the facial features of the virtual object and naturally integrates the facial features based on the facial prior information and the target audio features.
  • the target expression sequence information and the reference frame are used to determine the target video data, the synchronization accuracy between the audio content in the target video data and the facial components (such as the mouth) movements of the virtual object on the screen can be significantly improved, the real texture of the mouth movement can be enhanced, and the identity characteristics of the virtual object can be accurately reflected, and the expression differences between individuals can be more accurately captured and reproduced.
  • the overall operating efficiency and stability are also ensured, which is particularly suitable for real-time video generation.
  • FIG2 shows a schematic diagram according to an embodiment of the present disclosure
  • FIG3 shows a flowchart of a method for processing audio data according to an embodiment of the present disclosure
  • FIG4A shows a schematic diagram of an improved neural network model according to an embodiment of the present disclosure
  • FIG4C shows a schematic diagram of an audio to expression converter according to an embodiment of the present disclosure
  • FIG4D shows a schematic diagram of an expression-to-face renderer according to an embodiment of the present disclosure
  • FIG5A shows a flowchart of a method for training a neural network according to an embodiment of the present disclosure
  • FIG5B shows a schematic diagram of training an audio-to-expression converter according to an embodiment of the present disclosure
  • FIG6 shows a comparison diagram of test results according to an embodiment of the present disclosure
  • FIG7 shows a schematic diagram of an electronic device according to an embodiment of the present disclosure
  • FIG8 shows a schematic diagram of the architecture of an exemplary computing device according to an embodiment of the present disclosure
  • FIG9 shows a schematic diagram of a storage medium according to an embodiment of the present disclosure.
  • the models that can be used in the embodiments of the present disclosure can all be artificial intelligence models, especially artificial intelligence-based neural network models.
  • artificial intelligence-based neural network models are implemented as acyclic graphs in which neurons are arranged in different layers.
  • a neural network model includes an input layer and an output layer, which are separated by at least one hidden layer.
  • the hidden layer transforms the input received by the input layer into a representation that is useful for generating output in the output layer.
  • the nodes are fully connected to the nodes in the adjacent layers via edges, and there are no edges between the nodes in each layer.
  • the data received at the nodes of the input layer of the neural network is propagated to the nodes of the output layer via any one of the hidden layers, activation layers, pooling layers, convolutional layers, etc.
  • the input and output of the neural network model can take various forms, and the present disclosure does not limit this.
  • Figure 1 shows a schematic diagram of an application scenario 100 according to an embodiment of the present disclosure, in which a server 110 and multiple terminals 120 are schematically shown.
  • the neural network model of the embodiment of the present disclosure can be specifically integrated into various electronic devices, for example, the server 110 and any electronic device in the multiple terminals 120 in Figure 1.
  • the neural network model can be integrated into the terminal 120.
  • the terminal 120 can be a mobile phone, a tablet computer, a laptop computer, a desktop computer, a personal computer (PC), a smart speaker or a smart watch, etc., but is not limited to this.
  • the neural network model can also be integrated into the server 110.
  • the server 110 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.
  • the terminal and the server can be directly or indirectly connected through wired or wireless communication, and the present disclosure does not limit this.
  • the computer device that performs inference using the neural network model of the embodiments of the present disclosure can be either a terminal device, a server, or a system consisting of a terminal device and a server.
  • the method for processing audio data of the embodiments of the present disclosure can be executed on a terminal device, on a server, or jointly by a terminal device and a server.
  • the terminal 110 and the server 120 comply with the data protection principles, respect the user's data rights, and protect the user's data security and privacy.
  • the terminal 110 and the server 120 will clearly inform the user of the purpose, method and scope of collecting, using, storing, transmitting and deleting the user's data, and obtain the user's consent.
  • the terminal 110 and the server 120 will take reasonable technical and management measures to prevent the user's data from being leaked, tampered with, damaged or lost.
  • the terminal 110 and the server 120 provider will regularly review and update the user's data and delete expired or useless data in a timely manner.
  • the cloud service provider using the embodiment of the present disclosure respects the user's rights to data access, correction, deletion, withdrawal of consent, complaints and claims, and provides convenient channels and procedures so that users can effectively exercise these rights.
  • the data analysis process using artificial intelligence technology in the terminal 120 or server 110 is based on the principles of legality, rationality, and transparency.
  • the data collected and processed by the artificial intelligence model according to the embodiments of the present disclosure is relevant, necessary, and appropriate for the prediction purpose and does not contain any personally identifiable information or sensitive information.
  • the neural network model according to the embodiments of the present disclosure adopts appropriate technical and organizational measures to protect the security and integrity of the data and prevent unauthorized access, use, or disclosure of the data.
  • the artificial intelligence-based neural network model will comply with relevant data protection regulations and ethical principles.
  • the neural network model is trained based on a large amount of anonymized and de-identified data and does not infringe on the privacy rights of any individual or group.
  • the artificial intelligence model has also undergone rigorous testing and evaluation to ensure that its output results are accurate and reliable and will not cause any misleading or discriminatory effects.
  • the artificial intelligence model is designed only to improve service quality and customer satisfaction and will not be used for any illegal or unethical purposes.
  • the neural network model will be reviewed and updated regularly to adapt to changes in the data environment and legal regulations.
  • audio-driven virtual face generation methods can be divided into two categories: person-specific methods and person-generic methods.
  • Person-specific methods require neural network models to be trained or fine-tuned on a specific individual before prediction, while person-generic methods can directly generate speaking facial videos of unseen virtual characters.
  • some solutions employ fusion modeling approaches, aiming to more efficiently and accurately capture mouth movement features. Specifically, they can predict target facial region labels based on audio data or extract key point information in an unsupervised environment. Although facial region labels and key points are relatively easy to obtain, the inherent sparsity of this information prevents these solutions from fully expressing complex facial movements such as pursing or closing the lips.
  • the disclosed embodiments improve the neural network model architecture.
  • the improved neural network model consists of two decoupled modules: an audio-to-expression transformer and an expression-to-face renderer.
  • the improved neural network model can process input data in three stages to achieve higher performance: preprocessing, audio-to-expression conversion, and expression-to-video frame reconstruction.
  • the preprocessing module can perform necessary feature extraction on the reference video data and target audio data to obtain the facial prior information and target audio features of the virtual object.
  • the audio-to-expression converter maps the received facial prior information and target audio features into corresponding target expression sequence information, especially for subtle expression changes in mouth movements.
  • the expression-to-video frame reconstruction stage the expression-to-facial renderer accurately generates target video data corresponding to the target audio data based on the target expression sequence information, the reference frame in the reference video data, and the reference head pose feature vector corresponding to the reference frame.
  • the target video data is precisely synchronized with the target audio data and has highly realistic mouth movements and facial expression details.
  • the improved multi-stage neural network model can significantly improve the synchronization accuracy between the audio content and the facial components (such as the mouth) movements of the virtual object on the screen, enhance the real texture of the mouth movement, more accurately capture and reproduce the differences in facial expressions between individuals, and, while improving the performance of the system, also ensure the overall operating efficiency and stability.
  • the facial components such as the mouth
  • embodiments of the present disclosure provide a method for processing audio data to utilize the improved neural network model for inference.
  • the method comprises: determining facial prior information of a virtual object based on reference video data; determining target audio features based on target audio data; determining target expression sequence information based on the facial prior information and the target audio features; and determining target video data corresponding to the target audio data based on the target expression sequence information, a reference frame in the reference video data, and a reference head pose feature vector corresponding to the reference frame.
  • the improved neural network inference process significantly improves the synchronization between the audio content and the facial components of the virtual object on screen by first determining the target expression sequence and then predicting the video frame based on this sequence.
  • This approach not only enhances the realism of facial movements but also accurately reflects the identity of the virtual object. While improving performance, it also ensures the efficiency and stability of the neural network, making it particularly suitable for real-time video generation.
  • the embodiment of the present disclosure also provides a method for training a neural network to train the improved neural network model.
  • the neural network includes an audio-to-expression converter and an expression-to-face renderer.
  • the method includes: training the audio-to-expression converter based on sample video data and sample audio data corresponding to the sample video data; and training the expression-to-face renderer based on sample facial prior information, sample reference frames, and sample source video frames.
  • the audio-to-expression converter and the expression-to-face renderer can be trained separately and independently, and the training processes of the two do not interfere with each other.
  • an improved first loss function and a modified second loss function are optionally introduced. These two loss functions optimize different aspects of the neural network's performance by adjusting the neural network parameters.
  • the first and second loss functions enable the neural network to more effectively learn and adapt to the characteristics of the input data, optimizing multiple aspects during the training process.
  • This loss function design which comprehensively considers different characteristics, helps improve the generalization ability and training efficiency of the neural network, thereby enhancing its performance in practical applications.
  • the following describes a method for processing audio data, a method and an apparatus for training a neural network according to embodiments of the present disclosure with reference to FIG. 2 to FIG. 9 .
  • FIG2 shows a schematic diagram according to an embodiment of the present disclosure.
  • the specific application scenario for implementing audio-driven virtual face generation technology is as follows: there is a video clip (hereinafter referred to as reference video data) that corresponds to an audio clip.
  • the video content shows the facial expressions of a virtual character speaking to the camera, especially the details of their mouth movements.
  • the virtual character utters a sentence, such as "What a nice day today.”
  • target audio data another different audio clip
  • the neural network model according to an embodiment of the present disclosure is mainly composed of two parts: an audio to expression converter and an expression to facial renderer.
  • the embodiment of the present disclosure extracts facial prior information from the reference video data, including but not limited to: reference facial shape information, reference expression information, and reference head pose information.
  • audio features are also extracted from the target audio data.
  • a three-dimensional deformable face model (3D Morphable Face Model, 3DMM) can be used as a three-dimensional morphological model to extract more accurate facial shape information, posture information and expression information (such as the reference facial shape information, reference expression information and reference head pose information and other facial prior information mentioned later) from the reference video data to achieve accurate mapping from video to expression sequence.
  • the three-dimensional morphological model according to the embodiment of the present disclosure helps the neural network to learn the movements and changes of facial components (for example, the mouth) more effectively and accurately.
  • the facial prior information extracted from the preprocessing will be input into the audio-to-expression converter to generate the target expression sequence information, so as to better preserve and reflect the identity characteristics of the virtual human body and subtle changes in emotional expression.
  • the target expression sequence information will be used as the input of the expression-to-face renderer, so that the renderer can use the improved facial identity alignment module to accurately generate the motion video images of the target facial components (for example, the mouth, eyes, ears, eyebrows and other components of the target face).
  • the expression-to-face renderer can use a facial individual alignment module (FIA), which only requires one reference video frame to obtain the target video frame that conforms to the facial personality characteristics of the virtual character when pronouncing the pronunciation.
  • FOA facial individual alignment module
  • the expression-to-face renderer according to the embodiment of the present disclosure improves the personalized expression of the target video frame, reduces the computing cost, and helps to improve the reasoning speed.
  • the disclosed embodiments can generate fine details of facial components (used to represent virtual organs) in real time without relying on complex and inefficient computing modules, significantly improving the fineness of video images and the efficiency of generating video data.
  • the following description of this disclosure mainly uses the mouth as an example, those skilled in the art should understand that the disclosed solutions can be applied to the generation of any facial organs related to the expression and emotional communication. Of course, this disclosure is not limited to this.
  • FIG3 shows a flowchart of a method 30 for processing audio data according to an embodiment of the present disclosure.
  • Method 30 may be performed at a terminal device or a server (such as the terminal 120 or the server 110 described in FIG1 ).
  • Method 30 includes the following operations S301 to S304 .
  • method 30 may also include more or fewer operations, and the present disclosure is not limited thereto.
  • operations S301 and S302 are operations in the pre-processing stage, which can be performed by any facial information extraction model.
  • Operation S303 can be performed in the conversion stage of audio information to expression information, and operation S304 can be performed in the expression to video frame reconstruction stage.
  • facial prior information of a virtual object is determined based on reference video data.
  • the video content involved in the reference video data includes a virtual object, which is an object with a face and can pronounce words, such as a real person or animal, a cartoon character, a virtual three-dimensional model, etc.
  • the reference video data can be obtained by capturing a real scene using a video capture device, or can be generated by software.
  • the reference video data may be input video data such as that shown in FIG2 , in which a virtual object says, "It's a nice day today.”
  • the video data includes multiple reference video frames, each of which is an image.
  • the image shows the head of the virtual object.
  • the virtual object's facial prior information refers to accurate facial information used to identify the virtual object's face and is used to reflect the virtual object's facial features.
  • the virtual object's facial prior information includes one or more of the following: reference facial shape information, reference expression information, and reference head pose information.
  • reference facial shape information reference facial shape information
  • reference expression information reference expression information
  • reference head pose information reference head pose information
  • the reference facial shape information can be information used to characterize the facial shape features of a virtual object.
  • the reference facial shape information can be extracted from any reference video frame, which may include the texture and 3D shape information of the virtual object's face.
  • an expressionless 3D facial avatar of the virtual object can be obtained by directly performing an affine transformation on a universal 3D standard virtual object.
  • a 3D standard virtual object also known as a white model virtual object, is a standardized model used to simulate human facial expressions and movements. These models typically have a uniform three-dimensional shape and structure and are used to represent the basic appearance of a virtual object.
  • the present disclosure is not limited to this.
  • reference expression information can be any information used to characterize the expressive characteristics of a virtual object, and is used to reflect the expressive characteristics of the virtual object when expressing emotions in different languages.
  • Reference expression information can be extracted from each reference video frame to obtain a reference expression feature vector ⁇ i corresponding to each reference video frame i .
  • Reference expression information can refer to the set of reference expression feature vectors corresponding to all reference video frames.
  • a 3D facial avatar of the virtual object with a specific expression in a particular frame can be obtained by directly performing an affine transformation on a universal 3D standard virtual object.
  • the present disclosure is not limited to this.
  • the reference head pose information is any information used to characterize the posture characteristics of the head of a virtual object.
  • the reference head pose information is extracted from each reference video frame to obtain a reference head pose feature vector corresponding to each reference video frame i, and the reference head pose information can refer to a set of reference head pose feature vectors corresponding to all reference video frames.
  • the universal 3D standard virtual object can be directly adjusted by rotation to achieve the head pose corresponding to the reference head pose feature vector.
  • the present disclosure is not limited to this.
  • target audio features are determined based on target audio data.
  • the audio identified by the target audio data is the audio that needs to be synthesized as an audio-virtual object, that is, the audio that needs to be emitted by the virtual object in the target video data determined in the subsequent S304.
  • the target audio data can be the target audio data as shown in Figure 2, where the audio corresponding to the input video data is "I love you".
  • the target audio data may include multiple audio frames, and the target audio features refer to the sound-related features extracted from these audio frames. These features may include spectral information, time domain information, sound intensity, etc. of the audio signal. Of course, the present disclosure is not limited to this.
  • target expression sequence information is determined based on the facial prior information and the target audio features.
  • the target expression sequence information can be reference expression information corresponding to the target audio feature.
  • the target expression sequence information is a collection of target expression feature vectors arranged in time sequence, and each target expression feature vector corresponds to a target audio frame, which is used to reflect the expression characteristics of the virtual object when speaking a specific sentence.
  • This mapping relationship can be in the form of many-to-one, one-to-one, or one-to-many.
  • a target expression feature vector can correspond to one frame of target audio frame, or it can correspond to multiple frames of target audio frames, or even multiple target expression feature vectors can correspond to a single target audio frame.
  • the present disclosure is not limited to this.
  • the target expression sequence information may be directly determined based solely on the reference facial shape information and reference expression information in the facial prior information, combined with the target audio features.
  • target video data corresponding to the target audio data is determined based on the target expression sequence information and a reference frame in the reference video data.
  • the virtual object involved in the target video data and the virtual object involved in the reference video data in S301 are the same virtual object, except that the virtual object in the target video data emits audio corresponding to the target audio data.
  • the target expression sequence information can accurately identify the expression characteristics of the virtual object when the virtual object emits the audio corresponding to the target audio characteristics, and the reference frame can identify the relevant information of the virtual object, the target expression sequence information can be used to accurately simulate the facial condition of the virtual object emitting the audio corresponding to the target audio data based on the guidance of the reference frame in the reference video data, thereby obtaining the target video data.
  • the number of target video frames in the target video data can match the number of audio frames in the target audio data.
  • Each target video frame shows a specific facial expression of the virtual object when speaking the corresponding audio frame.
  • the mouth movement (i.e., mouth shape) of the virtual object when speaking the corresponding audio frame should correspond to the pronunciation of the audio frame. The consistency between the facial expression of the virtual object presented in the target video frame and its pronunciation makes the generated video more realistic and lifelike.
  • the reference frame I r in the reference video data can be any video frame in the reference video data, which is used to provide the basic shape and texture of the virtual object as reference information during the expression-to-video-frame reconstruction phase.
  • the same reference frame I r can be used, or different reference frames can be used, although this disclosure is not limited thereto.
  • a source video frame can be extracted from the reference video data; the source video frame can be masked based on the target expression sequence to obtain a first video frame, wherein, in the first video frame, part of the facial components of the virtual object are replaced by black pixel blocks; and based on the first video frame and the reference frame, the facial prior information is used to determine the target video data corresponding to the target audio data.
  • the source video frame can be one or more reference frames.
  • This application also does not limit the number of source video frames used when determining the target data.
  • it can be one source video frame or multiple source video frames.
  • masking the source video frame based on the target expression sequence to obtain a first video frame includes: determining a pixel area corresponding to partial facial components of the virtual object in the source video frame based on the target expression sequence and the reference facial shape information; and modifying the pixel values in the pixel area to preset values.
  • the method of determining the target video data corresponding to the target audio data based on the first video frame and the reference frame using the facial prior information includes: encoding the first video frame and the reference frame using a shared weight encoder to obtain the encoding features of the first video frame and the encoding features of the reference frame; using the facial prior information and the target expression sequence, and decoding the encoding features of the first video frame and the encoding features of the reference frame using a decoder to obtain the target video data corresponding to the target audio data, wherein the target video data includes a target video frame, and the target video frame includes predicted values of partial facial components of the virtual object replaced by black pixel blocks.
  • the first video frame may be generated based on the target expression sequence information.
  • the first video frame The facial components in the video frame are masked, that is, the facial components in the video frame are replaced by black pixel blocks
  • the target video frame is determined based on the source video frame I s and the reference frame I r .
  • the following description will be made by taking the mask of the mouth component (also referred to as the mouth mask) as an example. Those skilled in the art will appreciate that this solution can also be applied to any facial component.
  • a mouth mask is used to mask facial structures related to mouth movement (e.g., the lower half of the face), while retaining variable facial structures unrelated to expression changes or the target audio frame.
  • a facial identity alignment module is used to predict pixel values at the mouth mask. These predicted pixel values at the mouth mask are then applied to the target video frame to obtain the target video frame. This disclosure is not limited to this.
  • a facial identity alignment operation is also performed, which uses a facial identity alignment module with a cross-attention structure to establish an effective connection between target expression related parameters (such as target expression feature vectors) and the effective facial texture of a single reference frame.
  • target expression related parameters such as target expression feature vectors
  • the module can achieve more precise lip shape control.
  • the cross-attention structure allows the module to pay attention to the target expression and the facial texture information extracted from a single reference frame at the same time. This attention mechanism enables the module to accurately adjust the shape and movement of the lips based on the target expression feature vector and the facial texture of the reference frame. By combining these two types of information, the module can better understand the features and structure of the face and accurately control the lip movement.
  • the method according to the embodiment of the present disclosure can improve the accuracy and credibility of the lip movements of virtual objects, enhance the communication effect and visual expressiveness of the video.
  • the embodiment of the present disclosure can generate fine details of facial organs in real time without relying on complex and inefficient computing modules, greatly improving the fineness of the video picture and the efficiency of generating video data.
  • Figure 4A shows a schematic diagram of an improved neural network model according to an embodiment of the present disclosure.
  • Figure 4B shows a schematic diagram of the audio-to-expression conversion stage according to an embodiment of the present disclosure.
  • Figure 4C shows a schematic diagram of the audio-to-expression converter according to an embodiment of the present disclosure.
  • Figure 4D shows a schematic diagram of the expression-to-facial renderer according to an embodiment of the present disclosure.
  • a deep 3D facial reconstruction model can be used to perform operation S301 to determine the virtual object's facial prior information based on reference video data.
  • Facial prior information includes reference facial shape information ( ⁇ 1 ), reference expression information ( ⁇ 1 , ..., ⁇ N ), and reference head pose information ( ⁇ 1 , ..., ⁇ N ).
  • the deep 3D facial reconstruction model can extract 3D coefficients, including shape coefficients, expression coefficients, and pose coefficients.
  • the deep 3D facial reconstruction model can utilize a hybrid loss function for weakly supervised learning that incorporates both low-level and perceptual-level information for supervision.
  • the deep 3D facial reconstruction model can utilize complementary information from different images for multi-image facial reconstruction, achieving shape aggregation.
  • the deep 3D facial reconstruction model is fast, accurate, and robust, and can handle occlusions and large poses.
  • various embodiments of the present disclosure may also utilize other neural network models to extract the virtual object's facial prior information.
  • a hidden-unit Bert model (hereinafter referred to as a HuBert model) can be used to perform self-supervised speech representation learning, and target audio features (w 1 ,..., w l ) are determined based on the target audio data by masking the hidden units.
  • the HuBert model uses a self-supervised learning method to learn high-quality speech representations by masking the hidden units.
  • various embodiments of the present disclosure may also use other neural network models to determine the target audio features.
  • the audio to expression converter can use a reference facial shape information ( ⁇ 1 ) and N reference expression information ( ⁇ 1 ,..., ⁇ N ) in the 3D coefficients, combined with l target audio features (w 1 ,...,w l ) to obtain target expression sequence information, which includes T target expression features N, l and T can be the same value or different values, and the present disclosure is not limited thereto.
  • each virtual object has a unique facial and mouth structure, and each virtual object also has different speaking habits.
  • facial prior information including reference facial shape information ( ⁇ 1 ) and N reference expression information ( ⁇ 1 ,..., ⁇ N ), can reflect the personality or identity characteristics of the virtual object.
  • the reference facial shape information ( ⁇ 1 ) characterizes the identity of the virtual object through the texture and shape characteristics of the face.
  • this reference facial shape information can be associated with natural facial dimensions and oral cavity proportions.
  • the reference expression information captures the individual's unique speaking style, which provides a difference from the standard shape for personalized virtual objects.
  • Reference expression information includes factors such as the amplitude of mouth movement and emotional expression. By comprehensively utilizing this information, facial expression sequences that match the target audio can be more accurately generated, making the virtual object's expression more vivid and natural.
  • the audio-to-expression converter is based on a transformer architecture, which includes an encoder and a decoder.
  • the step of determining target expression sequence information optionally includes: encoding the reference facial shape information ( ⁇ 1 ), the reference expression information ( ⁇ 1 ,..., ⁇ N ), and the target audio features (w 1 ,...,w l ) to obtain a latent vector (Z); and decoding the latent vector (Z) to determine the target expression sequence information.
  • Figure 4B further illustrates some details of the audio-to-expression converter.
  • relevant information such as facial shape and expression related to the virtual object can be accurately associated with the different audios involved in the target audio features, so that the obtained target expression sequence information can more naturally identify the facial expression of the virtual object when emitting the target audio features.
  • the reference facial shape information ⁇ 1 in the 3D coefficient (which will serve as the default reference facial shape information), N reference expression information ( ⁇ 1 ,..., ⁇ N ), and l target audio features (w 1 ,...,w l ) can be passed through a fully connected network to obtain embedding vectors. These embedding vectors are then concatenated in time order to obtain a total of 1+N+1 tokens, which are input into the Multi-Head Self-Attention (MHSA) (also known as the Transformer Encoder) for encoding to obtain the latent vector Z.
  • MHSA Multi-Head Self-Attention
  • the expression query vector qt (initially set to zero) is combined with the hidden vector Z and fed into the Multi-Head Cross-Attention (MHCA) network (also known as the transformer decoder) for decoding to obtain T target expression features.
  • MHCA Multi-Head Cross-Attention
  • the expression-to-face renderer first obtains any reference video frame from the reference video data as a source video frame I s .
  • the mth reference video frame can be extracted from the reference video data as the source video frame.
  • the present disclosure is not limited to this.
  • the expression to face renderer can be based on a target expression feature vector in the target expression sequence information and the source video frame I s , generating the first video frame
  • the first video frame The facial components in are masked.
  • the area corresponding to the masked facial components is learnable, that is, the size and shape of the area can be associated with both the expression information and the identity information of the virtual object.
  • the first video frame is generated The process is learnable. As shown in FIG4B , the source video frame I s can be selectively masked, allowing for more accurate control over the modification of facial component areas. First, based on the predicted expression feature vector, 3D vertices are estimated and then projected onto the image. Next, based on these 3D vertices, the mouth and neck areas can be drawn and filled with black pixel blocks (or pixel blocks with pixel values of any preset values). That is, relative to the source video frame, the first video frame has a mask area, which belongs to the source video frame that is masked. The mask area of the first video frame can be a black pixel block. In this area of the source video frame, part of the facial components of the virtual object are included.
  • the facial area that needs to be predicted can be accurately identified, and this area is directly related to the target audio features emitted by the virtual object, thereby improving the pertinence and accuracy of the prediction.
  • First video frame The reference frame Ir in the reference video data is input to a shared weight encoder to process the two images to extract multi-scale features. Specifically, the shared weight encoder converts the first video frame It is encoded as feature Fs , and the reference frame Ir is encoded as feature Fr.
  • the shared weight encoder is composed of d stacked residual blocks, so the encoder is also called a d-level encoder. At each level (residual block), the image resolution is halved while the feature dimension of the extracted is increased. This setting allows for efficient extraction of multi-scale features in the shared weight encoder. Since the weights of the encoder are shared, the first video frame can be processed in parallel. and the reference frame I r . Specifically, the encoder extracts features of various scales from the input image through multiple levels of processing. For the i-th level in the d-level encoder, its multi-scale features can be calculated as follows.
  • SE i represents the i-th level of the encoder.
  • SE i represents the i-th level of the encoder.
  • a representation containing features of different scales and abstract levels is obtained. These features can capture different details and structural information of the input image.
  • the reference frame I r obtains a set of feature vectors of d sizes, that is, and Since the first video frame Sharing encoder weights with the reference frame I r allows for more efficient processing of the source and reference frames, extracting their shared feature representations. This parallel computing approach not only speeds up processing but also ensures that the feature representations of the source and reference frames are compared in the same space, facilitating subsequent matching and fusion.
  • the expression to face renderer will use the decoder including the face identity alignment module to transform the first video frame
  • the corresponding feature Fs and the corresponding feature Fr of the reference frame Ir are decoded to obtain the target video frame.
  • the improved neural network inference process by first determining the target expression sequence and then predicting the video frame based on this sequence, significantly improves the synchronization between the audio content and the facial components of the virtual object on the screen.
  • This approach not only enhances the realism of facial movements but also accurately reflects the identity of the virtual object. While improving performance, it also ensures the efficiency and stability of the neural network, making it particularly suitable for real-time video generation.
  • FIG4D shows more example details of the decoder.
  • the decoder is composed of d stacked decoding layers, so the decoder is also called a d-level decoder.
  • the decoder includes d decoding layers, and the input of each decoding layer includes the output of the previous decoding layer, the facial prior information, and the encoded features of the reference frame.
  • the output of each decoding layer is the predicted value of the target video frame. Assume that the input features of each level of the decoder are The output features are
  • the 3D coefficient Before being fused to the decoder through the adapter (AdalN) module, dimension mapping is performed through a three-layer MLP. This process aims to map the 3D coefficients into a feature space compatible with the decoder network for subsequent feature fusion and generation processes.
  • the input features of each level of the decoder are First upsample to increase the resolution to get the features feature
  • the 3D coefficient Modulated to match the shape of specific facial components, thus better controlling the movement of facial components and the generation of facial expressions.
  • the modulated features are compared with the features from the reference frame I r Aligned at the same resolution, the two are fused to control face generation and aggregate texture.
  • the above process can be expressed as follows.
  • FIA i is a first-level (or one-layer) decoding layer in the decoder including the face identity alignment module.
  • a fusion strategy can be further adopted.
  • the output of the last decoding layer of the decoder can be fused with the output of the first decoding layer of the decoder to obtain the final output.
  • This fusion strategy aims to retain only the pixels corresponding to the predicted facial components while eliminating background interference, making the generated image clearer and focused on facial features. The process can be expressed as follows.
  • Figure 5A shows a flow chart of a method 50 for training a neural network according to an embodiment of the present disclosure, which can be used to train the improved neural network model shown in Figures 4A to 4C.
  • Figure 5B shows a schematic diagram of training an audio to expression converter according to an embodiment of the present disclosure.
  • Method 50 may be performed at a terminal device or a server (such as the terminal 120 or the server 110 described in FIG1 ).
  • Method 50 includes the following operations S501 to S506 .
  • method 50 may also include more or fewer operations, and the present disclosure is not limited thereto.
  • sample facial prior information of a virtual object is determined based on sample video data.
  • sample target audio features are determined based on sample audio data corresponding to the sample video data.
  • the facial prior information obtained based on sample video data includes: sample facial shape information ( ⁇ 1 ), sample target expression sequence information (e.g., sample expression information ⁇ 1 ,..., ⁇ N ), and sample head pose information ( ⁇ 1 ,..., ⁇ N ).
  • sample facial shape information ⁇ 1
  • sample target expression sequence information e.g., sample expression information ⁇ 1 ,..., ⁇ N
  • sample head pose information ⁇ 1 ,..., ⁇ N
  • This facial prior information can be directly obtained through preprocessing.
  • Such facial prior information can serve as the true value of the virtual object's facial prior information.
  • the preprocessing process has been described in detail above and will not be repeated here in this disclosure.
  • the sample audio data corresponding to the sample video data can be extracted to directly determine the sample audio features corresponding to the sample audio data.
  • the sample audio data corresponding to the sample video data refers to, for example, the audio data containing the speech "The weather is so nice today" when a virtual person in the sample video data is saying "The weather is so nice today.”
  • the audio-to-expression converter is used to determine the predicted value of the target sample expression sequence information.
  • the audio-to-expression converter is trained using a first loss function.
  • the sample facial prior information may include the true value of the sample target expression sequence information
  • operation S503 includes: masking the true value of the facial prior information of the virtual object to obtain the masked facial prior information of the virtual object; based on the masked facial prior information of the virtual object and the sample audio features, using the audio-to-expression converter to determine the predicted value of the facial prior information of the sample target expression sequence information of the virtual object.
  • the present disclosure is not limited to this.
  • the audio to expression converter in training can be used to predict the predicted value of the masked portion of the sample target expression sequence information of the virtual object (for example, ).
  • the training goal of the audio-to-expression converter is to make the true value of the facial prior information ⁇ t and the predicted value Therefore, the parameters of each neuron in the audio-to-expression converter can be continuously adjusted to make the first loss function reach an extreme value.
  • the present disclosure is not limited to this.
  • the first loss function for training the audio-to-expression converter may include, but is not limited to, mean square error (MSE) loss and 3D vertex loss.
  • MSE mean square error
  • MSE mean square error
  • the 3D vertex loss is calculated by combining the predicted expression feature vectors
  • the predicted positions of 3D vertices are calculated using the facial shape information ⁇ 1 and the reference head pose information ⁇ . These vertices are key points (e.g., mouth corners) selected from a facial region (e.g., mouth) to evaluate the distance between the true and predicted values of the key points in that facial region.
  • the generated 3D vertices can be optionally aligned with the mouth key points.
  • V denote the 3D vertices calculated from the coefficients.
  • the 3D vertex loss LV can be described as follows:
  • the present disclosure is not limited thereto.
  • the training process of the expression-to-face renderer is described based on operations S505 and S506.
  • the training process of the audio-to-expression converter and the training process of the expression-to-face renderer can be independent of each other.
  • the input data used in the training process of the expression-to-face renderer can be the sample facial prior information (which optionally includes the true value of the sample target expression sequence information), the sample reference frame, and the sample source video frame. These input data can all be obtained in the preprocessing process (i.e., operations S501 and S502) without being processed by the audio-to-expression converter.
  • the audio-to-expression converter and the expression-to-face renderer can also be trained jointly, and the present disclosure is not limited to this.
  • the sample facial prior information includes the true value of the sample target expression sequence information.
  • a sample reference frame and a sample source video frame are determined based on the sample video data.
  • the expression-to-face renderer is trained using a second loss function based on the sample facial prior information, the sample reference frame, and the sample source video frame.
  • the sample source video frame is masked based on the true value of the target expression sequence to obtain a masked sample source video frame; based on the sample facial prior information and the true values of the sample reference frame and the sample target expression sequence information, the expression-to-face renderer is used to determine the predicted value of the sample source video frame for the masked sample source video frame; and based on the predicted value and the true value of the sample source video frame, the expression-to-face renderer is trained using the second loss function; wherein the second loss function includes a combination of one or more of the following items: pixel loss, perceptual loss, adversarial loss and local pixel loss.
  • the sample face prior information includes 3D coefficients
  • the sample source video frame can first be masked (i.e., a masked sample source video frame is obtained, in which facial components are replaced by black pixel blocks).
  • the predicted value of the sample source video frame is then determined based on these 3D coefficients and the sample reference frame.
  • the training goal of the expression-to-face renderer is to ensure that the true value of the sample source video frame is sufficiently close to the predicted value. Therefore, the parameters of each neuron in the expression-to-face renderer can be continuously adjusted to maximize the second loss function.
  • the present disclosure is not limited to this.
  • the second loss function used to train the expression-to-video renderer may adopt a combination of multiple loss functions to constrain the synchronization between facial components and expressions to improve the visual quality of the generated video frames.
  • pixel loss can be used to measure the pixel-level difference between the generated image and the target image.
  • the pixel difference between the true value I s promotes the accuracy and clarity of the generated video frame.
  • the pixel loss L 1 can be calculated as:
  • the present disclosure is not limited thereto.
  • perceptual loss is based on the features extracted by deep learning models (such as VGG network) (i.e., perceptual information obtained from images) to measure the prediction value of generating sample source video frames.
  • the perceptual difference between the generated image and the true value I s helps improve the realism and naturalness of the generated image to better match the target expression.
  • the perceptual loss can be calculated as:
  • the adversarial loss evaluates the predicted value of a sample source video frame via a discriminator network in a generative adversarial network (GAN) framework.
  • the generator network aims to fool the discriminator into being unable to distinguish the predicted values of the sample source video frames. and the true value I s , thereby improving the authenticity and realism of the image.
  • the adversarial loss can be calculated as: where D( ⁇ ) represents the discriminator.
  • local pixel loss focuses on enhancing pixel-level details in specific areas, such as the details of the mouth. By introducing additional loss terms for specific facial areas, the generated image is made clearer and more realistic in details such as the mouth.
  • the local pixel loss can be calculated as: Where M' represents the binary mask of the target facial region.
  • the second loss function is obtained by weighted summing the above loss functions.
  • the above loss functions act together in the training process of the expression-to-video renderer to ensure that the generated video can accurately reflect the target expression while maintaining a high level of visual quality and detail.
  • ⁇ 1 to ⁇ 4 are all hyperparameters, which can be set to any other values in actual applications, and the present disclosure is not limited thereto.
  • an improved first loss function and a modified second loss function are optionally introduced. These two loss functions optimize different aspects of the neural network's performance by adjusting the neural network parameters.
  • the first and second loss functions enable the neural network to more effectively learn and adapt to the characteristics of the input data, optimizing multiple aspects during the training process.
  • This loss function design which comprehensively considers different characteristics, helps improve the generalization ability and training efficiency of the neural network, thereby enhancing its performance in practical applications.
  • An embodiment of the present disclosure also provides a device for processing audio data, which includes: a first preprocessing module for determining facial prior information of a virtual object based on reference video data; a second preprocessing module for determining target audio features based on target audio data; and an audio-to-expression converter module for determining target expression sequence information based on the facial prior information and the target audio features; and an expression-to-facial renderer for determining target video data corresponding to the target audio data based on the target expression sequence information and a reference frame in the reference video data.
  • the facial prior information includes one or more of the following: reference facial shape information, reference expression information, and reference head pose information.
  • the audio-to-expression converter module is further configured to:
  • the hidden vector is decoded to determine the target expression sequence information.
  • the expression-to-face renderer is further configured to:
  • target video data corresponding to the target audio data is determined using the facial prior information.
  • the expression-to-face renderer is further configured to:
  • the pixel values in the pixel area are modified to preset values to obtain the first video frame.
  • the expression-to-face renderer is further configured to:
  • a decoder is used to decode the first encoding feature and the second encoding feature to obtain target video data corresponding to the target audio data, wherein the target video data includes a target video frame, and an area of black pixel blocks in the target video frame is replaced by a predicted value of a partial facial component of the virtual object.
  • the decoder includes multiple cascaded decoding layers, the input of each decoding layer includes the output of the previous decoding layer, the facial prior information and the encoding features of the reference frame, and the output of each decoding layer is the predicted value of the target video frame.
  • the facial features of the virtual object can be accurately identified, and thus the target expression sequence information that conforms to the facial features of the virtual object and naturally integrates the facial features can be determined based on the facial prior information and the target audio features.
  • the target expression sequence information and the reference frame are used to determine the target video data, the synchronization accuracy between the audio content in the target video data and the facial components (such as the mouth) of the virtual object on the screen can be significantly improved, the real texture of the mouth movement can be enhanced, and the identity characteristics of the virtual object can be accurately reflected.
  • the expression differences between individuals can be more accurately captured and reproduced.
  • the overall operating efficiency and stability are also ensured, making it particularly suitable for real-time video generation.
  • the present disclosure also provides a device for training a neural network, wherein the neural network includes an audio-to-expression converter and an expression-to-face renderer, and the device includes:
  • a first determining module is used to determine sample facial prior information of a virtual object based on sample video data
  • a second determining module configured to determine a sample target audio feature based on the sample audio data corresponding to the sample video data
  • a third determination module is configured to determine a predicted value of target sample expression sequence information using the audio-to-expression converter based on sample facial prior information and sample target audio features;
  • a first training module is configured to train the audio-to-expression converter using a first loss function based on a predicted value of the target sample expression sequence information
  • a fourth determining module configured to determine a sample reference frame and a sample source video frame based on the sample video data
  • the second training module is used to train the expression-to-face renderer based on the sample facial prior information, the sample reference frame, and the sample source video frame using a second loss function.
  • the sample facial prior information includes a true value of sample target expression sequence information
  • the third determination module is further configured to:
  • the audio-to-expression converter is used to determine a predicted value of the sample target expression sequence information of the virtual object.
  • the sample facial prior information includes a true value of sample target expression sequence information
  • the second training module is further configured to:
  • the expression to face renderer is trained using the second loss function
  • the second loss function includes one or more of the following: pixel loss, perceptual loss, adversarial loss, and local pixel loss.
  • an improved first loss function and a modified second loss function are optionally introduced. These two loss functions optimize different aspects of the neural network's characteristics, respectively. By adjusting the neural network parameters, the performance of the trained neural network is improved.
  • the first and second loss functions enable the neural network to more effectively learn and adapt to the characteristics of the input data, and optimize multiple aspects during the training process.
  • This loss function design which comprehensively considers different characteristics, helps improve the generalization ability and training efficiency of the neural network, thereby improving its performance in practical applications.
  • the predicted result of the video frame obtained according to the solution of the embodiment of the present disclosure is very close to the real result of the video frame, and the degree of restoration of the detailed texture is much higher than that of the traditional solution.
  • FIG7 shows a schematic diagram of an electronic device 2000 according to an embodiment of the present disclosure.
  • the electronic device 2000 may include one or more processors 2010 and one or more memories 2020.
  • the memory 2020 stores computer-readable code, which, when executed by the one or more processors 2010, may execute the method described above.
  • the processor in the embodiments of the present disclosure may be an integrated circuit chip having signal processing capabilities.
  • the processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component.
  • DSP digital signal processor
  • ASIC application-specific integrated circuit
  • FPGA field-programmable gate array
  • the methods, operations, and logic block diagrams disclosed in the embodiments of the present disclosure may be implemented or executed.
  • the general-purpose processor may be a microprocessor or any conventional processor, and may be an X86 architecture or an ARM architecture.
  • various example embodiments of the present disclosure may be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device.
  • firmware or software may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device.
  • blocks, devices, systems, techniques, or methods described herein may be implemented, as non-limiting examples, in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or a controller or other computing device, or some combination thereof.
  • the method or apparatus according to an embodiment of the present disclosure may also be implemented with the aid of the architecture of the computing device 3000 shown in FIG9 .
  • the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input/output component 3060, a hard disk 3070, and the like.
  • the storage device in the computing device 3000 such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and/or communication of the method provided by the present disclosure, as well as program instructions executed by the CPU.
  • the computing device 3000 may further include a user interface 3080.
  • the architecture shown in FIG8 is merely exemplary. When implementing different devices, one or more components in the computing device shown in FIG8 may be omitted according to actual needs.
  • FIG9 shows a schematic diagram of a storage medium 4000 according to the present disclosure.
  • the computer storage medium 4020 stores computer-readable instructions 4010.
  • the computer-readable storage medium in the embodiments of the present disclosure can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memory.
  • the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory.
  • the volatile memory can be a random access memory (RAM), which is used as an external cache.
  • RAM dynamic random access memory
  • DRAM dynamic random access memory
  • SDRAM synchronous dynamic random access memory
  • DDR SDRAM double data rate synchronous dynamic random access memory
  • ESDRAM enhanced synchronous dynamic random access memory
  • SLDRAM synchronous linked dynamic random access memory
  • DR RAM direct memory bus random access memory
  • an embodiment of the present application further provides a storage medium, which is used to store a computer program, and the computer program is used to execute the method provided by the above embodiment.
  • An embodiment of the present application further provides a computer program product including a computer program, which, when executed on a computer, enables the computer to execute the method provided in the above embodiment.
  • each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical functions.
  • the functions marked in the boxes can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved.
  • each box in the block diagram and/or flowchart, and the combination of boxes in the block diagram and/or flowchart can be implemented using a dedicated hardware-based system that performs the specified functions or operations, or can be implemented using a combination of dedicated hardware and computer instructions.
  • various example embodiments of the present disclosure may be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device.
  • firmware or software may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device.
  • blocks, devices, systems, techniques, or methods described herein may be implemented, as non-limiting examples, in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or a controller or other computing device, or some combination thereof.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Databases & Information Systems (AREA)
  • Acoustics & Sound (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • Quality & Reliability (AREA)
  • Computational Linguistics (AREA)
  • Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Processing Or Creating Images (AREA)

Abstract

本公开涉及一种处理音频数据的方法、训练神经网络的方法、处理音频数据的装置、电子设备及计算机可读存储介质。所述方法包括：基于参考视频数据，确定虚拟对象的面部先验信息；基于目标音频数据，确定目标音频特征；基于所述面部先验信息和所述目标音频特征，确定目标表情序列信息；以及基于所述目标表情序列信息和所述参考视频数据中的参考帧，确定与所述目标音频数据对应的目标视频数据。本公开能够显著提升音频内容与虚拟对象的面部组件(例如嘴部)动作之间的同步精度，增强嘴部动作的真实质感，更准确地捕捉和再现个体之间表情差异，并且，在提升系统性能表现的同时，也提高了整体系统的运行效率和稳定性。

Description

处理音频数据的方法、训练神经网络的方法和相关装置
本申请要求于2024年02月29日提交中国专利局、申请号为2024102321570、申请名称为“处理音频数据的方法和装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本公开涉及人工智能服务领域，更具体地涉及处理音频数据。
背景技术
近年来，基于音频数据生成视频数据的技术迅速发展，特别是由音频驱动的虚拟人脸生成(audio-driven face generation)技术，有着广泛的应用前景。这项技术能够依据输入的音频信号解析出情绪、语调等信息，并据此实时控制虚拟人物的表情与动作，实现高度同步的表现。在娱乐产业、游戏开发以及人机交互界面、虚拟现实会议等方面，该技术已经展现出了广泛的应用价值，并随着深度学习和神经网络技术的持续革新，其表现力和逼真度正不断提升。
尽管如此，当前的音频驱动虚拟人脸生成技术仍面临一系列技术挑战。首要挑战是音频数据与嘴部运动之间难以做到精确地同步，从而导致所生成的面部动画往往难以与语音内容完美契合。其次，在处理复杂的面部微观细节和纹理方面，当前的技术尚无法达到理想的真实视觉画面效果，使得虚拟人脸的真实性低。此外，如何确保不同虚拟对象间呈现与其独特的个性一致的表情特征，从而真实反映个体差异性，也是当前亟待解决的问题。最后，提升系统运行效率及整体稳定性，以适应更多实时应用场景的需求，同样是该技术未来改进的重要方向。
因此，音频驱动的虚拟人脸生成技术仍需要进一步优化和改进。
发明内容
本公开实施例提供了一种处理音频数据的方法、训练神经网络的方法、处理音频数据的装置、电子设备及计算机可读存储介质。
本公开实施例提供了一种处理音频数据的方法，所述方法包括：基于参考视频数据，确定虚拟对象的面部先验信息；基于目标音频数据，确定目标音频特征；基于所述面部先验信息和所述目标音频特征，确定目标表情序列信息；以及基于所述目标表情序列信息和所述参考视频数据中的参考帧，确定与所述目标音频数据对应的目标视频数据。
本公开实施例提供了一种训练神经网络的方法，其中，所述神经网络包括音频至表情转换器和表情至脸部渲染器，所述方法包括：基于样本视频数据，确定虚拟对象的样本面部先验信息；基于所述样本视频数据对应的样本音频数据，确定样本目标音频特征；基于样本面部先验信息和样本目标音频特征，利用所述音频至表情转换器确定目标样本表情序列信息的预测值；基于所述目标样本表情序列信息的预测值，利用第一损失函数，对所述音频至表情转换器进行训练；基于所述样本视频数据，确定样本参考帧和样本源视频帧；以及基于所述样本面部先验信息、所述样本参考帧、和所述样本源视频帧，利用第二损失函数，对所述表情至脸部渲染器进行训练。
本公开实施例提供了一种处理音频数据的装置，所述装置包括：第一预处理模块，被配置为：基于参考视频数据，确定虚拟对象的面部先验信息：第二预处理模块，被配置为：基于目标音频数据，确定目标音频特征；以及音频至表情转换器模块，被配置为：基于所述面部先验信息和所述目标音频特征，确定目标表情序列信息；表情至面部渲染器，被配置为：基于所述目标表情序列信息和所述参考视频数据中的参考帧，确定与所述目标音频数据对应的目标视频数据。
再一方面，本申请实施例提供一种计算机设备，包括：
处理器、通信接口、存储器和通信总线；
其中，所述处理器、所述通信接口和所述存储器通过所述通信总线完成相互间的通信；所述通信接口为通信模块的接口；
所述存储器，用于存储计算机程序，并将所述计算机程序传输给所述处理器；处理器，用于调用存储器中计算机程序执行以上方面的方法。
又一方面，本申请实施例提供一种存储介质，所述存储介质用于存储计算机程序，所述计算机程序用于执行以上方面的方法。
又一方面，本申请实施例提供了一种包括计算机程序的计算机程序产品，当其在计算机上运行时，使得所述计算机执行以上方面的方法。
本公开实施例通过确定参考视频数据中虚拟对象的面部先验信息，可以准确的标识虚拟对象的面部特点，从而可以基于面部先验信息和目标音频特征确定出符合虚拟对象面部特点且自然融合该面部特点的目标表情序列信息。当采用目标表情序列信息和参考帧确定目标视频数据时，能够显著提升目标视频数据中音频内容与屏幕上虚拟对象的面部组件(例如嘴部)动作之间的同步精度，增强嘴部动作的真实质感，还能够精确反映虚拟对象的身份特性，更准确地捕捉和再现个体之间表情差异，并且，在提升系统性能表现的同时，也确保了整体运行效率和稳定性，特别适用于实时视频生成。
附图说明
图1是示出根据本公开的实施例的场景的示例示意图；
图2示出了根据本公开实施例的示意图；
图3示出了根据本公开的实施例的处理音频数据的方法的流程图；
图4A示出了根据本公开的实施例的改进的神经网络模型的示意图；
图4B示出了根据本公开的实施例的音频至表情转换阶段的示意图；
图4C示出了根据本公开的实施例的音频至表情转换器的示意图；
图4D示出了根据本公开的实施例的表情至面部渲染器的示意图；
图5A示出了根据本公开的实施例的训练神经网络的方法的流程图；
图5B示出了根据本公开的实施例的训练音频至表情转换器的示意图；
图6示出了根据本公开的实施例的试验结果对比图；
图7示出了根据本公开的实施例的电子设备的示意图；
图8示出了根据本公开的实施例的示例性计算设备的架构的示意图；
图9示出了根据本公开的实施例的存储介质的示意图。
具体实施方式
为了使得本公开的目的、技术方案和优点更为明显，下面将参照附图详细描述根据本公开的示例实施例。显然，所描述的实施例仅仅是本公开的一部分实施例，而不是本公开的全部实施例，应理解，本公开不受这里描述的示例实施例的限制。
在本说明书和附图中，具有基本上相同或相似操作和元素用相同或相似的附图标记来表示，且对这些操作和元素的重复描述将被省略。同时，在本公开的描述中，术语“第一”、“第二”等仅用于区分描述，而不能理解为指示或暗示相对重要性或排序。
可用于本公开实施例的模型都可以是人工智能模型，尤其是基于人工智能的神经网络模型。通常，基于人工智能的神经网络模型被实现为无环图，其中神经元布置在不同的层中。通常，神经网络模型包括输入层和输出层，输入层和输出层通过至少一个隐藏层分开。隐藏层将由输入层接收到的输入变换为对在输出层中生成输出有用的表示。节点经由边全连接至相邻层中的节点，并且每个层内的节点之间不存在边。在神经网络的输入层的节点处接收的数据经由隐藏层、激活层、池化层、卷积层等中的任意一项被传播至输出层的节点。神经网络模型的输入输出可以采用各种形式，本公开对此不作限制。
首先，参照图1描述根据本公开的实施例的处理音频数据的方法以及相应的装置等的应用场景。图1示出了根据本公开的实施例的应用场景100的示意图，其中示意性地示出了服务器110和多个终端120。
本公开实施例的神经网络模型具体可以集成在各种电子设备中，例如，图1中的服务器110和多个终端120中的任意电子设备。比如，神经网络模型可以集成在终端120中。终端120可以是手机、平板电脑、笔记本电脑、台式计算机、个人计算机(PC，Personal Computer)、智能音箱或智能手表等，但并不局限于此。又比如，神经网络模型还可以集成在服务器110中。服务器110可以是独立的物理服务器，也可以是多个物理服务器构成的服务器集群或者分布式系统，还可以是提供云服务、云数据库、云计算、云函数、云存储、网络服务、云通信、中间件服务、域名服务、安全服务、内容分发网络(CDN，Content Delivery Network)、以及大数据和人工智能平台等基础云计算服务的云服务器。终端以及服务器可以通过有线或无线通信方式进行直接或间接地连接，本公开在此不做限制。
可以理解的是，应用本公开实施例的神经网络模型来进行推理的计算机设备既可以是终端设备，也可以是服务器，还可以是由终端设备和服务器组成的系统。本公开实施例的处理音频数据的方法可以是在终端上执行的，也可以是在服务器上执行的，还可以是由终端设备和服务器共同执行的。
值得注意的是，根据本公开的实施例的终端110和服务器120均遵守数据保护原则，尊重用户的数据权利，保障用户的数据安全和隐私。根据本公开的实施例的终端110和服务器120将明确告知用户，收集、使用、存储、传输和删除用户的数据的目的、方式和范围，并征得用户的同意。根据本公开的实施例的终端110和服务器120将采取合理的技术和管理措施，防止用户的数据被泄露、篡改、损坏或丢失。根据本公开的实施例的终端110和服务器120提供商将定期对用户的数据进行审查和更新，及时删除过期或无用的数据。此外，采用本公开实施例的云服务提供商尊重用户的数据访问、更正、删除、撤回同意、投诉和索赔等权利，并提供便捷的渠道和程序，让用户能够有效地行使这些权利。
更进一步地，在终端120或服务器110中利用人工智能技术进行数据分析的过程是基于合法、合理和透明的原则进行的。根据本公开的实施例的人工智能模型收集和处理的数据是与预测目的相关、必要和恰当的，不包含任何个人身份信息或敏感信息。根据本公开的实施例的神经网络模型采用了合适的技术和组织措施，保护数据的安全和完整，防止数据被未经授权的访问、使用或泄露。
根据本公开的实施例的基于人工智能的神经网络模型将遵守相关的数据保护法规和伦理原则。该神经网络模型是基于大量的匿名化和去标识化的数据训练出来的，不侵犯任何个人或团体的隐私权。人工智能模型也经过了严格的测试和评估，以确保其输出结果是准确和可靠的，不会造成任何误导或歧视。人工智能模型仅是为了提高服务质量和客户满意度而设计的，不会用于任何非法或不道德的目的。此外，神经网络模型将被定期审查和更新，以适应数据环境和法律规范的变化。
传统上，由音频驱动虚拟人脸生成方法主要可以被分为两类：个体特定(Person-Specific)的方法和通用个体适用的(Person-Generic)方法。个体特定方法中使用的神经网络模型在预测前需要对特定个体进行训练或微调，而通用个体适用的方法则能够直接生成未见过的虚拟人物的说话面部视频。
为了解决以上两种的方法中的音画同步问题，在一些方案中，引入了嘴部同步判别器或唇读网络来增强嘴部区域的可理解性。然而，这样的方案仍不能解决音画同步的问题。为了提高视频画面的质量，一些方案采用了扩散模型或对抗生成模型来尝试提高画面的保真度。然而，这样的方案在效率、视觉质量和嘴部动作精度之间难以取得平衡。
此外，在构建音频数据与面部表情之间的对应关系时，一些方案采用融合建模方法，旨在更高效、精确地捕捉嘴部动作特征。具体而言，它们能够依据音频数据来预测目标面部的区域标签或在无监督环境下抽取关键点信息。尽管面部区域标签和关键点相对便捷获取，但由于这些信息的内在稀疏性特点，这些方案无法充分表达诸如嘴唇噘起、紧闭等复杂面部动作。
本公开实施例对神经网络模型进行架构上的改进。相较于传统的适用于音频驱动虚拟人脸生成的神经网络模型的单一模块结构，改进后的神经网络模型由两个可解耦的模块构成：音频至表情转换器(audio-to-expression transformer)和表情至面部渲染器(expression-to-face renderer)。
由此，改进后的神经网络模型可以对输入数据进行三个阶段的分别处理以实现更高的性能。这三个阶段分别为：在预处理阶段、音频到表情转换阶段和表情到视频帧重建阶段。
具体地，首先，在预处理阶段，可以通过预处理模块对参考视频数据和目标音频数据进行必要的特征提取以获得虚拟对象的面部先验信息和目标音频特征。接着，在音频到表情转换阶段，音频至表情转换器将接收到的面部先验信息和目标音频特征映射成对应的目标表情序列信息，特别是针对嘴部动作的细微表情变化。最后，在表情到视频帧重建阶段，表情至面部渲染器基于所述目标表情序列信息、所述参考视频数据中的参考帧、所述参考帧对应的参考头部位姿特征向量，精确地生成与所述目标音频数据对应的目标视频数据，该目标视频数据与目标音频数据精准同步且具有高度真实感的嘴部运动及面部表情细节。
相较于传统的神经网络模型的单一阶段的方案，即直接输入音频数据和视频数据以生成更新后的视频画面，根据本公开实施例的改进后的多阶段神经网络模型能够显著提升音频内容与屏幕上虚拟对象的面部组件(例如嘴部)动作之间的同步精度，增强嘴部动作的真实质感，更准确地捕捉和再现个体之间表情差异，并且，在提升系统性能表现的同时，也确保了整体运行效率和稳定性。
针对改进后的神经网络模型，本公开实施例提供了一种处理音频数据的方法，以利用改进后的神经网络模型进行推理。所述方法包括：基于参考视频数据，确定虚拟对象的面部先验信息；基于目标音频数据，确定目标音频特征；基于所述面部先验信息和所述目标音频特征，确定目标表情序列信息；以及基于所述目标表情序列信息、所述参考视频数据中的参考帧、所述参考帧对应的参考头部位姿特征向量，确定与所述目标音频数据对应的目标视频数据。
在利用改进的神经网络进行推理的过程中，通过先确定目标表情序列，然后基于此预测视频帧的步骤，显著提升了音频内容与屏幕上虚拟对象面部组件的动作同步性。这种方法不仅增强了面部动作的真实感，还能够准确地反映虚拟对象的身份特性。在提高性能的同时，也确保了神经网络的运行效率和稳定性，特别适用于实时视频生成。
类似的，针对改进后的神经网络模型，本公开实施例还提供了一种训练神经网络的方法，以对改进后的神经网络模型进行训练。所述神经网络包括音频至表情转换器和表情至脸部渲染器，所述方法包括：基于样本视频数据和所述样本视频数据对应的样本音频数据，对所述音频至表情转换器进行训练；以及基于样本面部先验信息、样本参考帧、样本源视频帧，对所述表情至脸部渲染器进行训练。音频至表情转换器和表情至脸部渲染器可以被分开独立训练，二者的训练过程互不干扰。
在改进的神经网络模型训练过程中，可选择地引入了改进的第一损失函数和第二损失函数。这两个损失函数分别针对不同方面的特性进行优化，通过对神经网络参数的调整，提高了训练后神经网络的性能表现。第一损失函数和第二损失函数可以使得神经网络能够更有效地学习和适应输入数据的特征，并在训练过程中从多个方面进行优化。这种综合考虑不同特性的损失函数设计，有助于提高神经网络的泛化能力和训练效率，进而改善其在实际应用中的性能表现。
下面参照图2至图9描述根据本公开的实施例的处理音频数据的方法、训练神经网络的方法和装置。
图2示出了根据本公开实施例的示意图。
如图2所示，用于实现音频驱动的虚拟人脸生成技术的具体应用场景为：已有一段视频素材(以下又称为参考视频数据)，该视频素材与一段音频相对应。视频内容展现了某个虚拟对象面对镜头说话的面部表情，尤其是嘴部运动的细节。在这段视频素材中，虚拟对象说出了一句话，例如“今天天气真好”。为了更快地制作出该虚拟对象更多的视频，需要将这段视频中的虚拟对象的口型同步至另一段不同的音频内容(以下又称为目标音频数据)，例如，“我爱你”。即通过技术手段，使原本在说“今天天气真好”的虚拟人物，基于另一段音频数据，生成目标视频数据。在该目标视频数据中，虚拟对象的口型动作能够精确匹配上说出“我爱你”时应有的形态和节奏。
如图2所示，根据本公开实施例的神经网络模型主要由两大部分构成：音频至表情转换器和表情至面部渲染器。
在预处理过程中，本公开实施例从参考视频数据中提取面部先验信息，包括但不限于：参考面部形状信息、参考表情信息、和参考头部位姿信息。同时，在处理过程中，还从目标音频数据中提取了音频特征。可选地，可以使用三维可变形人脸模型(3D Morphable Face Model，3DMM)作为三维形态模型，来从参考视频数据中提取更准确的面部形状信息、姿态信息和表情信息(如后续提及的参考面部形状信息、参考表情信息和参考头部位姿信息等面部先验信息)，以实现从视频到表情序列的准确映射。相较于需要在面部区域标记关键点的传统方法，根据本公开实施例的三维形态模型有助于神经网络更有效、准确地学习面部组件(例如，嘴部)的动作以及变化。
在音频到表情转换阶段，预处理中提取的面部先验信息将输入至音频至表情转换器，以生成目标表情序列信息，以此来更好地保留并反映虚拟人体的身份特性及情绪表达的细微变化。
在表情到视频帧重建阶段，目标表情序列信息将作为表情至脸部渲染器的输入，以使得该渲染器能够利用改进的面部身份对齐模块，精确地生成目标面部组件(例如，目标面部的嘴部、眼部、耳朵、眉毛等组件)的运动视频画面。可选地，表情至面部渲染器可以使用基于面部个体对齐模块(FIA)，其仅需要一个参考视频帧就能得到符合虚拟人物在发音时的面部个性特征的目标视频帧。相比于采用传统方法，根据本公开实施例的表情至面部渲染器提高了目标视频帧的个性化表达，减少了运算成本，有助于提高推理速度。
本公开实施例能够在实时状态下生成精细的面部组件(其用于表征虚拟器官)的细节，无需依赖复杂且效率较低的计算模块，大幅度地提高了视频画面的精细度和生成视频数据的效率。虽然本公开后续主要以嘴部为例进行说明，但本领域技术人员应当理解，本公开的方案可以适用于任何与表情和情绪传达有关的面部器官的生成。当然本公开并不以此为限。
图3示出了根据本公开的实施例的处理音频数据的方法30的流程图。
方法30可以在终端设备或服务器(如图1所述的终端120或服务器110)处执行。其中，方法30包括以下的操作S301至操作S304。当然方法30还可以包括更多或更少的操作，本公开并不以此为限。
可选地，操作S301和S302为预处理阶段中的操作，其可以由任意的面部信息提取模型执行。在音频信息到表情信息的转换阶段，可以执行操作S303，而在表情到视频帧重建阶段可以执行操作S304。
在操作S301中，基于参考视频数据，确定虚拟对象的面部先验信息。
可选地，参考视频数据所涉及视频内容中包括虚拟对象，该虚拟对象为具有面部且可发音的对象，例如可以是真实的人、动物，也可以是卡通形象、虚拟三维模型等。参考视频数据可以是通过视频采集设备采集真实场景得到的，也可以是通过软件生成的。
参考视频数据可以是如图2所示的输入视频数据，在该输入视频数据中，虚拟对象说出“今天天气真好”。视频数据包括多个参考视频帧，每一个参考视频帧都是一张图像。图像中呈现了虚拟对象的头部。
虚拟对象的面部先验信息是指用于标识虚拟对象面部的面部准确信息，用于体现该虚拟对象的面部特征。可选地，虚拟对象的面部先验信息包括以下各项中的一项或多项：参考面部形状信息、参考表情信息、和参考头部位姿信息。当然，本公开并不限于此。
例如，参考面部形状信息可以是用于表征虚拟对象的面部形状特征的信息。可选地。参考面部形状信息可以从任意一帧参考视频帧中提取得到，其可以包含虚拟对象面部的纹理和3D形状信息。可选地，基于参考面部形状信息，可以通过对通用3D标准虚拟对象直接进行仿射变换，得到虚拟对象无表情的3D面部头像。3D标准虚拟对象又称为白模虚拟对象，是一种用于模拟人类面部表情和动作的标准化模型。这些模型通常具有统一的三维形状和结构，用于代表一个虚拟对象的基本外观。当然，本公开并不限于此。
例如，参考表情信息可以是用于表征虚拟对象的表情特性的任意信息，其用于体现虚拟对象在不同语言表达情绪下的表情特性。参考表情信息可以从每个参考视频帧中提取，从而得到每个参考视频帧i对应的参考表情特征向量βi。而参考表情信息可以指代所有参考视频帧对应的参考表情特征向量的集合。可选地，基于参考表情特征向量和参考面部形状信息，可通过对通用3D标准虚拟对象直接进行仿射变换，得到在某一帧中虚拟对象在特定表情下的3D面部头像。当然，本公开并不限于此。
例如，参考头部位姿信息用于表征虚拟对象的头部的位姿特性的任意信息。参考头部位姿信息从每个参考视频帧中提取，从而得到每个参考视频帧i对应的参考头部位姿特征向量，而参考头部位姿信息可以指代所有参考视频帧对应的参考头部位姿特征向量的集合。可选地，基于参考头部位姿特征向量，通过旋转，可以直接调整通用3D标准虚拟对象达到参考头部位姿特征向量对应的头部位姿。当然，本公开并不限于此。
在操作S302中，基于目标音频数据，确定目标音频特征。
目标音频数据所标识的音频，为需要进行音频-虚拟对象合成的音频，即需要在后续S304所确定的目标视频数据中，通过虚拟对象发出的音频。可选地，目标音频数据可以是如图2所示的目标音频数据，在该在该输入视频数据对应的音频为“我爱你”。目标音频数据可以包括多个音频帧，而目标音频特征则是指从这些音频帧中提取出来的与声音相关的特征。这些特征可能包括音频信号的频谱信息、时域信息、声音强度等。当然，本公开并不限于此。
在操作S303中，基于所述面部先验信息和所述目标音频特征，确定目标表情序列信息。
可选地，目标表情序列信息可以是与所述目标音频特征相对应的参考表情信息。目标表情序列信息为按时序排列的目标表情特征向量的集合，每个目标表情特征向量与目标音频帧相对应，用于反映虚拟对象在说出特定词句时的表情特性。这种映射关系可以是多对一、一对一、或一对多的形式。例如，一个目标表情特征向量可以对应于一帧目标音频帧，也可以对应于多帧目标音频帧，甚至可能多个目标表情特征向量对应于单个目标音频帧。当然，本公开并不限于此。
可选地，可以仅基于面部先验信息中的参考面部形状信息和参考表情信息，结合目标音频特征直接确定目标表情序列信息。当然，本公开并不以此为限。
在操作S304中，基于所述目标表情序列信息和所述参考视频数据中的参考帧，确定与所述目标音频数据对应的目标视频数据。
目标视频数据中涉及的虚拟对象和S301的参考视频数据中涉及的虚拟对象为同一个虚拟对象，只不过，目标视频数据中的虚拟对象会发出目标音频数据对应的音频。
由于目标表情序列信息能够准确的标识出虚拟对象在发出目标音频特征所对应音频的过程中，虚拟对象的表情特征，而参考帧能标识出虚拟对象的相关信息，故通过目标表情序列信息，可以基于参考视频数据中的参考帧的指导，可以准确模拟出虚拟对象发出目标音频数据对应音频的面部情况，从而得到目标视频数据。
可选地，目标视频数据的目标视频帧数量可以与目标音频数据中的音频帧数量相匹配。每个目标视频帧都展示了虚拟对象在说出对应音频帧时的特定面部表情。可选地，虚拟对象在说出对应音频帧时的嘴部运动(即嘴型)应该与该音频帧的发音相对应。目标视频帧呈现的虚拟对象的面部表情与其发音之间的一致，从而使得生成的视频更加真实和逼真。
可选地，参考视频数据中的参考帧Ir可以是参考视频数据中的任意一帧视频帧，其用于在表情到视频帧重建阶段提供虚拟对象的基本形状和纹理作为参考信息即可。具体到目标视频数据中的每个目标视频帧的生成过程，既可以使用相同的参考帧Ir也可以使用不同的参考帧，本公开并不以此为限。
可选地，在S304中，可以从所述参考视频数据中提取源视频帧；基于所述目标表情序列对所述源视频帧进行掩码，以得到第一视频帧，其中，在第一视频帧中，所述虚拟对象的部分面部组件被黑色像素块代替；以及基于第一视频帧和所述参考帧，利用所述面部先验信息，确定与所述目标音频数据对应的目标视频数据。
本申请不限定参考帧和源视频帧的关系，例如源视频帧可以是某个或某些参考帧，本申请也不限定在确定目标说数据时，所使用源视频帧的数量，例如可以是一个源视频帧，也可以是多个源视频帧。
可选地，所述基于所述目标表情序列对所述源视频帧进行掩码，以得到第一视频帧包括：基于所述目标表情序列和所述参考面部形状信息，确定在所述源视频帧中所述虚拟对象的部分面部组件对应的像素区域；以及将所述像素区域中的像素值修改为预设值。
可选地，所述基于第一视频帧和所述参考帧，利用所述面部先验信息，确定与所述目标音频数据对应的目标视频数据包括：利用共享权重的编码器来对所述第一视频帧和所述参考帧进行编码，以获取所述第一视频帧的编码特征和所述参考帧的编码特征；利用所述面部先验信息和所述目标表情序列，利用解码器对所述第一视频帧的编码特征和所述参考帧的编码特征进行解码，以获得与所述目标音频数据对应的目标视频数据，其中，所述目标视频数据包括目标视频帧，所述目标视频帧包括被黑色像素块代替的所述虚拟对象的部分面部组件的预测值。
例如，可以基于所述目标表情序列信息，生成第一视频帧(其中，第一视频帧中的面部组件被掩码，即该视频帧中的面部组件被黑色像素块代替)，接着基于所述源视频帧Is和参考帧Ir，确定目标视频帧以下以嘴部组件的掩码(又称为嘴部掩码)为例进行说明，本领域技术人员可以理解该方案还可以适用于任意的面部组件。
具体而言，对于每一个视频帧，通过嘴部掩码将与嘴部运动有关的面部结构进行遮蔽(例如下半张脸)，保留与表情变换或目标音频帧无关的面部可变结构。接着，利用参考视频数据中的参考帧Ir和包括可学习的嘴部掩码的视频帧，利用面部身份对齐模块预测嘴部掩码处的像素值并将所预测的嘴部掩码处的像素值填充至目标视频帧处，以得到目标视频帧。本公开并不以此为限。
可选地，在操作S304中还进行了面部身份的对齐操作，其利用交叉注意力结构的具有面部身份对齐模块，在目标表情相关参数(例如目标表情特征向量)和单个参考帧的有效面部纹理之间建立有效的联系。通过这种结构，模块能够实现更加精确的唇形控制。具体而言，交叉注意力结构允许模块同时关注目标表情和从单个参考帧中提取的面部纹理信息。这种注意力机制使得模块能够根据目标表情特征向量和参考帧的面部纹理，精确地调整唇部的形状和运动。通过将这两种信息结合起来，模块能够更好地理解面部的特征和结构，并对唇部运动进行精准控制。
根据本公开实施例的方法可以提高虚拟对象的口型动作的精确性和可信度，增强了视频的沟通效果和视觉表现力本公开实施例能够在实时状态下生成精细的面部器官的细节，无需依赖复杂且效率较低的计算模块，大幅度地提高了视频画面的精细度和生成视频数据的效率。
图4A示出了根据本公开的实施例的改进的神经网络模型的示意图。图4B示出了根据本公开的实施例的音频至表情转换阶段的示意图。图4C示出了根据本公开的实施例的音频至表情转换器的示意图。图4D示出了根据本公开的实施例的表情至面部渲染器的示意图。
如图4A所示，在预处理阶段，可以使用深度3D面部重建模型执行操作S301，以基于参考视频数据，确定虚拟对象的面部先验信息。面部先验信息包括参考面部形状信息(α1)、参考表情信息(β1,…,βN)、和参考头部位姿信息(ρ1,…,ρN)。给定一系列包括虚拟对象面部的视频帧，深度3D面部重建模型能够提取3D系数，其中包括形状(shape)系数、表情(expression)系数和姿势(pose)系数。可选地，深度3D面部重建模型可以利用混合损失函数进行弱监督学习，该混合损失函数考虑了低层次和感知水平的信息以进行监督。同时，深度3D面部重建模型还可以利用来自不同图像的互补信息进行多图像面部重建，实现了形状聚合。深度3D面部重建模型具有快速、准确和鲁棒的特点，能够应对遮挡和大姿态的情况。虽然如此，本公开的各项实施例还可以使用其他的神经网络模型来提取虚拟对象的面部先验信息。
可选地，在操作S302中，可以利用隐藏单元(hidden-unit)的Bert模型(以下又称为HuBert模型)进行自监督语音表示学习，通过对隐藏单元的掩码预测来基于目标音频数据确定目标音频特征(w1,…,wl)。Hubert模型采用了一种自监督学习方法，通过掩码预测隐藏单元，从而学习到了高质量的语音表示。虽然如此，本公开的各项实施例还可以使用其他的神经网络模型来确定目标音频特征。
在音频信息到表情信息的转换阶段，在操作S303中，音频至表情转换器可以利用3D系数中的一个参考面部形状信息(α1)和N个参考表情信息(β1,…,βN)，结合l个目标音频特征(w1,…,wl)得到目标表情序列信息，其包括T个目标表情特征N，l和T可以为相同值也可以为不同值，本公开并不以此为限。
举例来说每个虚拟对象的个体面部和嘴部结构都具有其独特性，同时每个虚拟对象还具有不同的说话习惯。在这种情况下，包括参考面部形状信息(α1)和N个参考表情信息(β1,…,βN)的面部先验信息能够体现虚拟对象的个性特征或身份特征。具体来说，参考面部形状信息(α1)通过面部的纹理和形状特性表征了虚拟对象的身份。可选地，该参考面部形状信息可以与自然面部尺寸和口腔比例相关联。另一方面，参考表情信息捕获了个体独特的说话风格，这为个性化的虚拟对象提供了与标准形状的差异。参考表情信息包括嘴部运动的幅度和情绪表达等因素。通过综合利用这些信息，可以更加准确地生成与目标音频相匹配的面部表情序列，从而使得虚拟对象的表情更加生动自然。
如图4A所示，音频至表情转换器是基于转换器(transformer)架构的，其包括一个编码器和一个解码器。在这样的架构下，当面部先验信息包括参考面部形状信息和参考表情信息时，确定目标表情序列信息的步骤可选地包括：对所述参考面部形状信息(α1)、所述参考表情信息(β1,…,βN)和所述目标音频特征(w1,…,wl,)进行编码，以得到隐藏向量(Z)；以及对所述隐藏向量(Z)进行解码，以确定所述目标表情序列信息。图4B进一步示出了音频至表情转换器的一些细节。
通过将参考面部形状信息和参考表情信息结合目标音频特征进行编码，可以将与虚拟对象相关的面部形状和表情等相关信息和目标音频特征中涉及的不同音频进行准确关联，以便得到的目标表情序列信息能够更为自然的标识出虚拟对象在发出目标音频特征时的面部表情。
具体地，3D系数中的参考面部形状信息α1(其将作为默认参考面部形状信息)、N个参考表情信息(β1,…,βN)和l个目标音频特征(w1,…,wl)可以分别通过全连接的网络以获得嵌入向量。然后，这些嵌入向量以时间顺序连接，得到总共1+N+l个词元(token)，这些词元被输入到多头自注意器(Multi-Head Self-Attention，MHSA)(也即转换器编码器)中进行编码，以得到隐藏向量Z。
Z＝MHSA(α1,β1,…,βN,w1,…,wl)                          (1)
表情查询向量qt(最初设置为零)与隐藏向量Z组合并馈送到多头交叉注意力(Multi-Head Cross-Attention，MHCA)网络(也即转换器解码器)中进行解码，以得到T个目标表情特征
在表情信息到视频数据的重建阶段，在操作S304中，表情至脸部渲染器首先从参考视频数据中获取任意一个参考视频帧作为源视频帧Is。举例来说，假设要生成第m帧目标视频帧，那么可以从参考视频数据中提取第m帧参考视频帧作为源视频帧。当然本公开并不限于此。
接着，表情至脸部渲染器可以基于目标表情序列信息中的一个目标表情特征向量和源视频帧Is，生成第一视频帧该第一视频帧中的面部组件被掩码。可选地，被掩码的面部组件对应的区域是可学习的，也即该区域的大小、形状可以与该虚拟对象的表情信息和身份信息均关联。
可选地，与传统方法不同，生成第一视频帧的过程是可学习的。如图4B所示，可以选择性地对源视频帧Is进行掩码，允许更准确地控制修改面部组件区域。首先，根据预测的表情特征向量，估计3D顶点，然后将其投影到图像上。接着，基于这些3D顶点，可以绘制并用黑色像素块(或像素值为任意预设值的像素块)填充嘴部和颈部区域。也就是说，相对于源视频帧来说，第一视频帧中具有掩码区域，属于被进行掩码处理的源视频帧，第一视频帧的掩码区域可以为黑色像素块，在源视频帧的这个区域中，包括了虚拟对象的部分面部组件。
生成掩码的过程如下：
其中，P是投影函数，并且τt指示t的平移矩阵。C是求解感兴趣的面部和颈部区域中的顶点Vxy的轮廓的函数，点Vxy的集合是S。
通过对源视频帧掩码，可以准确的标识出需要预测的面部区域，且该区域与虚拟对象发出目标音频特征直接相关，从而提高预测的针对性和准确性。
第一视频帧和所述参考视频数据中的参考帧Ir将共同输入至一个共享权重的编码器，以处理两个图像以提取多尺度特征。具体地，该共享权重编码器将第一视频帧编码为特征Fs，并将参考帧Ir编码为特征Fr。
可选地，如图4A所示，共享权重的编码器由d个堆叠的残差块组成，因此该编码器也称为d级编码器。在每个级(残差块)，图像分辨率减半，同时所提取的特征维度增加。这种设置允许在共享权重的编码器中有效地提取多尺度特征。由于编码器的权重是共享的，因此可以并行化地对第一视频帧和参考帧Ir进行编码。具体地，编码器通过多个级别的处理，从输入图像中提取出各种尺度的特征。针对d级编码器中的第i级，其多尺寸特征可以被计算如下。
其中，SEi表示编码器的第i级。在每一级处理结束时，都会得到一个包含不同尺度和抽象程度特征的表示。这些特征可以捕捉到输入图像的不同细节和结构信息。由此，针对第一视频帧和参考帧Ir得到d个尺寸的特征向量的集合，也即和 由于第一视频帧和参考帧Ir共享编码器的权重，其能够以更高效的方式处理源视频帧和参考帧，提取出它们共享的特征表示。这种并行计算方式不仅加快了处理速度，还确保了源视频帧和参考帧的特征表示在同一空间上进行比较，有利于后续的匹配和融合。
接着，表情至脸部渲染器将利用包括面部身份对齐模块的解码器来对第一视频帧对应的特征Fs和参考帧Ir对应特征Fr进行解码，以得到目标视频帧。
可见，在利用改进的神经网络进行推理的过程中，通过先确定目标表情序列，然后基于此预测视频帧的步骤，显著提升了音频内容与屏幕上虚拟对象面部组件的动作同步性。这种方法不仅增强了面部动作的真实感，还能够准确地反映虚拟对象的身份特性。在提高性能的同时，也确保了神经网络的运行效率和稳定性，特别适用于实时视频生成。
图4D示出了解码器更多的示例细节。具体地，解码器由d个堆叠的解码层组成，因此该解码器也称为d级解码器。例如，所述解码器包括d个解码层，每个解码层的输入包括前一个解码层的输出、所述面部先验信息和所述参考帧的编码特征，每个解码层的输出为目标视频帧的预测值。假设解码器的每个级的输入特征为输出特征为
如图4D所示，3D系数在通过适应器(AdalN)模块融合至解码器之前，通过三层MLP进行维度映射。这一过程旨在将3D系数映射到与解码器网络兼容的特征空间中，以便后续的特征融合和生成过程。由此，解码器的每个级的输入特征先经过上采样以增加分辨率，以得到特征特征再经过3D系数调制，以匹配特定的面部组件的形状，从而更好地控制面部组件运动和面部表情的生成。调制后的特征与来自参考帧Ir的特征以相同的分辨率对准，二者进行融合以控制面部生成和聚合纹理。以上过程可以被表述如下。
FIAi为包括面部身份对齐模块的解码器中的一级(或一层)解码层。
此外，为了避免目标视频帧中产生不必要的背景，并确保最终的结果更加真实和精确，还可以进一步采用融合策略。具体而言，在本公开的一些实施例中，可以解码器最后一层解码层的输出结果与解码器第一层解码层的输出进行融合，以获取最终的输出。这种融合策略旨在仅保留所预测的面部组件对应的像素，同时消除背景干扰，使得生成的图像更加清晰和专注于面部特征。该过程可以被表述如下。
图5A示出了根据本公开的实施例的训练神经网络的方法50的流程图，其可以对图4A至图4C所示的改进后的神经网络模型进行训练。图5B示出了根据本公开的实施例的训练音频至表情转换器的示意图。
方法50可以在终端设备或服务器(如图1所述的终端120或服务器110)处执行。其中，方法50包括以下的操作S501至操作S506。当然方法50还可以包括更多或更少的操作，本公开并不以此为限。
在操作S501中，基于样本视频数据，确定虚拟对象的样本面部先验信息。在操作S502中，基于所述样本视频数据对应的样本音频数据，确定样本目标音频特征。
例如，假设基于样本视频数据获取的面部先验信息包括：样本面部形状信息(α1)、样本目标表情序列信息(例如，样本表情信息β1,…,βN)、和样本头部位姿信息(ρ1,…,ρN)。这些面部先验信息可以直接经过预处理得到。这样的面部先验消息可以作为虚拟对象的面部先验信息的真实值。已经在前述详细的描述了预处理的过程，本公开在此不再赘述。
接着，参考以上详细描述的预处理过程，还可以对样本视频数据相对应的样本音频数据进行提取以直接确定样本音频数据对应的样本音频特征。这里与样本视频数据相对应的样本音频数据是指：假设在样本视频数据中虚拟人正在说出“今天天气真好”，就是包含了"今天天气真好"这段语音的音频数据。
接下来，基于操作S503和S504描述音频至表情转换器的训练过程。本领域技术人员应当理解本公开并不以此为限。
在操作S503中，基于样本面部先验信息和样本目标音频特征，利用所述音频至表情转换器确定目标样本表情序列信息的预测值。在操作S504中，基于所述目标样本表情序列信息的预测值，利用第一损失函数，对所述音频至表情转换器进行训练。可选地，所述样本面部先验信息可以包括样本目标表情序列信息的真实值，操作S503包括：对所述虚拟对象的面部先验信息的真实值进行掩码以获得掩码后的所述虚拟对象的面部先验信息；基于所述掩码后的所述虚拟对象的面部先验信息和所述样本音频特征，利用所述音频至表情转换器确定所述虚拟对象的样本目标表情序列信息面部先验信息的预测值。当然，本公开并不以此为限。
可选地，为了确定虚拟对象的样本目标表情序列信息的预测值，首先需要对其的真实值进行掩码处理，以获得掩码后的虚拟对象的样本目标表情序列信息。这个过程包括将真实值中的一部分信息进行隐藏或遮盖，从而生成一个带有掩码的版本。掩码后的信息可以是根据特定的条件或者规则进行遮盖，例如，隐藏特定的参考表情信息βt。接着，基于这些掩码后的样本目标表情序列信息(也即不包括特定的参考表情信息βt的面部先验信息)和样本音频特征，可以利用训练中的音频至表情转换器来预测虚拟对象的样本目标表情序列信息掩码部分的预测值(例如，)。对音频至表情转换器的训练目标是使得面部先验信息的真实值βt与预测值足够接近。因此，可以不断地调整音频至表情转换器中的各个神经元的参数以使得第一损失函数达到极值。当然本公开并不以此为限。
可选地，如图5B所示，用于训练音频到表情转换器的第一损失函数可以包括但不限于均方误差(MSE)损失和3D顶点损失。
例如，均方误差(MSE)损失用于计算表情特征向量的预测值和真实值之间的误差。假设预测的表情特征向量为为，而真实的表情特征向量为βt，则MSE损失可以表示为：
可选地，3D顶点损失通过组合预测的表情特征向量面部形状信息α1、和参考头部位姿信息ρ来计算3D顶点的预测位置。这些顶点是从面部区域(例如嘴部)选择的关键点(例如嘴角)，用于评估该面部区域的关键点的真实值和预测值之间的距离。通过最小化关键点的位置的预测值和真实值之间的差，可选确保所生成的3D顶点与嘴部关键点的准确对准。令V表示从系数计算的3D顶点。3D顶点损失LV可以描述如下：
由此，第一损失函数可以被表述为La2e＝LMSE+0.1*CV。其中，0.1为超参数，其还可以被设置为其他的任意值。本公开并不以此为限。
接下来，基于操作S505和S506描述表情至脸部渲染器的训练过程。本领域技术人员应当理解本公开并不以此为限。音频至表情转换器的训练过程和表情至脸部渲染器的训练过程可以是相互独立的。具体地，如以下详述的，表情至脸部渲染器的训练过程中使用的输入数据可以是所述样本面部先验信息(其可选地包括所述样本目标表情序列信息的真实值)、所述样本参考帧、和所述样本源视频帧。这些输入数据均可以在预处理的过程(也即操作S501和S502)中获得，而无需经过音频至表情转换器处理。当然，在另一些实施例中，音频至表情转换器和表情至脸部渲染器也可以是联合训练的，本公开并不以此为限。
具体地，所述样本面部先验信息包括样本目标表情序列信息的真实值。在操作S505中，基于所述样本视频数据，确定样本参考帧和样本源视频帧。接着，在操作S506中，基于所述样本面部先验信息、所述样本参考帧、和所述样本源视频帧，利用第二损失函数，对所述表情至脸部渲染器进行训练。
可选地，在操作S506中，基于所述目标表情序列的真实值，对所述样本源视频帧进行掩码，以获得经掩码的样本源视频帧；基于所述样本面部先验信息和、所述样本参考帧和所述样本目标表情序列信息的真实值，针对所述经掩码的样本源视频帧，利用所述表情至脸部渲染器确定所述样本源视频帧的预测值；以及基于所述样本源视频帧的预测值和真实值，利用所述第二损失函数，对所述表情至脸部渲染器进行训练；其中，第二损失函数包括以下各项中的一项或多项的组合：像素损失、感知损失、对抗性损失和局部像素损失。
可选地，所述样本面部先验信息包括3D系数可以参照以上详述的方法，可以首先对样本源视频帧进行掩码(也即得到掩码后的样本源视频帧，其面部组件被黑色像素块代替)。然后再基于这些3D系数、所述样本参考帧来确定所述样本源视频帧的预测值。对表情至脸部渲染器的训练目标是使得样本源视频帧的真实值与预测值足够接近。因此，可以不断地调整表情至脸部渲染器中的各个神经元的参数以使得第二损失函数达到极值。当然本公开并不以此为限。
可选地，用于训练表情到视频渲染器的第二损失函数可以采用多个损失函数的组合来约束面部组件与表情之间的同步性，以提高所生成的视频帧的视觉质量。
例如，像素损失可以用于衡量生成的图像与目标图像之间的像素级别的差异。通过最小化样本源视频帧的预测值和真实值Is之间的像素差异，促进所生成的视频帧的准确性和清晰度。作为一个示例。像素损失L1可以被计算为：当然本公开并不以此为限。
例如，感知损失基于深度学习模型(如VGG网络)提取的特征(也即从图像中获取的感知信息)，衡量生成样本源视频帧的预测值和真实值Is之间的感知差异。感知损失帮助提高生成图像的真实感和自然度，以更好地匹配目标表情。感知损失可以被计算为：
例如，对抗性损失通过生成对抗网络(GAN)框架中的鉴别器网络来评估样本源视频帧的预测值的真实性。生成器网络旨在欺骗鉴别器，使其无法区分样本源视频帧的预测值和真实值Is，从而提高图像的真实性和逼真度。对抗性损失可以被计算为： 其中D(·)表示鉴别器。
例如，局部像素损失专注于增强特定区域的像素级别细节，例如嘴部的细节。通过引入针对特定面部区域的额外损失项，使生成的图像在嘴部等细节上更加清晰和逼真。局部像素损失可以被计算为：其中，M'表示目标面部区域的二进制掩码。
通过根据对以上各项损失函数进行加权求和以得到第二损失函数。以上各项损失函数共同作用于表情到视频渲染器的训练过程中，以确保生成的视频能够准确地反映目标表情，同时保持视觉质量和细节的高水平。第二损失函数可以被计算为Le2f＝λ1*L1+λ2*L2+λ3*L3+λ4*L4，其中，λ1＝1,λ2＝1,λ3＝0.1,λ4＝1。λ1至λ4均为超参数，其可以在实际应用中设置为其他任意值，本公开并不以此为限。
在改进的神经网络模型训练过程中，可选择地引入了改进的第一损失函数和第二损失函数。这两个损失函数分别针对不同方面的特性进行优化，通过对神经网络参数的调整，提高了训练后神经网络的性能表现。第一损失函数和第二损失函数可以使得神经网络能够更有效地学习和适应输入数据的特征，并在训练过程中从多个方面进行优化。这种综合考虑不同特性的损失函数设计，有助于提高神经网络的泛化能力和训练效率，进而改善其在实际应用中的性能表现。
本公开实施例还提供了处理音频数据的装置，所述装置包括：第一预处理模块，用于基于参考视频数据，确定虚拟对象的面部先验信息：第二预处理模块，用于基于目标音频数据，确定目标音频特征；以及音频至表情转换器模块，用于基于所述面部先验信息和所述目标音频特征，确定目标表情序列信息；表情至面部渲染器，用于基于所述目标表情序列信息和所述参考视频数据中的参考帧，确定与所述目标音频数据对应的目标视频数据。
在一种可能的实现方式中，所述面部先验信息包括以下各项中的一项或多项：参考面部形状信息、参考表情信息、和参考头部位姿信息。
在一种可能的实现方式中，当所述面部先验信息包括所述参考面部形状信息和所述参考表情信息时，所述音频至表情转换器模块还用于：
对所述参考面部形状信息、所述参考表情信息和所述目标音频特征进行编码，以得到隐藏向量；以及
对所述隐藏向量进行解码，以确定所述目标表情序列信息。
在一种可能的实现方式中，所述表情至面部渲染器还用于：
从所述参考视频数据中提取源视频帧；
基于所述目标表情序列对所述源视频帧进行掩码，以得到第一视频帧，其中，在所述第一视频帧中，所述虚拟对象的部分面部组件被黑色像素块代替；以及
基于所述第一视频帧和所述参考帧，利用所述面部先验信息，确定与所述目标音频数据对应的目标视频数据。
在一种可能的实现方式中，所述表情至面部渲染器还用于：
基于所述目标表情序列和所述参考面部形状信息，确定在所述源视频帧中所述虚拟对象的部分面部组件对应的像素区域；以及
将所述像素区域中的像素值修改为预设值，得到所述第一视频帧。
在一种可能的实现方式中，所述表情至面部渲染器还用于：
利用共享权重的编码器来对所述第一视频帧和所述参考帧进行编码，以获取所述第一视频帧的第一编码特征和所述参考帧的第二编码特征；
利用所述面部先验信息和所述目标表情序列，利用解码器对所所述第一编码特征和所述第二编码特征进行解码，以获得与所述目标音频数据对应的目标视频数据，其中，所述目标视频数据包括目标视频帧，所述目标视频帧中黑色像素块的区域被代替为所述虚拟对象的部分面部组件的预测值。
在一种可能的实现方式中，所述解码器包括多个级联的解码层，每个解码层的输入包括前一个解码层的输出、所述面部先验信息和所述参考帧的编码特征，每个解码层的输出为目标视频帧的预测值。
可见，通过确定参考视频数据中虚拟对象的面部先验信息，可以准确的标识虚拟对象的面部特点，从而可以基于面部先验信息和目标音频特征确定出符合虚拟对象面部特点且自然融合该面部特点的目标表情序列信息。当采用目标表情序列信息和参考帧确定目标视频数据时，能够显著提升目标视频数据中音频内容与屏幕上虚拟对象的面部组件(例如嘴部)动作之间的同步精度，增强嘴部动作的真实质感，还能够精确反映虚拟对象的身份特性，更准确地捕捉和再现个体之间表情差异，并且，在提升系统性能表现的同时，也确保了整体运行效率和稳定性，特别适用于实时视频生成。
本公开实施例还提供了一种训练神经网络的装置，所述神经网络包括音频至表情转换器和表情至脸部渲染器，所述装置包括：
第一确定模块，用于基于样本视频数据，确定虚拟对象的样本面部先验信息；
第二确定模块，用于基于所述样本视频数据对应的样本音频数据，确定样本目标音频特征；
第三确定模块，用于基于样本面部先验信息和样本目标音频特征，利用所述音频至表情转换器确定目标样本表情序列信息的预测值；
第一训练模块，用于基于所述目标样本表情序列信息的预测值，利用第一损失函数，对所述音频至表情转换器进行训练；
第四确定模块，用于基于所述样本视频数据，确定样本参考帧和样本源视频帧；以及
第二训练模块，用于基于所述样本面部先验信息、所述样本参考帧、和所述样本源视频帧，利用第二损失函数，对所述表情至脸部渲染器进行训练。
在一种可能的实现方式中，所述样本面部先验信息包括样本目标表情序列信息的真实值，所述第三确定模块还用于：
对所述虚拟对象的面部先验信息的真实值进行掩码以获得掩码后的所述虚拟对象的面部先验信息；
基于所述掩码后的所述虚拟对象的面部先验信息和所述样本音频特征，利用所述音频至表情转换器确定所述虚拟对象的样本目标表情序列信息的预测值。
在一种可能的实现方式中，所述样本面部先验信息包括样本目标表情序列信息的真实值，所述第二训练模块还用于：
基于所述样本目标表情序列的真实值，对所述样本源视频帧进行掩码，以获得经掩码的样本源视频帧；
基于所述样本面部先验信息和所述样本参考帧，针对所述经掩码的样本源视频帧，利用所述表情至脸部渲染器确定所述样本源视频帧的预测值；以及
基于所述样本源视频帧的预测值和真实值，利用所述第二损失函数，对所述表情至脸部渲染器进行训练；
其中，第二损失函数包括以下各项中的一项或多项的组合：像素损失、感知损失、对抗性损失和局部像素损失。
可见，在改进的神经网络模型训练过程中，可选择地引入了改进的第一损失函数和第二损失函数。这两个损失函数分别针对不同方面的特性进行优化，通过对神经网络参数的调整，提高了训练后神经网络的性能表现。第一损失函数和第二损失函数可以使得神经网络能够更有效地学习和适应输入数据的特征，并在训练过程中从多个方面进行优化。这种综合考虑不同特性的损失函数设计，有助于提高神经网络的泛化能力和训练效率，进而改善其在实际应用中的性能表现。
如图6所示，根据本公开实施例的方案得到视频帧的预测结果与视频帧的真实结果非常接近，其细节纹理的还原程度远高于传统方案。
根据本公开的又一方面，还提供了一种电子设备，用于实施根据本公开的实施例的方法。图7示出了根据本公开的实施例的电子设备2000的示意图。
如图8所示，所述电子设备2000可以包括一个或多个处理器2010，和一个或多个存储器2020。其中，所述存储器2020中存储有计算机可读代码，所述计算机可读代码当由所述一个或多个处理器2010运行时，可以执行如上所述的方法。
本公开实施例中的处理器可以是一种集成电路芯片，具有信号的处理能力。上述处理器可以是通用处理器、数字信号处理器(DSP)、专用集成电路(ASIC)、现成可编程门阵列(FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。可以实现或者执行本公开实施例中的公开的各方法、操作及逻辑框图。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等，可以是X86架构或ARM架构的。
一般而言，本公开的各种示例实施例可以在硬件或专用电路、软件、固件、逻辑，或其任何组合中实施。某些方面可以在硬件中实施，而其他方面可以在可以由控制器、微处理器或其他计算设备执行的固件或软件中实施。当本公开实施例的各方面被图示或描述为框图、流程图或使用某些其他图形表示时，将理解此处描述的方框、装置、系统、技术或方法可以作为非限制性的示例在硬件、软件、固件、专用电路或逻辑、通用硬件或控制器或其他计算设备，或其某些组合中实施。
例如，根据本公开的实施例的方法或装置也可以借助于图9所示的计算设备3000的架构来实现。如图8所示，计算设备3000可以包括总线3010、一个或多个CPU 3020、只读存储器(ROM)3030、随机存取存储器(RAM)3040、连接到网络的通信端口3050、输入/输出组件3060、硬盘3070等。计算设备3000中的存储设备，例如ROM 3030或硬盘3070可以存储本公开提供的方法的处理和/或通信使用的各种数据或文件以及CPU所执行的程序指令。计算设备3000还可以包括用户界面3080。当然，图8所示的架构只是示例性的，在实现不同的设备时，根据实际需要，可以省略图8示出的计算设备中的一个或多个组件。
根据本公开的又一方面，还提供了一种计算机可读存储介质。图9示出了根据本公开的存储介质4000的示意图。
如图9所示，所述计算机存储介质4020上存储有计算机可读指令4010。当所述计算机可读指令4010由处理器运行时，可以执行参照以上附图描述的根据本公开的实施例的方法。本公开实施例中的计算机可读存储介质可以是易失性存储器或非易失性存储器，或可包括易失性和非易失性存储器两者。非易失性存储器可以是只读存储器(ROM)、可编程只读存储器(PROM)、可擦除可编程只读存储器(EPROM)、电可擦除可编程只读存储器(EEPROM)或闪存。易失性存储器可以是随机存取存储器(RAM)，其用作外部高速缓存。通过示例性但不是限制性说明，许多形式的RAM可用，例如静态随机存取存储器(SRAM)、动态随机存取存储器(DRAM)、同步动态随机存取存储器(SDRAM)、双倍数据速率同步动态随机存取存储器(DDRSDRAM)、增强型同步动态随机存取存储器(ESDRAM)、同步连接动态随机存取存储器(SLDRAM)和直接内存总线随机存取存储器(DR RAM)。应注意，本文描述的方法的存储器旨在包括但不限于这些和任意其它适合类型的存储器。应注意，本文描述的方法的存储器旨在包括但不限于这些和任意其它适合类型的存储器。
另外，本申请实施例还提供了一种存储介质，所述存储介质用于存储计算机程序，所述计算机程序用于执行上述实施例提供的方法。
本申请实施例还提供了一种包括计算机程序的计算机程序产品，当其在计算机上运行时，使得计算机执行上述实施例提供的方法。
需要说明的是，附图中的流程图和框图，图示了按照本公开各种实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分，所述模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个接连地表示的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
一般而言，本公开的各种示例实施例可以在硬件或专用电路、软件、固件、逻辑，或其任何组合中实施。某些方面可以在硬件中实施，而其他方面可以在可以由控制器、微处理器或其他计算设备执行的固件或软件中实施。当本公开实施例的各方面被图示或描述为框图、流程图或使用某些其他图形表示时，将理解此处描述的方框、装置、系统、技术或方法可以作为非限制性的示例在硬件、软件、固件、专用电路或逻辑、通用硬件或控制器或其他计算设备，或其某些组合中实施。
在上面详细描述的本公开的示例实施例仅仅是说明性的，而不是限制性的。本领域技术人员应该理解，在不脱离本公开的原理和精神的情况下，可对这些实施例或其特征进行各种修改和组合，这样的修改应落入本公开的范围内。

Claims (15)

  1. 一种处理音频数据的方法，所述方法由计算机设备执行，所述方法包括：
    基于参考视频数据，确定虚拟对象的面部先验信息；
    基于目标音频数据，确定目标音频特征；
    基于所述面部先验信息和所述目标音频特征，确定目标表情序列信息；以及
    基于所述目标表情序列信息和所述参考视频数据中的参考帧，确定与所述目标音频数据对应的目标视频数据。
  2. 如权利要求1所述的处理音频数据的方法，所述面部先验信息包括以下各项中的一项或多项：参考面部形状信息、参考表情信息、和参考头部位姿信息。
  3. 如权利要求2所述的处理音频数据的方法，当所述面部先验信息包括所述参考面部形状信息和所述参考表情信息时，所述基于所述面部先验信息和所述目标音频特征，确定目标表情序列信息包括：
    对所述参考面部形状信息、所述参考表情信息和所述目标音频特征进行编码，以得到隐藏向量；以及
    对所述隐藏向量进行解码，以确定所述目标表情序列信息。
  4. 如权利要求2所述的处理音频数据的方法，所述基于所述目标表情序列信息和所述参考视频数据中的参考帧，确定与所述目标音频数据对应的目标视频数据包括：
    从所述参考视频数据中提取源视频帧；
    基于所述目标表情序列对所述源视频帧进行掩码，以得到第一视频帧，其中，在所述第一视频帧中，所述虚拟对象的部分面部组件被黑色像素块代替；以及
    基于所述第一视频帧和所述参考帧，利用所述面部先验信息，确定与所述目标音频数据对应的目标视频数据。
  5. 如权利要求4所述的处理音频数据的方法，所述基于所述目标表情序列对所述源视频帧进行掩码，以得到第一视频帧包括：
    基于所述目标表情序列和所述参考面部形状信息，确定在所述源视频帧中所述虚拟对象的部分面部组件对应的像素区域；以及
    将所述像素区域中的像素值修改为预设值，得到所述第一视频帧。
  6. 如权利要求4所述的处理音频数据的方法，所述基于第一视频帧和所述参考帧，利用所述面部先验信息，确定与所述目标音频数据对应的目标视频数据包括：
    利用共享权重的编码器来对所述第一视频帧和所述参考帧进行编码，以获取所述第一视频帧的第一编码特征和所述参考帧的第二编码特征；
    利用所述面部先验信息和所述目标表情序列，利用解码器对所所述第一编码特征和所述第二编码特征进行解码，以获得与所述目标音频数据对应的目标视频数据，其中，所述目标视频数据包括目标视频帧，所述目标视频帧中黑色像素块的区域被代替为所述虚拟对象的部分面部组件的预测值。
  7. 如权利要求6所述的处理音频数据的方法，所述解码器包括多个级联的解码层，每个解码层的输入包括前一个解码层的输出、所述面部先验信息和所述参考帧的编码特征，每个解码层的输出为目标视频帧的预测值。
  8. 一种训练神经网络的方法，所述神经网络包括音频至表情转换器和表情至脸部渲染器，所述方法由计算机设备执行，所述方法包括：
    基于样本视频数据，确定虚拟对象的样本面部先验信息；
    基于所述样本视频数据对应的样本音频数据，确定样本目标音频特征；
    基于样本面部先验信息和样本目标音频特征，利用所述音频至表情转换器确定目标样本表情序列信息的预测值；
    基于所述目标样本表情序列信息的预测值，利用第一损失函数，对所述音频至表情转换器进行训练；
    基于所述样本视频数据，确定样本参考帧和样本源视频帧；以及
    基于所述样本面部先验信息、所述样本参考帧、和所述样本源视频帧，利用第二损失函数，对所述表情至脸部渲染器进行训练。
  9. 如权利要求8所述的训练神经网络的方法，所述样本面部先验信息包括样本目标表情序列信息的真实值，所述利用所述音频至表情转换器确定样本目标表情序列信息的预测值包括：
    对所述虚拟对象的面部先验信息的真实值进行掩码以获得掩码后的所述虚拟对象的面部先验信息；
    基于所述掩码后的所述虚拟对象的面部先验信息和所述样本音频特征，利用所述音频至表情转换器确定所述虚拟对象的样本目标表情序列信息的预测值。
  10. 如权利要求8所述的训练神经网络的方法，所述样本面部先验信息包括样本目标表情序列信息的真实值，所述对所述表情至脸部渲染器进行训练包括：
    基于所述样本目标表情序列的真实值，对所述样本源视频帧进行掩码，以获得经掩码的样本源视频帧；
    基于所述样本面部先验信息和所述样本参考帧，针对所述经掩码的样本源视频帧，利用所述表情至脸部渲染器确定所述样本源视频帧的预测值；以及
    基于所述样本源视频帧的预测值和真实值，利用所述第二损失函数，对所述表情至脸部渲染器进行训练；
    其中，第二损失函数包括以下各项中的一项或多项的组合：像素损失、感知损失、对抗性损失和局部像素损失。
  11. 一种处理音频数据的装置，所述装置包括：
    第一预处理模块，用于基于参考视频数据，确定虚拟对象的面部先验信息：
    第二预处理模块，用于基于目标音频数据，确定目标音频特征；
    音频至表情转换器模块，用于基于所述面部先验信息和所述目标音频特征，确定目标表情序列信息；
    表情至面部渲染器，用于基于所述目标表情序列信息和所述参考视频数据中的参考帧，确定与所述目标音频数据对应的目标视频数据。
  12. 一种训练神经网络的装置，所述神经网络包括音频至表情转换器和表情至脸部渲染器，所述装置包括：
    第一确定模块，用于基于样本视频数据，确定虚拟对象的样本面部先验信息；
    第二确定模块，用于基于所述样本视频数据对应的样本音频数据，确定样本目标音频特征；
    第三确定模块，用于基于样本面部先验信息和样本目标音频特征，利用所述音频至表情转换器确定目标样本表情序列信息的预测值；
    第一训练模块，用于基于所述目标样本表情序列信息的预测值，利用第一损失函数，对所述音频至表情转换器进行训练；
    第四确定模块，用于基于所述样本视频数据，确定样本参考帧和样本源视频帧；以及
    第二训练模块，用于基于所述样本面部先验信息、所述样本参考帧、和所述样本源视频帧，利用第二损失函数，对所述表情至脸部渲染器进行训练。
  13. 一种计算机设备，所述计算机设备包括：
    处理器、通信接口、存储器和通信总线；
    其中，所述处理器、所述通信接口和所述存储器通过所述通信总线完成相互间的通信；所述通信接口为通信模块的接口；
    所述存储器，用于存储计算机程序，并将所述计算机程序传输给所述处理器；
    所述处理器，用于调用存储器中计算机程序执行权利要求1-7任意一项所述的方法，或者执行权利要求8-10任意一项所述的方法。
  14. 一种存储介质，所述存储介质用于存储计算机程序，所述计算机程序用于执行权利要求1-7任意一项所述的方法，或者执行权利要求8-10任意一项所述的方法。
  15. 一种包括计算机程序的计算机程序产品，当其在计算机上运行时，使得所述计算机执行权利要求1-7任意一项所述的方法，或者执行权利要求8-10任意一项所述的方法。
PCT/CN2025/078027 2024-02-29 2025-02-19 处理音频数据的方法、训练神经网络的方法和相关装置 Pending WO2025180271A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202410232157.0A CN120568108A (zh) 2024-02-29 2024-02-29 处理音频数据的方法和装置
CN202410232157.0 2024-02-29

Publications (2)

Publication Number Publication Date
WO2025180271A1 true WO2025180271A1 (zh) 2025-09-04
WO2025180271A9 WO2025180271A9 (zh) 2025-10-30

Family

ID=96831776

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2025/078027 Pending WO2025180271A1 (zh) 2024-02-29 2025-02-19 处理音频数据的方法、训练神经网络的方法和相关装置

Country Status (2)

Country Link
CN (1) CN120568108A (zh)
WO (1) WO2025180271A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN121095849A (zh) * 2025-11-07 2025-12-09 备至科技(安吉)有限公司 面部反应生成方法、装置、电子设备、存储介质及程序产品
CN121937597A (zh) * 2026-01-19 2026-04-28 科大讯飞股份有限公司 数字人生成模型训练方法、数字人生成方法及相关装置

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN121883862B (zh) * 2026-03-19 2026-06-02 合肥工业大学 基于混合卷积与物理约束的固废焚烧工况识别方法及系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115209180A (zh) * 2022-06-02 2022-10-18 阿里巴巴（中国）有限公司 视频生成方法以及装置
US11562597B1 (en) * 2022-06-22 2023-01-24 Flawless Holdings Limited Visual dubbing using synthetic models
CN115883753A (zh) * 2022-11-04 2023-03-31 网易（杭州）网络有限公司 视频的生成方法、装置、计算设备及存储介质
CN117237521A (zh) * 2023-02-27 2023-12-15 长城信息股份有限公司 语音驱动人脸生成模型构建方法、目标人说话视频生成方法
CN117528135A (zh) * 2023-11-06 2024-02-06 上海智臻智能网络科技股份有限公司 语音驱动的脸部视频生成方法、装置、电子设备及介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115209180A (zh) * 2022-06-02 2022-10-18 阿里巴巴（中国）有限公司 视频生成方法以及装置
US11562597B1 (en) * 2022-06-22 2023-01-24 Flawless Holdings Limited Visual dubbing using synthetic models
CN115883753A (zh) * 2022-11-04 2023-03-31 网易（杭州）网络有限公司 视频的生成方法、装置、计算设备及存储介质
CN117237521A (zh) * 2023-02-27 2023-12-15 长城信息股份有限公司 语音驱动人脸生成模型构建方法、目标人说话视频生成方法
CN117528135A (zh) * 2023-11-06 2024-02-06 上海智臻智能网络科技股份有限公司 语音驱动的脸部视频生成方法、装置、电子设备及介质

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN121095849A (zh) * 2025-11-07 2025-12-09 备至科技(安吉)有限公司 面部反应生成方法、装置、电子设备、存储介质及程序产品
CN121937597A (zh) * 2026-01-19 2026-04-28 科大讯飞股份有限公司 数字人生成模型训练方法、数字人生成方法及相关装置

Also Published As

Publication number Publication date
WO2025180271A9 (zh) 2025-10-30
CN120568108A (zh) 2025-08-29

Similar Documents

Publication Publication Date Title
CN113901894B (zh) 一种视频生成方法、装置、服务器及存储介质
Tang et al. Lgm: Large multi-view gaussian model for high-resolution 3d content creation
CN111754596B (zh) 编辑模型生成、人脸图像编辑方法、装置、设备及介质
US10896535B2 (en) Real-time avatars using dynamic textures
CN111401216B (zh) 图像处理、模型训练方法、装置、计算机设备和存储介质
US11581020B1 (en) Facial synchronization utilizing deferred neural rendering
US11582519B1 (en) Person replacement utilizing deferred neural rendering
WO2024051445A1 (zh) 图像生成方法以及相关设备
CN114187165B (zh) 图像处理方法和装置
WO2025246674A1 (zh) 图像处理方法、装置、电子设备、计算机可读存储介质及计算机程序产品
WO2025180271A9 (zh) 处理音频数据的方法、训练神经网络的方法和相关装置
CN113705295A (zh) 对象姿态迁移方法、装置、设备及存储介质
CN111860380A (zh) 人脸图像生成方法、装置、服务器及存储介质
He et al. Emotalk3d: High-fidelity free-view synthesis of emotional 3d talking head
CN118175324A (zh) 用于视频生成的多维生成框架
CN113542758B (zh) 生成对抗神经网络辅助的视频压缩和广播
CN117668291A (zh) 用于视频处理的方法、电子设备和计算机程序产品
CN111275778A (zh) 人脸简笔画生成方法及装置
Gowda et al. From pixels to portraits: A comprehensive survey of talking head generation techniques and applications
US11954779B2 (en) Animation generation method for tracking facial expression and neural network training method thereof
CN117115310A (zh) 一种基于音频和图像的数字人脸生成方法及系统
CN114694065B (zh) 视频处理方法、装置、计算机设备及存储介质
WO2021170230A1 (en) Devices and methods for providing images and image capturing based on a text and providing images as a text
Kumar et al. Steganography-based facial re-enactment using generative adversarial networks
CN115631285B (zh) 基于统一驱动的人脸渲染方法、装置、设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 25760813

Country of ref document: EP

Kind code of ref document: A1