WO2023088080A1 - Speaking video generation method and apparatus, and electronic device and storage medium - Google Patents

Abstract

Disclosed are a speaking video generation method and apparatus, and a device and a storage medium. The method comprises: acquiring phoneme features and acoustic features of voice drive data, wherein the voice drive data comprises at least one of audio and text; acquiring at least one set of facial key point information of a target object in a first image according to the phoneme features and the acoustic features; according to the at least one set of facial key point information and a second image including the face of the target object, obtaining at least one target facial image corresponding to the voice drive data, wherein a set area comprising a specific part of the target object in the second image is blocked; and obtaining a speaking video of the target object according to the voice drive data and the at least one target facial image.

Description

Talking video generation method, device, electronic device and storage medium

Cross References to Related Applications

This patent application claims priority to Chinese Patent Application No. 202111386695.8 filed on November 22, 2021, the entire contents of which are incorporated herein by reference.

technical field

The present disclosure relates to the technical field of computer vision, and in particular to a method, device, device and storage medium for generating a speaking video.

Background technique

Talking video generation technology is an important technology used in voice-driven character images and cross-modal video generation, and plays a key role in the commercialization of virtual digital objects. At present, the corresponding mouth shape image is usually determined according to the voice frame, so as to obtain a series of mouth shape images corresponding to the output voice to generate a speaking video. However, the accuracy of the speaker's mouth shape in the video generated by this method is low and the mouth shape changes abruptly. .

Contents of the invention

An embodiment of the present disclosure provides a solution for generating a talking video.

According to the first aspect of the present disclosure, there is provided a method for generating a talking video, the method comprising: acquiring phoneme features and acoustic features of sound-driven data, the sound-driven data including at least one of audio and text; according to the The phoneme feature and the acoustic feature obtain at least one set of facial key point information of the target object in the first image; according to the at least one set of facial key point information and the second image containing the face of the target object, obtain At least one target facial image corresponding to the sound driving data, wherein a set area including a specific part of the target object in the second image is blocked; according to the sound driving data and the at least one target Facial image to obtain the speaking video of the target object.

In combination with any implementation manner provided by the present disclosure, the acquiring the phoneme features and acoustic features of the sound driving data includes: acquiring the phonemes contained in the audio corresponding to the sound driving data and the time stamps corresponding to each phoneme, and obtaining the sound The phoneme feature of the driving data; performing feature extraction on the audio corresponding to the sound driving data to obtain the acoustic features of the sound driving data.

In combination with any implementation manner provided by the present disclosure, the acquiring at least one set of facial key point information of the target object in the first image according to the phoneme features and the acoustic features includes: acquiring multiple phoneme features contained in the phoneme features Sub-phoneme features and sub-acoustic features corresponding to the plurality of sub-phoneme features; the sub-phoneme features and corresponding sub-acoustic features are input to the face key point extraction network, and the sub-phoneme features and the sub-acoustic features are obtained. Corresponding facial key point information.

In combination with any implementation manner provided by the present disclosure, the facial key point information includes 3D facial key point information, and according to the at least one set of facial key point information and the second image containing the face of the target object Before obtaining at least one target facial image corresponding to the sound driving data, the method further includes: projecting the 3D facial key point information onto a 2D plane, and obtaining the 3D facial key point information corresponding to 2D facial key point information; using the 2D facial key point information to update the facial key point information.

In combination with any implementation manner provided by the present disclosure, at least one target face corresponding to the sound driving data is obtained according to the at least one set of face key point information and the second image containing the face of the target object Before the image, the method also includes: performing filtering processing on multiple groups of face key point information, so that the change between the face key point information of each image frame and the face key point information of adjacent frames of the image frame The quantity satisfies the set conditions.

In combination with any implementation manner provided by the present disclosure, at least one target face corresponding to the sound driving data is obtained according to the at least one set of facial key point information and the second image containing the face of the target object The internal image includes: inputting each group of face key point information and the second image to the face complement network to obtain a target face image corresponding to the face key point information, wherein the face complement The whole network is used to complement the occluded set area in the second image according to the key point information of the face.

In combination with any implementation manner provided by the present disclosure, the obtaining the speaking video of the target object according to the sound driving data and the at least one target facial image includes: combining the at least one target facial image with the set fused with a given background image to obtain a first image sequence; according to the audio corresponding to the first image sequence and the sound driving data, a speaking video of the target object is obtained.

In combination with any of the implementations provided in the present disclosure, the facial key point extraction network is trained using phoneme feature samples and corresponding acoustic feature samples, wherein the phoneme feature samples and the acoustic feature samples include labeled facial key points point information.

In combination with any of the implementations provided in the present disclosure, the facial key point extraction network is trained in the following manner: according to the phoneme feature samples and the corresponding acoustic feature samples, the initial facial key point extraction network is trained, and the network When the change of the loss meets the convergence condition, the training is completed to obtain the facial key point extraction network, wherein the network loss includes the difference between the facial key point information predicted by the initial neural network and the marked facial key point information. difference.

In combination with any implementation manner provided in the present disclosure, the phoneme feature sample and the acoustic feature sample are obtained by marking the object's facial key point information on the phoneme feature and the acoustic feature of an object's audio.

In combination with any implementation manner provided by the present disclosure, the phoneme feature sample and the acoustic feature sample are obtained in the following manner: acquiring a speaking video of the object; acquiring multiple facial images according to the speaking video, and combining with each At least one audio frame corresponding to the facial image; obtaining phoneme features and acoustic features of at least one audio frame corresponding to each facial image; obtaining facial key point information according to the plurality of facial images, and according to The facial key point information marks the phoneme features and the acoustic features to obtain the phoneme feature samples and the acoustic feature samples.

In combination with any of the embodiments provided in the present disclosure, the face completion network is trained by generating an adversarial network, the generation adversarial network includes the face completion network and the first identification network, and the trained network loss includes : The first loss, which is used to indicate the difference between the face completion image output by the face completion network and the complete face image, wherein the complete face image is corresponding to the facial key point information A face image; a second loss, which is used to indicate the difference between the classification result output by the first discrimination network for the input image and the annotation information of the input image, wherein the annotation information indicates that the input image is the The face complement image output by the face complement network may be a real face image.

In combination with any implementation manner provided by the present disclosure, the generation confrontation network further includes a second discriminant network, and the trained network loss further includes: a third loss, which is used to instruct the second discriminative network to be effective for the face complement The difference between the discriminative result and the true corresponding result of the correspondence between the full image and the phoneme features.

According to a second aspect of the present disclosure, there is provided an apparatus for generating a speaking video, the apparatus comprising: a first acquisition unit configured to acquire phoneme features and acoustic features of sound driving data, the sound driving data including audio, text At least one item; a second obtaining unit, configured to obtain at least one set of facial key point information of the target object in the first image according to the phoneme feature and the acoustic feature; a first obtaining unit, configured to obtain at least one set of face key point information according to the at least one Combining face key point information and a second image containing the face of the target object to obtain at least one target face image corresponding to the sound driving data, wherein the second image includes the face of the target object The set area of the specific part is blocked; the second obtaining unit is configured to obtain the speaking video of the target object according to the sound driving data and the at least one target facial image.

According to a third aspect of the present disclosure, there is provided an electronic device, the device includes a memory and a processor, the memory is used to store computer instructions executable on the processor, and the processor is used to execute the computer instructions At this time, the method for generating a speaking video described in any implementation manner provided by the present disclosure is implemented.

According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method for generating a talking video in any implementation manner provided by the present disclosure is implemented.

The speaking video generation method, device, device, and computer-readable storage medium of one or more embodiments of the present disclosure acquire at least one set of facial key points of the target object in the first image according to the phoneme features and acoustic features of the voice driving data information; and according to the at least one set of face key point information and the second image containing the face of the target object, at least one target face image corresponding to the sound driving data is obtained, wherein the second A set area including a specific part of the target object in the image is blocked; finally, according to the sound driving data and the at least one target face image, a speaking video of the target object is obtained. In the embodiment of the present disclosure, a target face image is generated according to the key face information of the target object corresponding to the sound driving data and the image of the target object covering a specific part, and the obtained speech video of the target object is described in The mouth shape of the target object has a high degree of matching with the voice driving data, and the mouth shape changes coherently, and the speaking state of the target object is real and natural.

Description of drawings

In order to more clearly illustrate the technical solutions in one or more embodiments of this specification, the following will briefly introduce the drawings that need to be used in the description of the embodiments. Obviously, the drawings in the following description are only one example of this specification. Or some embodiments described in multiple embodiments, for those skilled in the art, other drawings can also be obtained according to these drawings without paying creative efforts.

FIG. 1 is a flow chart of a method for generating a speaking video proposed by at least one embodiment of the present disclosure;

Fig. 2 is a flow chart of a face key point extraction network training method proposed by at least one embodiment of the present disclosure;

Fig. 3 is a flowchart of a sample acquisition method proposed by at least one embodiment of the present disclosure;

Fig. 4 is a flowchart of another method for generating a speaking video proposed by at least one embodiment of the present disclosure;

Fig. 5 is a schematic diagram of the speaking video generation method shown in Fig. 4;

Fig. 6 is a schematic diagram of acquiring face key point information in a method for generating a speaking video proposed by at least one embodiment of the present disclosure;

Fig. 7 is a schematic structural diagram of a speaking video generation device proposed by at least one embodiment of the present disclosure;

Fig. 8 is a schematic structural diagram of an electronic device proposed by at least one embodiment of the present disclosure.

Detailed ways

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatuses and methods consistent with aspects of the present disclosure as recited in the appended claims.

The term "and/or" in this article is just an association relationship describing associated objects, which means that there can be three relationships, for example, A and/or B can mean: A exists alone, A and B exist simultaneously, and there exists alone B these three situations. In addition, the term "at least one" herein means any one of a variety or any combination of at least two of the more, for example, including at least one of A, B, and C, which may mean including from A, Any one or more elements selected from the set formed by B and C.

At least one embodiment of the present disclosure provides a method for generating a talking video, and the method may be executed by an electronic device such as a terminal device or a server. The terminal device may be a fixed terminal or a mobile terminal, such as a mobile phone, a tablet computer, a game machine, a desktop computer, an advertising machine, an all-in-one machine, a vehicle-mounted device, etc., and the server includes a local server or a cloud server. The method can also be implemented by a processor invoking computer-readable instructions stored in a memory.

Fig. 1 shows a flowchart of a method for generating a speaking video according to at least one embodiment of the present disclosure. As shown in Fig. 1 , the method includes steps 101-104.

In step 101, phoneme features and acoustic features of sound driving data are acquired.

A phoneme is the smallest unit of speech constituting a syllable. The audio corresponding to the sound driving data may contain one or more phonemes, and phoneme features may include features representing the start and end times of pronunciation of each phoneme. Taking the audio segment "Hello" as an example, the phoneme features of the sound-driven data may include, for example: n[0,0.2], i3[0.2,0.4], h[0.5,0.7], ao3[0.7 ,1.2], wherein, the data in [] indicates the starting and ending time of the pronunciation of the corresponding phoneme, and the unit is, for example, seconds. In the embodiment of the present disclosure, the phoneme features of the sound driving data may be obtained by acquiring the phonemes contained in the audio corresponding to the sound driving data and the time stamps corresponding to each phoneme.

The acoustic features are mainly used to describe the pronunciation characteristics of the audio, and the acoustic features include but are not limited to at least one of linear prediction parameters, Mel-frequency cepstral coefficients, perceptual linear high-efficiency coefficients, and the like. In the embodiment of the present disclosure, the acoustic features are, for example, Mel-frequency cepstral coefficients. In the embodiment of the present disclosure, the acoustic features of the sound driving data may be obtained by performing feature extraction on the audio corresponding to the sound driving data.

The sound driving data is pre-stored in the electronic device that executes the talking video generation method, or in other devices other than the electronic device, or is collected on-site by a sound collection device, etc. Sources of sound driver data are restricted. In the embodiment of the present disclosure, the sound driving data may include at least one of audio and text.

In the case where the sound driving data only includes audio, the text (for example, text information) corresponding to the audio can be determined by performing speech recognition on the audio;

In the case where the sound driving data only includes text, the text information corresponding to the text can be converted into audio (for example, a speech segment) by performing speech synthesis on the text;

Where the sound-driven data includes audio and text, the audio and text correspond to the same pronunciation. For example, the text is "Hello", and the audio in the sound driving data is a speech segment that emits the sound of "Hello".

In some embodiments, the phoneme features of the sound driving data can be obtained by performing an alignment operation on the audio and text corresponding to the sound driving data. Wherein, the alignment operation refers to aligning each speech segment in the audio with the phoneme in the text corresponding to the speech segment, that is, determining when the pronunciation corresponding to the text in the audio starts to be pronounced. By aligning the audio and text, on the one hand, the phonemes contained in the audio are determined, and at the same time, the time stamp corresponding to each phoneme can be obtained according to the duration of the pronunciation, so that the phoneme features of the sound driving data can be obtained.

Still taking "Hello" as an example, after aligning the audio and text, it can be determined that the pronunciation of the phoneme "n" will be pronounced in 0-0.2 seconds, and the pronunciation of "i3" will be pronounced in 0.2-0.4 seconds, etc., Thus, the phoneme features of the sound driving data can be obtained. Those skilled in the art should understand that the phoneme features of the sound driving data may also be acquired in other ways, which is not limited in this embodiment of the present disclosure.

In the embodiment of the present disclosure, by utilizing the phoneme features, the target object may be driven in any manner of audio or text to generate a speaking video of the target object.

In step 102, at least one set of facial key point information of the target object in the first image is obtained according to the phoneme feature and the acoustic feature.

When the target object (for example, a person) makes different voices, the mouth shape will change accordingly. Therefore, the position of the target object's mouth area or the facial key points in the set area containing the mouth area will change accordingly. The change. It can be known that the phoneme features and acoustic features of an audio frame correspond to a set of facial key point information of the target object. When the target object pronounces a certain phoneme, the facial key point information corresponding to its face can be determined. Wherein, the facial key point information includes the facial key points of the target object (for example, key points corresponding to facial features and facial contours) in the image (for example, the first image) containing the face of the target object location information. In the present disclosure, the information of each facial key point at the same time may be referred to as a set of facial key point information.

Taking the generation of the speaking video of the target object in the first image as an example, in this step, at least one set of facial key point information of the target object in the first image is obtained according to the phoneme features and acoustic features of the voice driving data. In the case that the audio corresponding to the sound driving data includes multiple phonemes, the facial key point information sequence corresponding to these phonemes and corresponding acoustic features can be obtained. Wherein, the facial key point information sequence includes multiple sets of facial key point information arranged in chronological order.

The embodiment of the present disclosure also utilizes the acoustic features on the basis of the phoneme features of the voice driving data, so that the acquired facial key point information can better match the pronunciation features of the audio corresponding to the voice driving data, so that the subsequent generated speech Video is more realistic.

In step 103, at least one target facial image corresponding to the sound driving data is obtained according to the at least one set of facial key point information and the second image including the target object's face.

Wherein, the second image is an image containing the face of the target object, and the second image can be obtained by performing occlusion processing on the first image, or by occluding another image different from the first image. An image is obtained by performing occlusion processing. For example, the first image is a face image of the target object A smiling, and another image different from the first image may be a face image of the target object A curling his lips.

In the second image, a set area including a specific part (for example, mouth) of the target object is blocked, and the set area includes an area where the position of key points on the face of the target object changes when the target object speaks For example, it may be the lower half of the face of the target object, or the facial area below the forehead, or the mouth area. Embodiments of the present disclosure do not limit the specific area to be blocked.

In some embodiments, the second image in which the set area is blocked may be generated by filling the set area with noise. Wherein, performing noise filling on the set area refers to setting each pixel in the set area with randomly generated pixel values. Those skilled in the art should understand that the blocking of the set area can also be performed in other ways, which is not limited in the present disclosure.

According to at least one set of facial key point information obtained in step 102, the blocked part (that is, the set area) in the second image can be complemented, so that the blocked set in the second image The distribution of facial keypoints in a given region is consistent with the phonemic and acoustic features of the voice-driven data. In this way, in the at least one target facial image generated according to the at least one set of facial key point information and the second image, the face in the area corresponding to the set area (that is, the completed area) The key point information is matched with the sound driving data.

In step 104, a speaking video of the target object is obtained according to the sound driving data and the at least one target facial image.

In the embodiment of the present disclosure, in the obtained speech video of the target object, the sound output by the target object is the audio corresponding to the sound driving data, and in each image frame of the speech video, the sound of the target object The face key point information corresponds to the phoneme feature and the acoustic feature of the output voice. The mouth shapes and speaking expressions of the target object in the generated speaking video are consistent with the pronunciation, so that the audience can feel that the target object is speaking.

In an embodiment of the present disclosure, at least one target facial image is generated according to at least one set of face key point information of the target object corresponding to the sound driving data and an image of the target object that blocks a specific part, and the obtained target object The mouth shape of the target object in the speaking video matches the voice driving data to a high degree, and the mouth shape changes coherently, and the speaking state of the target object is real and natural.

In some embodiments, the at least one target face image can be fused with the set background image to obtain the first video, and the target target can be obtained according to the audio corresponding to the first video and the sound driving data. Subject's speaking video. In an example, the pixels of the face area in the target face image may be used as foreground pixels to be superimposed on the set background image, so as to realize the fusion of the target face image and the set background image. Those skilled in the art should understand that multiple ways can be used to fuse the target face image and the set background image, which is not limited in the present disclosure

Through the above method, the speaking video of the target object in any background can be generated, which enriches the application scenarios of the method for generating the speaking video.

In some embodiments, at least one set of facial key point information of the target object in the first image corresponding to the phoneme feature and the acoustic feature may be obtained by using a facial key point extraction network.

First, multiple sub-phoneme features included in the phoneme features and sub-acoustic features corresponding to the multiple sub-phoneme features are acquired.

In an example, a plurality of sub-phoneme features included in the phoneme feature and sub-acoustic sub-features corresponding to the plurality of sub-phoneme features may be obtained by performing a sliding window on the phoneme feature and the acoustic feature of the sound driving data. feature. For example, the phoneme feature and the acoustic feature can be divided into multiple sub-phoneme features and sub-acoustic features according to the length of the time window. Specifically, in the process of sliding the window on the phoneme features and acoustic features of the sound driving data, the phoneme features and acoustic features in the time window obtained after each sliding window operation can be used as sub-phoneme features and sub-phoneme features. Acoustic features, and sub-phoneme features and sub-acoustic features in the same time window correspond to the same speech segment.

Next, the sub-phoneme features and corresponding sub-acoustic features are input to the trained facial key point extraction network to obtain facial key point information corresponding to the sub-phoneme features and the sub-acoustic features. Specifically, multiple sub-phoneme features and corresponding multiple sub-acoustic features may be input into the facial key point extraction network in time sequence in the form of multiple sub-phoneme feature-sub-acoustic feature pairs. The facial key point extraction network is used to determine a corresponding set of facial key point information according to each sub-phoneme feature-sub-acoustic feature pair. After inputting all the sub-phoneme feature-sub-acoustic feature pairs into the facial key point extraction network, multiple sets of facial key point information corresponding to the voice driving data can be obtained.

In the embodiment of the present disclosure, through the trained facial key point extraction network, the facial key point information corresponding to each sub-phoneme feature-sub-acoustic feature pair can be obtained, and the pronunciation, mouth shape and speaking of the target object can be realized. Good match for expressions.

In the embodiment of the present disclosure, the facial key point extraction network may be a three-dimensional 3D facial key point extraction network, that is, the output facial key point information is 3D facial key point information, and the 3D facial key point In addition to the position information of the facial key points, the point information also includes the depth information of the facial key points; the facial key point extraction network can also be a two-dimensional 2D facial key point extraction network, that is The output facial key point information is 2D facial key point information.

In the case that the facial key point information is 3D facial key point information, according to the at least one set of facial key point information and the second image containing the face of the target object, the sound corresponding to the sound is obtained. Before driving at least one target facial image corresponding to the data, the method further includes: projecting the 3D facial key point information onto a 2D plane to obtain 2D facial key point information corresponding to the 3D facial key point information ; Utilizing the 2D facial key point information to update the facial key point information. Afterwards, according to at least one set of 2D face key point information and a second image containing the face of the target object, at least one target face image corresponding to the sound driving data is obtained; finally, according to the sound driving data and the at least one target face image to obtain a speaking video of the target object.

In some embodiments, multiple groups of face key point information can be filtered so that the face key point information of each image frame in the finally obtained talking video is consistent with the adjacent frames of the image frame (including the previous frame and The amount of change between the face key point information of the next frame) satisfies the setting condition, which can include, for example, the position of each face key point in an image frame and the corresponding face key point in the adjacent frame. The variations between the positions are all smaller than the set threshold. Vibrating frames with large changes in face key point information can be filtered out by the above method, so as to avoid sudden changes in mouth shape in the generated speaking video.

In an example, the moving average processing of the consecutive frames corresponding to the multiple sets of facial key point information may be implemented by performing Gaussian filtering on the multiple sets of facial key point information in a time window. Wherein, the moving average processing refers to carrying out a weighted average of the value of the facial key point of each frame and the value of the facial key point of the adjacent frame, and updating the value of the facial key point of the frame by using the result of the weighted average.

In some embodiments, at least one target face image corresponding to the sound driving data may be obtained by inputting each set of face key point information and a second image containing the face of the target object into the face A part completion network to obtain a target face image corresponding to the facial key point information, wherein the face completion network is used to set the occlusion in the second image according to the facial key point information area to complete.

In the embodiment of the present disclosure, the masked set area in the second image is complemented according to the face key point information through the face completion network, so that the face key point information of the set area Consistent with the input facial key point information, so that the target object's mouth shape and speaking expression match the voice, and use the face complement network to complement the blocked set area in the second image It can generate high-definition target face images.

In some embodiments, the facial key point extraction network can be obtained by using phoneme feature samples and acoustic feature samples for training. The training method may be executed by a server, and the server executing the training method may be different from the device executing the above-mentioned talking video generation method.

FIG. 2 shows a training method for a facial key point extraction network proposed by at least one embodiment of the present disclosure. As shown in FIG. 2 , the training method includes steps 201-202.

In step 201, a phoneme feature sample and a corresponding acoustic feature sample are acquired, and the phoneme feature sample and the acoustic feature sample include marked facial key point information of the target object. Wherein, the phoneme feature sample and the corresponding acoustic feature sample are obtained based on the same speech segment, and the facial key point information marked in the phoneme feature sample and the corresponding acoustic feature sample are the same.

In step 202, according to the phoneme feature sample and the corresponding acoustic feature sample, the initial facial key point extraction network is trained, and when the change of the network loss meets the convergence condition, the training is completed to obtain the facial key point extraction network, Wherein, the network loss includes the difference between the facial key point information predicted by the initial neural network and the labeled facial key point information.

In some embodiments, the phoneme feature samples and the acoustic feature samples are obtained by marking the object's facial key point information on the phoneme features and acoustic features of the audio of an object. In an example, the phoneme feature samples and corresponding acoustic feature samples may be acquired through the method shown in FIG. 3 .

In step 301, the talking video of the subject is acquired. Wherein, the object may be the above-mentioned target object whose talking video is to be generated, or may be a different object from the target object.

In an example, when it is desired to generate a speaking video of a certain target object, the existing speaking video of the target object is acquired to obtain phoneme feature samples and acoustic feature samples.

In step 302, a plurality of facial images and at least one audio frame corresponding to each of the facial images are acquired according to the talking video.

By splitting the speaking video, a speech segment corresponding to the speaking video and a plurality of facial images included in the speaking video are obtained. Wherein, the multiple audio frames in the speech segment have a corresponding relationship with the multiple facial images.

In step 303, phoneme features and acoustic features of at least one audio frame corresponding to each of the facial images are acquired.

According to the corresponding relationship between the plurality of facial images and the plurality of audio frames in the speech segment, the phoneme features and acoustic features of at least one audio frame corresponding to any facial image are acquired.

In step 304, facial key point information is obtained according to the plurality of facial images, and the phoneme feature and the acoustic feature are marked according to the facial key point information to obtain the phoneme feature sample and the Acoustic feature samples described above.

In the embodiment of the present disclosure, by generating phoneme feature samples and acoustic feature samples according to the existing speech video of the target object for which the speech video is to be generated, the phoneme feature and acoustic feature samples of the speech spoken by the target object can be accurately established. The association between acoustic features and facial keypoint information can better realize the training of facial keypoint generation network.

In some embodiments, the face completion network can be trained using a generative adversarial network. The training method may be executed by a server, and the server executing the training method may be different from the device executing the above-mentioned speaking video generation method.

Wherein, the generation confrontation network includes the face completion network and the first identification network, and the face completion network is used to analyze the input occluded face image (that is, the one with the occlusion area) according to the facial key point information. face image) to generate a face complement image, wherein the occluded face image is obtained by blocking a set area including a specific part (for example, mouth) in the complete face image, so The complete face image may be the face image corresponding to the face key point information; the generated face complement image and the real face image are randomly input into the first identification network, and the first identification network Outputting an identification result for the input image, that is, judging whether the input image is a face complement image or a real face image.

The loss of using the generation confrontation network to train the face completion network includes:

The first loss is used to indicate the difference between the face completion image output by the face completion network and the complete face image, wherein the complete face image is the face corresponding to the facial key point information internal image;

The second loss is used to indicate the difference between the classification result output by the first discrimination network for the input image and the annotation information of the input image, wherein the annotation information indicates that the input image is the face complement The face complement image output by the whole network may be a real face image.

The training is completed when the variation of the training loss satisfies the convergence condition, and the face completion network is obtained.

In the embodiment of the present disclosure, using a generative confrontation network to train the face completion network can improve the accuracy of the face completion image output by the face completion network, which is conducive to improving the generated The image quality of the speaking video of the target subject.

In some embodiments, a second discriminant network for judging whether the face complement image is aligned with phoneme features can also be added to assist the training of the face complement network. In the training method, the face complement image output by the face complement network is input to the second discrimination network.

In addition to the above-mentioned first loss and second loss, the loss of this training also includes a third loss, and the third loss is used to indicate that the second discriminant network is for the difference between the face complement image and the phoneme feature The difference between the corresponding (eg, aligned) discriminative result and the true corresponding result.

By adding the second discrimination network to train the face completion network, the alignment effect between phoneme features and facial key points is further improved, which is beneficial to improve the quality of speaking videos.

Combining the flow chart of the speaking video generation method shown in FIG. 4, the schematic diagram of the speaking video generation method shown in FIG. 5, and the schematic diagram of obtaining facial key point information shown in FIG. The method is described.

In step 401, an alignment operation is performed on the audio and text corresponding to the sound driving data to obtain phoneme features of the sound driving data.

The audio corresponding to the sound driving data is, for example, as shown in FIG. 6 , which is the speech segment of "Hello", and the text corresponding to the sound driving data is the text of "Hello". By performing an alignment operation on the audio and the text, the phoneme features of the sound driving data are obtained.

As shown in Figure 5, in the case where the sound driving data only includes audio, the text corresponding to the audio can be determined by performing speech recognition on the audio; in the case where the sound driving data only includes text, Text information corresponding to the text may be converted into audio by performing speech synthesis on the text.

In step 402, feature extraction is performed on the audio corresponding to the sound driving data to obtain the Mel cepstrum feature of the sound driving data, that is, Mel frequency cepstral coefficients.

In step 403, a plurality of sub-phoneme features contained in the phoneme feature and sub-acoustic features corresponding to the plurality of sub-phoneme features are obtained by performing a sliding window on the phoneme feature and the acoustic feature of the sound driving data . The time window is shown as a dotted box in FIG. 6 , and the arrow shows the sliding direction of the time window. During the sliding process, the phoneme features and acoustic features obtained in each time window are sub-phoneme features and sub-acoustic features, and the sub-phoneme features and sub-acoustic features in the same time window correspond to the same speech segment.

In step 404, the sub-phoneme features and the corresponding sub-acoustic features are input to the trained facial key point extraction network to obtain facial key point information corresponding to the sub-phoneme features and the sub-acoustic features. As shown in FIG. 6 , for each obtained sub-phoneme feature and sub-acoustic feature in a time window, the face key point extraction network outputs face key point information corresponding to the time window.

Exemplarily, the facial key point extraction network is a 3D facial key point extraction network, and correspondingly, the obtained facial key point information is 3D facial key point information.

In step 405, 2D facial key point information corresponding to the 3D facial key point information is acquired.

In step 406, filter processing is performed on multiple sets of 2D facial key point information, so that the variation between the 2D facial key point information of each image frame and the facial key point information of adjacent frames meets the set condition.

In step 407, each set of filtered 2D facial key point information and the second image are input to the face completion network to obtain a target facial image corresponding to the 2D facial key point information, Wherein, the second image is an occluded face image, and the lower half of the face in the second image is filled with noise for occlusion.

In step 408, the multi-frame target facial images obtained in step 407 (for example, the speaker's facial images) are fused with the background image to obtain a first image sequence.

In step 409, a speaking video of the target object is obtained according to the first image sequence and the audio corresponding to the sound driving data.

Fig. 7 is a schematic structural diagram of a talking video generating device proposed by at least one embodiment of the present disclosure; as shown in Fig. 7 , the device includes: a first acquiring unit 701, configured to acquire phoneme features and acoustic features of sound driving data, so The sound driving data includes at least one of audio and text; the second acquisition unit 702 is configured to acquire at least one set of facial key point information of the target object in the first image according to the phoneme features and the acoustic features; the second An obtaining unit 703, configured to obtain at least one target facial image corresponding to the sound driving data according to the at least one set of facial key point information and the second image containing the target object's face, wherein, A set area including a specific part of the target object in the second image is blocked; a second obtaining unit 704 is configured to obtain the target object according to the sound driving data and the at least one target facial image talking video.

In combination with any implementation manner provided in the present disclosure, the first acquisition unit is specifically configured to: acquire the phonemes contained in the audio corresponding to the sound driving data and the time stamps corresponding to each phoneme, and obtain the phoneme features of the sound driving data ; Performing feature extraction on the audio corresponding to the sound driving data to obtain the acoustic features of the sound driving data.

In combination with any implementation manner provided in the present disclosure, the second acquisition unit is specifically configured to: acquire a plurality of sub-phoneme features included in the phoneme feature and sub-acoustic features corresponding to the plurality of sub-phoneme features; The features and corresponding sub-acoustic features are input to the facial key point extraction network to obtain facial key point information corresponding to the sub-phoneme features and the sub-acoustic features.

In combination with any of the implementations provided in the present disclosure, the facial key point information includes 3D facial key point information, and the device further includes a projection unit, configured to use the at least one set of facial key point information and include the The second image of the face of the target object, before obtaining at least one target face image corresponding to the sound driving data, project the 3D facial key point information onto a 2D plane to obtain the 3D facial key 2D facial key point information corresponding to the point information; using the 2D facial key point information to update the facial key point information.

In combination with any of the implementations provided in the present disclosure, the device further includes a filtering unit, configured to obtain, according to the at least one set of face key point information and the second image containing the face of the target object, a Before the at least one target facial image corresponding to the sound driving data, multiple groups of facial key point information are filtered, so that the facial key point information of each image frame and the facial key point information of adjacent frames The amount of change satisfies the set condition.

In combination with any implementation manner provided by the present disclosure, the first obtaining unit is specifically configured to: input each set of facial key point information and the second image to the face completion network, and obtain the facial key point The target face image corresponding to the information, wherein the face completion network is used to complete the blocked set area in the second image according to the facial key point information.

In combination with any implementation manner provided in the present disclosure, the second obtaining unit is specifically configured to: fuse the at least one target face image with the set background image to obtain a first image sequence; according to the first image sequence The audio corresponding to the sound driving data is used to obtain the speaking video of the target object.

In combination with any of the implementations provided in the present disclosure, the facial key point extraction network is trained using phoneme feature samples and corresponding acoustic feature samples, wherein the phoneme feature samples and the acoustic feature samples include labeled facial key points point information.

In combination with any of the implementations provided in the present disclosure, the facial key point extraction network is trained in the following manner: according to the phoneme feature samples and the corresponding acoustic feature samples, the initial facial key point extraction network is trained, and the network When the change of the loss meets the convergence condition, the training is completed to obtain the facial key point extraction network, wherein the network loss includes the difference between the facial key point information predicted by the initial neural network and the marked facial key point information. difference.

In combination with any implementation manner provided in the present disclosure, the phoneme feature sample and the acoustic feature sample are obtained by marking the object's facial key point information on the phoneme feature and the acoustic feature of an object's audio.

In combination with any implementation manner provided by the present disclosure, the phoneme feature sample and the acoustic feature sample are obtained in the following manner: acquiring a speaking video of the object; acquiring multiple facial images according to the speaking video, and combining with each At least one audio frame corresponding to the facial image; obtaining phoneme features and acoustic features of at least one audio frame corresponding to each facial image; obtaining facial key point information according to the plurality of facial images, and according to The facial key point information marks the phoneme feature and the acoustic feature to obtain the phoneme feature sample and the acoustic feature sample.

In combination with any of the embodiments provided in the present disclosure, the face completion network is trained by generating an adversarial network, the generation adversarial network includes the face completion network and the first identification network, and the trained network loss includes : The first loss, which is used to indicate the difference between the face completion image output by the face completion network and the complete face image, wherein the complete face image is corresponding to the facial key point information A face image; a second loss, which is used to indicate the difference between the classification result output by the first discrimination network for the input image and the annotation information of the input image, wherein the annotation information indicates that the input image is the The face complement image output by the face complement network may be a real face image.

In combination with any implementation manner provided by the present disclosure, the generation confrontation network further includes a second discrimination network, and the trained network loss further comprises: a third loss, which is used to indicate that the second discrimination network is suitable for the face complement The discrepancy between the discriminative results corresponding to the full image and phoneme features and the true corresponding results.

At least one embodiment of the present disclosure also provides an electronic device, as shown in FIG. 8 , the device includes a memory and a processor, the memory is used to store computer instructions that can be run on the processor, and the processor is used to execute the described The computer instructions implement the method for generating a talking video in any embodiment of the present disclosure.

At least one embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the method for generating a talking video in any embodiment of the present disclosure is implemented.

Those skilled in the art should understand that one or more embodiments of this specification may be provided as a method, system or computer program product. Accordingly, one or more embodiments of the present description may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of the present description may employ a computer program embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein. The form of the product.

Each embodiment in this specification is described in a progressive manner, the same and similar parts of each embodiment can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the data processing device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for relevant parts, please refer to part of the description of the method embodiment.

The foregoing describes specific embodiments of this specification. Other implementations are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. Multitasking and parallel processing are also possible or may be advantageous in certain embodiments.

Embodiments of the subject matter and functional operations described in this specification can be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or in A combination of one or more of . Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, that is, one or more of computer program instructions encoded on a tangible, non-transitory program carrier for execution by or to control the operation of data processing apparatus. Multiple modules. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode and transmit information to a suitable receiver device for transmission by the data The processing means executes. A computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

Computers suitable for the execution of a computer program include, for example, general and/or special purpose microprocessors, or any other type of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory and/or a random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices for storing data, such as magnetic or magneto-optical disks, or optical disks, to receive data therefrom or to It transmits data, or both. However, it is not necessary for a computer to have such a device. In addition, a computer may be embedded in another device such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a device such as a Universal Serial Bus (USB) ) portable storage devices like flash drives, to name a few.

Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (such as EPROM, EEPROM, and flash memory devices), magnetic disks (such as internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions or of what may be claimed, but rather as primarily describing features of particular embodiments of particular inventions. Certain features that are described in this specification in multiple embodiments can also be implemented in combination in a single embodiment. On the other hand, various features that are described in a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may function in certain combinations as described above and even be initially so claimed, one or more features from a claimed combination may in some cases be removed from that combination and the claimed A protected combination can point to a subcombination or a variant of a subcombination.

Similarly, while operations are depicted in the figures in a particular order, this should not be construed as requiring that those operations be performed in the particular order shown, or sequentially, or that all illustrated operations be performed, to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can often be integrated together in a single software product in, or packaged into multiple software products.

Thus, certain embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some implementations, multitasking and parallel processing may be advantageous.

The above descriptions are only preferred embodiments of one or more embodiments of this specification, and are not intended to limit one or more embodiments of this specification. Within the spirit and principles of one or more embodiments of this specification, Any modification, equivalent replacement, improvement, etc. should be included in the scope of protection of one or more embodiments of this specification.

Claims (16)

A method for generating a speaking video, characterized in that the method comprises: Obtain phoneme features and acoustic features of sound-driven data, where the sound-driven data includes at least one of audio and text; Acquiring at least one set of facial key point information of the target object in the first image according to the phoneme feature and the acoustic feature; According to the at least one set of face key point information and the second image containing the face of the target object, at least one target face image corresponding to the sound driving data is obtained, wherein the second image includes The set area of the specific part of the target object is blocked; A speaking video of the target object is obtained according to the sound driving data and the at least one target facial image. The method according to claim 1, wherein said obtaining phoneme features and acoustic features of the sound driving data comprises: Obtaining the phonemes contained in the audio corresponding to the sound driving data and the timestamps corresponding to each phoneme, and obtaining the phoneme characteristics of the sound driving data; Feature extraction is performed on the audio corresponding to the sound driving data to obtain the acoustic features of the sound driving data. The method according to claim 1 or 2, wherein said obtaining at least one set of face key point information of the target object in the first image according to the phoneme feature and the acoustic feature comprises: Obtaining a plurality of sub-phoneme features contained in the phoneme feature and sub-acoustic features corresponding to the plurality of sub-phoneme features; The sub-phoneme features and the corresponding sub-acoustic features are input to the facial key point extraction network to obtain facial key point information corresponding to the sub-phoneme features and the sub-acoustic features. The method according to any one of claims 1 to 3, wherein the facial key point information includes 3-dimensional (3D) facial key point information, and according to the at least one set of facial key point information and The second image containing the face of the target object, before obtaining at least one target face image corresponding to the sound driving data, the method further includes: Projecting the 3D face key point information onto a 2-dimensional (2D) plane to obtain 2D face key point information corresponding to the 3D face key point information; Utilizing the 2D facial key point information to update the facial key point information. The method according to any one of claims 1 to 4, characterized in that, according to the at least one set of face key point information and the second image containing the face of the target object, the sound drive is obtained. Before at least one target facial image corresponding to the data, the method also includes: Filtering is performed on multiple sets of facial key point information, so that the variation between the facial key point information of each image frame and the facial key point information of adjacent frames of the image frame satisfies a set condition. The method according to any one of claims 1 to 5, characterized in that, according to the at least one set of face key point information and the second image containing the face of the target object, the sound At least one target face image corresponding to the driving data, including: Each set of face key point information and the second image are input to the face complement network to obtain a target face image corresponding to the face key point information, wherein the face complement network is used to The key point information of the face complements the blocked set area in the second image. The method according to any one of claims 1 to 6, wherein the obtaining the speaking video of the target object according to the sound driving data and the at least one target facial image comprises: Fusing the at least one target face image with the set background image to obtain a first image sequence; According to the first image sequence and the audio corresponding to the sound driving data, a speaking video of the target object is obtained. The method according to any one of claims 3 to 7, wherein the facial key point extraction network is trained by using phoneme feature samples and corresponding acoustic feature samples, wherein the phoneme feature samples and the acoustic feature samples Feature samples include labeled facial keypoint information. The method according to claim 8, wherein the facial key point extraction network is trained in the following manner: According to the phoneme feature sample and the corresponding acoustic feature sample, the initial facial key point extraction network is trained, and when the change of network loss meets the convergence condition, the training is completed to obtain the facial key point extraction network, wherein the network The loss includes the difference between the facial key point information predicted by the initial neural network and the labeled facial key point information. The method according to claim 8 or 9, wherein the phoneme feature sample and the acoustic feature sample are obtained by marking the object's facial key point information on the phoneme feature and acoustic feature of an object's audio . The method according to claim 10, wherein the phoneme feature samples and the acoustic feature samples are obtained in the following manner: Obtain a speaking video of the subject; Obtaining a plurality of facial images according to the speaking video, and at least one audio frame corresponding to each of the facial images; Obtain phoneme features and acoustic features of at least one audio frame corresponding to each of the facial images; Facial key point information is acquired according to the plurality of facial images, and the phoneme feature and the acoustic feature are marked according to the facial key point information to obtain the phoneme feature sample and the acoustic feature sample. The method according to any one of claims 6 to 11, wherein the face completion network is trained by generating an adversarial network, and the generating adversarial network includes the face completion network and a first identification network , the network loss of the training includes: The first loss is used to indicate the difference between the face completion image output by the face completion network and the complete face image, wherein the complete face image is the face corresponding to the facial key point information internal image; The second loss is used to indicate the difference between the classification result output by the first discrimination network for the input image and the annotation information of the input image, wherein the annotation information indicates that the input image is the face complement The face complement image output by the whole network may be a real face image. The method according to claim 12, wherein the generation confrontation network also includes a second discrimination network, and the network loss of the training also includes: The third loss is used to indicate the difference between the discrimination result of the second discrimination network for the correspondence between the face complement image and the phoneme feature and the real correspondence result. A talking video generation device, characterized in that the device comprises: A first acquisition unit, configured to acquire phoneme features and acoustic features of sound-driven data, where the sound-driven data includes at least one of audio and text; A second acquiring unit, configured to acquire at least one set of facial key point information of the target object in the first image according to the phoneme feature and the acoustic feature; The first obtaining unit is configured to obtain at least one target facial image corresponding to the sound driving data according to the at least one set of facial key point information and the second image containing the target object's face, wherein, A set area including a specific part of the target object in the second image is blocked; The second obtaining unit is configured to obtain the speaking video of the target object according to the sound driving data and the at least one target facial image. An electronic device, characterized in that the device comprises a memory and a processor, the memory is used to store computer instructions executable on the processor, and the processor is used to implement claim 1 when executing the computer instructions to the method described in any one of 13. A computer-readable storage medium on which a computer program is stored, wherein the program implements the method according to any one of claims 1 to 13 when the program is executed by a processor.

Images