CN112383722A - Method and apparatus for generating video - Google Patents
Method and apparatus for generating video Download PDFInfo
- Publication number
- CN112383722A CN112383722A CN202011273138.0A CN202011273138A CN112383722A CN 112383722 A CN112383722 A CN 112383722A CN 202011273138 A CN202011273138 A CN 202011273138A CN 112383722 A CN112383722 A CN 112383722A
- Authority
- CN
- China
- Prior art keywords
- audio
- target person
- face
- key points
- video
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/265—Mixing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/02—Feature extraction for speech recognition; Selection of recognition unit
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Signal Processing (AREA)
- Computational Linguistics (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Acoustics & Sound (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Embodiments of the present disclosure disclose methods and apparatus for generating audio. One embodiment of the method comprises: acquiring acoustic features extracted from audio; determining face key points of a target person aiming at audio according to the acoustic characteristics, wherein the audio does not belong to the audio of the target person; and generating a video of the target person according to the key points of the face. This embodiment enables a convenient conversion from given audio to video.
Description
Technical Field
Embodiments of the present disclosure relate to the field of computer technologies, and in particular, to a method and an apparatus for generating a video.
Background
In recent years, a large number of applications of the stream type have been introduced to provide users with a large amount of information of various types such as text, audio, video, images, etc. through a stream page, and a large number of applications of the education or entertainment interaction type have been additionally included to provide users with various interesting functions. The main characteristic of these applications is that a large amount of contents are generated by users, each user can make and upload various contents such as video and images, and can browse the contents made by other users.
These application platforms are constantly developing and launching services for various functions based on various content production needs of users. Among them, providing functional services such as type conversion between information using multimodal techniques is an application direction in research and development. For example, some application platforms have provided various functional services that can convert speech directly into text, convert text directly into speech, change the gender of dubbing personnel in video, automatically add audio to video, automatically add video to audio, and the like.
Disclosure of Invention
Embodiments of the present disclosure propose methods and apparatuses for generating video.
In a first aspect, an embodiment of the present disclosure provides a method for generating a video, the method including: acquiring acoustic features extracted from audio; determining face key points of a target person aiming at audio according to the acoustic characteristics, wherein the audio does not belong to the audio of the target person; and generating a video of the target person according to the key points of the face.
In a second aspect, an embodiment of the present disclosure provides an apparatus for generating a video, the apparatus including: an acquisition unit configured to acquire acoustic features extracted from audio; a determining unit configured to determine a face key point of a target person for the audio according to the acoustic feature, wherein the audio does not belong to the audio of the target person; and the generating unit is configured to generate the video of the target person according to the key points of the human face.
In a third aspect, an embodiment of the present disclosure provides an electronic device, including: one or more processors; storage means for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the method as described in any implementation of the first aspect.
In a fourth aspect, embodiments of the present disclosure provide a computer-readable medium on which a computer program is stored, which computer program, when executed by a processor, implements the method as described in any of the implementations of the first aspect.
The method and the device for generating the video provided by the embodiment of the disclosure realize the visualization of the audio by acquiring the acoustic features extracted from the audio of a given non-target person, determining the face key points of the target person for the audio according to the acoustic features, and then generating the video of the target person according to the face key points of the target person, namely, a method for generating the video by directly converting the audio is provided, so that a plurality of content producers can conveniently make the video.
Drawings
Other features, objects and advantages of the disclosure will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:
FIG. 1 is an exemplary system architecture diagram in which one embodiment of the present disclosure may be applied;
FIG. 2 is a flow diagram for one embodiment of a method for generating video in accordance with the present disclosure;
FIG. 3 is a flow diagram of yet another embodiment of a method for generating video in accordance with the present disclosure;
FIG. 4 is a schematic diagram of one application scenario of a method for generating video in accordance with an embodiment of the present disclosure;
FIG. 5 is a schematic block diagram illustrating one embodiment of an apparatus for generating video in accordance with the present disclosure;
FIG. 6 is a schematic structural diagram of an electronic device suitable for use in implementing embodiments of the present disclosure.
Detailed Description
The present disclosure is described in further detail below with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the disclosure and are not limiting of the disclosure. It should be further noted that, for the convenience of description, only the portions relevant to the present disclosure are shown in the drawings.
It should be noted that, in the present disclosure, the embodiments and features of the embodiments may be combined with each other without conflict. The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
Fig. 1 shows an exemplary architecture 100 to which embodiments of the method for generating video or the apparatus for generating video of the present disclosure may be applied.
As shown in fig. 1, the system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the terminal devices 101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
The terminal devices 101, 102, 103 interact with a server 105 via a network 104 to receive or send messages or the like. Various client applications may be installed on the terminal devices 101, 102, 103. For example, browser-like applications, search-like applications, social platform software, instant messaging tools, educational-like applications, live-broadcast-like applications, information-flow-like applications, and the like.
The terminal apparatuses 101, 102, and 103 may be hardware or software. When the terminal devices 101, 102, 103 are hardware, they may be various electronic devices including, but not limited to, smart phones, tablet computers, e-book readers, laptop portable computers, desktop computers, and the like. When the terminal apparatuses 101, 102, 103 are software, they can be installed in the electronic apparatuses listed above. It may be implemented as multiple pieces of software or software modules (e.g., multiple pieces of software or software modules to provide distributed services) or as a single piece of software or software module. And is not particularly limited herein.
The server 105 may be a server providing various services, such as a server providing back-end support for client applications installed on the terminal devices 101, 102, 103. The server 105 may acquire audio from the terminal devices 101, 102, and 103, extract acoustic features, generate face key points of the target person for the audio according to the acoustic features, and generate a video of the target person by using the obtained face key points of the target person. Further, the server 105 may also feed the generated video of the target person to the terminal devices 101, 102, 103 for presentation.
Note that the audio may be directly stored locally in the server 105, and the server 105 may directly extract the locally stored audio and extract the acoustic features for processing, in which case, the terminal apparatuses 101, 102, and 103 and the network 104 may not be present.
It should be noted that the method for generating video provided by the embodiment of the present disclosure is generally performed by the server 105, and accordingly, the apparatus for generating video is generally disposed in the server 105.
It should be further noted that the terminal devices 101, 102, 103 may also generate the face key points of the target person for the audio according to the acoustic features, and then generate the video of the target person by using the obtained face key points of the target person. At this time, the method for generating the video may be executed by the terminal apparatuses 101, 102, 103, and accordingly, the apparatus for generating the video may be provided in the terminal apparatuses 101, 102, 103. At this point, the exemplary system architecture 100 may not have the server 105 and the network 104.
The server 105 may be hardware or software. When the server 105 is hardware, it may be implemented as a distributed server cluster composed of a plurality of servers, or may be implemented as a single server. When the server 105 is software, it may be implemented as multiple pieces of software or software modules (e.g., multiple pieces of software or software modules used to provide distributed services), or as a single piece of software or software module. And is not particularly limited herein.
It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
With continued reference to fig. 2, a flow 200 of one embodiment of a method for generating video in accordance with the present disclosure is shown. The method for generating video comprises the following steps:
In the present embodiment, the audio may be audio of arbitrary content. The audio frequency can be preset by technicians according to actual application requirements, and can also be set according to actual application scenes. For example, the audio may be audio input at a terminal device by a user of the terminal device (e.g., terminal devices 101, 102, 103, etc. shown in fig. 1).
The acoustic characteristics of the audio may refer to acoustic information contained by the audio. The acoustic information indicated by the acoustic signature may be preset by a technician according to the actual application requirements. For example, the acoustic features may include fundamental frequency features, formant features, and the like.
For any audio, the existing various methods for extracting acoustic features (such as signal processing methods, deep learning-based acoustic feature extraction methods, and the like) can be used for extracting the acoustic features of the audio.
In this embodiment, the executing subject of the method for generating video (e.g., server 105 shown in fig. 1) may retrieve acoustic features extracted from audio locally or from another storage device.
It should be noted that the process of extracting the acoustic features from the audio may be performed by the execution subject and the acoustic features may be stored locally. At this time, the execution subject may directly acquire the acoustic features from the local. The process of extracting the acoustic features from the audio may also be executed and stored by other electronic devices (such as the terminal devices 101, 102, 103 shown in fig. 1 or other servers, etc.), and at this time, the executing body may obtain the acoustic features from the other electronic devices.
In this embodiment, the target person may be a person specified in advance by a technician or a user according to actual application requirements. The audio may not belong to the audio of the targeted person, i.e. the audio is not generated by the targeted person reading the corresponding content.
The acoustic characteristics of the targeted person for the audio may indicate acoustic characteristics of the audio that the targeted person generated when reading the content of the audio. The face keypoints of the target person for the audio may indicate keypoints of the target person's face when reading the content of the audio.
Since the face of the target person and the generated audio usually have different changes when the target person reads different audio contents, the acoustic features of the target person for the audio may be different for different audios, and the face key points of the target person for the audio may also be different. The face key points can be represented by using the position coordinates of the key points on the face.
After obtaining the acoustic features extracted from the audio, various methods may be employed to determine the face key points of the target person for the corresponding audio according to the extracted acoustic features. For example, a large amount of videos of the recorded target person (such as speaking or singing) may be collected in advance, and then the presented key points of the face of the target person may be extracted from the recorded videos by using various existing key point extraction methods, and simultaneously, the acoustic features of the speaking or singing data of the target person may be extracted. Then, the mapping relation between the acoustic features and key points of the face of the target person can be counted based on methods such as curve fitting, so that the key points of the face of the target person corresponding to the acoustic features can be determined according to the acoustic features of any audio.
And step 203, generating a video of the target person according to the key points of the face.
In this embodiment, the video of the target person may refer to a video in which the face of the target person is presented. After the face key points of the target person for the audio are obtained, the video of the target person can be generated by utilizing various existing image processing methods and video processing methods in combination with actual application requirements.
For example, an image of the target person may be obtained from a local or connected other device, then the key points of the face of the target person displayed in the image are determined, and then the key points are adjusted to the determined key points of the face, so as to obtain an adjusted image of the target person. Then, the adjusted image of the target person can be used to create a video of the target person.
In some optional implementations of the embodiment, after obtaining the acoustic features of the target person for the audio, the audio may be added to the generated video of the target person, so that the video with the audio of other persons and presented with the target person may be obtained. Specifically, audio may be added to the generated video of the target person using various existing methods for merging audio and video.
Based on this, for many video producers, the audio of themselves or the favorite audio of themselves can be added to the generated video of the specified person, enabling more flexible video creation.
Optionally, the audio may include at least one of: speech data, singing data. Wherein the singing data may refer to data generated for various forms of singing. Therefore, the data type of the audio which can be processed can be enriched, and the flexibility of conversion from the audio to the video is further improved.
In some optional implementation manners of this embodiment, after the acoustic features extracted from the audio are acquired, the key point determination model corresponding to the target person, trained in advance, may be used to determine the face key points of the target person for the audio according to the acquired acoustic features.
The key point determination model corresponding to the target person can represent the corresponding relation between the acoustic features of the audio of other persons and the face key points of the target person aiming at the audio. The key point determination model corresponding to the target person can be obtained by utilizing pre-collected data training of the target person.
As an example, a large number of audios and videos (i.e., including audio and video) in which a target person speaks or sings may be pre-recorded, then key points of the face of the target person to be presented may be extracted from the recorded video, acoustic features of the target person may be extracted from the recorded audio, and the extracted key points and corresponding acoustic features may be used as training data.
Then, various types of untrained or trained artificial neural networks can be obtained to serve as initial key point determination models, acoustic features in training data serve as input of the initial key point determination models, key points corresponding to the input acoustic features serve as expected output of the initial key point determination models, parameters of the initial key point determination models are continuously adjusted according to values of loss functions by using algorithms such as gradient descent, back propagation and the like until preset training stop conditions are met (for example, values of the loss functions meet certain conditions and the like), and at the moment, the initial key point determination models obtained through training can serve as the key point determination models corresponding to the target people.
End-to-end conversion from the acoustic features of any audio to the key points of the face of the target person can be realized by utilizing the key point determination model, so that the complexity and the speed of the multi-mode conversion process can be reduced, and the efficiency of making the video of the target person is improved.
In some optional implementation manners of this embodiment, the face key points of the target person determined by using the key point determination model corresponding to the target person may specifically include at least one group of face key points, and each group of face key points may represent one frame of face image, that is, each group of face key points may be used to generate one frame of face image of the target person.
Each group of face key points may include a target number of key points to respectively represent different face parts. In particular, the target number may be set by a technician according to the actual application requirements.
At this time, after obtaining at least one group of face key points of the target person, face images corresponding to the respective groups of face key points may be generated to obtain a face image set, and then the face image set is used to generate a video of the target person.
For example, an image of the target person is obtained in advance, and then, for each group of face key points, the face image corresponding to the group of face key points is obtained by adjusting the key points of the face displayed in the image of the target person to be the group of face key points. And then, the obtained face images are used for making a video of the target person.
The multi-frame face image of the target person is generated by taking the frame as a unit, and then the video of the target person is made by utilizing the multi-frame face image, so that the flexibility and the naturalness of video making can be further improved, and the smoothness of the video is also favorably ensured.
In some optional implementations of the present embodiment, the keypoint determination model may include a feature conversion model and a prediction model. The feature conversion model may be used to convert the acoustic features into corresponding text features, that is, to implement conversion from the acoustic features of any audio to the text features of the text content corresponding to the audio. The prediction model can be used for determining the face key points of the target person according to the text features, namely, the conversion from the text features corresponding to any audio to the face key points of the target person is realized.
At this time, after the acoustic features extracted from the audio of other people are acquired, the acoustic features may be input to the feature conversion model, so as to obtain text features corresponding to the acoustic features. And then inputting the obtained text features into a prediction model to obtain the face key points of the target person aiming at the audio of other persons.
The specific character indicated by the text character can be set by a technician according to the actual application requirement and application scenario. For example, the text features of the text may include prosodic features, phonetic features, tonal features, pitch features, and the like.
Alternatively, the feature transformation model may be a model constructed based on posterior probability of speech (PPG). At this time, the feature transformation model may be constructed based on a network model structure that implements voice transformation based on the existing voice posterior probability.
The feature conversion model and the prediction model included in the key point determination model can be obtained by joint training of pre-collected data of the target person. For example, a large amount of audio and video (i.e., including audio and video) of a target person speaking or singing is pre-recorded, then key points of the face of the target person are extracted from the recorded video, acoustic features of the target person are extracted from the recorded audio, and the extracted key points and the corresponding acoustic features are used as training data.
Then, the acoustic features of the target person can be used as the input of the feature conversion model, the output result of the feature conversion model is input into the prediction model, the face key points of the target person are used for monitoring the output of the prediction model, and the network parameters of the feature conversion model and the prediction model are continuously adjusted until the preset training stop condition is reached.
Or, in the training process, the acoustic features of the target person may be used as the input of the feature conversion model, then the text features corresponding to the input acoustic features are obtained as the expected output of the feature conversion model, the training of the feature conversion model is completed first, then the network parameters of the feature conversion model are fixed, and then the training of the prediction model is completed.
The key point determination model of the target person is composed of a feature conversion model for realizing conversion from acoustic features to text features and a prediction model for realizing conversion from the text features to human face key points of the target person, so that audio can be more accurately converted into the audio of the target person according to the text features, the accuracy of an output result of the key point determination model of the target person is improved, and the quality of the generated video of the target person is improved.
The method provided by the above embodiment of the present disclosure proposes a method for generating a video from audio direct conversion. Specifically, for any given audio, according to the acoustic features of the audio, face key points of target persons other than the person to which the audio belongs for the audio can be generated, then videos of the target persons are generated by using the face key points, and then the videos of the target persons and the given audio are combined, so that videos which have the audio of other persons and are presented with the target persons can be obtained.
With further reference to fig. 3, a flow 300 of yet another embodiment of a method for generating a video is shown. The flow 300 of the method for generating a video comprises the steps of:
And 302, determining at least one group of face key points of the target person aiming at the audio according to the acoustic characteristics by utilizing a pre-trained key point determination model corresponding to the target person.
In this embodiment, the image generation model may represent a correspondence between a group of face key points and a frame of face image. The image generation model corresponding to the target person can be obtained by utilizing the pre-collected data training of the target person.
As an example, training data may be obtained first, and then an image generation model corresponding to the target person may be obtained through training using the training data. Specifically, a large number of videos of a target person during speaking and/or singing can be prerecorded, then each frame of image is extracted from the videos, and a face key point group corresponding to each frame of image is determined. Then, the extracted frame images and the corresponding key point groups can be used as training data.
Then, various untrained or trained artificial neural networks can be obtained as initial image generation models, the key point groups in the training data are used as input of the initial image generation models, the images corresponding to the input key point groups are used as expected output of the initial image generation models, parameters of the initial image generation models are continuously adjusted according to the values of the loss functions by using algorithms such as gradient descent, back propagation and the like until preset training stop conditions are reached (for example, the values of the loss functions meet certain conditions and the like), and at the moment, the initial image generation models obtained through training can be used as the image generation models corresponding to the target people.
And 304, generating a video of the target person by using the face images respectively corresponding to the face key points of each group.
The specific execution process of the content not described in detail in the steps 301-305 can refer to the related description in the corresponding embodiment of fig. 2, and will not be described herein again.
With continued reference to fig. 4, fig. 4 is an illustrative application scenario 400 of the method for generating video according to the present embodiment. In the application scenario of fig. 4, the user may specify a broadcast play and a target person in the terminal 401 that he uses. For example, the user may select his favorite radio play that wants to be videoed, while specifying the target person whose sound is deemed to conform to the story of the radio play.
The acoustic features 402 of the series may then be input into a key point determination module 403 of the target person, resulting in a number of face key point groups 404 for the series for the target person. Next, an image 406 of the target person corresponding to each face key point group is generated by using the image generation model 405, and a video 407 of the target person is created from the obtained images of the plurality of target persons.
Thereafter, the video 407 of the target person and the selected play may be merged to obtain a videoed play 408, and the videoed play 408 may be transmitted to the terminal 401. A user may upload a videotaped series 408 made by the user in his favorite application platform to be shared with other users in the application platform.
The process of generating the face image of the target person according to the face key points of the target person by using the pre-trained image generation model corresponding to the target person is highlighted by the flow of the method for generating the video provided by the embodiment of the disclosure, that is, the end-to-end face image generation is realized.
With further reference to fig. 5, as an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of an apparatus for generating a video, which corresponds to the method embodiment shown in fig. 2, and which is particularly applicable in various electronic devices.
As shown in fig. 5, the apparatus 500 for generating video provided by the present embodiment includes an acquisition unit 501, a determination unit 502, and a generation unit 503. Wherein the obtaining unit 501 is configured to obtain acoustic features extracted from audio; the determining unit 502 is configured to determine face key points of the target person for audio according to the acoustic features, wherein the audio does not belong to the audio of the target person; the generating unit 503 is configured to generate a video of the target person based on the face key points.
In the present embodiment, in the apparatus 500 for generating a video: the specific processing of the obtaining unit 501, the determining unit 502, and the generating unit 503 and the technical effects thereof can refer to the related descriptions of step 201, step 202, and step 203 in the corresponding embodiment of fig. 2, which are not repeated herein.
The apparatus provided by the above embodiment of the present disclosure acquires, by an acquisition unit, acoustic features extracted from an audio; the determining unit determines face key points of a target person aiming at audio according to the acoustic features, wherein the audio does not belong to the audio of the target person; the generating unit generates the video of the target person according to the key points of the face, so that the video of the audio is realized.
Referring now to FIG. 6, a schematic diagram of an electronic device (e.g., the server of FIG. 1) 600 suitable for use in implementing embodiments of the present disclosure is shown. The server shown in fig. 6 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 6, electronic device 600 may include a processing means (e.g., central processing unit, graphics processor, etc.) 601 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)602 or a program loaded from a storage means 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data necessary for the operation of the electronic apparatus 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
Generally, the following devices may be connected to the I/O interface 605: input devices 606 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 607 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 608 including, for example, tape, hard disk, etc.; and a communication device 609. The communication means 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. While fig. 6 illustrates an electronic device 600 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided. Each block shown in fig. 6 may represent one device or may represent multiple devices as desired.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication means 609, or may be installed from the storage means 608, or may be installed from the ROM 602. The computer program, when executed by the processing device 601, performs the above-described functions defined in the methods of embodiments of the present disclosure.
It should be noted that the computer readable medium described in the embodiments of the present disclosure may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In embodiments of the disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In embodiments of the present disclosure, however, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for embodiments of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In accordance with one or more embodiments of the present disclosure, there is provided a method for generating a video, the method comprising: acquiring acoustic features extracted from audio; determining face key points of a target person aiming at audio according to the acoustic characteristics, wherein the audio does not belong to the audio of the target person; and generating a video of the target person according to the key points of the face.
In accordance with one or more embodiments of the present disclosure, the method further comprises: and adding the audio which does not belong to the target person to the video.
According to one or more embodiments of the present disclosure, determining a face key point of a target person for audio according to an acoustic feature includes: and inputting the acoustic features into a pre-trained key point determination model corresponding to the target person to obtain the face key points of the target person aiming at the audio.
According to one or more embodiments of the present disclosure, the key point determination model includes a feature conversion model and a prediction model, wherein the feature conversion model is used for converting acoustic features into corresponding text features, and the prediction model is used for determining the face key points of the target person according to the text features.
According to one or more embodiments of the present disclosure, the face key points include at least one group of face key points, and each group of face key points is used for representing one frame of face image.
According to one or more embodiments of the present disclosure, generating a video of a target person according to face key points includes: generating a face image corresponding to each group of face key points in at least one group of face key points to obtain a face image set; and generating a video of the target person by using the face image set.
According to one or more embodiments of the present disclosure, generating a face image corresponding to each group of face key points in at least one group of face key points includes: and for each group of face key points in at least one group of face key points, generating a face image corresponding to the group of face key points according to the group of face key points by utilizing a pre-trained image generation model corresponding to the target person.
According to one or more embodiments of the present disclosure, the feature conversion model is a model constructed based on a speech posterior probability.
In accordance with one or more embodiments of the present disclosure, the audio includes at least one of: speech data, singing data.
In accordance with one or more embodiments of the present disclosure, there is provided an apparatus for generating a video, the apparatus including: an acquisition unit configured to acquire acoustic features extracted from audio; the determining unit is configured to determine a face key point of the target person aiming at the audio according to the acoustic features, wherein the audio does not belong to the audio of the target person; and the generating unit is configured to generate the video of the target person according to the key points of the human face.
According to one or more embodiments of the present disclosure, the apparatus further comprises: an adding unit configured to add audio for the video.
According to one or more embodiments of the present disclosure, the determining unit is further configured to: and inputting the acoustic features into a pre-trained key point determination model corresponding to the target person to obtain the face key points of the target person aiming at the audio.
According to one or more embodiments of the present disclosure, the key point determination model includes a feature conversion model and a prediction model, wherein the feature conversion model is used for converting acoustic features into corresponding text features, and the prediction model is used for determining the face key points of the target person according to the text features.
According to one or more embodiments of the present disclosure, the face key points include at least one group of face key points, and each group of face key points is used for representing one frame of face image.
According to one or more embodiments of the present disclosure, the generating unit is further configured to: generating a face image corresponding to each group of face key points in at least one group of face key points to obtain a face image set; and generating a video of the target person by using the face image set.
According to one or more embodiments of the present disclosure, the generating unit is further configured to: and for each group of face key points in at least one group of face key points, generating a face image corresponding to the group of face key points according to the group of face key points by utilizing a pre-trained image generation model corresponding to the target person.
According to one or more embodiments of the present disclosure, the feature conversion model is a model constructed based on a speech posterior probability.
In accordance with one or more embodiments of the present disclosure, the audio includes at least one of: speech data, singing data.
The units described in the embodiments of the present disclosure may be implemented by software or hardware. The described units may also be provided in a processor, and may be described as: a processor includes an acquisition unit, a determination unit, and a generation unit. Where the names of the units do not in some cases constitute a limitation of the unit itself, for example, the acquiring unit may also be described as a "unit that acquires acoustic features extracted from the audio".
As another aspect, the present disclosure also provides a computer-readable medium. The computer readable medium may be embodied in the electronic device described above; or may exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: acquiring acoustic features extracted from audio; determining face key points of a target person aiming at audio according to the acoustic characteristics, wherein the audio does not belong to the audio of the target person; and generating a video of the target person according to the key points of the face.
The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the disclosure in the embodiments of the present disclosure is not limited to the particular combination of the above-described features, but also encompasses other embodiments in which any combination of the above-described features or their equivalents is possible without departing from the scope of the present disclosure. For example, the above features and (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure are mutually replaced to form the technical solution.
Claims (12)
1. A method for generating video, comprising:
acquiring acoustic features extracted from audio;
determining a face key point of a target person aiming at the audio according to the acoustic features, wherein the audio does not belong to the audio of the target person;
and generating the video of the target person according to the face key points.
2. The method of claim 1, wherein the method further comprises:
and adding the audio which does not belong to the target person to the video.
3. The method of claim 1, wherein the determining, from the acoustic features, face keypoints for the audio by the target person comprises:
and inputting the acoustic features into a pre-trained key point determination model corresponding to the target person to obtain the face key points of the target person aiming at the audio.
4. The method of claim 3, wherein the keypoint determination model comprises a feature conversion model for converting acoustic features into corresponding text features and a prediction model for determining face keypoints of the target person from text features.
5. The method of any of claims 1-4, wherein the face keypoints comprise at least one group of face keypoints, and each group of face keypoints is used for representing a frame of face image.
6. The method of claim 5, wherein the generating a video of the target person from the face keypoints comprises:
generating face images respectively corresponding to each group of face key points in the at least one group of face key points to obtain a face image set;
and generating the video of the target person by using the face image set.
7. The method of claim 6, wherein the generating of the face images corresponding to the sets of face key points in the at least one set of face key points comprises:
and for each group of face key points in the at least one group of face key points, generating a face image corresponding to the group of face key points according to the group of face key points by utilizing a pre-trained image generation model corresponding to the target person.
8. The method of claim 4, wherein the feature transformation model is a model constructed based on a speech posterior probability.
9. The method of claim 1, wherein the audio comprises at least one of: speech data, singing data.
10. An apparatus for generating video, wherein the apparatus comprises:
an acquisition unit configured to acquire acoustic features extracted from audio;
a determining unit configured to determine a face key point of a target person for the audio according to the acoustic feature, wherein the audio does not belong to the audio of the target person;
and the generating unit is configured to generate the video of the target person according to the face key points.
11. An electronic device, comprising:
one or more processors;
a storage device having one or more programs stored thereon;
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-9.
12. A computer-readable medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011273138.0A CN112383722B (en) | 2020-11-13 | 2020-11-13 | Method and apparatus for generating video |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011273138.0A CN112383722B (en) | 2020-11-13 | 2020-11-13 | Method and apparatus for generating video |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112383722A true CN112383722A (en) | 2021-02-19 |
| CN112383722B CN112383722B (en) | 2023-04-07 |
Family
ID=74583983
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011273138.0A Active CN112383722B (en) | 2020-11-13 | 2020-11-13 | Method and apparatus for generating video |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112383722B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109377539A (en) * | 2018-11-06 | 2019-02-22 | 北京百度网讯科技有限公司 | Method and apparatus for generating animation |
| CN110688911A (en) * | 2019-09-05 | 2020-01-14 | 深圳追一科技有限公司 | Video processing method, device, system, terminal equipment and storage medium |
| US10645520B1 (en) * | 2019-06-24 | 2020-05-05 | Facebook Technologies, Llc | Audio system for artificial reality environment |
| CN111415677A (en) * | 2020-03-16 | 2020-07-14 | 北京字节跳动网络技术有限公司 | Method, apparatus, device and medium for generating video |
| CN111432233A (en) * | 2020-03-20 | 2020-07-17 | 北京字节跳动网络技术有限公司 | Method, apparatus, device and medium for generating video |
-
2020
- 2020-11-13 CN CN202011273138.0A patent/CN112383722B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109377539A (en) * | 2018-11-06 | 2019-02-22 | 北京百度网讯科技有限公司 | Method and apparatus for generating animation |
| US10645520B1 (en) * | 2019-06-24 | 2020-05-05 | Facebook Technologies, Llc | Audio system for artificial reality environment |
| CN110688911A (en) * | 2019-09-05 | 2020-01-14 | 深圳追一科技有限公司 | Video processing method, device, system, terminal equipment and storage medium |
| CN111415677A (en) * | 2020-03-16 | 2020-07-14 | 北京字节跳动网络技术有限公司 | Method, apparatus, device and medium for generating video |
| CN111432233A (en) * | 2020-03-20 | 2020-07-17 | 北京字节跳动网络技术有限公司 | Method, apparatus, device and medium for generating video |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112383722B (en) | 2023-04-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109377539B (en) | Method and apparatus for generating animation | |
| US11158102B2 (en) | Method and apparatus for processing information | |
| CN111599343B (en) | Method, apparatus, device and medium for generating audio | |
| US11151765B2 (en) | Method and apparatus for generating information | |
| CN111402842B (en) | Method, apparatus, device and medium for generating audio | |
| EP3889912B1 (en) | Method and apparatus for generating video | |
| US11308671B2 (en) | Method and apparatus for controlling mouth shape changes of three-dimensional virtual portrait | |
| JP6971292B2 (en) | Methods, devices, servers, computer-readable storage media and computer programs for aligning paragraphs and images | |
| CN107481715B (en) | Method and apparatus for generating information | |
| CN111899720A (en) | Method, apparatus, device and medium for generating audio | |
| CN111798821B (en) | Sound conversion method, device, readable storage medium and electronic equipment | |
| CN107705782B (en) | Method and device for determining phoneme pronunciation duration | |
| CN110880198A (en) | Animation generation method and device | |
| CN113205793B (en) | Audio generation method and device, storage medium and electronic equipment | |
| CN110472558B (en) | Image processing method and device | |
| JP2023552854A (en) | Human-computer interaction methods, devices, systems, electronic devices, computer-readable media and programs | |
| CN115691544A (en) | Training of virtual image mouth shape driving model and driving method, device and equipment thereof | |
| CN112364144A (en) | Interaction method, device, equipment and computer readable medium | |
| CN113257218B (en) | Speech synthesis method, device, electronic equipment and storage medium | |
| CN112381926A (en) | Method and apparatus for generating video | |
| CN113923390A (en) | Video recording method, device, equipment and storage medium | |
| CN112383721B (en) | Method, apparatus, device and medium for generating video | |
| WO2023061229A1 (en) | Video generation method and device | |
| CN111862933A (en) | Method, apparatus, device and medium for generating synthesized speech | |
| CN112383722B (en) | Method and apparatus for generating video |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |