WO2026001087A1 - 视频生成方法、装置、设备及介质 - Google Patents

视频生成方法、装置、设备及介质

Info

Publication number
WO2026001087A1
WO2026001087A1 PCT/CN2025/082423 CN2025082423W WO2026001087A1 WO 2026001087 A1 WO2026001087 A1 WO 2026001087A1 CN 2025082423 W CN2025082423 W CN 2025082423W WO 2026001087 A1 WO2026001087 A1 WO 2026001087A1
Authority
WO
WIPO (PCT)
Prior art keywords
target
prompt
network
networks
target object
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/CN2025/082423
Other languages
English (en)
French (fr)
Inventor
邓启力
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Zitiao Network Technology Co Ltd
Original Assignee
Beijing Zitiao Network Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Zitiao Network Technology Co Ltd filed Critical Beijing Zitiao Network Technology Co Ltd
Publication of WO2026001087A1 publication Critical patent/WO2026001087A1/zh
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/472—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/488—Data services, e.g. news ticker
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof

Definitions

  • This disclosure relates to the field of multimedia processing technology, and in particular to a video generation method, apparatus, device, and medium.
  • This disclosure provides a video generation method, apparatus, device, and medium.
  • This disclosure provides a video generation method, the method comprising: acquiring a target prompt image and at least two prompt texts; wherein the target prompt image includes a first target object, and the at least two prompt texts are used to describe different object actions; based on the target prompt image and the at least two prompt texts, generating a target video containing a second target object using a target network model; wherein the second target object has at least some of the external features of the first target object, and the target video presents a dynamic scene of the second target object performing the different object actions.
  • the target network model includes multiple generator networks, and the multiple generator networks are arranged in parallel; the target prompt image is used as the input of each generator network, and the at least two prompt texts are used as the input of at least two generator networks, with different prompt texts corresponding to different generator networks.
  • the step of generating a target video containing a second target object using a target network model based on the target prompt image and the at least two prompt texts includes: obtaining object text features and prompt text features corresponding to each of the at least two prompt texts based on the target prompt image and the at least two prompt texts; inputting the object text features into each of the generation networks, and inputting the prompt text features corresponding to each of the at least two prompt texts into different generation networks; generating output images through the multiple generation networks based on their respective input information; wherein the output images contain the second target object, and the action form of the second target object presented in different output images is different; and obtaining a target video containing the second target object based on the parallel arrangement order of the multiple generation networks and the output images of the multiple generation networks.
  • the target network model further includes a conditional injection network and at least two text encoders; the step of obtaining object text features and corresponding prompt text features of the at least two prompt texts based on the target prompt image and the at least two prompt texts includes: using an object recognition model to obtain a first target object feature in the target prompt image, and mapping the first target object feature to object text features through the conditional injection network; using the at least two text encoders to encode the at least two prompt texts respectively to obtain corresponding prompt text features of the at least two prompt texts.
  • generating output images based on the input information of the multiple generator networks includes: generating output images using a temporal attention mechanism based on the corresponding input information, target association information, and parallel arrangement order of the multiple generator networks; wherein the target association information includes target features output by the same specified network layer corresponding to each of the multiple generator networks.
  • the plurality of generative networks are divided into a first type of network and a second type of network.
  • the input of the first type of network includes the prompt text
  • the second type of network is the network other than the first type of network among the plurality of generative networks.
  • the second target object in the output image corresponding to the first type of network has a first action form, which corresponds to the object action described by the prompt text of the first type of network.
  • the second target object in the output image corresponding to the second type of network has a second action form.
  • the second action form corresponding to the second type of network located between two adjacent first type of networks is a gradient form used to connect the first action forms corresponding to the two adjacent first type of networks.
  • the multiple generator networks have the same structure and share parameters; the input of each generator network also includes a noise map, and the noise maps of different generator networks are different.
  • the method further includes: performing special effects processing on the target video to obtain a special effects video.
  • This disclosure also provides a video generation apparatus, comprising: an acquisition module for acquiring a target prompt image and at least two prompt texts; wherein the target prompt image includes a first target object, and the at least two prompt texts describe different object actions; and a video generation module for generating a target video containing a second target object based on the target prompt image and the at least two prompt texts using a target network model; wherein the second target object has at least some of the external features of the first target object, and the target video presents a dynamic scene of the second target object performing the different object actions.
  • This disclosure also provides an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the video generation method provided in this disclosure.
  • This disclosure also provides a computer-readable storage medium storing a computer program for performing the video generation method provided in this disclosure.
  • Figure 1 is a flowchart illustrating a video generation method provided in an embodiment of this disclosure
  • Figure 2 is a schematic diagram of the structure of a target network model provided in an embodiment of this disclosure
  • Figure 3 is a schematic diagram of the structure of a target network model provided in an embodiment of this disclosure.
  • Figure 4 is a schematic diagram of the structure of a generative network provided in an embodiment of this disclosure.
  • Figure 5 is a schematic diagram illustrating the principle of a temporal attention mechanism provided in an embodiment of this disclosure
  • Figure 6 is a schematic diagram of a video generation device provided in an embodiment of this disclosure.
  • Figure 7 is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure.
  • FIG. 1 is a flowchart illustrating a video generation method according to an embodiment of this disclosure. This method can be executed by a video generation device, which can be implemented using software and/or hardware and is generally integrated into an electronic device. As shown in Figure 1, the method mainly includes the following steps S102 to S104:
  • Step S102 Obtain a target prompt image and at least two prompt texts; wherein the target prompt image includes a first target object, and the at least two prompt texts are used to describe different object actions.
  • the first target object can be a person, animal, vehicle, robot, etc.
  • the object's actions described in the prompt text can be opening its eyes, closing its eyes, looking up, looking down, walking, running, etc.
  • Step S104 Based on the target cue image and at least two cue texts, a target video containing a second target object is generated using a target network model; wherein the second target object has at least some of the external features of the first target object, and the target video presents dynamic scenes of the second target object performing different object actions. For example, assuming the two input cue texts are used to describe the actions of opening and closing eyes, the target video can present dynamic scenes of the second target object from opening to closing its eyes.
  • the target network model is a generative model.
  • the target network model in this disclosure can be trained using a basic diffusion model and a LoRA (Low-Rank Adaptation) model.
  • This target network model can generate object images with a specific style. Taking a person as an example, the target network model can generate images containing virtual character images with a specific style, such as a cartoon virtual character or a virtual character in ancient costume with the external features of the first target object.
  • the parameters of the basic diffusion model can be adjusted based on LoRA technology.
  • LoRA technology is a lightweight model fine-tuning technique that can assist model training by inserting some network layers (which can be called LoRA layers) into the basic diffusion model.
  • the embodiments of this disclosure can also add specific functional modules, such as a temporal attention module, to the original network structure of the generative model.
  • a temporal attention module such as a temporal attention module
  • the target network model can generate multi-frame images with the action gradient effect of the second target object based on the temporal attention mechanism, thereby forming a target video.
  • the model is basically only provided with one prompt text, while the embodiments of this disclosure will obtain at least two prompt texts, which helps the model to controllably and reliably generate object animation videos that present different object actions based on at least two prompt texts.
  • the technical solution provided in this disclosure can efficiently and conveniently generate a target video containing a second target object based on a target prompt image (including a first target object) and at least two prompt texts (used to describe different object actions) using a target network model, without requiring professional animation video production. This significantly reduces the cost and efficiency of animation video generation. Furthermore, under the constraint of the target prompt image, the second target object in the target video possesses at least some of the external features of the first target object, and under the constraint of the at least two prompt texts, the second target object can perform different object actions.
  • This method can achieve the effect of generating personalized videos for the first target object, and the constraints of the target prompt image and at least two prompt texts also make the video generation process highly controllable, generating an animated video that presents a second target object with similar external features to the first target object. This is also highly entertaining and engaging for users, effectively meeting their needs.
  • the target network model comprises multiple generator networks arranged in parallel.
  • the target cue image serves as input to each generator network, and at least two cue texts serve as input to at least two generator networks. Different cue texts correspond to different generator networks.
  • the multiple generator networks have identical structures and share parameters, facilitating training and information exchange, and allowing for better control over the matching degree of the output image.
  • Each generator network's input also includes a noise map. The noise maps for different generator networks are different, and the noise maps for each generator network can be randomly generated, which can be achieved using relevant techniques, and will not be elaborated upon here.
  • FIG. 2 shows a schematic diagram of a target network model, illustrating parallel generator networks 1, 2, ..., 10 to N.
  • Each generator network has an output image, and all output images arranged in sequence constitute the target video.
  • the input of each generator network includes the target prompt image and the corresponding noise image, but only some generator networks have prompt text in their input.
  • generator network 2 does not have corresponding prompt text, which can be considered as its prompt text input being empty or default.
  • Figure 2 is only for illustrative purposes. In practical applications, corresponding prompt text can be set for each generator network.
  • generator networks whose inputs include prompt text can be flexibly specified according to requirements, without any restrictions here.
  • step S104 which is the step of generating a target video containing a second target object using a target network model based on the target prompt image and at least two prompt texts, can be performed with reference to steps A to C below:
  • Step A involves obtaining the object text features and the corresponding text features for each of the at least two prompt texts, based on the target prompt image and at least two prompt texts.
  • the target network model also includes a conditional injection network and at least two text encoders. Step A can be performed as follows: Steps A1 to A2
  • Step A1 involves using an object recognition model to obtain the features of the first target object in the target prompt image, and then mapping these features to object text features via a conditional injection network, injecting the object text features into the generation network.
  • This embodiment does not limit the structure of the conditional injection network; it may include functional units such as feature mapping units.
  • the conditional injection network can not only further extract features from the received data but also perform processing such as feature mapping, mapping the object features (i.e., the first target object features) to prompt text features, and injecting these prompt text features as conditional information into the generation process of the target network. This can guide image generation to a certain extent, ensuring that the second target object contained in the output image has similar external features to the first target object.
  • Step A2 Encode at least two prompt texts using at least two text encoders to obtain the prompt text features corresponding to each of the at least two prompt texts.
  • Step B involves inputting the object text features into each generator network, and inputting the prompt text features corresponding to at least two prompt texts into different generator networks; generating output images through multiple generator networks based on their respective input information; wherein, the output images contain the second target object, and the action form of the second target object presented in different output images is different.
  • the target network model provided in this disclosure has two significant differences from existing conventional video generation models or text-to-image models: (1) It is not limited to generating images or videos using only text, but introduces target prompt images on the basis of prompt text, thereby injecting the features of the first target object into the target network model, which helps to achieve personalized customization effects for the first target object. (2) Existing technologies basically only provide one prompt text for the generation model. Even if the generation model used has multiple generation networks, the multiple generation networks share the same prompt text. However, this disclosure can obtain at least two prompt texts for describing the actions of different objects, and can be input into different generation networks respectively. Some generation networks may not have prompt text, but learn the content to be generated based on the feature information of other networks and according to the temporal attention mechanism. The above methods can effectively ensure the accuracy and controllability of the object actions presented in the generated animation video.
  • Figure 3 shows a schematic diagram of a target network model.
  • Figure 3 further illustrates the object recognition model, the conditional injection network, and the text encoder.
  • the conditional injection network may include an adapter unit.
  • the conditional injection network maps the received features of the first target object to the corresponding feature space.
  • the mapped features i.e., object text features
  • This portion of the network could be, for example, the cross-attention unit of the denoising network (e.g., Unet) in the generator network, or a LoRA layer added to the generator network.
  • conditional injection network can inject conditional information such as the features of the first target object into the image generation process, so that the second target object contained in the output image of the generator network can reflect at least some of the external features of the first target object to a certain extent.
  • conditional information such as the features of the first target object into the image generation process
  • the second target object contained in the output image of the generator network can reflect at least some of the external features of the first target object to a certain extent.
  • multiple generator networks generate their respective output images based on their corresponding input information, target association information, and parallel arrangement order, using a temporal attention mechanism.
  • the target association information includes target features output by the same specified network layer corresponding to each of the multiple generator networks.
  • the same specified network layer corresponding to each of the multiple generator networks can refer to a specific network layer of the same Unet unit of the multiple generator networks.
  • Features at each position in the feature information output by the specified network layer or features at a specified position can be used as target features.
  • FIG. 4 a schematic diagram of a generative network structure can be referenced in Figure 4.
  • the diagram illustrates that the generative network includes an encoder, a denoising network, and a decoder.
  • the encoder's input is a noise map
  • the decoder's output is an image. It also shows that features output from the conditional injection network can be injected into each Unet unit.
  • the input to the generative network contains prompt text
  • the prompt text features will also be input to each Unet unit, although this is not shown in Figure 4.
  • Figure 4 uses a Unet network as an example for the denoising network, comprising multiple Unet units (Unet Blocks). The difference from a conventional Unet network lies in the insertion of a temporal attention module.
  • This module primarily processes information based on a temporal attention mechanism. Specifically, it utilizes this mechanism to achieve information interaction between N generative networks.
  • Each generative network can determine its own required features based on the feature information obtained from other generative networks, thereby ensuring that the output images corresponding to the N generative networks can sequentially achieve the animation effect of the gradual change in the action of the second target object, such as a dynamic effect of opening the eyes, gradually closing the upper and lower eyelids, and finally closing the eyes completely.
  • temporal attention modules at the same position can perform temporal attention processing based on their respective target association information and the order of the target networks.
  • target association information 1 includes the feature information output by the same specified network layer of the first module (the first Unet unit and/or the first temporal attention module) of the N generative networks.
  • the positions of different temporal attention modules are different, and the corresponding target association information is different.
  • temporal attention modules can be implemented using relevant technologies, such as the Animate diff module, which can be integrated into models such as generative networks and learn motion priors from video datasets, thereby extending the text-based image model into a video generation model.
  • the temporal attention mechanism allows the generative network in the target network model to adaptively focus on temporal information when processing N sequence data, capturing dependencies and contextual information in the sequence data, and assigning different weights to features with the same position in the N sequence data.
  • the temporal attention mechanism can obtain the target features output by the same specified network layer of each of the N generative networks.
  • the features with the same position in the output features of the specified network layer of the i-th Unet unit of each of the N generative networks can be assigned corresponding weights and weighted fusion processing can be performed.
  • the output features of the specified network layer of the i-th Unet unit are used as an example.
  • Figure 5 uses gray grids to illustrate the features with the same position in the output features of the specified network layer of the i-th Unet unit of different generative networks, which are the aforementioned target features.
  • each feature in the 3*3 features shown in Figure 5 can be used as the target feature in turn, and combined with the corresponding features of other generative networks for weighted fusion processing, so as to obtain the feature corresponding to the corresponding position of the generative network based on the weighted fusion features, thereby ensuring the rationality of the output image of the generative network.
  • the output image of generator network 1 is a person with open eyes
  • the output image of generator network 10 is a person with closed eyes
  • the distance between the upper and lower eyelids of the person in the output images of generator networks 2 to 9 is different, such as gradually decreasing distance, thus presenting a gradual transition effect of the person's movement from open eyes to closed eyes.
  • the temporal attention mechanism can reasonably and reliably determine the content to be output by itself, thereby generating an animated video with continuous action.
  • Step C Based on the parallel arrangement of multiple generator networks and the output images of each generator network, a target video containing the second target object is obtained.
  • Each generator network has a corresponding sequence number. Based on the sequence number of each generator network, the parallel arrangement order of multiple generator networks can be determined. Then, based on the output images of each of the multiple generator networks, the target video can be obtained by sorting them sequentially according to their sequence numbers.
  • multiple generator networks are divided into a first type of network and a second type of network.
  • the input of the first type of network includes prompt text
  • the second type of network consists of all generator networks except the first type.
  • the input of the second type of network does not include prompt text, and the input content corresponding to its prompt text is empty or the prompt text is default.
  • generator network 1, generator network 10, and generator network N in Figure 2 are first type networks
  • generator networks 2 to 9 are second type networks.
  • the second target object in the output image corresponding to the first type of network has a first action form, which corresponds to the object action described in the prompt text of the first type of network; the second target object in the output image corresponding to the second type of network has a second action form; and the second action form corresponding to the second type of network located between two adjacent first type networks is a gradual transition form used to connect the first action forms corresponding to the two adjacent first type networks.
  • the prompt text of generator network 1 describes the action of opening eyes, and the first action form is the open eyes form;
  • the prompt text of generator network 10 describes the action of closing eyes, and the corresponding first action form is the closed eyes form;
  • the second action forms generated by generator networks 2 to 9 are intermediate forms that gradually transition from open eyes to closed eyes.
  • the specific implementation principle can be referred to the aforementioned related content, and will not be repeated here.
  • the video generation method provided in this disclosure further includes: applying special effects processing to the target video to obtain a special effects video.
  • This special effects processing includes, but is not limited to, speed adjustment and masking, and may specifically depend on the application scenario and user needs of the target video; therefore, no limitation is placed on the special effects processing method herein.
  • the first target object can be the streamer in the target live streaming room
  • the target prompt image is the streamer's image.
  • at least two prompt texts describing actions can be preset.
  • an animated video of a virtual object with the streamer's external characteristics can be generated using a target network model, and this virtual object can perform the actions described in the prompt texts.
  • This animated video can serve as a customized gift video for the streamer.
  • the animated video can be played on the interface of all users in the target live streaming room, presenting the visual experience of giving a customized image gift to the streamer, enhancing the fun and appeal of the live stream.
  • the method provided by this disclosure can quickly and conveniently generate corresponding personalized animations for streamers, and the object actions in the animation can be adjusted according to needs, which can better meet user needs.
  • one or more embodiments of this disclosure can reduce the cost of generating animated videos and improve the efficiency of video production. They also make the video generation process more controllable, generating animated videos that can present a second target object with similar external characteristics to the first target object. These videos are also more interesting and engaging for users, and can better meet user needs.
  • FIG. 6 is a schematic diagram of the structure of a video generation apparatus provided in this disclosure. This apparatus can be implemented by software and/or hardware, and is generally integrated into an electronic device. As shown in Figure 6, the video generation apparatus includes:
  • the acquisition module 602 is used to acquire a target prompt image and at least two prompt texts; wherein, the target prompt image includes a first target object, and the at least two prompt texts are used to describe different object actions;
  • the video generation module 604 is used to generate a target video containing a second target object based on a target prompt image and at least two prompt texts using a target network model; wherein the second target object has at least some of the external features of the first target object, and the target video presents dynamic scenes of the second target object performing different object actions.
  • the apparatus provided in this disclosure can efficiently and conveniently generate a target video containing a second target object based on a target prompt image (including a first target object) and at least two prompt texts (used to describe different object actions) using a target network model, without requiring professional video production. This significantly reduces video generation costs and improves efficiency. Furthermore, under the constraint of the target prompt image, the second target object in the target video possesses at least some of the external features of the first target object, and under the constraint of the at least two prompt texts, the second target object can perform different object actions.
  • This method can achieve the effect of generating personalized videos for the first target object, and the constraints of the target prompt image and at least two prompt texts also make the video generation process highly controllable, generating animated videos that present a second target object with similar external features to the first target object. This provides users with greater interest and appeal, effectively meeting their needs.
  • the target network model includes multiple generator networks arranged in parallel; the target cue image is used as input to each generator network, and the at least two cue texts are used as input to at least two generator networks, with different cue texts corresponding to different generator networks.
  • the video generation module 604 is specifically used to: obtain object text features and corresponding prompt text features for each of the at least two prompt texts based on the target prompt image and the at least two prompt texts; input the object text features into each of the generation networks, and input the corresponding prompt text features for each of the at least two prompt texts into different generation networks; generate respective output images through the multiple generation networks based on their respective input information; wherein the output images contain the second target object, and the action form of the second target object presented in different output images is different; and obtain a target video containing the second target object based on the parallel arrangement order of the multiple generation networks and the respective output images of the multiple generation networks.
  • the video generation module 604 is specifically used to: obtain the first target object features in the target prompt image using an object recognition model, and map the first target object features into object text features through the conditional injection network; and encode the at least two prompt texts using the at least two text encoders to obtain the prompt text features corresponding to each of the at least two prompt texts.
  • the video generation module 604 is specifically used to: generate output images of each of the multiple generation networks based on their respective input information, target association information, and parallel arrangement order using a temporal attention mechanism; wherein, the target association information includes target features output by the same specified network layer corresponding to each of the multiple generation networks.
  • the plurality of generative networks are divided into a first type of network and a second type of network.
  • the input of the first type of network includes the prompt text
  • the second type of network is any network other than the first type of network among the plurality of generative networks.
  • the second target object in the output image corresponding to the first type of network has a first action form, which corresponds to the object action described by the prompt text of the first type of network.
  • the second target object in the output image corresponding to the second type of network has a second action form.
  • the second action form corresponding to the second type of network located between two adjacent first type of networks is a gradient form used to connect the first action forms corresponding to the two adjacent first type of networks.
  • the plurality of generating networks have the same structure and share parameters; the input of each of the plurality of generating networks also includes a noise map, and the noise maps of different generating networks are different.
  • the apparatus further includes a special effects processing module for performing special effects processing on the target video to obtain a special effects video.
  • the video generation apparatus provided in this disclosure can execute the video generation method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of executing the method.
  • This disclosure provides an electronic device, which includes: a storage device storing a computer program thereon; and a processing device for executing the computer program in the storage device to implement the steps of any method of this disclosure.
  • the terminal devices in embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers.
  • mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers.
  • PDAs personal digital assistants
  • PADs tablet computers
  • PMPs portable multimedia players
  • in-vehicle terminals e.g., in-vehicle navigation terminals
  • fixed terminals such as digital TVs and desktop computers.
  • the electronic device shown in FIG7 is
  • the electronic device 700 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703.
  • the RAM 703 also stores various programs and data required for the operation of the electronic device 700.
  • the processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704.
  • An input/output (I/O) interface 705 is also connected to the bus 704.
  • I/O interface 705 input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709.
  • Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data.
  • Figure 7 illustrates electronic device 700 with various devices, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.
  • embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts.
  • the computer program can be downloaded and installed from a network via communication device 709, or installed from storage device 708, or installed from ROM 702.
  • processing device 701 it performs the functions defined in the methods of embodiments of this disclosure.
  • embodiments of this disclosure can also be computer program products, comprising computer program instructions that, when executed by a processor, cause the processor to perform the image processing methods provided in the embodiments of this disclosure.
  • the computer program product can be written in any combination of one or more programming languages to perform the operations of the embodiments of this disclosure.
  • the programming languages include object-oriented programming languages such as Java and C++, as well as conventional procedural programming languages such as C or similar languages.
  • the program code can be executed entirely on a user's computing device, partially on a user's device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.
  • embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the video generation method provided in embodiments of this disclosure.
  • the computer-readable storage medium may be any combination of one or more readable media.
  • a readable medium may be a readable signal medium or a readable storage medium.
  • a readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
  • This disclosure also provides a computer program product, including a computer program/instructions that, when executed by a processor, implement the video generation method of this disclosure.
  • a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information.
  • This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.
  • sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format.
  • the pop-up window can also include a selection control allowing the user to choose "agree” or “disagree” to provide personal information to the electronic device.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Databases & Information Systems (AREA)
  • Human Computer Interaction (AREA)
  • Processing Or Creating Images (AREA)

Abstract

本公开实施例涉及一种视频生成方法、装置、设备及介质，其中该方法包括：获取目标提示图像以及至少两个提示文本；其中，目标提示图像中包括第一目标对象，至少两个提示文本用于描述不同的对象动作；基于目标提示图像与至少两个提示文本，利用目标网络模型生成包含有第二目标对象的目标视频；其中，第二目标对象具有第一目标对象的至少部分外在特征，且目标视频呈现出第二目标对象执行不同的对象动作的动态画面。

Description

视频生成方法、装置、设备及介质
相关申请的交叉引用
本申请要求于2024年6月26日提交的，申请号为202410841770.2、发明名称为“视频生成方法、装置、设备及介质”的中国专利申请的优先权，该申请的全部内容通过引用结合在本申请中。
技术领域
本公开涉及多媒体处理技术领域，尤其涉及一种视频生成方法、装置、设备及介质。
背景技术
随着多媒体技术的发展，越来越多的场景都需要富有感染力或趣味性的视频来提升用户的视觉体验。以直播场景为例，可以基于某用户选择赠予主播的虚拟礼物而在所有观众们的终端界面上呈现出相应的礼物动画视频，在社交场景中，用户之间也可以互相赠予带有趣味性的动画视频。
发明内容
本公开提供了一种视频生成方法、装置、设备及介质。
本公开实施例提供了一种视频生成方法，所述方法包括：获取目标提示图像以及至少两个提示文本；其中，所述目标提示图像中包括第一目标对象，所述至少两个提示文本用于描述不同的对象动作；基于所述目标提示图像与所述至少两个提示文本，利用目标网络模型生成包含有第二目标对象的目标视频；其中，所述第二目标对象具有所述第一目标对象的至少部分外在特征，且所述目标视频呈现出所述第二目标对象执行所述不同的对象动作的动态画面。
可选的，所述目标网络模型包含多个生成网络，且多个所述生成网络并行排列；所述目标提示图像用于作为每个所述生成网络的输入，所述至少两个提示文本用于作为至少两个所述生成网络的输入，不同的所述提示文本对应的生成网络不同。
可选的，所述基于所述目标提示图像与所述至少两个提示文本，利用目标网络模型生成包含有第二目标对象的目标视频，包括：基于所述目标提示图像与所述至少两个提示文本，获取对象文本特征和所述至少两个提示文本各自对应的提示文本特征；将所述对象文本特征输入至每个所述生成网络，以及将所述至少两个提示文本各自对应的提示文本特征输入至不同的生成网络；通过所述多个生成网络基于各自的输入信息生成各自的输出图像；其中，所述输出图像包含所述第二目标对象，不同输出图像中呈现的所述第二目标对象的动作形态不同；基于所述多个生成网络的并行排列顺序及所述多个生成网络各自的输出图像，得到包含有第二目标对象的目标视频。
可选的，所述目标网络模型还包括条件注入网络以及至少两个文本编码器；所述基于所述目标提示图像与所述至少两个提示文本，获取对象文本特征和所述至少两个提示文本各自对应的提示文本特征，包括：利用对象识别模型获取所述目标提示图像中的第一目标对象特征，并通过所述条件注入网络将所述第一目标对象特征映射为对象文本特征；利用所述至少两个文本编码器对所述至少两个提示文本分别进行编码处理，得到所述至少两个提示文本各自对应的提示文本特征。
可选的，所述通过所述多个生成网络基于各自的输入信息生成各自的输出图像，包括：通过所述多个生成网络基于各自对应的输入信息、目标关联信息以及并行排列顺序，利用时序注意力机制生成各自的输出图像；其中，所述目标关联信息包括所述多个生成网络各自对应的相同指定网络层输出的目标特征。
可选的，所述多个生成网络划分为第一类网络和第二类网络，所述第一类网络的输入包括所述提示文本，所述第二类网络是所述多个生成网络中除所述第一类网络之外的网络；所述第一类网络对应的输出图像中的第二目标对象具有第一动作形态，所述第一动作形态对应所述第一类网络的提示文本所描述的对象动作；所述第二类网络对应的输出图像中的第二目标对象具有第二动作形态；且，位于相邻的两个第一类网络之间的第二类网络对应的第二动作形态，是用于衔接所述相邻的两个第一类网络对应的第一动作形态的渐变形态。
可选的，所述多个生成网络的结构均相同，且所述多个生成网络共享参数；所述多个生成网络各自的输入还包括噪声图，不同的所述生成网络的噪声图不同。
可选的，所述方法还包括：对所述目标视频进行特效处理，得到特效视频。
本公开实施例还提供了一种视频生成装置，包括：获取模块，用于获取目标提示图像以及至少两个提示文本；其中，所述目标提示图像中包括第一目标对象，所述至少两个提示文本用于描述不同的对象动作；视频生成模块，用于基于所述目标提示图像与所述至少两个提示文本，利用目标网络模型生成包含有第二目标对象的目标视频；其中，所述第二目标对象具有所述第一目标对象的至少部分外在特征，且所述目标视频呈现出所述第二目标对象执行所述不同的对象动作的动态画面。
本公开实施例还提供了一种电子设备，所述电子设备包括：处理器；用于存储所述处理器可执行指令的存储器；所述处理器，用于从所述存储器中读取所述可执行指令，并执行所述指令以实现如本公开实施例提供的视频生成方法。
本公开实施例还提供了一种计算机可读存储介质，所述存储介质存储有计算机程序，所述计算机程序用于执行如本公开实施例提供的视频生成方法。
应当理解，本部分所描述的内容并非旨在标识本公开的实施例的关键或重要特征，也不用于限制本公开的范围。本公开的其它特征将通过以下的说明书而变得容易理解。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本公开的实施例，并与说明书一起用于解释本公开的原理。
为了更清楚地说明本公开实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，对于本领域普通技术人员而言，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1为本公开实施例提供的一种视频生成方法的流程示意图；
图2为本公开实施例提供的一种目标网络模型的结构示意图；
图3为本公开实施例提供的一种目标网络模型的结构示意图；
图4为本公开实施例提供的一种生成网络的结构示意图；
图5为本公开实施例提供的一种时序注意力机制的原理示意图；
图6为本公开实施例提供的一种视频生成装置的结构示意图；
图7为本公开实施例提供的一种电子设备的结构示意图。
具体实施方式
如上文所述，越来越多的场景都需要富有感染力或趣味性的视频来提升用户的视觉体验。然而，现有技术大多需要专业人士制作动画视频，生成动画视频所需的成本较高，效率低下。而借助人工智能生成动画视频的方式可控性较差，也难以满足个性化定制需求。
为了能够更清楚地理解本公开的上述目的、特征和优点，下面将对本公开的方案进行进一步描述。需要说明的是，在不冲突的情况下，本公开的实施例及实施例中的特征可以相互组合。
在下面的描述中阐述了很多具体细节以便于充分理解本公开，但本公开还可以采用其他不同于在此描述的方式来实施；显然，说明书中的实施例只是本公开的一部分实施例，而不是全部的实施例。
图1为本公开实施例提供的一种视频生成方法的流程示意图，该方法可以由视频生成装置执行，其中该装置可以采用软件和/或硬件实现，一般可集成在电子设备中。如图1所示，该方法主要包括如下步骤S102～步骤S104：
步骤S102，获取目标提示图像以及至少两个提示文本；其中，目标提示图像中包括第一目标对象，至少两个提示文本用于描述不同的对象动作。
本公开实施例对第一目标对象不进行限制，诸如，第一目标对象可以为人物、动物、车辆、机器人等。本公开实施例对提示文本所描述的对象动作不进行限制，诸如，该对象动作可以为睁眼、闭眼、抬头、低头、走、跑等。
步骤S104，基于目标提示图像与至少两个提示文本，利用目标网络模型生成包含有第二目标对象的目标视频；其中，第二目标对象具有第一目标对象的至少部分外在特征，且目标视频呈现出第二目标对象执行不同的对象动作的动态画面。诸如，假设输入的两个提示文本用于描述睁眼和闭眼两个动作，目标视频能够呈现出第二目标对象由睁眼到闭眼的动态画面。
本公开实施例对目标网络模型的结构不进行限制，示例性地，目标网络模型为生成模型。在实际应用中，可以采用基础的扩散模型及LoRA(Low-Rank Adaptation)模型训练得到本公开实施例中的目标网络模型，该目标网络模型能够生成具有特定风格的对象图像，以第一目标对象为人示例，则该目标网络模型可生成包含有特定风格的虚拟人物形象的图像，如第二目标对象是具有第一目标对象的外在特征的卡通虚拟人物、古装虚拟人物等。具体的，可以基于LoRA技术对基础扩散模型进行参数调整，LoRA技术是一种轻量级的模型微调技术，可借助在基础扩散模型中插入一些网络层(可称为LoRA层)辅助模型训练，能够在使用少量具有特定对象的样本数据的情况下，对大型的基础模型进行定制化微调，从而生成具有特定对象生成能力的生成模型。此外，本公开实施例还可以在生成模型的原有网络结构基础上，新增诸如时序注意力模块等特定的功能模块，借助该功能模块基于时序注意力机制实现目标网络模型生成具有第二目标对象的动作渐变效果的多帧图像，从而形成目标视频。
应当说明的是，在相关的生成式人工智能技术中，基本只是为模型提供一个提示文本，而本公开实施例会获取至少两个提示文本，有助于使模型基于至少两个提示文本可控且可靠地生成呈现有执行不同的对象动作的对象动画视频。
本公开实施例提供的上述技术方案，能够基于目标提示图像(包括第一目标对象)与至少两个提示文本(用于描述不同的对象动作)，利用目标网络模型高效便捷地生成包含有第二目标对象的目标视频，而无需专业人士制作动画视频，可较好地降低动画视频的生成成本，提升动画视频的生成效率；而且在目标提示图像的约束下，目标视频中的第二目标对象具有第一目标对象的至少部分外在特征，在至少两个提示文本的约束下，第二目标对象可执行不同的对象动作。上述方式可以实现针对第一目标对象生成个性化定制视频的效果，且通过目标提示图像以及至少两个提示文本的约束，也使视频生成过程具有较强的可控性，生成能够呈现出与第一目标对象外在特征相近的第二目标对象的动画视频，对于用户而言也具有较强的趣味性和感染力，能够较好地满足用户需求。
在一些实施方式中，目标网络模型包含多个生成网络，且多个生成网络并行排列；目标提示图像用于作为每个生成网络的输入，至少两个提示文本用于作为至少两个生成网络的输入，不同的提示文本对应的生成网络不同。在一些具体的实施示例中，多个生成网络的结构均相同，且多个生成网络共享参数，更便于训练及信息交互，并可较好地把控输出图像的匹配度。多个生成网络各自的输入还包括噪声图，不同的生成网络的噪声图不同，各个生成网络对应的噪声图可以是随机生成的，可采用相关技术实现，在此不再赘述。
为便于理解，可参见图2所示的一种目标网络模型的结构示意图，示意出并行的生成网络1、生成网络2、……、生成网络10至生成网络N。每个生成网络分别具有输出图像，所有输出图像按序排列即为目标视频。如图2可见，每个生成网络的输入均包含目标提示图像和相应的噪声图，但只有部分生成网络的输入包括提示文本，诸如生成网络2没有相应的提示文本，可以视为其提示文本的输入内容为空或者缺省。当然图2仅为示意性说明，在实际应用中也可以为每个生成网络分别设置相应的提示文本，另外在提示文本的数量少于生成网络的数量的情况下，可以根据需求灵活指定输入包含提示文本的生成网络，在此不进行限制。
在前述基础上，上述步骤S104，也即基于目标提示图像与至少两个提示文本，利用目标网络模型生成包含有第二目标对象的目标视频的步骤，可以参照如下步骤A～步骤C执行：
步骤A，基于目标提示图像与至少两个提示文本，获取对象文本特征和至少两个提示文本各自对应的提示文本特征；在一些具体的实施示例中，目标网络模型还包括条件注入网络以及至少两个文本编码器，步骤A可以参照如下步骤A1～步骤A2执行：
步骤A1，利用对象识别模型获取目标提示图像中的第一目标对象特征，并通过条件注入网络将第一目标对象特征映射为对象文本特征，以将对象文本特征注入至生成网络中。本公开实施例对条件注入网络的结构不进行限制，诸如其可以包含特征映射单元等功能单元，条件注入网络不仅能够对接收到的数据进行进一步特征提取，也可以进行诸如特征映射等处理，将对象特征(也即第一目标对象特征)映射为提示文本特征，并将提示文本特征作为条件信息注入至目标网络的生成过程中，在一定程度上能够引导图像生成，保障输出图像中所包含的第二目标对象具有与第一目标对象相似的外在特征。
步骤A2，利用至少两个文本编码器对至少两个提示文本分别进行编码处理，得到至少两个提示文本各自对应的提示文本特征。
步骤B，将对象文本特征输入至每个生成网络，以及将至少两个提示文本各自对应的提示文本特征输入至不同的生成网络；通过多个生成网络基于各自的输入信息生成各自的输出图像；其中，输出图像包含第二目标对象，不同输出图像中呈现的第二目标对象的动作形态不同。
另外应当说明的是，本公开实施例提供的目标网络模型与现有常规的视频生成模型或者文生图模型相比，有两个明显的区别在于：(1)不局限于仅采用文本生成图像或视频，而是在提示文本的基础上引入目标提示图像，从而将第一目标对象的特征注入至目标网络模型中，有助于针对第一目标对象实现个性化定制效果。(2)现有技术基本仅为生成模型提供一个提示文本，即便采用的生成模型存在多个生成网络，多个生成网络也均共享同一个提示文本，而本公开实施例可以获取至少两个用于描述不同对象动作的提示文本，并且可分别输入至不同的生成网络，部分生成网络也可不具有提示文本，而是基于其它网络的特征信息，依据时序注意力机制等方式学习自身所需生成的内容，上述方式可以有效保障生成的动画视频中所呈现的对象动作的准确性及可控性。
为便于理解，在图2的基础上，可参见图3所示的一种目标网络模型的结构示意图，在图3中进一步示意出对象识别模型、条件注入网络以及文本编码器。示例性地，条件注入网络可以包含Adapter单元。条件注入网络可将接收到的第一目标对象特征映射至相应的特征空间，映射所得的特征(也即对象文本特征)可通过生成网络中的部分网络注入至生成网络的图像生成过程中，上述部分网络诸如可以是生成网络中的去噪网络(例如Unet)的交叉注意力单元，还可以是生成网络中增设的LoRA层等。通过上述方式，条件注入网络可将第一目标对象特征等条件信息注入至图像生成过程中，从而使生成网络的输出图像中所包含的第二目标对象能够在一定程度上反映出第一目标对象的至少部分外在特征。同样，通过将提示文本特征输入至生成网络，可以有效约束生成网络的输出图像中所包含的第二目标对象能够执行提示文本所描述的对象动作。
在一些具体的实施示例中，通过多个生成网络基于各自对应的输入信息、目标关联信息以及并行排列顺序，利用时序注意力机制生成各自的输出图像；其中，目标关联信息包括多个生成网络各自对应的相同指定网络层输出的目标特征，在实际应用中，多个生成网络各自对应的相同指定网络层诸如可以是指多个生成网络的同一Unet单元的某特定网络层，指定网络层输出的特征信息中的每个位置的特征或者指定位置的特征均可作为目标特征。
为便于理解，可参照图4所示的一种生成网络的结构示意图，示意出生成网络包括编码器、去噪网络和解码器，其中，编码器的输入为噪声图，解码器的输出为图像，且也示意出可将条件注入网络输出的特征注入至各个Unet单元，另外，在实际应用中，生成网络的输入若包含提示文本，则提示文本特征也会输入至各Unet单元，图4中对此并未示意。图4中以去噪网络为Unet网络为例进行示意，包括多个Unet单元(Unet Block)，与常规Unet网络的区别在于插入了时序注意力模块，该模块主要基于时序注意力机制进行信息处理，具体利用时序注意力机制实现N个生成网络之间的信息交互，使每个生成网络可以基于获取到的其它生成网络的特征信息并确定自身所需生成的特征，从而保障N个生成网络对应的输出图像可以按序实现第二目标对象动作渐变的动画效果，诸如实现从睁眼、上下眼皮逐渐闭合到完全闭眼的动态效果。如图4可知，在N个并行的生成网络中，具有相同位置的时序注意力模块可基于各自对应的目标关联信息、目标网络的排列顺序等信息进行时序注意力处理，以目标关联信息1为例，包含N个生成网络对应的首个模块(首个Unet单元和/或首个时序注意力模块)的相同指定网络层输出的特征信息，不同时序注意力模块的位置不同，对应的目标关联信息不同。在实际应用中，时序注意力模块可以采用相关技术实现，例如可以采用Animate diff模块(动画差异模块)实现，其能够整合到诸如生成网络等模型中，并从视频数据集中学习运动先验，从而可将文生图模型扩展为视频生成模型。
具体的，时序注意力机制允许目标网络模型中的生成网络在处理N个序列数据时，能够自适应地关注时序信息，捕捉序列数据中的依赖关系和上下文信息，并为N个序列数据中具有相同位置的特征分配不同的权重，参照图5所示的一种时序注意力机制的原理示意图，对于每个生成网络而言，利用时序注意力机制可以获取N个生成网络各自对应的相同指定网络层输出的目标特征，诸如，对N个生成网络各自第i个Unet单元的指定网络层的输出特征中相同位置的特征分配相应权重并进行加权融合处理，在图5中以第i个Unet单元的指定网络层的输出特征为例，图5用灰色格子示意出不同生成网络的第i个Unet单元的指定网络层的输出特征中相同位置的特征，也即前述目标特征，在实际应用中，图5中所示的3*3特征中的每个特征均可依次作为目标特征，并结合其它生成网络的相应特征进行加权融合处理，从而基于加权融合特征得到该生成网络的相应位置对应的特征，以此保障该生成网络的输出图像的合理性。诸如，生成网络1的输出图像为人物睁眼图，生成网络10的输出图像为人物闭眼图，生成网络2～生成网络9的输出图像的人物的上下眼皮之间的距离不同，诸如距离逐渐缩小，从而呈现出人物从睁眼开始逐渐过渡到闭眼的动作渐变效果。可以理解的是，虽然可能只有部分网络的输入包含用于描述动作的提示文本，诸如生成网络1的提示文本描述睁眼动作，生成网络10的提示文本描述闭眼动作，但是对于输入不包含提示文本(可理解为提示文本为空或者提示文本缺省)的生成网络2～生成网络9而言，基于所获得的其它生成网络的目标关联信息以及多个网络的排列顺序，利用时序注意力机制便可合理可靠地确定其自身所需输出的内容，从而生成具有动作连续性的动画视频。
步骤C，基于多个生成网络的并行排列顺序及多个生成网络各自的输出图像，得到包含有第二目标对象的目标视频。
每个生成网络都具有相应的排列序号，基于各生成网络的排列序号可知多个生成网络的并行排列顺序，然后基于多个生成网络各自的输出图像按照序号进行依次排序，即可得到目标视频。
在一些实施方式中，多个生成网络划分为第一类网络和第二类网络，第一类网络的输入包括提示文本，第二类网络是多个生成网络中除第一类网络之外的网络；换言之，第二类网络的输入不包括提示文本，其提示文本对应的输入内容为空或者提示文本缺省。示例性地，图2中的生成网络1、生成网络10、生成网络N为第一类网络，生成网络2～生成网络9为第二类网络。
第一类网络对应的输出图像中的第二目标对象具有第一动作形态，第一动作形态对应第一类网络的提示文本所描述的对象动作；第二类网络对应的输出图像中的第二目标对象具有第二动作形态；且，位于相邻的两个第一类网络之间的第二类网络对应的第二动作形态，是用于衔接相邻的两个第一类网络对应的第一动作形态的渐变形态。诸如，生成网络1的提示文本描述睁眼动作，第一动作形态为睁眼形态，生成网络10的提示文本描述闭眼动作，相应的第一动作形态为闭眼形态；生成网络2～生成网络9生成的第二动作形态为由睁眼逐步衔接过渡到闭眼的中间形态。具体实现原理可参照前述相关内容，在此不再赘述。
在一些实施方式中，本公开实施例提供的上述视频生成方法还包括：对目标视频进行特效处理，得到特效视频。该特效处理包括但不限于变速处理、蒙版处理等，具体可取决于目标视频的应用场景及用户需求，在此对特效处理方式不进行限制。
本公开实施例对目标视频的应用场景也不进行限制，诸如可以为直播场景、社交场景等。以下以直播场景为例，第一目标对象可以为目标直播间中的主播，目标提示图像即为主播图像，然后可预先设置至少两个用于描述动作的提示文本，之后可利用目标网络模型生成具有主播外在特征的虚拟对象的动画视频，且该虚拟对象能够执行提示文本所描述的动作。该动画视频可作为针对主播定制化的礼物视频，在观众发起特定的礼物赠送请求时，即可在目标直播间的所有用户端的界面上播放该动画视频，呈现出给主播赠送定制化形象礼物的观感，增强了直播趣味性和感染力。可以理解的是，倘若采用现有技术，则需要专业人士才可为主播制作个性化动画视频，所需成本较高且效率低下，在现如今主播数量较多的情况下更无法一一为主播定制动画视频，而通过本公开实施例提供的上述方式，可以快速便捷地针对主播生成相应的个性化定制动画，且可根据需求调整动画中的对象动作，可较好地满足用户需求。换而言之，本公开的一个或多个实施例可降低动画视频生成成本并提升视频制作效率，也使视频生成过程具有较强的可控性，生成能够呈现出与第一目标对象外在特征相近的第二目标对象的动画视频，对用户也具有较强的趣味性和感染力，能够较好地满足用户需求。
对应于前述视频生成方法，本公开实施例进一步提供了一种视频生成装置，图6为本公开实施例提供的一种视频生成装置的结构示意图，该装置可由软件和/或硬件实现，一般可集成在电子设备中，如图6所示，视频生成装置包括：
获取模块602，用于获取目标提示图像以及至少两个提示文本；其中，目标提示图像中包括第一目标对象，至少两个提示文本用于描述不同的对象动作；
视频生成模块604，用于基于目标提示图像与至少两个提示文本，利用目标网络模型生成包含有第二目标对象的目标视频；其中，第二目标对象具有第一目标对象的至少部分外在特征，且目标视频呈现出第二目标对象执行不同的对象动作的动态画面。
本公开实施例提供的上述装置，能够基于目标提示图像(包括第一目标对象)与至少两个提示文本(用于描述不同的对象动作)，利用目标网络模型高效便捷地生成包含有第二目标对象的目标视频，而无需专业人士制作视频，可较好地降低视频生成成本，提升视频生成效率；而且在目标提示图像的约束下，目标视频中的第二目标对象具有第一目标对象的至少部分外在特征，在至少两个提示文本的约束下，第二目标对象可执行不同的对象动作。上述方式可以实现针对第一目标对象生成个性化定制视频的效果，且通过目标提示图像以及至少两个提示文本的约束，也使视频生成过程具有较强的可控性，生成能够呈现出与第一目标对象外在特征相近的第二目标对象的动画视频，对于用户而言也具有较强的趣味性和感染力，能够较好地满足用户需求。
在一些实施方式中，所述目标网络模型包含多个生成网络，且多个所述生成网络并行排列；所述目标提示图像用于作为每个所述生成网络的输入，所述至少两个提示文本用于作为至少两个所述生成网络的输入，不同的所述提示文本对应的生成网络不同。
在一些实施方式中，所述视频生成模块604具体用于：基于所述目标提示图像与所述至少两个提示文本，获取对象文本特征和所述至少两个提示文本各自对应的提示文本特征；将所述对象文本特征输入至每个所述生成网络，以及将所述至少两个提示文本各自对应的提示文本特征输入至不同的生成网络；通过所述多个生成网络基于各自的输入信息生成各自的输出图像；其中，所述输出图像包含所述第二目标对象，不同输出图像中呈现的所述第二目标对象的动作形态不同；基于所述多个生成网络的并行排列顺序及所述多个生成网络各自的输出图像，得到包含有第二目标对象的目标视频。
在一些实施方式中，所述视频生成模块604具体用于：利用对象识别模型获取所述目标提示图像中的第一目标对象特征，并通过所述条件注入网络将所述第一目标对象特征映射为对象文本特征；利用所述至少两个文本编码器对所述至少两个提示文本分别进行编码处理，得到所述至少两个提示文本各自对应的提示文本特征。
在一些实施方式中，所述视频生成模块604具体用于：通过所述多个生成网络基于各自对应的输入信息、目标关联信息以及并行排列顺序，利用时序注意力机制生成各自的输出图像；其中，所述目标关联信息包括所述多个生成网络各自对应的相同指定网络层输出的目标特征。
在一些实施方式中，所述多个生成网络划分为第一类网络和第二类网络，所述第一类网络的输入包括所述提示文本，所述第二类网络是所述多个生成网络中除所述第一类网络之外的网络；所述第一类网络对应的输出图像中的第二目标对象具有第一动作形态，所述第一动作形态对应所述第一类网络的提示文本所描述的对象动作；所述第二类网络对应的输出图像中的第二目标对象具有第二动作形态；且，位于相邻的两个第一类网络之间的第二类网络对应的第二动作形态，是用于衔接所述相邻的两个第一类网络对应的第一动作形态的渐变形态。
在一些实施方式中，所述多个生成网络的结构均相同，且所述多个生成网络共享参数；所述多个生成网络各自的输入还包括噪声图，不同的所述生成网络的噪声图不同。
在一些实施方式中，所述装置还包括特效处理模块，用于对所述目标视频进行特效处理，得到特效视频。
本公开实施例所提供的视频生成装置可执行本公开任意实施例所提供的视频生成方法，具备执行方法相应的功能模块和有益效果。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的装置实施例的具体工作过程，可以参考方法实施例中的对应过程，在此不再赘述。
本公开实施例提供了一种电子设备，电子设备包括：存储装置，其上存储有计算机程序；处理装置，用于执行所述存储装置中的所述计算机程序，以实现本公开中任一项方法的步骤。
下面参考图7，其示出了适于用来实现本公开实施例的电子设备700的结构示意图。本公开实施例中的终端设备可以包括但不限于诸如移动电话、笔记本电脑、数字广播接收器、PDA(个人数字助理)、PAD(平板电脑)、PMP(便携式多媒体播放器)、车载终端(例如车载导航终端)等等的移动终端以及诸如数字TV、台式计算机等等的固定终端。图7示出的电子设备仅仅是一个示例，不应对本公开实施例的功能和使用范围带来任何限制。
如图7所示，电子设备700可以包括处理装置(例如中央处理器、图形处理器等)701，其可以根据存储在只读存储器(ROM)702中的程序或者从存储装置708加载到随机访问存储器(RAM)703中的程序而执行各种适当的动作和处理。在RAM 703中，还存储有电子设备700操作所需的各种程序和数据。处理装置701、ROM 702以及RAM 703通过总线704彼此相连。输入/输出(I/O)接口705也连接至总线704。
通常，以下装置可以连接至I/O接口705：包括例如触摸屏、触摸板、键盘、鼠标、摄像头、麦克风、加速度计、陀螺仪等的输入装置706；包括例如液晶显示器(LCD)、扬声器、振动器等的输出装置707；包括例如磁带、硬盘等的存储装置708；以及通信装置709。通信装置709可以允许电子设备700与其他设备进行无线或有线通信以交换数据。虽然图7示出了具有各种装置的电子设备700，但是应理解的是，并不要求实施或具备所有示出的装置。可以替代地实施或具备更多或更少的装置。
特别地，根据本公开的实施例，上文参考流程图描述的过程可以被实现为计算机软件程序。例如，本公开的实施例包括一种计算机程序产品，其包括承载在非暂态计算机可读介质上的计算机程序，该计算机程序包含用于执行流程图所示的方法的程序代码。在这样的实施例中，该计算机程序可以通过通信装置709从网络上被下载和安装，或者从存储装置708被安装，或者从ROM 702被安装。在该计算机程序被处理装置701执行时，执行本公开实施例的方法中限定的上述功能。
除了上述方法和设备以外，本公开的实施例还可以是计算机程序产品，其包括计算机程序指令，所述计算机程序指令在被处理器运行时使得所述处理器执行本公开实施例所提供的图像处理方法。所述计算机程序产品可以以一种或多种程序设计语言的任意组合来编写用于执行本公开实施例操作的程序代码，所述程序设计语言包括面向对象的程序设计语言，诸如Java、C++等，还包括常规的过程式程序设计语言，诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算设备上执行、部分地在用户设备上执行、作为一个独立的软件包执行、部分在用户计算设备上部分在远程计算设备上执行、或者完全在远程计算设备或服务器上执行。
此外，本公开的实施例还可以是计算机可读存储介质，其上存储有计算机程序指令，所述计算机程序指令在被处理器运行时使得所述处理器执行本公开实施例所提供的视频生成方法。
所述计算机可读存储介质可以采用一个或多个可读介质的任意组合。可读介质可以是可读信号介质或者可读存储介质。可读存储介质例如可以包括但不限于电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。可读存储介质的更具体的例子(非穷举的列表)包括：具有一个或多个导线的电连接、便携式盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。
本公开实施例还提供了一种计算机程序产品，包括计算机程序/指令，该计算机程序/指令被处理器执行时实现本公开实施例中的视频生成方法。
可以理解的是，在使用本公开各施例公开的技术方案之前，均应当依据相关法律法规通过恰当的方式对本公开所涉及个人信息的类型、使用范围、使用场景等告知用户并获得用户的授权。
例如，在响应于接收到用户的主动请求时，向用户发送提示信息，以明确地提示用户，其请求执行的操作将需要获取和使用到用户的个人信息。从而，使得用户可以根据提示信息来自主地选择是否向执行本公开技术方案的操作的电子设备、应用程序、服务器或存储介质等软件或硬件提供个人信息。
作为一种可选的但非限定性的实现方式，响应于接收到用户的主动请求，向用户发送提示信息的方式例如可以是弹窗的方式，弹窗中可以以文字的方式呈现提示信息。此外，弹窗中还可以承载供用户选择“同意”或者“不同意”向电子设备提供个人信息的选择控件。
可以理解的是，上述通知和获取用户授权过程仅是示意性的，不对本公开的实现方式构成限定，其他满足相关法律法规的方式也可应用于本公开的实现方式中。
需要说明的是，在本文中，诸如“第一”和“第二”等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括所述要素的过程、方法、物品或者设备中还存在另外的相同要素。
以上所述仅是本公开的具体实施方式，使本领域技术人员能够理解或实现本公开。对这些实施例的多种修改对本领域的技术人员来说将是显而易见的，本文中所定义的一般原理可以在不脱离本公开的精神或范围的情况下，在其它实施例中实现。因此，本公开将不会被限制于本文所述的这些实施例，而是要符合与本文所公开的原理和新颖特点相一致的最宽的范围。

Claims (11)

  1. 一种视频生成方法，其中包括：
    获取目标提示图像以及至少两个提示文本；其中，所述目标提示图像中包括第一目标对象，所述至少两个提示文本用于描述不同的对象动作；
    基于所述目标提示图像与所述至少两个提示文本，利用目标网络模型生成包含有第二目标对象的目标视频；其中，所述第二目标对象具有所述第一目标对象的至少部分外在特征，且所述目标视频呈现出所述第二目标对象执行所述不同的对象动作的动态画面。
  2. 根据权利要求1所述的方法，其中所述目标网络模型包含多个生成网络，且多个所述生成网络并行排列；所述目标提示图像用于作为每个所述生成网络的输入，所述至少两个提示文本用于作为至少两个所述生成网络的输入，不同的所述提示文本对应的生成网络不同。
  3. 根据权利要求2所述的方法，其中所述基于所述目标提示图像与所述至少两个提示文本，利用目标网络模型生成包含有第二目标对象的目标视频，包括：
    基于所述目标提示图像与所述至少两个提示文本，获取对象文本特征和所述至少两个提示文本各自对应的提示文本特征；
    将所述对象文本特征输入至每个所述生成网络，以及将所述至少两个提示文本各自对应的提示文本特征输入至不同的生成网络；通过所述多个生成网络基于各自的输入信息生成各自的输出图像；其中，所述输出图像包含所述第二目标对象，不同输出图像中呈现的所述第二目标对象的动作形态不同；
    基于所述多个生成网络的并行排列顺序及所述多个生成网络各自的输出图像，得到包含有第二目标对象的目标视频。
  4. 根据权利要求3所述的方法，其中所述目标网络模型还包括条件注入网络以及至少两个文本编码器；所述基于所述目标提示图像与所述至少两个提示文本，获取对象文本特征和所述至少两个提示文本各自对应的提示文本特征，包括：
    利用对象识别模型获取所述目标提示图像中的第一目标对象特征，并通过所述条件注入网络将所述第一目标对象特征映射为对象文本特征；
    利用所述至少两个文本编码器对所述至少两个提示文本分别进行编码处理，得到所述至少两个提示文本各自对应的提示文本特征。
  5. 根据权利要求3所述的方法，其中所述通过所述多个生成网络基于各自的输入信息生成各自的输出图像，包括：
    通过所述多个生成网络基于各自对应的输入信息、目标关联信息以及并行排列顺序，利用时序注意力机制生成各自的输出图像；其中，所述目标关联信息包括所述多个生成网络各自对应的相同指定网络层输出的目标特征。
  6. 根据权利要求2所述的方法，其中所述多个生成网络划分为第一类网络和第二类网络，所述第一类网络的输入包括所述提示文本，所述第二类网络是所述多个生成网络中除所述第一类网络之外的网络；
    所述第一类网络对应的输出图像中的第二目标对象具有第一动作形态，所述第一动作形态对应所述第一类网络的提示文本所描述的对象动作；
    所述第二类网络对应的输出图像中的第二目标对象具有第二动作形态；且，位于相邻的两个第一类网络之间的第二类网络对应的第二动作形态，是用于衔接所述相邻的两个第一类网络对应的第一动作形态的渐变形态。
  7. 根据权利要求2所述的方法，其中所述多个生成网络的结构均相同，且所述多个生成网络共享参数；所述多个生成网络各自的输入还包括噪声图，不同的所述生成网络的噪声图不同。
  8. 根据权利要求1所述的方法，其中所述方法还包括：
    对所述目标视频进行特效处理，得到特效视频。
  9. 一种视频生成装置，其中包括：
    获取模块，用于获取目标提示图像以及至少两个提示文本；其中，所述目标提示图像中包括第一目标对象，所述至少两个提示文本用于描述不同的对象动作；
    视频生成模块，用于基于所述目标提示图像与所述至少两个提示文本，利用目标网络模型生成包含有第二目标对象的目标视频；其中，所述第二目标对象具有所述第一目标对象的至少部分外在特征，且所述目标视频呈现出所述第二目标对象执行所述不同的对象动作的动态画面。
  10. 一种电子设备，其中所述电子设备包括：
    存储装置，其上存储有计算机程序；
    处理装置，用于执行所述存储装置中的所述计算机程序，以实现权利要求1-8中任一项所述的视频生成方法的步骤。
  11. 一种计算机可读存储介质，其中所述存储介质存储有计算机程序，所述计算机程序用于执行上述权利要求1-8中任一所述的视频生成方法。
PCT/CN2025/082423 2024-06-26 2025-03-13 视频生成方法、装置、设备及介质 Pending WO2026001087A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202410841770.2A CN121217962A (zh) 2024-06-26 2024-06-26 视频生成方法、装置、设备及介质
CN202410841770.2 2024-06-26

Publications (1)

Publication Number Publication Date
WO2026001087A1 true WO2026001087A1 (zh) 2026-01-02

Family

ID=98112437

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2025/082423 Pending WO2026001087A1 (zh) 2024-06-26 2025-03-13 视频生成方法、装置、设备及介质

Country Status (2)

Country Link
CN (1) CN121217962A (zh)
WO (1) WO2026001087A1 (zh)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110047121A (zh) * 2019-03-20 2019-07-23 北京字节跳动网络技术有限公司 端到端的动画生成方法、装置及电子设备
US20210150357A1 (en) * 2018-11-14 2021-05-20 Nvidia Corporation Smoothing regularization for a generative neural network
CN116309965A (zh) * 2022-12-30 2023-06-23 魔珐(上海)信息科技有限公司 动画生成方法及装置，计算机可读存储介质、终端
CN117633296A (zh) * 2023-12-06 2024-03-01 浙江阿里巴巴机器人有限公司 视频生成及其模型训练方法、电子设备
CN117896592A (zh) * 2023-12-26 2024-04-16 北京字跳网络技术有限公司 视频生成方法、装置、可读介质及电子设备
CN118138854A (zh) * 2024-03-28 2024-06-04 云知声智能科技股份有限公司 视频生成方法、装置、计算机设备及介质
CN118229815A (zh) * 2024-03-19 2024-06-21 北京百度网讯科技有限公司 视频生成方法、深度学习模型的训练方法、装置、设备和存储介质

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210150357A1 (en) * 2018-11-14 2021-05-20 Nvidia Corporation Smoothing regularization for a generative neural network
CN110047121A (zh) * 2019-03-20 2019-07-23 北京字节跳动网络技术有限公司 端到端的动画生成方法、装置及电子设备
CN116309965A (zh) * 2022-12-30 2023-06-23 魔珐(上海)信息科技有限公司 动画生成方法及装置，计算机可读存储介质、终端
CN117633296A (zh) * 2023-12-06 2024-03-01 浙江阿里巴巴机器人有限公司 视频生成及其模型训练方法、电子设备
CN117896592A (zh) * 2023-12-26 2024-04-16 北京字跳网络技术有限公司 视频生成方法、装置、可读介质及电子设备
CN118229815A (zh) * 2024-03-19 2024-06-21 北京百度网讯科技有限公司 视频生成方法、深度学习模型的训练方法、装置、设备和存储介质
CN118138854A (zh) * 2024-03-28 2024-06-04 云知声智能科技股份有限公司 视频生成方法、装置、计算机设备及介质

Also Published As

Publication number Publication date
CN121217962A (zh) 2025-12-26

Similar Documents

Publication Publication Date Title
US10620948B2 (en) Application system for multiuser creating and editing of applications
US10950020B2 (en) Real-time AR content management and intelligent data analysis system
US20180143741A1 (en) Intelligent graphical feature generation for user content
CN111970571B (zh) 视频制作方法、装置、设备及存储介质
JP7337172B2 (ja) 音声パケット推薦方法、装置、電子機器およびプログラム
WO2015018299A1 (zh) 用户界面实现方法及装置
US20250005836A1 (en) Dynamic real time avatar-based ai communication system
CN115965791A (zh) 图像生成方法、装置及电子设备
WO2025260813A1 (zh) 视频生成方法、电子设备及计算机可读存储介质
CN112040311A (zh) 视频图像补帧方法、装置、设备及可存储介质
US12499515B2 (en) System and method for efficient scene continuity in visual and multimedia using generative artificial intelligence
CN116437153A (zh) 虚拟模型的预览方法、装置、电子设备及存储介质
CN116233561A (zh) 一种虚拟礼物生成方法、装置、设备及介质
WO2025067533A1 (zh) 一种文字处理方法、装置、计算机设备、存储介质及计算机程序产品
CN115136595B (zh) 用于流式传输沉浸式媒体的方法及装置
US20140282000A1 (en) Animated character conversation generator
CN120429508A (zh) 模型训练、资源推荐方法及设备
WO2026001087A1 (zh) 视频生成方法、装置、设备及介质
EP3389049B1 (en) Enabling third parties to add effects to an application
CN120010839A (zh) 基于基础大语言模型的前端页面代码生成方法及装置
CN117714813A (zh) 视频生成方法、装置、介质及设备
CN116977159A (zh) 图像处理方法、装置、电子设备及计算机可读存储介质
CN116600173A (zh) 元宇宙直播间的虚拟资源的编辑方法、装置、设备及介质
CN116309970A (zh) 车用虚拟数字形象生成方法、装置、电子设备及存储介质
WO2026001080A1 (zh) 视频生成方法、装置、设备及介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 25824482

Country of ref document: EP

Kind code of ref document: A1