WO2025201082A1 - 视频生成的方法、装置、设备和存储介质 - Google Patents

视频生成的方法、装置、设备和存储介质

Info

Publication number
WO2025201082A1
WO2025201082A1 PCT/CN2025/082436 CN2025082436W WO2025201082A1 WO 2025201082 A1 WO2025201082 A1 WO 2025201082A1 CN 2025082436 W CN2025082436 W CN 2025082436W WO 2025201082 A1 WO2025201082 A1 WO 2025201082A1
Authority
WO
WIPO (PCT)
Prior art keywords
video
predicted
model
quality
sample
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/CN2025/082436
Other languages
English (en)
French (fr)
Inventor
陈伟峰
吴捷
吉雅太
肖学锋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Zitiao Network Technology Co Ltd
Original Assignee
Beijing Zitiao Network Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Zitiao Network Technology Co Ltd filed Critical Beijing Zitiao Network Technology Co Ltd
Publication of WO2025201082A1 publication Critical patent/WO2025201082A1/zh
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/23418—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234336—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by media transcoding, e.g. video is transformed into a slideshow of still pictures or audio is converted into text
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/44008—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
    • H04N21/440236—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by media transcoding, e.g. video is transformed into a slideshow of still pictures, audio is converted into text

Definitions

  • Example embodiments of the present disclosure generally relate to the field of computer technology, and more particularly, to methods, devices, apparatuses, and computer-readable storage media for video generation.
  • Text-to-video generation allows users to input text, and the model can automatically generate a video corresponding to the text or edit existing videos.
  • Current methods face significant challenges in video generation continuity (consistency between the subject and background, and the ability to preserve motion patterns) and video quality (both subjective and objective quality).
  • a method for video generation includes: obtaining input information for a trained video generation model, the input information including at least a text description of a target video to be generated; and generating the target video based on the input information using the trained video generation model, wherein the video generation model is trained by: determining a reward score for the predicted video based on at least one of quality of the predicted video and motion consistency between the predicted video and a sample video, the predicted video being generated by the trained video generation model based on at least a sample text description matching the sample video, and training the video generation model based on a predetermined training objective, wherein the training objective is configured to increase or maximize the reward score.
  • an electronic device in a third aspect of the present disclosure, includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.
  • a computer program product which includes a computer program, and when the computer program is executed by a processor, the method of the first aspect is implemented.
  • FIG1 shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented
  • FIG2 shows a schematic diagram of the input and architecture of a video generation model according to some embodiments of the present disclosure
  • FIG4 shows a reward feedback learning algorithm according to some embodiments of the present disclosure
  • FIG5 shows a schematic diagram of an environment in which embodiments of the present disclosure can be implemented
  • FIG7 shows a block diagram of an apparatus for video generation according to some embodiments of the present disclosure.
  • the term “including” and similar terms should be understood as open inclusion, i.e., “including but not limited to”.
  • the term “based on” should be understood as “based at least in part on”.
  • the term “one embodiment” or “the embodiment” should be understood as “at least one embodiment”.
  • the term “some embodiments” should be understood as “at least some embodiments”.
  • Other explicit and implicit definitions may be included below.
  • a prompt message is sent to the user to clearly remind the user that the operation requested to be performed will require obtaining and using the user's personal information, so that the user can independently choose whether to provide personal information to the electronic device, application, server or storage medium and other software or hardware that performs the operation of the technical solution of the present disclosure based on the prompt message.
  • a prompt message in response to receiving a user's active request, may be sent to the user, for example, in the form of a pop-up window, in which the prompt message may be presented in text form.
  • the pop-up window may also include a selection control for the user to select "agree” or “disagree” to provide personal information to the electronic device.
  • model can learn the association between corresponding inputs and outputs from training data, so that after training is completed, corresponding outputs can be generated for given inputs.
  • the generation of the model can be based on machine learning technology.
  • Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multiple layers of processing units.
  • a neural network model is an example of a model based on deep learning.
  • model may also be referred to as “machine learning model”, “learning model”, “machine learning network” or “learning network”, and these terms are used interchangeably in this article.
  • a “neural network” is a machine learning network based on deep learning.
  • a neural network is capable of processing inputs and providing corresponding outputs. It typically includes an input layer, an output layer, and one or more hidden layers between the input and output layers. Neural networks used in deep learning applications typically include many hidden layers, thereby increasing the depth of the network. The layers of a neural network are connected in sequence so that the output of the previous layer is provided as input to the next layer, where the input layer receives the input of the neural network and the output of the output layer serves as the final output of the neural network.
  • Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each of which processes the input from the previous layer.
  • machine learning can be roughly divided into three stages, namely the training stage, the testing stage, and the application stage (also called the inference stage).
  • the training stage a given model can be trained using a large amount of training data, and the parameter values are continuously updated iteratively until the model can obtain consistent inferences that meet the expected goals from the training data.
  • the model can be considered to be able to learn the association between input and output (also called input-to-output mapping) from the training data.
  • the parameter values of the trained model are determined.
  • the testing stage the test input is applied to the trained model to test whether the model can provide the correct output, thereby determining the performance of the model.
  • the testing stage can sometimes be integrated into the training stage.
  • the trained model can be used to process the actual model input based on the parameter values obtained through training to determine the corresponding model output.
  • Reinforcement learning also known as reinforcement learning, evaluation learning, or enhanced learning, is a machine learning technique used to describe and solve the problem of an intelligent agent learning strategies to maximize rewards or achieve specific goals during its interaction with the environment. Reinforcement learning focuses on the interaction between the agent and the environment, and its goal is generally to maximize rewards. In other words, reinforcement learning is a learning mechanism that learns how to map states to behaviors in order to maximize rewards. Such an intelligent agent needs to continuously experiment in the environment, continuously optimizing the state-behavior relationship through feedback (rewards) provided by the environment.
  • Reinforcement learning systems generally involve four elements: policy, reward, value, and environment or model. The following sections introduce these four elements separately.
  • a policy defines the actions a model should take in a given state, essentially mapping states to actions.
  • a state refers to the state perceived by the model.
  • the policy is the core of a reinforcement learning system, as it determines the actions to take in each state.
  • the policy itself can be a specific mapping or a random distribution.
  • Rewards define the objective of a reinforcement learning problem. At each time step, the environment sends a scalar value to the reinforcement learning system. Rewards determine how well a model performs. Therefore, reward signals are the primary factor influencing the policy. The model's task is to maximize the total reward accumulated over a period of time.
  • Value is a crucial concept in reinforcement learning. Unlike immediate rewards, a value function measures long-term benefits. It evaluates the benefits of a current action from a long-term perspective, rather than focusing solely on the immediate reward. Calculating the value function requires analyzing transitions between states.
  • the environment also known as the model, is used to predict the next state and corresponding reward after a state and action are given.
  • the main goal of the model is to interact with the environment at each time step (taking the state as input) to find the optimal policy ⁇ so as to reach the goal while maximizing the cumulative reward (expected return) over the entire time period.
  • the model takes the state s as input and returns the action a to be performed.
  • the expected return Rt is the sum of the rewards from the current time step to the last time step t.
  • model training through reinforcement learning (RL) and feedback is also proposed.
  • Such a training scheme is also called reinforcement learning from human feedback (RLHF).
  • the target model to be trained is also called the action model (actor model), which is used to map the state s in the reinforcement learning environment to the action a.
  • the state s in reinforcement learning corresponds to the model input, and the action a corresponds to the model output.
  • the output of the target model is the action logic, which includes the score determined by the target model for each potential action based on the input, and the action with the highest score is determined as the final selection of the target model.
  • the reward model is configured to determine the reward score of the output estimated by the target model for the input.
  • the reward model is used to evaluate the action a and state s of the target model, and determine the reward score based on the quality of the model output. In RLHF, it is expected to maximize the reward score.
  • electronic device 110 can be any type of device with computing capabilities, including terminal devices or server devices.
  • the terminal device can be any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, desktop computer, laptop computer, notebook computer, netbook computer, tablet computer, media computer, multimedia tablet, personal communication system (PCS) device, personal navigation device, personal digital assistant (PDA), audio/video player, digital camera/camcorder, positioning device, television receiver, radio broadcast receiver, e-book device, gaming device or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof.
  • the server device can include, for example, a computing system/server, such as a mainframe, an edge computing node, a computing device in a cloud environment, and the like.
  • T2I text-to-image
  • T2V text-to-video
  • Video quality can include both technical quality and aesthetic quality.
  • Technical quality is characterized by fewer artifacts and reduced blur, while aesthetic quality is measured subjectively by human perception of visual appeal.
  • Motion consistency includes object consistency (the subject and background remain unchanged between frames) and motion smoothness (the motion follows physical principles).
  • an improved video generation scheme is proposed. Specifically, input information for a trained video generation model is obtained, and the input information at least includes a text description of a target video to be generated.
  • a target video is generated based on the input information, and the video generation model is trained by: determining a reward score for the predicted video based on at least one of the quality of the predicted video and the motion consistency between the predicted video and the sample video, respectively, the predicted video is generated by the video generation model being trained based on at least a sample text description that matches the sample video, and training the video generation model based on a predetermined training target, and the training target is configured to increase or maximize the reward score.
  • the reward score of the predicted video is determined based on the quality of the predicted video and the motion consistency between the predicted video and the motion video, and the optimization direction of the video generation model is guided by reward feedback learning, thereby improving the quality and motion consistency of the generated predicted video.
  • FIG2 shows a schematic diagram 200 of the input and architecture of the video generation model 120 according to some embodiments of the present disclosure.
  • the input information at least includes a text description 205 of the target video 112 to be generated.
  • input information 102 also includes a reference video 210 used as a control condition for video generation.
  • Reference video 210 may include multiple frames.
  • a control map extractor 215 may be used to extract reference control maps from reference video 210, such as edge maps and depth maps 220.
  • the extracted edge maps or depth maps 220 provide rich information about the three-dimensional structure of the scene, highlighting structural features and simplifying image information, thereby reducing the complexity of subsequent processing and analysis and improving computational efficiency.
  • the video generation model 120 includes a diffusion probability model
  • the input information 102 also includes noise information sampled from a noise distribution.
  • the diffusion probability model will be briefly introduced below.
  • the backward denoising process is often learned by optimizing a variational constraint on the log-likelihood, which can be expressed as follows:
  • the model performing the backward denoising process can first start sampling from the noise distribution q noise (x (T) ) and iteratively denoise by using p ⁇ (x (t-1)
  • the first frame of the reference video 210 or the edge map or depth map 220 extracted from the reference video 210 can be introduced as a content prior to help generate more general videos.
  • noise information sampled from the noise distribution is added to each frame except the first frame, and the video generation model 120 uses the content prior to learn denoising.
  • the video generation model 120 can be trained using the following loss function:
  • the diffusion probability model ⁇ predicts the noise at time step t based on the conditional text description c p , the control map c f , and the input x t of the first frame v 1 . Based on this, the video generation model 120 does not need to memorize the video content in the training set, but instead focuses on learning to reconstruct motion, achieving better generation results with fewer training resources.
  • the video generation model also includes an attention-based module.
  • a one-dimensional temporal attention layer/convolution layer 235 is added for each two-dimensional spatial attention layer/convolution layer 230.
  • a spatiotemporal self-attention mechanism 240 can be employed, in which spatial and temporal relationships are jointly modeled to capture dependencies across frames. Each frame is processed by a two-dimensional spatial attention layer/convolution layer 230, and then these frame-level features are passed together to a trainable one-dimensional temporal attention layer/convolution layer 235 for frame modeling.
  • the spatial self-attention mechanism can be adjusted by merging spatiotemporal self-attention across frames, which can be formulated as:
  • the input information 102 may also include a motion prior 245.
  • the motion prior 245 may include residuals.
  • a residual-based noise prior can be used. Specifically, this can be accomplished by calculating pixel residuals between consecutive frames and then initializing the noise distribution accordingly after downsampling. This ensures that static regions exhibit consistent noise, while dynamic regions exhibit different noise patterns.
  • a threshold can be used to distinguish between static and dynamic regions, thereby providing control over the smoothness of the generated video.
  • motion prior 245 can include optical flow information.
  • a noise prior based on optical flow can be introduced.
  • optical flow can be calculated between consecutive frames in pixel space and then downsampled to the latent space. This can improve the consistency of the video generated by the model.
  • the trained video generation model 120 can be used to generate a target video 112 based on the input information 102.
  • the video generation model 120 is trained in the following manner: based on at least one of the quality of the predicted video and the motion consistency between the predicted video and the sample video, a reward score of the predicted video is determined.
  • the predicted video is generated by the video generation model 120 being trained based on at least a sample text description that matches the sample video.
  • the video generation model is trained based on a predetermined training objective, and the training objective is configured to increase or maximize the reward score.
  • the sample video here can be a true value video used to guide the training process of the video generation model 120. Under a training strategy with the optimization goal of improving or maximizing the reward score, the video generation model 120 can learn the sample video and its corresponding text description, so that when given a new, unseen input, it can generate a predicted video close to the sample video.
  • a quality reward model can be used to determine a quality reward score for a predicted video based on the quality of the predicted video.
  • the quality of the predicted video here can include subjective quality and objective quality. The following describes how to determine a reward score for a predicted video using a reward model with reference to FIG3 , which shows a schematic diagram 300 of a training video generation model 120 according to some embodiments of the present disclosure.
  • a subjective quality reward model 310 can be used to determine a first quality reward score for the predicted video 305.
  • Subjective quality reward model 310 can be trained in a supervised manner using a training set of images or videos, including data such as the ground-truth quality reward scores for each image or video in the training set.
  • the ground-truth quality reward scores are provided by users and are subjectively measured based on human perception of visual appeal.
  • Subjective quality reward model 310 also known as an aesthetic scoring model, is trained using human rating data on images, with higher scores associated with higher aesthetic appeal.
  • an objective quality reward model 315 is used to determine a second quality reward score for the predicted video 305.
  • the objective quality reward model is configured to determine the quality reward score based on the presence of at least one quality-influencing factor in the input video.
  • quality-influencing factors may include image artifacts, noise, blur, etc. If a quality-influencing factor is present in the predicted video 305, the video score is low; otherwise, the score is high. It should be understood that more, fewer, or different objective quality-influencing factors may be provided as needed to objectively assess video quality.
  • the loss function for quality can be expressed as a weighted sum of a subjective quality reward score and an objective quality reward score:
  • R qt represents the objective quality reward score of each frame
  • R qa represents the subjective quality reward score of each frame
  • b qt represents the upper bound of the objective quality reward model 315
  • b qa represents the upper bound of the subjective quality reward model 310
  • ⁇ qt and ⁇ qa represent weights respectively.
  • the quality reward model can guide the learning direction of the video generation model 120 and improve the training efficiency, thereby generating a predicted video 305 with better quality.
  • a motion reward score for the predicted video 305 can be determined based on the motion consistency between the predicted video 305 and the sample video.
  • the training objective is configured to increase or maximize the quality reward score and increase or maximize the motion reward score.
  • sample optical flow information of the sample video and predicted optical flow information of the predicted video 305 can be determined, and a first motion reward score for the predicted video 305 can be determined based on the difference between the predicted optical flow information and the sample optical flow information.
  • the optical flow between each frame of the training input video i.e., the sample video before noise addition in the pre-training phase
  • the optical flow between each frame of the output video i.e., the predicted video 305
  • the difference between the output optical flow and the input optical flow is calculated and used as the reward score. The smaller the difference between the output optical flow and the input optical flow, the higher the reward score.
  • a motion reward model 320 may be used to determine a first motion reward score for the predicted video 305.
  • the motion reward model 320 is configured to determine the first motion reward score based on a difference between the predicted optical flow information and the sample optical flow information.
  • the loss function for motion consistency may be expressed as a weighted sum of the first motion reward score and the second motion reward score:
  • v represents a sample video
  • R mf and R mr represent the motion reward score based on optical flow (first motion reward score) and the motion reward score based on optical flow (second motion reward score), respectively
  • ⁇ mf and ⁇ mr represent weights, respectively.
  • the total spatiotemporal reward loss can be obtained.
  • the total spatiotemporal reward loss can be expressed as the sum of the above motion loss and mass:
  • FIG. 5 shows a schematic diagram of an environment 500 in which embodiments of the present disclosure can be implemented.
  • the model is generally shown to involve different stages, including a training stage 502 and an application stage 506. After the training stage is completed, there may also be a testing stage, which is not shown in the figure.
  • the obtained model 505 has trained parameter values and can be provided to the model application system 530 for use.
  • the model 505 can be used to process the corresponding target input 532 in the actual scene and provide the corresponding target output 534.
  • the model application system 530 can be configured to implement the electronic device 110 of Figure 1.
  • the model training system 510 and the model application system 530 may include any computing system with computing capabilities, such as various computing devices/systems, terminal devices, servers, etc.
  • Terminal devices may include any type of mobile, fixed, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, or any combination thereof, including accessories and peripherals of these devices or any combination thereof.
  • Servers include, but are not limited to, mainframe computers, edge computing nodes, computing devices in cloud environments, etc.
  • the components and arrangements in the environment 500 shown in FIG5 are merely examples, and a computing system suitable for implementing the exemplary implementations described in the present disclosure may include one or more different components, other components, and/or different arrangements.
  • a computing system suitable for implementing the exemplary implementations described in the present disclosure may include one or more different components, other components, and/or different arrangements.
  • the model training system 510 and the model application system 530 may be integrated into the same system or device. Implementations of the present disclosure are not limited in this respect.
  • the electronic device 110 obtains input information for a trained video generation model, where the input information includes at least a text description of a target video to be generated.
  • the electronic device 110 generates a target video based on the input information using a trained video generation model, wherein the video generation model is trained by determining a reward score for the predicted video based on at least one of the quality of the predicted video and the consistency of motion between the predicted video and the sample video, wherein the predicted video is generated by the trained video generation model based on at least a sample text description that matches the sample video, and training the video generation model based on a predetermined training objective, wherein the training objective is configured to increase or maximize the reward score.
  • the training of the video generation model can be implemented locally on the electronic device 110 or can be implemented remotely. In the case of remote implementation, the electronic device 110 can obtain the trained video generation model from the remote device for use.
  • determining a reward score for a predicted video includes at least one of the following: determining a quality reward score for the predicted video based on the quality of the predicted video using a quality reward model; and determining a motion reward score for the predicted video based on motion consistency between the predicted video and a sample video; and wherein the training objective is configured to increase or maximize the quality reward score and increase or maximize the motion reward score.
  • determining a quality reward score for a predicted video includes at least one of the following: determining a first quality reward score for the predicted video using a subjective quality reward model, wherein the subjective quality reward model is supervisedly trained using the following training data: a training image set or a training video set, and a true quality reward score for each image in the training image set or each video in the training video set, wherein the true quality reward score is provided by a user; and determining a second quality reward score for the predicted video using an objective quality reward model, wherein the objective quality reward model is configured to determine the quality reward score based on the presence of at least one quality influencing factor in the input video.
  • determining the motion reward score of the predicted video includes: determining sample optical flow information of the sample video and predicted optical flow information of the predicted video; and determining a first motion reward score of the predicted video based on the difference between the predicted optical flow information and the sample optical flow information.
  • determining the motion reward score of the predicted video includes: determining sample residual information of the sample video based on pixel differences between consecutive video frames of the sample video; determining prediction residual information of the predicted video based on pixel differences between consecutive video frames of the predicted video; and determining a second motion reward score of the predicted video based on the difference between the prediction residual information and the sample residual information.
  • the input information further includes a reference video used as a control condition for video generation, or includes at least one of an edge map and a depth map extracted from the reference video.
  • FIG. 7 shows a block diagram of an apparatus 700 for video generation according to some embodiments of the present disclosure.
  • Apparatus 700 may be implemented as or included in electronic device 110 of Figure 1.
  • Each module/component in apparatus 700 may be implemented by hardware, software, firmware, or any combination thereof.
  • the device 700 includes an input information acquisition module 710, which is configured to obtain input information for a trained video generation model, wherein the input information includes at least a text description of a target video to be generated.
  • the device 700 also includes a target video generation module 720, which is configured to generate a target video based on the input information using the trained video generation model, wherein the video generation model is trained by: determining a reward score for the predicted video based on at least one of the quality of the predicted video and the motion consistency between the predicted video and a sample video, wherein the predicted video is generated by the video generation model being trained based on at least a sample text description that matches the sample video, and training the video generation model based on a predetermined training objective, wherein the training objective is configured to increase or maximize the reward score.
  • the target video generation module 720 includes a reward score determination module configured to determine a quality reward score of the predicted video based on the quality of the predicted video using a quality reward model; and to determine a motion reward score of the predicted video based on the motion consistency between the predicted video and the sample video; and wherein the training objective is configured to increase or maximize the quality reward score and increase or maximize the motion reward score.
  • the reward score determination module includes a quality reward score determination module, which is configured to determine a first quality reward score for the predicted video using a subjective quality reward model, wherein the subjective quality reward model is supervisedly trained using the following training data: a training image set or a training video set, and a true quality reward score for each image in the training image set or each video in the training video set, wherein the true quality reward score is provided by a user; and determine a second quality reward score for the predicted video using an objective quality reward model, wherein the objective quality reward model is configured to determine the quality reward score based on the presence of at least one quality influencing factor in the input video.
  • the reward score determination module includes a first motion reward score determination module, which is configured to determine sample optical flow information of the sample video and predicted optical flow information of the predicted video; and determine a first motion reward score of the predicted video based on the difference between the predicted optical flow information and the sample optical flow information.
  • the reward score determination module includes a second motion reward score determination module, which is configured to determine sample residual information of the sample video based on pixel differences between consecutive video frames of the sample video; determine prediction residual information of the predicted video based on pixel differences between consecutive video frames of the predicted video; and determine a second motion reward score of the predicted video based on the difference between the prediction residual information and the sample residual information.
  • a second motion reward score determination module which is configured to determine sample residual information of the sample video based on pixel differences between consecutive video frames of the sample video; determine prediction residual information of the predicted video based on pixel differences between consecutive video frames of the predicted video; and determine a second motion reward score of the predicted video based on the difference between the prediction residual information and the sample residual information.
  • the input information further includes a reference video used as a control condition for video generation, or includes at least one of an edge map and a depth map extracted from the reference video.
  • the video generation model also includes an attention-based module.
  • FIG8 shows a block diagram of an electronic device 800 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the electronic device 800 shown in FIG8 is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. The electronic device 800 shown in FIG8 may be used to implement the electronic device 110 of FIG1 or the apparatus 700 of FIG7.
  • electronic device 800 is in the form of a general-purpose computing device.
  • Components of electronic device 800 may include, but are not limited to, one or more processors or processing units 810, memory 820, storage device 830, one or more communication units 840, one or more input devices 850, and one or more output devices 860.
  • Processing unit 810 may be a real or virtual processor and is capable of performing various processes according to programs stored in memory 820. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to enhance the parallel processing capabilities of electronic device 800.
  • the electronic device 800 typically includes a plurality of computer storage media. Such media can be any available media accessible to the electronic device 800, including but not limited to volatile and non-volatile media, removable and non-removable media.
  • the memory 820 can be a volatile memory (e.g., registers, cache, random access memory (RAM)), a non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof.
  • the storage device 830 can be a removable or non-removable medium and can include a machine-readable medium, such as a flash drive, a disk, or any other medium that can be used to store information and/or data and can be accessed within the electronic device 800.
  • the electronic device 800 may further include additional removable/non-removable, volatile/non-volatile storage media.
  • a disk drive for reading from or writing to a removable, non-volatile disk (e.g., a “floppy disk”) and an optical drive for reading from or writing to a removable, non-volatile optical disk may be provided.
  • each drive may be connected to a bus (not shown) by one or more data media interfaces.
  • the memory 820 may include a computer program product 825 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.
  • the communication unit 840 enables communication with other electronic devices via a communication medium. Additionally, the functions of the components of the electronic device 800 can be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the electronic device 800 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.
  • PC network personal computer
  • the input device 850 may be one or more input devices, such as a mouse, keyboard, or trackball.
  • the output device 860 may be one or more output devices, such as a display, a speaker, or a printer.
  • the electronic device 800 may also communicate with one or more external devices (not shown) via the communication unit 840 as needed, such as a storage device, a display device, or the like, with one or more devices that allow a user to interact with the electronic device 800, or with any device that allows the electronic device 800 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input/output (I/O) interface (not shown).
  • I/O input/output
  • a computer-readable storage medium on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above.
  • a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.
  • These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions/actions specified in one or more blocks in the flowchart and/or block diagram.
  • These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and/or other device to operate in a specific manner.
  • the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions/actions specified in one or more blocks in the flowchart and/or block diagram.
  • Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions/actions specified in one or more boxes in the flowchart and/or block diagram.
  • each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and a part for a module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification.
  • the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved.
  • each box in the block diagram and/or flow chart, and the combination of the boxes in the block diagram and/or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Image Analysis (AREA)

Abstract

根据本公开的实施例，提供了视频生成的方法、装置、设备和存储介质。该方法包括：获得针对经训练的视频生成模型的输入信息，输入信息至少包括对要生成的目标视频的文本描述；以及利用经训练的视频生成模型，基于输入信息来生成目标视频，视频生成模型通过以下被训练：基于预测视频的质量和预测视频与样本视频之间运动一致性中的至少一项，分别确定预测视频的奖励得分，预测视频由正在训练的视频生成模型至少基于与样本视频匹配的样本文本描述来生成，以及基于预定训练目标来训练视频生成模型，训练目标被配置为使奖励得分增加或最大化。

Description

视频生成的方法、装置、设备和存储介质
本申请要求2024年03月25日递交的、标题为“视频生成的方法、装置、设备和存储介质”、申请号为202410346959.4的中国发明专利申请的优先权，该申请的全部内容通过引用结合在本申请中。
技术领域
本公开的示例实施例总体涉及计算机技术领域，特别地涉及视频生成的方法、装置、设备和计算机可读存储介质。
背景技术
近年来，基于文本的视觉内容生成领域快速发展。文本生成视频允许用户输入文本，模型可以自动生成对应文本的视频或者编辑已有的视频。当前方法在视频生成的连续性(主体、背景前后的一致性，运动规律的保持能力)和视频生成质量(视频的主观质量和客观质量)存在很大的挑战。
发明内容
在本公开的第一方面，提供了一种视频生成的方法。该方法包括：获得针对经训练的视频生成模型的输入信息，输入信息至少包括对要生成的目标视频的文本描述；以及利用经训练的视频生成模型，基于输入信息来生成目标视频，视频生成模型通过以下被训练：基于预测视频的质量和预测视频与样本视频之间运动一致性中的至少一项，分别确定预测视频的奖励得分，预测视频由正在训练的视频生成模型至少基于与样本视频匹配的样本文本描述来生成，以及基于预定训练目标来训练视频生成模型，训练目标被配置为使奖励得分增加或最大化。
在本公开的第二方面，提供了一种视频生成的装置。该装置包括：输入信息获得模块，被配置为获得针对经训练的视频生成模型的输入信息，输入信息至少包括对要生成的目标视频的文本描述；以及目标视频生成模块，被配置为利用经训练的视频生成模型，基于输入信息来生成目标视频，视频生成模型通过以下被训练：基于预测视频的质量和预测视频与样本视频之间运动一致性中的至少一项，分别确定预测视频的奖励得分，预测视频由正在训练的视频生成模型至少基于与样本视频匹配的样本文本描述来生成，以及基于预定训练目标来训练视频生成模型，训练目标被配置为使奖励得分增加或最大化。
在本公开的第三方面，提供了一种电子设备。该设备包括至少一个处理单元；以及至少一个存储器，至少一个存储器被耦合到至少一个处理单元并且存储用于由至少一个处理单元执行的指令。指令在由至少一个处理单元执行时使设备执行第一方面的方法。
在本公开的第四方面，提供了一种计算机可读存储介质。该介质上存储有计算机程序，计算机程序被处理器执行时实现第一方面的方法。
在本公开的第五方面，提供了一种计算机程序产品。该计算机程序产品包括计算机程序，计算机程序被处理器执行时实现第一方面的方法。
应当理解，该部分中所描述的内容并非旨在限定本公开的实施例的关键特征或重要特征，也不用于限制本公开的范围。本公开的其它特征将通过以下的描述而变得容易理解。
附图说明
结合附图并参考以下详细说明，本公开各实施例的上述和其他特征、优点及方面将变得更加明显。在附图中，相同或相似的附图标记表示相同或相似的元素，其中：
图1示出了能够在其中实现本公开的实施例的示例环境的示意图；
图2示出了根据本公开的一些实施例的视频生成模型的输入及架构的示意图；
图3示出了根据本公开的一些实施例的训练视频生成模型的示意图；
图4示出了根据本公开一些实施例的奖励反馈学习算法；
图5示出了能够在其中实现本公开的实施例的环境的示意图；
图6示出了根据本公开的一些实施例的用于模型训练的过程的示意图；
图7示出了根据本公开的一些实施例的用于视频生成的装置的框图；以及
图8示出了其中可以实现本公开的一个或多个实施例的电子设备的框图。
具体实施方式
下面将参照附图更详细地描述本公开的实施例。虽然附图中示出了本公开的某些实施例，然而应当理解的是，本公开可以通过各种形式来实现，而且不应该被解释为限于这里阐述的实施例，相反，提供这些实施例是为了更加透彻和完整地理解本公开。应当理解的是，本公开的附图及实施例仅用于示例性作用，并非用于限制本公开的保护范围。
在本公开的实施例的描述中，术语“包括”及其类似用语应当理解为开放性包含，即“包括但不限于”。术语“基于”应当理解为“至少部分地基于”。术语“一个实施例”或“该实施例”应当理解为“至少一个实施例”。术语“一些实施例”应当理解为“至少一些实施例”。下文还可能包括其他明确的和隐含的定义。
可以理解的是，本技术方案所涉及的数据(包括但不限于数据本身、数据的获得或使用)应当遵循相应法律法规及相关规定的要求。
可以理解的是，在使用本公开各实施例公开的技术方案之前，均应当根据相关法律法规通过适当的方式对本公开所涉及个人信息的类型、使用范围、使用场景等告知用户并获得用户的授权。
例如，在响应于接收到用户的主动请求时，向用户发送提示信息，以明确地提示用户，其请求执行的操作将需要获得和使用到用户的个人信息，从而使得用户可以根据提示信息来自主地选择是否向执行本公开技术方案的操作的电子设备、应用程序、服务器或存储介质等软件或硬件提供个人信息。
作为一种可选的但非限制性的实现方式，响应于接收到用户的主动请求，向用户发送提示信息的方式，例如可以是弹窗的方式，弹窗中可以以文字的方式呈现提示信息。此外，弹窗中还可以承载供用户选择“同意”或“不同意”向电子设备提供个人信息的选择控件。
可以理解的是，上述通知和获得用户授权过程仅是示意性的，不对本公开的实现方式构成限定，其他满足相关法律法规的方式也可应用于本公开的实现方式中。
如本文中所使用的，术语“模型”可以从训练数据中学习到相应的输入与输出之间的关联关系，从而在训练完成后可以针对给定的输入，生成对应的输出。模型的生成可以基于机器学习技术。深度学习是一种机器学习算法，通过使用多层处理单元来处理输入和提供相应输出。神经网络模型是基于深度学习的模型的一个示例。在本文中，“模型”也可以被称为“机器学习模型”、“学习模型”、“机器学习网络”或“学习网络”，这些术语在本文中可互换地使用。
“神经网络”是一种基于深度学习的机器学习网络。神经网络能够处理输入并且提供相应输出，其通常包括输入层和输出层以及在输入层与输出层之间的一个或多个隐藏层。在深度学习应用中使用的神经网络通常包括许多隐藏层，从而增加网络的深度。神经网络的各个层按顺序相连，从而前一层的输出被提供作为后一层的输入，其中输入层接收神经网络的输入，而输出层的输出作为神经网络的最终输出。神经网络的每个层包括一个或多个节点(也称为处理节点或神经元)，每个节点处理来自上一层的输入。
通常，机器学习大致可以包括三个阶段，即训练阶段、测试阶段和应用阶段(也称为推理阶段)。在训练阶段，给定的模型可以使用大量的训练数据进行训练，不断迭代更新参数值，直到模型能够从训练数据中获得一致的满足预期目标的推理。通过训练，模型可以被认为能够从训练数据中学习从输入到输出之间的关联(也称为输入到输出的映射)。训练后的模型的参数值被确定。在测试阶段，将测试输入应用到训练后的模型，以测试模型是否能够提供正确的输出，从而确定模型的性能。测试阶段有时候可以被融合在训练阶段。在应用或推理阶段，已训练的模型可以被用于基于训练得到的参数值，对实际的模型输入进行处理，确定对应的模型输出。
强化学习，又称再励学习、评价学习或增强学习，是机器学习技术之一，用于描述和解决智能体在与环境的交互过程中通过学习策略以达成回报最大化或实现特定目标的问题。强化学习关注智能体与环境之间的交互，并且强化学习的目标一般是追求最大回报。换句话说，强化学习是一种学习如何从状态映射到行为以使得获取的奖励最大的学习机制。这样的智能体需要不断地在环境中进行实验，通过环境给予的反馈(奖励)来不断优化状态-行为的对应关系。
强化学习系统一般涉及四个要素：策略(policy)，奖励(reward)，价值(value)以及环境或者说是模型(model)。接下来对这四个要素分别进行介绍。
策略定义了模型在给定状态下所做出的动作，即从状态到动作的映射。状态指的是模型所感知到的状态。通常，策略是强化学习系统的核心，因为通过策略可以确定在每个状态下的动作。根据配置需要，策略本身可以是具体的映射也可以是随机的分布。
奖励定义了强化学习问题的目标。在每个时间步骤内，环境向强化学习发出的标量值即为奖励。奖励能够定义模型表现好坏。因此奖励信号是影响策略的主要因素。模型的任务就是使得一个时段内积累的总奖励值最大。
价值(Value)，或者说价值函数，这是强化学习中非常重要的概念，与奖励的即时性不同，价值函数是对长期收益的衡量。从一个长期的角度来评判当前动作的收益，而不仅仅盯着眼前的奖励。价值函数的计算需要对状态之间的转移进行分析。
环境，也称为模型。在给出了状态与动作后，通过环境可以预测接下来的状态和对应的奖励。
假设S是有限状态空间，A是每个状态s∈S的动作空间；p是从第t步的状态st到第t+1步状态st+1的状态转移概率，R是动作a∈A执行后获得的即时奖励值。模型的主要目标是在每个时间步骤中与环境交互(将状态作为输入)，以找到最优策略π，以便在整个时间段内最大化累积奖励(预期回报)的同时达到目标。模型将状态s作为输入，并返回要执行的动作a。在特定时间步t，预期返回Rt是当前时间步到最后时间步t的奖励总和。在采取动作时，模型基于之前的经验(利用)和收集新的经验(探索)之间做出选择，以便在未来做出更好的决策。
在强化学习基础上，还提出了通过强化学习(RL)和反馈的方式的模型训练，这样的训练方案也称为来自人类反馈的强化学习(RLHF)。在RLHF的模型训练系统中，要训练的目标模型也称为动作模型(actor model)，其用于将强化学习环境中的状态s映射到动作a。强化学习中的状态s对应于模型输入，动作a对应于模型输出。目标模型的输出是动作逻辑，其包括由目标模型在输入的基础上，针对每个潜在动作确定的得分，得分最高的动作被确定为目标模型的最终选择的生成。奖励模型(reward model)被配置为确定目标模型为输入所估计的输出的奖励得分。奖励模型用于评估目标模型的动作a和状态s，并基于模型输出的质量来确定奖励得分。在RLHF中，期望最大化奖励得分。
图1示出了本公开的实施例能够在其中实现的示例环境100的示意图。在环境100中，电子设备110可以利用视频生成模型120来执行视频生成任务。在一些实现中，电子设备110可以基于输入信息102，利用视频生成模型120来生成目标视频112。
在图1中，电子设备110可以是任何类型的具有计算能力的设备，包括终端设备或服务端设备。终端设备可以是任何类型的移动终端、固定终端或便携式终端，包括移动手机、台式计算机、膝上型计算机、笔记本计算机、上网本计算机、平板计算机、媒体计算机、多媒体平板、个人通信系统(PCS)设备、个人导航设备、个人数字助理(PDA)、音频/视频播放器、数码相机/摄像机、定位设备、电视接收器、无线电广播接收器、电子书设备、游戏设备或者前述各项的任何组合，包括这些设备的配件和外设或者其任何组合。服务端设备例如可以包括计算系统/服务器，诸如大型机、边缘计算节点、云环境中的计算设备，等等。
应当理解，仅出于示例性的目的描述环境100的结构和功能，而不暗示对于本公开的范围的任何限制。
当前的文本到图像(text-to-image，简称T2I)模型在大规模的图像-文本对上训练，展示了在用户提供的文本提示的指导下生成高质量图像的能力。基于这些预先训练的T2I模型，个性化生成和条件生成对生成的图像提供了更细粒度的控制。在图像生成方面取得的成功已无缝扩展到视频生成，其中文本到视频(text-to-video，简称T2V)模型在生成由文本提示驱动的连贯视频方面表现出色。
但是，T2V模型仍然面临着制作高质量和运动一致性视频的挑战。视频质量可以包括技术质量和美学质量，技术质量以较少的瑕疵和减少的模糊为特征，美学质量以人类视觉吸引力的感知为主观衡量标准。运动一致性包括对象一致性(在帧之间主体和背景保持不变)和运动平滑性(运动遵循物理原理)。
为了提升视频生成的质量，在本公开的实施例中，提出了一种改进的视频生成方案。具体地，获得针对经训练的视频生成模型的输入信息，该输入信息至少包括对要生成的目标视频的文本描述。利用经训练的视频生成模型，基于输入信息来生成目标视频，视频生成模型通过以下被训练：基于预测视频的质量和预测视频与样本视频之间运动一致性中的至少一项，分别确定预测视频的奖励得分，预测视频由正在训练的视频生成模型至少基于与样本视频匹配的样本文本描述来生成，以及基于预定训练目标来训练视频生成模型，训练目标被配置为使奖励得分增加或最大化。
根据本公开的方案，基于预测视频的质量和预测视频与运动视频之间的运动一致性确定预测视频的奖励得分，通过奖励反馈学习的方式来指导视频生成模型的优化方向，从而可以提升生成的预测视频的质量和运动一致性。
以下将继续参考附图描述本公开的一些示例实施例。
图2示出了根据本公开的一些实施例的视频生成模型120的输入及架构的示意图200。
如图2所示，为了使视频生成模型120生成视频，需要获得针对经训练的视频生成模型120的输入信息。输入信息至少包括对要生成的目标视频112的文本描述205。
在一些实施例中，输入信息102还包括用作视频生成的控制条件的参考视频210。这里的参考视频210可以包括多帧图像。进一步地，可以利用控制图提取器215提取参考视频210的参考控制图，例如，边缘图和深度图220等。基于此，提取的边缘图或深度图220的提供了关于场景三维结构的丰富信息，突出了结构特征，简化了图像信息，从而有助于减少后续处理和分析的复杂性，提高计算效率。
在一些实施例中，视频生成模型120包括扩散概率模型，并且输入信息102还包括从噪声分布中采样的噪声信息。为更好理解，下文中将先简单介绍扩散概率模型。
扩散概率模型是生成式模型的一类，但其数据生成处理基于一对马尔科夫过程，即前向扩散过程和后向去噪过程。前向扩散过程(被表示为：是逐步干扰数据x(0)～q(x(0))，通过T个逐步加噪步骤x(1：T)＝x1,…,x(t-1),x(t),…,x(T)，得到静态噪声分布x(T)～qnoise。通过模型训练，所学习到的后向去噪过程(被表示为：执行相反的过程，逐步将样本朝向数据分布去噪，得到数据x(0)～q(x(0))。由此可见，后向去噪过程可以对应于期望的数据建模过程，最终得到期望的数据。
在一些实现中，为了将模型(表示为：pθ(x(0)))拟合到数据分布q(x(0))，后向去噪过程的学习通常是通过优化对数似然的变分约束来实现，这可以被表示为如下：
在完成学习后，执行后向去噪过程的模型能够首先从噪声分布qnoise(x(T))开始采样，并通过利用pθ(x(t-1)|x(t))进行迭代去噪，直到获得期望的数据。
为了继承图像领域的知识，可以引入参考视频210或者从参考视频210提取的边缘图或深度图220的第一帧作为内容先验，以帮助生成更通用的视频。具体地，向除了第一帧之外的每个帧添加从噪声分布中采样的噪声信息，视频生成模型120利用内容先验学习去噪。示例性地，可以使用以下公式化的损失函数训练视频生成模型120：
其中ε是真值噪音，扩散概率模型θ利用在时间步t，基于条件文本描述cp，控制图cf和第一帧v1的输入xt预测噪音。基于此，视频生成模型120不需要记忆训练集中的视频内容，而是关注学习重建运动，可以利用更少的训练资源实现更好的生成效果。
在一些实施例中，视频生成模型还包括基于注意力的模块。在经训练的视频生成模型120的架构225中，针对每个二维空间注意力层/卷积层230增加一维时间注意力层/卷积层235。为了进一步促进帧建模，可以采用时空自注意力机制240，其中空间和时间关系被联合建模，以捕获跨帧的依赖关系。每个帧经过二维的空间注意力层/卷积层230处理，随后这些帧级特征被共同传递到可训练的一维的时间注意力层/卷积层235用于帧建模。此外，为了实现细粒度建模，可以通过跨帧合并时空自注意力来调整空间自注意力机制，其可以公式化为：
其中表示帧i的令牌序列，表示N个帧的级联。在公式(4)中，连接N个帧的特征K、V，使得每个位置具有所有视频帧的全局感知，并倾向于生成更一致的结果。
在一些实施例中，输入信息102还可以包括运动先验245。这里的运动先验245可以包括残差。为了在静态区域中保持一致的噪声，并在动态区域中引入变化的噪声，可以使用了基于残差的噪声先验。具体地，可以通过计算连续帧之间的像素残差，然后在下采样后相应地初始化噪声分布。基于此，确保了静态区域表现出相同的噪声，而动态区域具有不同的噪声模式。此外，可以利用阈值区分静态区域和动态区域，从而提供对所生成视频的平滑度的控制。
备选地或附加地，运动先验245可以包括光流信息。为了使生成的视频流与帧中描绘的运动对齐，可以引入基于光流的噪声先验。具体地，可以计算像素空间中连续帧之间的光流，然后将光流信息下采样到潜在空间。基于此，可以提升模型生成的视频的一致性。
在获得了输入信息102之后，可以利用经训练的视频生成模型120，基于输入信息102来生成目标视频112。视频生成模型120通过以下方式被训练：基于预测视频的质量和预测视频与样本视频之间运动一致性中的至少一项，分别确定预测视频的奖励得分。预测视频由正在训练的视频生成模型120至少基于与样本视频匹配的样本文本描述来生成。基于预定训练目标来训练视频生成模型，训练目标被配置为使奖励得分增加或最大化。这里的样本视频可以是真值视频，用于指导视频生成模型120的训练过程。在以提高或最大化奖励得分为优化目标的训练策略下，视频生成模型120可以学习样本视频与其对应的文本描述，从而在给定新的、未见过的输入时，能够生成接近样本视频的预测视频。
在一些实施例中，可以利用质量奖励模型，基于预测视频的质量，确定预测视频的质量奖励得分。这里的预测视频的质量可以包括主观质量和客观质量。以下将参考图3说明利用奖励模型确定预测视频的奖励得分，图3示出了根据本公开的一些实施例的训练视频生成模型120的示意图300。
如图3所示，在一些实施例中，可以利用主观质量奖励模型310，确定预测视频305的第一质量奖励得分。主观质量奖励模型310可以使用训练图像集或训练视频集，针对训练图像集中的各个图像或训练视频集中的各个视频的真值质量奖励得分等数据来进行有监督训练。真值质量奖励得分由用户提供，根据人类视觉吸引力的感知进行主观测量。主观质量奖励模型310也可以被称为美学打分模型，该模型由人类对图像的打分数据训练而来，美感高则得分高。
在一些实施例中，利用客观质量奖励模型315，确定预测视频305的第二质量奖励得分。客观质量奖励模型被配置为基于输入视频中至少一个质量影响因素的存在来确定质量奖励得分。示例性地，质量影响因素可以包括残影、噪声、模糊等。如果预测视频305出现质量影响因素，则该视频的得分低，否则得分高。应当理解，根据需要，还可以设置更多、更少或不同的客观质量影响因素，用于从客观评估视频的质量。
在一些实施例中，针对质量的损失函数可以表示为主观质量奖励得分和客观质量奖励得分的加权和：
Lquality＝λqt·ReLU(bqt-Rqt(v))+，λqa·R，L(bqa-Rqa(v′)). (5)
其中Rqt表示逐帧的客观质量奖励得分，Rqa表示逐帧的主观质量奖励得分，bqt表示客观质量奖励模型315的上边界，bqa表示主观质量奖励模型310的上边界，λqt和λqa分别表示权重。
基于此，质量奖励模型可以指导视频生成模型120的学习方向，提高训练效率，从而可以生成质量更好的预测视频305。
在一些实施例中，可以基于预测视频305与样本视频之间运动一致性，确定预测视频305的运动奖励得分。训练目标被配置为使质量奖励得分增加或最大化，并且使运动奖励得分增加或最大化。
在一些实施例中，可以确定样本视频的样本光流信息和预测视频305的预测光流信息，并基于预测光流信息与样本光流信息之间的差异，确定预测视频305的第一运动奖励得分。可以训练输入视频(即，预训练阶段加噪前的样本视频)每帧之间的光流作为真值，输出视频(即，预测视频305)每帧之间的光流作为预测的运动信息。计算输出光流和输入光流的差异，以此作为奖励得分。输出光流和输入光流的差异越小，奖励得分越高。
在一些实施例中，可以利用运动奖励模型320，确定预测视频305的第一运动奖励得分。运动奖励模型320被配置为基于预测光流信息和样本光流信息的差异来确定第一运动奖励得分。
基于此，通过降低或最小化预测光流信息和样本光流信息的差异，可以生成一致性更好的预测视频305。
在一些实施例中，可以基于样本视频的连续视频帧之间的像素差异，确定样本视频的样本残差信息。基于预测视频305的连续视频帧之间的像素差异，确定预测视频305的预测残差信息。基于预测残差信息与样本残差信息之间的差异，确定预测视频305的第二运动奖励得分。这里的残差信息可以是视频帧中同一个位置的像素值差。可以将预测残差信息与样本残差信息之间的差异作为奖励得分，差异越小，奖励得分越高。
在一些实施例中，针对运动一致性的损失函数可以被表示为第一运动奖励得分和第二运动奖励得分的加权和：
Lmotion＝-λmr·Rmr(v，v′)-λmf·Rmf(v,v′)，  (6)
其中v表示样本视频，v′＝Decoder(x′0)表示预测视频305，Rmf和Rmr分别表示基于光流的运动奖励得分(第一运动奖励得分)和基于光流的运动奖励得分(第二运动奖励得分)，λmf和λmr分别表示权重。
基于此，通过降低或最小化预测残差信息与样本残差信息之间的差异，可以生成一致性更好的预测视频305。
在一些实施例中，在得到奖励得分并计算出损失之后，可以得到总的时空奖励损失。总的时空奖励损失可以被表示为以上运动损失和质量之和：
图4示出了根据本公开一些实施例的奖励反馈学习算法400。在一些实施例中，可以利用图4所示的奖励反馈学习的算法对视频生成模型120进行优化。继续参考图3，具体而言，可以随机采样一个噪声XT325，利用视频生成模型120其进行迭代推理得到Xt+1330，在该推理中，推理梯度不回传。接下来，继续进行推理得到Xt335，在该推理中，具有梯度。然后，从Xt335推理到X0340，并将X0340解码为预测视频305。将预测视频305输入到奖励模型得到奖励得分，然后计算出损失，并用该损失优化视频生成模型120。以此方式，经过优化后，视频的主观美学质量、客观质量、连续性都有较大的提升，如图3中的预测视频展示区域345所示。
应当理解，除图4所示的示例算法外，还可以通过任何其他训练和梯度回传算法来训练视频生成模型120。在一些实施例中，在视频生成模型120的训练过程中，各个奖励模型，例如主观质量奖励模型310、客观质量奖励模型315和运动奖励模型320的参数不变。也就是说，这些奖励模型可以被预先训练或配置好，以用于支持对视频生成模型120的训练。
图5示出了能够在其中实现本公开的实施例的环境500的示意图。在图5的环境500中，总体示出了模型涉及不同阶段，包括训练阶段502和应用阶段506。在训练阶段完成后还可以有测试阶段，图中未示出。
在训练阶段502，模型训练系统510被配置为利用训练数据集512来执行模型505的训练。模型505例如可以是图1中的视频生成模型120。在训练开始时，模型可以具有初始参数值。训练过程是要基于训练数据来将模型505的参数值更新到期望值。
在应用阶段506，所获得的模型505具有已训练的参数值，可以被提供给模型应用系统530来使用。在应用阶段506，可以利用模型505来处理实际场景中的对应目标输入532，并且提供相应的目标输出534。模型应用系统530可以被配置为实现图1的电子设备110。
在图5中，模型训练系统510和模型应用系统530可以包括具有计算能力的任何计算系统，例如各种计算设备/系统、终端设备、服务器等。终端设备可以涉及任意类型的移动终端、固定终端或便携式终端，包括移动手机、台式计算机、膝上型计算机、笔记本计算机、上网本计算机、平板计算机、媒体计算机、多媒体平板、或者前述各项的任意组合，包括这些设备的配件和外设或者其任意组合。服务器包括但不限于大型机、边缘计算节点、云环境中的计算设备，等等。
应当理解，图5示出的环境500中的部件和布置仅仅是示例，适于用于实现本公开所描述的示例性实现方式的计算系统可以包括一个或多个不同的部件、其他部件和/或不同的布置方式。例如，虽然被示出为是分离的，但模型训练系统510和模型应用系统530可以集成在相同系统或设备中。本公开的实现方式在此方面不受限制。
图6示出了根据本公开的一些实施例的用于视频生成的过程600的示意图。过程600可以被实现在图1的电子设备110处。
在框610，电子设备110获得针对经训练的视频生成模型的输入信息，输入信息至少包括对要生成的目标视频的文本描述。
在框620，电子设备110利用经训练的视频生成模型，基于输入信息来生成目标视频，视频生成模型通过以下被训练：基于预测视频的质量和预测视频与样本视频之间运动一致性中的至少一项，分别确定预测视频的奖励得分，预测视频由正在训练的视频生成模型至少基于与样本视频匹配的样本文本描述来生成，以及基于预定训练目标来训练视频生成模型，训练目标被配置为使奖励得分增加或最大化。视频生成模型的训练可以被实现在电子设备110本地或者可以被远程实现。在远程实现的情况下，电子设备110可以从远程设备获得经训练的视频生成模型来使用。
在一些实施例中，确定预测视频的奖励得分包括以下至少一项：利用质量奖励模型，基于预测视频的质量，确定预测视频的质量奖励得分；以及基于预测视频与样本视频之间运动一致性，确定预测视频的运动奖励得分；并且其中训练目标被配置为使质量奖励得分增加或最大化，并且使运动奖励得分增加或最大化。
在一些实施例中，确定预测视频的质量奖励得分包括以下至少一项：利用主观质量奖励模型，确定预测视频的第一质量奖励得分，其中主观质量奖励模型利用以下训练数据来被有监督训练：训练图像集或训练视频集，以及针对训练图像集中的各个图像或训练视频集中的各个视频的真值质量奖励得分，真值质量奖励得分由用户提供；以及利用客观质量奖励模型，确定预测视频的第二质量奖励得分，其中客观质量奖励模型被配置为基于输入视频中至少一个质量影响因素的存在来确定质量奖励得分。
在一些实施例中，确定预测视频的运动奖励得分包括：确定样本视频的样本光流信息和预测视频的预测光流信息；以及基于预测光流信息与样本光流信息之间的差异，确定预测视频的第一运动奖励得分。
在一些实施例中，确定预测视频的运动奖励得分包括：基于样本视频的连续视频帧之间的像素差异，确定样本视频的样本残差信息；基于预测视频的连续视频帧之间的像素差异，确定预测视频的预测残差信息；以及基于预测残差信息与样本残差信息之间的差异，确定预测视频的第二运动奖励得分。
在一些实施例中，输入信息还包括用作视频生成的控制条件的参考视频，或者包括从参考视频提取的边缘图和深度图中的至少一项。
在一些实施例中，视频生成模型还包括基于注意力的模块。
图7示出了根据本公开的一些实施例的用于视频生成的装置700的框图。装置700可以被实现为或者被包括在图1的电子设备110处。装置700中的各个模块/组件可以由硬件、软件、固件或者它们的任意组合来实现。
如图所示，装置700包括输入信息获得模块710，被配置为获得针对经训练的视频生成模型的输入信息，输入信息至少包括对要生成的目标视频的文本描述。装置700还包括目标视频生成模块720，被配置为利用经训练的视频生成模型，基于输入信息来生成目标视频，视频生成模型通过以下被训练：基于预测视频的质量和预测视频与样本视频之间运动一致性中的至少一项，分别确定预测视频的奖励得分，预测视频由正在训练的视频生成模型至少基于与样本视频匹配的样本文本描述来生成，以及基于预定训练目标来训练视频生成模型，训练目标被配置为使奖励得分增加或最大化。
在一些实施例中，目标视频生成模块720包括奖励得分确定模块，被配置为利用质量奖励模型，基于预测视频的质量，确定预测视频的质量奖励得分；以及基于预测视频与样本视频之间运动一致性，确定预测视频的运动奖励得分；并且其中训练目标被配置为使质量奖励得分增加或最大化，并且使运动奖励得分增加或最大化。
在一些实施例中，奖励得分确定模块包括质量奖励得分确定模块，被配置为利用主观质量奖励模型，确定预测视频的第一质量奖励得分，其中主观质量奖励模型利用以下训练数据来被有监督训练：训练图像集或训练视频集，以及针对训练图像集中的各个图像或训练视频集中的各个视频的真值质量奖励得分，真值质量奖励得分由用户提供；以及利用客观质量奖励模型，确定预测视频的第二质量奖励得分，其中客观质量奖励模型被配置为基于输入视频中至少一个质量影响因素的存在来确定质量奖励得分。
在一些实施例中，奖励得分确定模块包括第一运动奖励得分确定模块，被配置为确定样本视频的样本光流信息和预测视频的预测光流信息；以及基于预测光流信息与样本光流信息之间的差异，确定预测视频的第一运动奖励得分。
在一些实施例中，奖励得分确定模块包括第二运动奖励得分确定模块，被配置为基于样本视频的连续视频帧之间的像素差异，确定样本视频的样本残差信息；基于预测视频的连续视频帧之间的像素差异，确定预测视频的预测残差信息；以及基于预测残差信息与样本残差信息之间的差异，确定预测视频的第二运动奖励得分。
在一些实施例中，输入信息还包括用作视频生成的控制条件的参考视频，或者包括从参考视频提取的边缘图和深度图中的至少一项。
在一些实施例中，视频生成模型包括扩散概率模型，并且输入信息还包括从噪声分布中采样的噪声信息。
在一些实施例中，视频生成模型还包括基于注意力的模块。
图8示出了其中可以实施本公开的一个或多个实施例的电子设备800的框图。应当理解，图8所示出的电子设备800仅仅是示例性的，而不应当构成对本文所描述的实施例的功能和范围的任何限制。图8所示出的电子设备800可以用于实现图1的电子设备110、或图7的装置700。
如图8所示，电子设备800是通用计算设备的形式。电子设备800的组件可以包括但不限于一个或多个处理器或处理单元810、存储器820、存储设备830、一个或多个通信单元840、一个或多个输入设备850以及一个或多个输出设备860。处理单元810可以是实际或虚拟处理器并且能够根据存储器820中存储的程序来执行各种处理。在多处理器系统中，多个处理单元并行执行计算机可执行指令，以提高电子设备800的并行处理能力。
电子设备800通常包括多个计算机存储介质。这样的介质可以是电子设备800可访问的任何可以获得的介质，包括但不限于易失性和非易失性介质、可拆卸和不可拆卸介质。存储器820可以是易失性存储器(例如寄存器、高速缓存、随机访问存储器(RAM))、非易失性存储器(例如，只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)、闪存)或它们的某种组合。存储设备830可以是可拆卸或不可拆卸的介质，并且可以包括机器可读介质，诸如闪存驱动、磁盘或者任何其他介质，其可以能够用于存储信息和/或数据并且可以在电子设备800内被访问。
电子设备800可以进一步包括另外的可拆卸/不可拆卸、易失性/非易失性存储介质。尽管未在图8中示出，可以提供用于从可拆卸、非易失性磁盘(例如“软盘”)进行读取或写入的磁盘驱动和用于从可拆卸、非易失性光盘进行读取或写入的光盘驱动。在这些情况中，每个驱动可以由一个或多个数据介质接口被连接至总线(未示出)。存储器820可以包括计算机程序产品825，其具有一个或多个程序模块，这些程序模块被配置为执行本公开的各种实施例的各种方法或动作。
通信单元840实现通过通信介质与其他电子设备进行通信。附加地，电子设备800的组件的功能可以以单个计算集群或多个计算机器来实现，这些计算机器能够通过通信连接进行通信。因此，电子设备800可以使用与一个或多个其他服务器、网络个人计算机(PC)或者另一个网络节点的逻辑连接来在联网环境中进行操作。
输入设备850可以是一个或多个输入设备，例如鼠标、键盘、追踪球等。输出设备860可以是一个或多个输出设备，例如显示器、扬声器、打印机等。电子设备800还可以根据需要通过通信单元840与一个或多个外部设备(未示出)进行通信，外部设备诸如存储设备、显示设备等，与一个或多个使得用户与电子设备800交互的设备进行通信，或者与使得电子设备800与一个或多个其他电子设备通信的任何设备(例如，网卡、调制解调器等)进行通信。这样的通信可以经由输入/输出(I/O)接口(未示出)来执行。
根据本公开的示例性实现方式，提供了一种计算机可读存储介质，其上存储有计算机可执行指令，其中计算机可执行指令被处理器执行以实现上文描述的方法。根据本公开的示例性实现方式，还提供了一种计算机程序产品，计算机程序产品被有形地存储在非瞬态计算机可读介质上并且包括计算机可执行指令，而计算机可执行指令被处理器执行以实现上文描述的方法。
这里参照根据本公开实现的方法、装置、设备和计算机程序产品的流程图和/或框图描述了本公开的各个方面。应当理解，流程图和/或框图的每个方框以及流程图和/或框图中各方框的组合，都可以由计算机可读程序指令实现。
这些计算机可读程序指令可以提供给通用计算机、专用计算机或其他可编程数据处理装置的处理单元，从而生产出一种机器，使得这些指令在通过计算机或其他可编程数据处理装置的处理单元执行时，产生了实现流程图和/或框图中的一个或多个方框中规定的功能/动作的装置。也可以把这些计算机可读程序指令存储在计算机可读存储介质中，这些指令使得计算机、可编程数据处理装置和/或其他设备以特定方式工作，从而，存储有指令的计算机可读介质则包括一个制造品，其包括实现流程图和/或框图中的一个或多个方框中规定的功能/动作的各个方面的指令。
可以把计算机可读程序指令加载到计算机、其他可编程数据处理装置、或其他设备上，使得在计算机、其他可编程数据处理装置或其他设备上执行一系列操作步骤，以产生计算机实现的过程，从而使得在计算机、其他可编程数据处理装置、或其他设备上执行的指令实现流程图和/或框图中的一个或多个方框中规定的功能/动作。
附图中的流程图和框图显示了根据本公开的多个实现的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段或指令的一部分，模块、程序段或指令的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个连续的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或动作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
以上已经描述了本公开的各实现，上述说明是示例性的，并非穷尽性的，并且也不限于所公开的各实现。在不偏离所说明的各实现的范围和精神的情况下，对于本技术领域的普通技术人员来说许多修改和变更都是显而易见的。本文中所用术语的选择，旨在最好地解释各实现的原理、实际应用或对市场中的技术的改进，或者使本技术领域的其他普通技术人员能理解本文公开的各个实现方式。

Claims (12)

  1. 一种视频生成的方法，包括：
    获得针对经训练的视频生成模型的输入信息，所述输入信息至少包括对要生成的目标视频的文本描述；以及
    利用所述经训练的视频生成模型，基于所述输入信息来生成所述目标视频，所述视频生成模型通过以下被训练：
    基于预测视频的质量和所述预测视频与样本视频之间运动一致性中的至少一项，分别确定所述预测视频的奖励得分，所述预测视频由正在训练的所述视频生成模型至少基于与所述样本视频匹配的样本文本描述来生成，以及
    基于预定训练目标来训练所述视频生成模型，所述训练目标被配置为使所述奖励得分增加或最大化。
  2. 根据权利要求1所述的方法，其中确定所述预测视频的奖励得分包括以下至少一项：
    利用质量奖励模型，基于所述预测视频的质量，确定所述预测视频的质量奖励得分；以及
    基于所述预测视频与样本视频之间运动一致性，确定所述预测视频的运动奖励得分；并且
    其中所述训练目标被配置为使所述质量奖励得分增加或最大化，并且使所述运动奖励得分增加或最大化。
  3. 根据权利要求2所述的方法，其中确定所述预测视频的质量奖励得分包括以下至少一项：
    利用主观质量奖励模型，确定所述预测视频的第一质量奖励得分，
    其中所述主观质量奖励模型利用以下训练数据来被有监督训练：训练图像集或训练视频集，以及针对所述训练图像集中的各个图像或所述训练视频集中的各个视频的真值质量奖励得分，所述真值质量奖励得分由用户提供；以及
    利用客观质量奖励模型，确定所述预测视频的第二质量奖励得分，
    其中所述客观质量奖励模型被配置为基于输入视频中至少一个质量影响因素的存在来确定质量奖励得分。
  4. 根据权利要求2所述的方法，其中确定所述预测视频的运动奖励得分包括：
    确定所述样本视频的样本光流信息和所述预测视频的预测光流信息；以及
    基于所述预测光流信息与所述样本光流信息之间的差异，确定所述预测视频的第一运动奖励得分。
  5. 根据权利要求2所述的方法，其中确定所述预测视频的运动奖励得分包括：
    基于所述样本视频的连续视频帧之间的像素差异，确定所述样本视频的样本残差信息；
    基于所述预测视频的连续视频帧之间的像素差异，确定所述预测视频的预测残差信息；以及
    基于所述预测残差信息与所述样本残差信息之间的差异，确定所述预测视频的第二运动奖励得分。
  6. 根据权利要求1所述的方法，其中所述输入信息还包括用作视频生成的控制条件的参考视频，或者包括从所述参考视频提取的边缘图和深度图中的至少一项。
  7. 根据权利要求1所述的方法，其中所述视频生成模型包括扩散概率模型，并且其中所述输入信息还包括从噪声分布中采样的噪声信息。
  8. 根据权利要求7所述的方法，其中所述视频生成模型还包括基于注意力的模块。
  9. 一种视频生成的装置，包括：
    输入信息获得模块，被配置为获得针对经训练的视频生成模型的输入信息，所述输入信息至少包括对要生成的目标视频的文本描述；以及
    目标视频生成模块，被配置为利用所述经训练的视频生成模型，基于所述输入信息来生成所述目标视频，所述视频生成模型通过以下被训练：
    基于预测视频的质量和所述预测视频与样本视频之间运动一致性中的至少一项，分别确定所述预测视频的奖励得分，所述预测视频由正在训练的所述视频生成模型至少基于与所述样本视频匹配的样本文本描述来生成，以及
    基于预定训练目标来训练所述视频生成模型，所述训练目标被配置为使所述奖励得分增加或最大化。
  10. 一种电子设备，包括：
    至少一个处理单元；以及
    至少一个存储器，所述至少一个存储器被耦合到所述至少一个处理单元并且存储用于由所述至少一个处理单元执行的指令，所述指令在由所述至少一个处理单元执行时使所述设备执行根据权利要求1至8中任一项所述的方法。
  11. 一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现根据权利要求1至8中任一项所述的方法。
  12. 一种计算机程序产品，包括计算机程序，所述计算机程序被处理器执行时实现根据权利要求1至8中任一项所述的方法。
PCT/CN2025/082436 2024-03-25 2025-03-13 视频生成的方法、装置、设备和存储介质 Pending WO2025201082A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202410346959.4 2024-03-25
CN202410346959.4A CN120711228A (zh) 2024-03-25 2024-03-25 视频生成的方法、装置、设备和存储介质

Publications (1)

Publication Number Publication Date
WO2025201082A1 true WO2025201082A1 (zh) 2025-10-02

Family

ID=97110154

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2025/082436 Pending WO2025201082A1 (zh) 2024-03-25 2025-03-13 视频生成的方法、装置、设备和存储介质

Country Status (2)

Country Link
CN (1) CN120711228A (zh)
WO (1) WO2025201082A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN120953453A (zh) * 2025-10-14 2025-11-14 阿里巴巴（中国）有限公司 模型训练方法、视频生成方法、电子设备及存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114598926A (zh) * 2022-01-20 2022-06-07 中国科学院自动化研究所 一种视频生成方法、装置、电子设备及存储介质
CN115240103A (zh) * 2022-06-21 2022-10-25 有米科技股份有限公司 基于视频和文本的模型训练方法及装置
CN117499711A (zh) * 2023-11-08 2024-02-02 腾讯科技（深圳）有限公司 视频生成模型的训练方法、装置、设备及存储介质
US11908180B1 (en) * 2023-03-24 2024-02-20 Google Llc Generating videos using sequences of generative neural networks
CN117668297A (zh) * 2023-12-05 2024-03-08 浙江阿里巴巴机器人有限公司 视频生成方法、电子设备及计算机可读存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114598926A (zh) * 2022-01-20 2022-06-07 中国科学院自动化研究所 一种视频生成方法、装置、电子设备及存储介质
CN115240103A (zh) * 2022-06-21 2022-10-25 有米科技股份有限公司 基于视频和文本的模型训练方法及装置
US11908180B1 (en) * 2023-03-24 2024-02-20 Google Llc Generating videos using sequences of generative neural networks
CN117499711A (zh) * 2023-11-08 2024-02-02 腾讯科技（深圳）有限公司 视频生成模型的训练方法、装置、设备及存储介质
CN117668297A (zh) * 2023-12-05 2024-03-08 浙江阿里巴巴机器人有限公司 视频生成方法、电子设备及计算机可读存储介质

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
WANG XIANG; ZHANG SHIWEI; YUAN HANGJIE; QING ZHIWU; GONG BIAO; ZHANG YINGYA; SHEN YUJUN; GAO CHANGXIN; SANG NONG: "A Recipe for Scaling up Text-to-Video Generation with Text-free Videos", 2024 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), IEEE, 16 June 2024 (2024-06-16), pages 6572 - 6582, XP034696440, DOI: 10.1109/CVPR52733.2024.00628 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN120953453A (zh) * 2025-10-14 2025-11-14 阿里巴巴（中国）有限公司 模型训练方法、视频生成方法、电子设备及存储介质

Also Published As

Publication number Publication date
CN120711228A (zh) 2025-09-26

Similar Documents

Publication Publication Date Title
Kwon et al. Diffusion models already have a semantic latent space
JP7047087B2 (ja) ビデオコンテンツを特徴付けるための深層強化学習フレームワーク
CN109919221B (zh) 基于双向双注意力机制图像描述方法
CN118015421A (zh) 使用预训练扩散模型的个性化单图像概念编码器
CN116975347B (zh) 图像生成模型训练方法及相关装置
JP2022530127A (ja) 教師なしデータ拡張を使用した機械学習モデルのトレーニング
CN113822318B (zh) 神经网络的对抗训练方法、装置、计算机设备及存储介质
US12445677B2 (en) Small and fast video processing networks via neural architecture search
CN111506814A (zh) 一种基于变分自注意力网络的序列推荐方法
CN117544832A (zh) 用于生成视频的方法、装置、设备和介质
US20240386529A1 (en) Generating domain-specific videos using diffusion models
CN108898221B (zh) 基于状态特征和后继特征的特征与策略的联合学习方法
CN118170992A (zh) 对象推荐方法、装置、介质和计算设备
CN114494713A (zh) 用于图像识别的方法和设备
WO2025201082A1 (zh) 视频生成的方法、装置、设备和存储介质
CN117408777A (zh) 基于扩散模型的模型优化方法、装置、设备、介质及产品
WO2025261282A1 (zh) 文生图模型、模型训练、图像生成方法、装置及电子设备
CN119583845A (zh) 视频生成方法及装置、计算机程序产品和电子设备
CN119003876A (zh) 用于内容推荐的方法、装置、设备和存储介质
WO2025050994A1 (zh) 用于图像生成的方法、装置、设备和存储介质
US20260057604A1 (en) Three-Dimensional Diffusion Models
CN119922393A (zh) 定制视频生成中的动作和外观
CN117727086A (zh) 图像处理方法、装置、计算机设备和图像处理系统
US12536624B1 (en) Auto-regressive video generation with long-term frame memory
CN119168793B (zh) 一种项目推荐方法、装置、介质和计算机设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 25776909

Country of ref document: EP

Kind code of ref document: A1