WO2025218166A1 - 视频处理的方法、装置、电子设备和计算机程序产品 - Google Patents

视频处理的方法、装置、电子设备和计算机程序产品

Info

Publication number
WO2025218166A1
WO2025218166A1 PCT/CN2024/133587 CN2024133587W WO2025218166A1 WO 2025218166 A1 WO2025218166 A1 WO 2025218166A1 CN 2024133587 W CN2024133587 W CN 2024133587W WO 2025218166 A1 WO2025218166 A1 WO 2025218166A1
Authority
WO
WIPO (PCT)
Prior art keywords
video
features
temporal
spatial
text
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/CN2024/133587
Other languages
English (en)
French (fr)
Inventor
聂宇翔
王涵
王彦杰
黄灿
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Douyin Vision Co Ltd
Original Assignee
Douyin Vision Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Douyin Vision Co Ltd filed Critical Douyin Vision Co Ltd
Publication of WO2025218166A1 publication Critical patent/WO2025218166A1/zh
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/44008—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G06F16/3331—Query processing
    • G06F16/334—Query execution
    • G06F16/3344—Query execution using natural language analysis
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/30—Semantic analysis
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
    • H04N21/84—Generation or processing of descriptive data, e.g. content descriptors

Definitions

  • the present application relates to the field of computer technology, and more particularly to a method, device, electronic device, and computer program product for video processing.
  • GLMs generative language models
  • multimodal models that integrate generative language models and image processing models are becoming increasingly important. These models can simultaneously process multiple data types, such as text and images, to more accurately understand and express complex semantic information. In areas such as natural language processing, computer vision, and intelligent conversation, multimodal models offer new possibilities for more comprehensive and intelligent human-computer interaction.
  • Embodiments of the present disclosure provide a method, apparatus, electronic device, computer program product, and medium for video processing.
  • a method for video processing includes determining temporal and spatial features of a video. Furthermore, the method includes generating a text output corresponding to the video using a spatiotemporal query conversion module based on the temporal and spatial features, wherein the spatiotemporal query conversion module includes a temporal query and a corresponding temporal expert network, and a spatial query and a corresponding spatial expert network.
  • a video processing apparatus includes a spatiotemporal feature determination module configured to determine temporal and spatial features of a video. Furthermore, the apparatus includes a text output generation module configured to generate a text output corresponding to the video based on the temporal and spatial features using a spatiotemporal query conversion module.
  • the spatiotemporal query conversion module includes a temporal query and a corresponding temporal expert network, as well as a spatial query and a corresponding spatial expert network.
  • an electronic device comprising a processor and a memory coupled to the processor, wherein the memory has instructions stored therein, and when the instructions are executed by the processor, the electronic device executes the method according to the first aspect.
  • a computer-readable storage medium is provided.
  • the computer program product is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions that, when executed, cause a computer to perform the steps of the method of the first aspect of the present disclosure.
  • a computer-readable storage medium wherein one or more computer instructions are stored on the computer-readable storage medium, wherein the one or more computer instructions are executed by a processor to implement the method according to the first aspect.
  • FIG1 shows a schematic diagram of an example environment in which devices and/or methods according to embodiments of the present disclosure may be implemented
  • FIG2 shows a flow chart of a method for video processing according to an embodiment of the present disclosure
  • FIG3 is a schematic diagram showing a process of processing a video using a video processing model according to an embodiment of the present disclosure
  • FIG4A is a schematic diagram showing the structure of a spatiotemporal query conversion module according to an embodiment of the present disclosure
  • FIG4B shows a schematic diagram of a matrix of a cross-attention mask according to an embodiment of the present disclosure
  • FIG5 is a schematic diagram showing a process of pre-training a video processing model according to an embodiment of the present disclosure
  • FIG6 is a schematic diagram showing a process of interacting with a video processing system according to an embodiment of the present disclosure
  • FIG7 shows a block diagram of a video processing apparatus according to an embodiment of the present disclosure.
  • FIG8 shows a block diagram of an electronic device according to an embodiment of the present disclosure.
  • multimodal models play an important role in various fields. They can perform well when processing text and image data. However, when it comes to processing video data, multimodal models perform poorly. Analysis has found that because video data has both spatial and temporal features, semantic alignment of these features is often difficult, resulting in poor performance of multimodal models for video.
  • an embodiment of the present disclosure proposes a video processing solution, which first generates the temporal features and spatial features of the video, and then uses the temporal features and spatial features of the video to generate a text output corresponding to the video using a spatiotemporal query conversion module, which includes a temporal query and a corresponding temporal expert network as well as a spatial query and a corresponding spatial expert network.
  • the embodiments of the present disclosure utilize a spatiotemporal query conversion module to effectively generate a spatiotemporal semantic alignment representation of the input video using temporal queries and corresponding temporal expert networks, as well as spatial queries and corresponding spatial expert networks.
  • This significantly enhances the visual language alignment capability of the video processing model, enabling it to better understand the video content.
  • the effectiveness of the video processing model also significantly improves.
  • the spatiotemporal query conversion module not only improves the performance of the processing model, but also helps lay the foundation for subsequent research and application of multimodal video processing.
  • FIG1 shows a schematic diagram of an example environment in which the device and/or method according to an embodiment of the present disclosure can be implemented.
  • the example environment 100 may include a computing device 110, which may be a user terminal, a mobile device, a computer, etc., or may be a computing system, a single server, a distributed server, or a cloud-based server.
  • the computing device 110 may receive a video 120.
  • the embodiments of the present disclosure may process multimodal data, i.e., data in a video modality and data in a text modality.
  • Multimodal data refers to a collection of data containing multiple types or forms, which may be from different sensors, devices, or sources, and typically include at least two of multiple forms such as text, images, audio, and video.
  • the computing device 110 may include a video processing system 130, which may generate temporal features 132 and spatial features 134 of the video 120. Unlike static images, videos not only have spatial information of image frames, but also contain temporal information between image frames. Therefore, it is necessary to capture not only the spatial features 134 of the static scene of the video 120, but also its temporal features 132 that change over time.
  • the spatiotemporal query conversion module 136 may semantically align the temporal features 132 and spatial features of the video 120.
  • the spatiotemporal query conversion module 136 may include a temporal query 138 and a corresponding temporal expert network 140, a spatial query 142 and a corresponding spatial expert network 144.
  • the video processing system 130 may generate a text output 150 corresponding to the video 120.
  • the video processing system 130 can generate a text output 150 corresponding to the video 120 based on the temporal features 132 and the spatial features 134 using a spatiotemporal query conversion module 136, where the spatiotemporal query conversion module 136 includes a temporal query 138 and a corresponding temporal expert network 140 and a spatial query 142 and a corresponding spatial expert network 144.
  • FIG2 shows a flow chart of a method 200 for video processing according to an embodiment of the present disclosure.
  • temporal and spatial features of a video may be determined.
  • the video processing system 130 may determine the temporal features 132 and spatial features 134 of the video 120.
  • a text output corresponding to the video can be generated based on the temporal features and the spatial features using a spatiotemporal query conversion module, the spatiotemporal query conversion module including a temporal query and a corresponding temporal expert network, and a spatial query and a corresponding spatial expert network.
  • the video processing system 130 can generate a text output 150 corresponding to the video 120 based on the temporal features 132 and the spatial features 134 using a spatiotemporal query conversion module 136, the spatiotemporal query conversion module 136 including a temporal query 138 and a corresponding temporal expert network 140, and a spatial query 142 and a corresponding spatial expert network 144.
  • the spatiotemporal query conversion module by utilizing the spatiotemporal query conversion module, it is possible to effectively generate a spatiotemporal semantic alignment representation of the input video, significantly enhancing the visual language alignment capability of the video processing model, enabling it to better understand the video content.
  • the spatiotemporal query conversion module also significantly improves.
  • the spatiotemporal query conversion module not only improves the performance of the processing model, but also helps lay the foundation for subsequent research and application of multimodal video processing.
  • FIG3 shows a schematic diagram of a process 300 for processing a video using a video processing model according to an embodiment of the present disclosure.
  • a video 302 can be input into a visual encoder.
  • the embodiment of the present disclosure does not impose any restrictions on the format of the video 302, and can be applied to various video formats, including but not limited to MP4, AVI, MOV, etc.
  • a corresponding semantic alignment representation can be generated by the spatiotemporal query conversion module.
  • the visual encoder 304 can receive the video 302 and generate corresponding visual features or visual representations.
  • the visual encoder is a neural network model for image processing and computer vision tasks, which can be used as a visual feature extractor, capable of converting input image data into high-dimensional feature vectors, which contain the semantic and visual information of the image. These feature vectors can be used for various tasks, such as image classification, target detection, image semantic segmentation, etc.
  • the attention pooling module 306 can receive the visual features of the video 302 and generate spatial features 308 and temporal features 310.
  • the attention pooling module 306 can be used to decouple the spatiotemporal features of the video. Explicit modeling of the spatiotemporal features of the video is crucial for the language model to effectively understand the video content, which allows the language model to capture rich semantic information, dynamic changes and contextual clues, thereby enhancing the language model's ability to understand the video content.
  • the attention pooling module 306 consists of a cross-attention layer and a feedforward layer, and can obtain the spatial features 308 and temporal features 310 of the video 302 through a learnable pooling process.
  • the input video is
  • T is the number of frames
  • H, W and C represent the height, width and number of channels of each image frame.
  • the initial video embedding is generated.
  • N represents the number of image blocks per frame
  • D represents the feature dimension.
  • the attention pooling module 306 can introduce two attention pooling queries to learn to extract the corresponding features.
  • the time pooling query can be used To embed the video Perform cross-attention operation to generate temporal features 310
  • the process can be expressed by formula (1) and formula (2):
  • CA(Q s ,x,x) represents the cross attention network
  • FFN( ⁇ ) represents the feedforward neural network.
  • spatial pooling can be used to query To transpose the video embedding
  • a cross-attention operation is performed to generate spatial features 308, which can be expressed by formula (3) and formula (4):
  • spatiotemporal query conversion module 312 can utilize temporal query 314, spatial query 316, and fused query 318 to process spatial features 308 and temporal features 310.
  • temporal query 314, spatial query 316, and fused query 318 can be considered feature extractors, extracting corresponding features from spatial features 308 and temporal features 310 to achieve semantic alignment.
  • temporal query 314 and spatial query 316 can have 32 feature dimensions, while the fused query can have 1 dimension.
  • only temporal query 314 and spatial query 316 may be included, without fused query 318.
  • the spatiotemporal query conversion module 312 can generate video features 320 for the video 302 and align the dimensions of the video features 320 with the dimensions of the language model 324 using a multi-layer perceptron (MLP) layer 322.
  • MLP multi-layer perceptron
  • the video features may have 1024 dimensions, while the input dimensions of the language model 324 are 4096. Therefore, the MLP layer 322 can be used to convert the dimensions of the video features 320 into the input dimensions of the language model 324, and further process the video features 320 to generate an output 328.
  • the video features and the language model input dimensions can also have other numbers, and the embodiments of the present disclosure are not limited in this regard.
  • the output 328 can be a summary or description of the video 302.
  • a prompt 326 can also be input to the language model 324 to instruct the language model 324 to generate the output indicated by the prompt 326.
  • the prompt 326 can be "What are the characters in the video doing?"
  • the language model 324 can then generate a corresponding output 328 based on its understanding of the video content.
  • output 328 may be "A girl is feeding a boy a pizza," and this process can be considered a video question answering task.
  • the video processing model of the present disclosure can also perform other tasks, including but not limited to multimodal instruction following, video object localization, video description generation, and video summary generation.
  • FIG. 4A shows a schematic diagram of the structure 400A of the spatiotemporal query conversion module according to an embodiment of the present disclosure.
  • a video 402 can generate visual features through a visual encoder 404, and generate spatial features 408 and temporal features 410 through an attention pooling module 406.
  • the spatiotemporal query conversion module 412 can connect the visual encoder and the language model to bridge the gap between visual representation and language patterns.
  • the spatiotemporal query conversion module 412 can include three expert networks, a temporal expert network 414, a spatial expert network 416, and a fusion expert network 418.
  • each expert network is an independent sub-model, and each expert network is responsible for processing a specific subspace or subtask.
  • the spatiotemporal query conversion module 412 dynamically combines the outputs of each expert network by selecting and weighting between different expert networks to effectively process the temporal features, spatial features, and spatiotemporal fusion features in the video data.
  • the spatiotemporal query conversion module 412 may also include learnable spatial queries 420, temporal queries 422, and fusion queries 424.
  • the spatial queries 420, temporal queries 422, and fusion queries 424 may respectively extract the spatial representation, temporal representation, and spatiotemporal fusion representation of the video 402.
  • a spatial query 420 with a feature dimension of 32 dimensions may be used
  • a temporal query 422 with a feature dimension of 32 dimensions may be used
  • a fusion query 424 with a feature dimension of 1 dimension may be used. It should be understood that query vectors of other dimensions may also be used, and the embodiments of the present disclosure do not limit the dimensions of the vectors.
  • the spatiotemporal query conversion module 412 may not include the fusion query 424 and the corresponding fusion expert network 418.
  • the spatial query 420, the temporal query 422, and the fusion query 424 interact through the self-attention network 426, and interact with the spatial features 408 and the temporal features 410 of the video 402 through the cross-attention network 428.
  • each query can also interact with the video description 430 through the same self-attention network 426.
  • the video description 430 can be a descriptive text corresponding to the video 402. For example, the video description 430 can be "A girl is feeding a boy a piece of pizza.”
  • the feedforward neural network 432 can receive the output of the self-attention network 426 to further encode the video description 430.
  • the embodiment of the present disclosure designs a matrix of cross-attention masks in the cross-attention network 428.
  • the cross-attention masks will be described below in conjunction with Figure 4B.
  • FIG4B shows a schematic diagram of a matrix 400B of a cross-attention mask according to an embodiment of the present disclosure.
  • a gray block indicates that the corresponding position of the matrix is "1," and a white block indicates that the corresponding position of the matrix is "0.”
  • the cross-attention mask can be used to control the visibility of spatial and temporal queries on various spatiotemporal features. Spatial and temporal queries are limited to focusing on their corresponding features, while fused queries can focus on all features. For example, the mask corresponding to spatial feature 452 of spatial query 450 is 1, indicating that spatial feature 452 is visible to spatial query 450.
  • the mask corresponding to temporal feature 462 of spatial query 450 is 0, indicating that temporal feature 462 is invisible to spatial query 450.
  • the mask corresponding to temporal feature 462 of temporal query 460 is 1, but the mask corresponding to spatial feature 452 is 0.
  • the mask corresponding to both spatial feature 452 and temporal feature 462 of fused query 470 is 1 because fused query 470 can focus on both spatial feature 452 and temporal feature 462 simultaneously.
  • the design of the cross-attention mask enables precise control of attention allocation when processing spatiotemporal features, thereby more effectively capturing important information in video data.
  • L VTM loss function 434, L VTC loss function 436, and L VTG loss function 438 can also be used to train spatiotemporal query conversion module 412.
  • L VTM loss function 434 is used to represent the video-text matching (VTM) loss function
  • L VTC loss function 436 is used to represent the video-text comparison (VTC) loss function
  • L VTG loss function 438 is used to represent the video-text generation (VTG) loss function.
  • FIG. 5 shows a schematic diagram of a process 500 of pre-training of a video processing model according to an embodiment of the present disclosure.
  • the spatiotemporal query conversion module is independently trained to perform a first stage of pre-training.
  • the spatiotemporal query conversion module is trained to extract the spatiotemporal video embedding that is most relevant to the video description.
  • the spatiotemporal query conversion module 412 is trained to extract the spatiotemporal video embedding that is most relevant to the video description 430.
  • the spatiotemporal query conversion module is trained by jointly optimizing the VTM loss function, the VTC loss function, and the VTG loss function.
  • the average values of the spatial query and the temporal query can be calculated separately, and then the average values can be connected with the classification labels and input into the binary classification task to predict whether the video and the video description match.
  • the spatiotemporal query conversion module is connected to the language model to perform the second stage of pre-training.
  • the spatiotemporal query conversion module 312 can be connected to the language model 324 to train the spatiotemporal query conversion module 312.
  • the parameters of the language model 324 can remain unchanged, which can speed up the model training process.
  • an MLP layer can be used to project the video features from the spatiotemporal query conversion module to the embedding space of the language model.
  • the MLP layer 332 can be used to connect the spatiotemporal query conversion module 312 and the language model 324, and the MLP layer 332 can project the video features 320 to the embedding space of the language model 324.
  • FIG6 shows a schematic diagram of a process 600 for interacting with a video processing system according to an embodiment of the present disclosure.
  • a user may input a video 602 into the video processing system, wherein the video processing system may include a video processing model according to an embodiment of the present disclosure.
  • video 602 may be a video related to a vehicle.
  • the video processing system may also display image frames below video 602.
  • the user may then interact with the video processing system.
  • the user may send a dialogue 604, “What happened in this video?”, and the video processing system may then generate a corresponding dialogue 606, “In this video, a car’s rearview mirror is damaged after a car accident.”
  • the video processing system may also support multiple rounds of dialogue.
  • the user may enter a dialogue 608 to inquire further about content related to video 602, and the video processing system may further generate a dialogue 610 to respond.
  • FIG7 shows a block diagram of a video processing apparatus 700 according to an embodiment of the present disclosure.
  • apparatus 700 includes a spatiotemporal feature determination module 702 configured to determine temporal features and spatial features of a video.
  • apparatus 700 also includes a text output generation module 704 configured to generate a text output corresponding to the video based on the temporal features and spatial features using a spatiotemporal query conversion module, wherein the spatiotemporal query conversion module includes a temporal query and a corresponding temporal expert network, as well as a spatial query and a corresponding spatial expert network.
  • Figure 8 shows a block diagram of an electronic device 800 according to certain embodiments of the present disclosure.
  • Figure 8 shows a block diagram of an electronic device 800 according to certain embodiments of the present disclosure.
  • the device 800 may be a device or apparatus described in an embodiment of the present disclosure.
  • the device 800 includes a central processing unit (CPU) and/or a graphics processing unit (GPU) 801, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 802 or computer program instructions loaded from a storage unit 808 into a random access memory (RAM) 803.
  • ROM read-only memory
  • RAM random access memory
  • Various programs and data required for the operation of the device 800 can also be stored in the RAM 803.
  • the CPU/GPU 801, ROM 802, and RAM 803 are connected to each other via a bus 804.
  • An input/output (I/O) interface 805 is also connected to the bus 804.
  • the device 800 may also include a coprocessor.
  • I/O interface 805 Various components in device 800 are connected to I/O interface 805, including an input unit 806, such as a keyboard, mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, optical disk, etc.; and a communication unit 809, such as a network card, modem, wireless communication transceiver, etc.
  • the communication unit 809 allows device 800 to exchange information/data with other devices via a computer network such as the Internet and/or various telecommunication networks.
  • the various methods or processes described above can be executed by CPU/GPU 801.
  • the methods can be implemented as computer software programs that are tangibly embodied in a machine-readable medium, such as storage unit 808.
  • part or all of the computer program can be loaded and/or installed on device 800 via ROM 802 and/or communication unit 809.
  • the computer program is loaded into RAM 803 and executed by CPU/GPU 801, one or more steps or actions in the methods or processes described above can be performed.
  • the methods and processes described above may be implemented as a computer program product.
  • the computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.
  • Computer-readable storage medium can be a tangible device that can keep and store the instructions used by the instruction execution device.
  • Computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device or any suitable combination thereof.More specific examples (non-exhaustive list) of computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove having instructions stored thereon, and any suitable combination thereof.
  • Computer-readable storage medium used herein is not interpreted as a transient signal itself, such as radio
  • the computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing/processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and/or a wireless network.
  • the network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers.
  • the network adapter card or network interface in each computing/processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing/processing device.
  • the computer program instructions for performing the disclosed operation can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data or source code or the object code written in any combination of one or more programming languages, programming languages include object-oriented programming languages, and conventional procedural programming languages.
  • Computer-readable program instructions can be performed completely on a user's computer, partially on a user's computer, performed as an independent software package, partly on a user's computer and partly on a remote computer, or performed completely on a remote computer or server.
  • the remote computer can be connected to the user's computer by any type of network-including local area network (LAN) or wide area network (WAN), or can be connected to an external computer (such as utilizing an internet service provider to connect by the internet).
  • LAN local area network
  • WAN wide area network
  • an external computer such as utilizing an internet service provider to connect by the internet.
  • this electronic circuit can perform computer-readable program instructions, thereby realizing various aspects of the present disclosure.
  • These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions/actions specified in one or more blocks in the flowchart and/or block diagram.
  • These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and/or other device to operate in a specific manner.
  • the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions/actions specified in one or more blocks in the flowchart and/or block diagram.
  • Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions/actions specified in one or more boxes in the flowchart and/or block diagram.
  • each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function.
  • the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved.
  • each box in the block diagram and/or flow chart, and the combination of the boxes in the block diagram and/or flow chart can be implemented by a special hardware-based system that performs the prescribed function or action, or can be implemented by a combination of special hardware and computer instructions.
  • Example 1 A method for video processing, comprising:
  • a text output corresponding to the video is generated using a spatiotemporal query conversion module, wherein the spatiotemporal query conversion module includes a temporal query and a corresponding temporal expert network and a spatial query and a corresponding spatial expert network.
  • Example 2 The method of Example 1, wherein the temporal features and the spatial features are generated by an attention pooling module, the method further comprising:
  • the temporal features and the spatial features are generated using the cross attention network and the feedforward neural network in the attention pooling module.
  • Example 3 The method according to any one of Examples 1-2, wherein the spatiotemporal query conversion module further comprises a fused query and a corresponding fused expert network.
  • Example 4 The method of any one of Examples 1-3, further comprising:
  • the spatiotemporal query conversion module is used to generate a text output indicated by the prompt text.
  • Example 5 The method of any one of Examples 1-4, wherein generating the text output of the prompt text indication comprises:
  • the text output indicated by the prompt text is generated.
  • Example 6 A method according to any one of Examples 1-5, wherein the spatiotemporal query conversion module further includes a cross-attention mask, wherein the cross-attention mask includes a temporal mask, a spatial mask, and a fusion mask.
  • Example 7 The method of any one of Examples 1-6, wherein the text output is generated by a video processing model, the video processing model including the spatiotemporal query conversion module, and the method further comprising:
  • the video processing model is trained based on the training video, the video description text of the training video and the training prompt text.
  • Example 8 The method of any of Examples 1-7, wherein training the video processing model comprises:
  • a visual encoder Based on the training video, a visual encoder generates training visual features
  • the attention pooling module Based on the training visual features, the attention pooling module generates training time features and training spatial features;
  • the spatiotemporal query conversion module Based on the training time features, the training space features and the video description text, the spatiotemporal query conversion module generates a conversion output;
  • the attention pooling module and the spatiotemporal query conversion module are trained.
  • Example 9 The method of any one of Examples 1-8, wherein training the attention pooling module and the spatiotemporal query transformation module comprises:
  • the attention pooling module and the spatiotemporal query conversion module are trained.
  • Example 10 The method of any one of Examples 1-9, further comprising:
  • the language model Based on the conversion output and the training prompt text, the language model outputs a target output indicated by the training prompt text;
  • the attention pooling module and the spatiotemporal query conversion module are trained, and the parameters of the language model remain unchanged during the training.
  • Example 11 A video processing apparatus, comprising:
  • a spatiotemporal feature determination module configured to determine temporal features and spatial features of a video
  • a text output generation module is configured to generate a text output corresponding to the video based on the temporal features and the spatial features using a spatiotemporal query conversion module, wherein the spatiotemporal query conversion module includes a temporal query and a corresponding temporal expert network and a spatial query and a corresponding spatial expert network.
  • Example 12 The apparatus of Example 11, wherein the temporal features and the spatial features are generated by an attention pooling module, the apparatus further comprising:
  • a visual feature generation module is configured to generate visual features of the video based on the video; and a spatiotemporal feature generation module is configured to generate the temporal features and the spatial features based on the visual features using the cross-attention network and the feedforward neural network in the attention pooling module.
  • Example 13 The apparatus of any one of Examples 11-12, wherein the spatiotemporal query conversion module further comprises a fused query and a corresponding fused expert network.
  • Example 14 The apparatus of any of Examples 11-13, further comprising:
  • a prompt text acquisition module configured to acquire the prompt text
  • the indication output generation module is configured to generate a text output of the prompt text indication based on the time feature, the spatial feature and the prompt text using the spatiotemporal query conversion module.
  • Example 15 The apparatus of any of Examples 11-14, wherein the instruction output generation module comprises:
  • a video feature generation module configured to generate video features based on the temporal features and the spatial features using the spatiotemporal query conversion module
  • the second indication output generating module is configured to generate the text output indicated by the prompt text based on the video feature and the prompt text.
  • Example 16 An apparatus according to any one of Examples 11-15, wherein the spatiotemporal query transformation module further comprises a cross-attention mask, wherein the cross-attention mask comprises a temporal mask, a spatial mask, and a fusion mask.
  • Example 17 The apparatus of any of Examples 11-16, wherein the text output is generated by a video processing model, the video processing model including the spatiotemporal query conversion module, the apparatus further comprising:
  • the video processing model training module is configured to train the video processing model based on the training video, the video description text of the training video and the training prompt text.
  • Example 18 The apparatus of any of Examples 11-17, wherein the video processing model training module comprises:
  • a training visual feature generation module configured to generate training visual features based on the training video by a visual encoder
  • a training spatiotemporal feature generation module configured to generate training temporal features and training spatial features based on the training visual features by the attention pooling module
  • a conversion output generating module configured to generate a conversion output by the spatiotemporal query conversion module based on the training time features, the training space features and the video description text;
  • the second training module of the processing model is configured to train the attention pooling module and the spatiotemporal query conversion module based on the conversion output.
  • Example 19 The apparatus of any of Examples 11-18, wherein the processing model second training module comprises:
  • the third training module of the processing model is configured to train the attention pooling module and the spatiotemporal query conversion module based on the video-text matching loss function, the video-text comparison loss function and the video positioning text generation loss function.
  • Example 20 The apparatus of any of Examples 11-19, further comprising:
  • a target output module configured to output a target output indicated by the training prompt text by a language model based on the conversion output and the training prompt text
  • the fourth training module of the processing model trains the attention pooling module and the spatiotemporal query conversion module based on the target output indicated by the training prompt text, and the parameters of the language model remain unchanged during the training.
  • Example 21 An electronic device comprising:
  • a text output corresponding to the video is generated using a spatiotemporal query conversion module, wherein the spatiotemporal query conversion module includes a temporal query and a corresponding temporal expert network and a spatial query and a corresponding spatial expert network.
  • Example 22 The electronic device of Example 21, wherein the temporal features and the spatial features are generated by an attention pooling module, and the method further comprises:
  • the temporal features and the spatial features are generated using the cross attention network and the feedforward neural network in the attention pooling module.
  • Example 23 The electronic device of any one of Examples 21-22, wherein the spatiotemporal query conversion module further comprises a fused query and a corresponding fused expert network.
  • Example 24 The electronic device of any of Examples 21-23, wherein the actions further comprise:
  • the spatiotemporal query conversion module is used to generate a text output indicated by the prompt text.
  • Example 25 The electronic device of any of Examples 21-24, wherein generating the text output of the prompt text indication comprises:
  • the text output indicated by the prompt text is generated.
  • Example 26 An electronic device according to any one of Examples 21-25, wherein the spatiotemporal query conversion module further comprises a cross-attention mask, wherein the cross-attention mask comprises a temporal mask, a spatial mask, and a fusion mask.
  • Example 27 The electronic device of any of Examples 21-26, wherein the text output is generated by a video processing model, the video processing model including the spatiotemporal query conversion module, the method further comprising:
  • the video processing model is trained based on the training video, the video description text of the training video and the training prompt text.
  • Example 28 The electronic device of any of Examples 21-27, wherein training the video processing model comprises:
  • a visual encoder Based on the training video, a visual encoder generates training visual features
  • the attention pooling module Based on the training visual features, the attention pooling module generates training time features and training spatial features;
  • the spatiotemporal query conversion module Based on the training time features, the training space features and the video description text, the spatiotemporal query conversion module generates a conversion output;
  • the attention pooling module and the spatiotemporal query conversion module are trained.
  • Example 29 The electronic device of any one of Examples 21-28, wherein training the attention pooling module and the spatiotemporal query conversion module comprises:
  • the attention pooling module and the spatiotemporal query conversion module are trained.
  • Example 30 The electronic device of any of Examples 21-29, wherein the actions further comprise:
  • the language model Based on the conversion output and the training prompt text, the language model outputs a target output indicated by the training prompt text;
  • the attention pooling module and the spatiotemporal query conversion module are trained, and the parameters of the language model remain unchanged during the training.
  • Example 31 A computer-readable storage medium having one or more computer instructions stored thereon, wherein the one or more computer instructions are executed by a processor to implement the method according to any one of Examples 1 to 10.
  • Example 32 A computer program product tangibly stored on a computer-readable medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method of any one of Examples 1 to 10.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

本公开的实施例涉及视频处理的方法、装置、电子设备和计算机程序产品。该方法包括确定视频的时间特征和空间特征。此外，该方法还包括基于时间特征和空间特征，使用时空查询转换模块生成与视频相对应的文本输出，时空查询转换模块包括时间查询和对应的时间专家网络以及空间查询和对应的空间专家网络。

Description

视频处理的方法、装置、电子设备和计算机程序产品
相关申请的交叉引用
本申请要求于2024年4月19日提交的，申请号为202410479933.7、发明名称为“视频处理的方法、装置、电子设备和计算机程序产品”的中国专利申请的优先权，该申请的全部内容通过引用结合在本申请中。
技术领域
本申请涉及计算机技术领域，尤其具体地涉及视频处理的方法、装置、电子设备以及计算机程序产品。
背景技术
随着人工智能技术的不断发展，生成式语言模型变得越来越重要。这些模型能够自动生成高质量、自然流畅的文本，为各种应用领域带来了新的可能性。从自然语言处理到创造性内容生成，生成式语言模型在对话系统、文本摘要、翻译和创意文案等方面发挥着关键作用。
随着多模态智能的需求不断增长，融合生成式语言模型和图像处理模型的多模态模型变得愈发重要。这种模型能够同时处理文本和图像等多种数据类型，从而更准确地理解和表达复杂的语义信息。在自然语言处理、计算机视觉和智能对话等领域，多模态模型为实现更全面、更智能的人机交互提供了新的可能性。
发明内容
本公开的实施例提供了一种视频处理的方法、装置、电子设备、计算机程序产品以及介质。
根据本公开的第一方面，提供了一种视频处理的方法。该方法包括确定视频的时间特征和空间特征。此外，该方法还包括基于时间特征和空间特征，使用时空查询转换模块生成与视频相对应的文本输出，时空查询转换模块包括时间查询和对应的时间专家网络以及空间查询和对应的空间专家网络。
根据本公开的第二方面，提供了一种视频处理的装置。该装置包括时空特征确定模块，被配置为确定视频的时间特征和空间特征。此外，该装置还包括文本输出生成模块，被配置为基于时间特征和空间特征，使用时空查询转换模块生成与视频相对应的文本输出，时空查询转换模块包括时间查询和对应的时间专家网络以及空间查询和对应的空间专家网络。。
根据本公开的第三方面，提供了一种电子设备。该电子设备包括处理器以及与处理器耦合的存储器，存储器具有存储于其中的指令，指令在被处理器执行时使电子设备执行根据第一方面所述的方法。
在本公开的第四方面中，提供了一种计算机可读存储介质。该计算机程序产品被有形地存储在非瞬态计算机可读介质上并且包括计算机可执行指令，所述计算机可执行指令在被执行时使得计算机执行本公开的第一方面的方法的步骤。
在本公开的第五方面中，提供了一种计算机可读存储介质。该计算机可读存储介质上存储有一条或多条计算机指令，其中一条或多条计算机指令被处理器执行以实现根据第一方面所述的方法。
发明内容部分是为了以简化的形式来介绍对概念的选择，它们在下文的具体实施方式中将被进一步描述。发明内容部分无意标识要求保护的主题的关键特征或主要特征，也无意限制要求保护的主题的范围。
附图说明
结合附图并参考以下详细说明，本公开各实施例的上述和其它特征、优点及方面将变得更加明显。在附图中，相同或相似的附图标记表示相同或相似的元素，其中：
图1示出了根据本公开的实施例的设备和/或方法可以在其中被实施的示例环境的示意图；
图2示出了根据本公开的实施例的视频处理的方法的流程图；
图3示出了根据本公开的实施例的利用视频处理模型处理视频的过程的示意图；
图4A示出了根据本公开的实施例的时空查询转换模块的结构的示意图；
图4B示出了根据本公开的实施例的交叉注意力掩码的矩阵的示意图；
图5示出了根据本公开的实施例的视频处理模型的预训练的过程的示意图；
图6示出了根据本公开的实施例的与视频处理系统进行交互的过程的示意图；
图7示出了根据本公开的实施例的视频处理的装置的框图；以及
图8示出了根据本公开的实施例的电子设备的框图。
在所有附图中，相同或相似参考数字表示相同或相似元素。
具体实施方式
可以理解的是，在使用本公开各实施例公开的技术方案之前，均应当依据相关法律法规通过恰当的方式对本公开所涉及个人信息的类型、使用范围、使用场景等告知用户并获得用户的授权。
下面将参照附图更详细地描述本公开的实施例。虽然附图中显示了本公开的某些实施例，然而应当理解的是，本公开可以通过各种形式来实现，而且不应该被解释为限于这里阐述的实施例，相反提供这些实施例是为了更加透彻和完整地理解本公开。应当理解的是，本公开的附图及实施例仅用于示例性作用，并非用于限制本公开的保护范围。
在本公开的实施例的描述中，术语“包括”及其类似用语应当理解为开放性包括，即“包括但不限于”。术语“基于”应当理解为“至少部分地基于”。术语“一个实施例”或“该实施例”应当理解为“至少一个实施例”。术语“第一”、“第二”等等可以指代不同的或相同的对象，除非明确说明。下文还可能包括其它明确的和隐含的定义。
如前所述，多模态模型在多个领域发挥着重要的作用，相关的多模态模型在处理文本和图像数据时，可以发挥不错的效果。然而，当涉及处理视频数据时，多模态模型的表现较差。经过分析发现，视频数据由于同时具有空间特征和时间特征，因此在进行特征的语义对齐时，往往存在困难，导致处理视频的多模态模型效果不佳。
为此，本公开的实施例提出了一种视频处理的方案，该方案首先生成视频的时间特征和空间特征，然后利用视频的时间特征和空间特征，使用时空查询转换模块生成与视频相对应的文本输出，该时空查询转换模块包括时间查询和对应的时间专家网络以及空间查询和对应的空间专家网络。
本公开的实施例通过时空查询转换模块，利用时间查询和对应的时间专家网络以及空间查询和对应的空间专家网络来有效生成输入视频的时空语义对齐表示，显著增强了视频处理模型的视觉语言对齐能力，使其更好地理解视频内容。随着模型对视频输入的理解能力的提升，视频处理模型的效果也显著提高。时空查询转换模块不仅能够提高处理模型的性能，还有助于为后续的多模态视频处理的研究和应用打下基础。
图1示出了根据本公开的实施例的设备和/或方法可以在其中被实施的示例环境的示意图。如图1所示，示例环境100可以包括计算设备110，其可以是用户终端、移动设备、计算机等，其也可以为计算系统、单个服务器、分布式服务器、或者基于云的服务器。计算设备110可以接收视频120。本公开的实施例可以处理多模态数据，即视频模态的数据和文本模态的数据。多模态数据指的是包含多种类型或形式的数据的集合，这些数据可以是来自不同传感器、设备或来源的，通常包括文本、图像、音频、视频等多种形式中的至少两种形式。
计算设备110可以包括视频处理系统130，视频处理系统130可以生成视频120的时间特征132和空间特征134。与静态图片不同，视频不仅具有图像帧的空间信息，还包含了图像帧之间的时序信息。因此，不仅需要捕获视频120的静态场景的空间特征134，还能需要捕获其随时间变化的时间特征132。时空查询转换模块136可以将视频120的时间特征132和空间特征进行语义对齐。时空查询转换模块136可以包括时间查询138以及对应的时间专家网络140，空间查询142以及对应的空间专家网络144。视频处理系统130可以生成与视频120相对应的文本输出150。视频处理系统130可以基于时间特征132和空间特征134，使用时空查询转换模块136生成与视频120相对应的文本输出150，时空查询转换模块136包括时间查询138和对应的时间专家网络140以及空间查询142和对应的空间专家网络144。
应当理解，仅出于示例性的目的来描述示例环境100中的架构和功能，而不暗示对本公开的范围的任何限制。本公开的实施例还可以被应用到具有不同的结构和/或功能的其他环境中。
下文将结合图2至图8详细描述根据本公开实施例的过程。为了便于理解，在下文描述中提及的具体数据均是示例性的，并不用于限定本公开的保护范围。可以理解，以下描述的实施例还可以包括未示出的附加动作和/或可以省略所示出的动作，本公开的范围在此方面不受限制。
图2示出了根据本公开的实施例的视频处理的方法200的流程图。在框202处，可以确定视频的时间特征和空间特征。例如，参考图1，视频处理系统130可以确定视频120的时间特征132和空间特征134。
在框204处，可以基于时间特征和空间特征，使用时空查询转换模块生成与视频相对应的文本输出，时空查询转换模块包括时间查询和对应的时间专家网络以及空间查询和对应的空间专家网络。例如，参考图1，视频处理系统130可以基于时间特征132和空间特征134，使用时空查询转换模块136生成与视频120相对应的文本输出150，时空查询转换模块136包括时间查询138和对应的时间专家网络140以及空间查询142和对应的空间专家网络144。
由此，根据本公开的实施例的方法200，通过利用时空查询转换模块，能够有效生成输入视频的时空语义对齐表示，显著增强了视频处理模型的视觉语言对齐能力，使其更好地理解视频内容。随着模型对视频输入的理解能力的提升，视频处理模型的效果也显著提高。此外，时空查询转换模块不仅能够提高处理模型的性能，还有助于为后续的多模态视频处理的研究和应用打下基础。
图3示出了根据本公开的实施例的利用视频处理模型处理视频的过程300的示意图。如图3所示，可以将视频302输入视觉编码器。本公开的实施例对于视频302的格式不做限制，可以适用于各种视频格式，包括但不限于MP4、AVI、MOV等，无论视频数据是由哪种设备或软件生成，都可以通过时空查询转换模块生成相应的语义对齐表示。视觉编码器304可以接收视频302，并生成对应的视觉特征或视觉表示。视觉编码器是用于图像处理和计算机视觉任务的神经网络模型，可以被用作视觉特征提取器，能够将输入的图像数据转换为高维特征向量，其中包含了图像的语义和视觉信息，这些特征向量可以被用于各种任务，如图像分类、目标检测、图像语义分割等。
注意力池化模块306可以接收视频302的视觉特征，并且生成空间特征308和时间特征310。注意力池化模块306可以用来对视频的时空特征进行解耦。视频的时空特征的显式建模对于语言模型有效理解视频内容至关重要，其允许语言模型捕捉丰富的语义信息、动态变化和上下文线索，从而增强语言模型理解视频内容的能力。注意力池化模块306由交叉注意力层和前馈层组成，可以通过可学习的池化过程获取视频302的空间特征308和时间特征310。例如，假设输入视频为其中T为帧数，H、W和C表示每个图像帧的高度、宽度和通道数。当视频302通过视觉编码器304后，生成初始视频嵌入其中N表示每帧的图像块的数目，D表示特征维度。
为了从视频302中抽取时空特征，注意力池化模块306可以引入两个注意力池化查询来学习提取相应的特征。具体地，可以利用时间池化查询来对视频嵌入进行交叉注意力操作来生成时间特征310该过程可以用公式(1)和公式(2)表示：
其中CA(Qs,x,x)表示交叉注意力网络，并且FFN(·)表示前馈神经网络。类似地，可以利用空间池化查询来对视频嵌入的转置进行交叉注意力操作来生成空间特征308，该过程可以用公式(3)和公式(4)表示：
如图3所示，时空查询转换模块312可以利用时间查询314、空间查询316和融合查询318来处理空间特征308和时间特征310。例如，可以将时间查询314、空间查询316和融合查询318看作是特征提取器，来从空间特征308和时间特征310提取相应的特征，以实现语义对齐。在一些实施例中，时间查询314和空间查询316的特征维度可以为32维，并且融合查询的维度可以为1维。在一些实施例中，可以仅包括时间查询314和空间查询316，而不包括融合查询318。
时空查询转换模块312可以生成视频302的视频特征320，并且通过多层感知机(MLP)层322，来将视频特征320的维度与语言模型324的维度进行对齐。例如，视频特征维度可以为1024维，而语言模型324的输入维度为4096维，因此可以利用MLP层322将视频特征320的维度转换为语言模型324的输入维度，来对视频特征320进行进一步地处理，以生成输出328。此外，视频特征维度和语言模型的输入维度还可以为其他数目，本公开的实施例对此不做限制。例如，输出328可以为视频302的总结或描述。在一些实施例中，还可以向语言模型324输入提示内容326，来指示语言模型324生成提示内容326所指示的输出。例如，提示内容326可以为“请问视频中的人物在干什么”，那么语言模型324可以根据其对于视频内容的理解，生成相应的输出328。例如，输出328可以为“一个女孩正在给一个男孩喂一块披萨”，并且该过程可以看作是视频问答任务。此外，本公开的视频处理模型还可以执行其他任务，包括但不限于多模态指令跟随、视频物体定位、视频描述生成以及视频总结生成等等。
图4A示出了根据本公开的实施例的时空查询转换模块的结构400A的示意图。如图4A所示，视频402可以通过视觉编码器404生成视觉特征，并且通过注意力池化模块406生成空间特征408和时间特征410。时空查询转换模块412可以连接视觉编码器和语言模型，弥合视觉表示和语言模式之间的差异。具体地，时空查询转换模块412可以包括三个专家网络，时间专家网络414、空间专家网络416和融合专家网络418。在时空查询转换模块412中，每个专家网络是一个独立的子模型，每个专家网络负责处理特定子空间或子任务。在训练过程中，时空查询转换模块412通过在不同的专家网络之间进行选择和加权来动态地组合各个专家网络的输出，来有效地处理视频数据中的时间特征、空间特征以及时空融合特征。
时空查询转换模块412还可以包括可学习的空间查询420、时间查询422和融合查询424。在一些实施例中，空间查询420、时间查询422和融合查询424可以分别提取视频402的空间表示、时间表示和时空融合表示。在一些实施例中，可以使用特征维度为32维的空间查询420，使用特征维度为32维的时间查询422，并且使用特征维度为1维的融合查询424。应当理解，还可以使用其他维度的查询向量，本公开的实施例对于向量的维度不做限制。此外，在一些实施例中，时空查询转换模块412可以不包括融合查询424以及相应的融合专家网络418。
空间查询420、时间查询422和融合查询424通过自注意力网络426进行交互，并通过交叉注意力网络428与视频402的空间特征408以及时间特征410进行交互。此外，各个查询还可以通过相同的自注意力网络426与视频描述430进行交互。视频描述430可以是与视频402相对应的描述文本。例如，视频描述430可以为“一个女孩正在给一个男孩喂一块披萨”。前馈神经网络432可以接收自注意力网络426的输出来对视频描述430进行进一步地编码。为了确保各个查询能够与相应特征进行交叉注意力运算，本公开的实施例在交叉注意力网络428中设计了交叉注意力掩码的矩阵，下面将结合图4B描述交叉注意力掩码。
图4B示出了根据本公开的实施例的交叉注意力掩码的矩阵400B的示意图。如图4B所示，其中灰色块表示矩阵的相应位置为“1”，白色块表示矩阵的相应位置为“0”。交叉注意力掩码可以控制用于控制空间查询和时间查询在各种时空特征上的可见性，空间查询和时间查询仅限于关注其相应的特征，而融合查询可以关注所有特征。例如，空间查询450与空间特征452相对应的掩码为1，表示空间查询450对空间特征452是可见的。此外，空间查询450与时间特征462相对应的掩码为0，表示空间查询450对时间特征462是不可见的。同样地。时间查询460与时间特征462相对应的掩码为1，但是与空间特征452相对应的掩码为0。此外，融合查询470与空间特征452和时间特征462相对应的掩码均为1，因为融合查询470可以同时关注空间特征452和时间特征462。交叉注意力掩码的设计使得能够在处理时空特征时精确地控制注意力的分配，从而更有效地捕捉视频数据中的重要信息。
返回参考图4A，不同的专家网络被用来并行处理不同的查询向量。此外，还可以利用LVTM损失函数434、LVTC损失函数436以及LVTG损失函数438来训练时空查询转换模块412。例如，LVTM损失函数434用来表示视频文本匹配(VTM)损失函数，LVTC损失函数436用来表示视频文本对比(VTC)损失函数，LVTG损失函数438用来表示视频定位文本生成(VTG)损失函数。
图5示出了根据本公开的实施例的视频处理模型的预训练的过程500的示意图。如图5所示，在框502处，对时空查询转换模块进行独立训练，进行第一阶段的预训练。在第一阶段的预训练中，时空查询转换模块被训练来提取与视频描述最相关的时空视频嵌入。结合图4A所述，时空查询转换模块412被训练来提取与视频描述430最相关的时空视频嵌入。在一些实施例中，在第一阶段的预训练中，通过联合优化VTM损失函数、VTC损失函数以及VTG损失函数来训练时空查询转换模块。在计算VTM损失函数时，可以分别计算空间查询和时间查询的平均值，然后可以将该平均值与分类标记连接起来，并将它们输入到二元分类任务中，以预测视频和视频描述是否匹配。
在框504处，将时空查询转换模块与语言模型连接，进行第二阶段的预训练。例如，结合图3所述，可以将时空查询转换模块312与语言模型324进行连接，来训练时空查询转换模块312。在一些实施例中，语言模型324的参数可以保持不变，这样可以加快模型训练的过程。此外，为了实现将时空查询转换模块与语言模型进行连接，可以使用MLP层将视频特征从时空查询转换模块投影到语言模型的嵌入空间。例如，结合图3所述，可以利用MLP层332来连接时空查询转换模块312和语言模型324，MLP层332可以将视频特征320投影到语言模型324的嵌入空间。
图6示出了根据本公开的实施例的与视频处理系统进行交互的过程600的示意图。如图6所示，用户可以向视频处理系统输入是视频602，其中视频处理系统可以具有根据本公开的实施例的视频处理模型。例如，视频602可以为与车辆有关的视频。在一些实施例中，视频处理系统还可以在视频602的下方显示图像帧。然后，用户可以与视频处理系统进行交互。例如，用户可以发送对话604“请问在这个视频中发生了什么”，然后视频处理系统可以生成与相应的对话606“在这个视频中，一辆汽车的后视镜在车祸后受损”。此外，视频处理系统还可以支持多轮对话。例如，用户可以输入对话608来进一步询问与视频602有关的内容，视频处理系统可以进一步生成对话610进行回答。
图7示出了根据本公开的实施例的视频处理的装置700的框图。如图7所示，装置700包括时空特征确定模块702，被配置为确定视频的时间特征和空间特征。此外，装置700还包括文本输出生成模块704，被配置为基于所述时间特征和所述空间特征，使用时空查询转换模块生成与所述视频相对应的文本输出，所述时空查询转换模块包括时间查询和对应的时间专家网络以及空间查询和对应的空间专家网络。
图8示出了根据本公开的某些实施例的电子设备800的框图。图8示出了根据本公开的某些实施例的电子设备800的框图，设备800可以是本公开的实施例所描述的设备或装置。如图8所示，设备800包括中央处理单元(CPU)和/或图形处理单元(GPU)801，其可以根据存储在只读存储器(ROM)802中的计算机程序指令或者从存储单元808加载到随机访问存储器(RAM)803中的计算机程序指令，来执行各种适当的动作和处理。在RAM 803中，还可以存储设备800操作所需的各种程序和数据。CPU/GPU 801、ROM 802以及RAM 803通过总线804彼此相连。输入/输出(I/O)接口805也连接至总线804。虽然未在图8中示出，设备800还可以包括协处理器。
设备800中的多个部件连接至I/O接口805，包括：输入单元806，例如键盘、鼠标等；输出单元807，例如各种类型的显示器、扬声器等；存储单元808，例如磁盘、光盘等；以及通信单元809，例如网卡、调制解调器、无线通信收发机等。通信单元809允许设备800通过诸如互联网的计算机网络和/或各种电信网络与其他设备交换信息/数据。
上文所描述的各个方法或过程可以由CPU/GPU 801来执行。例如，在一些实施例中，方法可被实现为计算机软件程序，其被有形地包含于机器可读介质，例如存储单元808。在一些实施例中，计算机程序的部分或者全部可以经由ROM 802和/或通信单元809而被载入和/或安装到设备800上。当计算机程序被加载到RAM 803并由CPU/GPU 801执行时，可以执行上文描述的方法或过程中的一个或多个步骤或动作。
在一些实施例中，以上所描述的方法和过程可以被实现为计算机程序产品。计算机程序产品可以包括计算机可读存储介质，其上载有用于执行本公开的各个方面的计算机可读程序指令。
计算机可读存储介质可以是可以保持和存储由指令执行设备使用的指令的有形设备。计算机可读存储介质例如可以是但不限于电存储设备、磁存储设备、光存储设备、电磁存储设备、半导体存储设备或者上述的任意合适的组合。计算机可读存储介质的更具体的例子(非穷举的列表)包括：便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、静态随机存取存储器(SRAM)、便携式压缩盘只读存储器(CD-ROM)、数字多功能盘(DVD)、记忆棒、软盘、机械编码设备、例如其上存储有指令的打孔卡或凹槽内凸起结构、以及上述的任意合适的组合。这里所使用的计算机可读存储介质不被解释为瞬时信号本身，诸如无线电波或者其他自由传播的电磁波、通过波导或其他传输媒介传播的电磁波(例如，通过光纤电缆的光脉冲)、或者通过电线传输的电信号。
本文所描述的计算机可读程序指令可以从计算机可读存储介质下载到各个计算/处理设备，或者通过网络、例如互联网、局域网、广域网和/或无线网下载到外部计算机或外部存储设备。网络可以包括铜传输电缆、光纤传输、无线传输、路由器、防火墙、交换机、网关计算机和/或边缘服务器。每个计算/处理设备中的网络适配卡或者网络接口从网络接收计算机可读程序指令，并转发该计算机可读程序指令，以供存储在各个计算/处理设备中的计算机可读存储介质中。
用于执行本公开操作的计算机程序指令可以是汇编指令、指令集架构(ISA)指令、机器指令、机器相关指令、微代码、固件指令、状态设置数据、或者以一种或多种编程语言的任意组合编写的源代码或目标代码，编程语言包括面向对象的编程语言，以及常规的过程式编程语言。计算机可读程序指令可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络—包括局域网(LAN)或广域网(WAN)—连接到用户计算机，或者，可以连接到外部计算机(例如利用互联网服务提供商来通过互联网连接)。在一些实施例中，通过利用计算机可读程序指令的状态信息来个性化定制电子电路，例如可编程逻辑电路、现场可编程门阵列(FPGA)或可编程逻辑阵列(PLA)，该电子电路可以执行计算机可读程序指令，从而实现本公开的各个方面。
这些计算机可读程序指令可以提供给通用计算机、专用计算机或其它可编程数据处理装置的处理单元，从而生产出一种机器，使得这些指令在通过计算机或其它可编程数据处理装置的处理单元执行时，产生了实现流程图和/或框图中的一个或多个框中规定的功能/动作的装置。也可以把这些计算机可读程序指令存储在计算机可读存储介质中，这些指令使得计算机、可编程数据处理装置和/或其他设备以特定方式工作，从而，存储有指令的计算机可读介质则包括一个制造品，其包括实现流程图和/或框图中的一个或多个框中规定的功能/动作的各个方面的指令。
也可以把计算机可读程序指令加载到计算机、其它可编程数据处理装置、或其它设备上，使得在计算机、其它可编程数据处理装置或其它设备上执行一系列操作步骤，以产生计算机实现的过程，从而使得在计算机、其它可编程数据处理装置、或其它设备上执行的指令实现流程图和/或框图中的一个或多个框中规定的功能/动作。
附图中的流程图和框图显示了根据本公开的多个实施例的设备、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个框可以代表一个模块、程序段或指令的一部分，所述模块、程序段或指令的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。在有些作为替换的实现中，框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个连续的框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这取决于所涉及的功能。也要注意的是，框图和/或流程图中的每个框、以及框图和/或流程图中的框的组合，可以用执行规定的功能或动作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
以上已经描述了本公开的各实施例，上述说明是示例性的，并非穷尽性的，并且也不限于所公开的各实施例。在不偏离所说明的各实施例的范围和精神的情况下，对于本技术领域的普通技术人员来说许多修改和变更都是显而易见的。本文中所用术语的选择，旨在最好地解释各实施例的原理、实际应用或对市场中技术的技术改进，或者使得本技术领域的其它普通技术人员能理解本文公开的各实施例。
下列出了本公开的一些示例实现。
示例1.一种视频处理的方法，包括：
确定视频的时间特征和空间特征；以及
基于所述时间特征和所述空间特征，使用时空查询转换模块生成与所述视频相对应的文本输出，所述时空查询转换模块包括时间查询和对应的时间专家网络以及空间查询和对应的空间专家网络。
示例2.根据示例1所述的方法，其中所述时间特征和所述空间特征通过注意力池化模块而被生成，所述方法还包括：
基于所述视频，生成所述视频的视觉特征；以及
基于所述视觉特征，利用所述注意力池化模块中的交叉注意力网络和前馈神经网络，生成所述时间特征和所述空间特征。
示例3.根据示例1-2中任一项所述的方法，其中所述时空查询转换模块还包括融合查询和对应的融合专家网络。
示例4.根据示例1-3中任一项所述的方法，还包括：
获取提示文本；以及
基于所述时间特征、所述空间特征以及所述提示文本，使用所述时空查询转换模块生成所述提示文本指示的文本输出。
示例5.根据示例1-4中任一项所述的方法，其中生成所述提示文本指示的所述文本输出包括：
基于所述时间特征和所述空间特征，利用所述时空查询转换模块生成视频特征；以及
基于所述视频特征和所述提示文本，生成所述提示文本指示的所述文本输出。
示例6.根据示例1-5中任一项所述的方法，其中时空查询转换模块还包括交叉注意力掩码，所述交叉注意力掩码包括时间掩码、空间掩码和融合掩码。
示例7.根据示例1-6中任一项所述的方法，其中所述文本输出由视频处理模型生成，所述视频处理模型包括所述时空查询转换模块，所述方法还包括：
基于训练视频、所述训练视频的视频描述文本和训练提示文本，训练所述视频处理模型。
示例8.根据示例1-7中任一项所述的方法，其中训练所述视频处理模型包括：
基于所述训练视频，由视觉编码器生成训练视觉特征；
基于所述训练视觉特征，由注意力池化模块生成训练时间特征和训练空间特征；
基于所述训练时间特征、所述训练空间特征和所述视频描述文本，由所述时空查询转换模块生成转换输出；以及
基于所述转换输出，训练所述注意力池化模块和所述时空查询转换模块。
示例9.根据示例1-8中任一项所述的方法，其中训练所述注意力池化模块和所述时空查询转换模块包括：
基于视频文本匹配损失函数、视频文本对比损失函数和视频定位文本生成损失函数，训练所述注意力池化模块和所述时空查询转换模块。
示例10.根据示例1-9中任一项所述的方法，还包括：
基于所述转换输出和所述训练提示文本，由语言模型输出所述训练提示文本指示的目标输出；以及
基于所述训练提示文本指示的目标输出，训练所述注意力池化模块和所述时空查询转换模块，所述语言模型的参数在训练中保持不变。
示例11.一种视频处理的装置，包括：
时空特征确定模块，被配置为确定视频的时间特征和空间特征；以及
文本输出生成模块，被配置为基于所述时间特征和所述空间特征，使用时空查询转换模块生成与所述视频相对应的文本输出，所述时空查询转换模块包括时间查询和对应的时间专家网络以及空间查询和对应的空间专家网络。
示例12.根据示例11所述的装置，其中所述时间特征和所述空间特征通过注意力池化模块而被生成，所述装置还包括：
视觉特征生成模块，被配置为基于所述视频，生成所述视频的视觉特征；以及时空特征生成模块，被配置为基于所述视觉特征，利用所述注意力池化模块中的交叉注意力网络和前馈神经网络，生成所述时间特征和所述空间特征。
示例13.根据示例11-12中任一项所述的装置，其中所述时空查询转换模块还包括融合查询和对应的融合专家网络。
示例14.根据示例11-13中任一项所述的装置，所述装置还包括：
提示文本获取模块，被配置为获取提示文本；以及
指示输出生成模块，被配置为基于所述时间特征、所述空间特征以及所述提示文本，使用所述时空查询转换模块生成所述提示文本指示的文本输出。
示例15.根据示例11-14中任一项所述的装置，其中所述指示输出生成模块包括：
视频特征生成模块，被配置为基于所述时间特征和所述空间特征，利用所述时空查询转换模块生成视频特征；以及
第二指示输出生成模块，被配置为基于所述视频特征和所述提示文本，生成所述提示文本指示的所述文本输出。
示例16.根据示例11-15中任一项所述的装置，其中时空查询转换模块还包括交叉注意力掩码，所述交叉注意力掩码包括时间掩码、空间掩码和融合掩码。
示例17.根据示例11-16中任一项所述的装置，其中所述文本输出由视频处理模型生成，所述视频处理模型包括所述时空查询转换模块，所述装置还包括：
视频处理模型训练模块，被配置为基于训练视频、所述训练视频的视频描述文本和训练提示文本，训练所述视频处理模型。
示例18.根据示例11-17中任一项所述的装置，其中所述视频处理模型训练模块包括：
训练视觉特征生成模块，被配置为基于所述训练视频，由视觉编码器生成训练视觉特征；
训练时空特征生成模块，被配置为基于所述训练视觉特征，由注意力池化模块生成训练时间特征和训练空间特征；
转换输出生成模块，被配置为基于所述训练时间特征、所述训练空间特征和所述视频描述文本，由所述时空查询转换模块生成转换输出；以及
处理模型第二训练模块，被配置为基于所述转换输出，训练所述注意力池化模块和所述时空查询转换模块。
示例19.根据示例11-18中任一项所述的装置，其中所述处理模型第二训练模块包括：
处理模型第三训练模块，被配置为基于视频文本匹配损失函数、视频文本对比损失函数和视频定位文本生成损失函数，训练所述注意力池化模块和所述时空查询转换模块。
示例20.根据示例11-19中任一项所述的装置，所述装置还包括：
目标输出成模块，被配置为基于所述转换输出和所述训练提示文本，由语言模型输出所述训练提示文本指示的目标输出；以及
处理模型第四训练模块，基于所述训练提示文本指示的目标输出，训练所述注意力池化模块和所述时空查询转换模块，所述语言模型的参数在训练中保持不变。
示例21.一种电子设备，包括：
处理器；以及
与所述处理器耦合的存储器，所述存储器具有存储于其中的指令，所述指令在被处理器执行时，使得所述电子设备执行动作，所述动作包括：
确定视频的时间特征和空间特征；以及
基于所述时间特征和所述空间特征，使用时空查询转换模块生成与所述视频相对应的文本输出，所述时空查询转换模块包括时间查询和对应的时间专家网络以及空间查询和对应的空间专家网络。
示例22.根据示例21所述的电子设备，其中所述时间特征和所述空间特征通过注意力池化模块而被生成，所述方法还包括：
基于所述视频，生成所述视频的视觉特征；以及
基于所述视觉特征，利用所述注意力池化模块中的交叉注意力网络和前馈神经网络，生成所述时间特征和所述空间特征。
示例23.根据示例21-22中任一项所述的电子设备，其中所述时空查询转换模块还包括融合查询和对应的融合专家网络。
示例24.根据示例21-23中任一项所述的电子设备，所述动作还包括：
获取提示文本；以及
基于所述时间特征、所述空间特征以及所述提示文本，使用所述时空查询转换模块生成所述提示文本指示的文本输出。
示例25.根据示例21-24中任一项所述的电子设备，其中生成所述提示文本指示的所述文本输出包括：
基于所述时间特征和所述空间特征，利用所述时空查询转换模块生成视频特征；以及
基于所述视频特征和所述提示文本，生成所述提示文本指示的所述文本输出。
示例26.根据示例21-25中任一项所述的电子设备，其中时空查询转换模块还包括交叉注意力掩码，所述交叉注意力掩码包括时间掩码、空间掩码和融合掩码。
示例27.根据示例21-26中任一项所述的电子设备，其中所述文本输出由视频处理模型生成，所述视频处理模型包括所述时空查询转换模块，所述方法还包括：
基于训练视频、所述训练视频的视频描述文本和训练提示文本，训练所述视频处理模型。
示例28.根据示例21-27中任一项所述的电子设备，其中训练所述视频处理模型包括：
基于所述训练视频，由视觉编码器生成训练视觉特征；
基于所述训练视觉特征，由注意力池化模块生成训练时间特征和训练空间特征；
基于所述训练时间特征、所述训练空间特征和所述视频描述文本，由所述时空查询转换模块生成转换输出；以及
基于所述转换输出，训练所述注意力池化模块和所述时空查询转换模块。
示例29.根据示例21-28中任一项所述的电子设备，其中训练所述注意力池化模块和所述时空查询转换模块包括：
基于视频文本匹配损失函数、视频文本对比损失函数和视频定位文本生成损失函数，训练所述注意力池化模块和所述时空查询转换模块。
示例30.根据示例21-29中任一项所述的电子设备，所述动作还包括：
基于所述转换输出和所述训练提示文本，由语言模型输出所述训练提示文本指示的目标输出；以及
基于所述训练提示文本指示的目标输出，训练所述注意力池化模块和所述时空查询转换模块，所述语言模型的参数在训练中保持不变。
示例31.一种计算机可读存储介质，其上存储有一条或多条计算机指令，其中所述一条或多条计算机指令被处理器执行以实现根据示例1至10中任一项所述的方法。
示例32.一种计算机程序产品，所述计算机程序产品被有形地存储在计算机可读介质上并且包括计算机可执行指令，所述计算机可执行指令在由设备执行时使所述设备执行根据示例1至10中任一项所述的方法。
尽管已经采用特定于结构特征和/或方法逻辑动作的语言描述了本公开，但是应当理解所附权利要求书中所限定的主题未必局限于上面描述的特定特征或动作。相反，上面所描述的特定特征和动作仅仅是实现权利要求书的示例形式。

Claims (13)

  1. 一种视频处理的方法，包括：
    确定视频的时间特征和空间特征；以及
    基于所述时间特征和所述空间特征，使用时空查询转换模块生成与所述视频相对应的文本输出，所述时空查询转换模块包括时间查询和对应的时间专家网络以及空间查询和对应的空间专家网络。
  2. 根据权利要求1所述的方法，其中所述时间特征和所述空间特征通过注意力池化模块而被生成，所述方法还包括：
    基于所述视频，生成所述视频的视觉特征；以及
    基于所述视觉特征，利用所述注意力池化模块中的交叉注意力网络和前馈神经网络，生成所述时间特征和所述空间特征。
  3. 根据权利要求2所述的方法，其中所述时空查询转换模块还包括融合查询和对应的融合专家网络。
  4. 根据权利要求1所述的方法，还包括：
    获取提示文本；以及
    基于所述时间特征、所述空间特征以及所述提示文本，使用所述时空查询转换模块生成所述提示文本指示的文本输出。
  5. 根据权利要求4所述的方法，其中生成所述提示文本指示的所述文本输出包括：
    基于所述时间特征和所述空间特征，利用所述时空查询转换模块生成视频特征；以及
    基于所述视频特征和所述提示文本，生成所述提示文本指示的所述文本输出。
  6. 根据权利要求5所述的方法，其中时空查询转换模块还包括交叉注意力掩码，所述交叉注意力掩码包括时间掩码、空间掩码和融合掩码。
  7. 根据权利要求1所述的方法，其中所述文本输出由视频处理模型生成，所述视频处理模型包括所述时空查询转换模块，所述方法还包括：
    基于训练视频、所述训练视频的视频描述文本和训练提示文本，训练所述视频处理模型。
  8. 根据权利要求7所述的方法，其中训练所述视频处理模型包括：
    基于所述训练视频，由视觉编码器生成训练视觉特征；
    基于所述训练视觉特征，由注意力池化模块生成训练时间特征和训练空间特征；
    基于所述训练时间特征、所述训练空间特征和所述视频描述文本，由所述时空查询转换模块生成转换输出；以及
    基于所述转换输出，训练所述注意力池化模块和所述时空查询转换模块。
  9. 根据权利要求8所述的方法，其中训练所述注意力池化模块和所述时空查询转换模块包括：
    基于视频文本匹配损失函数、视频文本对比损失函数和视频定位文本生成损失函数，训练所述注意力池化模块和所述时空查询转换模块。
  10. 根据权利要求8所述的方法，还包括：
    基于所述转换输出和所述训练提示文本，由语言模型输出所述训练提示文本指示的目标输出；以及
    基于所述训练提示文本指示的目标输出，训练所述注意力池化模块和所述时空查询转换模块，所述语言模型的参数在训练中保持不变。
  11. 一种视频处理的装置，包括：
    时空特征确定模块，被配置为确定视频的时间特征和空间特征；以及
    文本输出生成模块，被配置为基于所述时间特征和所述空间特征，使用时空查询转换模块生成与所述视频相对应的文本输出，所述时空查询转换模块包括时间查询和对应的时间专家网络以及空间查询和对应的空间专家网络。
  12. 一种电子设备，包括：
    处理器；以及
    与所述处理器耦合的存储器，所述存储器具有存储于其中的指令，所述指令在被处理器执行时，使得所述电子设备执行根据权利要求1至10中任一项所述的方法。
  13. 一种计算机程序产品，所述计算机程序产品被有形地存储在非瞬态计算机可读介质上并且包括计算机可执行指令，所述计算机可执行指令用于执行根据权利要求1至10中的任一项所述的方法。
PCT/CN2024/133587 2024-04-19 2024-11-21 视频处理的方法、装置、电子设备和计算机程序产品 Pending WO2025218166A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202410479933.7 2024-04-19
CN202410479933.7A CN118590707A (zh) 2024-04-19 2024-04-19 视频处理的方法、装置、电子设备和计算机程序产品

Publications (1)

Publication Number Publication Date
WO2025218166A1 true WO2025218166A1 (zh) 2025-10-23

Family

ID=92536279

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2024/133587 Pending WO2025218166A1 (zh) 2024-04-19 2024-11-21 视频处理的方法、装置、电子设备和计算机程序产品

Country Status (2)

Country Link
CN (1) CN118590707A (zh)
WO (1) WO2025218166A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118590707A (zh) * 2024-04-19 2024-09-03 抖音视界有限公司 视频处理的方法、装置、电子设备和计算机程序产品

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111723238A (zh) * 2019-03-22 2020-09-29 曜科智能科技（上海）有限公司 视频多事件的裁剪及文本描述方法及其装置、设备和介质
CN117351387A (zh) * 2023-09-12 2024-01-05 北京百度网讯科技有限公司 视频对话及模型训练方法、装置、设备和存储介质
CN117370604A (zh) * 2023-10-09 2024-01-09 中国民航大学 基于视频时空场景图融合推理的视频描述生成方法及系统
US20240061998A1 (en) * 2022-08-21 2024-02-22 Nec Laboratories America, Inc. Concept-conditioned and pretrained language models based on time series to free-form text description generation
CN118590707A (zh) * 2024-04-19 2024-09-03 抖音视界有限公司 视频处理的方法、装置、电子设备和计算机程序产品

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109978021B (zh) * 2019-03-07 2022-09-16 北京大学深圳研究生院 一种基于文本不同特征空间的双流式视频生成方法
US11288438B2 (en) * 2019-11-15 2022-03-29 Salesforce.Com, Inc. Bi-directional spatial-temporal reasoning for video-grounded dialogues
US11210523B2 (en) * 2020-02-06 2021-12-28 Mitsubishi Electric Research Laboratories, Inc. Scene-aware video dialog
CN111652202B (zh) * 2020-08-10 2020-12-01 浙江大学 利用自适应的时空图模型通过提升视频-语言表征学习来解决视频问答问题的方法及其系统
CN112261491B (zh) * 2020-12-22 2021-04-16 北京达佳互联信息技术有限公司 视频时序标注方法、装置、电子设备及存储介质
US11989941B2 (en) * 2021-11-16 2024-05-21 Salesforce, Inc. Systems and methods for video and language pre-training
CN114428866B (zh) * 2022-01-26 2025-05-02 杭州电子科技大学 一种基于面向对象的双流注意力网络的视频问答方法
CN115438169A (zh) * 2022-11-08 2022-12-06 苏州浪潮智能科技有限公司 一种文本与视频的互检方法、装置、设备及存储介质
CN116013299A (zh) * 2022-11-10 2023-04-25 江南大学 一种局部语义指导的多特征融合视频文本生成方法
CN116721372B (zh) * 2023-06-02 2025-09-05 桂林电子科技大学 一种视频描述方法、装置、系统以及存储介质
CN117113270A (zh) * 2023-08-03 2023-11-24 中山大学 一种基于改进对齐方法的知识融合多模态交互方法及装置
CN117251821B (zh) * 2023-09-13 2026-04-28 山东大学 一种视频-语言理解方法及系统

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111723238A (zh) * 2019-03-22 2020-09-29 曜科智能科技（上海）有限公司 视频多事件的裁剪及文本描述方法及其装置、设备和介质
US20240061998A1 (en) * 2022-08-21 2024-02-22 Nec Laboratories America, Inc. Concept-conditioned and pretrained language models based on time series to free-form text description generation
CN117351387A (zh) * 2023-09-12 2024-01-05 北京百度网讯科技有限公司 视频对话及模型训练方法、装置、设备和存储介质
CN117370604A (zh) * 2023-10-09 2024-01-09 中国民航大学 基于视频时空场景图融合推理的视频描述生成方法及系统
CN118590707A (zh) * 2024-04-19 2024-09-03 抖音视界有限公司 视频处理的方法、装置、电子设备和计算机程序产品

Also Published As

Publication number Publication date
CN118590707A (zh) 2024-09-03

Similar Documents

Publication Publication Date Title
JP7429734B2 (ja) マルチモーダルデータ連合学習モデルトレーニング方法及び装置
CN114565812B (zh) 语义分割模型的训练方法、装置和图像的语义分割方法
CN114092759A (zh) 图像识别模型的训练方法、装置、电子设备及存储介质
KR102601159B1 (ko) 버추얼 휴먼 인터렉션 생성 장치 및 그에 대한 방법
CN111382647A (zh) 一种图片处理方法、装置、设备及存储介质
US20230139994A1 (en) Method for recognizing dynamic gesture, device, and storage medium
CN118644596A (zh) 一种人脸关键点运动图像生成方法以及相关设备
CN114399646A (zh) 一种基于Transformer结构的图像描述方法和装置
CN120849867B (zh) 一种基于多模态大模型的三维场景理解与指令分析方法
WO2025218166A1 (zh) 视频处理的方法、装置、电子设备和计算机程序产品
CN115762484A (zh) 用于语音识别的多模态数据融合方法、装置、设备及介质
CN119538967A (zh) 一种基于多模态理解的数字人交互方法及系统
CN118781234A (zh) 图像数据处理方法和模型训练方法
CN117010501B (zh) 一种基于场景图神经网络推理机制的视觉问答系统及方法
CN120833425B (zh) 基于多模态统一嵌入空间引导的听话人视频生成方法
CN116704588A (zh) 面部图像的替换方法、装置、设备及存储介质
CN119228853B (zh) 基于选择性状态空间的目标跟踪方法、系统、装置、介质
WO2022165132A1 (en) Keypoint based action localization
US20260004495A1 (en) Method, electronic device, and computer program product for generating video
CN120472359A (zh) 视频语言模型的训练方法以及人体交互行为识别方法
CN119296104A (zh) 基于三维高斯泼溅的多模态三维实例分割方法
CN118196796A (zh) 基于大语言模型驱动的三维语义占用预测方法及装置
US12112402B2 (en) Method, electronic device, and computer program product for processing target object
CN116862951A (zh) 一种基于Transformer的轻量化目标识别与跟踪系统及方法
Yang et al. AWAD-Fusion: Dynamic multi-sensor fusion framework for robust 3D object detection in adverse weather

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24935694

Country of ref document: EP

Kind code of ref document: A1