WO2025256268A1 - 多模态数据处理方法、装置、电子设备、计算机可读存储介质及计算机程序产品 - Google Patents

多模态数据处理方法、装置、电子设备、计算机可读存储介质及计算机程序产品

Info

Publication number
WO2025256268A1
WO2025256268A1 PCT/CN2025/090125 CN2025090125W WO2025256268A1 WO 2025256268 A1 WO2025256268 A1 WO 2025256268A1 CN 2025090125 W CN2025090125 W CN 2025090125W WO 2025256268 A1 WO2025256268 A1 WO 2025256268A1
Authority
WO
WIPO (PCT)
Prior art keywords
embedding
image
text
multimodal
predicted
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/CN2025/090125
Other languages
English (en)
French (fr)
Inventor
杨帅
葛玉莹
葛艺潇
单瀛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology (Shenzhen) Co Ltd
Original Assignee
Tencent Technology (Shenzhen) Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology (Shenzhen) Co Ltd filed Critical Tencent Technology (Shenzhen) Co Ltd
Publication of WO2025256268A1 publication Critical patent/WO2025256268A1/zh
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
    • G06F16/75—Clustering; Classification
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
    • G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/783—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/27—Regression, e.g. linear or logistic regression
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G06N3/0442—Recurrent networks, e.g. Hopfield networks characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0475—Generative networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/094—Adversarial learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/04—Inference or reasoning models
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/62—Text, e.g. of license plates, overlay texts or captions on TV images
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition
    • G06V30/18—Extraction of features or characteristics of the image
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs

Definitions

  • This application relates to the field of computer technology, and in particular to a multimodal data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product.
  • This application provides a multimodal data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can effectively improve the performance of multimodal large language models and thus improve the quality of multimodal content generation.
  • embodiments of this application provide a multimodal data processing method applied to electronic devices, including:
  • the embedding of the reference text, the embedding of the start marker of the reference visual feature, the embedding of the reference visual feature, and the embedding of the end marker of the reference visual feature are determined by a multimodal large language model.
  • the embeddings of the reference text, the start marker, the reference visual features, and the end marker are concatenated into a target embedding sequence. Attention processing is performed on the embeddings of the start marker, the end marker, and the embeddings selected by the sliding window in the target embedding sequence to output a prediction sequence.
  • a predicted image and predicted text are generated based on the predicted sequence, wherein the predicted text is used to describe the content in the predicted image.
  • embodiments of this application also provide a multimodal data processing apparatus, including:
  • the acquisition module is configured to acquire a reference image and reference text, wherein the reference text is used to describe the content in the reference image;
  • the first processing module is configured to extract reference visual features of the reference image; and determine the embedding of the reference text, the embedding of the start marker of the reference visual features, the embedding of the reference visual features, and the embedding of the end marker of the reference visual features through a multimodal large language model.
  • the second processing module is configured to, based on the multimodal large language model, concatenate the embeddings of the reference text, the start marker, the reference visual features, and the end marker into a target embedding sequence, perform attention processing based on the embeddings of the start marker, the end marker, and the embeddings selected by the sliding window in the target embedding sequence, and output a prediction sequence.
  • the generation module is configured to generate a predicted image and predicted text based on the predicted sequence, wherein the predicted text is used to describe the content in the predicted image.
  • embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described multimodal data processing method.
  • embodiments of this application also provide a computer-readable storage medium storing a computer program, which is executed by a processor to implement the above-described multimodal data processing method.
  • embodiments of this application also provide a computer program product, which includes a computer program stored in a computer-readable storage medium.
  • a processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the multimodal data processing method described above.
  • the embodiments of this application include at least the following beneficial effects: by acquiring a reference image and reference text used to describe the content in the reference image, reference visual features of the reference image are extracted, and start and end markers are added to both ends of the reference visual features, which facilitates the subsequent differentiation of the reference visual features and reference text based on the start and end markers.
  • the reference text and the reference visual features with start and end markers added to both ends are input into a multimodal large language model.
  • the multimodal large language model can better understand the relationship between the text and the image, thereby generating more semantically consistent prediction results.
  • the multimodal large language model can better capture and utilize contextual information, while optimizing the generation of text and images and improving the accuracy of prediction.
  • a prediction image and prediction text describing the prediction image are generated.
  • attention processing is performed on the embeddings of the start marker, the end marker, and the embeddings selected by the sliding window in the target embedding sequence.
  • the sliding window mechanism significantly reduces the computational complexity, enabling the model to maintain efficient inference speed in the generation of long sequences.
  • the multimodal large language model can still focus on the information carried by the embeddings of the start marker, the end marker, and the embeddings selected by the sliding window in the target embedding sequence through attention processing, thereby reducing the perplexity of the multimodal large language model during inference, effectively improving the performance of the multimodal large language model, and thus improving the quality of multimodal content generation.
  • Figure 1 is a schematic diagram of an optional implementation environment provided in an embodiment of this application.
  • FIG. 2 is a schematic diagram of an optional multimodal data processing method provided in an embodiment of this application.
  • FIG. 3 is a schematic diagram of an optional attention processing method provided in an embodiment of this application.
  • Figure 4 is a schematic diagram of an optional process for obtaining predicted visual features according to an embodiment of this application.
  • Figure 5 is a schematic diagram of an optional process for obtaining the embedding of reference visual features according to an embodiment of this application;
  • Figure 6 is a schematic diagram of an optional process for training a multimodal large language model provided in an embodiment of this application;
  • Figure 7 is a schematic diagram of an optional process of the joint training feature extractor and the diffusion model provided in the embodiments of this application;
  • Figure 8 is a schematic diagram of an optional first process for retraining the diffusion model provided in an embodiment of this application.
  • Figure 9 is a schematic diagram of an optional second process for retraining the diffusion model provided in an embodiment of this application.
  • Figure 10 is a schematic diagram of an optional attention score distribution provided in an embodiment of this application.
  • Figure 11 is an optional bar chart of the FID score provided in an embodiment of this application.
  • Figure 12 is an optional pie chart of style consistency comparison results provided in an embodiment of this application.
  • Figure 13 is an optional pie chart of the narrative attractiveness comparison results provided in an embodiment of this application.
  • Figure 14 is an optional pie chart of the image-text consistency comparison results provided in the embodiments of this application.
  • Figure 15 is a schematic diagram of an optional structure of the multimodal data processing device provided in an embodiment of this application.
  • Figure 16 is a partial structural block diagram of the terminal provided in an embodiment of this application.
  • FIG 17 is a partial structural block diagram of the server provided in an embodiment of this application.
  • the target object's permission or consent when processing data related to the characteristics of the target object, such as target object attribute information or attribute information sets, is required, the target object's permission or consent will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards.
  • the target object can be a user. Additionally, when this embodiment needs to obtain target object attribute information, separate permission or consent from the target object will be obtained through pop-ups or redirection to a confirmation page. Only after obtaining the target object's separate permission or consent will the necessary target object-related data for the normal operation of this embodiment be acquired.
  • module refers to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof.
  • a processor or multiple processors or memory
  • each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.
  • Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing.
  • Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology applied to the cloud computing business model. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing.
  • AI Artificial Intelligence
  • AI is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.
  • AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence.
  • AI technology is a comprehensive discipline involving a wide range of fields, encompassing both hardware and software technologies.
  • Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operating/interactive systems, and mechatronics.
  • Pre-trained models also known as large models or foundational models, can be widely applied to downstream tasks in various AI fields after fine-tuning.
  • AI software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, and machine learning/
  • Machine Learning is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance.
  • Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence.
  • Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instruction-based learning.
  • a reference image is visual data used in computer vision and multimedia applications for comparison, analysis, or generation of targets. It is a two-dimensional digital matrix where each element (pixel) represents a color or intensity value.
  • Reference images can be of various types, including but not limited to RGB color images, grayscale images, binary images, depth images, and thermal imaging images.
  • the sources of reference images are wide-ranging, including photographs of natural scenes, works of art, medical images, satellite images, animations, or movie stills.
  • Reference text is textual data used to describe or annotate visual content, such as a reference image. It can be a description, label, title, description, story, dialogue, or any other form of text about the image content, designed to provide semantic understanding, contextual information, or emotional expression of the visual data. Reference text can be manually written or generated using natural language processing techniques.
  • Visual features are abstract information extracted from images using computer vision algorithms, used to describe the content, structure, or attributes of an image. These features can be low-level (such as color, texture, edges, and shape) or high-level (such as object category, scene type, action, and expression). Visual features are usually represented as numerical vectors or matrices and can be used for tasks such as image recognition, classification, retrieval, and generation.
  • Start markers are special symbols or marks used in sequence data processing to indicate the beginning of a sequence. In natural language processing and sequence generation tasks, start markers are often used to prompt the model to begin generating or processing new sequences. Start markers are usually preset symbols or characters; for example, the default start marker is ⁇ BoI>.
  • An end marker is a special symbol or mark used in sequence data processing to indicate the end of a sequence.
  • the end marker tells the model that the generation or processing of a sequence is complete. In generation tasks, the end marker can be used to stop further generation, ensuring that the generated sequence has appropriate length and integrity.
  • the end marker is usually a preset symbol or character; for example, the default end marker is ⁇ EoI>.
  • Predicted lexical units are the basic units used in text analysis and generation. They represent the smallest linguistic unit that a language model considers when predicting the next lexical unit. Predicted lexical units can be complete words, single characters, subwords, or other linguistic units of granularity, depending on the segmentation strategy and model architecture used.
  • embodiments of this application provide a multimodal data processing method, apparatus, electronic device, and storage medium, which can effectively improve the performance of multimodal large language models and thus improve the quality of multimodal content generation.
  • the implementation environment includes a terminal 101 and a server 102, wherein the terminal 101 and the server 102 are connected through a communication network.
  • server 102 can obtain a reference image and reference text sent by terminal 101, wherein the reference text is used to describe the content in the reference image; extract reference visual features of the reference image, and input the reference text and reference visual features with start and end markers added to both ends into a multimodal large language model; determine a target embedding sequence obtained by concatenating the embedding of the reference text, the embedding of the start marker, the embedding of the reference visual features, and the embedding of the end marker based on the multimodal large language model; perform attention processing based on the embedding of the start marker, the embedding of the end marker, and the embedding selected by the sliding window in the target embedding sequence, and output a prediction sequence; generate a prediction image and prediction text based on the prediction sequence, wherein the prediction text is used to describe the content in the prediction image; and send the prediction image and prediction text to terminal 101.
  • Server 102 acquires a reference image and reference text describing the content in the reference image, extracts reference visual features from the reference image, and adds start and end markers to both ends of the reference visual features. This facilitates subsequent differentiation between the reference visual features and the reference text based on the start and end markers. Based on this, the reference text and the reference visual features with added start and end markers are input into a multimodal large language model.
  • the multimodal large language model outputs a prediction sequence, and then generates a prediction image and prediction text describing the prediction image based on the prediction sequence.
  • the multimodal large language model can still focus on the information carried by the embeddings of the start and end markers, as well as the embeddings selected by the sliding window in the target embedding sequence, through attention processing. This reduces the perplexity of the multimodal large language model during inference, effectively improves the performance of the multimodal large language model, and ultimately improves the quality of multimodal content generation.
  • the terminal is used to collect reference images and reference text for art creation, and the server generates predicted images and predicted text that meet the requirements of art creation based on the reference images and reference text, as an art creation assistance tool.
  • the terminal is used to collect reference images and reference text
  • the server generates predicted images and predicted text based on the reference images and reference text for making teaching materials and training content.
  • the terminal is used to collect reference images and reference text
  • the server generates predicted images and predicted text based on the reference images and reference text to provide personalized content recommendations.
  • the terminal is used to collect reference images and reference text
  • the server generates predicted images and predicted text that meet the business requirements set by the advertiser based on the reference images and reference text.
  • Server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Additionally, server 102 can also be a node server in a blockchain network.
  • Terminal 101 can be a mobile phone, computer, smart voice interaction device, smart home appliance, vehicle terminal, etc., but is not limited to these.
  • Terminal 101 and server 102 can be directly or indirectly connected through wired or wireless communication, and this application embodiment does not impose any limitations.
  • the multimodal data processing method can be executed by a server, or by a terminal, or by a server in conjunction with a terminal.
  • the multimodal data processing method includes, but is not limited to, the following steps 201 to 204.
  • the reference image is visual modal data, which refers to a two-dimensional digital matrix composed of pixels.
  • the reference image can be an RGB image, where each pixel corresponds to three color channels. It can also be a grayscale image, a binary image, a depth image, etc., and this embodiment does not limit the specific type of image.
  • the source of the reference image can be diverse; for example, it can be an image from an animation clip, a movie screenshot, or a photographic work.
  • Reference visual features refer to more abstract and informative features extracted from a reference image.
  • Reference visual features may include one visual feature vector or multiple visual feature vectors of the same dimension; for example, reference visual features may include 256 visual feature vectors of 768 dimensions.
  • Reference visual features can be obtained by extracting features from the reference image using feature extractors, such as Convolutional Neural Networks (CNNs), Generative Adversarial Networks (GANs), and Contrastive Language-Image Pre-training (CLIP) models.
  • feature extractors such as Convolutional Neural Networks (CNNs), Generative Adversarial Networks (GANs), and Contrastive Language-Image Pre-training (CLIP) models.
  • Reference visual features can also be implemented through any of the following processing methods:
  • Transfer learning Based on a pre-trained model, visual features for a specific task are extracted through transfer learning. This method can leverage existing knowledge to improve the accuracy of feature extraction.
  • Unsupervised learning Using unsupervised learning methods, such as autoencoders, variational autoencoders (VAE), and generative adversarial networks (GAN), these methods can learn the features of an image without labels.
  • VAE variational autoencoders
  • GAN generative adversarial networks
  • Multi-task learning Through multi-task learning, multiple related tasks are learned simultaneously, such as image classification, object detection, and semantic segmentation, thereby extracting more comprehensive and robust visual features.
  • "extracting reference visual features from a reference image” can be achieved by performing the following processing: converting the reference image into a feature map through a convolutional layer, the feature map includes low-level features of the reference image, such as edges and textures; flattening the feature map into a sequence of feature vectors, each feature vector in the feature vector sequence corresponding to a location or region in the reference image; for each feature vector in the feature vector sequence, performing self-attention calculation on the feature vector to determine the weight of the feature vector, the weight representing the degree of correlation between the current feature vector and other feature vectors; and performing a weighted sum of multiple feature vectors based on the weights to obtain the reference visual features.
  • This application's embodiments extract reference visual features from a reference image using a self-attention mechanism. This allows the information of the entire reference image to be considered when processing the feature vector at each location, thereby capturing the global contextual relationships. Through weight allocation, more attention is paid to local details in the reference image, improving the accuracy of feature extraction.
  • the start marker is usually a preset symbol or character used to indicate the beginning of the sequence. It can be located at the first position of the sequence. For example, the start marker is preset to ⁇ BoI>.
  • the end marker is also a preset symbol or character used to indicate the end of the sequence. It can be located at the last position of the sequence. For example, the end marker is preset to ⁇ EoI>. It can be seen that both the start marker and the end marker can be regarded as text modal data.
  • the two ends of the reference visual feature refer to the beginning and end of the reference visual feature.
  • the start marker and the end marker are added to the two ends of the reference visual feature respectively. For example, the start marker is added to the beginning of the reference visual feature and the end marker is added to the end of the reference visual feature.
  • the start marker, the reference visual feature and the end marker can exist in combination.
  • Multimodal refers to information from different senses or sources, such as vision, hearing, and touch.
  • Multimodal large language models are an extension of large language models. Compared to large language models (LLMs) that can only process input data corresponding to the text modality, multimodal large language models can process input data corresponding to the text modality, as well as input data corresponding to other modalities besides the text modality, such as visual modality, audio modality, etc.
  • LLMs large language models
  • the multimodal large language model can distinguish between reference visual features and reference text based on start and end markers.
  • Reference text, start markers, and end markers are all input data corresponding to the text modality.
  • the multimodal large language model can effectively process reference text, start markers, and end markers.
  • Reference visual features are input data corresponding to the visual modality, and the multimodal large language model can also effectively process reference visual features.
  • the data can be input in various ways.
  • the reference text and reference visual features with start and end markers respectively added to both ends can be input into the multimodal large language model as a batch of data at the same time; another example is to input the reference text and reference visual features with start and end markers respectively added to both ends into the multimodal large language model sequentially; when using the sequential input method, the reference text can be input into the multimodal large language model first, or the reference visual features with start and end markers added can be input into the multimodal large language model first.
  • This embodiment of the application does not limit this.
  • Step 203 Based on the multimodal large language model, determine the target embedding sequence obtained by concatenating the embedding of the reference text, the embedding of the start marker, the embedding of the reference visual features, and the embedding of the end marker. Perform attention processing based on the embedding of the start marker, the embedding of the end marker, and the embedding selected by the sliding window in the target embedding sequence to obtain the prediction sequence.
  • the multimodal large language model can determine the embedding of the corresponding modal data through different embedding modules.
  • the multimodal large language model includes a word embedding module and a graph embedding module.
  • the word embedding module is used to process the input data corresponding to the text modality.
  • the embedding of the reference text, the embedding of the start tag and the embedding of the end tag can be determined.
  • the embedding of the reference text includes the embedding of each word in the reference text.
  • each word, start tag, and end tag of the reference text are recorded as lexical units in a lexicon.
  • Each lexical unit in the lexicon is assigned a unique index.
  • the lexicon can be used to determine the index corresponding to each word, start tag, and end tag of the reference text.
  • the word embedding module maintains a word embedding matrix that stores the word embeddings of each lexical unit.
  • the word embedding module can query the corresponding word embedding in the word embedding matrix based on the index, that is, obtain the embedding of the reference text, the embedding of the start tag, and the embedding of the end tag.
  • the graph embedding module is used to process the input data corresponding to the visual modality.
  • the embedding of the reference visual features can be determined.
  • the reference visual feature includes a visual feature vector
  • the embedding of the reference visual feature is the embedding of the visual feature vector.
  • the reference visual feature includes multiple visual feature vectors
  • the embedding of the reference visual feature includes the embedding of each visual feature vector.
  • the embedding of the visual feature vector has the same dimension as the word embedding mentioned above.
  • the embeddings of the reference text, the start marker, the reference visual features, and the end marker can be directly added together in a preset order to obtain the target embedding sequence.
  • the preset order can be: embedding of the reference text, embedding of the start marker, embedding of the reference visual features, and embedding of the end marker.
  • a multimodal large language model can be used to assign weights to the embeddings of the reference text, the start marker, the reference visual features, and the end marker, and a weighted sum can be performed based on these weights to obtain the target embedding sequence.
  • the embeddings of the reference text (a), the start marker (b), the reference visual feature (c), and the end marker (d) are added together to form the target embedding sequence a+b+c+d.
  • the weight of the reference text embedding (a) is w1
  • the weight of the start marker embedding (b) is w2
  • the weight of the reference visual feature embedding (c) is w3
  • the weight of the end marker embedding (d) is w4 .
  • the target embedding sequence can be represented as a ⁇ w1 + b ⁇ w2 + c ⁇ w3 + d ⁇ w4 .
  • This application's embodiments through weight allocation, can flexibly adjust the proportion of different modal information according to task requirements, making the target embedding sequence more suitable for the task.
  • the weighted summation process effectively fuses information from different modalities, enhancing the model's ability to understand multimodal data.
  • the weight allocation mechanism helps maintain semantic consistency between textual and visual information, enabling the target embedding sequence to better reflect the semantic content of the input data.
  • the target embedding sequence is the input sequence of the multimodal large language model
  • the prediction sequence is the output sequence of the multimodal large language model.
  • the output sequence consists of all the output data of the multimodal large language model.
  • the sliding window is used to determine the attention range of the multimodal large language model.
  • the attention range refers to the specific range that the multimodal large language model focuses on in the input sequence when inferring output data.
  • the embedding of the start marker and the embedding of the end marker can also be selected by the sliding window.
  • the attention range includes the embedding of the start marker and the embedding of the end marker.
  • the attention range includes the embedding of the start marker and the embedding of the end marker that are not selected by the sliding window.
  • the number of embeddings of the start marker or the number of embeddings of the end marker that are not selected by the sliding window can be zero. This application embodiment does not limit this.
  • perplexity can be determined by performing the following steps: For each word in the sequence, calculate the prediction probability of that word given all preceding words, and determine the logarithm of the prediction probability; determine the sum of the logarithms of all words, and determine the ratio of the sum to the length of the sequence to obtain the average log probability; determine the negative of the average log probability, and raise the negative to the power of 2 to obtain the perplexity.
  • the prediction probability is p ⁇ sub>i ⁇ /sub>
  • the logarithm of the prediction probability is log ⁇ sub>2 ⁇ /sub>p ⁇ sub> i ⁇ /sub> . If there are N words in total, the sum of the logarithms of all words is...
  • the average log probability is The negative of the mean logarithm is Confusion level
  • the attention range of the multimodal large language model is relatively narrow. Compared with global attention processing of the target embedding sequence, it can effectively reduce memory usage and significantly improve inference efficiency.
  • Step 204 Generate a predicted image and predicted text based on the predicted sequence.
  • predicted text describes the content of the predicted image and is typically text without special markers.
  • the predicted image is visual modality data, while the predicted text is text modality data.
  • the predicted image is an image associated with a reference image. For example, when using a multimodal large language model for story generation, the generated predicted image is the reference image in the next context; similarly, when using a multimodal large language model for step generation, the generated predicted image is the reference image in the next step; and again, when using a multimodal large language model for content augmentation, the generated predicted image is an image similar to the reference image.
  • markers include, but are not limited to, at least one of the following:
  • Start Token Typically used to mark the beginning of a sequence, such as when generating text or image descriptions, the start token helps the model know where the generation process begins.
  • End Token Used to mark the end of a sequence, ensuring the model knows when to stop generating.
  • Separator Token Used to separate different parts of input data. For example, in multimodal input, separator tokens can be used to separate text and visual features.
  • Mask tokens In some pre-training tasks, such as the masked language model task in BERT, mask tokens are used to cover up certain words in the input sequence, and the model needs to predict the masked words.
  • Class tokens In some models, class tokens are used to represent the classification information of the entire sequence. For example, in image classification tasks, class tokens can be used to represent the category of an image.
  • Position Token Used to represent the position information of each element in the sequence, which is very important for models that need to consider order information.
  • the multimodal large language model can learn the content format of alternating text modality and visual modality, outputting a prediction sequence for generating alternating text modality data and visual modality data, and then generating alternating predicted images and predicted text based on the prediction sequence.
  • the predicted images and predicted text belong to multimodal content.
  • the multimodal large language model can reduce the perplexity of multimodal large language model inference by focusing on the information carried by the embedding of the start tag, the embedding of the end tag, and the embedding selected by the sliding window in the target embedding sequence through attention processing, thereby effectively improving the performance of the multimodal large language model and thus improving the quality of multimodal content generation. It can obtain rich text content and diverse visual content, making the predicted text and predicted image closely intertwined.
  • the prediction sequence consists of all the output data of the multimodal large language model. Whenever the multimodal large language model infers a new output data, the length of the prediction sequence increases.
  • a stopping condition can be set for the multimodal large language model, which can continuously infer new output data until the stopping condition is reached.
  • the stopping condition is that the length of the prediction sequence reaches a preset length threshold. Since the length of the prediction sequence is adjustable, the alternating prediction of images and prediction of text can be single-round or multi-round. This embodiment of the application does not limit this.
  • the prediction sequence includes predicted terms and predicted visual features. Attention processing is performed based on the embeddings of the start marker, the end marker, and the embeddings selected by the sliding window in the target embedding sequence to output the prediction sequence. For example, attention processing can be performed based on the embeddings of the start marker, the end marker, and the embeddings selected by the sliding window in the target embedding sequence to generate predicted embeddings. Predicted embeddings belonging to the text type are classified to output predicted terms. Predicted embeddings belonging to the visual type are mapped to output predicted visual features.
  • predicted lexical units are the basic units in text analysis, and predicted lexical units can be words, characters, subwords, etc.; predicted visual features are located in the visual feature space, and predicted visual features are used to indicate more abstract and informative features of an image.
  • the multimodal large language model Based on this, the multimodal large language model generates a predictive embedding in each round of inference.
  • the multimodal large language model can consider the information carried by the start tag embedding, the end tag embedding, and the embedding selected by the sliding window in the target embedding sequence when generating predictive embeddings, thereby improving the relevance and coherence of the generated content.
  • the multimodal large language model can generate predictive embeddings belonging to text type and predictive embeddings belonging to visual type. Predictive embeddings belonging to text type refer to those that can determine the output data corresponding to the text modality, while predictive embeddings belonging to visual type refer to those that can determine the output data corresponding to the visual modality.
  • the model classifies them to output predictive words corresponding to the text modality.
  • the model maps them to output predictive visual features corresponding to the visual modality.
  • the multimodal large language model has a Transformer decoding layer.
  • the Transformer decoding layer can perform self-attention processing on the input data to generate predictive embeddings.
  • the multimodal large language model processes multimodal data, it can alternately generate a set of predictive embeddings belonging to the text type and a set of predictive embeddings belonging to the visual type, and then alternately output a set of predictive lexical units corresponding to the text modality and a set of predictive visual features corresponding to the visual modality.
  • FIG3 is an optional visualization diagram of attention processing provided in the embodiments of this application; a beginning marker is added to the beginning of the reference text; a target embedding sequence is determined by concatenating the embeddings of the reference text, the beginning marker, the end marker, and the reference visual features based on a multimodal large language model; attention processing is performed based on the beginning marker embedding, the end marker embedding, and the embeddings selected by the sliding window in the target embedding sequence, and a predicted sequence is output.
  • the target embedding sequence is determined by concatenating the embeddings of the beginning marker, the reference text, the beginning marker, the end marker, and the reference visual features based on a multimodal large language model; attention processing is performed based on the beginning marker embedding, the beginning marker embedding, the end marker embedding, and the embeddings selected by the sliding window in the target embedding sequence, and a predicted embedding is generated; the predicted embeddings belonging to the text type are classified and predicted words are output; the predicted embeddings belonging to the visual type are mapped and predicted visual features are output.
  • the beginning mark is a preset symbol or character.
  • the beginning mark is preset to ⁇ BoS>.
  • One or more beginning marks can be added to the beginning of the reference text. This application embodiment does not limit this.
  • attention processing enables multimodal large language models to consider the information carried by the embeddings of the beginning and end tags, the beginning tag, the end tag, and the embeddings selected by the sliding window in the target embedding sequence when generating predictive embeddings, which can further improve the relevance and coherence of the generated content.
  • self-attention processing is performed based on the embedding of the start marker, the embedding of the end marker, and the embedding selected by the sliding window in the target embedding sequence to generate a predicted embedding.
  • the embedding of the start marker, the embedding of the end marker, and the embedding selected by the sliding window in the target embedding sequence can be concatenated to form an embedding sequence to be processed; a first key matrix and a first value matrix are constructed based on the embedding sequence to be processed; a first query matrix is constructed based on the embedding at the last position in the embedding sequence to be processed; and attention processing is performed based on the first query matrix, the first key matrix, and the first value matrix to generate a predicted embedding.
  • the process involves concatenating the embedding sequence to be processed, which consists of the embeddings of the start marker, the end marker, and the embeddings selected by the sliding window in the target embedding sequence, in the order they appear in the target embedding sequence.
  • the first key matrix is the product of the process to be processed embedding sequence and the learnable first key parameter matrix.
  • the first value matrix is the product of the process to be processed embedding sequence and the learnable first value parameter matrix. Both the first key matrix and the first value matrix include the hidden states of each embedding in the process to be processed embedding sequence.
  • the first query matrix is the product of the embedding at the last position in the process to be processed and the learnable first query parameter matrix.
  • the first query matrix includes the hidden state of the embedding at the last position in the process to be processed embedding sequence.
  • the hidden state can be a vector representation processed by the self-attention layer and the feedforward neural network layer. This vector representation includes the model's implicit understanding of each position in the input sequence.
  • each embedding in the sequence has a dimension of 1*768, the first query parameter matrix, the first key parameter matrix, and the first value parameter matrix all have a dimension of 768*768,
  • the embedding sequence to be processed is multiplied by the first key parameter matrix to obtain a 50*768-dimensional first key matrix
  • the embedding sequence to be processed is multiplied by the first value parameter matrix to obtain a 50*768-dimensional first value matrix
  • the embedding at the last position in the embedding sequence to be processed is multiplied by the first query parameter matrix to obtain a 1*768-dimensional first query matrix.
  • the first query matrix indicates the information that needs to be focused on, it is specifically used to find the information most relevant to the last position of the embedding in the sequence to be processed.
  • the first key matrix and the first value matrix both contain information about each embedding in the sequence to be processed.
  • the first key matrix is used to help the multimodal large language model determine the importance of each embedding in the sequence to be processed to the last position of the embedding in the sequence to be processed.
  • the first value matrix is used to generate predicted embeddings according to their importance.
  • the multimodal large language model performs attention processing on the first query matrix, the first key matrix, and the first value matrix to generate predicted embeddings.
  • the source of the first query matrix is the same as the source of the first key matrix and the first value matrix, it is actually a self-attention processing of the first query matrix, the first key matrix, and the first value matrix to generate predicted embeddings.
  • the multimodal large language model can pay more attention to the important information in the sequence to be processed and ignore the secondary information, efficiently utilize and integrate input information, and enhance the model's ability to understand and generate complex outputs.
  • the sequence is loaded into memory and attention processing is performed. Before attention processing, the sequence is positionally encoded based on the relative positions of each embedding in memory, resulting in positional encoding vectors for each embedding. These positional encoding vectors are then used to adjust the embeddings, providing the multimodal large language model with relative positional information for each embedding. This allows the model to learn richer semantic representations. Furthermore, since the sliding window size is fixed and the total number of embeddings for the start and end markers is relatively small, the attention scope of the multimodal large language model is relatively narrow. Compared to loading the entire target embedding sequence into memory for global attention processing, this approach effectively reduces memory usage and significantly improves inference efficiency.
  • self-attention processing is performed based on the embedding of the start marker, the embedding of the end marker, and the embedding selected by the sliding window in the target embedding sequence to generate a predicted embedding.
  • the embedding of the start marker, the embedding of the end marker, and the embedding selected by the sliding window in the target embedding sequence can be concatenated to form an embedding sequence to be processed; a first key matrix and a first value matrix are constructed based on the embedding sequence to be processed; a first query matrix is constructed based on the embedding at the last position in the embedding sequence to be processed; and attention processing is performed based on the first query matrix, the first key matrix, and the first value matrix to generate a predicted embedding.
  • the predicted embedding can be determined in several ways.
  • the first way of determining the predicted embedding is described in detail below.
  • attention processing is performed based on a first query matrix, a first key matrix, and a first value matrix to generate a predicted embedding.
  • an attention score matrix may be obtained by multiplying the transpose of the first query matrix and the first key matrix, adjusting the attention score matrix based on a scaling factor, and normalizing the adjusted attention score matrix to obtain an attention weight matrix.
  • the attention weight matrix includes the attention weights corresponding to each embedding in the sequence of embeddings to be processed.
  • the predicted embedding is then generated based on the product of the adjusted attention weight matrix and the first value matrix.
  • the larger the attention weight corresponding to the embedding in the sequence to be processed the higher the importance of the embedding to the last position in the sequence. Conversely, the smaller the attention weight corresponding to the embedding in the sequence, the lower the importance of the embedding to the last position in the sequence.
  • multimodal large language models can focus more on important information in the embedding sequence to be processed and ignore secondary information through attention weight matrix, efficiently utilize and integrate input information, and enhance the model's ability to understand and generate complex outputs.
  • Ol represents the predicted embedding
  • Ql represents the first query matrix
  • Kl represents the first key matrix. This is the transpose of the first key matrix.
  • d be the scaling factor
  • dk be the dimension of each embedding in the sequence of embeddings to be processed.
  • the adjusted attention score matrix is shown, where softmax is the normalization function. This refers to normalizing the adjusted attention score matrix, where V ⁇ sub> l ⁇ /sub> is the first value matrix.
  • the second method for determining the prediction embedding is described in detail below.
  • attention processing is performed based on a first query matrix, a first key matrix, and a first value matrix to generate a predicted embedding.
  • an attention weight matrix may be determined based on the first query matrix and the first key matrix, wherein the attention weight matrix includes the attention weights corresponding to each embedding in the sequence of embeddings to be processed; the attention weights corresponding to the embeddings of the start tag or the embeddings of the end tag are adjusted to be greater than or equal to a preset weight threshold; and a predicted embedding is generated based on the product of the adjusted attention weight matrix and the first value matrix.
  • the attention weight corresponding to the embedding of the starting label is greater than or equal to the weight threshold.
  • the attention weight corresponding to the embedding of the starting label is adjusted so that the attention weight corresponding to the embedding of the starting label is greater than or equal to the weight threshold.
  • the attention weight corresponding to the embedding of the ending label is less than the weight threshold, the attention weight corresponding to the embedding of the ending label is adjusted so that the attention weight corresponding to the embedding of the ending label is greater than or equal to the weight threshold.
  • the attention weight matrix ensures that the attention weights corresponding to the embeddings of the start and end markers are both greater than or equal to the weight thresholds. Since the attention weights corresponding to the embeddings of the start and end markers are relatively high, the multimodal large language model can maintain attention to the key information carried by the embeddings of the start and end markers when generating predicted embeddings through the attention weight matrix.
  • the target difference between the attention weight corresponding to the initial label embedding and the weight threshold is -0.05.
  • the attention weight corresponding to the initial label embedding needs to be increased by 0.05.
  • the target difference is multiplied by the ratio between the attention weights corresponding to the other embeddings in the embedding sequence to be processed to obtain the adjustment amount of the attention weights corresponding to the other embeddings in the embedding sequence to be processed.
  • the attention weights corresponding to the other embeddings in the embedding sequence to be processed are adjusted based on the adjustment amount to ensure that the sum of the attention weights corresponding to each embedding in the embedding sequence to be processed is equal to one.
  • the predicted embeddings belonging to the visual type are mapped to obtain predicted visual features.
  • a second key matrix and a second value matrix may be constructed based on the predicted embeddings belonging to the visual type; multiple pre-trained first learnable vectors are obtained, and a second query matrix is constructed based on the multiple first learnable vectors, wherein the total dimension of the multiple first learnable vectors is the same as the dimension of the reference visual features; attention processing is performed based on the second query matrix, the second key matrix, and the second value matrix to obtain predicted visual features.
  • the second key matrix is the product of the predicted embedding belonging to the visual type and the learnable second key parameter matrix.
  • the second value matrix is the product of the predicted embedding belonging to the visual type and the learnable second value parameter matrix. Both the second key matrix and the second value matrix include the latent state of the predicted embedding belonging to the visual type.
  • the second query matrix is the product of multiple first learnable vectors and the learnable second query parameter matrix. The second query matrix includes the latent state of multiple first learnable vectors.
  • the learnable second key parameter matrix and the learnable second value parameter matrix can be obtained by performing the following processes: obtaining an initialized deep learning model, an initialized second key parameter matrix, and an initialized second value parameter matrix; predicting the input sequence using the initialized deep learning model to obtain the predicted output; determining the loss based on the input sequence and the true label of the input sequence; determining the gradient of the initialized second key parameter matrix and the initialized second value parameter matrix based on the loss using the backpropagation algorithm; updating the values of the initialized second key parameter matrix and the initialized second value parameter matrix by calling the optimizer according to the gradient until the model converges, thus obtaining the learnable second key parameter matrix and the learnable second value parameter matrix.
  • the second key matrix and the second value matrix are constructed from multiple consecutive predictive embeddings belonging to the visual type. Assuming both the second key matrix and the second value matrix are constructed from 64 consecutive predictive embeddings belonging to the visual type, with each predictive embedding having a dimension of 1*768, the reference visual feature having a dimension of 256*768, and 256 first learnable vectors, each with a dimension of 1*768, for a total dimension of 256*768, the second query matrix is constructed from these 256 first learnable vectors.
  • the second query parameters ...
  • the dimensions of the matrix, the second key parameter matrix, and the second value parameter matrix are all 768*768.
  • Multiplying the 64 predicted embeddings belonging to the visual type with the second key parameter matrix yields a 64*768-dimensional second key matrix.
  • Multiplying the 64 predicted embeddings belonging to the visual type with the second value parameter matrix yields a 64*768-dimensional second value matrix.
  • Multiplying the 256 first learnable vectors with the second query parameter matrix yields a 256*768-dimensional second query matrix. Attention processing is then performed based on the second query matrix, the second key matrix, and the second value matrix to obtain 256*768-dimensional predicted visual features.
  • the above-mentioned "attention processing based on the second query matrix, the second key matrix, and the second value matrix to obtain predicted visual features” can be determined by performing the following process: determining the candidate weight matrix based on the second query matrix and the second key matrix, wherein the candidate weight matrix includes the attention weights corresponding to each embedding in the prediction sequence; generating the predicted visual features based on the product of the candidate weight matrix and the second value matrix.
  • the predicted embedding output by the multimodal large language model is located in the text embedding space.
  • This predicted embedding contains linguistic information.
  • the source of the second query matrix is different from the source of the second key matrix and the second value matrix, it is actually a cross-attention processing of the second query matrix, the second key matrix, and the second value matrix to obtain the predicted visual features. This aligns the predicted visual features with multiple first learnable vectors in the visual feature space, thereby mapping the predicted embedding belonging to the visual type to predicted visual features with visual information.
  • Figure 4 is a schematic diagram of an optional process for obtaining predicted visual features provided by an embodiment of this application; the multimodal large language model internally sets up N stacked first query Transformer layers.
  • the first query Transformer layer includes a first self-attention sublayer, a first cross-attention sublayer, and a first feedforward sublayer.
  • the second Transformer layer is used to map the predicted embeddings belonging to the visual type.
  • the first self-attention sublayer can perform self-attention processing on the multiple first learnable vectors to obtain multiple first hidden vectors.
  • the first cross-attention sublayer can perform cross-attention processing on the multiple first hidden vectors and the predicted embeddings belonging to the visual type to obtain multiple second hidden vectors.
  • the predicted visual features are obtained by mapping the multiple second hidden vectors through the first feedforward sublayer.
  • the first query Transformer layer can be pre-trained in conjunction with the text Transformer layer, which is used to encode the text. During the pre-training process, the output of the first query Transformer layer is aligned with the output of the text Transformer layer. By pre-training the first query Transformer layer, multiple first learnable vectors with visual information can be learned. This allows the first query Transformer layer to filter useful linguistic information from the predicted embedding through cross-attention processing and transform the useful linguistic information from the text embedding space to the visual feature space, which can effectively improve the quality of the predicted image.
  • a target embedding sequence is determined based on a multimodal large language model by concatenating the embeddings of the reference text, the start marker, the reference visual features, and the end marker. For example, based on the multimodal large language model, a third key matrix and a third value matrix are constructed according to the reference visual features; multiple pre-trained second learnable vectors are obtained, and a third query matrix is constructed based on the multiple second learnable vectors, wherein the dimension of the second learnable vectors is the same as the dimension of the embedding of the reference text; attention processing is performed based on the third query matrix, the third key matrix, and the third value matrix to obtain the embedding of the reference visual features; and the embeddings of the reference text, the start marker, the reference visual features, and the end marker are concatenated to obtain the target embedding sequence.
  • the reference text, start tag, and end tag are all input data corresponding to the text modality, they can be embedded separately.
  • the embedding of the reference text, the start tag, and the end tag can be determined.
  • the embedding of the reference text, the start tag, and the end tag are all in the text embedding space, and the dimensions of the embedding of the reference text, the start tag, and the end tag are consistent.
  • the third key matrix is the product of the reference visual features and the learnable third key parameter matrix.
  • the third value matrix is the product of the reference visual features and the learnable third value parameter matrix. Both the third key matrix and the third value matrix include the latent state of the reference visual features.
  • the third query matrix is the product of multiple second learnable vectors and the learnable third query parameter matrix. The third query matrix includes the latent state of multiple second learnable vectors.
  • the third key matrix and the third value matrix are constructed from reference visual features of dimension 256*768.
  • the third query matrix is constructed from the 64 second learnable vectors.
  • the dimensions of the third query parameter matrix, the third key parameter matrix, and the third value parameter matrix are all 768*768.
  • Multiplying the reference visual features by the third key parameter matrix yields a 256*768-dimensional third key matrix.
  • Multiplying the first feature with the third value parameter matrix yields a 256*768 dimensional third value matrix.
  • Multiplying the 64 second learnable vectors with the third query parameter matrix yields a 64*768 dimensional third query matrix.
  • Attention processing is then performed based on the third query matrix, the third key matrix, and the third value matrix to obtain 64 1*768 dimensional embeddings of reference visual features. Therefore, when the predicted word output by the multimodal large language model is the starting word, the 1st to 64th predicted embeddings generated subsequently are used to determine the predicted visual features, and the 65th predicted embedding generated subsequently is used to determine the predicted word.
  • the reference visual features extracted from the reference image reside in the visual feature space.
  • These reference visual features contain visual information.
  • the reference visual features need to be mapped to obtain the embeddings of the reference visual features in the text embedding space.
  • a third query matrix is first constructed from multiple second learnable vectors, along with a third key matrix and a third value matrix constructed from the reference visual features. Then, attention processing is applied to the third query matrix, third key matrix, and third value matrix. Since the source of the third query matrix is different from the sources of the third key matrix and the third value matrix, cross-attention processing is actually performed on the third query matrix, third key matrix, and third value matrix to obtain the embeddings of the reference visual features.
  • FIG. 5 is a schematic diagram of an optional process for obtaining the embedding of reference visual features according to an embodiment of this application; the multimodal large language model internally has N stacked second query Transformer layers.
  • the second query Transformer layer includes a second self-attention sublayer, a second cross-attention sublayer, and a second feedforward sublayer.
  • the second Transformer layer is used to map the reference visual features.
  • the second self-attention sublayer can perform self-attention processing on the multiple second learnable vectors to obtain multiple third hidden vectors.
  • the second cross-attention sublayer can perform cross-attention processing on the multiple third hidden vectors and the reference visual features to obtain multiple fourth hidden vectors.
  • the second feedforward sublayer maps the multiple fourth hidden vectors to obtain the embedding of the reference visual features.
  • the second query Transformer layer can be pre-trained in conjunction with the image Transformer layer, which is used to encode the image. During the pre-training process, the output of the second query Transformer layer is aligned with the output of the image Transformer layer. By pre-training the second query Transformer layer, multiple second learnable vectors with linguistic information can be learned. This allows the second query Transformer layer to filter useful visual information from the predicted embeddings through cross-attention processing and transform the useful visual information from the visual feature space to the text embedding space. This can effectively reduce the burden on multimodal large language models to learn and align visual and text modalities.
  • FIG6 is a schematic diagram of an optional process for generating a predicted image provided by an embodiment of the present application; a predicted image and predicted text are generated based on a predicted sequence, for example, the predicted text can be generated based on predicted words; a randomly generated first noise image is obtained, the first noise image is input into a diffusion model, and the predicted visual features are used as diffusion conditions of the diffusion model to denoise the first noise image to generate a predicted image.
  • the first noisy image is randomly generated, and its pixels are random.
  • the size of the first noisy image can be the same as that of the reference image, that is, the height and width of the first noisy image are the same as those of the reference image, so that the predicted image generated based on the denoising of the first noisy image has the same size as the reference image.
  • the diffusion model has two main processes: a forward process and a backward process.
  • the forward process also known as the diffusion process, involves gradually adding Gaussian noise to the image until it becomes completely random noise.
  • the diffusion process can be a parameterized Markov chain.
  • the backward process also known as the inverse diffusion process, involves using a series of Markov chains to gradually remove the prediction noise at each time step, thereby recovering the data from the Gaussian noise. Therefore, the diffusion model can denoise the first noisy image based on the predicted visual features during the backward process, thereby generating the predicted image. In other words, the denoising process of the diffusion model yields a predicted image that meets the predicted visual features.
  • the diffusion model includes the Unet network, which can denoise noisy images in a backward process.
  • cross-attention processing is applied to the output of the current feature extractor based on predicted visual features to obtain cross-attention features. These cross-attention features are then input to the next feature extractor. Therefore, for each feature extractor in the Unet network, cross-attention processing is applied to the output of the current feature extractor using predicted visual features. This cross-attention processing helps the diffusion model focus more on key regions in the image, improving the accuracy of denoising.
  • the feature extractors in the Unet network can be connected to cross-attention layers. In these layers, query features are determined based on the current feature extractor output, key features and value features are determined based on predicted visual features, and then cross-attention features are determined using the query features, key features, and value features.
  • a diffusion model is used to gradually remove the predicted noise from the first noisy image through multiple time steps, thereby recovering a high-quality, detailed predicted image.
  • the predicted visual features are injected into the diffusion model when predicting noise at each time step, which is equivalent to using the predicted visual features as a global latent code, i.e., global features.
  • the diffusion model considers the predicted visual features in each generation process to ensure that the predicted image matches the predicted visual features, effectively improving the accuracy of the predicted image and thus improving the quality of the predicted image.
  • the multimodal data processing method before inputting the reference text and reference visual features with start and end markers respectively added to both ends into the multimodal large language model, the multimodal data processing method further includes: acquiring a first sample image, a first sample text, a target image, and target text, wherein the first sample text is used to describe the content in the first sample image, the target text is used to describe the content in the target image, and the target image is an image associated with the first sample image; extracting a first visual feature from the first sample image; inputting the first sample text and the first visual feature with start and end markers respectively added to both ends into the multimodal large language model for inference; determining a first loss for inference using the multimodal large language model based on the target text, and determining a second loss for inference using the multimodal large language model based on the target image; and training the multimodal large language model based on the first loss and the second loss.
  • both the first sample image and the target image are visual modal data. Similar to the reference text, both the first sample text and the target text are text modal data.
  • the target image is associated with the first sample image. For example, when using a multimodal large language model for story generation, the target image is the image of the first sample image in the next scene. For another example, when using a multimodal large language model for step generation, the target image is the image of the first sample image in the next step. For yet another example, when using a multimodal large language model for content enhancement, the target image is an image similar to the first sample image.
  • Figure 7 is a schematic diagram of an optional process for training a multimodal large language model provided in an embodiment of this application.
  • the multimodal large language model Before generating predicted text and predicted images using a multimodal large language model, the multimodal large language model needs to be trained first. During the model training process, features are extracted from the first sample image to obtain the first visual features. The first sample text and the first visual features with start and end markers added to both ends are input into the trained multimodal large language model for inference. Similar to the process of generating predicted text, the multimodal large language model can output the corresponding training words and thus generate the corresponding training text. Similarly, similar to the process of generating predicted images, the multimodal large language model can output the corresponding training visual features.
  • a first loss is determined by the target text and training text
  • a second loss is determined by extracting the target visual features from the target image and then by combining the target visual features and training visual features. Training the multimodal large language model based on the first and second losses can improve the prediction accuracy of the multimodal large language model.
  • the first loss can be calculated by inputting the target text and training text into the cross-entropy loss function
  • the second loss can be calculated by inputting the target visual features and training visual features into the cosine similarity loss function.
  • the first loss can also be calculated in other ways
  • the second loss can also be calculated in other ways.
  • the loss function used to determine the first loss and the second loss can also include any of the following: mean squared error loss function, multi-label classification loss function, and triplet loss function. This application embodiment does not limit the specific loss function.
  • the multimodal large language model during model training, the multimodal large language model generates an output sequence by performing the task of predicting the next embedding.
  • the first sample text, start tag, first visual feature, end tag, target text, and target image are concatenated and input into the multimodal large language model. Since the target text and target image are both labeled data, using the target text and target image as input to the multimodal large language model allows the multimodal large language model to generate training text and training images without waiting to generate the embedding at a certain position before generating the embedding at the next position. This enables parallel training, improves training efficiency, and makes it easier for the multimodal large language model to converge during training.
  • the first loss and the second loss are backpropagated to update the parameters of the multimodal large language model.
  • the process of calculating the first loss and the second loss and updating the parameters is repeated multiple times until the loss converges, at which point the iteration process is stopped, and the trained multimodal large language model is obtained.
  • backpropagation is implemented using the backpropagation algorithm, which calculates the gradient of each neuron from the output layer to the input layer and updates the neuron's weights and biases based on the gradients.
  • Gradient descent is used to continuously update the parameters, thus reducing the loss value.
  • Various gradient descent algorithms can be used, such as batch gradient descent, stochastic gradient descent, adaptive gradient descent, and momentum gradient descent.
  • the multimodal large language model is trained based on a first loss and a second loss.
  • the first and second losses can be weighted and summed based on their respective weights to obtain the model loss, and then the multimodal large language model is trained based on this model loss. Therefore, by determining an appropriate model loss, the training performance of the multimodal large language model can be improved.
  • loss weights can be determined in a variety of ways.
  • the loss weights are preset. For example, the loss weights corresponding to the first loss and the second loss are both set to 1. Or, the loss weight corresponding to the first loss is set to 1, and the loss weight corresponding to the second loss is set to 10.
  • the multimodal large language model is trained based on a first loss and a second loss.
  • the first loss and the second loss can be concatenated and input into a regression model for prediction to obtain the loss weights corresponding to the first loss and the second loss respectively.
  • the regression model and the multimodal large language model are jointly trained.
  • the first loss and the second loss are weighted and summed based on the loss weights to obtain the model loss, and the multimodal large language model is trained based on the model loss.
  • the weights of the first loss L1 are:
  • the weight of the second loss L2 is
  • the model loss obtained by weighted summation of the first and second losses is:
  • the first loss and the second loss are concatenated and input into the regression model.
  • the regression model can predict the loss weights corresponding to the first loss and the second loss respectively.
  • the loss weights are used to characterize the contribution of the first loss and the second loss to the model loss.
  • the first loss and the second loss are weighted and summed based on the loss weights to obtain the model loss.
  • the first loss and the second loss can be adaptively adjusted by the loss weights to avoid overfitting.
  • FIG8 is a schematic diagram of an optional first process of the joint training feature extractor and diffusion model provided in the embodiments of this application; the first visual feature is extracted by the feature extractor.
  • the multimodal data processing method further includes: acquiring a second sample image, inputting the second sample image into the feature extractor, and extracting the second visual feature of the second sample image; acquiring a randomly generated second noise image, inputting the second noise image into the diffusion model, using the second visual feature as the diffusion condition of the diffusion model to denoise the second noise image, and generating a reconstructed image; determining a third loss based on the difference between the reconstructed image and the second sample image, and jointly training the feature extractor and the diffusion model based on the third loss.
  • the second sample image is also visual modality data.
  • a feature extractor is used to extract features from the second sample image, extracting more abstract and informative second visual features.
  • the second noise image is randomly generated, and its pixels exhibit randomness.
  • the size of the second noise image can be the same as that of the second sample image; that is, the height and width of the second noise image are the same as those of the second sample image, ensuring that the reconstructed image generated based on the denoising of the second noise image has the same size as the second sample image.
  • the diffusion model is used to gradually remove the predicted noise of the second noisy image through multiple time steps, thereby recovering a high-quality, detailed second sample image.
  • the second visual features are injected into the diffusion model when predicting noise at each time step. This is equivalent to using the second visual features as a global latent code, i.e., global features.
  • the diffusion model considers the second visual features in each generation process to ensure that the second sample image matches the second visual features.
  • feature extraction is performed on the second sample image using a feature extractor, which is equivalent to labeling the second sample image.
  • the second visual features are then used as diffusion conditions for the diffusion model to generate the reconstructed image, which is equivalent to delabeling the second visual features.
  • the third loss is determined by the difference between the reconstructed image and the second sample image.
  • the feature extractor and the diffusion model are jointly trained based on the third loss, ensuring efficient synergy between the feature extractor and the diffusion model. This allows the feature extractor to learn to extract more meaningful second visual features from the second sample image, and the diffusion model to use the second visual features as a guide for denoising, which helps to generate more accurate reconstructed images during the denoising process.
  • This approach can simultaneously improve the feature extraction capability of the feature extractor and the image reconstruction capability of the diffusion model.
  • the third loss can be calculated by inputting the reconstructed image and the second sample image into the mean squared error loss function.
  • the third loss can also be calculated in other ways.
  • the loss function for determining the third loss includes, but is not limited to, any of the following: cross-entropy loss function, multi-label classification loss function, and triplet loss function. This application embodiment does not limit the specific loss.
  • the feature extractor may be frozen.
  • the first visual feature is extracted by the feature extractor.
  • the multimodal data processing method further includes: acquiring a second sample image; inputting the second sample image into the feature extractor to extract the second visual feature of the second sample image; acquiring a randomly generated second noisy image; inputting the second noisy image into a diffusion model; using the second visual feature as a diffusion condition for the diffusion model to denoise the second noisy image and generate a reconstructed image; determining a third loss based on the difference between the reconstructed image and the second sample image; and training the diffusion model based on the third loss. Based on this, the image reconstruction capability of the diffusion model can be improved.
  • the feature extractor and diffusion model can be trained in stages.
  • the parameters of the diffusion model are frozen, and the fourth visual feature of the reconstructed image is extracted.
  • the fifth loss is determined based on the second and fourth visual features.
  • the parameters of the feature extractor are updated based on the fifth loss to obtain the trained feature extractor.
  • the parameters of the trained feature extractor are frozen, and the parameters of the diffusion model are updated based on the third loss to obtain the trained diffusion model.
  • the trained feature extractor and the trained diffusion model are fine-tuned based on the third loss until both converge.
  • This application embodiment employs phased training. In each of the first two phases, only one model is trained, and each model can be optimized independently on its specific task, effectively utilizing computational resources. Training each model individually makes it easier to understand the role and performance of each model, while in the joint training phase, the contribution of each model to the final result becomes clearer.
  • FIG9 is a schematic diagram of an optional second process for retraining the diffusion model provided in the embodiments of this application; after training the multimodal large language model based on the first loss and the second loss, the multimodal data processing method further includes: acquiring a third sample image, a second sample text, and a label image, wherein the second sample text is used to describe the content in the third sample image, and the label image is an image associated with the third sample image; extracting the third visual features of the third sample image, and inputting the second sample text and the third visual features with start and end marks added at both ends respectively into the trained multimodal large language model for inference; acquiring a randomly generated third noise image, inputting the third noise image into the trained diffusion model, and using the visual features output by the trained multimodal large language model as the diffusion condition of the diffusion model to denoise the third noise image to generate an associated image; determining a fourth loss based on the associated image and the label image, and retraining the diffusion model based on the fourth loss.
  • both the third sample image and the label image are visual modal data.
  • the second visual feature refers to a more abstract and informative feature extracted from the third sample image.
  • the third noisy image is randomly generated, and its pixels are random.
  • the size of the third noisy image can be the same as that of the third sample image; that is, the height and width of the third noisy image are the same as those of the third sample image, so that the reconstructed image generated based on the denoising of the third noisy image has the same size as the third sample image.
  • the label image is an image associated with the third sample image.
  • the label image is the image of the third sample image in the next scene.
  • the label image is the image of the third sample image in the next step.
  • the label image is an image that is similar to the third sample image.
  • a diffusion model is used to progressively remove predicted noise from the third noisy image across multiple time steps. Starting from the last time step, noise is gradually reduced to recover a high-quality, detailed third sample image.
  • second visual features are injected into the diffusion model when predicting noise at each time step. This is equivalent to using the second visual features as a global latent code.
  • the attention weight between the second visual features and the global latent code is calculated. Based on the calculated attention weight, the information of the global latent code is fused into the second visual features.
  • the noise at the current time step is predicted using the second visual features that have been fused with the global latent code information.
  • the predicted noise is then removed from the third noisy image to obtain the denoised third sample image. This ensures that the diffusion model considers the second visual features in each generation process to guarantee that the third sample image matches the second visual features.
  • feature extraction is performed on the third sample image to obtain the third visual feature.
  • the second sample text and the third visual feature with start and end markers added to both ends are input into the trained multimodal large language model for inference. Similar to the output process of predicting visual features, the multimodal large language model can output the corresponding visual features. Then, the visual features output by the multimodal large language model are used as the diffusion condition of the diffusion model to denoise the third noisy image and generate associated images.
  • the fourth loss is determined by the associated images and the label images. Based on the fourth loss, the diffusion model is trained again. The diffusion model can generate associated images that are closer to the label images, which is equivalent to fine-tuning the diffusion model to better align the output space of the multimodal large language model with the visual feature space and enhance the style consistency of the generated images.
  • the fourth loss can be calculated by inputting the associated image and the label image into the mean squared error loss function.
  • the fourth loss can also be calculated in other ways.
  • the loss function for determining the fourth loss includes, but is not limited to, any of the following: cross-entropy loss function, multi-label classification loss function, and triplet loss function. This application embodiment does not limit the specific loss.
  • acquiring a first sample image, first sample text, a target image, and target text can be achieved by acquiring multiple consecutive video frames and subtitle text corresponding to each video frame; calling a text description model to perform text generation processing on each video frame to generate description text corresponding to each video frame; constructing a prompt instruction for prompting content enhancement based on the description text; concatenating the prompt instruction, each subtitle text, and the corresponding description text and inputting them into a target large language model; performing content enhancement on each subtitle text based on the target large language model to obtain enhanced text corresponding to each video frame; determining the last video frame as the target image, the remaining video frames as the first sample image, the enhanced text corresponding to the target image as the target text, and the enhanced text corresponding to the first sample image as the first sample text.
  • the multiple consecutive video frames can be obtained from video sources such as animation, movies, and TV programs.
  • the animation video can be cut according to continuous scenes, with one frame captured for each scene, or the animation video can be cut at fixed time intervals, with one frame captured every certain period of time. Then, the multiple consecutive frames obtained are used as multiple consecutive video frames.
  • Each video frame can be an RGB image, grayscale image, binary image, depth image, etc., and this application embodiment does not limit the types of images.
  • Subtitle text refers to the textual display of spoken dialogue, sound effects, musical cues, and other auditory information in video content such as animation, movies, and television programs. Subtitle text is usually synchronized with the audio in multimedia content, allowing viewers to obtain the corresponding content by reading the subtitle text even when they do not hear the sound or need assistance to understand it, thereby improving the viewing experience of the video content.
  • the text description model can perform object detection on video frames and obtain object detection results including the detected objects.
  • the text description model can also extract features from video frames and obtain screen features including scene features and action features of the detected objects. Then, based on the object detection results and screen features, text prediction is performed on the video frames to obtain the corresponding descriptive text for the video frames.
  • the descriptive text is used to characterize the background information of the video frames, such as people, environment, and events.
  • the prompt instruction could be "Please refer to the following description text to enhance the content of the subtitle text.”
  • the prompt instruction, each subtitle text, and the corresponding description text are concatenated and input into the target large language model.
  • the model input format of the target large language model can be "Description text 1, Subtitle text 1, Description text 2, Subtitle text 2" or "Description text 1, Description text 2, Subtitle text 1, Subtitle text 2".
  • the target large language model can integrate the background information carried by the description text into the subtitle text, thereby enhancing the content of the subtitle text.
  • video frames and their corresponding subtitle texts are first acquired. Then, descriptive text for the video frames is generated. Next, the subtitle text is augmented using a target large language model based on the descriptive text to obtain augmented text. Then, video frames and their corresponding augmented texts are used as image-text pairs to construct a dataset.
  • the dataset contains high-resolution video frames and high-quality augmented texts, and can contain a large number of image-text pairs.
  • the last video frame in a series of consecutive video frames is determined as the target image, and the remaining video frames in the series of consecutive video frames are determined as the first sample images.
  • the target image is an image associated with the first sample image, and the target image is used to indicate the next scene of the first sample image.
  • the augmented text corresponding to the target image is determined as the target text
  • the augmented text corresponding to the first sample image is determined as the first sample text.
  • the multimodal large language model is trained using the first sample image, first sample text, target image, and target text, enabling the multimodal large language model to effectively handle tasks such as story generation and step generation, thereby improving the quality of multimodal content generation by the multimodal large language model.
  • the training phase is described in detail below.
  • the training phase can be divided into three sub-phases.
  • the following explanation uses the server (e.g., server 102) as the execution subject.
  • the diffusion model and feature extractor are pre-trained.
  • a second sample image is acquired and input into a feature extractor to extract the second visual features of the second sample image.
  • a randomly generated second noisy image is obtained, and the second noisy image is input into the diffusion model.
  • the second visual feature is used as the diffusion condition of the diffusion model to denoise the second noisy image and generate a reconstructed image.
  • a third loss is determined based on the difference between the reconstructed image and the second sample image, and the feature extractor and diffusion model are jointly trained based on the third loss.
  • the multimodal large language model is trained.
  • the text description model is called to generate text for each video frame, producing description text for each video frame.
  • a prompt instruction for content enhancement based on the reference description text is constructed.
  • the prompt instruction, each subtitle text and the corresponding description text are concatenated and input into the target large language model.
  • the content enhancement of each subtitle text is performed to obtain the enhanced text corresponding to each video frame.
  • the last video frame is determined as the target image
  • the remaining video frames are determined as the first sample image
  • the enhanced text corresponding to the target image is determined as the target text
  • the enhanced text corresponding to the first sample image is determined as the first sample text.
  • the first sample text is used to describe the content in the first sample image
  • the target text is used to describe the content in the target image
  • the target image is the image associated with the first sample image.
  • the first visual features of the first sample image are extracted, and the first visual features with start and end marks added to both ends are input into the multimodal large language model for inference.
  • a first loss is determined based on the target text to perform inference using the multimodal large language model, and a second loss is determined based on the target image to perform inference using the multimodal large language model.
  • the first loss and the second loss are concatenated and input into the regression model for prediction to obtain the loss weights corresponding to the first loss and the second loss respectively.
  • the regression model is jointly trained with the multimodal large language model.
  • the first loss and the second loss are weighted and summed based on the loss weights to obtain the model loss, and the multimodal large language model is trained based on the model loss.
  • the diffusion model is fine-tuned.
  • the second sample text is used to describe the content in the third sample image
  • the label image is an image associated with the third sample image
  • the third visual features of the third sample image are extracted, and the second sample text and the third visual features with start and end markers added to both ends are input into the trained multimodal large language model for inference.
  • a randomly generated third noisy image is obtained and input into the trained diffusion model.
  • the visual features output by the trained multimodal large language model are used as the diffusion conditions of the diffusion model to denoise the third noisy image and generate the associated image.
  • a fourth loss is determined based on the associated image and the label image, and the diffusion model is trained again based on the fourth loss.
  • the reasoning phase will be described in detail below, taking the server (e.g., server 102) as the execution subject as an example.
  • the reference visual features of the reference image are extracted, and the reference text and the reference visual features with start and end marks added to both ends are input into the multimodal large language model.
  • a third key matrix and a third value matrix are constructed according to the reference visual features.
  • attention processing is performed based on the third query matrix, the third key matrix, and the third value matrix to obtain the embedding of the reference visual features.
  • the embeddings of the reference text, the start marker, the reference visual features, and the end marker are concatenated to obtain the target embedding sequence.
  • the embeddings of the start marker, the end marker, and the embeddings selected by the sliding window in the target embedding sequence are concatenated into the embedding sequence to be processed.
  • the first key matrix and the first value matrix are constructed based on the embedded sequence to be processed.
  • the first query matrix is constructed based on the embedding at the last position in the embedding sequence to be processed.
  • the attention weight matrix is determined based on the first query matrix and the first key matrix, wherein the attention weight matrix includes the attention weight corresponding to each embedding in the embedding sequence to be processed.
  • the attention weights corresponding to the embedding of the start tag or the embedding of the end tag are adjusted to be greater than or equal to the preset weight threshold.
  • a predictive embedding is generated based on the product of the adjusted attention weight matrix and the first value matrix.
  • the predicted embeddings belonging to the text type are classified, and the predicted words are output.
  • a second key matrix and a second value matrix are constructed based on the predictive embeddings belonging to the visual type.
  • multiple pre-trained first learnable vectors are obtained, and a second query matrix is constructed based on the multiple first learnable vectors, wherein the total dimension of the multiple first learnable vectors is the same as the dimension of the reference visual features.
  • attention processing is performed based on the second query matrix, the second key matrix, and the second value matrix to obtain the predicted visual features.
  • a randomly generated first noisy image is obtained, and the first noisy image is input into the diffusion model.
  • the predicted visual features are used as the diffusion conditions of the diffusion model to denoise the first noisy image and generate a predicted image.
  • reference visual features of the reference image are extracted. Start and end markers are added to both ends of the reference visual features, which facilitates the subsequent differentiation of the reference visual features and reference text based on the start and end markers.
  • the reference text and the reference visual features with start and end markers added to both ends are input into a multimodal large language model.
  • the multimodal large language model outputs a prediction sequence, and then generates a prediction image and prediction text describing the prediction image based on the prediction sequence.
  • the multimodal large language model can still focus on the information carried by the embeddings of the start marker, the embeddings of the end marker, and the embeddings selected by the sliding window in the target embedding sequence through attention processing, thereby reducing the perplexity of the multimodal large language model during inference, effectively improving the performance of the multimodal large language model, and thus improving the quality of multimodal content generation.
  • Figure 10 is a schematic diagram of an optional distribution of attention score provided in an embodiment of this application.
  • the test embedding sequence is obtained by concatenating the embeddings of the beginning marker, the test text, the start marker, the end marker, and the test visual features.
  • the test visual features are obtained by feature extraction from the test image.
  • the attention score distribution of all embeddings is plotted in the same distribution diagram.
  • the horizontal axis represents the index of the key matrix, which indicates the corresponding embedding
  • the vertical axis represents the index of the query matrix, which indicates the corresponding embedding.
  • images are generated using the Latent Diffusion Model (LDM), the StoryGen model, and the multimodal large language model provided in this application embodiment, respectively.
  • LDM Latent Diffusion Model
  • SGen the StoryGen model
  • multimodal large language model provided in this application embodiment, respectively.
  • the evaluation metrics of each model in FID and CLIP scores can be obtained, as shown in Table 1 below:
  • FID Frechet Inception Distance
  • the FID score is a measure of the distance between the feature vectors of the real image and the generated image. The lower the FID score, the smaller the distance between the feature vectors of the real image and the generated image, and the higher the quality of the image generated by the model.
  • CLIP stands for Contrastive Language-Image Pre-training, and the CLIP score is used to evaluate the relevance between the image and the text. The higher the CLIP score, the higher the relevance between the generated image and the generated text, and the higher the quality of the image generated by the model.
  • the multimodal large language model provided in this application embodiment has the lowest FID score and the highest CLIP score. Therefore, the multimodal large language model provided in this application embodiment generates the highest quality images, and the multimodal large language model provided in this application embodiment has the best performance.
  • each comparative test generates a test image and test text, and then the performance of the two models is determined based on the results of each comparative test.
  • the input content of the multimodal large language model and the image text generation model is consistent, thereby ensuring the reliability of the comparison results.
  • the style consistency score of the two models needs to be calculated.
  • the style consistency score is used to characterize the style similarity between the generated content and the input content.
  • the style consistency score is higher, the style similarity between the generated content and the input content is higher, and vice versa.
  • the probability of the multimodal large language model having a higher style consistency score than the image text generation model is calculated to obtain the winning probability; the probability of the multimodal large language model having a higher style consistency score than the image text generation model is calculated to obtain the tie probability; and the probability of the multimodal large language model having a lower style consistency score than the image text generation model is calculated to obtain the losing probability.
  • the narrative appeal of the generated content of the two models can be determined.
  • narrative appeal can be determined by the audience's voting results on the appeal of the generated content, or it can be determined based on the comments on the generated content.
  • the probability of the multimodal large language model having greater narrative appeal than the image text generation model is obtained; the probability of the multimodal large language model having equal narrative appeal as the image text generation model is obtained; and the probability of the multimodal large language model having less narrative appeal than the image text generation model is obtained.
  • the image-text consistency score of the two models needs to be calculated.
  • the image-text consistency score is used to characterize the correlation between the test image and the test text.
  • the correlation between the test image and the test text is higher, and vice versa.
  • the probability of the multimodal large language model having a higher image-text consistency score than the image-text generation model is obtained; the probability of the multimodal large language model having a higher image-text consistency score than the image-text generation model is obtained; and the probability of a tie is obtained.
  • the probability of the multimodal large language model having a lower image-text consistency score than the image-text generation model is obtained.
  • Figure 11 is an optional bar chart of FID scores provided in an embodiment of this application.
  • the FID score of the multimodal large language model provided in this application embodiment is low. Therefore, the image quality generated by the multimodal large language model provided in this application embodiment is high.
  • Figure 12 is an optional pie chart of style consistency comparison results provided in an embodiment of this application.
  • the probability that the style consistency score of the multimodal large language model provided in this application embodiment is greater than that of the image text generation model is 16.1%, i.e., the winning probability is 16.1%.
  • the probability that the style consistency score of the multimodal large language model provided in this application embodiment is equal to that of the image text generation model is 71.7%, i.e., the tie probability is 71.7%.
  • the probability that the style consistency score of the multimodal large language model provided in this application embodiment is greater than that of the image text generation model is 12.2%, i.e., the failure probability is 12.2%. It is evident that the style consistency of the multimodal large language model provided in this application embodiment is superior to the style consistency of the image text generation model provided by related technologies.
  • Figure 13 is an optional pie chart of the narrative attractiveness comparison results provided in an embodiment of this application.
  • the probability that the narrative appeal of the multimodal large language model provided in this application embodiment is greater than that of the image text generation model is 51.7%, i.e., the probability of winning is 51.7%.
  • the probability that the narrative appeal of the multimodal large language model provided in this application embodiment is equal to that of the image text generation model is 12.8%, i.e., the probability of a tie is 12.8%.
  • the probability that the narrative appeal of the multimodal large language model provided in this application embodiment is greater than that of the image text generation model is 35.6%, i.e., the probability of losing is 35.6%. It is evident that the narrative appeal of the multimodal large language model provided in this application embodiment is significantly better than that of the image text generation model provided by related technologies.
  • Figure 14 is an optional pie chart of the image-text consistency comparison results provided in the embodiments of this application.
  • the probability that the image-text consistency score of the multimodal large language model provided in this application embodiment is greater than that of the image-text consistency score of the image-text generation model is 47.8%, i.e., the winning probability is 47.8%.
  • the probability that the image-text consistency score of the multimodal large language model provided in this application embodiment is equal to that of the image-text generation model is 35.0%, i.e., the tie probability is 35.0%.
  • the probability that the image-text consistency score of the multimodal large language model provided in this application embodiment is greater than that of the image-text generation model is 17.2%, i.e., the failure probability is 17.2%. It is evident that the image-text consistency of the multimodal large language model provided in this application embodiment is significantly better than that of the image-text generation model provided by related technologies.
  • the multimodal large language model provided in this application embodiment has a lower FID score, better style consistency, better narrative appeal, and better image-text consistency. Therefore, the multimodal large language model provided in this application embodiment has better performance.
  • the multimodal data processing method provided in this application embodiment can effectively reduce memory usage and significantly reduce inference time, thereby improving inference efficiency; compared with the sliding window approach, the multimodal data processing method provided in this application embodiment can significantly reduce FID score and increase CLIP score, effectively improving the performance of multimodal large language model, and thus improving the quality of multimodal content generation.
  • Figure 14 is an optional pie chart of the image-text consistency comparison results provided in the embodiments of this application.
  • the multimodal data processing method provided in this application embodiment can be applied to a variety of scenarios.
  • educational step-by-step graphics and text can be expanded. These graphics and text can include experimental steps, knowledge points, and so on.
  • a reference image and reference text for the first step are obtained, and then the reference visual features of the reference image are extracted.
  • the reference text and the reference visual features with start and end markers added to both ends are input into a multimodal large language model.
  • a prediction sequence is output.
  • prediction images and prediction text for subsequent steps are generated. This expands the step-by-step graphics and text, and the prediction images and text can maintain stylistic consistency, content attractiveness, and image-text consistency, thereby improving the overall quality of the step-by-step graphics and text.
  • story content can be expanded by obtaining reference images and reference text for a certain story scenario. Then, reference visual features of the reference images can be extracted. The reference text and reference visual features with start and end markers added to both ends are input into a multimodal large language model. Based on the multimodal large language model, a prediction sequence is output. Then, based on the prediction sequence, prediction images and prediction text for subsequent story scenarios are generated. This expands the story content, and the prediction images and prediction text can maintain stylistic consistency, content appeal, and image-text consistency, thereby improving the overall quality of the story content.
  • steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this embodiment, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.
  • the multimodal data processing apparatus 1500 includes:
  • the acquisition module 1501 is configured to acquire a reference image and reference text, wherein the reference text is used to describe the content in the reference image;
  • the first processing module 1502 is configured to extract reference visual features from the reference image and input the reference text and the reference visual features with start and end marks added to both ends into the multimodal large language model.
  • the second processing module 1503 is configured to determine the target embedding sequence obtained by concatenating the embedding of the reference text, the embedding of the start marker, the embedding of the reference visual features, and the embedding of the end marker based on the multimodal large language model, perform attention processing based on the embedding of the start marker, the embedding of the end marker, and the embedding selected by the sliding window in the target embedding sequence, and output the prediction sequence.
  • the generation module 1504 is configured to generate a predicted image and predicted text based on the predicted sequence, wherein the predicted text is used to describe the content in the predicted image.
  • the prediction sequence includes predicted words and predicted visual features.
  • the second processing module 1503 is further configured to: perform attention processing based on the embedding of the start marker, the embedding of the end marker, and the embedding selected by the sliding window in the target embedding sequence to generate a prediction embedding; classify the prediction embeddings belonging to the text type and output the prediction words; and map the prediction embeddings belonging to the visual type and output the prediction visual features.
  • the second processing module 1503 is further configured to: concatenate the embedding of the start marker, the embedding of the end marker, and the embedding selected by the sliding window in the target embedding sequence into an embedding sequence to be processed; construct a first key matrix and a first value matrix based on the embedding sequence to be processed; construct a first query matrix based on the embedding at the last position in the embedding sequence to be processed; and perform attention processing based on the first query matrix, the first key matrix, and the first value matrix to generate a predicted embedding.
  • the second processing module 1503 is further configured to: determine an attention weight matrix based on a first query matrix and a first key matrix, wherein the attention weight matrix includes the attention weights corresponding to each embedding in the embedding sequence to be processed; adjust the attention weights corresponding to the embeddings of the start tag or the embeddings of the end tag to be greater than or equal to a preset weight threshold; and generate a predicted embedding based on the product of the adjusted attention weight matrix and the first value matrix.
  • the second processing module 1503 is further configured to: determine an attention score matrix based on the product of the transpose of the first query matrix and the first key matrix; adjust the attention score matrix based on a scaling factor; and normalize the adjusted attention score matrix to obtain an attention weight matrix.
  • the second processing module 1503 is further configured to: construct a second key matrix and a second value matrix based on the predicted embedding belonging to the visual type; obtain a plurality of pre-trained first learnable vectors, construct a second query matrix based on the plurality of first learnable vectors, wherein the total dimension of the plurality of first learnable vectors is the same as the dimension of the reference visual feature; and perform attention processing based on the second query matrix, the second key matrix and the second value matrix to obtain the predicted visual feature.
  • the second processing module 1503 is further configured to: determine a candidate weight matrix based on the second query matrix and the second key matrix, wherein the candidate weight matrix includes attention weights corresponding to each embedding in the prediction sequence; and generate predicted visual features based on the product of the candidate weight matrix and the second value matrix.
  • the generation module 1504 is further configured to: generate predicted text based on predicted terms; obtain a randomly generated first noise image; input the first noise image into a diffusion model; use the predicted visual features as diffusion conditions of the diffusion model to denoise the first noise image and generate a predicted image.
  • the second processing module 1503 is further configured to: construct a third key matrix and a third value matrix based on a multimodal large language model and reference visual features; obtain multiple pre-trained second learnable vectors and construct a third query matrix based on the multiple second learnable vectors, wherein the dimension of the second learnable vectors is the same as the dimension of the embedding of the reference text; perform attention processing based on the third query matrix, the third key matrix and the third value matrix to obtain the embedding of the reference visual features; and concatenate the embedding of the reference text, the embedding of the start marker, the embedding of the reference visual features and the embedding of the end marker to obtain the target embedding sequence.
  • the multimodal data processing apparatus further includes a first training module (not shown in the figure), the first training module being configured to: acquire a first sample image, a first sample text, a target image, and target text, wherein the first sample text is used to describe the content in the first sample image, the target text is used to describe the content in the target image, and the target image is an image associated with the first sample image; extract a first visual feature from the first sample image, input the first sample text and the first visual feature with start and end marks added to both ends respectively into a multimodal large language model for inference; determine a first loss for inference using the multimodal large language model based on the target text, determine a second loss for inference using the multimodal large language model based on the target image; and train the multimodal large language model based on the first loss and the second loss.
  • a first training module being configured to: acquire a first sample image, a first sample text, a target image, and target text, wherein the first sample text is used to describe the content in the first sample image,
  • the first training module is further configured to: concatenate the first loss and the second loss and input them into the regression model for prediction to obtain the loss weights corresponding to the first loss and the second loss respectively, wherein the regression model and the multimodal large language model are jointly trained; the first loss and the second loss are weighted and summed based on the loss weights to obtain the model loss, and the multimodal large language model is trained based on the model loss.
  • the multimodal data processing apparatus further includes a second training module (not shown in the figure), the second training module being configured to: acquire a second sample image, input the second sample image into a feature extractor, and extract a second visual feature from the second sample image; acquire a randomly generated second noise image, input the second noise image into a diffusion model, use the second visual feature as a diffusion condition of the diffusion model to denoise the second noise image, and generate a reconstructed image; determine a third loss based on the difference between the reconstructed image and the second sample image, and jointly train the feature extractor and the diffusion model based on the third loss.
  • a second training module being configured to: acquire a second sample image, input the second sample image into a feature extractor, and extract a second visual feature from the second sample image; acquire a randomly generated second noise image, input the second noise image into a diffusion model, use the second visual feature as a diffusion condition of the diffusion model to denoise the second noise image, and generate a reconstructed image; determine a third loss
  • the second training module is further configured to: freeze the parameters of the diffusion model, extract the fourth visual feature of the reconstructed image, determine the fifth loss based on the second visual feature and the fourth visual feature, update the parameters of the feature extractor based on the fifth loss, and obtain the trained feature extractor; freeze the parameters of the trained feature extractor, update the parameters of the diffusion model based on the third loss, and obtain the trained diffusion model; and fine-tune the trained feature extractor and the trained diffusion model based on the third loss.
  • the multimodal data processing apparatus further includes a third training module (not shown in the figure), the third training module being configured to: acquire a third sample image, a second sample text, and a label image, wherein the second sample text is used to describe the content in the third sample image, and the label image is an image associated with the third sample image; extract the third visual features of the third sample image, and input the second sample text and the third visual features with start and end marks added to both ends respectively into the trained multimodal large language model for inference; acquire a randomly generated third noisy image, input the third noisy image into the trained diffusion model, and use the visual features output by the trained multimodal large language model as the diffusion condition of the diffusion model to denoise the third noisy image to generate an associated image; determine a fourth loss based on the associated image and the label image, and train the diffusion model again based on the fourth loss.
  • a third training module (not shown in the figure)
  • the third training module being configured to: acquire a third sample image, a second sample text, and a label image, wherein
  • the first training module is further configured to: acquire multiple consecutive video frames and subtitle text corresponding to each video frame; call a text description model to perform text generation processing on each video frame to generate description text corresponding to each video frame; construct a prompt instruction for prompting content enhancement based on the description text; concatenate the prompt instruction, each subtitle text, and the corresponding description text and input them into a target large language model; perform content enhancement on each subtitle text based on the target large language model to obtain enhanced text corresponding to each video frame; determine the last video frame as the target image, determine the remaining video frames as the first sample image, determine the enhanced text corresponding to the target image as the target text, and determine the enhanced text corresponding to the first sample image as the first sample text.
  • the aforementioned multimodal data processing device 1500 and multimodal data processing method are based on the same inventive concept. They acquire a reference image and reference text describing the content of the reference image, extract reference visual features from the reference image, and add start and end markers to both ends of the reference visual features. This facilitates subsequent differentiation between the reference visual features and the reference text based on the start and end markers. Furthermore, the reference text and the reference visual features with added start and end markers are input into a multimodal large language model. The multimodal large language model outputs a predicted sequence, and then generates... The predicted image and the predicted text used to describe the predicted image are generated.
  • the multimodal large language model can still focus on the information carried by the embeddings of the start tag, the end tag, and the embeddings selected by the sliding window in the target embedding sequence through attention processing. This reduces the perplexity of the multimodal large language model during inference, effectively improves the performance of the multimodal large language model, and thus improves the quality of multimodal content generation.
  • the electronic device provided in this application embodiment for executing the above-described multimodal data processing method can be a terminal.
  • the terminal includes: a camera assembly 1610, a first memory 1620, an input unit 1630, a display unit 1640, a sensor 1650, an audio circuit 1660, a wireless fidelity (WiFi) module 1670, a first processor 1680, and a first power supply 1690, etc.
  • WiFi wireless fidelity
  • the terminal structure shown in Figure 16 does not constitute a limitation on the terminal, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.
  • the camera assembly 1610 can be used to capture images or videos.
  • the camera assembly 1610 includes a front-facing camera and a rear-facing camera.
  • the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal.
  • there are at least two rear-facing cameras which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions.
  • the first memory 1620 can be used to store software programs and modules.
  • the first processor 1680 executes various terminal functions and data processing by running the software programs and modules stored in the first memory 1620.
  • the input unit 1630 can be used to receive input numeric or character information, and to generate key signal inputs related to terminal settings and function control.
  • the input unit 1630 may include a touch panel 1631 and other input devices 1632.
  • the display unit 1640 can be used to display input or provided information, as well as various menus of the terminal.
  • the display unit 1640 may include a display panel 1641.
  • Audio circuitry 1660, speaker 1661, and microphone 1662 provide an audio interface.
  • the first power source 1690 can be AC power, DC power, a disposable battery, or a rechargeable battery.
  • the number of sensors 1650 can be one or more, and these sensors 1650 include, but are not limited to: accelerometers, gyroscopes, pressure sensors, optical sensors, etc.
  • An accelerometer can detect the magnitude of acceleration along the three axes of a coordinate system established by the terminal.
  • an accelerometer can be used to detect the components of gravitational acceleration along the three axes.
  • the first processor 1680 can control the display unit 1640 to display the user interface in either a horizontal or vertical view based on the gravitational acceleration signal acquired by the accelerometer.
  • the accelerometer can also be used for games or for acquiring user motion data.
  • the gyroscope sensor can detect the terminal's orientation and rotation angle. It can work in conjunction with an accelerometer to collect 3D user movements on the terminal. Based on the data collected by the gyroscope sensor, the first processor 1680 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.
  • motion sensing e.g., changing the UI based on the user's tilt
  • image stabilization during shooting e.g., game control, and inertial navigation.
  • the pressure sensor can be disposed on at least one of the side bezel of the terminal and the lower layer of the display unit 1640.
  • the first processor 1680 can perform left/right hand recognition or quick operation based on the grip signal collected by the pressure sensor.
  • the first processor 1680 can control the operable controls on the UI interface based on the user's pressure operation on the display unit 1640.
  • the operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.
  • the first processor 1680 can control the display brightness of the display unit 1640 based on the ambient light intensity collected by the optical sensor. For example, when the ambient light intensity is high, the display brightness of the display unit 1640 is increased; when the ambient light intensity is low, the display brightness of the display unit 1640 is decreased. In another embodiment, the first processor 1680 can also dynamically adjust the shooting parameters of the camera assembly 1610 based on the ambient light intensity collected by the optical sensor.
  • the first processor 1680 included in the terminal can execute the multimodal data processing method of the previous embodiment.
  • the electronic device provided in this application embodiment for executing the above-described multimodal data processing method can also be a server.
  • the server can vary considerably due to different configurations or performance. It may include one or more second processors 1710 and second memories 1730, and one or more storage media 1740 (e.g., one or more mass storage devices) for storing application programs 1743 or data 1742.
  • the second memory 1730 and storage media 1740 can be temporary or persistent storage.
  • the program stored in the storage media 1740 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server.
  • the second processor 1710 may be configured to communicate with the storage media 1740 and execute a series of instruction operations in the storage media 1740 on the server.
  • the server may also include at least one of the following: one or more secondary power supplies 1720, one or more wired or wireless network interfaces 1750, one or more input/output interfaces 1760, and one or more operating systems 1741, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.
  • the second processor 1710 in the server can be used to execute multimodal data processing methods.
  • This application also provides a computer-readable storage medium for storing a computer program for executing the multimodal data processing methods of the foregoing embodiments.
  • This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium.
  • a processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the multimodal data processing method described above.
  • At least one (item) refers to one or more, and “more than one” refers to two or more. "At least one of the two” is used to describe the association relationship of the associated objects, indicating that there can be three relationships. For example, “at least one of A and B” can mean: only A exists, only B exists, and A and B exist simultaneously, where A and B can be singular or plural. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items.
  • At least one of a, b, or c can mean: a, b, c, "a and b", “a and c", “b and c", or "a and b and c", where a, b, and c can be single or multiple.
  • the disclosed systems, apparatuses, and methods can be implemented in other ways.
  • the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods.
  • multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.
  • the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.
  • the units described as separate components may or may not be physically separate.
  • the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
  • the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
  • the integrated unit can be implemented in hardware or as a software functional unit.
  • the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.
  • This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application.
  • the aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Molecular Biology (AREA)
  • Multimedia (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Databases & Information Systems (AREA)
  • Library & Information Science (AREA)
  • Medical Informatics (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Signal Processing (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

本申请公开了一种多模态数据处理方法、装置、电子设备及存储介质，该方法包括：获取参考图像以及参考文本；提取参考图像的参考视觉特征；通过多模态大语言模型确定的嵌入、参考视觉特征的起始标记的嵌入、参考视觉特征的嵌入以及参考视觉特征的结束标记的嵌入；基于多模态大语言模型，将参考文本的嵌入、起始标记的嵌入、参考视觉特征的嵌入以及结束标记的嵌入，拼接为目标嵌入序列，基于起始标记的嵌入、结束标记的嵌入以及目标嵌入序列中被滑动窗口选中的嵌入进行注意力处理，输出预测序列；基于预测序列生成预测图像以及预测文本。

Description

多模态数据处理方法、装置、电子设备、计算机可读存储介质及计算机程序产品
相关申请的交叉引用
本申请基于申请号为2024107494580、申请日为2024年06月11日的中国专利申请提出，并要求该中国专利申请的优先权，该中国专利申请的全部内容在此引入本申请作为参考。
技术领域
本申请涉及计算机技术领域，尤其涉及一种多模态数据处理方法、装置、电子设备、计算机可读存储介质及计算机程序产品。
背景技术
随着图像生成和开放形式文本生成技术的显著进步，图像数据和文本数据的多模态数据处理已经成为一个越来越引人注目的领域。这项任务需要充分理解互相关联的图像数据以及文本数据，由于文本数据不仅具有抽象和叙述性质，而且与复杂的图像数据紧密交织在一起，因此，在推理过程中，当需要进行连续多个轮次的多模态数据处理时，通常会导致模型的性能下降，影响了多模态内容的生成质量。
发明内容
以下是对本申请实施例详细描述的主题的概述。本概述并非是为了限制权利要求的保护范围。
本申请实施例提供了一种多模态数据处理方法、装置、电子设备、计算机可读存储介质及计算机程序产品，能够有效地提升多模态大语言模型的性能，进而提升多模态内容的生成质量。
一方面，本申请实施例提供了一种多模态数据处理方法，应用于电子设备，包括：
获取参考图像以及参考文本，其中，所述参考文本用于描述所述参考图像中的内容；
提取所述参考图像的参考视觉特征；
通过多模态大语言模型确定所述参考文本的嵌入、所述参考视觉特征的起始标记的嵌入、所述参考视觉特征的嵌入以及所述参考视觉特征的结束标记的嵌入；
基于所述多模态大语言模型，将所述参考文本的嵌入、所述起始标记的嵌入、所述参考视觉特征的嵌入以及所述结束标记的嵌入，拼接为目标嵌入序列，基于所述起始标记的嵌入、所述结束标记的嵌入以及所述目标嵌入序列中被滑动窗口选中的嵌入进行注意力处理，输出预测序列；
基于所述预测序列生成预测图像以及预测文本，其中，所述预测文本用于描述所述预测图像中的内容。
另一方面，本申请实施例还提供了一种多模态数据处理装置，包括：
获取模块，配置为获取参考图像以及参考文本，其中，所述参考文本用于描述所述参考图像中的内容；
第一处理模块，配置为提取所述参考图像的参考视觉特征；通过多模态大语言模型确定所述参考文本的嵌入、所述参考视觉特征的起始标记的嵌入、所述参考视觉特征的嵌入以及所述参考视觉特征的结束标记的嵌入；
第二处理模块，配置为基于所述多模态大语言模型，将所述参考文本的嵌入、所述起始标记的嵌入、所述参考视觉特征的嵌入以及所述结束标记的嵌入，拼接为目标嵌入序列，基于所述起始标记的嵌入、所述结束标记的嵌入以及所述目标嵌入序列中被滑动窗口选中的嵌入进行注意力处理，输出预测序列；
生成模块，配置为基于所述预测序列生成预测图像以及预测文本，其中，所述预测文本用于描述所述预测图像中的内容。
另一方面，本申请实施例还提供了一种电子设备，包括存储器和处理器，所述存储器存储有计算机程序，所述处理器执行所述计算机程序时实现上述的多模态数据处理方法。
另一方面，本申请实施例还提供了一种计算机可读存储介质，所述存储介质存储有计算机程序，所述计算机程序被处理器执行实现上述的多模态数据处理方法。
另一方面，本申请实施例还提供了一种计算机程序产品，该计算机程序产品包括计算机程序，该计算机程序存储在计算机可读存介质中。计算机设备的处理器从计算机可读存储介质读取该计算机程序，处理器执行该计算机程序，使得该计算机设备执行实现上述的多模态数据处理方法。
本申请实施例至少包括以下有益效果：通过获取参考图像以及用于描述参考图像中的内容的参考文本，提取参考图像的参考视觉特征，在参考视觉特征的两端分别添加起始标记以及结束标记，能够便于后续基于起始标记以及结束标记区分出参考视觉特征以及参考文本，在此基础上，将参考文本与在两端分别添加了起始标记以及结束标记的参考视觉特征输入至多模态大语言模型，通过将文本和参考视觉特征嵌入到同一语义空间，多模态大语言模型可以更好地理解文本和图像之间的关系，从而生成更符合语义的预测结果，通过联合多个嵌入，多模态大语言模型能够更好地捕捉和利用上下文信息，同时优化文本和图像的生成，提高预测的准确性。基于多模态大语言模型输出预测序列，再基于预测序列生成预测图像以及用于描述预测图像的预测文本，由于在推理的过程中，基于起始标记的嵌入、结束标记的嵌入以及目标嵌入序列中被滑动窗口选中的嵌入进行注意力处理，通过限制注意力计算的范围，滑动窗口机制显著降低了计算复杂度，使得模型能够在长序列生成中保持高效的推理速度，因此，即便是进行连续多个轮次的多模态数据处理，多模态大语言模型也能够通过注意力处理关注起始标记的嵌入、结束标记的嵌入以及目标嵌入序列中被滑动窗口选中的嵌入所携带的信息，从而降低多模态大语言模型推理时的困惑度，有效地提升多模态大语言模型的性能，进而提升多模态内容的生成质量。
本申请实施例的其他特征和优点将在随后的说明书中阐述，并且，部分地从说明书中变得显而易见，或者通过实施本申请实施例而了解。
附图说明
附图用来提供对本申请实施例技术方案的进一步理解，并且构成说明书的一部分，与本申请实施例一起用于解释本申请实施例的技术方案，并不构成对本申请实施例技术方案的限制。
图1为本申请实施例提供的一种可选的实施环境的示意图；
图2为本申请实施例提供的多模态数据处理方法的一种可选的流程示意图；
图3为本申请实施例提供的注意力处理的一种可选的可视化示意图；
图4为本申请实施例提供的得到预测视觉特征的一种可选的流程示意图；
图5为本申请实施例提供的得到参考视觉特征的嵌入的一种可选的流程示意图；
图6为本申请实施例提供的训练多模态大语言模型的一种可选的流程示意图；
图7为本申请实施例提供的联合训练特征提取器与所述扩散模型的一种可选的流程示意图；
图8为本申请实施例提供的再次训练扩散模型的一种可选的第一流程示意图；
图9为本申请实施例提供的再次训练扩散模型的一种可选的第二流程示意图；
图10为本申请实施例提供的注意力分数分布的一种可选的分布示意图；
图11为本申请实施例提供的FID分数的一种可选的柱状示意图；
图12为本申请实施例提供的风格一致性比对结果的一种可选的饼状示意图；
图13为本申请实施例提供的叙述吸引力比对结果的一种可选的饼状示意图；
图14为本申请实施例提供的图文一致性比对结果的一种可选的饼状示意图；
图15为本申请实施例提供的多模态数据处理装置的一种可选的结构示意图；
图16为本申请实施例提供的终端的部分结构框图；
图17为本申请实施例提供的服务器的部分结构框图。
具体实施方式
为了使本申请实施例的目的、技术方案及优点更加清楚明白，以下结合附图及实施例，对本申请实施例进行进一步详细说明。应当理解，此处所描述的具体实施例仅用以解释本申请实施例，并不用于限定本申请实施例。
需要说明的是，在本申请实施例中，当涉及到需要根据目标对象属性信息或属性信息集合等与目标对象特性相关的数据进行相关处理时，都会先获得目标对象的许可或者同意，而且，对这些数据的收集、使用和处理等，都会遵守相关法律法规和标准。其中，目标对象可以是用户。此外，当本申请实施例需要获取目标对象属性信息时，会通过弹窗或者跳转到确认页面等方式获得目标对象的单独许可或者单独同意，在明确获得目标对象的单独许可或者单独同意之后，再获取用于使本申请实施例能够正常运行的必要的目标对象相关数据。
本申请实施例中，术语“模块”或“单元”是指有预定功能的计算机程序或计算机程序的一部分，并与其他相关部分一起工作以实现预定目标，并且可以通过使用软件、硬件(如处理电路或存储器)或其组合来全部或部分实现。同样的，一个处理器(或多个处理器或存储器)可以用来实现一个或多个模块或单元。此外，每个模块或单元都可以是包含该模块或单元功能的整体模块或单元的一部分。
为便于理解本申请实施例提供的技术方案，这里先对本申请实施例使用的一些关键名词进行解释：
1)云技术(Cloud technology)，是指在广域网或局域网内将硬件、软件、网络等系列资源统一起来，实现数据的计算、储存、处理和共享的一种托管技术。云技术基于云计算商业模式应用的网络技术、信息技术、整合技术、管理平台技术、应用技术等的总称，可以组成资源池，按需所用，灵活便利。云计算技术将变成重要支撑。技术网络系统的后台服务需要大量的计算、存储资源，如视频网站、图片类网站和更多的门户网站。伴随着互联网行业的高度发展和应用，将来每个物品都有可能存在自己的识别标志，都需要传输到后台系统进行逻辑处理，不同程度级别的数据将会分开处理，各类行业数据皆需要强大的系统后盾支撑，只能通过云计算来实现。
2)人工智能(Artificial Intelligence,AI)是利用数字计算机或者数字计算机控制的机器模拟、延伸和扩展人的智能，感知环境、获取知识并使用知识获得最佳结果的理论、方法、技术及应用系统。换句话说，人工智能是计算机科学的一个综合技术，它企图了解智能的实质，并生产出一种新的能以人类智能相似的方式做出反应的智能机器。人工智能也就是研究各种智能机器的设计原理与实现方法，使机器具有感知、推理与决策的功能。人工智能技术是一门综合学科，涉及领域广泛，既有硬件层面的技术也有软件层面的技术。人工智能基础技术一般包括如传感器、专用人工智能芯片、云计算、分布式存储、大数据处理技术、预训练模型技术、操作/交互系统、机电一体化等。其中，预训练模型又称大模型、基础模型，经过微调后可以广泛应用于人工智能各大方向下游任务。人工智能软件技术主要包括计算机视觉技术、语音处理技术、自然语言处理技术以及机器学习/深度学习等几大方向。
3)机器学习(Machine Learning，简称ML)是一门多领域交叉学科，涉及概率论、统计学、逼近论、凸分析、算法复杂度理论等多门学科。专门研究计算机怎样模拟或实现人类的学习行为，以获取新的知识或技能，重新组织已有的知识结构使之不断改善自身的性能。机器学习是人工智能的核心，是使计算机具有智能的根本途径，其应用遍及人工智能的各个领域。机器学习和深度学习通常包括人工神经网络、置信网络、强化学习、迁移学习、归纳学习、示教学习等技术。
4)参考图像，是指在计算机视觉和多媒体应用中，作为比较、分析或生成目标的视觉数据。它是一个二维的数字矩阵，其中，每个元素(像素)代表颜色或强度值。参考图像可以是各种类型的图像，包括但不限于RGB彩色图像、灰度图像、二值图像、深度图像、热成像图像等。参考图像的来源广泛，可以是自然场景的照片、艺术作品、医学影像、卫星图像、动画或电影截图等。
5)参考文本，是用于描述或注释视觉内容的文本数据，例如用于描述参考图像。它可以是关于图像内容的描述、标签、标题、说明、故事、对话或其他任何形式的文本，旨在提供对视觉数据的语义理解、上下文信息或情感表达。参考文本可以是人工编写的，也可以是通过自然语言处理技术生成的。
6)视觉特征，是通过计算机视觉算法从图像中提取的抽象信息，用于描述图像的内容、结构或属性。这些特征可以是低级的(如颜色、纹理、边缘、形状)或高级的(如物体类别、场景类型、动作、表情等)。视觉特征通常表示为数值向量或矩阵，可以用于图像识别、分类、检索、生成等任务。
7)起始标记，是在序列数据处理中使用的特殊符号或标记，用于指示一个序列的开始。在自然语言处理和序列生成任务中，起始标记常常用于提示模型开始生成或处理新的序列。起始标记通常为预设的符号或者字符，例如，起始标记预设为<BoI>。
8)结束标记，是在序列数据处理中使用的特殊符号或标记，用于指示一个序列的结束。结束标记用于告诉模型一个序列的生成或处理已经完成。在生成任务中，结束标记可以用于停止进一步的生成过程，确保生成的序列具有适当的长度和完整性。结束标记通常为预设的符号或者字符，例如，结束标记预设为<EoI>。
9)预测词元，是文本分析和生成过程中使用的基本单位，它代表了语言模型在预测下一个词元时所考虑的最小语言单位。预测词元可以是完整的词语、单个字符、子词(subword)或其他粒度的语言单位，具体取决于所使用的分词策略和模型的架构。
随着图像生成和开放形式文本生成技术的显著进步，图像数据和文本数据的多模态数据处理已经成为一个越来越引人注目的领域。这项任务需要充分理解互相关联的图像数据以及文本数据，由于文本数据不仅具有抽象和叙述性质，而且与复杂的图像数据紧密交织在一起，因此，在推理过程中，当需要进行连续多个轮次的多模态数据处理时，通常会导致模型的性能下降，影响了多模态内容的生成质量。
基于此，本申请实施例提供了一种多模态数据处理方法、装置、电子设备及存储介质，能够有效地提升多模态大语言模型的性能，进而提升多模态内容的生成质量。
参照图1，图1为本申请实施例提供的一种可选的实施环境的示意图，该实施环境包括终端101和服务器102，其中，终端101和服务器102之间通过通信网络连接。
示例性地，服务器102可以获取终端101发送的参考图像以及参考文本，其中，参考文本用于描述参考图像中的内容；提取参考图像的参考视觉特征，将参考文本与在两端分别添加了起始标记以及结束标记的参考视觉特征输入至多模态大语言模型；基于多模态大语言模型确定由参考文本的嵌入、起始标记的嵌入、参考视觉特征的嵌入以及结束标记的嵌入拼接得到的目标嵌入序列，基于起始标记的嵌入、结束标记的嵌入以及目标嵌入序列中被滑动窗口选中的嵌入进行注意力处理，输出预测序列；基于预测序列生成预测图像以及预测文本，其中，预测文本用于描述预测图像中的内容；服务器102将预测图像以及预测文本发送至终端101。
服务器102通过获取参考图像以及用于描述参考图像中的内容的参考文本，提取参考图像的参考视觉特征，在参考视觉特征的两端分别添加起始标记以及结束标记，能够便于后续基于起始标记以及结束标记区分出参考视觉特征以及参考文本，在此基础上，将参考文本与在两端分别添加了起始标记以及结束标记的参考视觉特征输入至多模态大语言模型，基于多模态大语言模型输出预测序列，再基于预测序列生成预测图像以及用于描述预测图像的预测文本，由于在推理的过程中，基于起始标记的嵌入、结束标记的嵌入以及目标嵌入序列中被滑动窗口选中的嵌入进行注意力处理，因此，即便是进行连续多个轮次的多模态数据处理，多模态大语言模型也能够通过注意力处理关注起始标记的嵌入、结束标记的嵌入以及目标嵌入序列中被滑动窗口选中的嵌入所携带的信息，从而降低多模态大语言模型推理时的困惑度，有效地提升多模态大语言模型的性能，进而提升多模态内容的生成质量。
以艺术创作辅助场景为例，终端用于采集艺术创作的参考图像和参考文本，服务器基于参考图像和参考文本生成符合艺术创作要求的预测图像以及预测文本，作为艺术创作辅助工具。
以教育培训场景为例，终端用于采集参考图像和参考文本，服务器基于参考图像和参考文本生成用于制作教学材料和培训内容的预测图像以及预测文本。
以社交媒体场景为例，终端用于采集参考图像和参考文本，服务器基于参考图像和参考文本生成用于提供个性化的内容推荐的预测图像以及预测文本。
以广告场景为例，终端用于采集参考图像和参考文本，服务器基于参考图像和参考文本生成符合广告主设定的业务需求的预测图像以及预测文本。
服务器102可以是独立的物理服务器，也可以是多个物理服务器构成的服务器集群或者分布式系统，还可以是提供云服务、云数据库、云计算、云函数、云存储、网络服务、云通信、中间件服务、域名服务、安全服务、CDN(Content Delivery Network，内容分发网络)、以及大数据和人工智能平台等基础云计算服务的云服务器。另外，服务器102还可以是区块链网络中的一个节点服务器。
终端101可以是手机、电脑、智能语音交互设备、智能家电、车载终端等，但并不局限于此。终端101以及服务器102可以通过有线或无线通信方式进行直接或间接地连接，本申请实施例在此不做限制。
本申请实施例提供的方法可应用于各种场景，包括但不限于云技术、人工智能、智慧交通、辅助驾驶等场景。
参照图2，图2为本申请实施例提供的多模态数据处理方法的一种可选的流程示意图，该多模态数据处理方法可以由服务器执行，或者也可以由终端执行，或者也可以由服务器配合终端执行，该多模态数据处理方法包括但不限于以下步骤201至步骤204。
步骤201：获取参考图像以及参考文本。
其中，参考图像为视觉模态数据，参考图像是指由像素组成的二维数字矩阵。参考图像可以是RGB图像，即每个像素点分别对应三个颜色通道，参考图像还可以是灰度图像、二值图像、深度图像等等，本申请实施例在此不作限定。参考图像的来源可以具备多样性，例如，参考图像可以是动画片段、电影截图、摄影作品等来源的图像。
其中，参考文本为文本模态数据，参考文本用于描述参考图像中的内容，即参考文本为参考图像的描述文本，例如，参考文本可以是描述参考图像的视觉要素、场景细节、情感氛围或情景故事等内容的文本，示例性地，当参考图像为动画片段时，参考文本可以为动画片段相关的字幕文本、剧本文本、资料文本等；可见，参考文本为参考图像提供更详细的信息，通过使用参考文本能够有效深化和扩展对参考图像的理解。
步骤202：提取参考图像的参考视觉特征，将参考文本与在两端分别添加了起始标记以及结束标记的参考视觉特征输入至多模态大语言模型。
其中，参考视觉特征是指从参考图像提取的更抽象且更具信息量的特征，参考视觉特征可包括一个视觉特征向量或者多个相同维度的视觉特征向量，例如，参考视觉特征包括256个768维的视觉特征向量；参考视觉特征可通过特征提取器对参考图像进行特征提取得到，例如，特征提取器包括卷积神经网络(Convolutional Neural Networks，CNN)、生成对抗网络、对比语言图像预训练模型(Contrastive Language–Image Pre-training，CLIP)等等。参考视觉特征还可以通过以下任意一种处理实现：
人工提取特征：使用人工设计的特征描述符，如尺度不变特征变换(Scale-Invariant Feature Transform，SIFT)、加速稳健特征(Speeded Up Robust Features，SURF)、方向梯度直方图(Histogram of Oriented Gradient，HOG)，这些方法通过计算图像的局部特征来描述图像内容。
深度学习特征：除了卷积神经网络，还可以使用其他深度学习模型，如循环神经网络(Recurrent Neural Network，RNN)、长短期记忆网络(Long Short-Term Memory，LSTM)、自注意力机制等，这些模型可以捕捉图像的全局和局部特征。
预训练模型：使用预训练的模型，如ImageNet上的预训练模型，这些模型已经学习了丰富的图像特征，可以直接用于特征提取。
迁移学习：在预训练模型的基础上，通过迁移学习来提取特定任务的视觉特征，这种方法可以利用已有的知识来提高特征提取的准确性。
无监督学习：使用无监督学习方法，如自编码器、变分自编码器(Variational AutoEncoder，VAE)、生成对抗网络(Generative Adversarial Network，GAN)等，这些方法可以在没有标签的情况下学习图像的特征。
多任务学习：通过多任务学习，同时学习多个相关任务，如图像分类、目标检测、语义分割等，从而提取更加全面和鲁棒的视觉特征。
示例的，以自注意力机制为例，“提取参考图像的参考视觉特征”可以通过执行以下处理实现：通过卷积层将参考图像转换为特征图，特征图包括参考图像的低级特征，如边缘、纹理等；将特征图展平为特征向量序列，特征向量序列中的每个特征向量对应于参考图像中的一个位置或区域；对于特征向量序列中的每个特征向量，对特征向量进行自注意力计算，确定特征向量的权重，权重表征当前特征向量与其他特征向量的相关程度；基于权重对多个特征向量进行加权求和，得到参考视觉特征。
本申请实施例通过自注意力机制提取参考图像的参考视觉特征，能够使在处理每个位置的特征向量时，考虑整个参考图像的信息，从而捕捉到全局的上下文关系。通过权重分配，对参考图像中的局部细节给予更多的关注，提高了特征提取的准确性。
其中，起始标记通常为预设的符号或者字符，用于指示序列的开始，可以位于序列的第一个位置，例如，起始标记预设为<BoI>，类似地，结束标记也为预设的符号或者字符，用于指示序列的结束，可以位于序列的最后一个位置，例如，结束标记预设为<EoI>，可见，起始标记和结束标记均可视为文本模态数据；参考视觉特征的两端是指参考视觉特征的首端和末端，在参考视觉特征的两端分别添加起始标记以及结束标记，例如在参考视觉特征的首端添加起始标记，以及在参考视觉特征的末端添加结束标记，起始标记、参考视觉特征和结束标记能够以组合的形式存在。
其中，多模态指的是来自不同感官或来源的信息，例如视觉、听觉、触觉等，多模态大语言模型是大语言模型的扩展形式，相较于仅能处理对应于文本模态的输入数据的大语言模型(Large Language Model，LLM)，多模态大语言模型不仅能够处理对应于文本模态的输入数据，还能够处理对应于除了文本模态之外的其他模态的输入数据，例如，其他模态包括视觉模态、音频模态等等。
基于此，对于属于视觉模态数据的参考图像，需要先将参考图像映射为固定维度的参考视觉特征，然后再将在两端分别添加了起始标记以及结束标记的参考视觉特征输入至多模态大语言模型，多模态大语言模型能够基于起始标记以及结束标记区分出参考视觉特征以及参考文本，参考文本、起始标记以及结束标记均为对应于文本模态的输入数据，多模态大语言模型能够有效处理参考文本、起始标记以及结束标记，而参考视觉特征为对应于视觉模态的输入数据，多模态大语言模型也能够有效处理参考视觉特征。
示例的，将参考文本与在两端分别添加了起始标记以及结束标记的参考视觉特征输入至多模态大语言模型时，可采用多种方式输入数据，例如，将参考文本与在两端分别添加了起始标记以及结束标记的参考视觉特征作为一批数据同时输入至多模态大语言模型；又例如，将参考文本与在两端分别添加了起始标记以及结束标记的参考视觉特征依次输入至多模态大语言模型；在采用依次输入数据的方式时，可先将参考文本输入至多模态大语言模型，也可先将添加了起始标记以及结束标记的参考视觉特征输入至多模态大语言模型，本申请实施例在此不作限定。
步骤203：基于多模态大语言模型确定由参考文本的嵌入、起始标记的嵌入、参考视觉特征的嵌入以及结束标记的嵌入拼接得到的目标嵌入序列，基于起始标记的嵌入、结束标记的嵌入以及目标嵌入序列中被滑动窗口选中的嵌入进行注意力处理，得到预测序列。
其中，多模态大语言模型可通过不同嵌入模块分别确定对应的模态数据的嵌入，例如，多模态大语言模型包括词嵌入模块和图嵌入模块，词嵌入模块用于处理对应于文本模态的输入数据，通过词嵌入模块分别对参考文本、起始标记以及结束标记进行嵌入处理，能够确定参考文本的嵌入、起始标记的嵌入以及结束标记的嵌入，参考文本的嵌入包括参考文本中各个词语的嵌入。
示例性地，参考文本的各个词语、起始标记以及结束标记均作为词元记录在词表中，在词表中每个词元都会被分配一个唯一的索引，通过词表能够确定参考文本的各个词语、起始标记以及结束标记各自对应的索引，词嵌入模块维护有存储了各个词元的词嵌入的词嵌入矩阵，词嵌入模块能够基于索引在词嵌入矩阵中查询到对应的词嵌入，即得到参考文本的嵌入、起始标记的嵌入以及结束标记的嵌入。
然后，图嵌入模块用于处理对应于视觉模态的输入数据，通过图嵌入模块对参考视觉特征进行嵌入处理，能够确定参考视觉特征的嵌入；当参考视觉特征包括一个视觉特征向量时，参考视觉特征的嵌入为视觉特征向量的嵌入，当参考视觉特征包括多个视觉特征向量时，参考视觉特征的嵌入包括各个视觉特征向量的嵌入，视觉特征向量的嵌入与上述的词嵌入的维度是相同的。
在一些实施例中，可以直接按照预设顺序，将参考文本的嵌入、起始标记的嵌入、参考视觉特征的嵌入以及结束标记的嵌入相加为目标嵌入序列，预设顺序可以为：参考文本的嵌入、起始标记的嵌入、参考视觉特征的嵌入、结束标记的嵌入。还可以通过多模态大语言模型为参考文本的嵌入、起始标记的嵌入、参考视觉特征的嵌入以及结束标记的嵌入分配各自的权重，基于权重对参考文本的嵌入、起始标记的嵌入、参考视觉特征的嵌入以及结束标记的嵌入进行加权求和，得到目标嵌入序列。
示例的，以直接拼接为例，将参考文本的嵌入a、起始标记的嵌入b、参考视觉特征的嵌入c以及结束标记的嵌入d相加为目标嵌入序列a+b+c+d。以分配权重进行加权求和为例，参考文本的嵌入a的权重为w1，起始标记的嵌入b的权重为w2，参考视觉特征的嵌入c的权重为w3，结束标记的嵌入d的权重为w4，则目标嵌入序列可以表示为a·w1+b·w2+c·w3+d·w4。
本申请实施例通过权重分配，可以根据任务的需要，灵活地调整不同模态信息的比重，使得目标嵌入序列更加符合任务的要求。加权求和的过程使得不同模态的信息得到了有效的融合，增强了模型对多模态数据的理解能力。权重分配机制有助于保持文本和视觉信息的语义一致性，使得目标嵌入序列能够更好地反映输入数据的语义内容。
其中，目标嵌入序列为多模态大语言模型的输入序列，预测序列为多模态大语言模型的输出序列，输出序列由多模态大语言模型的所有输出数据组成；滑动窗口用于确定多模态大语言模型的注意力范围，注意力范围是指多模态大语言模型在推理输出数据时，在输入序列中所关注的特定范围；利用多模态大语言模型进行连续多个轮次的多模态数据处理时，每当推理出一个输出数据时，在目标嵌入序列的末端添加该输出数据对应的嵌入，再基于新的目标嵌入序列推理出下一个输出数据，滑动窗口会随着目标嵌入序列的变长而滑动，目标嵌入序列的最后一个嵌入始终为被滑动窗口选中的嵌入，滑动窗口的大小通常是固定的，例如，滑动窗口的大小设为5，假设目标嵌入序列的长度为30，首次推理时，被滑动窗口选中的嵌入包括目标嵌入序列中的最后5个嵌入，第二次推理时，目标嵌入序列的长度更新为31，此时，被滑动窗口选中的嵌入包括新的目标嵌入序列中的最后5个嵌入。
基于此，多模态大语言模型的注意力范围除了包括被滑动窗口选中的嵌入之外，注意力范围还包括起始标记的嵌入以及结束标记的嵌入，多模态大语言模型能够通过注意力处理关注起始标记的嵌入、结束标记的嵌入以及目标嵌入序列中被滑动窗口选中的嵌入所携带的信息，在处理目标嵌入序列时，相较于只关注被滑动窗口选中的嵌入，通过保留起始标记的嵌入以及结束标记的嵌入，能够扩展模型的记忆，使得模型保持对起始标记的嵌入以及结束标记的嵌入所携带的关键信息的关注，避免因滑动窗口的滑动而导致信息丢失，有助于多模态大语言模型在处理长序列时保持注意力分布均匀和稳定，从而降低多模态大语言模型推理时的困惑度，有效地提升多模态大语言模型的性能，例如，利用多模态大语言模型进行故事生成时，能够提高生成内容的连贯性，以及保持风格和角色的一致性。
值得注意的是，起始标记的嵌入以及结束标记的嵌入也可被滑动窗口选中，此处，注意力范围包括起始标记的嵌入以及结束标记的嵌入，例如，注意力范围包括未被滑动窗口选中的起始标记的嵌入以及结束标记的嵌入，未被滑动窗口选中的起始标记的嵌入或者结束标记的嵌入的数量均可为零，本申请实施例在此不作限定。
其中，困惑度是衡量模型对给定输入序列时推理输出数据的不确定性的度量，当困惑度越低时，模型的预测效果越好，反之，当困惑度越高时，模型的预测效果越差。通常情况下，可将困惑度作为评估多模态大语言模型性能的重要指标。
示例的，困惑度可以通过执行以下处理确定：对于序列中的每个词，计算模型在给定前面所有词的情况下，预测这个词的预测概率，确定预测概率的对数；确定所有词的对数的加和，确定加和与序列的长度的比值，得到平均对数概率；确定平均对数概率的相反数，针对相反数以2为底数取幂，得到困惑度。例如，针对序列中的每个词ni，预测概率为pi，预测概率的对数为log2pi，若一共有N个词，则所有词的对数的加和为平均对数概率为平均对数概率的相反数为困惑度为
另外，由于滑动窗口的大小固定，且起始标记的嵌入以及结束标记的嵌入的总数量相对较少，因此，多模态大语言模型的注意力范围相对较窄，相较于对目标嵌入序列进行全局注意力处理，能够提高有效降低内存使用，并显著提高推理效率。
步骤204：基于预测序列生成预测图像以及预测文本。
其中，预测文本用于描述预测图像中的内容，预测文本通常为不包括特殊标记的文本。预测图像为视觉模态数据，预测文本为文本模态数据；预测图像是与参考图像关联的图像，例如，利用多模态大语言模型进行故事生成时，生成的预测图像是参考图像在下一个情景的图像，又例如，利用多模态大语言模型进行步骤生成时，生成的预测图像是参考图像在下一个步骤的图像，又例如，利用多模态大语言模型进行内容增强时，生成的预测图像是与参考图像相近的图像。
示例的，特殊标记包括但不限于以下至少一种：
起始标记(Start Token)：通常用于标记序列的开始，例如在生成文本或图像描述时，起始标记可以帮助模型知道生成过程的起点。
结束标记(End Token)：用于标记序列的结束，确保模型知道何时停止生成。
分隔标记(Separator Token)：用于分隔不同部分的输入数据，例如在多模态输入中，分隔标记可以用来分隔文本和视觉特征。
掩码标记(Mask Token)：在一些预训练任务中，如BERT中的掩码语言模型任务，掩码标记用于掩盖输入序列中的某些词，模型需要预测被掩盖的词。
类标记(Class Token)：在某些模型中，类标记用于表示整个序列的分类信息，例如在图像分类任务中，类标记可以用来表示图像的类别。
位置标记(Position Token)：用于表示序列中每个元素的位置信息，这对于需要考虑顺序信息的模型非常重要。
基于此，由于参考文本和参考图像是交替的，所以多模态大语言模型能够学习到交替的文本模态和视觉模态的内容格式，输出用于生成交替的文本模态数据和视觉模态数据的预测序列，进而基于预测序列生成交替的预测图像和预测文本，预测图像和预测文本属于多模态内容；多模态大语言模型能够通过注意力处理关注起始标记的嵌入、结束标记的嵌入以及目标嵌入序列中被滑动窗口选中的嵌入所携带的信息，从而降低多模态大语言模型推理时的困惑度，有效地提升多模态大语言模型的性能，进而提升多模态内容的生成质量，能够得到丰富的文本内容以及多样的视觉内容，使得预测文本与预测图像紧密交织在一起。
示例的，预测序列由多模态大语言模型的所有输出数据组成，每当多模态大语言模型推理出一个新的输出数据时，预测序列的长度就会增加，可设置多模态大语言模型的停止条件，多模态大语言模型能够持续推理出新的输出数据，直至达到停止条件，例如，停止条件为预测序列的长度达到预设的长度阈值，由于预测序列的长度可调整，所以，交替的预测图像和预测文本可以是单轮或者多轮的，本申请实施例在此不作限定。
在一些实施例中，预测序列包括预测词元和预测视觉特征，基于起始标记的嵌入、结束标记的嵌入以及目标嵌入序列中被滑动窗口选中的嵌入进行注意力处理，输出预测序列，例如，可以是基于起始标记的嵌入、结束标记的嵌入以及目标嵌入序列中被滑动窗口选中的嵌入进行注意力处理，生成预测嵌入；对属于文本类型的预测嵌入进行分类，输出预测词元；对属于视觉类型的预测嵌入进行映射，输出预测视觉特征。
其中，预测词元是文本分析时的基本单位，预测词元可以是词语、字符、子词等；预测视觉特征处于视觉特征空间，预测视觉特征用于指示图像的更抽象且更具信息量的特征。
基于此，多模态大语言模型在连续多个轮次的推理过程中，每个轮次生成一个预测嵌入，通过注意力处理使得多模态大语言模型在生成预测嵌入时，能够考虑到起始标记的嵌入、结束标记的嵌入以及目标嵌入序列中被滑动窗口选中的嵌入所携带的信息，从而提高生成内容的相关性和连贯性；另外，多模态大语言模型能够生成包括属于文本类型的预测嵌入和属于视觉类型的预测嵌入，属于文本类型的预测嵌入是指通过该预测嵌入能够确定对应于文本模态的输出数据，属于视觉类型的预测嵌入是指通过该预测嵌入能够确定对应于视觉模态的输出数据，在生成属于文本类型的预测嵌入时，对属于文本类型预测嵌入进行分类，能够输出对应于文本模态的预测词元，在生成属于视觉类型的预测嵌入时，对属于视觉类型预测嵌入进行映射，能够输出对应于视觉模态的预测视觉特征。
示例的，多模态大语言模型内部设置有Transformer解码层，将起始标记的嵌入、结束标记的嵌入以及目标嵌入序列中被滑动窗口选中的嵌入输入至Transformer解码层后，Transformer解码层能够对输入数据进行自注意力处理，生成预测嵌入，多模态大语言模型进行多模态数据处理时，能够交替生成一组属于文本类型的预测嵌入以及一组属于视觉类型的预测嵌入，进而交替输出一组对应于文本模态的预测词元以及一组对应于视觉模态的预测视觉特征。
在一些实施例中，参考图3，图3为本申请实施例提供的注意力处理的一种可选的可视化示意图；参考文本的首端添加有首端标记，基于多模态大语言模型确定由参考文本的嵌入、起始标记的嵌入、结束标记的嵌入以及参考视觉特征的嵌入拼接得到的目标嵌入序列，基于起始标记的嵌入、结束标记的嵌入以及目标嵌入序列中被滑动窗口选中的嵌入进行注意力处理，输出预测序列，例如，可以是基于多模态大语言模型确定由首端标记的嵌入、参考文本的嵌入、起始标记的嵌入、结束标记的嵌入以及参考视觉特征的嵌入拼接得到的目标嵌入序列，基于首端标记的嵌入、起始标记的嵌入、结束标记的嵌入以及目标嵌入序列中被滑动窗口选中的嵌入进行注意力处理，生成预测嵌入；对属于文本类型的预测嵌入进行分类，输出预测词元；对属于视觉类型的预测嵌入进行映射，输出预测视觉特征。
其中，首端标记为预设的符号或者字符，例如，首端标记预设为<BoS>,参考文本的首端可以添加一个或者多个首端标记，本申请实施例在此不作限定。
基于此，通过注意力处理使得多模态大语言模型在生成预测嵌入时，能够考虑到首端标记的嵌入、起始标记的嵌入、结束标记的嵌入以及目标嵌入序列中被滑动窗口选中的嵌入所携带的信息，能够进一步提高生成内容的相关性和连贯性。
在一些实施例中，基于起始标记的嵌入、结束标记的嵌入以及目标嵌入序列中被滑动窗口选中的嵌入进行自注意力处理，生成预测嵌入，例如，可以是将起始标记的嵌入、结束标记的嵌入以及目标嵌入序列中被滑动窗口选中的嵌入拼接为待处理嵌入序列；基于待处理嵌入序列构建第一键矩阵和第一价值矩阵；基于待处理嵌入序列中最后一个位置的嵌入构建第一查询矩阵；基于第一查询矩阵、第一键矩阵和第一价值矩阵进行注意力处理，生成预测嵌入。
其中，待处理嵌入序列是由起始标记的嵌入、结束标记的嵌入以及目标嵌入序列中被滑动窗口选中的嵌入按照在目标嵌入序列中的顺序拼接而成，第一键矩阵为待处理嵌入序列与可学习的第一键参数矩阵的乘积，第一价值矩阵为待处理嵌入序列与可学习的第一价值参数矩阵的乘积，第一键矩阵和第一价值矩阵均包括待处理嵌入序列中的各个嵌入的隐状态，第一查询矩阵为待处理嵌入序列中最后一个位置的嵌入与可学习的第一查询参数矩阵的乘积，第一查询矩阵包括待处理嵌入序列中最后一个位置的嵌入的隐状态，其中，隐状态可以为经过自注意力层和前馈神经网络层处理后的向量表示，该向量表示包括模型对输入序列中各个位置的隐含理解。
示例性地，假设待处理嵌入序列包括50个嵌入，待处理嵌入序列中的各个嵌入的维度均为1*768，第一查询参数矩阵、第一键参数矩阵和第一价值参数矩阵的维度均为768*768，待处理嵌入序列与第一键参数矩阵相乘得到50*768维的第一键矩阵，待处理嵌入序列与第一价值参数矩阵相乘得到50*768维的第一价值矩阵，待处理嵌入序列中最后一个位置的嵌入与第一查询参数矩阵相乘得到1*768维的第一查询矩阵。
基于此，由于第一查询矩阵指示当前需要关注的信息，第一查询矩阵具体用于查找与待处理嵌入序列中最后一个位置的嵌入最相关的信息，第一键矩阵和第一价值矩阵均包含待处理嵌入序列中的各个嵌入的信息，第一键矩阵用于供多模态大语言模型确定待处理嵌入序列中的各个嵌入对待处理嵌入序列中最后一个位置的嵌入的重要程度，第一价值矩阵用于按照重要程度生成预测嵌入，多模态大语言模型对第一查询矩阵、第一键矩阵和第一价值矩阵进行注意力处理生成预测嵌入，由于第一查询矩阵的来源与第一键矩阵以及第一价值矩阵的来源是相同的，实际上是对第一查询矩阵、第一键矩阵和第一价值矩阵进行自注意力处理生成预测嵌入，多模态大语言模型在生成预测嵌入时，能够更关注待处理嵌入序列中的重要信息而忽略次要信息，高效地利用和整合输入信息，增强模型理解和生成复杂输出的能力。
需要说明的是，在确定待处理嵌入序列后，将待处理嵌入序列加载至内存并进行注意力处理，在进行注意力处理之前，会基于待处理嵌入序列中的各个嵌入在内存中的相对位置对待处理嵌入序列进行位置编码，得到待处理嵌入序列中的各个嵌入对应的位置编码向量，然后基于位置编码向量对待处理嵌入序列中的各个嵌入进行调整，能够为多模态大语言模型提供待处理嵌入序列中的各个嵌入的相对位置信息，使得多模态大语言模型能够学习更丰富的内容语义表示；另外，由于滑动窗口的大小固定，且起始标记的嵌入以及结束标记的嵌入的总数量相对较少，因此，多模态大语言模型的注意力范围相对较窄，相较于将整个目标嵌入序列加载至内存中进行全局注意力处理，能够提高有效降低内存使用，并显著提高推理效率。
在一些实施例中，基于起始标记的嵌入、结束标记的嵌入以及目标嵌入序列中被滑动窗口选中的嵌入进行自注意力处理，生成预测嵌入，例如，可以是将起始标记的嵌入、结束标记的嵌入以及目标嵌入序列中被滑动窗口选中的嵌入拼接为待处理嵌入序列；基于待处理嵌入序列构建第一键矩阵和第一价值矩阵；基于待处理嵌入序列中最后一个位置的嵌入构建第一查询矩阵；基于第一查询矩阵、第一键矩阵和第一价值矩阵进行注意力处理，生成预测嵌入。
示例的，预测嵌入可通过多种方式确定，下面对第一种确定预测嵌入的方式进行详细描述。
在一些实施例中，基于第一查询矩阵、第一键矩阵和第一价值矩阵进行注意力处理，生成预测嵌入，例如，可以是基于第一查询矩阵和第一键矩阵的转置的乘积得到注意力分数矩阵，基于缩放因子对注意力分数矩阵进行调整，对调整后的注意力分数矩阵进行归一化处理，得到注意力权重矩阵，其中，注意力权重矩阵包括待处理嵌入序列中的各个嵌入各自对应的注意力权重；基于调整后的注意力权重矩阵与第一价值矩阵的乘积生成预测嵌入。
其中，当待处理嵌入序列中的嵌入对应的注意力权重越大时，代表该嵌入对待处理嵌入序列中最后一个位置的嵌入的重要程度越高，反之，当待处理嵌入序列中的嵌入对应的注意力权重越小时，代表该嵌入对待处理嵌入序列中最后一个位置的嵌入的重要程度越低。
基于此，多模态大语言模型在生成预测嵌入时，能够通过注意力权重矩阵更关注待处理嵌入序列中的重要信息而忽略次要信息，高效地利用和整合输入信息，增强模型理解和生成复杂输出的能力。
示例的，预测嵌入的计算公式如公式(1)：
其中，Ol为预测嵌入，Ql为第一查询矩阵，Kl为第一键矩阵，为第一键矩阵的转置，为缩放因子，dk为待处理嵌入序列中的各个嵌入的维度，为调整后的注意力分数矩阵，softmax是归一化函数，是指对调整后的注意力分数矩阵进行归一化处理，Vl为第一价值矩阵。
下面对第二种确定预测嵌入的方式进行详细描述。
在一些实施例中，基于第一查询矩阵、第一键矩阵和第一价值矩阵进行注意力处理，生成预测嵌入，例如，可以是基于第一查询矩阵和第一键矩阵确定注意力权重矩阵，其中，注意力权重矩阵包括待处理嵌入序列中的各个嵌入各自对应的注意力权重；将起始标记的嵌入或者结束标记的嵌入各自对应的注意力权重，调整至大于或者等于预设的权重阈值；基于调整后的注意力权重矩阵与第一价值矩阵的乘积生成预测嵌入。
其中，多模态大语言模型在生成预测嵌入时，在确定注意力权重矩阵之后，需要判断起始标记的嵌入对应的注意力权重是否大于或者等于权重阈值，当起始标记的嵌入对应的注意力权重小于权重阈值时，对起始标记的嵌入对应的注意力权重进行调整，使得起始标记的嵌入对应的注意力权重大于或者等于权重阈值，类似地，也需要判断结束标记的嵌入对应的注意力权重是否大于或者等于权重阈值，当结束标记的嵌入对应的注意力权重小于权重阈值时，对结束标记的嵌入对应的注意力权重进行调整，使得结束标记的嵌入对应的注意力权重大于或者等于权重阈值。
基于此，在注意力权重矩阵中确保起始标记的嵌入以及结束标记的嵌入各自对应的注意力权重均大于或者等于权重阈值，由于起始标记的嵌入以及结束标记的嵌入各自对应的注意力权重相对较高，所以多模态大语言模型在生成预测嵌入时，能够通过注意力权重矩阵保持对起始标记的嵌入以及结束标记的嵌入所携带的关键信息的关注，避免因滑动窗口的滑动而导致信息丢失，有助于多模态大语言模型在处理长序列时保持注意力分布均匀和稳定，从而降低多模态大语言模型推理时的困惑度，有效地提升多模态大语言模型的性能，还能够更关注待处理嵌入序列中的重要信息而忽略次要信息，高效地利用和整合输入信息，增强模型理解和生成复杂输出的能力。
示例性地，当起始标记的嵌入对应的注意力权重为0.2，权重阈值为0.25时，起始标记的嵌入对应的注意力权重与权重阈值之间的目标差值为-0.05，需要将起始标记的嵌入对应的注意力权重增大0.05，然后将目标差值与待处理嵌入序列中其余嵌入对应的注意力权重之间的比例相乘，得到待处理嵌入序列中其余嵌入对应的注意力权重的调整量，然后基于调整量对待处理嵌入序列中其余嵌入对应的注意力权重进行调整，以确保待处理嵌入序列中各个嵌入对应的注意力权重之和等于一。
在一些实施例中，对属于视觉类型的预测嵌入进行映射，得到预测视觉特征，例如，可以是根据属于视觉类型的预测嵌入构建第二键矩阵和第二价值矩阵；获取预先训练好的多个第一可学习向量，根据多个第一可学习向量构建第二查询矩阵，其中，多个第一可学习向量的总维度与参考视觉特征的维度相同；基于第二查询矩阵、第二键矩阵和第二价值矩阵进行注意力处理，得到预测视觉特征。
其中，第二键矩阵为属于视觉类型的预测嵌入与可学习的第二键参数矩阵的乘积，第二价值矩阵为属于视觉类型的预测嵌入与可学习的第二价值参数矩阵的乘积，第二键矩阵和第二价值矩阵均包括属于视觉类型的预测嵌入的隐状态，第二查询矩阵为多个第一可学习向量与可学习的第二查询参数矩阵的乘积，第二查询矩阵包括多个第一可学习向量的隐状态。
示例的，可学习的第二键参数矩阵和可学习的第二价值参数矩阵可以通过执行以下处理得到：获取初始化的深度学习模型、初始化的第二键参数矩阵和初始化的第二价值参数矩阵；通过初始化的深度学习模型对输入序列进行预测，得到预测输出；基于输入序列与输入序列的真实标签确定损失；通过反向传播算法，基于损失确定针对初始化的第二键参数矩阵和初始化的第二价值参数矩阵的梯度，根据梯度调用优化器更新初始化的第二键参数矩阵和初始化的第二价值参数矩阵的值，直至模型收敛，得到可学习的第二键参数矩阵和可学习的第二价值参数矩阵。
通常情况下，第二键矩阵和第二价值矩阵由多个连续的属于视觉类型的预测嵌入构建得到，假设第二键矩阵和第二价值矩阵均由64个连续的属于视觉类型的预测嵌入构建得到，预测嵌入的维度为1*768，参考视觉特征的维度为256*768，第一可学习向量的数量为256个，每个第一可学习向量的维度为1*768，256个第一可学习向量的总维度为256*768，第二查询矩阵由256个维度为第一可学习向量构建得到，第二查询参数矩阵、第二键参数矩阵和第二价值参数矩阵的维度均为768*768，则64个属于视觉类型的预测嵌入与第二键参数矩阵相乘得到64*768维的第二键矩阵，64个属于视觉类型的预测嵌入与第二价值参数矩阵相乘得到64*768维的第二价值矩阵，256个第一可学习向量与第二查询参数矩阵相乘得到256*768维的第二查询矩阵，基于第二查询矩阵、第二键矩阵和第二价值矩阵进行注意力处理，得到256*768维的预测视觉特征。
这里，上述“基于第二查询矩阵、第二键矩阵和第二价值矩阵进行注意力处理，得到预测视觉特征”可以通过执行以下处理确定：基于第二查询矩阵和第二键矩阵确定候选权重矩阵，其中，候选权重矩阵包括预测序列中的各个嵌入各自对应的注意力权重；基于候选权重矩阵与第二价值矩阵的乘积生成预测视觉特征。
基于此，多模态大语言模型输出的预测嵌入处于文本嵌入空间，该预测嵌入具有语言信息，为了确保预测图像的有效生成，需要对属于视觉类型的预测嵌入进行映射，得到处于视觉特征空间的预测视觉特征，例如，先由多个第一可学习向量构建的第二查询矩阵，以及由属于视觉类型的预测嵌入构建的第二键矩阵和第二价值矩阵，然后对第二查询矩阵、第二键矩阵和第二价值矩阵进行注意力处理，由于第二查询矩阵的来源与第二键矩阵以及第二价值矩阵的来源是不同的，实际上是对第二查询矩阵、第二键矩阵和第二价值矩阵进行交叉注意力处理得到预测视觉特征，使得预测视觉特征与处于视觉特征空间的多个第一可学习向量对齐,实现将属于视觉类型的预测嵌入映射为具有视觉信息的预测视觉特征。
示例的，参考图4，图4为本申请实施例提供的得到预测视觉特征的一种可选的流程示意图；多模态大语言模型内部设置有N个堆叠的第一查询Transformer层，第一查询Transformer层包括第一自注意力子层、第一交叉注意力子层和第一前馈子层，第二Transformer层用于对属于视觉类型的预测嵌入进行映射，将多个第一可学习向量以及属于视觉类型的预测嵌入输入至第一查询Transformer层后，第一自注意力子层能够对多个第一可学习向量进行自注意力处理，得到多个第一隐藏向量，第一交叉注意力子层能够对多个第一隐藏向量以及属于视觉类型的预测嵌入进行交叉注意力处理，得到多个第二隐藏向量，通过第一前馈子层对多个第二隐藏向量进行映射，得到预测视觉特征。
另外，第一查询Transformer层可结合文本Transformer层进行预训练，文本Transformer层用于对文本进行编码，在预训练过程中，将第一查询Transformer层的输出与文本Transformer层的输出进行对齐，通过对第一查询Transformer层进行预训练，能够学习到具有视觉信息的多个第一可学习向量，使得第一查询Transformer层能够通过交叉注意力处理从预测嵌入中筛选出有用的语言信息，并将有用的语言信息从文本嵌入空间转换至视觉特征空间，能够有效提高预测图像的质量。
值得注意的是，与将多模态大语言模型的输出空间与视觉特征空间对齐类似，多模态大语言模型在处理参考视觉特征的嵌入时，需要多模态大语言模型的输入空间与视觉特征空间对齐，即对参考视觉特征的嵌入进行映射，下面对参考视觉特征的映射过程进行详细描述。
在一些实施例中，基于多模态大语言模型确定由参考文本的嵌入、起始标记的嵌入、参考视觉特征的嵌入以及结束标记的嵌入拼接得到的目标嵌入序列，例如，可以是基于多模态大语言模型，根据参考视觉特征构建第三键矩阵和第三价值矩阵；获取预先训练好的多个第二可学习向量，根据多个第二可学习向量构建第三查询矩阵，其中，第二可学习向量维度与参考文本的嵌入的维度相同；基于第三查询矩阵、第三键矩阵和第三价值矩阵进行注意力处理，得到参考视觉特征的嵌入；将参考文本的嵌入、起始标记的嵌入、参考视觉特征的嵌入以及结束标记的嵌入进行拼接得到目标嵌入序列。
其中，由于参考文本、起始标记和结束标记均为对应于文本模态的输入数据，可分别对参考文本、起始标记以及结束标记进行嵌入处理，能够确定参考文本的嵌入、起始标记的嵌入以及结束标记的嵌入，在对参考文本进行嵌入处理时，需要先对参考文本进行分词处理，再对参考文本的分词结果进行嵌入处理；参考文本的嵌入、起始标记的嵌入以及结束标记的嵌入均处于文本嵌入空间，参考文本的嵌入、起始标记的嵌入以及结束标记的嵌入的维度是一致的。
其中，第三键矩阵为参考视觉特征与可学习的第三键参数矩阵的乘积，第三价值矩阵为参考视觉特征与可学习的第三价值参数矩阵的乘积，第三键矩阵和第三价值矩阵均包括参考视觉特征的隐状态，第三查询矩阵为多个第二可学习向量与可学习的第三查询参数矩阵的乘积，第三查询矩阵包括多个第二可学习向量的隐状态。
示例性地，第三键矩阵和第三价值矩阵由维度为256*768的参考视觉特征构建得到，第二可学习向量的数量为64个，每个第二可学习向量的维度为1*768，每个第二可学习向量维度与参考文本的嵌入的维度相同，第三查询矩阵由64个第二可学习向量构建得到，第三查询参数矩阵、第三键参数矩阵和第三价值参数矩阵的维度均为768*768，则参考视觉特征与第三键参数矩阵相乘得到256*768维的第三键矩阵，参考视觉特征与第三价值参数矩阵相乘得到256*768维的第三价值矩阵，64个第二可学习向量与第三查询参数矩阵相乘得到64*768维的第三查询矩阵，基于第三查询矩阵、第三键矩阵和第三价值矩阵进行注意力处理，得到64个维度为1*768的参考视觉特征的嵌入，因此，当多模态大语言模型输出的预测词元为起始词元时，后续生成的第1个至第64个的预测嵌入用于确定预测视觉特征，后续生成的第65个预测嵌入用于确定预测词元。
基于此，由参考图像提取得到的参考视觉特征处于视觉特征空间，参考视觉特征具有视觉信息，为了确保多模态大语言模型能够有效处理输入数据，需要对参考视觉特征进行映射，得到处于文本嵌入空间的参考视觉特征的嵌入，例如，先由多个第二可学习向量构建的第三查询矩阵，以及由参考视觉特征构建的第三键矩阵和第三价值矩阵，然后对第三查询矩阵、第三键矩阵和第三价值矩阵进行注意力处理，由于第三查询矩阵的来源与第三键矩阵以及第三价值矩阵的来源是不同的，实际上是对第三查询矩阵、第三键矩阵和第三价值矩阵进行交叉注意力处理得到参考视觉特征的嵌入，使得参考视觉特征的嵌入与处于文本嵌入空间的第二可学习向量对齐,实现将参考视觉特征映射为具有语言信息的参考视觉特征的嵌入，使得参考文本的嵌入、起始标记的嵌入、参考视觉特征的嵌入以及结束标记的嵌入均处于文本嵌入空间，能够有效减轻多模态大语言模型的学习对齐视觉模态与文本模态的负担，从而缓解了遗忘问题。
示例的，参考图5，图5为本申请实施例提供的得到参考视觉特征的嵌入的一种可选的流程示意图；多模态大语言模型内部设置有N个堆叠的第二查询Transformer层，第二查询Transformer层包括第二自注意力子层、第二交叉注意力子层和第二前馈子层，第二Transformer层用于对参考视觉特征进行映射，将多个第二可学习向量以及参考视觉特征输入至第二查询Transformer层后，第二自注意力子层能够对多个第二可学习向量进行自注意力处理，得到多个第三隐藏向量，第二交叉注意力子层能够对多个第三隐藏向量以及参考视觉特征进行交叉注意力处理，得到多个第四隐藏向量，通过第二前馈子层对多个第四隐藏向量进行映射，得到参考视觉特征的嵌入。
另外，第二查询Transformer层可结合图像Transformer层进行预训练，图像Transformer层用于对图像进行编码，在预训练过程中，将第二查询Transformer层的输出与图像Transformer层的输出进行对齐，通过对第二查询Transformer层进行预训练，能够学习到具有语言信息的多个第二可学习向量，使得第二查询Transformer层能够通过交叉注意力处理从预测嵌入中筛选出有用的视觉信息，并将有用的视觉信息从视觉特征空间转换至文本嵌入空间，能够有效减轻多模态大语言模型的学习对齐视觉模态与文本模态的负担。
在一些实施例中，参考图6，图6为本申请实施例提供的生成预测图像的一种可选的流程示意图；基于预测序列生成预测图像以及预测文本，例如，可以是基于预测词元生成预测文本；获取随机生成的第一噪声图像，将第一噪声图像输入至扩散模型，将预测视觉特征作为扩散模型的扩散条件对第一噪声图像进行去噪处理，生成预测图像。
其中，第一噪声图像是随机生成的，第一噪声图像的像素具备随机性。第一噪声图像的尺寸与参考图像的尺寸可以是相同的，即第一噪声图像的高度与参考图像的高度相同，第一噪声图像的宽度与参考图像的宽度相同，使得基于第一噪声图像去噪生成的预测图像与参考图像的尺寸相同。
其中，扩散模型(Diffusion Model)有两个主要的过程:前向过程和后向过程；前向过程又称为扩散过程，扩散模型在前向过程中对图像逐步加入高斯噪声直至图像成为完全随机噪声，示例性地，扩散过程整体可以是一个参数化的马尔可夫链(Markov chain)；后向过程又称为逆扩散过程，扩散模型在后向过程中利用一系列马尔可夫链在每个时间步逐步去除预测噪声，从而从高斯噪声中恢复数据；因此，扩散模型能够在后向过程中基于预测视觉特征对第一噪声图像进行去噪处理，从而生成预测图像，即通过扩散模型的去噪处理得到满足预测视觉特征的预测图像。
示例的，扩散模型包括Unet网络，Unet网络可以在后向过程对噪声图像进行去噪处理，对于Unet网络中的任意一个特征提取器，基于预测视觉特征对当前的特征提取器的输出进行交叉注意力处理，得到交叉注意力特征，将交叉注意力特征输入至下一个特征提取器，因此，对于Unet网络中的每一个特征提取器，利用预测视觉特征对当前的特征提取器输出进行交叉注意力处理，交叉注意力处理可以帮助扩散模型更加关注图像中的关键区域，提高去噪的准确性。Unet网络的特征提取器可连接有交叉注意力层，在交叉注意力层中基于当前的特征提取器输出确定查询特征，基于预测视觉特征确定键特征和价值特征，进而通过查询特征、键特征和价值特征确定交叉注意力特征。
基于此，利用扩散模型通过多个时间步逐步去除第一噪声图像的预测噪声，从而恢复得到高质量、细节丰富的预测图像；基于全局注意力机制，在每个时间步预测噪声时都将预测视觉特征注入扩散模型，相当于将预测视觉特征作为全局潜码，即全局特征(Global Latent)，通过交叉注意力处理，使得扩散模型在每一步生成过程中都考虑预测视觉特征以确保预测图像与预测视觉特征相匹配，有效提高预测图像的准确性，从而提高预测图像的质量。
在一些实施例中，将参考文本与在两端分别添加了起始标记以及结束标记的参考视觉特征输入至多模态大语言模型之前，多模态数据处理方法还包括：获取第一样本图像、第一样本文本、目标图像以及目标文本，其中，第一样本文本用于描述第一样本图像中的内容，目标文本用于描述目标图像中的内容，目标图像为与第一样本图像关联的图像；提取第一样本图像的第一视觉特征，将第一样本文本与在两端分别添加了起始标记以及结束标记的第一视觉特征输入至多模态大语言模型进行推理；基于目标文本确定多模态大语言模型进行推理的第一损失，基于目标图像确定多模态大语言模型进行推理的第二损失；基于第一损失和第二损失对多模态大语言模型进行训练。
其中，与参考图像类似，第一样本图像和目标图像均为视觉模态数据，与参考文本类似，第一样本文本和目标文本均为文本模态数据，目标图像与第一样本图像相关联，例如，利用多模态大语言模型进行故事生成时，目标图像是第一样本图像在下一个情景的图像，又例如，利用多模态大语言模型进行步骤生成时，目标图像是第一样本图像在下一个步骤的图像，又例如，利用多模态大语言模型进行内容增强时，目标图像是与第一样本图像相近的图像。
示例的，参考图7，图7为本申请实施例提供的训练多模态大语言模型的一种可选的流程示意图。
其中，利用多模态大语言模型生成预测文本和预测图像之前，需要先对多模态大语言模型进行训练，在模型训练过程中，对第一样本图像进行特征提取，得到第一视觉特征，将第一样本文本与在两端分别添加了起始标记以及结束标记的第一视觉特征输入至训练后的多模态大语言模型进行推理，与预测文本的生成过程类似，多模态大语言模型能够输出相应的训练词元，进而生成相应的训练文本，类似地，与预测图像的生成过程类似，多模态大语言模型能够输出相应的训练视觉特征。
基于此，通过目标文本和训练文本确定第一损失，通过提取目标图像的目标视觉特征,然后通过目标视觉特征和训练视觉特征确定第二损失，基于第一损失和第二损失对多模态大语言模型进行训练，能够提高多模态大语言模型的预测准确度。
示例的，第一损失可通过将目标文本和训练文本输入至交叉熵损失函数进行计算得到，第二损失可通过将目标视觉特征和训练视觉特征输入至余弦相似性损失函数进行计算得到，除此之外，第一损失也可以通过其他方式计算得到，第二损失也可以通过其他方式计算得到，用于确定第一损失和第二损失的损失函数还可以包括以下任意一种：均方误差损失函数、多标签分类损失函数和三元组损失函数，本申请实施例在此不作限定。
在一些实施例中，在模型训练过程中，多模态大语言模型通过执行预测下一个嵌入的任务来生成输出序列，第一样本文本、起始标记、第一视觉特征、结束标记、目标文本以及目标图像拼接后输入多模态大语言模型，由于目标文本和目标图像均为标签数据，将目标文本以及目标图像作为多模态大语言模型的输入，使得多模态大语言模型在生成训练文本以及训练图像时，无需等待生成某个位置的嵌入后，才生成下一个位置的嵌入，能够进行并行训练，提高训练效率，多模态大语言模型在训练过程中更容易收敛。
在一些实施例中，将第一损失和第二损失进行反向传播，从而更新多模态大语言模型的参数，多次迭代计算第一损失和第二损失以及更新参数的过程，直至损失收敛，停止迭代过程，得到训练后的多模态大语言模型。
示例的，反向传播是通过反向传播算法实现的，从输出层到输入层计算每个神经元的梯度，并根据梯度更新神经元的权重和偏置，采用梯度下降的方式不断更新参数，使得损失值降低。梯度下降可以采用各种梯度下降的算法，例如批量梯度下降算法、随机梯度下降算法、自适应梯度下降算法和动量梯度下降算法。
在一些实施例中，基于第一损失和第二损失对多模态大语言模型进行训练，例如，可以是基于损失权重对第一损失和第二损失进行加权求和，得到模型损失，基于模型损失对多模态大语言模型进行训练。基于此，通过确定合适的模型损失，能够提高多模态大语言模型的训练效果。
示例的，损失权重可以通过多种方式确定，在一种确定损失权重的方式中，损失权重是预设的，例如，第一损失和第二损失各自对应的损失权重均设为1，又例如，第一损失对应的损失权重设为1，第二损失对应的损失权重设为10。
下面对另一种确定损失权重的方式进行详细描述。
在一些实施例中，基于第一损失和第二损失对多模态大语言模型进行训练，例如，可以是将第一损失和第二损失拼接后输入回归模型进行预测，得到第一损失和第二损失各自对应的损失权重，其中，回归模型与多模态大语言模型联合训练；基于损失权重对第一损失和第二损失进行加权求和，得到模型损失，基于模型损失对多模态大语言模型进行训练。
示例的，第一损失L1的权重为第二损失L2的权重为则对第一损失和第二损失进行加权求和得到的模型损失为
基于此，将第一损失和第二损失拼接后输入回归模型，回归模型能够预测出第一损失和第二损失各自对应的损失权重，损失权重用于表征第一损失和第二损失对模型损失的贡献程度，基于损失权重对第一损失和第二损失进行加权求和，得到模型损失，能够通过损失权重自适应调节第一损失和第二损失，能够避免出现过拟合，通过对回归模型与多模态大语言模型进行联合训练，回归模型能够更准确地预测出损失权重，提高多模态大语言模型的训练效果，从而提高多模态大语言模型的预测准确度。
在一些实施例中，参考图8，图8为本申请实施例提供的联合训练特征提取器与扩散模型的一种可选的第一流程示意图；第一视觉特征通过特征提取器提取得到，提取第一样本图像的第一视觉特征之前，多模态数据处理方法还包括：获取第二样本图像，将第二样本图像输入特征提取器，提取第二样本图像的第二视觉特征；获取随机生成的第二噪声图像，将第二噪声图像输入至扩散模型，将第二视觉特征作为扩散模型的扩散条件对第二噪声图像进行去噪处理，生成重建图像；基于重建图像和第二样本图像之间的差异确定第三损失，基于第三损失对特征提取器和扩散模型进行联合训练。
其中，与参考图像类似，第二样本图像也为视觉模态数据，特征提取器用于对第二样本图像进行特征提取，从第二样本图像提取的更抽象且更具信息量的第二视觉特征；第二噪声图像是随机生成的，第二噪声图像的像素具备随机性。第二噪声图像的尺寸与第二样本图像的尺寸可以是相同的，即第二噪声图像的高度与第二样本图像的高度相同，第二噪声图像的宽度与第二样本图像的宽度相同，使得基于第二噪声图像去噪生成的重建图像与第二样本图像的尺寸相同。
其中，利用扩散模型通过多个时间步逐步去除第二噪声图像的预测噪声，从而恢复得到高质量、细节丰富的第二样本图像；基于全局注意力机制，在每个时间步预测噪声时都将第二视觉特征注入扩散模型，相当于将第二视觉特征作为全局潜码，即全局特征，通过交叉注意力处理，使得扩散模型在每一步生成过程中都考虑第二视觉特征以确保第二样本图像与第二视觉特征相匹配。
基于此，利用特征提取器对第二样本图像进行特征提取，相当于对第二样本图像进行标记化，将第二视觉特征作为扩散模型的扩散条件生成重建图像，相当于对第二视觉特征进行去标记化，通过重建图像和第二样本图像之间的差异确定第三损失，然后基于第三损失对特征提取器和扩散模型进行联合训练，确保了特征提取器和扩散模型之间的高效协同作用，使得特征提取器学习从第二样本图像中提取更有意义的第二视觉特征，扩散模型能够将第二视觉特征作为去噪处理的指导，有助于在去噪过程中生成更准确的重建图像，能够同时提高特征提取器的特征提取能力以及扩散模型的图像重建能力。
示例的，第三损失可通过将重建图像与第二样本图像输入至均方误差损失函数进行计算得到，除此之外，第三损失也可以通过其他方式计算得到，确定第三损失的损失函数包括但不限于以下任意一种：交叉熵损失函数、多标签分类损失函数和三元组损失函数，本申请实施例在此不作限定。
在一些实施例中，特征提取器可以是冻结的，第一视觉特征通过特征提取器提取得到，提取第一样本图像的第一视觉特征之前，多模态数据处理方法还包括：获取第二样本图像，将第二样本图像输入特征提取器，提取第二样本图像的第二视觉特征；获取随机生成的第二噪声图像，将第二噪声图像输入至扩散模型，将第二视觉特征作为扩散模型的扩散条件对第二噪声图像进行去噪处理，生成重建图像；基于重建图像和第二样本图像之间的差异确定第三损失，基于第三损失对扩散模型进行训练。基于此，能够提高扩散模型的图像重建能力。
这里，可以针对特征提取器和扩散模型进行分阶段训练，第一阶段冻结扩散模型的参数，提取重建图像的第四视觉特征；基于第二视觉特征与第四视觉特征确定第五损失；基于第五损失更新特征提取器的参数，得到训练后的特征提取器。第二阶段冻结训练后的特征提取器的参数，基于第三损失更新扩散模型的参数，得到训练后的扩散模型；基于第三损失对训练后的特征提取器和训练后的扩散模型进行微调，直至特征提取器和扩散模型均收敛。
本申请实施例通过分阶段训练，在前两个阶段中的每个阶段，只训练一个模型，每个模型可以在其特定的任务上单独优化，可以有效地利用计算资源。单独训练每个模型可以更容易地理解每个模型的作用和性能，在联合训练阶段，可以更清楚地看到每个模型对最终结果的贡献。
在一些实施例中，参考图9，图9为本申请实施例提供的再次训练扩散模型的一种可选的第二流程示意图；基于第一损失和第二损失对多模态大语言模型进行训练之后，多模态数据处理方法还包括：获取第三样本图像、第二样本文本和标签图像，其中，第二样本文本用于描述第三样本图像中的内容，标签图像为与第三样本图像关联的图像；提取第三样本图像的第三视觉特征，将第二样本文本与在两端分别添加了起始标记以及结束标记的第三视觉特征输入至训练后的多模态大语言模型进行推理；获取随机生成的第三噪声图像，将第三噪声图像输入至训练后的扩散模型，将训练后的多模态大语言模型输出的视觉特征作为扩散模型的扩散条件对第三噪声图像进行去噪处理，生成关联图像；根据关联图像和标签图像确定第四损失，基于第四损失再次对扩散模型进行训练。
其中，与参考图像类似，第三样本图像和标签图像均为视觉模态数据，第二视觉特征是指从第三样本图像提取的更抽象且更具信息量的特征；第三噪声图像是随机生成的，第三噪声图像的像素具备随机性。第三噪声图像的尺寸与第三样本图像的尺寸可以是相同的，即第三噪声图像的高度与第三样本图像的高度相同，第三噪声图像的宽度与第三样本图像的宽度相同，使得基于第三噪声图像去噪生成的重建图像与第三样本图像的尺寸相同。
其中，标签图像为与第三样本图像关联的图像，例如，利用多模态大语言模型进行故事生成时，标签图像是第三样本图像在下一个情景的图像，又例如，利用多模态大语言模型进行步骤生成时，标签图像是第三样本图像在下一个步骤的图像，又例如，利用多模态大语言模型进行内容增强时，标签图像是与第三样本图像相近的图像。
其中，利用扩散模型通过多个时间步逐步去除第三噪声图像的预测噪声，从最后一个时间步开始，逐步减少噪声，从而恢复得到高质量、细节丰富的第三样本图像；基于全局注意力机制，在每个时间步预测噪声时都将第二视觉特征注入扩散模型，相当于将第二视觉特征作为全局潜码，即全局特征(Global Latent)，通过交叉注意力处理，计算第二视觉特征与全局潜码之间的注意力权重，根据计算得到的注意力权重，将全局潜码的信息融合到第二视觉特征中，通过融合了全局潜码信息的第二视觉特征，预测当前时间步的噪声，将预测的噪声从第三噪声图像中去除，得到去噪后的第三样本图像，使得扩散模型在每一步生成过程中都考虑第二视觉特征以确保第三样本图像与第二视觉特征相匹配。
基于此，对第三样本图像进行特征提取，得到第三视觉特征，将第二样本文本与在两端分别添加了起始标记以及结束标记的第三视觉特征输入至训练后的多模态大语言模型进行推理，与预测视觉特征的输出过程类似，多模态大语言模型能够输出相应的视觉特征，然后多模态大语言模型输出的视觉特征作为扩散模型的扩散条件对第三噪声图像进行去噪处理，生成关联图像，通过关联图像和标签图像确定第四损失，基于第四损失再次对扩散模型进行训练，扩散模型能够生成更接近标签图像的关联图像，相当于对扩散模型进行微调，以更好地将多模态大语言模型的输出空间与视觉特征空间对齐，增强生成图像的风格一致性。
示例的，第四损失可通过将关联图像和标签图像输入至均方误差损失函数进行计算得到，除此之外，第四损失也可以通过其他方式计算得到，确定第四损失的损失函数包括但不限于以下任意一种：交叉熵损失函数、多标签分类损失函数和三元组损失函数，本申请实施例在此不作限定。
在一些实施例中，获取第一样本图像、第一样本文本、目标图像以及目标文本，体可以是获取多个连续的视频帧以及各个视频帧对应的字幕文本；调用文本描述模型分别对各个视频帧进行文本生成处理，生成各个视频帧对应的描述文本；构建用于提示参照描述文本进行内容增强的提示指令，将提示指令、各个字幕文本和对应的描述文本拼接后输入目标大语言模型，基于目标大语言模型分别对各个字幕文本进行内容增强，得到各个视频帧对应的增强文本；将最后一个视频帧确定为目标图像，将其余的视频帧确定为第一样本图像，将目标图像对应的增强文本确定为目标文本，将第一样本图像对应的增强文本确定为第一样本文本。
其中，多个连续的视频帧可以从动画、电影、电视节目等视频来源得到，以动画为例，可以按照连续的情景对动画视频进行截取，每个情景截取一个画面，也可以按照固定的时间间隔对动画视频进行截取，每隔一段时间截取一个画面，然后将截取得到的多个连续的画面作为多个连续的视频帧；各个视频帧可以是RGB图像、灰度图像、二值图像、深度图像等等，本申请实施例在此不作限定。
其中，字幕文本是指动画、电影、电视节目等视频内容中的口语对话、声音效果、音乐提示以及其他听觉信息的文字展示，字幕文本通常与多媒体内容中的音频同步，使得观众能够在未听到声音或需要辅助理解的情况下，通过阅读字幕文本来获取相应的内容，提高视频内容的观看效果。
其中，文本描述模型能够对视频帧进行目标检测，得到包括检测对象的目标检测结果，文本描述模型还能够对视频帧进行特征提取，得到包括场景特征以及检测对象的动作特征的画面特征，然后根据目标检测结果和画面特征对视频帧进行文本预测，得到视频帧对应的描述文本，描述文本用于表征视频帧的人物、环境、事件等背景信息。
其中，目标大语言模型属于大语言模型，大语言模型是使用大量文本数据训练的深度学习模型，可以生成自然语言文本或理解语言文本的含义；大语言模型一般采用循环神经网络(RNN)或变种，如长短时记忆网络(LSTM)和门控循环单元(Gated Recurrent Unit，GRU)，以捕捉文本序列中的上下文信息，从而实现自然语言文本的生成、语言模型评估、文本分类、情感分析等任务；在自然语言处理领域，大语言模型已经被广泛应用，例如语音识别、机器翻译、自动摘要、对话系统、智能问答等。
示例的，提示指令可以为“请参考以下描述文本，对字幕文本进行内容增强”，将提示指令、各个字幕文本和对应的描述文本拼接后输入目标大语言模型，假设描述文本和字幕文本的数量是两个，目标大语言模型的模型输入形式可以为“描述文本1、字幕文本1、描述文本2、字幕文本2”，目标大语言模型的模型输入形式也可以为“描述文本1、描述文本2、字幕文本1、字幕文本2”，目标大语言模型能够将描述文本所携带的背景信息融合到字幕文本中，实现对字幕文本的内容增强。
基于此，先获取视频帧以及对应的字幕文本，然后生成视频帧的描述文本，再通过目标大语言模型基于描述文本对字幕文本进行内容增强，得到增强文本，然后将视频帧和对应的增强文本作为图像文本对构建数据集，数据集包含了高分辨率的视频帧以及高质量的增强文本，而且数据集可以包含大量的图像文本对，然后将多个连续的视频帧中的最后一个视频帧确定为目标图像，将多个连续的视频帧中的其余视频帧确定为第一样本图像，使得目标图像为与第一样本图像关联的图像，而且目标图像用于指示第一样本图像的下一个情景，在此基础上，将目标图像对应的增强文本确定为目标文本，以及将第一样本图像对应的增强文本确定为第一样本文本，确保目标文本用于描述目标图像中的内容，并确保第一样本文本用于描述第一样本图像中的内容，通过第一样本图像、第一样本文本、目标图像以及目标文本训练多模态大语言模型，使得多模态大语言模型能够有效处理故事生成、步骤生成等任务，提升多模型大语言模型对多模态内容的生成质量。
下面详细说明多模态数据处理方法的完整过程，完整过程具体包括训练阶段和推理阶段。
下面对训练阶段进行详细描述，训练阶段可分为三个训练子阶段，下面以服务器(例如服务器102)是执行主体为例进行说明。
在第一个训练子阶段中，对扩散模型和特征提取器进行预训练。
首先，获取第二样本图像，将第二样本图像输入特征提取器，提取第二样本图像的第二视觉特征。
然后，获取随机生成的第二噪声图像，将第二噪声图像输入至扩散模型，将第二视觉特征作为扩散模型的扩散条件对第二噪声图像进行去噪处理，生成重建图像。
然后，基于重建图像和第二样本图像之间的差异确定第三损失，基于第三损失对特征提取器和扩散模型进行联合训练。
在第二个训练子阶段中，对多模态大语言模型进行训练。
首先，获取多个连续的视频帧以及各个视频帧对应的字幕文本。
然后，调用文本描述模型分别对各个视频帧进行文本生成处理，生成各个视频帧对应的描述文本。
然后，构建用于提示参照描述文本进行内容增强的提示指令，将提示指令、各个字幕文本和对应的描述文本拼接后输入目标大语言模型，基于目标大语言模型分别对各个字幕文本进行内容增强，得到各个视频帧对应的增强文本。
然后，将最后一个视频帧确定为目标图像，将其余的视频帧确定为第一样本图像，将目标图像对应的增强文本确定为目标文本，将第一样本图像对应的增强文本确定为第一样本文本，其中，第一样本文本用于描述第一样本图像中的内容，目标文本用于描述目标图像中的内容，目标图像为与第一样本图像关联的图像。
然后，提取第一样本图像的第一视觉特征，将第一样本文本与在两端分别添加了起始标记以及结束标记的第一视觉特征输入至多模态大语言模型进行推理。
然后，基于目标文本确定多模态大语言模型进行推理的第一损失，基于目标图像确定多模态大语言模型进行推理的第二损失。
然后，将第一损失和第二损失拼接后输入回归模型进行预测，得到第一损失和第二损失各自对应的损失权重，其中，回归模型与多模态大语言模型联合训练。
然后，基于损失权重对第一损失和第二损失进行加权求和，得到模型损失，基于模型损失对多模态大语言模型进行训练。
在第三个训练子阶段中，对扩散模型进行微调。
首先，获取第三样本图像、第二样本文本和标签图像，其中，第二样本文本用于描述第三样本图像中的内容，标签图像为与第三样本图像关联的图像。
然后，提取第三样本图像的第三视觉特征，将第二样本文本与在两端分别添加了起始标记以及结束标记的第三视觉特征输入至训练后的多模态大语言模型进行推理。
然后，获取随机生成的第三噪声图像，将第三噪声图像输入至训练后的扩散模型，将训练后的多模态大语言模型输出的视觉特征作为扩散模型的扩散条件对第三噪声图像进行去噪处理，生成关联图像。
然后，根据关联图像和标签图像确定第四损失，基于第四损失再次对扩散模型进行训练。
下面对推理阶段进行详细描述，下面以服务器(例如服务器102)是执行主体为例进行说明。
首先，获取参考图像以及参考文本，其中，参考文本用于描述参考图像中的内容。
然后，提取参考图像的参考视觉特征，将参考文本与在两端分别添加了起始标记以及结束标记的参考视觉特征输入至多模态大语言模型。
然后，基于多模态大语言模型，根据参考视觉特征构建第三键矩阵和第三价值矩阵。
然后，获取预先训练好的多个第二可学习向量，根据多个第二可学习向量构建第三查询矩阵，其中，第二可学习向量维度与参考文本的嵌入的维度相同。
然后，基于第三查询矩阵、第三键矩阵和第三价值矩阵进行注意力处理，得到参考视觉特征的嵌入。
然后，将参考文本的嵌入、起始标记的嵌入、参考视觉特征的嵌入以及结束标记的嵌入进行拼接得到目标嵌入序列。
然后，将起始标记的嵌入、结束标记的嵌入以及目标嵌入序列中被滑动窗口选中的嵌入拼接为待处理嵌入序列。
然后，基于待处理嵌入序列构建第一键矩阵和第一价值矩阵。
然后，基于待处理嵌入序列中最后一个位置的嵌入构建第一查询矩阵。
然后，基于第一查询矩阵和第一键矩阵确定注意力权重矩阵，其中，注意力权重矩阵包括待处理嵌入序列中的各个嵌入各自对应的注意力权重。
然后，将起始标记的嵌入或者结束标记的嵌入各自对应的注意力权重，调整至大于或者等于预设的权重阈值。
然后，基于调整后的注意力权重矩阵与第一价值矩阵的乘积生成预测嵌入。
然后，对属于文本类型的预测嵌入进行分类，输出预测词元。
然后，根据属于视觉类型的预测嵌入构建第二键矩阵和第二价值矩阵。
然后，获取预先训练好的多个第一可学习向量，根据多个第一可学习向量构建第二查询矩阵，其中，多个第一可学习向量的总维度与参考视觉特征的维度相同。
然后，基于第二查询矩阵、第二键矩阵和第二价值矩阵进行注意力处理，得到预测视觉特征。
然后，基于预测词元生成预测文本。
然后，获取随机生成的第一噪声图像，将第一噪声图像输入至扩散模型，将预测视觉特征作为扩散模型的扩散条件对第一噪声图像进行去噪处理，生成预测图像。
基于此，通过获取参考图像以及用于描述参考图像中的内容的参考文本，提取参考图像的参考视觉特征，在参考视觉特征的两端分别添加起始标记以及结束标记，能够便于后续基于起始标记以及结束标记区分出参考视觉特征以及参考文本，在此基础上，将参考文本与在两端分别添加了起始标记以及结束标记的参考视觉特征输入至多模态大语言模型，基于多模态大语言模型输出预测序列，再基于预测序列生成预测图像以及用于描述预测图像的预测文本，由于在推理的过程中，基于起始标记的嵌入、结束标记的嵌入以及目标嵌入序列中被滑动窗口选中的嵌入进行注意力处理，因此，即便是进行连续多个轮次的多模态数据处理，多模态大语言模型也能够通过注意力处理关注起始标记的嵌入、结束标记的嵌入以及目标嵌入序列中被滑动窗口选中的嵌入所携带的信息，从而降低多模态大语言模型推理时的困惑度，有效地提升多模态大语言模型的性能，进而提升多模态内容的生成质量。
下面对多模态大语言模型构建注意力的原理进行详细描述。
示例的，参考图10，图10为本申请实施例提供的注意力分数分布的一种可选的分布示意图。
其中，将首端标记的嵌入、测试文本的嵌入、起始标记的嵌入、结束标记的嵌入以及测试视觉特征的嵌入拼接得到测试嵌入序列，测试视觉特征是由测试图像进行特征提取得到的，利用多模态大语言模型对测试嵌入序列进行全局注意力处理时，先通过测试嵌入序列中各个嵌入确定查询矩阵与键矩阵，然后根据查询矩阵与键矩阵的转置的乘积确定所有嵌入的注意力分数分布。
基于此，如图10所示，将所有嵌入的注意力分数分布绘制在同一个分布示意图中，横坐标为键矩阵的索引，键矩阵的索引用于指示对应的嵌入，纵坐标为查询矩阵的索引，查询矩阵的索引用于指示对应的嵌入，可见，大多数查询主要关注首端标记的嵌入、标点符号的嵌入、起始标记的嵌入以及结束标记的嵌入，由于标点符号的嵌入的范数很小，所以标点符号的嵌入对推理结果的贡献较低，因此，在构建多模态大语言模型的注意力时，仅需要通过注意力处理关注起始标记的嵌入、结束标记的嵌入以及目标嵌入序列中被滑动窗口选中的嵌入所携带的信息即可，能够降低多模态大语言模型推理时的困惑度，有效地提升多模态大语言模型的性能，进而提升多模态内容的生成质量，还能够提高有效降低内存使用，并显著提高推理效率。
下面对本申请实施例提供的多模态大语言模型与其他模型的评估方式进行详细描述。
在第一种评估方式中，分别利用潜在扩散模型(Latent Diffusion Model，LDM)、故事生成模型(StoryGen)以及本申请实施例提供的多模态大语言模型生成图像，能够得到各个模型在FID和CLIP分数的评估指标，如下表1所示：
表1
其中，FID全称为Frechet Inception Distance，FID分数为真实图像和生成图像的特征向量之间距离的一种度量，当FID分数越低时，真实图像和生成图像的特征向量之间距离越小，模型生成的图像质量越高；CLIP全称为Contrastive Language-Image Pre-training，CLIP分数用于评估图像与文本之间的相关性，当CLIP分数越高时，生成图像和生成文本的之间的相关性最高，模型生成的图像质量越高。
可见，本申请实施例提供的多模态大语言模型的FID分数最低，CLIP分数越高，所以，本申请实施例提供的多模态大语言模型生成的图像质量最高，本申请实施例提供的多模态大语言模型的性能最优。
在第二种评估方式中，对本申请实施例提供的多模态大语言模型以及相关技术提供的图像文本生成模型进行多个对比测试，每个对比测试都能生成测试图像和测试文本，然后基于每个对比测试的结果确定两种模型的性能。其中，在每个对比测试中，多模态大语言模型与图像文本生成模型的输入内容都是一致的，从而确保对比结果的可靠性。
然后，在各个对比测试中，都需要计算两种模型的风格一致性分数，风格一致性分数用于表征生成内容与输入内容的风格相似度，当风格一致性分数越高时，生成内容与输入内容的风格相似度越高，反之，当风格一致性分数越低时，生成内容与输入内容的风格相似度越低，在此基础上，在所有对比测试中，统计多模态大语言模型的风格一致性分数大于图像文本生成模型的风格一致性分数的概率，得到获胜概率；统计多模态大语言模型的风格一致性分数等于图像文本生成模型的风格一致性分数的概率，得到平局概率；以及统计多模态大语言模型的风格一致性分数小于图像文本生成模型的风格一致性分数的概率，得到失败概率。
然后，在各个对比测试中，可确定两种模型的生成内容的叙述吸引力，例如，可以通过观众对生成内容吸引力的投票结果确定叙述吸引力，也可以基于生成内容的评论结果确定叙述吸引力，在此基础上，在所有对比测试中，统计多模态大语言模型的叙述吸引力大于图像文本生成模型的叙述吸引力的概率，得到获胜概率；统计多模态大语言模型的叙述吸引力等于图像文本生成模型的叙述吸引力的概率，得到平局概率；以及统计多模态大语言模型的叙述吸引力小于图像文本生成模型的叙述吸引力的概率，得到失败概率。
然后，在各个对比测试中，都需要计算两种模型的图文一致性分数，图文一致性分数用于表征测试图像与测试文本之间的相关性，当图文一致性分数越高时，测试图像与测试文本之间的相关性越高，反之，当图文一致性分数越低时，测试图像与测试文本之间的相关性越低，在此基础上，在所有对比测试中，统计多模态大语言模型的图文一致性分数大于图像文本生成模型的图文一致性分数的概率，得到获胜概率；统计多模态大语言模型的图文一致性分数等于图像文本生成模型的图文一致性分数的概率；得到平局概率，以及统计多模态大语言模型的图文一致性分数小于图像文本生成模型的图文一致性分数的概率，得到失败概率。
示例的，参考图11，图11为本申请实施例提供的FID分数的一种可选的柱状示意图。
可见，本申请实施例提供的多模态大语言模型的FID分数较低，所以，本申请实施例提供的多模态大语言模型生成的图像质量较高。
示例的，参考图12，图12为本申请实施例提供的风格一致性比对结果的一种可选的饼状示意图。
可见，本申请实施例提供的多模态大语言模型的风格一致性分数大于图像文本生成模型的风格一致性分数的概率为16.1％，即获胜概率为16.1％，本申请实施例提供的多模态大语言模型的风格一致性分数等于图像文本生成模型的风格一致性分数的概率为71.7％，即平局概率为71.7％，本申请实施例提供的多模态大语言模型的风格一致性分数大于图像文本生成模型的风格一致性分数的概率为12.2％，即失败概率为12.2％，可见，本申请实施例提供的多模态大语言模型的风格一致性优于相关技术提供的图像文本生成模型的风格一致性。
示例的，参考图13，图13为本申请实施例提供的叙述吸引力比对结果的一种可选的饼状示意图。
可见，本申请实施例提供的多模态大语言模型的叙述吸引力大于图像文本生成模型的叙述吸引力的概率为51.7％，即获胜概率为51.7％，本申请实施例提供的多模态大语言模型的叙述吸引力等于图像文本生成模型的叙述吸引力的概率为12.8％，即平局概率为12.8％，本申请实施例提供的多模态大语言模型的叙述吸引力大于图像文本生成模型的叙述吸引力的概率为35.6％，即失败概率为35.6％，可见，本申请实施例提供的多模态大语言模型的叙述吸引力显著优于相关技术提供的图像文本生成模型的叙述吸引力。
示例的，参考图14，图14为本申请实施例提供的图文一致性比对结果的一种可选的饼状示意图。
可见，本申请实施例提供的多模态大语言模型的图文一致性分数大于图像文本生成模型的图文一致性分数的概率为47.8％，即获胜概率为47.8％，本申请实施例提供的多模态大语言模型的图文一致性分数等于图像文本生成模型的图文一致性分数的概率为35.0％，即平局概率为35.0％，本申请实施例提供的多模态大语言模型的图文一致性分数大于图像文本生成模型的图文一致性分数的概率为17.2％，即失败概率为17.2％，可见，本申请实施例提供的多模态大语言模型的图文一致性显著优于相关技术提供的图像文本生成模型的图文一致性。
基于此，相较于相关技术提供的图像文本生成模型，本申请实施例提供的多模态大语言模型的FID分数较低、风格一致性较优、叙述吸引力较优以及图文一致性较优，因此，本申请实施例提供的多模态大语言模型的性能较优。
在第三种评估方式中，在多模态大语言模型中，分别利用本申请实施例提供的多模态数据处理方法所采用的注意力构建方式以及其他注意力构建方式进行推理，能够得到不同注意力构建方式的评估指标，如下表2所示：
表2
可见，相较于采用全局注意力的方式，本申请实施例提供的多模态数据处理方法能够提高有效降低内存使用，并显著减少推理时间，从而提高推理效率；相较于采用滑动窗口的方式，本申请实施例提供的多模态数据处理方法能够显著降低FID分数以及提高CLIP分数，有效地提升多模态大语言模型的性能，进而提升多模态内容的生成质量。
示例的，参考图14，图14为本申请实施例提供的图文一致性比对结果的一种可选的饼状示意图。
可见，本申请实施例提供的多模态数据处理方法可以应用于多种场景。
例如，在教育领域，可对教育相关的步骤图文进行扩展，步骤图文可包括实验步骤图文、知识点图文等等，获取首个步骤的参考图像以及参考文本，然后提取参考图像的参考视觉特征，将参考文本与在两端分别添加了起始标记以及结束标记的参考视觉特征输入至多模态大语言模型，基于多模态大语言模型输出预测序列，然后基于预测序列生成后续步骤的预测图像以及预测文本，实现对步骤图文进行扩展，预测图像以及预测文本能够保持风格一致性、内容吸引力以及图像文本一致性，从而提高步骤图文的整体质量。
例如，在娱乐领域，可对故事内容进行扩展，获取某个故事情景的参考图像以及参考文本，然后提取参考图像的参考视觉特征，将参考文本与在两端分别添加了起始标记以及结束标记的参考视觉特征输入至多模态大语言模型，基于多模态大语言模型输出预测序列，然后基于预测序列生成后续故事情景的预测图像以及预测文本，实现对故事内容进行扩展，预测图像以及预测文本能够保持风格一致性、内容吸引力以及图像文本一致性，从而提高故事内容的整体质量。
可以理解的是，虽然上述各个流程图中的各个步骤按照箭头的指示依次显示，但是这些步骤并不是必然按照箭头指示的顺序依次执行。除非本实施例中有明确的说明，这些步骤的执行并没有严格的顺序限制，这些步骤可以以其它的顺序执行。而且，上述流程图中的至少一部分步骤可以包括多个步骤或者多个阶段，这些步骤或者阶段并不必然是在同一时间执行完成，而是可以在不同的时间执行，这些步骤或者阶段的执行顺序也不必然是依次进行，而是可以与其它步骤或者其它步骤中的步骤或者阶段的至少一部分轮流或者交替地执行。
参照图15，图15为本申请实施例提供的多模态数据处理装置的一种可选的结构示意图，该多模态数据处理装置1500包括：
获取模块1501，配置为获取参考图像以及参考文本，其中，参考文本用于描述参考图像中的内容；
第一处理模块1502，配置为提取参考图像的参考视觉特征，将参考文本与在两端分别添加了起始标记以及结束标记的参考视觉特征输入至多模态大语言模型；
第二处理模块1503，配置为基于多模态大语言模型确定由参考文本的嵌入、起始标记的嵌入、参考视觉特征的嵌入以及结束标记的嵌入拼接得到的目标嵌入序列，基于起始标记的嵌入、结束标记的嵌入以及目标嵌入序列中被滑动窗口选中的嵌入进行注意力处理，输出预测序列；
生成模块1504，配置为基于预测序列生成预测图像以及预测文本，其中，预测文本用于描述预测图像中的内容。
在一些实施例中，预测序列包括预测词元和预测视觉特征，上述第二处理模块1503还配置为：基于起始标记的嵌入、结束标记的嵌入以及目标嵌入序列中被滑动窗口选中的嵌入进行注意力处理，生成预测嵌入；对属于文本类型的预测嵌入进行分类，输出预测词元；对属于视觉类型的预测嵌入进行映射，输出预测视觉特征。
在一些实施例中，上述第二处理模块1503还配置为：将起始标记的嵌入、结束标记的嵌入以及目标嵌入序列中被滑动窗口选中的嵌入拼接为待处理嵌入序列；基于待处理嵌入序列构建第一键矩阵和第一价值矩阵；基于待处理嵌入序列中最后一个位置的嵌入构建第一查询矩阵；基于第一查询矩阵、第一键矩阵和第一价值矩阵进行注意力处理，生成预测嵌入。
在一些实施例中，上述第二处理模块1503还配置为：基于第一查询矩阵和第一键矩阵确定注意力权重矩阵，其中，注意力权重矩阵包括待处理嵌入序列中的各个嵌入各自对应的注意力权重；将起始标记的嵌入或者结束标记的嵌入各自对应的注意力权重，调整至大于或者等于预设的权重阈值；基于调整后的注意力权重矩阵与第一价值矩阵的乘积生成预测嵌入。
在一些实施例中，上述第二处理模块1503还配置为：基于第一查询矩阵和第一键矩阵的转置的乘积，确定注意力分数矩阵；基于缩放因子对注意力分数矩阵进行调整，对调整后的注意力分数矩阵进行归一化处理，得到注意力权重矩阵。
在一些实施例中，上述第二处理模块1503还配置为：根据属于视觉类型的预测嵌入构建第二键矩阵和第二价值矩阵；获取预先训练好的多个第一可学习向量，根据多个第一可学习向量构建第二查询矩阵，其中，多个第一可学习向量的总维度与参考视觉特征的维度相同；基于第二查询矩阵、第二键矩阵和第二价值矩阵进行注意力处理，得到预测视觉特征。
在一些实施例中，上述第二处理模块1503还配置为：基于第二查询矩阵和第二键矩阵确定候选权重矩阵，其中，候选权重矩阵包括预测序列中的各个嵌入各自对应的注意力权重；基于候选权重矩阵与第二价值矩阵的乘积生成预测视觉特征。
在一些实施例中，上述生成模块1504还配置为：基于预测词元生成预测文本；获取随机生成的第一噪声图像，将第一噪声图像输入至扩散模型，将预测视觉特征作为扩散模型的扩散条件对第一噪声图像进行去噪处理，生成预测图像。
在一些实施例中，上述第二处理模块1503还配置为：基于多模态大语言模型，根据参考视觉特征构建第三键矩阵和第三价值矩阵；获取预先训练好的多个第二可学习向量，根据多个第二可学习向量构建第三查询矩阵，其中，第二可学习向量维度与参考文本的嵌入的维度相同；基于第三查询矩阵、第三键矩阵和第三价值矩阵进行注意力处理，得到参考视觉特征的嵌入；将参考文本的嵌入、起始标记的嵌入、参考视觉特征的嵌入以及结束标记的嵌入进行拼接得到目标嵌入序列。
在一些实施例中，多模态数据处理装置还包括第一训练模块(图中未示出)，第一训练模块配置为：获取第一样本图像、第一样本文本、目标图像以及目标文本，其中，第一样本文本用于描述第一样本图像中的内容，目标文本用于描述目标图像中的内容，目标图像为与第一样本图像关联的图像；提取第一样本图像的第一视觉特征，将第一样本文本与在两端分别添加了起始标记以及结束标记的第一视觉特征输入至多模态大语言模型进行推理；基于目标文本确定多模态大语言模型进行推理的第一损失，基于目标图像确定多模态大语言模型进行推理的第二损失；基于第一损失和第二损失对多模态大语言模型进行训练。
在一些实施例中，上述第一训练模块还配置为：将第一损失和第二损失拼接后输入回归模型进行预测，得到第一损失和第二损失各自对应的损失权重，其中，回归模型与多模态大语言模型联合训练；基于损失权重对第一损失和第二损失进行加权求和，得到模型损失，基于模型损失对多模态大语言模型进行训练。
在一些实施例中，多模态数据处理装置还包括第二训练模块(图中未示出)，第二训练模块配置为：获取第二样本图像，将第二样本图像输入特征提取器，提取第二样本图像的第二视觉特征；获取随机生成的第二噪声图像，将第二噪声图像输入至扩散模型，将第二视觉特征作为扩散模型的扩散条件对第二噪声图像进行去噪处理，生成重建图像；基于重建图像和第二样本图像之间的差异确定第三损失，基于第三损失对特征提取器和扩散模型进行联合训练。
在一些实施例中，上述第二训练模块还配置为：冻结扩散模型的参数，提取重建图像的第四视觉特征，基于第二视觉特征与第四视觉特征确定第五损失，基于第五损失更新特征提取器的参数，得到训练后的特征提取器；冻结训练后的特征提取器的参数，基于第三损失更新扩散模型的参数，得到训练后的扩散模型；基于第三损失对训练后的特征提取器和训练后的扩散模型进行微调。
在一些实施例中，多模态数据处理装置还包括第三训练模块(图中未示出)，第三训练模块配置为：获取第三样本图像、第二样本文本和标签图像，其中，第二样本文本用于描述第三样本图像中的内容，标签图像为与第三样本图像关联的图像；提取第三样本图像的第三视觉特征，将第二样本文本与在两端分别添加了起始标记以及结束标记的第三视觉特征输入至训练后的多模态大语言模型进行推理；获取随机生成的第三噪声图像，将第三噪声图像输入至训练后的扩散模型，将训练后的多模态大语言模型输出的视觉特征作为扩散模型的扩散条件对第三噪声图像进行去噪处理，生成关联图像；根据关联图像和标签图像确定第四损失，基于第四损失再次对扩散模型进行训练。
在一些实施例中，上述第一训练模块还配置为：获取多个连续的视频帧以及各个视频帧对应的字幕文本；调用文本描述模型分别对各个视频帧进行文本生成处理，生成各个视频帧对应的描述文本；构建用于提示参照描述文本进行内容增强的提示指令，将提示指令、各个字幕文本和对应的描述文本拼接后输入目标大语言模型，基于目标大语言模型分别对各个字幕文本进行内容增强，得到各个视频帧对应的增强文本；将最后一个视频帧确定为目标图像，将其余的视频帧确定为第一样本图像，将目标图像对应的增强文本确定为目标文本，将第一样本图像对应的增强文本确定为第一样本文本。
上述多模态数据处理装置1500与多模态数据处理方法基于相同的发明构思，通过获取参考图像以及用于描述参考图像中的内容的参考文本，提取参考图像的参考视觉特征，在参考视觉特征的两端分别添加起始标记以及结束标记，能够便于后续基于起始标记以及结束标记区分出参考视觉特征以及参考文本，在此基础上，将参考文本与在两端分别添加了起始标记以及结束标记的参考视觉特征输入至多模态大语言模型，基于多模态大语言模型输出预测序列，再基于预测序列生成预测图像以及用于描述预测图像的预测文本，由于在推理的过程中，基于起始标记的嵌入、结束标记的嵌入以及目标嵌入序列中被滑动窗口选中的嵌入进行注意力处理，因此，即便是进行连续多个轮次的多模态数据处理，多模态大语言模型也能够通过注意力处理关注起始标记的嵌入、结束标记的嵌入以及目标嵌入序列中被滑动窗口选中的嵌入所携带的信息，从而降低多模态大语言模型推理时的困惑度，有效地提升多模态大语言模型的性能，进而提升多模态内容的生成质量。
本申请实施例提供的用于执行上述多模态数据处理方法的电子设备可以是终端，参照图16，图16为本申请实施例提供的终端的部分结构框图，该终端包括：摄像头组件1610、第一存储器1620、输入单元1630、显示单元1640、传感器1650、音频电路1660、无线保真(wireless fidelity，WiFi)模块1670、第一处理器1680、以及第一电源1690等部件。本领域技术人员可以理解，图16中示出的终端结构并不构成对终端的限定，可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件布置。
摄像头组件1610可用于采集图像或视频。可选地，摄像头组件1610包括前置摄像头和后置摄像头。通常，前置摄像头设置在终端的前面板，后置摄像头设置在终端的背面。在一些实施例中，后置摄像头为至少两个，分别为主摄像头、景深摄像头、广角摄像头、长焦摄像头中的任意一种，以实现主摄像头和景深摄像头融合实现背景虚化功能、主摄像头和广角摄像头融合实现全景拍摄以及VR(Virtual Reality，虚拟现实)拍摄功能或者其它融合拍摄功能。
第一存储器1620可用于存储软件程序以及模块，第一处理器1680通过运行存储在第一存储器1620的软件程序以及模块，从而执行终端的各种功能应用以及数据处理。
输入单元1630可用于接收输入的数字或字符信息，以及产生与终端的设置以及功能控制有关的键信号输入。示例的，输入单元1630可包括触摸面板1631以及其他输入装置1632。
显示单元1640可用于显示输入的信息或提供的信息以及终端的各种菜单。显示单元1640可包括显示面板1641。
音频电路1660、扬声器1661，传声器1662可提供音频接口。
第一电源1690可以是交流电、直流电、一次性电池或可充电电池。
传感器1650的数量可以为一个或者多个，该一个或多个传感器1650包括但不限于：加速度传感器、陀螺仪传感器、压力传感器、光学传感器等等。其中：
加速度传感器可以检测以终端建立的坐标系的三个坐标轴上的加速度大小。比如，加速度传感器可以用于检测重力加速度在三个坐标轴上的分量。第一处理器1680可以根据加速度传感器采集的重力加速度信号，控制显示单元1640以横向视图或纵向视图进行用户界面的显示。加速度传感器还可以用于游戏或者用户的运动数据的采集。
陀螺仪传感器可以检测终端的机体方向及转动角度，陀螺仪传感器可以与加速度传感器协同采集用户对终端的3D动作。第一处理器1680根据陀螺仪传感器采集的数据，可以实现如下功能：动作感应(比如根据用户的倾斜操作来改变UI)、拍摄时的图像稳定、游戏控制以及惯性导航。
压力传感器可以设置在终端的侧边框和显示单元1640的下层中的至少之一。当压力传感器设置在终端的侧边框时，可以检测用户对终端的握持信号，由第一处理器1680根据压力传感器采集的握持信号进行左右手识别或快捷操作。当压力传感器设置在显示单元1640的下层时，由第一处理器1680根据用户对显示单元1640的压力操作，实现对UI界面上的可操作性控件进行控制。可操作性控件包括按钮控件、滚动条控件、图标控件、菜单控件中的至少一种。
光学传感器用于采集环境光强度。在一个实施例中，第一处理器1680可以根据光学传感器采集的环境光强度，控制显示单元1640的显示亮度。例如，当环境光强度较高时，调高显示单元1640的显示亮度；当环境光强度较低时，调低显示单元1640的显示亮度。在另一个实施例中，第一处理器1680还可以根据光学传感器采集的环境光强度，动态调整摄像头组件1610的拍摄参数。
在本实施例中，该终端所包括的第一处理器1680可以执行前面实施例的多模态数据处理方法。
本申请实施例提供的用于执行上述多模态数据处理方法的电子设备也可以是服务器，参照图17，图17为本申请实施例提供的服务器的部分结构框图，服务器可因配置或性能不同而产生比较大的差异，可以包括一个或一个以上第二处理器1710和第二存储器1730，一个或一个以上存储应用程序1743或数据1742的存储介质1740(例如一个或一个以上海量存储装置)。其中，第二存储器1730和存储介质1740可以是短暂存储或持久存储。存储在存储介质1740的程序可以包括一个或一个以上模块(图示没标出)，每个模块可以包括对服务器中的一系列指令操作。在一些实施例中，第二处理器1710可以设置为与存储介质1740通信，在服务器上执行存储介质1740中的一系列指令操作。
服务器还可以包括以下至少之一：一个或一个以上第二电源1720，一个或一个以上有线或无线网络接口1750，一个或一个以上输入输出接口1760，一个或一个以上操作系统1741，例如Windows ServerTM，Mac OS XTM，UnixTM，LinuxTM，FreeBSDTM等等。
服务器中的第二处理器1710可以用于执行多模态数据处理方法。
本申请实施例还提供一种计算机可读存储介质，计算机可读存储介质用于存储计算机程序，计算机程序用于执行前述各个实施例的多模态数据处理方法。
本申请实施例还提供了一种计算机程序产品，该计算机程序产品包括计算机程序，该计算机程序存储在计算机可读存介质中。计算机设备的处理器从计算机可读存储介质读取该计算机程序，处理器执行该计算机程序，使得该计算机设备执行实现上述的多模态数据处理方法。
本申请的说明书及上述附图中的术语“第一”、“第二”、“第三”、“第四”等(如果存在)是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。应该理解这样使用的数据在适当情况下可以互换，以便这里描述的本申请实施例，例如能够以除了在这里图示或描述的那些以外的顺序实施。此外，术语“包括”和“具有”以及他们的任何变形，意图在于覆盖不排他的包含，例如，包含了一系列步骤或单元的过程、方法、系统、产品或装置不必限于清楚地列出的那些步骤或单元，而是可包括没有清楚地列出的或对于这些过程、方法、产品或装置固有的其它步骤或单元。
应当理解，在本申请实施例中，“至少一个(项)”是指一个或者多个，“多个”是指两个或两个以上。“二者中的至少一个”，用于描述关联对象的关联关系，表示可以存在三种关系，例如，“A和B中的至少一个”可以表示：只存在A，只存在B以及同时存在A和B三种情况，其中A，B可以是单数或者复数。“以下至少一项(个)”或其类似表达，是指这些项中的任意组合，包括单项(个)或复数项(个)的任意组合。例如，a，b或c中的至少一项(个)，可以表示：a，b，c，“a和b”，“a和c”，“b和c”，或“a和b和c”，其中a，b，c可以是单个，也可以是多个。
应了解，在本申请实施例的描述中，多个(或多项)的含义是两个以上，大于、小于、超过等理解为不包括本数，以上、以下、以内等理解为包括本数。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统，装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请实施例的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机装置(可以是个人计算机，服务器，或者网络装置等)执行本申请各个实施例方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(Read-Only Memory，简称ROM)、随机存取存储器(Random Access Memory，简称RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
还应了解，本申请实施例提供的各种实施方式可以任意进行组合，以实现不同的技术效果。
以上是对本申请的较佳实施进行了具体说明，但本申请并不局限于上述实施方式，熟悉本领域的技术人员在不违背本申请精神的共享条件下还可作出种种等同的变形或替换，这些等同的变形或替换均包括在本申请权利要求所限定的范围内。

Claims (19)

  1. 一种多模态数据处理方法，应用于电子设备，包括：
    获取参考图像以及参考文本，其中，所述参考文本用于描述所述参考图像中的内容；
    提取所述参考图像的参考视觉特征；
    通过多模态大语言模型确定所述参考文本的嵌入、所述参考视觉特征的起始标记的嵌入、所述参考视觉特征的嵌入以及所述参考视觉特征的结束标记的嵌入；
    基于所述多模态大语言模型，将所述参考文本的嵌入、所述起始标记的嵌入、所述参考视觉特征的嵌入以及所述结束标记的嵌入，拼接为目标嵌入序列，基于所述起始标记的嵌入、所述结束标记的嵌入以及所述目标嵌入序列中被滑动窗口选中的嵌入进行注意力处理，输出预测序列；
    基于所述预测序列生成预测图像以及预测文本，其中，所述预测文本用于描述所述预测图像中的内容。
  2. 根据权利要求1所述的多模态数据处理方法，其中，所述预测序列包括预测词元和预测视觉特征，所述基于所述起始标记的嵌入、所述结束标记的嵌入以及所述目标嵌入序列中被滑动窗口选中的嵌入进行注意力处理，输出预测序列，包括：
    基于所述起始标记的嵌入、所述结束标记的嵌入以及所述目标嵌入序列中被滑动窗口选中的嵌入进行注意力处理，生成预测嵌入；
    对属于文本类型的所述预测嵌入进行分类，输出所述预测词元；
    对属于视觉类型的所述预测嵌入进行映射，输出所述预测视觉特征。
  3. 根据权利要求2所述的多模态数据处理方法，其中，所述基于所述起始标记的嵌入、所述结束标记的嵌入以及所述目标嵌入序列中被滑动窗口选中的嵌入进行注意力处理，生成预测嵌入，包括：
    将所述起始标记的嵌入、所述结束标记的嵌入以及所述目标嵌入序列中被滑动窗口选中的嵌入拼接为待处理嵌入序列；
    基于所述待处理嵌入序列构建第一键矩阵和第一价值矩阵；
    基于所述待处理嵌入序列中最后一个位置的嵌入构建第一查询矩阵；
    基于所述第一查询矩阵、所述第一键矩阵和所述第一价值矩阵进行注意力处理，生成预测嵌入。
  4. 根据权利要求3所述的多模态数据处理方法，其中，所述基于所述第一查询矩阵、所述第一键矩阵和所述第一价值矩阵进行注意力处理，生成预测嵌入，包括：
    基于所述第一查询矩阵和所述第一键矩阵确定注意力权重矩阵，其中，所述注意力权重矩阵包括所述待处理嵌入序列中的各个嵌入各自对应的注意力权重；
    将所述起始标记的嵌入或者所述结束标记的嵌入各自对应的所述注意力权重，调整至大于或者等于预设的权重阈值；
    基于调整后的所述注意力权重矩阵与所述第一价值矩阵的乘积生成预测嵌入。
  5. 根据权利要求4所述的多模态数据处理方法，其中，所述基于所述第一查询矩阵和所述第一键矩阵确定注意力权重矩阵，包括：
    基于所述第一查询矩阵和所述第一键矩阵的转置的乘积，确定注意力分数矩阵；
    基于缩放因子对所述注意力分数矩阵进行调整，对调整后的所述注意力分数矩阵进行归一化处理，得到注意力权重矩阵。
  6. 根据权利要求2所述的多模态数据处理方法，其中，所述对属于视觉类型的所述预测嵌入进行映射，得到预测视觉特征，包括：
    根据属于视觉类型的所述预测嵌入构建第二键矩阵和第二价值矩阵；
    获取预先训练好的多个第一可学习向量，根据所述多个第一可学习向量构建第二查询矩阵，其中，所述多个第一可学习向量的总维度与所述参考视觉特征的维度相同；
    基于所述第二查询矩阵、所述第二键矩阵和所述第二价值矩阵进行注意力处理，得到预测视觉特征。
  7. 根据权利要求6所述的多模态数据处理方法，所述基于所述第二查询矩阵、所述第二键矩阵和所述第二价值矩阵进行注意力处理，得到预测视觉特征，包括：
    基于所述第二查询矩阵和所述第二键矩阵确定候选权重矩阵，其中，所述候选权重矩阵包括所述预测序列中的各个嵌入各自对应的注意力权重；
    基于所述候选权重矩阵与所述第二价值矩阵的乘积生成预测视觉特征。
  8. 根据权利要求2所述的多模态数据处理方法，其中，所述基于所述预测序列生成预测图像以及预测文本，包括：
    基于所述预测词元生成预测文本；
    获取随机生成的第一噪声图像，将所述预测视觉特征作为扩散模型的扩散条件，基于所述扩散模型对所述第一噪声图像进行去噪处理，生成预测图像。
  9. 根据权利要求1至8任一项所述的多模态数据处理方法，其中，所述基于所述多模态大语言模型确定由所述参考文本的嵌入、所述起始标记的嵌入、所述参考视觉特征的嵌入以及所述结束标记的嵌入拼接得到的目标嵌入序列，包括：
    基于所述多模态大语言模型，根据所述参考视觉特征构建第三键矩阵和第三价值矩阵；
    获取预先训练好的多个第二可学习向量，根据多个所述第二可学习向量构建第三查询矩阵，其中，所述第二可学习向量维度与所述参考文本的嵌入的维度相同；
    基于所述第三查询矩阵、所述第三键矩阵和所述第三价值矩阵进行注意力处理，得到所述参考视觉特征的嵌入；
    将所述参考文本的嵌入、所述起始标记的嵌入、所述参考视觉特征的嵌入以及所述结束标记的嵌入进行拼接得到目标嵌入序列。
  10. 根据权利要求1至9任意一项所述的多模态数据处理方法，其中，所述将所述参考文本与在两端分别添加了起始标记以及结束标记的所述参考视觉特征输入至多模态大语言模型之前，所述多模态数据处理方法还包括：
    获取第一样本图像、第一样本文本、目标图像以及目标文本，其中，所述第一样本文本用于描述所述第一样本图像中的内容，所述目标文本用于描述所述目标图像中的内容，所述目标图像为与所述第一样本图像关联的图像；
    提取所述第一样本图像的第一视觉特征，将所述第一样本文本与在两端分别添加了所述起始标记以及所述结束标记的所述第一视觉特征输入至所述多模态大语言模型进行推理；
    基于所述目标文本确定所述多模态大语言模型进行推理的第一损失，基于所述目标图像确定所述多模态大语言模型进行推理的第二损失；
    基于所述第一损失和所述第二损失对所述多模态大语言模型进行训练。
  11. 根据权利要求10所述的多模态数据处理方法，其中，所述基于所述第一损失和所述第二损失对所述多模态大语言模型进行训练，包括：
    将所述第一损失和所述第二损失拼接后输入回归模型进行预测，得到所述第一损失和所述第二损失各自对应的损失权重，其中，所述回归模型是与所述多模态大语言模型联合训练的；
    基于所述损失权重对所述第一损失和所述第二损失进行加权求和，得到模型损失，基于所述模型损失对所述多模态大语言模型进行训练。
  12. 根据权利要求10所述的多模态数据处理方法，其中，所述第一视觉特征通过特征提取器提取得到，所述提取所述第一样本图像的第一视觉特征之前，所述多模态数据处理方法还包括：
    获取第二样本图像，通过所述特征提取器提取所述第二样本图像的第二视觉特征；
    获取随机生成的第二噪声图像，将所述第二视觉特征作为所述扩散模型的扩散条件，通过所述扩散模型对所述第二噪声图像进行去噪处理，生成重建图像；
    基于所述重建图像和所述第二样本图像之间的差异确定第三损失，基于所述第三损失对所述特征提取器和所述扩散模型进行联合训练。
  13. 根据权利要求12所述的多模态数据处理方法，其中，所述基于所述第三损失对所述特征提取器和所述扩散模型进行联合训练，包括：
    冻结所述扩散模型的参数，提取所述重建图像的第四视觉特征，基于所述第二视觉特征与所述第四视觉特征确定第五损失，基于所述第五损失更新所述特征提取器的参数，得到训练后的所述特征提取器；
    冻结训练后的所述特征提取器的参数，基于所述第三损失更新所述扩散模型的参数，得到训练后的所述扩散模型；
    基于所述第三损失对训练后的所述特征提取器和训练后的所述扩散模型进行微调。
  14. 根据权利要求12所述的多模态数据处理方法，其中，所述基于所述第一损失和所述第二损失对所述多模态大语言模型进行训练之后，所述多模态数据处理方法还包括：
    获取第三样本图像、第二样本文本和标签图像，其中，所述第二样本文本用于描述所述第三样本图像中的内容，所述标签图像为与所述第三样本图像关联的图像；
    提取所述第三样本图像的第三视觉特征，将所述第二样本文本与在两端分别添加了所述起始标记以及所述结束标记的所述第三视觉特征输入至训练后的所述多模态大语言模型进行推理；
    获取随机生成的第三噪声图像，将训练后的所述多模态大语言模型输出的视觉特征作为所述扩散模型的扩散条件，通过所述扩散模型对所述第三噪声图像进行去噪处理，生成关联图像；
    根据所述关联图像和所述标签图像确定第四损失，基于所述第四损失再次对所述扩散模型进行训练。
  15. 根据权利要求10所述的多模态数据处理方法，其中，获取第一样本图像、第一样本文本、目标图像以及目标文本，包括：
    获取多个连续的视频帧以及各个所述视频帧对应的字幕文本；
    调用文本描述模型分别对各个所述视频帧进行文本生成处理，生成各个所述视频帧对应的描述文本；
    构建用于提示参照所述描述文本进行内容增强的提示指令，将所述提示指令、各个所述字幕文本和对应的所述描述文本拼接后输入目标大语言模型，基于所述目标大语言模型分别对各个所述字幕文本进行内容增强，得到各个所述视频帧对应的增强文本；
    将最后一个所述视频帧确定为目标图像，将其余的所述视频帧确定为第一样本图像，将所述目标图像对应的所述增强文本确定为目标文本，将所述第一样本图像对应的所述增强文本确定为第一样本文本。
  16. 一种多模态数据处理装置，包括：
    获取模块，配置为获取参考图像以及参考文本，其中，所述参考文本用于描述所述参考图像中的内容；
    第一处理模块，配置为提取所述参考图像的参考视觉特征；通过多模态大语言模型确定所述参考文本的嵌入、所述参考视觉特征的起始标记的嵌入、所述参考视觉特征的嵌入以及所述参考视觉特征的结束标记的嵌入；
    第二处理模块，配置为基于所述多模态大语言模型，将所述参考文本的嵌入、所述起始标记的嵌入、所述参考视觉特征的嵌入以及所述结束标记的嵌入，拼接为目标嵌入序列，基于所述起始标记的嵌入、所述结束标记的嵌入以及所述目标嵌入序列中被滑动窗口选中的嵌入进行注意力处理，输出预测序列；
    生成模块，配置为基于所述预测序列生成预测图像以及预测文本，其中，所述预测文本用于描述所述预测图像中的内容。
  17. 一种电子设备，包括存储器和处理器，所述存储器存储有计算机程序，所述处理器执行所述计算机程序时实现权利要求1至15任意一项所述的多模态数据处理方法。
  18. 一种计算机可读存储介质，所述存储介质存储有计算机程序，所述计算机程序被处理器执行时实现权利要求1至15任意一项所述的多模态数据处理方法。
  19. 一种计算机程序产品，包括计算机程序，所述计算机程序被处理器执行时实现权利要求1至15任意一项所述的多模态数据处理方法。
PCT/CN2025/090125 2024-06-11 2025-04-21 多模态数据处理方法、装置、电子设备、计算机可读存储介质及计算机程序产品 Pending WO2025256268A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202410749458.0 2024-06-11
CN202410749458.0A CN121117257A (zh) 2024-06-11 2024-06-11 内容生成方法、装置、电子设备及存储介质

Publications (1)

Publication Number Publication Date
WO2025256268A1 true WO2025256268A1 (zh) 2025-12-18

Family

ID=97940177

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2025/090125 Pending WO2025256268A1 (zh) 2024-06-11 2025-04-21 多模态数据处理方法、装置、电子设备、计算机可读存储介质及计算机程序产品

Country Status (2)

Country Link
CN (1) CN121117257A (zh)
WO (1) WO2025256268A1 (zh)

Cited By (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN121388196A (zh) * 2025-12-19 2026-01-23 江西省科学院能源研究所 基于ai的文本生成方法及系统
CN121404318A (zh) * 2025-12-30 2026-01-27 新紫光集团有限公司 一种自动驾驶、模型微调方法、装置、系统及车辆
CN121458831A (zh) * 2025-12-25 2026-02-03 杭州海康威视数字技术股份有限公司 基于多模态区域语义对齐的扩散生成安全优化方法及设备
CN121480739A (zh) * 2026-01-08 2026-02-06 清华大学 面向大语言模型的多模态推理系统的训练方法、装置、设备及介质
CN121547667A (zh) * 2026-01-16 2026-02-17 乐田智作(湖南)影视技术服务有限公司 一种基于记忆感知协同的直播长视频剪辑方法及装置
CN121614283A (zh) * 2026-02-03 2026-03-06 北京面壁智能科技有限责任公司 多模态端侧模型的异构计算资源动态调度方法及系统
CN121617109A (zh) * 2026-01-29 2026-03-06 苏州元脑智能科技有限公司 预训练语言模型输出词元的控制方法及电子设备
CN121660097A (zh) * 2026-01-13 2026-03-13 西安交通大学 视觉大语言模型幻觉缓解方法及相关装置
CN121747988A (zh) * 2026-02-26 2026-03-27 中南大学 基于多模态大语言模型与链式推理的医疗分析方法及系统
CN121831976A (zh) * 2026-03-16 2026-04-10 港理大(杭州)技术创新研究院有限公司 台风强度预测方法和装置、电子设备及存储介质
CN121859858A (zh) * 2026-03-17 2026-04-14 杭州万像天影影视科技有限公司 基于生成式大模型的影视脚本生成方法及系统
CN121860871A (zh) * 2026-03-18 2026-04-14 大连理工大学 对比参数解耦的多模态图像合成方法
CN121955751A (zh) * 2026-03-30 2026-05-01 安徽国麒科技有限公司 一种基于时序与多模态模型融合的电池寿命预测方法及装置
CN122112663A (zh) * 2026-04-10 2026-05-29 青岛海尔科技有限公司 基于多模态的图像识别方法、存储介质及电子装置
CN122133708A (zh) * 2026-05-08 2026-06-02 齐鲁工业大学(山东省科学院) 基于扩散模型的多模态情感分析方法、系统、设备及介质
CN122172351A (zh) * 2026-05-09 2026-06-09 南京信息工程大学 一种基于大语言模型的跨模态多源数据台风路径预测方法及系统
CN122287708A (zh) * 2026-05-28 2026-06-26 上海燧原科技股份有限公司 一种基于扩散模型的注意力加速方法、装置、设备及介质
CN122414416A (zh) * 2026-06-17 2026-07-17 中国科学技术大学 一种基于gw距离改进mllm模型的视觉推理方法及装置、介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180348890A1 (en) * 2017-06-05 2018-12-06 Facebook, Inc. Systems and methods for multilingual emoji search
CN114898192A (zh) * 2022-05-30 2022-08-12 阿里巴巴（中国）有限公司 模型训练方法、预测方法、设备、存储介质及程序产品
CN115115913A (zh) * 2022-06-02 2022-09-27 北京科技大学 一种数据处理方法、装置、电子设备及存储介质
CN116796287A (zh) * 2023-06-16 2023-09-22 平安科技（深圳）有限公司 图文理解模型的预训练方法、装置、设备及存储介质
CN117171382A (zh) * 2023-07-28 2023-12-05 宁波善德电子集团有限公司 一种基于综合特征和自然语言的车辆视频检索方法
CN117437317A (zh) * 2023-10-25 2024-01-23 腾讯科技（深圳）有限公司 图像生成方法、装置、电子设备、存储介质和程序产品

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180348890A1 (en) * 2017-06-05 2018-12-06 Facebook, Inc. Systems and methods for multilingual emoji search
CN114898192A (zh) * 2022-05-30 2022-08-12 阿里巴巴（中国）有限公司 模型训练方法、预测方法、设备、存储介质及程序产品
CN115115913A (zh) * 2022-06-02 2022-09-27 北京科技大学 一种数据处理方法、装置、电子设备及存储介质
CN116796287A (zh) * 2023-06-16 2023-09-22 平安科技（深圳）有限公司 图文理解模型的预训练方法、装置、设备及存储介质
CN117171382A (zh) * 2023-07-28 2023-12-05 宁波善德电子集团有限公司 一种基于综合特征和自然语言的车辆视频检索方法
CN117437317A (zh) * 2023-10-25 2024-01-23 腾讯科技（深圳）有限公司 图像生成方法、装置、电子设备、存储介质和程序产品

Cited By (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN121388196A (zh) * 2025-12-19 2026-01-23 江西省科学院能源研究所 基于ai的文本生成方法及系统
CN121458831A (zh) * 2025-12-25 2026-02-03 杭州海康威视数字技术股份有限公司 基于多模态区域语义对齐的扩散生成安全优化方法及设备
CN121404318A (zh) * 2025-12-30 2026-01-27 新紫光集团有限公司 一种自动驾驶、模型微调方法、装置、系统及车辆
CN121480739A (zh) * 2026-01-08 2026-02-06 清华大学 面向大语言模型的多模态推理系统的训练方法、装置、设备及介质
CN121660097A (zh) * 2026-01-13 2026-03-13 西安交通大学 视觉大语言模型幻觉缓解方法及相关装置
CN121547667A (zh) * 2026-01-16 2026-02-17 乐田智作(湖南)影视技术服务有限公司 一种基于记忆感知协同的直播长视频剪辑方法及装置
CN121617109B (zh) * 2026-01-29 2026-04-03 苏州元脑智能科技有限公司 预训练语言模型输出词元的控制方法及电子设备
CN121617109A (zh) * 2026-01-29 2026-03-06 苏州元脑智能科技有限公司 预训练语言模型输出词元的控制方法及电子设备
CN121614283A (zh) * 2026-02-03 2026-03-06 北京面壁智能科技有限责任公司 多模态端侧模型的异构计算资源动态调度方法及系统
CN121747988A (zh) * 2026-02-26 2026-03-27 中南大学 基于多模态大语言模型与链式推理的医疗分析方法及系统
CN121831976A (zh) * 2026-03-16 2026-04-10 港理大(杭州)技术创新研究院有限公司 台风强度预测方法和装置、电子设备及存储介质
CN121859858A (zh) * 2026-03-17 2026-04-14 杭州万像天影影视科技有限公司 基于生成式大模型的影视脚本生成方法及系统
CN121860871A (zh) * 2026-03-18 2026-04-14 大连理工大学 对比参数解耦的多模态图像合成方法
CN121955751A (zh) * 2026-03-30 2026-05-01 安徽国麒科技有限公司 一种基于时序与多模态模型融合的电池寿命预测方法及装置
CN122112663A (zh) * 2026-04-10 2026-05-29 青岛海尔科技有限公司 基于多模态的图像识别方法、存储介质及电子装置
CN122133708A (zh) * 2026-05-08 2026-06-02 齐鲁工业大学(山东省科学院) 基于扩散模型的多模态情感分析方法、系统、设备及介质
CN122172351A (zh) * 2026-05-09 2026-06-09 南京信息工程大学 一种基于大语言模型的跨模态多源数据台风路径预测方法及系统
CN122287708A (zh) * 2026-05-28 2026-06-26 上海燧原科技股份有限公司 一种基于扩散模型的注意力加速方法、装置、设备及介质
CN122414416A (zh) * 2026-06-17 2026-07-17 中国科学技术大学 一种基于gw距离改进mllm模型的视觉推理方法及装置、介质

Also Published As

Publication number Publication date
CN121117257A (zh) 2025-12-12

Similar Documents

Publication Publication Date Title
CN121117257A (zh) 内容生成方法、装置、电子设备及存储介质
CN113723166B (zh) 内容识别方法、装置、计算机设备和存储介质
CN110852256A (zh) 时序动作提名的生成方法、装置、设备及存储介质
CN117033609A (zh) 文本视觉问答方法、装置、计算机设备和存储介质
CN112085120B (zh) 多媒体数据的处理方法、装置、电子设备及存储介质
CN119255028B (zh) 音频生成方法、装置、电子设备及存储介质
CN113515669B (zh) 基于人工智能的数据处理方法和相关设备
CN118661202A (zh) 自动的视频和音频注释技术
CN114495916B (zh) 背景音乐的插入时间点确定方法、装置、设备和存储介质
CN115129935B (zh) 视频封面确定方法、装置、设备、存储介质及产品
CN118824282A (zh) 模型训练方法、装置、电子设备及存储介质
CN118644411A (zh) 视频生成方法、装置、电子设备及存储介质
CN112037239A (zh) 基于多层次显式关系选择的文本指导图像分割方法
CN119128200B (zh) 图像转换方法、系统、计算机设备以及存储介质
CN116932788B (zh) 封面图像提取方法、装置、设备及计算机存储介质
CN116975294A (zh) 文本类型确定方法、模型训练方法、装置及电子设备
US12548599B2 (en) Generating contextual transition effects in videos
CN120409657B (zh) 多模态大模型驱动的人物知识图谱构建方法及系统
CN115114477B (zh) 一种视频的信息处理方法、装置、计算机设备和存储介质
CN119646181A (zh) 内容生成方法、装置、电子设备及存储介质
CN117292122B (zh) 一种rgb-d显著性物体检测、语义分割方法及系统
CN114218438B (zh) 视频数据处理方法、装置、电子设备和计算机存储介质
CN110969187B (zh) 一种图谱迁移的语义分析方法
CN117218635A (zh) 字幕识别方法、装置、设备、存储介质及程序产品
CN117036845A (zh) 模型处理方法、装置、设备、存储介质和计算机程序产品

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 25820948

Country of ref document: EP

Kind code of ref document: A1