CN117876940B - Video language task execution and model training method, device, equipment, and medium - Google Patents

Abstract

The present invention discloses a method, device, equipment and medium for executing a video language task and training a model thereof, which are applied to the field of video understanding technology. The method comprises inputting a video sample with a text label, a video parameter to be learned and a frame parameter to be learned into a video language model, a visual language pre-training model extracting visual features and parameter features, a video frame adapter converting visual features into frame visual information that meets the requirements of the visual language pre-training model based on the frame parameters to be learned, and a video adapter extracting video visual information based on the video parameters to be learned; the video language model is iteratively updated according to the loss information between the frame visual information, the video visual information and the text semantic features until the preset model training end condition is met. The present invention can solve the problems of slow convergence of the video language model in the related art and time-consuming and resource-consuming training, and can effectively improve the training efficiency of the video language model and save the computing resources required for model training.

Description

Video language task execution and model training method, device, equipment, and medium

Technical Field

The present invention relates to the field of video understanding technology, and in particular to a method, device, electronic device and readable storage medium for executing a video language task and training a model thereof.

Background technique

The video language model can understand the intrinsic relationship between the visual modality and the language modality, and can be used to perform video language-related tasks, including but not limited to video content understanding and classification tasks, video subtitle translation and generation tasks.

The video language model in the related technology has the problems of weak correlation between visual modality and text modality and different focus ranges of text on video, which leads to slow convergence of video language model and time-consuming and resource-consuming training.

In view of this, improving the training efficiency of video language models is a technical problem that technical personnel in the field need to solve.

Summary of the invention

The present invention provides a method, device, electronic device and readable storage medium for executing a video language task and training a model thereof, which can effectively improve the training efficiency of the video language model and save the computing resources required for model training.

In order to solve the above technical problems, the present invention provides the following technical solutions:

On one hand, the present invention provides a video language model training method, comprising:

Obtain a video sample data set carrying text description labels, pre-set video parameters to be learned, and frame parameters to be learned;

The video samples in the video sample data set, the video parameters to be learned and the frame parameters to be learned are input into the video language model; the video language model includes a target visual language pre-training model, a video frame adapter and a video adapter; the target visual language pre-training model is used to extract visual features and parameter features and input them into the video frame adapter and the video adapter respectively, the video frame adapter is used to convert the visual features into frame visual information that meets the requirements of the target visual language pre-training model, and the video adapter is used to extract video visual information;

Iteratively updating the video language model according to the frame visual information, the video visual information, and the loss information of the text semantic features until a preset model training end condition is met;

The parameter features corresponding to the video parameters to be learned are input into the video adapter, and the parameter features corresponding to the frame parameters to be learned are input into the video frame adapter, so as to obtain text-related visual information using the frame parameters to be learned.

In a first exemplary embodiment, the parameter feature corresponding to the frame parameter to be learned is a frame parameter feature, the text description label includes a video frame description text label, the text semantic feature corresponding to the video frame description text label is a video frame text feature, and the video frame adapter includes a frame input layer, a text encoding layer, a cross-modal fusion layer, a feature enhancement layer, and a frame output layer;

Among them, the frame input layer is used to receive the splicing result of the frame parameter features and the video frame text features; the text encoding layer is used to encode the splicing result based on the current attention mask to obtain the frame parameter encoding features; the cross-modal fusion layer is used to perform cross-modal fusion processing on the frame parameter encoding features and the visual features; the feature enhancement layer is used to perform feature enhancement processing on the fusion result and input the enhanced features to the text encoding layer; the frame output layer is used to output frame visual information.

In a second exemplary implementation, the cross-modal fusion layer is a cross-modal attention mechanism layer, and the cross-modal fusion processing of the frame parameter encoding features and the visual features includes:

The frame parameter encoding features are used as query vectors, and the visual features are used as a set of value vectors and key vectors. The frame parameter encoding features and the visual features are encoded based on a cross-modal attention mechanism to serve as a fusion result.

In a third exemplary embodiment, the feature enhancement layer includes a first feature enhancement layer, an interactive feature extraction layer, and a second feature enhancement layer;

The first feature enhancement layer is used to perform layer normalization processing on the fusion result and obtain a first interactive enhancement feature through a residual connection;

The interactive feature extraction layer is used to extract the first interactive enhancement feature to obtain a second interactive enhancement feature;

The second feature enhancement layer is used to perform layer normalization processing on the second interactive enhancement feature and connect it through a residual connection.

In a fourth exemplary embodiment, the training process of the video frame adapter includes:

Extract the features of the frame visual information corresponding to the current frame to obtain the image frame features corresponding to the current frame image;

Extract the video frame text features corresponding to the current frame to obtain the image frame text features corresponding to the current frame image;

The video frame adapter is iteratively updated according to the loss information between each image frame feature and the corresponding image frame text feature.

In a fifth exemplary implementation, the iterative updating of the video frame adapter according to the loss information between each image frame feature and the corresponding image frame text feature includes:

Determining a frame-text matching loss by predicting whether the image frame features and the image frame text features are a positive match or a negative mismatch using the video frame adapter;

By comparing the similarity between the image frame features and the image frame text features, the frame-text comparison loss is determined;

Masking off some video frame text features, predicting the masked video frame text features through the video frame adapter trained based on the image frame text features corresponding to the remaining video frame text features and each image frame feature, and determining the text generation loss;

A loss function of the video frame adapter is determined according to the frame-text matching loss, the frame-text contrast loss and the text generation loss.

In a sixth exemplary implementation, determining the frame-text comparison loss by comparing the similarity between the image frame feature and the image frame text feature includes:

The positively matched image frame features and image frame text features are taken as a group of positive samples, and the negatively mismatched image frame features and image frame text features are taken as a group of negative samples;

Calculate the positive similarity between the image frame features and the image frame text features in each group of positive samples, and calculate the negative similarity between the image frame features and the image frame text features in each group of negative samples;

A frame-text contrast loss is determined by comparing the positive similarity and the negative similarity.

In a seventh exemplary implementation, determining the frame-text comparison loss by comparing the similarity between the image frame feature and the image frame text feature includes:

The contrast loss function relationship is called to calculate the frame-text contrast loss; the contrast loss function relationship is: ;

where Loss _ITG is _the frame-text contrast loss, _exp represents the exponential function, Zi is the i -th image frame feature, Ti is the image frame text feature that matches the i -th image frame feature, Tj is the _j - th image frame feature that does not match the image frame text feature, N _ITG is the total number of matches between image frame text features and image frame features, θ represents the similarity between image frame features and image frame text features, and τ is the parameter to be optimized.

In an eighth exemplary embodiment, determining the loss function of the video frame adapter according to the frame-text matching loss, the frame-text contrast loss, and the text generation loss includes:

Determine an image frame-image frame text loss according to the frame-text matching loss, the frame-text comparison loss and the text generation loss;

Masking a target image frame of the video sample, predicting the target image frame through a video frame adapter trained based on image frame text features corresponding to the masked video sample and features of each image frame, and determining a video frame mask loss;

A loss function of the video frame adapter is determined according to the image frame-image frame text loss and the video frame mask loss.

In a ninth exemplary implementation, determining the video frame mask loss includes:

The video frame mask loss function relationship is called to calculate the video frame mask loss; the video frame mask loss function relationship is:

;

Among them, Loss _MTF is the video frame mask loss, is the expectation of random distribution within a small batch of video samples, D represents random distribution, V represents image frame features, V _m represents target image frame, O( V _m ) represents target image frame features, /> is the image frame feature of the video sample that is not masked, T represents/> The corresponding image frame text features, /> is the image frame text feature of the target image frame, k is the kth image frame feature that is masked inside the small batch video sample, K is the number of images that are masked inside the small batch video sample, and model represents the prediction result.

In a tenth exemplary embodiment, the parameter features corresponding to the video parameters to be learned are video parameter features, and the video adapter includes a video input layer, a parameter encoder layer, a feature fusion layer, a feature extraction layer, and a video output layer;

Among them, the video input layer is used to receive the joint features of the visual features and the frame visual information; the parameter encoder layer is used to encode the video parameter features to obtain video parameter coding features; the feature fusion layer is used to fuse the video parameter coding features and the joint features; the feature extraction layer is used to extract features from the fusion processing results and transmit the extracted features to the parameter encoder layer; the video output layer is used to output video visual information.

In an eleventh exemplary embodiment, the feature fusion layer includes a first video feature enhancement layer, a cross-modal learning layer, and a second video feature enhancement layer;

The first video feature enhancement layer is used to perform residual connection on the video parameter coding feature and the video parameter feature, and perform layer normalization processing to obtain parameter enhancement features;

The cross-modal learning layer is used to fuse the video parameter encoding feature and the joint feature based on a cross-modal attention mechanism, taking the parameter enhancement feature as a query vector and the joint feature as a set of value vectors and key vectors to obtain a multi-modal fusion feature;

The second video feature enhancement layer performs residual connection on the multimodal fusion features and performs layer normalization processing to obtain a fusion processing result.

In a twelfth exemplary embodiment, the video language model further comprises a docking network layer; the docking network layer comprises a first transformer model, a video feature extraction layer and a joint layer;

Among them, the first converter model is used to fuse the visual features based on the self-attention mechanism to obtain visual fusion features; the video feature extraction layer is used to extract features from the visual fusion features and convert the dimensions of the extracted features into the same dimensions as the input dimensions of the video adapter; the joint layer is used to combine the frame visual information and the output of the video feature extraction layer, and input the joint features into the video adapter.

In a thirteenth exemplary implementation, the text description tag includes a video description text tag, the text semantic feature corresponding to the video description text tag is a video text feature, and the training process of the video adapter includes:

Extracting video features of the video visual information;

Extracting encoded text features corresponding to the video text features;

The video adapter is iteratively updated according to the loss information between the video feature and the encoded text feature.

In a fourteenth exemplary implementation, the loss information between the video feature and the encoded text feature includes:

The video-text loss calculation formula is called to calculate the video-text loss of the video adapter, and the video-text loss calculation formula is:

;

Wherein, Loss _G is the video-text loss, _NG is the total number of matches between the video features and the encoded text features in the current batch, is the i 'th video feature in the current batch,/> is the encoded text feature that matches the i 'th video feature,/> is the j'th encoded text feature that does not match the i'th video feature, θ represents the similarity between the video feature and the encoded text feature, and τ is the parameter to be optimized.

In a fifteenth exemplary implementation, the step of inputting the video sample, the preset video parameters to be learned, and the frame parameters to be learned into the video language model includes:

Performing image sampling processing on the video sample to obtain multiple frames of sample images;

Extracting image features of each frame of sample image using an image encoder of the target visual language pre-training model to obtain visual features;

Extracting text semantic features of the text description label of the video sample using the text encoder of the target visual language pre-trained model;

The text encoder of the target visual language pre-training model is used to extract parameter features corresponding to the video parameters to be learned and the frame parameters to be learned respectively.

In a sixteenth exemplary implementation, the text encoder using the target visual language pre-training model extracts parameter features corresponding to the video parameters to be learned and the frame parameters to be learned, respectively, including:

Using the text encoder of the target visual language pre-training model to randomly initialize the frame parameters to be learned, and using the random initialization results of the frame parameters to be learned as frame parameter features;

The text encoder of the target visual language pre-training model is used to encode the video parameters to be learned based on the current attention mask to obtain video parameter features.

In a seventeenth exemplary implementation, the text description tag includes a video description text tag and a video frame description text tag, and the text encoder using the target visual language pre-trained model to extract text semantic features of the text description tag of the video sample includes:

Extracting video text features of the video description text tag using a text encoder of the target visual language pre-trained model;

Using the text encoder of the target visual language pre-trained model to perform lemma processing on the video frame description text label, and performing word embedding processing on the lemma processing result to obtain video frame text features;

The text encoder of the target visual language pre-trained model is used to encode the video description text label based on the current attention mask to obtain video text features.

In an eighteenth exemplary implementation, the image encoder using the target visual language pre-trained model extracts image features of each frame of sample image to obtain visual features, including:

Divide the current frame image into multiple image blocks with non-overlapping contents;

Convert each graphic block into a one-dimensional representation through linear mapping, and add position encoding information to the corresponding image block;

The image block after linear mapping and position encoding is input into the encoder of the second transformer model, and feature extraction is performed on the output of the encoder of the second transformer model to obtain the visual features of the video sample.

In a nineteenth exemplary implementation, the parameter feature corresponding to the frame parameter to be learned is a frame parameter feature, the text description label includes a video frame description text label and a video description text label, the text semantic feature corresponding to the video frame description text label is a video frame text feature, the text semantic feature corresponding to the video description text label is a video text feature, and the training process of the video language model includes:

Taking the video frame description text label, the frame parameters to be learned, and the video sample data set as input, freezing the image encoder of the target visual language pre-trained model, and using the frame parameters to be learned to obtain visual information corresponding to the video frame description text label, the video frame adapter is trained;

When the video frame adapter training is completed, the video frame description text label, the video frame description text label, the frame parameters to be learned, the video parameters to be learned, and the video sample data set are used as input to train the video adapter.

In a twentieth exemplary implementation, before training the video adapter, the method further includes:

When a learning rate adjustment instruction is received, the current learning rate is updated according to a new learning rate of the learning rate adjustment instruction; the new learning rate is less than the current learning rate.

In a twenty-first exemplary implementation, the training of the video frame adapter includes:

Call the video frame adapter loss function to train the video frame adapter; the video frame adapter loss function is:

;

Among them, Loss _frame represents _the video frame adapter loss function, Loss _ITM is the frame-text matching loss, Loss _ITC is the text generation loss, Loss _ITG is the frame-text contrast loss, Loss _MEF is the video frame mask loss, α0 is the frame-text matching loss coefficient, α1 is the text generation loss coefficient, α2 is the frame-text contrast loss coefficient, _and β is _the video frame mask loss coefficient.

In a twenty-second exemplary implementation, the training of the video frame adapter includes:

Calling a video language loss function to train the video adapter; the video language loss function is:

;

Among them, Loss represents the video language loss function, α is the coefficient of the video frame adapter loss function, Loss _G is the video-text loss, and γ is the coefficient of the video-text loss function.

A second aspect of the present invention provides a video language task execution method, comprising:

Using any of the above video language model training methods, a video language model is trained to obtain a video language model;

Obtaining a video language task to be performed and a corresponding video language task training sample set;

Based on the video language task, fine-tuning the video language model using the video language task training sample set;

The video language task is performed using the fine-tuned video language model.

In a first exemplary implementation, the video language task to be performed is a video content understanding task, and the video language task training sample set is a video sample set of multiple video samples carrying video content labels; based on the video language task, fine-tuning the video language model using the video language task training sample set includes:

Based on the video content understanding task, the video language model is fine-tuned using the video sample set, so as to perform the video content understanding task using the fine-tuned video language model.

A third aspect of the present invention provides a video language model training device, comprising:

A data acquisition module is used to acquire a video sample data set carrying text description labels, pre-set video parameters to be learned, and frame parameters to be learned;

An input data processing module is used to input the video samples in the video sample data set, the video parameters to be learned and the frame parameters to be learned into the video language model; the video language model includes a target visual language pre-training model, a video frame adapter and a video adapter; the target visual language pre-training model is used to extract visual features and parameter features and input them into the video frame adapter and the video adapter respectively, the video frame adapter is used to convert the visual features into frame visual information that meets the requirements of the target visual language pre-training model, and the video adapter is used to extract video visual information; wherein the parameter features corresponding to the video parameters to be learned are input into the video adapter, and the parameter features corresponding to the frame parameters to be learned are input into the video frame adapter, so as to obtain text-related visual information using the frame parameters to be learned;

The model parameter updating module is used to iteratively update the video language model according to the frame visual information, the video visual information and the loss information of the text semantic features until the preset model training end condition is met.

A fourth aspect of the present invention provides a video language task execution device, comprising:

A model training module, used to train a video language model using the video language model training method described in any one of the above items;

A data acquisition module, used to acquire the video language task to be performed and the corresponding video language task sample set;

A model fine-tuning module, used to fine-tune the video language model based on the video language task using the video language task sample set;

The task execution module is used to execute the video language task using the fine-tuned video language model.

The fifth aspect of the present invention also provides an electronic device, comprising a processor, wherein the processor is used to implement the steps of the video language model training method as described in any of the preceding items when executing a computer program stored in a memory.

Finally, the present invention also provides a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the video language model training method as described in any of the preceding items are implemented.

The technical solution provided by the present invention has the advantages of using a video frame adapter to convert the visual features of a video sample into frame visual information that meets the requirements of a visual language pre-training model, using the frame parameters to be learned to learn text-related visual information, assisting the model in establishing associations between different frames and guiding the model to focus on different visual information, and being able to solve the problem of weak correlation between video modality and language modality, and realizing the adaptation of the visual language pre-training model to the video frame. The video adapter can be used to integrate frame text visual information, solve the problem of the focus range of the language modality, use the video parameters to be learned to assist the model in establishing semantic correspondences and understanding the global information of the video on the entire video sequence, integrate the global video information and frame text visual information, and solve the information loss of frame text visual information due to attention bias, thereby making full use of the visual information of the video at different levels such as local details and global semantics, improving the model's visual understanding ability of the video, and improving the representation ability of the video language model, so that the video language model converges quickly during the training process, improving the training efficiency of the video language model, and saving the computing resources required for training. Furthermore, based on the existing visual language pre-training model, a video language model is constructed by adding video frame adapters and video adapters. There is no need to change the original visual language pre-training model structure, no need to redesign all network structures, and no need to retrain the video language model with a large amount of video text data. It is possible to migrate the rich visual representation and powerful cross-modal interaction capabilities of the visual language pre-training model to video language tasks, which not only retains the performance of the original model, but also enhances its scalability and flexibility.

In addition, the present invention also provides a corresponding video language task execution method, implementation device, electronic device and readable storage medium for the video language model training method, which further makes the method more practical. The video language task execution method, device, electronic device and readable storage medium have corresponding advantages.

It is to be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions of the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

FIG1 is a flow chart of a video language model training method provided by the present invention;

FIG2 is a schematic diagram of a structure of a video frame adapter provided by the present invention in an exemplary application scenario;

FIG3 is a schematic diagram of a training process of a video frame adapter in an exemplary application scenario provided by the present invention;

FIG4 is a schematic diagram of a structure of a video adapter provided by the present invention in an exemplary application scenario;

FIG5 is a schematic diagram of a structure of a video adapter and a docking network provided by the present invention in an exemplary application scenario;

FIG6 is a schematic diagram of a training process of a video adapter in an exemplary application scenario provided by the present invention;

FIG7 is a flow chart of a method for executing a video language task provided by the present invention;

FIG8 is a schematic diagram of a hardware framework of a video language task execution method provided by the present invention in an exemplary application scenario;

FIG9 is a schematic diagram of the structure of a video language model provided by the present invention in an exemplary application scenario;

FIG10 is a structural diagram of a specific implementation of the video language model training device provided by the present invention;

FIG11 is a structural diagram of a specific implementation of the video language task execution device provided by the present invention;

FIG. 12 is a structural diagram of a specific implementation of an electronic device provided by the present invention.

Detailed ways

In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Among them, the terms "first", "second", "third", "fourth", etc. in the specification and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations of the two are intended to cover non-exclusive inclusions. The term "exemplary" means "used as an example, embodiment or illustrative". Any embodiment described here as "exemplary" is not necessarily interpreted as being superior or better than other embodiments.

The video language model is a cross-modal model that can deeply understand the intrinsic relationship between the visual modality and the language modality. It is widely used in various application scenarios related to video language. For example, in the application scenarios of searching and annotating videos, the video language model can help users quickly locate and understand the video content. With the ability of the video language model to accurately analyze user interests and preferences, it can also promote the development of video summary generation and personalized recommendation systems. In addition, the video language model also understands video content such as actual scenes, objects, actions, and emotions, which is conducive to the development of related fields of natural language description, such as the generation of titles, subtitles, and storylines. For the application scenarios of video question-answering and dialogue, the video language model enables computers to understand and accurately respond to video-related questions.

However, the current video language model has the problem of language modality focus range, that is, the range of videos it focuses on is different. For example, the text description may focus on certain frames, a certain video segment, or the entire video. This leads to slow training convergence of the video language model, and the final execution results of the video language task do not meet the needs of users. In addition, the video review text description of the current video language model is short. The sparsity of the video text description makes it more difficult to model the association between video features and text descriptions. The training convergence of the video language model is slow, and the performance of the video language model is poor. In the current video language model training process, there is an inconsistency between the visual information in the sample video and the semantic information in the description. The scene in the video may not be able to establish a clear correspondence with the specific semantic concepts in the description, resulting in a weakened correlation between the description and the video. The training convergence of the video language model is slow, and the performance of the video language model cannot meet the needs of users. Furthermore, video language tasks usually require the training of complex deep learning models. These models have a large number of parameters and hierarchical structures, which increases the demand for computing and storage resources required for the model training process. Moreover, within a certain range, the structure of the model and the size of the training samples are positively correlated with the model performance. In order to obtain a video language model with good performance, the training and tuning of the model requires multiple iterations and experiments, which further increases the training time and the consumption of computing resources.

It can be seen that the video language model in the related art has the problem that the video and language are weakly correlated, and the text has different focus ranges on the video, which leads to slow convergence of the video language model and time-consuming and resource-consuming training. In addition, the video language pre-training model in the related art requires redesigning all network structures, and then retraining the model with a large amount of video text data. First, the video features and text features are extracted, and then the visual information and text information are mapped to a unified semantic space through contrastive learning or Transform (converter model) based on the attention mechanism. This kind of training model from scratch is time-consuming and labor-intensive, usually requires a lot of computing resources, and has poor flexibility and scalability.

In view of this, in order to solve the problems of time-consuming and resource-consuming video language pre-training, weak correlation between video and language, and language modality focusing on the range of video modality in the related art, the present invention is based on the visual language pre-training model in the related art. Without changing the original model parameters, by adding a video frame adapter, a video adapter, all video frames share a frame parameter to be learned, and a small number of video parameters to be learned, the video language pre-training model adaptation can be completed, and the migration of the rich visual representation of the visual language pre-training model and the powerful cross-modal interaction ability can be adapted to the video language task, that is, the original model performance is retained, while its scalability and flexibility are enhanced, and the training efficiency of the video language model can be effectively improved, and the computing resources required for model training can be saved. After introducing the technical solution of the present invention, various non-limiting embodiments of the present invention are described in detail below. In order to better illustrate the present invention, numerous specific details are given in the specific embodiments below. Those skilled in the art should understand that the present invention can also be implemented without these specific details. In some other examples, the methods and means well known to those skilled in the art are not described in detail to highlight the main purpose of the present invention.

First, please refer to FIG. 1, which is a flow chart of a video language model training method provided by this embodiment. This embodiment may include the following contents:

S101: Obtain a video sample data set carrying text description labels, pre-set video parameters to be learned, and frame parameters to be learned.

In this embodiment, the video sample data set is a training sample data set used for training the video language model, which may include a large number of video samples covering a variety of rich scenes, and the number of video samples can be flexibly determined according to actual needs, which does not affect the implementation of the present invention. Each video sample has a text description label, which is used to describe the video sample, which is text data. The video samples in the video sample data set are pre-labeled with text descriptions, and the labeled information is the text description label.

Among them, the frame parameter to be learned is a variable or a set of variables that enables the video frame to learn text-related visual information during the video language model training process. It can be flexibly determined according to the specific actual application scenario. The frame parameter to be learned is a learnable parameter of the video frame. All video frames share a frame parameter to be learned. The shared frame parameter to be learned helps the video language model to associate and understand the correlation between different video frames. In a video, different frames may contain the same person, the same scene, or related objects. By sharing the frame parameter to be learned, the model can be assisted in establishing associations between different frames. For example, it assists other frames in finding related people or objects, thereby improving the accuracy and coherence of the video language model in multi-frame video understanding and analysis. In addition, since the visual information corresponding to the text can be obtained through the frame parameter to be learned, different frames may obtain different visual information, which is equivalent to providing different perspectives and information. By sharing the frame parameter to be learned between different frames, the video language model can be guided to pay attention to this information. At the same time, these complementary information can be comprehensively utilized to improve the video language model's understanding and representation capabilities for video language tasks. The video parameters to be learned are also a variable or a set of variables that enable the video to learn and obtain visual information about the text during the video language model training process. They can be flexibly determined according to specific practical application scenarios. The video parameters to be learned are learnable parameters of the video, which can assist the video language model in establishing semantic correspondences and understanding the global information of the video over the entire video sequence. Since the video parameters to be learned enable the video language model to establish semantic correspondences and understand the global information of the video over the entire video sequence, and the frame parameters to be learned enable the video language model to perform local alignment and understanding at the video frame level. By combining the video parameters to be learned and the frame parameters to be learned, fine-grained alignment across time scales is achieved, and the visual information of the video at different levels such as local details and global semantics is fully utilized, so as to improve the video language model's ability to understand and express the global and local features of the video or video sample to be processed, improve the video language model's visual understanding ability of the video or video sample to be processed, and improve the representation ability of the video language model.

S102: Inputting video samples, to-be-learned video parameters, and to-be-learned frame parameters in the video sample data set into a video language model.

In this step, the video language model is a pre-trained model framework pre-built based on the target visual language pre-trained model, the video frame adapter and the video adapter, that is, the model structure of the video language model includes the target visual language pre-trained model, the video frame adapter and the video adapter. The target visual language pre-trained model is used to extract visual features and parameter features and input them into the video frame adapter and the video adapter respectively. The target visual language pre-trained model is a pre-trained model that can simultaneously understand and fuse visual and language information to achieve cross-modal vision and language. The target visual language pre-trained model can learn the relationship between visual information and language information by pre-training on large-scale image-text pair data, so that the model can simultaneously process and understand image and text information, and show good performance and generalization ability in multiple visual and language tasks.

The target visual language pre-training model of this embodiment may adopt any VLP (Vision-Language Pretraining) model in the related art, including but not limited to ViLBERT (Vision and Language Bert (Bidirectional Encoder Representations from Transformers), Visual Language Bert) model, LXMERT (Language-Visual Multimodal Bert, cross-modal Bert framework for visual and language understanding), Video BERT model, Visual BERT model, UniVL (Unified Video and Language Pre-Training Model, video language pre-training) model, UNITER (Universal Image-Text Representation, universal image text representation), CLIP (Contrastive Language-Image Pretraining) model, OSCAR (Object-Semantics Aligned Pretraining, object semantics alignment pre-training) model, MOCA (Montreal Cognitive Assessment, Montreal cognitive assessment) model. Among them, the VideoBERT model is a joint model that can be used for video and language representation learning. Through the pre-training process, information can be extracted and understood from videos and related texts. This type of model allows the model to understand visual concepts in videos and how these concepts are associated with natural language descriptions. The UniVL model is a unified video and language pre-training model for multimodal understanding and generation. It achieves a unified representation of video and language by jointly optimizing understanding and generation tasks. This approach enables the model to understand and generate language descriptions related to videos. The VisualBERT model is a simple and efficient baseline model for video and language learning. It combines visual and language information and learns multimodal representations through pre-training and fine-tuning tasks. The MOCA model learns video and text representations through a memory-enhanced contrastive learning method. It constructs positive and negative pairs between videos and texts for self-supervised training and uses a memory mechanism to enhance the effect of contrastive learning. The ViLBERT model learns the relationship between images and texts by jointly training image encoders and text encoders, achieving cross-modal understanding and reasoning of images and texts. The UNITER model is pre-trained on large-scale image-text pair data, learns visual and language representations, and has the ability to understand and generate multimodal information. The CLIP model is a visual language pre-training model that is pre-trained through contrastive learning. It trains the model by learning the similarity between images and texts, giving it a strong ability to match images and texts, and can judge the relevance of images and texts. The OSCAR model is a VLP model that aims at object and semantic alignment. It learns visual and language representations by jointly training visual and language encoders, and can achieve alignment and association between images and texts.

In this step, after selecting a corresponding number of video samples from the video sample data set according to the pre-set training parameters and inputting them into the video language model, the image encoder of the target visual language pre-training model is used to extract the visual features of the video samples, and the text encoder of the target visual language pre-training model is used to extract the text semantic features of the text description labels of the video samples, and extract the parameter features corresponding to the video parameters to be learned and the frame parameters to be learned. The image encoder of the target visual language pre-training model transmits the extracted visual features to the video frame adapter and the video adapter respectively, and inputs the parameter features corresponding to the video parameters to be learned into the video adapter, and the text encoder of the target visual language pre-training model inputs the parameter features corresponding to the frame parameters to be learned into the video frame adapter.

In this embodiment, since the image and text of the target visual language pre-training model often have a weak correlation, and this weak correlation is more serious in the video language data, the target visual language pre-training model can extract rich semantic information image features, but these image features have strong semantic consistency and generalization performance, but still cannot meet the adaptation of video frames, so they need to be converted into frame features that meet the requirements of the video language pre-training model through a video frame adapter. In other words, the video frame adapter obtains text-related visual information based on the received visual features by forcing the frame parameters to be learned, and converts the visual features into frame visual information that meets the requirements of the target visual language pre-training model. It can be understood that the frame text visual information extracted by the video frame adapter is fine-grained information at the frame level, lacking understanding of the entire video. On this basis, the video adapter combines the visual features extracted by the image encoder of the target visual language pre-training model, and learns the visual information of the video text related to the video parameters to be learned based on the video adapter. In other words, the video adapter receives the visual features, learns the text information of the video through the video parameters to be learned, and assists the model to establish a semantic correspondence relationship for the entire video sequence and understand the global information of the video, and extracts the video visual information.

S103: Iteratively update the video language model according to the loss information of the frame visual information, the video visual information and the text semantic features until the preset model training end condition is met.

After the frame visual information and video visual information corresponding to the video sample are obtained in the previous step, they are compared with the text semantic features of the text description label corresponding to the video sample, and the model parameters of the video language model are updated by continuously narrowing the gap between the two. For example, the batch random gradient descent method can be used to train the video language model until the preset model training end condition is met. The preset model training end condition can be, for example, the number of iterations reaching a preset value, or the video language model converges, or the accuracy of the video language model reaches a preset accuracy threshold, which does not affect the implementation of this application. Before the gradient update iteration, the model needs to initialize the gradient descent algorithm, set epoch (training cycle), batch_size (batch size), weight update cycle t, iteration (number of iterations). For example, the total number of video samples contained in the video sample data set can be 60,000, and the video language model is trained for at least 100 training cycles. A training cycle refers to the non-repetitive use of all training samples in the training set to update the model parameters of the neural network. Each time a batch of data is taken to update the model parameters of the video language model to complete a training process. In the gradient update iteration process, 500 video samples are used for each iteration update. These 500 video samples are called a batch of data, that is, batch_size samples. The number of iterations, iteration, refers to the number of times batch_size samples are used for training. The number of iterations to complete an epoch, iteration = 60000/500 = 120. The weight update cycle refers to updating the weight every t iterations during video language model training. When the preset model training end condition is reached, the video language model is a trained video language model, which can make full use of the visual information of the video at different levels, and effectively improve the understanding and inference ability of the video content by jointly learning the representation of video and language, thereby realizing the transfer of the capabilities of the visual language pre-training model to the video language pre-training task, which not only retains the performance of the original model, but also enhances its scalability and flexibility, making it suitable for a wider range of multimodal application fields.

In the technical solution provided in this embodiment, a video frame adapter is used to convert the visual features of a video sample into frame visual information that meets the requirements of a visual language pre-training model, and the frame parameters to be learned are used to learn text-related visual information, assist the model in establishing associations between different frames, and guide the model to focus on different visual information, which can solve the problem of weak correlation between video modality and language modality, and achieve the adaptation of the visual language pre-training model to the video frame. The video adapter can be used to integrate frame text visual information, solve the problem of the focus range of the language modality, and use the video parameters to be learned to assist the model in establishing semantic correspondences and understanding the global information of the video on the entire video sequence, integrate the global video information and frame text visual information, and solve the information loss of frame text visual information due to attention bias, thereby making full use of the visual information of the video at different levels such as local details and global semantics, improving the model's visual understanding ability of the video, and improving the representation ability of the video language model, so that the video language model converges quickly during the training process, improving the training efficiency of the video language model, and saving the computing resources required for training. Furthermore, based on the existing visual language pre-training model, a video language model is constructed by adding video frame adapters and video adapters. There is no need to change the original visual language pre-training model structure, no need to redesign all network structures, and no need to retrain the video language model with a large amount of video text data. It is possible to migrate the rich visual representation and powerful cross-modal interaction capabilities of the visual language pre-training model to video language tasks, which not only retains the performance of the original model, but also enhances its scalability and flexibility.

In the above embodiment, there is no limitation on the structure of the video frame adapter. Based on the above embodiment, as shown in FIG2 , the present invention further provides an exemplary structure of the video frame adapter, which may include the following contents:

In this embodiment, the video frame adapter may include a frame input layer, a text encoding layer, a cross-modal fusion layer, a feature enhancement layer, and a frame output layer. Among them, the frame input layer is used to receive the splicing result of the frame parameter feature and the video frame text feature; the text encoding layer has a built-in text encoder, which is used to encode the splicing result based on the current attention mask to obtain the frame parameter encoding feature and the frame text encoding feature; the cross-modal fusion layer is used to perform cross-modal fusion processing on the frame parameter encoding feature and the visual feature; the feature enhancement layer is used to perform feature enhancement processing on the fusion result and input the enhanced feature to the text encoding layer, repeat the above process multiple times until the first preset number of repetitions is reached, such as M1 times, M1 can be 200, to obtain the frame visual information; the frame output layer is used to output the frame visual information.

Among them, for the convenience of representation, the parameter features corresponding to the frame parameters to be learned can be defined as frame parameter features, and the text description label includes a video frame description text label and a video description text label. The video description text label is the text data corresponding to the entire video, and the video frame description text label is the text data corresponding to the current frame image. For example, the video description text label is that the indicator light on the server panel is flashing, and the video frame description text label is that the indicator light on the server panel is lit and emits green light. For the convenience of description, the text semantic features corresponding to the video frame description text label can be defined as video frame text features. The frame parameter feature is the feature output by the text encoder of the target visual language pre-training model after encoding the frame parameters to be learned, and the video frame text feature is the feature after the text encoder of the target visual language pre-training model encodes the video frame description text label. The frame parameters to be learned and the video frame description text label can be encoded accordingly according to the model type adopted by the target visual language pre-training model, and the present invention does not impose any restrictions on it. The frame input layer will pre-set the attention mask, and the value of the attention mask can correspond to which input feature is encoded. For example, the attention mask For example, it can be expressed as , where /> is the mask of the i-th frame parameter to be learned,/> The mask of the i-th word. The attention mask gives different types of mask values, representing the encoding of different types of data. For example, when only the learning frame parameter encoding is treated, the video frame description text label is masked out, and the setting can be/> is 1, if the input contains a video frame description text label, let/> If the video frame description text label is not input, it can be left unchanged. When only encoding the video frame description text label, mask out the frame parameters to be learned, and set /> If the input contains frame parameters to be learned, you can set it to 1. is 0. If there is no frame parameter to be learned in the input, it can be left unset. If the frame parameters to be learned and the video frame description text labels are encoded at the same time, all values in the attention mask can be set to 1, which indicates that the two modalities are encoded simultaneously. The frame text encoding feature is the feature obtained after the video frame text feature is encoded by the text encoder of the video frame adapter, and the frame parameter encoding feature is the feature obtained after the frame parameter feature is encoded by the text encoder of the video frame adapter. The text encoder built into the text encoding layer can be the same type as the text encoder of the target visual language pre-training model, or it can be different, which does not affect the implementation of the present invention. The cross-modal fusion layer can use any mechanism that can realize cross-modal data fusion for data fusion, forcing the frame parameters to be learned to extract text-related visual features, and at the same time, it is a bridge to promote the information interaction and integration of the two modalities. The cross-modal fusion layer can be built-in with any method that can extract the intrinsic relationship between different modal data and fuse different modal data, such as any attention mechanism or cross-modal attention mechanism. Taking the cross-modal attention mechanism as an example, the cross-modal fusion layer can be a cross-modal attention mechanism layer, and the frame parameter encoding features can be used as query vectors, and the visual features can be used as a set of value vectors and key vectors. The frame parameter encoding features and visual features are encoded based on the cross-modal attention mechanism, and the encoding results are used as fusion results. In order to further improve the accuracy of fusion feature extraction and accelerate the convergence speed of the video frame adapter, the model convergence can be accelerated by layer normalization, and the generalization ability can be further improved by the residual structure. Correspondingly, the feature enhancement layer may include a first feature enhancement layer, an interactive feature extraction layer, and a second feature enhancement layer; the first feature enhancement layer is used to perform layer normalization on the fusion result, and obtain the first interactive enhancement feature through residual connection; the interactive feature extraction layer is used to extract the first interactive enhancement feature to obtain the second interactive enhancement feature; the second feature enhancement layer is used to perform layer normalization on the second interactive enhancement feature and obtain the second interactive enhancement feature through residual connection. The interactive feature extraction layer can be any model structure that can extract deep features, such as a feedforward neural network or a fully connected neural network or a video feature extraction layer. The feedforward neural network maps the data to a high-dimensional space and then to a low-dimensional space through linear transformation, thereby extracting deeper features. For example, the visual feature corresponding to the current image frame is / > , encoding features of frame parameters based on cross-modal attention mechanism/> The result of encoding with visual features can be expressed as/> , , after the layer normalization of the first feature enhancement layer and the residual connection, the first interactive enhancement feature is obtained/> , where / > is the residual coefficient, the interactive feature extraction layer is a feed-forward neural network (also known as Feed forward), which performs a Feed forward operation on the first interactive enhancement feature to obtain the second interactive enhancement feature/> . Again through layer normalization and residual connection, we get /> , where/> is the residual coefficient. The obtained feat ₂ is input into the text encoding layer, and the first attention mask representation method is adopted. The above steps are performed in sequence, and the first preset number of repetitions is repeated to obtain the final frame feature. (i.e., frame visual information), whose dimension is the same as the input /> same.

In order to make the encoding process more clear to the technicians in the relevant field, this embodiment takes the target visual language pre-trained model as CLIP as an example, and the corresponding text encoder is CLIP text Encoder (also known as CLIP text encoder). Exemplarily, the text encoder of the target visual language pre-trained model can be used to randomly initialize the frame parameters to be learned, and the randomly initialized results of the frame parameters to be learned are used as frame parameter features; the text encoder of the target visual language pre-trained model is used to tokenize the video frame description text label, and the tokenization result is word embedded to obtain the video frame text feature. For example, the frame parameters to be learned are randomly initialized (such as can be defined as frame learnablecontext), and the result can be recorded as , where /> is a set of real numbers, C is the embedding length, and D is the embedding dimension. The dimension of D is equal to the dimension of the video frame text feature after tokenization. When the learning frame parameters are initialized, C and D can use random values with a mean of 0 and a variance of 1. The result of tokenization of the video frame description text label can be recorded as , text is the text label describing the video frame, /> For any BERT model, the word_embedding process can be performed on the lemma result to obtain the video frame text features. The word_embedding result can be expressed as/> . word_embedding is the embedding corresponding to the video frame adapter. Taking the target visual language pre-training model as CLIP as an example, the corresponding text encoder is CLIP text Encoder (also known as CLIP text encoder), and the output result corresponds to/> 、/> . The random initialization result flc of the frame parameters before learning and the text features of the video frame/> The splicing result is used as the input of the frame input layer of the video frame adapter. ,in, is the frame parameter feature corresponding to the i-th frame parameter to be learned,/> is the embedding of the i-th word, i.e., the text feature of the video frame. The text encoder built into the text encoding layer of the video frame adapter is used to encode the random initialization result flc and the video frame text feature. The encoding result can be recorded as/> , , where the first c data are the frame parameter encoding features corresponding to the frame parameter features, which can be used Indicates that the last m data are the encoding of the video frame text features, that is, the frame text encoding features, which can be / > In this embodiment, both the frame parameters to be learned and the video frame description text labels need to be encoded, so the attention mask here adopts the third method mentioned above.

As can be seen from the above, this embodiment transfers and adapts the visual information learned by the target visual language pre-training model through the video frame adapter, completes the frame feature extraction of the video language pre-training model, and solves the problem of weak correlation between video and language at the frame level. By setting layer normalization processing, the convergence speed of the video frame adapter can be accelerated, and the generalization ability can be further improved through the residual structure, which is conducive to improving the training efficiency and performance of the entire video language model.

Based on the model structure of the video frame adapter determined in any of the above embodiments, this embodiment also requires training the video frame adapter. The training process of the video frame adapter may include the following contents:

Extract the features of the frame visual information corresponding to the current frame to obtain the image frame features corresponding to the current frame image; extract the features of the video frame text features corresponding to the current frame to obtain the image frame text features corresponding to the current frame image; iteratively update the video frame adapter according to the loss information between each image frame feature and the corresponding image frame text feature.

In this embodiment, as shown in FIG3 , a network structure connected to a video frame adapter can be used to extract video frame text features and frame visual information features. A fully connected layer can be used to obtain the features of the frame visual information corresponding to the current frame, which can be defined as the image frame features ,/> , the input dimension of the fully connected layer is The dimension of the output dimension is the same as the image frame text feature. The feedforward neural network can be used to extract the features of the video frame text feature corresponding to the current frame, which can be defined as the image frame text feature/> Since the input layer of the video frame adapter is the concatenation result of the frame parameter features and the video frame text features, the image frame text features corresponding to the current frame image can also directly extract the features corresponding to the frame text encoding features corresponding to the current frame. Correspondingly, , the fully connected layer input dimension here is/> Dimension, the output dimension is the same as the image frame features .

The above embodiment does not impose any limitation on how to iteratively update the video frame adapter according to the loss information between each image frame feature and the corresponding image frame text feature, that is, it does not impose any limitation on the loss function adopted by the video frame adapter. Of course, as a simple implementation, the loss function of the video frame adapter can be directly based on the mean square error or cross entropy error or other related technologies. In order to improve the performance of the video frame adapter, this embodiment also provides a method for determining the loss function of the video frame adapter, which may include the following contents:

The frame-to-text matching loss is determined by using a video frame adapter to predict whether the image frame features and the image frame text features are a positive match or a negative mismatch; the frame-to-text comparison loss is determined by comparing the similarity between the image frame features and the image frame text features; some video frame text features are masked off, and the masked video frame text features are predicted by using a video frame adapter trained based on the image frame text features corresponding to the remaining video frame text features and each image frame feature to determine the text generation loss; the loss function of the video frame adapter is determined based on the frame-to-text matching loss, the frame-to-text comparison loss and the text generation loss.

Among them, the frame-text matching loss is the loss of whether the video frame matches the text. The purpose is to learn the binary classification task of fine-grained alignment between video frames and text representations. The video adapter is required to predict whether an image-text pair is a positive match or a negative mismatch, that is, , where V and T are the video frame representation and the text representation respectively, matched means matching the video frame representation and the text representation, i.e. positive matching, unmatched means mismatching the video frame representation and the text representation, i.e. negative mismatching, and the frame-text contrast loss is the contrast between the video frame and the text, and its purpose is to achieve their alignment by maximizing the mutual information between the video frame representation and the text representation. Exemplarily, the calculation process of the frame-text contrast loss may include: taking the positively matched image frame features and the image frame text features as a group of positive samples, and taking the negatively mismatched image frame features and the image frame text features as a group of negative samples; calculating the positive similarity between the image frame features and the image frame text features in each group of positive samples, and calculating the negative similarity between the image frame features and the image frame text features in each group of negative samples; and determining the frame-text contrast loss by comparing the positive similarity and the negative similarity. In other words, the loss is achieved by comparing the video frame-text similarity of the positive sample pair with the video frame-text similarity of the negative sample. In this embodiment, the visual information related to the text can be obtained through the frame parameters to be learned, and then the similarity is calculated with the text representation. The text generation loss is the text generation loss based on the video frame. By masking off part of the text, the masked part of the text is predicted by the image frame and the remaining text, and then the final loss can be obtained by cross entropy. In order to further improve the joint understanding ability of the image frame and the text, based on the above embodiment, the video frame mask loss can also be added in the process of calculating the loss function of the video adapter. Exemplarily, a certain frame feature in the video sample can be masked first, and the masked video frame feature can be predicted by other frames and frame texts, and the corresponding video frame mask loss can be obtained by comparing the gap between the real and the predicted. Based on this, the calculation process of the loss function of the video frame adapter may include: determining the image frame-image frame text loss according to the frame-text matching loss, the frame-text comparison loss and the text generation loss; masking the target image frame of the video sample, predicting the target image frame through the video frame adapter trained based on the image frame text feature corresponding to the masked video sample and each image frame feature, and determining the video frame mask loss; determining the loss function of the video frame adapter according to the image frame-image frame text loss and the video frame mask loss.

In order to improve the model training efficiency, the contrast loss function relationship and the video frame mask loss function relationship can be stored locally in advance, and the frame-text contrast loss can be calculated by calling the contrast loss function relationship, and the video frame mask loss function relationship can be called to calculate the video frame mask loss. The contrast loss function relationship can be expressed as:

;

The video frame mask loss function relationship can be expressed as:

;

Where, Loss _ITG is the frame-text contrast loss, exp represents the exponential function, Zi is the i-th image frame feature, Ti _is the image _frame text feature that matches the i -th image frame feature, Tj is the _j - th image frame feature that does not match the image frame text feature, N _ITG is the total number of matches between the image frame text feature and the image frame feature, θ represents the similarity between the image frame feature and the image frame text feature, and τ is the parameter to be optimized. Loss _MTF is the video frame mask loss,

is the expectation of random distribution within a small batch of video samples, D represents random distribution, V represents image frame features, V _m represents target image frame, O( V _m ) represents target image frame features, /> is the image frame feature of the video sample that is not masked, T represents/> The corresponding image frame text features, /> is the image frame text feature of the target image frame, k is the kth image frame feature that is masked inside the small batch video sample, K is the number of images that are masked inside the small batch video sample, and model represents the prediction result.

As can be seen from the above, the image frame-image frame text loss of this embodiment completes the alignment and learning between modalities from three different levels, which helps the video adapter learn the fine-grained correspondence between image frames and texts, improves the performance of semantic understanding and cross-modal tasks, and enhances the representation ability of the video language model on multimodal data. Further predicting the masked image frame through other frames, other frames corresponding to text, and mask frames corresponding to text helps the video adapter learn richer modal representations, thereby improving the ability to jointly understand image frames and texts.

The above embodiment does not impose any limitation on the structure of the video adapter. Based on the above embodiment, as shown in FIG4 , the present invention further provides an exemplary structure of the video adapter, which may include the following contents:

The video adapter includes a video input layer, a parameter encoder layer, a feature fusion layer, a feature extraction layer and a video output layer; wherein the video input layer is used to receive the joint features of the visual features and the frame visual information; the parameter encoder layer is used to encode the video parameter features to obtain the video parameter encoding features; the feature fusion layer is used to fuse the video parameter encoding features and the joint features; the feature extraction layer is used to extract features from the fusion processing results and transmit the extracted features to the parameter encoder layer; repeat multiple times until reaching a second preset number of repetitions, such as M2 times, M2 can be 300, to obtain video visual information, and the video output layer outputs the video visual information.

In this embodiment, for the convenience of representation, the parameter features corresponding to the video parameters to be learned can be defined as video parameter features, and the text semantic features corresponding to the video description text labels can be defined as video text features. The video parameter features are the features output by the text encoder of the target visual language pre-training model after encoding the video parameters to be learned, and the video text features are the features after the text encoder of the target visual language pre-training model encodes the video description text labels. The video parameters to be learned and the video description text labels can be encoded accordingly according to the model type adopted by the target visual language pre-training model, and the present invention does not impose any restrictions on them. Similarly, the text encoder of the target visual language pre-training model will also pre-set an attention mask, and the value of the attention mask can correspond to which input feature is encoded. Correspondingly, the text encoder of the target visual language pre-training model can be used to encode the video parameters to be learned based on the current attention mask to obtain the video parameter features. The text encoder of the target visual language pre-training model is used to extract the video text features of the video description text labels. Taking the target visual language pre-training model as CLIP as an example, the video parameters to be learned and the video text description labels are encoded by CLIP Text Encoder, and the encoding result can be recorded as ,Correspondingly, the input of CLIP Text Encoder ,/> is the video parameter to be learned,/> Describes the text label for the video; output of CLIPText Encoder/> , where the encoding of the first c video parameters to be learned is also the video parameter feature, recorded as/> , the last m are video text features, denoted as/> ,Here the attention mask uses the third method mentioned above to set the corresponding value.

Among them, the parameter encoder layer can use any self-attention mechanism to analyze the video parameter features. Encoding, accordingly, the parameter encoder layer can be represented as a self-attention mechanism layer, as shown in Figure 5, the video parameter encoding feature is used to represent the output of the parameter encoder layer, which can be defined as/> ,/> , self_attention represents any self-attention mechanism, such as MultiHeadAttention, the input dimension and output dimension are the same, and header can be set to 6. The feature fusion layer is used to learn the relevant visual information of the input video parameter encoding features, and any attention mechanism can be used to learn, such as self-attention mechanism, cross attention mechanism, which does not affect the implementation of the present invention. In order to further improve the accuracy of fusion feature extraction and accelerate the convergence speed of the video adapter, the model convergence can be accelerated by layer normalization, and the generalization ability can be further improved by the residual structure. Accordingly, the feature fusion layer can adopt a cross-attention mechanism to learn the relevant visual information of the video parameter coding features. Accordingly, the feature fusion layer can adopt a cross-attention mechanism layer, which may include a first video feature enhancement layer, a cross-modal learning layer and a second video feature enhancement layer; wherein the first video feature enhancement layer is used to perform residual connection on the video parameter coding features and the video parameter features, and perform layer normalization processing to obtain parameter enhancement features; the cross-modal learning layer is used to perform fusion processing on the video parameter coding features and the joint features based on the cross-modal attention mechanism, with the parameter enhancement features as the query vector and the joint features as a set of value vectors and key vectors to obtain multi-modal fusion features; the second video feature enhancement layer performs residual connection on the multi-modal fusion features, and performs layer normalization processing to obtain a fusion processing result. The feature extraction layer can be any model structure that can extract deep features, such as a feedforward neural network or a fully connected neural network layer or a video feature extraction layer. The feedforward neural network maps the data to a high-dimensional space and then to a low-dimensional space through linear transformation, thereby extracting deeper features. For example, based on/> Perform residual connection and layer normalization, β ₀ is the residual coefficient, and learn the visual information related to the video learning parameters through the cross-modal cross attention mechanism, that is, /> . By /> Perform residual connection again and do layer normalization,/> is the residual coefficient. Feed forward the feature, that is

. Will/> As the input of the corresponding step of the parameter encoder layer, repeat the above process M2 times to obtain the video learnable parameters to obtain the video visual information related to the text/> .

Based on the above embodiment, it can be understood that the video frame adapter can obtain the visual information of the text through the frame parameters to be learned, but although this information is beneficial to the visual information extraction of the entire video, it also causes attention bias, resulting in a lack of relevant video visual information. In order to further improve the performance of the final video language model, this embodiment can be used to extract the visual features of the original frame visual information, that is, the output of the image encoder of the target visual language pre-training model. Encode again to obtain the visual information of the entire video, and obtain the complete visual information of the video through further fusion extraction to fill the information loss caused by attention bias of the video frame adapter, which may include the following:

As shown in FIG5 , the video language model further includes a docking network layer; the docking network layer includes a first transformer model, a video feature extraction layer and a joint layer; wherein the first transformer model is used to fuse visual features based on a self-attention mechanism to obtain visual fusion features; the video feature extraction layer is used to extract features from the visual fusion features and convert the dimensions of the extracted features into the same dimensions as the input dimensions of the video adapter; the joint layer is used to combine the frame visual information and the output of the video feature extraction layer, and input the joint features into the video adapter. In this embodiment, the first transformer model can be a Transformer, which is used for The Transformer self-attention mechanism can be used to fuse visual information to obtain

TF is any Transformer encoder model. For example, the simplest MultiHeadAttention mechanism can be used. The input dimension is / > and output dimensions/> Similarly, the header can be set to 3. Then the video information is obtained through the video feature extraction layer, which is a combination of a linear rectification function and a fully connected layer, which can be, for example, a multi-layer perceptron MLP, /> , through the video feature extraction layer, the visual semantic information is further abstracted and combined to obtain richer visual information of the video. Convert to the same dimensions as the video frame adapter. Get the result of the video frame adapter processing/> , that is, the frame visual information related to the video frame text, the joint layer joint frame visual information/> and video visual information/> , which can be defined as , where/> .

From the above, it can be seen that this embodiment integrates the frame text visual information and the global video information through the above-mentioned video adapter, which can solve the information loss of the frame text visual information caused by attention bias. By setting the layer normalization processing, the convergence speed of the video adapter can be accelerated. At the same time, the generalization ability is further improved through the residual structure, which is conducive to improving the training efficiency and performance of the entire video language model.

Based on the model structure of the video adapter determined in any of the above embodiments, this embodiment also requires training the video adapter. As shown in FIG6 , the training process of the video adapter may include the following contents:

Extract video features of video visual information; extract coded text features corresponding to video text features; iteratively update the video adapter according to loss information between the video features and the coded text features.

In this embodiment, a network structure connected to a video adapter can be used to extract video text features and video visual information features. A fully connected layer can be used to obtain video visual information. The corresponding video features can be defined as video features/> ,/> , the input dimension of the fully connected layer is The dimension of the output is the same as /> The same. The features of video text features can be extracted using a feedforward neural network, which can be defined as encoded text features/> ,/> , where the input dimension of the feedforward neural network is/> Dimension, output dimension and /> same .

The above embodiment does not impose any limitation on how to iteratively update the video adapter according to the loss information between the video features and the encoded text features, that is, it does not impose any limitation on the loss function adopted by the video adapter. Of course, as a simple implementation method, the loss function of the video adapter can be directly based on the mean square error or the cross entropy error or other related technologies. In order to improve the performance of the video adapter, this embodiment also provides a method for determining the loss function of the video adapter, which may include the following contents:

The video-text loss calculation relationship is stored locally in advance, the video-text loss calculation relationship is called, and the video-text loss of the video adapter is calculated. The video-text loss calculation relationship can be expressed as: ;

Among them, Loss _G is the video-text loss, _NG is the total number of matches between video features and encoded text features in the current batch, is the i 'th video feature in the current batch,/> is the encoded text feature that matches the i 'th video feature,/> is the j'th encoded text feature that does not match the i'th video feature, θ represents the similarity between the video feature and the encoded text feature, and τ is the parameter to be optimized. Among them, for the calculation relationship involved in the present invention, since the log base is a fixed number and does not affect the model training process, the base can usually be omitted, and the technicians in the relevant field can select the required base according to the actual situation, which does not affect the implementation of the present invention.

From the above, we can see that by determining the loss function based on the similarity of visual features and text representations, we can train a video adapter with better performance and improve the performance of the video language model.

The above embodiment does not limit how to use the target visual language pre-training model to process the input video sample, the preset video parameters to be learned, and the frame parameters to be learned. Based on this, this embodiment also provides an exemplary implementation method, which may include the following content:

Perform image sampling processing on the video samples to obtain multiple frames of sample images; use the image encoder of the target visual language pre-training model to extract image features of each frame of the sample image to obtain visual features; use the text encoder of the target visual language pre-training model to extract text semantic features of the text description label of the video samples; use the text encoder of the target visual language pre-training model to respectively extract parameter features corresponding to the video parameters to be learned and the frame parameters to be learned.

In order to enable the image encoder of the target visual language pre-training model to quickly output visual features, video frame splitting can also be performed, that is, the continuous video stream is converted into a single image frame process. A fixed time interval can be set in the time dimension, and N frames are selected by uniform sampling. The result after sampling a single video is recorded as , where N is the number of samples and V is the result after the current video is frame extracted.

In order to extract visual features with rich semantics and achieve effective representation of image visual information and cross-modal learning capabilities, the process of the image encoder of the target visual language pre-training model extracting image features of each frame of sample images may include: dividing the current frame image into multiple image blocks with non-overlapping content; converting each graphic block into a one-dimensional representation through linear mapping, and adding position coding information to the corresponding image block; inputting the image blocks after linear mapping and position coding into the encoder of the second converter model, and extracting features from the output of the encoder of the second converter model to obtain the visual features of the video sample. The video frame features extracted by the image encoder based on the target visual language pre-training model are also visual features, and the final result can be represented as , where/> is the frame video feature. Taking CLIP as an example, the target visual language pre-training model uses Vision Transformer (ViT, visual encoder) to extract image visual features. The input image is first divided into non-overlapping tiles of size 16x16. These tiles represent the local area of the image. Each tile is linearly mapped to convert it from a two-dimensional representation to a one-dimensional representation. At the same time, a position code is added to each tile to capture its spatial position information in the image. The tile sequence after linear mapping and position coding passes through the 12-layer Transformer encoder. The Transformer encoder uses a self-attention mechanism to integrate contextual information in the tile sequence, model the global information of the image, and promote interaction and fusion between different tiles. At the output of the Transformer encoder, in order to extract image features and transfer feature information between different positions to capture details and semantic associations in the image, a multi-layer perceptron can be used for feature extraction. Through the above steps, visual features with rich semantics can be extracted from the image. These features can capture the content and semantic information of the image for cross-modal matching and learning with text input. The extracted image feature dimension is 197x768, where 197 represents the length of the tile sequence and 768 is the feature dimension of the tile.

From the above, we can see that through video frame decomposition and the above-mentioned visual feature extraction method, visual features with rich semantics can be extracted, the effective representation of image visual information and cross-modal learning capabilities can be achieved, and the performance of the video language model can be effectively improved.

Furthermore, considering the video language data or video language tasks, the weak correlation between video and text will make it more difficult for the video language model to understand the semantics and associate the modalities. If end-to-end direct training is adopted, the model converges slowly. Therefore, the present invention also provides a training method for the video language model, which may include the following contents:

The video frame description text label, the frame parameters to be learned, and the video sample data set are used as inputs, and the video frame adapter is trained by freezing the image encoder of the target visual language pre-trained model and using the frame parameters to be learned to obtain the visual information corresponding to the video frame description text label; when the video frame adapter training is completed, the video frame description text label, the video frame description text label, the frame parameters to be learned, the video parameters to be learned, and the video sample data set are used as inputs to train the video adapter. Of course, before model training, it is necessary to pre-set the relevant parameters of the model training, including but not limited to setting the dimensions of the frame parameters to be learned, such as C is 256,768; setting the number of training rounds, learning rate, optimizer, frame splitting interval, image size input to the image encoder, and image enhancement method. On the basis of completing the video frame adaptation, when training the video adapter, the learning rate of the video frame adapter can be lowered, that is, when receiving the learning rate adjustment instruction, the current learning rate is updated according to the new learning rate of the learning rate adjustment instruction; the new learning rate is less than the current learning rate, such as reducing the previous 3e-3 to 5e-4. When training the video adapter, in addition to the video part, the input includes the video frame text description label, the video text description label, the frame parameters to be learned, and the video parameters to be learned. That is, the current input is , where /> is the video text description label and the video parameters to be learned,/> 、/> They are the text description label of the video frame and the frame parameters to be learned. When training the video adapter, by combining the frame parameters to be learned and the video parameters to be learned, fine-grained alignment across time scales can be achieved, improving the model's ability to understand and express the global and local features of the video. By combining the video frame adapter and the video adapter, feature extraction and adaptation are completed at the video frame level and the video level respectively, which can make full use of visual information at different levels in the video, including local details and global semantics, thereby improving the model's visual understanding of the video.

Furthermore, in order to improve the efficiency of model training, the video frame adapter loss function and the video language loss function can be pre-stored and directly called during training. When training a video frame adapter, the video frame adapter loss function can be called to calculate the loss function of the video frame adapter; when training a video adapter, the video language loss function can be called to calculate the loss function of the video adapter. Among them, the video frame adapter loss function is:

;

The video language loss function is:

;

Wherein, Loss represents the video language loss function, Loss _frame represents the video frame adapter loss function, Loss _ITM is the frame-text matching loss, Loss _ITC is the text generation loss, Loss _ITG is the frame-text contrast loss, Loss _MEF is the video frame mask loss, α ₀ is the frame-text matching loss coefficient, α ₁ is the text generation loss coefficient, α ₂ is the frame-text contrast loss coefficient, and β is the video frame mask loss coefficient. Loss represents the video language loss function, α is the video frame adapter loss function coefficient, Loss _G is the video-text loss, and γ is the video-text loss function coefficient.

From the above, it can be seen that this embodiment first trains the visual language model on the video frame adapter, and then completes the learning of the video adapter on the basis of reducing the learning intensity of the video frame adapter, thereby completing the training of the entire video language pre-training model, which can improve the semantic understanding ability of the video language model, improve the convergence speed of the video language model, and further improve the training speed and performance of the video language model.

It can be understood that in the process of processing video language related tasks, the pre-trained model includes a pre-training process and a fine-tuning process. The fine-tuning process is the process of applying the pre-trained model to the data set of the current downstream video language application task and adapting the model parameters to its own data set. The above embodiment trains a video language model suitable for video language tasks. The video language model is a pre-trained large model with strong generalization ability. This embodiment fine-tunes the pre-trained large model to obtain a video language model for performing a specified video language task. Based on this, the present invention also provides a video language task execution method, please refer to Figure 7, which may include the following contents:

S701: training a video language model;

S702: Obtain a video language task to be executed and a corresponding video language task sample set;

S703: Based on the video language task, fine-tune the video language model using the video language task sample set;

S704: Perform a video language task using the fine-tuned video language model.

In the pre-training stage, it is generally based on a large-scale corpus, for a specific language model training task, to train a large-scale neural network algorithm structure to learn and implement, and the large-scale neural network algorithm structure and parameters finally obtained are the pre-training model, that is, the video language model recorded in the above embodiment. In this embodiment, the video language model is trained using the video language model training method recorded in any of the previous embodiments. In the fine-tuning stage, small-scale training is performed for specific task objectives (downstream tasks) and task data (downstream data) to achieve slight adjustments to the parameters of the video language model, and finally obtain a video language model adapted to specific tasks and data. In this embodiment, the video language task to be executed and the corresponding video language task training sample set are obtained; based on the video language task, the video language model is fine-tuned using the video language task training sample set; the downstream task is the video language task to be executed, and the task data is the video language task training sample set. Finally, the video language task is executed using the fine-tuned video language model.

Among them, the video language tasks to be performed include but are not limited to video content understanding and classification tasks, video subtitle generation and translation tasks, video question-answering tasks, video summary and highlight generation tasks, and video retrieval and recommendation tasks. Among them, the video content understanding and classification task refers to the use of video language models to understand video content and classify it into different categories, such as movies, sports events, news reports, etc., such as video classification and video library content management. The video subtitle generation and translation task is to use the video language model to understand video content and dialogue, automatically generate subtitles, and even perform multi-language translation. For example, automatically generate subtitles for movies or TV shows, and access cross-language video content. The video question-answering task refers to the use of video language models to understand video content and answer questions about the video, such as interactive learning on educational platforms and automatic question answering in customer service. The video summary and highlight generation task refers to the use of video language models to automatically identify key moments in the video and generate summaries or highlight clips, which are suitable for quick browsing of long video content. For example, the wonderful moments of sports events are replayed, and the key content summary of conference recordings. Video retrieval and recommendation tasks refer to improving the accuracy and relevance of video searches by understanding video content and user queries, such as search and recommendation on online video platforms, and video retrieval in digital libraries. The video language task training sample set is the training sample data set corresponding to the above-mentioned video language task to be performed, that is, the training sample set used by the pre-trained model in the process of fine-tuning for the video language task to be performed. Taking the video language task to be performed as a video question-answering task as an example, the video language task training sample set includes multiple different types of video samples, and each video sample is pre-labeled with the corresponding question and the corresponding answer label manually or automatically. Taking the video language task to be performed as a video content understanding task as an example, the video language task training sample set is a video sample set of multiple video samples carrying video content labels, that is, the video language task training sample set is a video content understanding task training sample set, and the video content understanding task training sample set includes multiple different types of video samples, and each video sample carries a label corresponding to the video content, so that the video language model can learn the video content, and automatically understand and measure the video content of the input video.

From the above, it can be seen that this embodiment obtains a pre-trained model by first training through the model training method recorded in the above embodiment, and then fine-tunes the parameters of the video language model through the downstream video language task to be applied, so as to obtain a video language model capable of executing downstream video language tasks, which is beneficial to improving the execution efficiency and execution accuracy of video language tasks and meeting the user's execution requirements for video language related tasks.

It should be noted that there is no strict order of execution between the steps in the present invention. As long as they comply with the logical order, these steps can be executed simultaneously or in a preset order. Figures 1 and 7 are only a schematic diagram and do not mean that this is the only execution order.

Finally, based on the technical solution of the present invention, some possible application scenarios involved in the technical solution of the present invention are introduced as examples in conjunction with FIG8. FIG8 is a schematic diagram of a hardware composition framework applicable to a video language task execution method provided by the present invention, which may include the following contents:

The hardware component framework may include a first electronic device 81 and a second electronic device 82, and the first electronic device 81 and the second electronic device 82 are connected via a network 83. The first electronic device 81 is deployed with a processor for executing the video language model training method described in any of the above embodiments, and sends the trained video language model to the second electronic device 82. The second electronic device 82 is deployed to provide an interface for human-computer interaction, and stores a video language model that has undergone a pre-training phase. When a video question-and-answer task is received, a training sample set corresponding to the training video question-and-answer task is obtained; based on the video question-and-answer task, the video language model is fine-tuned using the video question-and-answer task sample set; and the video question-and-answer task is performed using the fine-tuned video language model.

Among them, the first electronic device 81 completes all or part of the steps in the video language model training recorded in the above embodiment. The video language model it builds is shown in Figure 9. The video language model includes an image encoder and a text encoder, a video frame adapter, a video adapter, and a video frame decomposition module of the target visual language pre-training model. Its input includes video frame description text labels, video frame description text labels, frame parameters to be learned, video parameters to be learned, and video sample data sets. The target visual language pre-training model is used to extract visual features, frame parameter features, and video parameter features, and input them into the video frame adapter and the video adapter respectively. The video frame adapter is used to convert visual features into frame visual information that meets the requirements of the target visual language pre-training model, and the video adapter is used to extract video visual information.

It should be noted that the above application scenarios are only shown to facilitate understanding of the concept and principle of the present invention, and the embodiments of the present invention are not limited in this respect. On the contrary, the embodiments of the present invention can be applied to any applicable scenario.

It can be seen from the above that this embodiment can improve the execution efficiency and execution accuracy of video question tasks and meet the user's execution requirements for video question and answer tasks.

The present invention also provides a corresponding device for the video language model training method and the video language task execution method, which further makes the method more practical. Among them, the device can be described from the perspective of functional modules and hardware. The video language model training device and the video language task execution device provided by the present invention are introduced below. The device is used to implement the video language model training method and the video language task execution method provided by the present invention. In this embodiment, the video language model training device and the video language task execution device may include or be divided into one or more program modules, and the one or more program modules are stored in a storage medium and executed by one or more processors to complete the corresponding video language model training method and video language task execution method disclosed in the first embodiment. The program module referred to in this embodiment refers to a series of computer program instruction segments that can complete specific functions, which is more suitable for describing the execution process of the video language model training device and the video language task execution device in the storage medium than the program itself. The following description will specifically introduce the functions of each program module of this embodiment. The video language model training device and the video language task execution device described below can be referenced to each other with the corresponding video language model training method and the video language task execution method described above.

From the perspective of functional modules, see FIG. 10 , which is a structural diagram of a video language model training device provided in this embodiment under a specific implementation mode, and the device may include:

The data acquisition module 101 is used to acquire a video sample data set carrying text description labels, preset video parameters to be learned, and frame parameters to be learned;

The input data processing module 102 is used to input the video samples, the video parameters to be learned and the frame parameters to be learned in the video sample data set into the video language model; the video language model includes a target visual language pre-training model, a video frame adapter and a video adapter; the target visual language pre-training model is used to extract visual features and parameter features and input them into the video frame adapter and the video adapter respectively, the video frame adapter is used to convert the visual features into frame visual information that meets the requirements of the target visual language pre-training model, and the video adapter is used to extract video visual information; wherein the parameter features corresponding to the video parameters to be learned are input into the video adapter, and the parameter features corresponding to the frame parameters to be learned are input into the video frame adapter, so as to obtain text-related visual information using the frame parameters to be learned;

The model parameter updating module 103 is used to iteratively update the video language model according to the loss information of the frame visual information, the video visual information and the text semantic features until the preset model training end condition is met.

Exemplarily, in some implementations of this embodiment, the video frame adapter may include a frame input layer, a text encoding layer, a cross-modal fusion layer, a feature enhancement layer, and a frame output layer;

Among them, the frame input layer is used to receive the splicing result of the frame parameter features and the video frame text features; the text encoding layer is used to encode the splicing result based on the current attention mask to obtain the frame parameter encoding features and the frame text encoding features; the cross-modal fusion layer is used to perform cross-modal fusion processing on the frame parameter encoding features and the visual features; the feature enhancement layer is used to perform feature enhancement processing on the fusion result and input the enhanced features to the text encoding layer; repeat multiple times until the first preset number of repetitions is reached to obtain the frame visual information; the frame output layer is used to output the frame visual information.

In some exemplary implementations of this embodiment, the cross-modal fusion layer is a cross-modal attention mechanism layer, which is used to use frame parameter encoding features as query vectors and visual features as a set of value vectors and key vectors, and encode frame parameter encoding features and visual features based on the cross-modal attention mechanism as a fusion result.

In some other exemplary implementations of the present embodiment, the feature enhancement layer includes a first feature enhancement layer, an interactive feature extraction layer and a second feature enhancement layer; the first feature enhancement layer is used to perform layer normalization on the fusion result, and obtain a first interactive enhancement feature through a residual connection; the interactive feature extraction layer is used to extract features from the first interactive enhancement feature to obtain a second interactive enhancement feature; the second feature enhancement layer is used to perform layer normalization on the second interactive enhancement feature, and obtain a second interactive enhancement feature through a residual connection.

In some other exemplary implementations of this embodiment, the model parameter updating module 103 includes a video frame adapter training module, and the video frame adapter training module can be used to:

Extract the features of the frame visual information corresponding to the current frame to obtain the image frame features corresponding to the current frame image; extract the frame text encoding features corresponding to the current frame to obtain the image frame text features corresponding to the current frame image; iteratively update the video frame adapter according to the loss information between each image frame feature and the corresponding image frame text feature.

As an exemplary implementation of the above embodiment, the video frame adapter training module may be further used for:

The frame-to-text matching loss is determined by using a video frame adapter to predict whether the image frame features and the image frame text features are a positive match or a negative mismatch; the frame-to-text comparison loss is determined by comparing the similarity between the image frame features and the image frame text features; some video frame text features are masked off, and the masked video frame text features are predicted by using a video frame adapter trained based on the image frame text features corresponding to the remaining video frame text features and each image frame feature to determine the text generation loss; the loss function of the video frame adapter is determined based on the frame-to-text matching loss, the frame-to-text comparison loss and the text generation loss.

As another exemplary implementation of the above embodiment, the above video frame adapter training module may be further used for:

The positively matched image frame features and image frame text features are taken as a group of positive samples, and the negatively mismatched image frame features and image frame text features are taken as a group of negative samples;

Calculate the positive similarity between the image frame features and the image frame text features in each group of positive samples, and calculate the negative similarity between the image frame features and the image frame text features in each group of negative samples;

The frame-text contrast loss is determined by comparing positive and negative similarities.

As an exemplary implementation of the above embodiment, the video frame adapter training module may be further used for:

Call the contrast loss function relationship to calculate the frame-text contrast loss; the contrast loss function relationship is: ;

where Loss _ITG is the frame-text contrast loss, Zi is the i -th image frame feature, _Ti is _the image frame text feature that matches the i -th image frame feature, Tj is the j _- th image frame feature that does not match the image frame text feature, N _ITG is the total number of matches between image frame text features and image frame features, θ represents the similarity between image frame features and image frame text features, and τ is the parameter to be optimized.

As another exemplary implementation of the above embodiment, the above video frame adapter training module may be further used for:

Determine the image frame-image frame text loss based on the frame-text matching loss, the frame-text contrast loss, and the text generation loss;

The target image frame of the masked video sample is predicted by a video frame adapter trained based on the image frame text features corresponding to the masked video sample and the features of each image frame to determine the video frame mask loss;

The loss function of the video frame adapter is determined based on the image frame-image frame text loss and the video frame mask loss.

As an exemplary implementation of the above embodiment, the video frame adapter training module may be further used for:

Call the video frame mask loss function relationship to calculate the video frame mask loss; the video frame mask loss function relationship is: ;

Among them, Loss _MTF is the video frame mask loss, is the expectation of random distribution within a small batch of video samples, D represents random distribution, V represents image frame features, V _m represents target image frame, O( V _m ) represents target image frame features, /> is the image frame feature of the video sample that is not masked, T represents/> The corresponding image frame text features, /> is the image frame text feature of the target image frame, k is the kth image frame feature that is masked inside the small batch video sample, K is the number of images that are masked inside the small batch video sample, and model represents the prediction result.

The parameter features corresponding to the video parameters to be learned are video parameter features, and the video adapter includes a video input layer, a parameter encoder layer, a feature fusion layer, a feature extraction layer and a video output layer;

Among them, the video input layer is used to receive the joint features of visual features and frame visual information; the parameter encoder layer is used to encode the video parameter features to obtain video parameter coding features; the feature fusion layer is used to fuse the video parameter coding features and the joint features; the feature extraction layer is used to extract features from the fusion processing results and transmit the extracted features to the parameter encoder layer; repeat multiple times until a second preset number of repetitions is reached to obtain video visual information, and the video output layer is used to output the video visual information.

Exemplarily, in some other implementations of this embodiment, the feature fusion layer includes a first video feature enhancement layer, a cross-modal learning layer, and a second video feature enhancement layer;

The first video feature enhancement layer is used to perform residual connection on the video parameter coding feature and the video parameter feature, and perform layer normalization processing to obtain the parameter enhancement feature;

The cross-modal learning layer is used to fuse the video parameter encoding features and the joint features based on the cross-modal attention mechanism, with the parameter enhanced features as the query vector and the joint features as a set of value vectors and key vectors to obtain multi-modal fusion features;

In the second video feature enhancement layer, residual connection is performed on the multimodal fusion features, and layer normalization is performed to obtain the fusion processing result.

In some exemplary implementations of this embodiment, the video language model further includes a docking network layer; the docking network layer includes a first transformer model, a video feature extraction layer, and a joint layer;

Among them, the first converter model is used to fuse visual features based on the self-attention mechanism to obtain visual fusion features; the video feature extraction layer is used to extract features from the visual fusion features and convert the dimensions of the extracted features into the same dimensions as the input dimensions of the video adapter; the union layer is used to combine the frame visual information and the output of the video feature extraction layer, and input the union features into the video adapter.

In some other exemplary implementations of this embodiment, the model parameter updating module 103 includes a video adapter training module, and the video adapter training module can be used to:

Extract video features of video visual information;

Extracting encoded text features corresponding to video text features;

The video adapter is iteratively updated based on the loss information between the video features and the encoded text features.

As an exemplary implementation of the above embodiment, the above video adapter training module may be further used for:

Call the video-text loss calculation formula to calculate the video-text loss of the video adapter. The video-text loss calculation formula is:

;

Among them, Loss _G is the video-text loss, _NG is the total number of matches between video features and encoded text features in the current batch, is the i 'th video feature in the current batch,/> is the encoded text feature that matches the i 'th video feature,/> is the j'th encoded text feature that does not match the i'th video feature, θ represents the similarity between the video feature and the encoded text feature, and τ is the parameter to be optimized.

Illustratively, in some other implementations of this embodiment, the input data processing module 102 may also be used for:

Performing image sampling processing on the video samples to obtain multiple frames of sample images;

The image encoder of the target visual language pre-trained model is used to extract the image features of each frame of the sample image to obtain the visual features;

The text encoder of the target visual language pre-trained model is used to extract the text semantic features of the text description labels of the video samples;

The text encoder of the target visual language pre-training model is used to extract parameter features corresponding to the video parameters to be learned and the frame parameters to be learned respectively.

In some exemplary implementations of this embodiment, the input data processing module 102 may be further used to:

The text encoder of the target visual language pre-trained model is used to randomly initialize the frame parameters to be learned, and the random initialization results of the frame parameters to be learned are used as frame parameter features;

The text encoder of the target visual language pre-trained model is used to encode the video parameters to be learned based on the current attention mask to obtain the video parameter features.

In some other exemplary implementations of this embodiment, the input data processing module 102 may be further used for:

Extract video text features of video description text labels using the text encoder of the target visual language pre-trained model;

The text encoder of the target visual language pre-trained model is used to tokenize the video frame description text label, and the tokenization result is processed by word embedding to obtain the video frame text feature;

Using the text encoder of the target visual language pre-trained model, the video description text label is encoded based on the current attention mask to obtain the video text features.

In some other exemplary implementations of this embodiment, the input data processing module 102 may be further used for:

Divide the current frame image into multiple image blocks with non-overlapping contents;

Convert each graphic block into a one-dimensional representation through linear mapping, and add position encoding information to the corresponding image block;

The image block after linear mapping and position encoding is input into the encoder of the second transformer model, and the output of the encoder of the second transformer model is subjected to feature extraction to obtain the visual features of the video sample.

Exemplarily, in some other implementations of this embodiment, the above-mentioned model parameter updating module 103 may also be used for:

Taking the video frame description text label, the frame parameters to be learned, and the video sample data set as input, the video frame adapter is trained by freezing the image encoder of the target visual language pre-trained model and using the frame parameters to be learned to obtain the visual information corresponding to the video frame description text label;

When the video frame adapter training is completed, the video frame description text label, the video frame description text label, the frame parameters to be learned, the video parameters to be learned, and the video sample data set are used as input to train the video adapter.

In some exemplary implementations of this embodiment, the above-mentioned model parameter updating module 103 may be further used for:

When a learning rate adjustment instruction is received, the current learning rate is updated according to the new learning rate of the learning rate adjustment instruction; the new learning rate is less than the current learning rate.

In some other exemplary implementations of this embodiment, the above-mentioned model parameter updating module 103 may be further used for:

Call the video frame adapter loss function to train the video frame adapter; the video frame adapter loss function is:

;

Among them, Loss _frame represents _the video frame adapter loss function, Loss _ITM is the frame-text matching loss, Loss _ITC is the text generation loss, Loss _ITG is the frame-text contrast loss, Loss _MEF is the video frame mask loss, α0 is the frame-text matching loss coefficient, α1 is the text generation loss coefficient, α2 is the frame-text contrast loss coefficient, _and β is _the video frame mask loss coefficient.

In some other exemplary implementations of this embodiment, the above-mentioned model parameter updating module 103 may be further used for:

Call the video language loss function to train the video adapter; the video language loss function is:

;

Among them, Loss represents the video language loss function, α is the coefficient of the video frame adapter loss function, Loss _G is the video-text loss, and γ is the coefficient of the video-text loss function.

From the perspective of functional modules, see FIG. 11 , which is a structural diagram of a video language task execution device provided in this embodiment under a specific implementation mode, and the device may include:

A model training module 111 is used to train and obtain a video language model;

A data acquisition module 112, used to acquire a video language task to be performed and a corresponding video language task sample set;

A model fine-tuning module 113, used to fine-tune the video language model based on the video language task using the video language task sample set;

The task execution module 114 is used to execute the video language task using the fine-tuned video language model.

The functions of the functional modules of the video language model training device and the video language task execution device of this embodiment can be specifically implemented according to the method in the above method embodiment. The specific implementation process can refer to the relevant description of the above method embodiment, which will not be repeated here.

It can be seen from the above that this embodiment can effectively improve the training efficiency of the video language model and save the computing resources required for model training.

The video language model training device and video language task execution device mentioned above are described from the perspective of functional modules. Furthermore, the present invention also provides an electronic device, which is described from the perspective of hardware. Figure 12 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention under one implementation. As shown in Figure 12, the electronic device includes a memory 120 for storing a computer program; a processor 121 for implementing the steps of the video language model training method and/or the video language task execution method mentioned in any of the above embodiments when executing the computer program.

Among them, the processor 121 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the processor 121 may also be a controller, a microcontroller, a microprocessor or other data processing chip. The processor 121 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 121 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 121 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 121 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

The memory 120 may include one or more computer-readable storage media, which may be non-transitory. The memory 120 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash storage devices. In some embodiments, the memory 120 may be an internal storage unit of an electronic device, such as a hard disk of a server. In other embodiments, the memory 120 may also be an external storage device of an electronic device, such as a plug-in hard disk equipped on a server, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Further, the memory 120 may also include both an internal storage unit of an electronic device and an external storage device. The memory 120 can not only be used to store application software and various types of data installed in the electronic device, such as: the code of the program in the process of executing the video language model training method and the video language task execution method, but also can be used to temporarily store data that has been output or is to be output. In this embodiment, the memory 120 is at least used to store the following computer program 1201, wherein, after the computer program is loaded and executed by the processor 121, the relevant steps of the video language model training method and the video language task execution method disclosed in any of the aforementioned embodiments can be implemented. In addition, the resources stored in the memory 120 may also include an operating system 1202 and data 1203, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 1202 may include Windows, Unix, Linux, etc. Data 1203 may include but is not limited to data corresponding to the video language model training results and the video language task execution results, etc.

In some embodiments, the electronic device may further include a display screen 122, an input/output interface 123, a communication interface 124 or a network interface, a power supply 125 and a communication bus 126. Among them, the display screen 122 and the input/output interface 123, such as a keyboard, belong to a user interface, and an exemplary user interface may also include a standard wired interface, a wireless interface, etc. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device, etc. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface. The communication interface 124 may exemplarily include a wired interface and/or a wireless interface, such as a WI-FI interface, a Bluetooth interface, etc., which is generally used to establish a communication connection between an electronic device and other electronic devices. The communication bus 126 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, FIG12 shows only one thick line, but this does not mean that there is only one bus or one type of bus.

Those skilled in the art will appreciate that the structure shown in FIG. 12 does not limit the electronic device and may include more or fewer components than shown in the figure, for example, may also include a sensor 127 for implementing various functions.

The functions of the functional modules of the electronic device described in this embodiment can be specifically implemented according to the method in the above method embodiment. The specific implementation process can refer to the relevant description of the above method embodiment, which will not be repeated here.

It can be seen from the above that this embodiment can effectively improve the training efficiency of the video language model and save the computing resources required for model training.

It is understandable that if the video language model training method and the video language task execution method in the above-mentioned embodiment are implemented in the form of a software functional unit and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (RandomAccess Memory, RAM), electrically erasable programmable ROM, register, hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, removable disk, CD-ROM, disk or optical disk, etc. Various media that can store program codes.

Based on this, the present invention also provides a readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the video language model training method and/or the video language task execution method described in any of the above embodiments.

In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the hardware disclosed in the embodiments, including devices and electronic devices, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description.

Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

The above is a detailed introduction to a video language task execution and model training method, device, electronic device and readable storage medium provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that based on the embodiments of the present invention, for ordinary technicians in this technical field, all other embodiments obtained without creative work are within the scope of protection of the present invention. Without departing from the principles of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims (26)

1. A video language model training method, characterized by comprising: Obtain a video sample data set carrying text description labels, pre-set video parameters to be learned, and frame parameters to be learned; The video samples in the video sample data set, the video parameters to be learned and the frame parameters to be learned are input into the video language model; the video language model includes a target visual language pre-training model, a video frame adapter and a video adapter; the target visual language pre-training model is used to extract visual features and parameter features and input them into the video frame adapter and the video adapter respectively, the video frame adapter is used to convert the visual features into frame visual information that meets the requirements of the target visual language pre-training model, and the video adapter is used to extract video visual information; Iteratively updating the video language model according to the frame visual information, the video visual information, and the loss information of the text semantic features until a preset model training end condition is met; wherein the parameter features corresponding to the video parameters to be learned are input into the video adapter, and the parameter features corresponding to the frame parameters to be learned are input into the video frame adapter, so as to obtain visual information related to the text using the frame parameters to be learned; The image encoder of the target visual language pre-training model is used to extract visual features of video samples, and the text encoder of the target visual language pre-training model is used to extract text semantic features of text description labels of video samples, and extract parameter features corresponding to the video parameters to be learned and the frame parameters to be learned; Among them, the parameter features corresponding to the frame parameters to be learned are frame parameter features, the text description labels include video frame description text labels, the text semantic features corresponding to the video frame description text labels are video frame text features, and the video frame adapter includes a frame input layer, a text encoding layer, a cross-modal fusion layer, a feature enhancement layer and a frame output layer; the frame input layer is used to receive the splicing result of the frame parameter features and the video frame text features; the text encoding layer is used to encode the splicing result based on the current attention mask to obtain the frame parameter encoding features; the cross-modal fusion layer is used to perform cross-modal fusion processing on the frame parameter encoding features and the visual features; the feature enhancement layer is used to perform feature enhancement processing on the fusion result and input the enhanced features into the text encoding layer; the frame output layer is used to output frame visual information; The parameter features corresponding to the video parameters to be learned are video parameter features, and the video adapter includes a video input layer, a parameter encoder layer, a feature fusion layer, a feature extraction layer and a video output layer; the video input layer is used to receive the joint features of the visual features and the frame visual information; the parameter encoder layer is used to encode the video parameter features to obtain video parameter encoding features; the feature fusion layer is used to fuse the video parameter encoding features and the joint features; the feature extraction layer is used to extract features from the fusion processing results and transmit the extracted features to the parameter encoder layer; the video output layer is used to output video visual information; Among them, each video sample has a text description label, the video description text label is the text data corresponding to the entire video, and the video frame description text label is the text data corresponding to the current frame image. When the frame visual information and video visual information corresponding to the video sample are obtained, they are compared with the text semantic features of the text description label corresponding to the video sample, and the model parameters of the video language model are updated by continuously narrowing the gap between the two. 2. The video language model training method according to claim 1, characterized in that the cross-modal fusion layer is a cross-modal attention mechanism layer, and the cross-modal fusion processing of the frame parameter encoding features and the visual features comprises: The frame parameter encoding features are used as query vectors, and the visual features are used as a set of value vectors and key vectors. The frame parameter encoding features and the visual features are encoded based on a cross-modal attention mechanism to serve as a fusion result. 3. The video language model training method according to claim 1, characterized in that the feature enhancement layer includes a first feature enhancement layer, an interactive feature extraction layer and a second feature enhancement layer; The first feature enhancement layer is used to perform layer normalization processing on the fusion result and obtain a first interactive enhancement feature through a residual connection; The interactive feature extraction layer is used to extract the first interactive enhancement feature to obtain a second interactive enhancement feature; The second feature enhancement layer is used to perform layer normalization processing on the second interactive enhancement feature and connect it through a residual connection. 4. The video language model training method according to claim 1, wherein the training process of the video frame adapter comprises: Extract the features of the frame visual information corresponding to the current frame to obtain the image frame features corresponding to the current frame image; Extract the video frame text features corresponding to the current frame to obtain the image frame text features corresponding to the current frame image; The video frame adapter is iteratively updated according to the loss information between each image frame feature and the corresponding image frame text feature. 5. The video language model training method according to claim 4, characterized in that the iterative updating of the video frame adapter according to the loss information between each image frame feature and the corresponding image frame text feature comprises: Determining a frame-text matching loss by predicting whether the image frame features and the image frame text features are a positive match or a negative mismatch using the video frame adapter; By comparing the similarity between the image frame features and the image frame text features, the frame-text comparison loss is determined; Masking off some video frame text features, predicting the masked video frame text features through the video frame adapter trained based on the image frame text features corresponding to the remaining video frame text features and each image frame feature, and determining the text generation loss; A loss function of the video frame adapter is determined according to the frame-text matching loss, the frame-text contrast loss and the text generation loss. 6. The video language model training method according to claim 5, characterized in that the determining the frame-text contrast loss by comparing the similarity between the image frame features and the image frame text features comprises: The positively matched image frame features and image frame text features are taken as a group of positive samples, and the negatively mismatched image frame features and image frame text features are taken as a group of negative samples; Calculate the positive similarity between the image frame features and the image frame text features in each group of positive samples, and calculate the negative similarity between the image frame features and the image frame text features in each group of negative samples; A frame-text contrast loss is determined by comparing the positive similarity and the negative similarity. 7. The video language model training method according to claim 5, characterized in that the determining the frame-text contrast loss by comparing the similarity between the image frame features and the image frame text features comprises: The contrast loss function relationship is called to calculate the frame-text contrast loss; the contrast loss function relationship is: ; where Loss _ITG is _the frame-text contrast loss, _exp represents the exponential function, Zi is the i -th image frame feature, Ti is the image frame text feature that matches the i -th image frame feature, Tj is the _j - th image frame feature that does not match the image frame text feature, N _ITG is the total number of matches between image frame text features and image frame features, θ represents the similarity between image frame features and image frame text features, and τ is the parameter to be optimized. 8. The video language model training method according to claim 5, characterized in that the loss function of the video frame adapter is determined according to the frame-text matching loss, the frame-text contrast loss and the text generation loss, comprising: Determine an image frame-image frame text loss according to the frame-text matching loss, the frame-text comparison loss and the text generation loss; Masking a target image frame of the video sample, predicting the target image frame through a video frame adapter trained based on image frame text features corresponding to the masked video sample and features of each image frame, and determining a video frame mask loss; A loss function of the video frame adapter is determined based on the image frame-to-image frame text loss and the video frame mask loss. 9. The video language model training method according to claim 8, wherein determining the video frame mask loss comprises: The video frame mask loss function relationship is called to calculate the video frame mask loss; the video frame mask loss function relationship is: ; Among them, Loss _MTF is the video frame mask loss, is the expectation of random distribution within a small batch of video samples, D represents random distribution, V represents image frame features, V _m represents target image frame, O( V _m ) represents target image frame features, /> is the image frame feature of the video sample that is not masked, T represents/> The corresponding image frame text features, /> is the image frame text feature of the target image frame, k is the kth image frame feature that is masked inside the small batch video sample, K is the number of images that are masked inside the small batch video sample, and model represents the prediction result. 10. The video language model training method according to claim 1, characterized in that the feature fusion layer includes a first video feature enhancement layer, a cross-modal learning layer and a second video feature enhancement layer; The first video feature enhancement layer is used to perform residual connection on the video parameter coding feature and the video parameter feature, and perform layer normalization processing to obtain parameter enhancement features; The cross-modal learning layer is used to fuse the video parameter encoding feature and the joint feature based on a cross-modal attention mechanism, taking the parameter enhancement feature as a query vector and the joint feature as a set of value vectors and key vectors to obtain a multi-modal fusion feature; The second video feature enhancement layer performs residual connection on the multimodal fusion features and performs layer normalization processing to obtain a fusion processing result. 11. The video language model training method according to claim 9, characterized in that the video language model further comprises a docking network layer; the docking network layer comprises a first converter model, a video feature extraction layer and a joint layer; Among them, the first converter model is used to fuse the visual features based on the self-attention mechanism to obtain visual fusion features; the video feature extraction layer is used to extract features from the visual fusion features and convert the dimensions of the extracted features into the same dimensions as the input dimensions of the video adapter; the joint layer is used to combine the frame visual information and the output of the video feature extraction layer, and input the joint features into the video adapter. 12. The video language model training method according to claim 9, characterized in that the text description tag includes a video description text tag, the text semantic feature corresponding to the video description text tag is a video text feature, and the training process of the video adapter includes: Extracting video features of the video visual information; Extracting encoded text features corresponding to the video text features; The video adapter is iteratively updated according to the loss information between the video feature and the encoded text feature. 13. The video language model training method according to claim 12, characterized in that the loss information between the video feature and the encoded text feature comprises: The video-text loss calculation formula is called to calculate the video-text loss of the video adapter, and the video-text loss calculation formula is: ; Wherein, Loss _G is the video-text loss, _NG is the total number of matches between the video features and the encoded text features in the current batch, is the i 'th video feature in the current batch,/> is the encoded text feature that matches the i 'th video feature,/> is the j'th encoded text feature that does not match the i'th video feature, θ represents the similarity between the video feature and the encoded text feature, and τ is the parameter to be optimized. 14. The video language model training method according to claim 1, characterized in that the step of inputting the video samples in the video sample data set, the video parameters to be learned, and the frame parameters to be learned into the video language model comprises: Performing image sampling processing on the video sample to obtain multiple frames of sample images; Extracting image features of each frame of sample image using an image encoder of the target visual language pre-training model to obtain visual features; Extracting text semantic features of the text description label of the video sample using the text encoder of the target visual language pre-trained model; The text encoder of the target visual language pre-training model is used to extract parameter features corresponding to the video parameters to be learned and the frame parameters to be learned respectively. 15. The video language model training method according to claim 14, characterized in that the text encoder using the target visual language pre-training model extracts parameter features corresponding to the video parameters to be learned and the frame parameters to be learned, respectively, comprising: Using the text encoder of the target visual language pre-training model to randomly initialize the frame parameters to be learned, and using the random initialization results of the frame parameters to be learned as frame parameter features; The text encoder of the target visual language pre-training model is used to encode the video parameters to be learned based on the current attention mask to obtain video parameter features. 16. The video language model training method according to claim 14, characterized in that the text description label includes a video description text label and a video frame description text label, and the text encoder of the target visual language pre-training model is used to extract the text semantic features of the text description label of the video sample, comprising: Extracting video text features of the video description text tag using a text encoder of the target visual language pre-trained model; Using the text encoder of the target visual language pre-trained model to perform lemma processing on the video frame description text label, and performing word embedding processing on the lemma processing result to obtain video frame text features; The text encoder of the target visual language pre-trained model is used to encode the video description text label based on the current attention mask to obtain video text features. 17. The video language model training method according to claim 14, characterized in that the step of extracting image features of each frame of sample image using the image encoder of the target visual language pre-training model to obtain visual features comprises: Divide the current frame image into multiple image blocks with non-overlapping contents; Convert each graphic block into a one-dimensional representation through linear mapping, and add position encoding information to the corresponding image block; The image block after linear mapping and position encoding is input into the encoder of the second transformer model, and feature extraction is performed on the output of the encoder of the second transformer model to obtain the visual features of the video sample. 18. The video language model training method according to any one of claims 1 to 17, characterized in that the parameter features corresponding to the frame parameters to be learned are frame parameter features, the text description labels include video frame description text labels and video description text labels, the text semantic features corresponding to the video frame description text labels are video frame text features, the text semantic features corresponding to the video description text labels are video text features, and the training process of the video language model comprises: Taking the video frame description text label, the frame parameters to be learned, and the video sample data set as input, freezing the image encoder of the target visual language pre-trained model, and using the frame parameters to be learned to obtain visual information corresponding to the video frame description text label, the video frame adapter is trained; When the video frame adapter training is completed, the video frame description text label, the video frame description text label, the frame parameters to be learned, the video parameters to be learned, and the video sample data set are used as input to train the video adapter. 19. The video language model training method according to claim 18, wherein the training of the video frame adapter comprises: Call the video frame adapter loss function to train the video frame adapter; the video frame adapter loss function is: ; Among them, Loss _frame represents _the video frame adapter loss function, Loss _ITM is the frame-text matching loss, Loss _ITC is the text generation loss, Loss _ITG is the frame-text contrast loss, Loss _MEF is the video frame mask loss, α0 is the frame-text matching loss coefficient, α1 is the text generation loss coefficient, α2 is the frame-text contrast loss coefficient, _and β is _the video frame mask loss coefficient. 20. The video language model training method according to claim 19, wherein the training of the video frame adapter comprises: Calling a video language loss function to train the video adapter; the video language loss function is: ; Among them, Loss represents the video language loss function, α is the coefficient of the video frame adapter loss function, Loss _G is the video-text loss, and γ is the video-text loss function. 21. A video language task execution method, characterized by comprising: Using the video language model training method according to any one of claims 1 to 20, training a video language model; Obtaining a video language task to be performed and a corresponding video language task training sample set; Based on the video language task, fine-tuning the video language model using the video language task training sample set; The video language task is performed using the fine-tuned video language model. 22. The video language task execution method according to claim 21, characterized in that the video language task to be executed is a video content understanding task, and the video language task training sample set is a video sample set of multiple video samples carrying video content labels; based on the video language task, fine-tuning the video language model using the video language task training sample set comprises: Based on the video content understanding task, the video language model is fine-tuned using the video sample set, so as to perform the video content understanding task using the fine-tuned video language model. 23. A video language model training device, comprising: A data acquisition module is used to acquire a video sample data set carrying text description labels, pre-set video parameters to be learned, and frame parameters to be learned; The input data processing module is used to input the video samples in the video sample data set, the video parameters to be learned and the frame parameters to be learned into the video language model; the video language model includes a target visual language pre-training model, a video frame adapter and a video adapter; the target visual language pre-training model is used to extract visual features and parameter features and input them into the video frame adapter and the video adapter respectively, the video frame adapter is used to convert the visual features into frame visual information that meets the requirements of the target visual language pre-training model, and the video adapter is used to extract video visual information; wherein the parameter features corresponding to the video parameters to be learned are input into the The video adapter, the parameter features corresponding to the frame parameters to be learned are input into the video frame adapter, so as to use the frame parameters to be learned to obtain text-related visual information; wherein the image encoder of the target visual language pre-training model is used to extract the visual features of the video sample, and the text encoder of the target visual language pre-training model is used to extract the text semantic features of the text description label of the video sample, and extract the parameter features corresponding to the video parameters to be learned and the frame parameters to be learned; the model parameter updating module is used to iteratively update the video language model according to the loss information of the frame visual information, the video visual information and the text semantic features until the preset model training end condition is met; Among them, the parameter features corresponding to the frame parameters to be learned are frame parameter features, the text description labels include video frame description text labels, the text semantic features corresponding to the video frame description text labels are video frame text features, and the video frame adapter includes a frame input layer, a text encoding layer, a cross-modal fusion layer, a feature enhancement layer and a frame output layer; the frame input layer is used to receive the splicing result of the frame parameter features and the video frame text features; the text encoding layer is used to encode the splicing result based on the current attention mask to obtain the frame parameter encoding features; the cross-modal fusion layer is used to perform cross-modal fusion processing on the frame parameter encoding features and the visual features; the feature enhancement layer is used to perform feature enhancement processing on the fusion result and input the enhanced features into the text encoding layer; the frame output layer is used to output frame visual information; The parameter features corresponding to the video parameters to be learned are video parameter features, and the video adapter includes a video input layer, a parameter encoder layer, a feature fusion layer, a feature extraction layer and a video output layer; the video input layer is used to receive the joint features of the visual features and the frame visual information; the parameter encoder layer is used to encode the video parameter features to obtain video parameter encoding features; the feature fusion layer is used to fuse the video parameter encoding features and the joint features; the feature extraction layer is used to extract features from the fusion processing results and transmit the extracted features to the parameter encoder layer; the video output layer is used to output video visual information; Among them, the model parameter updating module is further used for: each video sample has a text description label, the video description text label is the text data corresponding to the entire video, and the video frame description text label is the text data corresponding to the current frame image. When the frame visual information and video visual information corresponding to the video sample are obtained, they are compared with the text semantic features of the text description label corresponding to the video sample, and the model parameters of the video language model are updated by continuously narrowing the gap between the two. 24. A video language task execution device, characterized by comprising: A model training module, used to train a video language model using the video language model training method according to any one of claims 1 to 20; A data acquisition module, used to acquire the video language task to be executed and the corresponding video language task sample set; A model fine-tuning module, used to fine-tune the video language model based on the video language task using the video language task sample set; The task execution module is used to execute the video language task using the fine-tuned video language model. 25. An electronic device, characterized in that it comprises a processor and a memory, wherein the processor is used to implement the steps of the video language model training method according to any one of claims 1 to 20 and/or the video language task execution method according to claim 22 or 21 when executing the computer program stored in the memory. 26. A readable storage medium, characterized in that a computer program is stored on the readable storage medium, and when the computer program is executed by a processor, the steps of the video language model training method according to any one of claims 1 to 20 and/or the video language task execution method according to claim 21 or 22 are implemented.