CN118692014A - Video tag identification method, device, equipment, medium and product - Google Patents

Abstract

The embodiment of the application discloses a video tag identification method, a device, equipment, a medium and a product, wherein the method comprises the following steps: acquiring a video to be identified, and extracting multi-modal features of the video to be identified to obtain first multi-modal features, wherein the first multi-modal features comprise a plurality of modal features respectively corresponding to a plurality of modes; searching a plurality of reference videos similar to the video to be identified according to the first multi-mode characteristics, wherein each reference video carries a video tag; building context learning information according to the second multi-mode features corresponding to each reference video, the video tags carried by each reference video and the first multi-mode features of the video to be identified; and identifying the video tag of the video to be identified according to the context learning information. The technical scheme of the embodiment of the application can rapidly and accurately identify the video tag of the obtained video.

Description

Video tag identification method, device, equipment, medium and product

Technical Field

The present application relates to the field of video tag technology, and in particular to a video tag recognition method, a video tag recognition device, an electronic device, a computer-readable storage medium, and a computer product.

Background Art

Video tag recognition is an important part of video content features. By automatically generating tags for massive user-generated content (UGC) videos, we can provide video content features of different granularities for downstream content distribution links (such as recommendation systems and content operations). However, due to the diversity of UGC video content, the tag libraries used in actual business scenarios often contain hundreds of thousands or even millions of tags, and accurately matching the corresponding video tags for each video faces huge challenges.

Summary of the invention

The embodiments of the present application provide a video tag recognition method, a video tag recognition device, an electronic device, a computer-readable storage medium, and a computer program product, which can quickly and accurately recognize the video tag of a video.

Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

According to one aspect of an embodiment of the present application, a video tag identification method is provided, including: obtaining a video to be identified, and performing multimodal feature extraction on the video to be identified to obtain a first multimodal feature, wherein the first multimodal feature includes multiple modal features corresponding to multiple modalities; retrieving multiple reference videos similar to the video to be identified based on the first multimodal feature, each of the reference videos carrying a video tag; constructing context learning information based on second multimodal features corresponding to each of the reference videos and video tags carried by each of the reference videos, as well as the first multimodal feature of the video to be identified; and identifying the video tag of the video to be identified based on the context learning information.

According to one aspect of an embodiment of the present application, a video tag identification device is provided, including: an acquisition module, used to acquire a video to be identified, and perform multimodal feature extraction on the video to be identified to obtain a first multimodal feature, wherein the first multimodal feature includes multiple modal features corresponding to multiple modalities; a retrieval module, used to retrieve multiple reference videos similar to the video to be identified based on the first multimodal feature, each of the reference videos carrying a video tag; a construction module, used to construct context learning information based on second multimodal features corresponding to each of the reference videos and video tags carried by each of the reference videos, as well as the first multimodal feature of the video to be identified; and an identification module, used to identify the video tag of the video to be identified based on the context learning information.

In one embodiment of the present application, the first multimodal feature includes a first video frame feature, and the second multimodal feature includes a second video frame feature; the construction module is further used to perform feature conversion processing on the first video frame feature to obtain a first visual text mark sequence, and perform feature conversion processing on the second video frame feature corresponding to each of the reference videos to obtain a second visual text mark sequence corresponding to each of the reference videos; obtain the first text information corresponding to the video to be identified and the second text information corresponding to each of the reference videos; construct the context learning information according to the second visual text mark sequence corresponding to each of the reference videos, the second text information and the video label, as well as the first visual text mark sequence and the first text information.

In one embodiment of the present application, the number of the first video frame features includes multiple; the construction module is further used to perform feature fusion processing on the multiple first video frame features to obtain target video features; and the target video features are aligned to a preset text feature space through a pre-trained feature alignment module to obtain the first visual text tag sequence.

In one embodiment of the present application, the construction module is further used to construct a learning example based on a first text tag used to isolate different information under the same video, and the second visual text tag sequence, the second text information and the video label corresponding to the same reference video, so as to obtain multiple learning examples corresponding to multiple reference videos; splicing the multiple learning examples to obtain learning information, wherein the multiple learning examples in the learning information are separated by the second text tag used to isolate information of different videos; constructing a recognition example based on the first text tag, the first visual text tag sequence and the first text information; and generating the contextual learning information based on the learning information and the recognition example.

In one embodiment of the present application, the acquisition module is further used to obtain the video title of the video to be identified, and the recognition text converted from the audio information of the video to be identified; generate a context description of the video to be identified based on the video scene of the video to be identified; and generate the first text information based on the video title, the recognition text and the context description.

In one embodiment of the present application, the first multimodal feature includes a target video feature and a first text feature; the retrieval module is used to obtain a pre-established video feature retrieval library and a text feature retrieval library, the video feature retrieval library includes a mapping relationship between each candidate video and the video feature corresponding to the candidate video, and the text feature retrieval library includes a mapping relationship between each candidate video and the text feature corresponding to the candidate video; the video feature retrieval library and the text feature retrieval library are respectively searched according to the target video feature, and the video feature retrieval library and the text feature retrieval library are respectively searched according to the first text feature to obtain a plurality of target candidate videos similar to the video to be identified; the reference video is selected from the plurality of target candidate videos according to the video similarity between the plurality of target candidate videos and the video to be identified.

In one embodiment of the present application, the multiple target candidate videos include multiple target candidate videos retrieved by each retrieval method; the retrieval module is further used to calculate the similarity between the video to be identified and each target candidate video for the multiple target candidate videos retrieved by each retrieval method; for each target candidate video, calculate the average similarity between the target candidate video and the video to be identified in each retrieval method; and select the reference video from the multiple target candidate videos based on the average similarity.

In one embodiment of the present application, the acquisition module is further used to extract multiple video frames from the video to be identified, and divide the multiple video frames into multiple segments; extract a target video frame from the multiple segments, and perform video feature extraction on the target video frame to obtain a first video frame feature; obtain the first text information corresponding to the video to be identified, and perform text feature extraction on the first text information to obtain a first text feature; obtain the first multimodal feature based on the first video frame feature and the first text feature.

In one embodiment of the present application, recognition is further used to perform sequence feature conversion on the context learning information to obtain a context learning sequence, wherein the sequence feature is an input feature supported by a label generation model; the context learning sequence is input into the label generation model, wherein the label generation model is obtained by freezing the original model parameters of a preset language model and adjusting the newly added model parameters of the language model according to the sample context learning sequence, wherein the newly added model parameters are related to a low-rank adaptive module introduced into the language model; and a target video label of the video to be identified output by the label generation model is obtained.

In one embodiment of the present application, the device also includes a training module for obtaining a first sample video feature, a first sample text information and a first sample video label corresponding to the first sample video, and a second sample video feature, a second sample text information and a second sample video label corresponding to the second sample video; performing feature alignment processing on the first sample video feature and the second sample video feature according to the pre-trained initial feature alignment module to obtain a first sample visual mark sequence and a second sample visual mark sequence, and constructing sample context learning information according to the first sample visual mark sequence, the first sample text information and the first sample video label, and the second sample visual mark sequence and the second sample text information; performing sequence feature conversion on the sample context learning information to obtain the sample context learning sequence; introducing the low-rank adaptive module into the language model to introduce the newly added model parameters into the language model through the low-rank adaptive module; keeping the original model parameters of the language model frozen, and adjusting the newly added model parameters of the language model according to the sample context learning sequence and the second sample video label to obtain the label generation model.

In one embodiment of the present application, the training module is also used to obtain a sample image and a sample description text corresponding to the sample image; perform feature extraction on the sample image to obtain a sample visual feature, and input the sample visual feature into the module to be trained so that the module to be trained aligns the sample visual feature to the input text feature space of the language model; obtain the sample text feature corresponding to the alignment description instruction, and input the sample text feature and the target sample visual text feature output by the feature alignment module into the language model, wherein the model parameters of the language model remain frozen; obtain the sample prediction description text output by the language model, and train the module to be trained according to the sample description text and the sample prediction description text to obtain the initial feature alignment module.

In one embodiment of the present application, the training module is further used to input the sample context learning sequence into the language model introduced into the low-rank adaptive module, and obtain the predicted sample label output by the language model; according to the difference between the predicted sample label and the second sample video label, the newly added model parameters of the language model are adjusted to obtain the label generation model; the training module is also used to adjust the module parameters of the initial feature alignment module according to the difference between the predicted sample label and the second sample video label.

According to one aspect of an embodiment of the present application, an embodiment of the present application provides an electronic device, comprising one or more processors; a storage device for storing one or more computer programs, when the one or more computer programs are executed by the one or more processors, the electronic device implements the video tag recognition method as described above.

According to one aspect of an embodiment of the present application, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor of an electronic device, the electronic device executes the video tag identification method as described above.

According to one aspect of an embodiment of the present application, an embodiment of the present application provides a computer program product, including a computer program, wherein the computer program is stored in a computer-readable storage medium, and a processor of an electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device performs the above-mentioned video tag recognition method.

In the technical solution provided in the embodiments of the present application, a video to be identified is obtained, and multimodal features are extracted for the video to be identified to generate a first multimodal feature containing multiple modal features, and then multiple reference videos similar to the video to be identified are retrieved based on the first multimodal feature, each reference video carries a video label, and the reference video not only provides rich comparison information, but also provides important contextual basis for label identification of the video to be identified; contextual learning information is constructed based on the second multimodal features corresponding to each reference video and the video labels carried by each reference video, as well as the first multimodal features of the video to be identified. Through the contextual learning information, label identification is not limited to feature analysis of a single modality, but uses labels and multimodal features of multiple reference videos for contextual comparison and learning, providing a reference for identifying video labels of the video to be identified, thereby quickly and accurately identifying video labels of the video to be identified, ensuring that the identified video labels are more accurate and reliable.

It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings.

FIG1 is a schematic diagram of an implementation environment involved in the present application.

FIG. 2 is a flow chart of a video tag recognition method shown in an exemplary embodiment of the present application.

FIG. 3 is a schematic diagram of another video tag recognition method shown in an exemplary embodiment of the present application.

FIG4-1 is a schematic diagram of context learning information shown in an exemplary embodiment of the present application.

FIG4-2 is a schematic diagram of another type of context learning information shown in an exemplary embodiment of the present application.

FIG. 5 is a flow chart of another video tag identification method shown in an exemplary embodiment of the present application.

FIG. 6 is a flow chart of another video tag identification method shown in an exemplary embodiment of the present application.

FIG. 7 is a flow chart of another video tag identification method shown in an exemplary embodiment of the present application.

FIG. 8 is a flow chart of another video tag identification method shown in an exemplary embodiment of the present application.

FIG. 9 is a training diagram of a feature training module shown in an exemplary embodiment of the present application.

FIG. 10 is a schematic diagram of an application chain of a video tag identification method shown in an exemplary embodiment of the present application.

FIG. 11 is a schematic diagram of the overall structural flow of video tag recognition shown in an exemplary embodiment of the present application.

FIG. 12 is a structural block diagram of a video tag identification device shown in an exemplary embodiment of the present application.

FIG. 13 shows a schematic diagram of the structure of a computer system of an electronic device suitable for implementing an embodiment of the present application.

DETAILED DESCRIPTION

Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.

The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all contents and operations, nor must they be executed in the order described. For example, some operations may be decomposed, while some operations may be combined or partially combined, so the actual execution order may change according to actual conditions.

It should also be noted that the "multiple" mentioned in this application refers to two or more than two. "And/or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and/or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character "/" generally indicates that the previous and next associated objects are in an "or" relationship.

The technical solutions of the embodiments of the present application are introduced in detail below.

Please refer to FIG1 , which is a schematic diagram of an implementation environment involved in the present application. The implementation environment includes a terminal 10 and a server 20 .

The terminal 10 is used for the video to be identified, and sends the video to be identified to the server 20 .

The server 20 is used to perform multimodal feature extraction on the video to be identified to obtain a first multimodal feature, where the first multimodal feature includes multiple modal features corresponding to multiple modalities, and retrieve multiple reference videos similar to the video to be identified based on the first multimodal feature, each reference video carrying a video tag, wherein the server also performs multimodal feature extraction on each reference video to obtain a second multimodal feature corresponding to each reference video, and then constructs context learning information based on the second multimodal features and video tags corresponding to each reference video, and the first multimodal feature of the video to be identified, and identifies the video tag of the video to be identified through the context learning information.

The server may also send the video tag of the video to be identified to the terminal, so that the terminal can classify the video or recommend the video according to the video tag.

In some embodiments, the server 20 may also obtain the video to be identified by itself, and then perform multimodal feature extraction, similar video retrieval, and context learning information construction, and identify the video tag of the video to be identified based on the context learning information.

In some embodiments, the terminal 10 can also independently implement the processing of video tag recognition, that is, the terminal 10 obtains the video to be identified to perform multimodal feature extraction, similar video retrieval and context learning information construction, and then identifies the video tag of the video to be identified.

Among them, the aforementioned terminal 10 can be any electronic device that can obtain modal data of the target object, such as a smart phone, a tablet, a laptop computer, a computer, an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, etc. The server 20 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network) and big data and artificial intelligence platforms. There is no restriction on this here.

The terminal 10 and the server 20 establish a communication connection in advance through a network, so that the terminal 10 and the server 20 can communicate with each other through the network. The network can be a wired network or a wireless network, and there is no limitation here.

It should be noted that in the specific implementation of the present application, at least one of the video to be identified and the reference video involves object-relatedness. When the embodiment of the present application is applied to a specific product or technology, it is necessary to obtain the object's permission or consent, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions.

The various implementation details of the technical solutions of the embodiments of the present application are elaborated in detail below.

As shown in Figure 2, Figure 2 is a flowchart of a video tag identification method shown in an embodiment of the present application. The method can be applied to the implementation environment shown in Figure 1. The method can be executed by a terminal or a server, or by a terminal and a server together. In the embodiment of the present application, the method is executed by a server as an example for explanation. The video tag identification method may include S210 to S240, which are described in detail as follows.

S210: Obtain a video to be identified, and perform multimodal feature extraction on the video to be identified to obtain a first multimodal feature, where the first multimodal feature includes multiple modal features corresponding to multiple modalities.

In the embodiment of the present application, the video to be identified refers to a video whose video tag has not yet been determined, and the video tag is a label used to identify and classify video content. The label can be a specific keyword, phrase, or any descriptive information related to the video theme.

A video to be identified is obtained, and multimodal feature extraction is performed on the video to be identified, wherein modality can be understood as different representations or acquisition methods of data, such as images and texts belong to different modalities, and multimodality includes at least two modalities. Multimodal feature extraction is performed on the video to be identified, and then the first multimodal feature obtained includes multiple modal features obtained by performing feature extraction on multiple modalities respectively. For example, feature extraction can be performed on the video content of the video to be identified, and feature extraction can be performed on the text information of the video to be identified. It can also be feature extraction on the audio information of the video to be identified. The first multimodal feature includes video features corresponding to the video content, text features corresponding to the text information, and audio features corresponding to the audio information.

In one example, the video to be identified includes information of multiple modalities, and feature extraction can be performed on a specific modality of the video to be identified according to the business scenario to which the video to be identified belongs or the instructions of the terminal; for example, if the business scenario to which the video to be identified belongs is a short video scene using a template, and the template includes uniform background music, feature extraction may not be performed on the audio information of the video to be identified, but feature extraction may be performed on the text and video content of the video to be identified.

In another example, feature extraction can be performed on characteristic modalities of the video to be identified based on the video type of the video to be identified. For example, for a music video, feature extraction can be performed on the video content and sound; for a TV/movie video segment, feature extraction can be performed on the video content, text, and sound of the video.

S220. Retrieve a plurality of reference videos similar to the video to be identified according to the first multimodal feature, each reference video carrying a video tag.

In an embodiment of the present application, a video library is pre-stored, and the video library includes multiple candidate videos carrying video tags. The video tags of the candidate videos can be obtained by identification through steps S210~S240, or can be determined by labeling by relevant personnel, which is not limited here.

After obtaining the first multimodal feature of the video to be identified, multiple reference videos similar to the video to be identified are retrieved from the video library according to the first multimodal feature, wherein the similarity between the retrieved reference videos and the video to be identified is greater than a preset similarity threshold.

In one example, the process of retrieving similar videos based on the first multimodal feature may be to compare each modal feature included in the first multimodal feature with the corresponding modal feature of the candidate video in the video library, and then select the reference video corresponding to the feature similarity higher than the preset similarity threshold; such as comparing the video features included in the first multimodal feature with the video features of each candidate video in the video library to select a first reference video set; comparing the text features included in the first multimodal feature with the text features of each candidate video in the video library to select a second reference video set; comparing the video features included in the first multimodal feature with the text features of each candidate video in the video library to select a third reference video set; comparing the text features included in the first multimodal feature with the video features of each candidate video in the video library to select a fourth reference video set, and obtaining a final reference video similar to the video to be identified based on the first to fourth reference video sets.

In another example, the process of retrieving similar videos based on the first multimodal feature may be to fuse the multiple modal features included in the first multimodal feature to obtain a fused feature, and compare the fused feature with the fused features of each candidate video in the video library to select a fifth reference video set.

In another example, a final reference video similar to the video to be identified can be obtained based on the above-mentioned first to fourth reference video sets and fifth reference video set. For example, the intersection video of the first to fourth reference video sets and the fifth reference video set is used as the final reference video; or the first to fourth reference video sets and the fifth reference video set are all used as the final reference video.

S230: Construct contextual learning information according to the second multimodal features corresponding to the reference videos, the video tags carried by the reference videos, and the first multimodal features of the video to be identified.

In an embodiment of the present application, it is necessary to perform multimodal feature extraction on each reference video to obtain a second multimodal feature, wherein the process of performing multimodal feature extraction on the video to be identified and the reference video is the same, and the modal feature type included in the first multimodal feature is the same as the modal feature type included in the second multimodal feature, for example, the feature type of the first multimodal feature includes video features and text features, and the feature type of the second multimodal feature also includes video features and text features.

It can be understood that there is a mapping relationship between the second multimodal features of the reference video and the video label, that is, the reference video is a learning example related to the video label, and then contextual learning information is constructed according to the second multimodal features corresponding to each reference video and the video label carried by each reference video, as well as the first multimodal features of the video to be identified. The contextual learning information includes the contextual information provided by the learning example, which is prompt information with analogy learning.

In one example, the context learning information may be obtained by splicing the second multimodal features corresponding to the reference videos, the video tags carried by the reference videos, and the first multimodal features of the video to be identified.

S240: Identify the video tag of the video to be identified based on the context learning information.

As described above, the contextual learning information includes the contextual information provided by the learning examples, and the video tags of the video to be identified are identified based on the contextual learning information. The contextual information provided by the learning examples can be used to guide the identification of the video tags of the video to be identified, and then the target video tags of the video to be identified are obtained.

In one example, the context learning information is input into a pre-trained label generation model, and the label generation model generates a video label for the video to be identified according to the context learning information.

In an embodiment of the present application, a video to be identified is obtained, and multimodal features are extracted for the video to be identified to generate a first multimodal feature containing multiple modal features, and then multiple reference videos similar to the video to be identified are retrieved based on the first multimodal feature, each reference video carries a video label, and the reference video not only provides rich comparison information, but also provides important contextual basis for label identification of the video to be identified; contextual learning information is constructed based on the second multimodal features corresponding to each reference video and the video labels carried by each reference video, as well as the first multimodal feature of the video to be identified. Through the contextual learning information, label identification is not limited to feature analysis of a single modality, but uses labels and multimodal features of multiple reference videos for contextual comparison and learning, providing a reference for identifying video labels of the video to be identified, reducing possible deviations under a single modality, and being able to quickly and accurately identify video labels of the video to be identified, ensuring that the identified video labels are more accurate and reliable.

In one embodiment of the present application, another video tag recognition method is provided. The video tag recognition method can be applied to the implementation environment shown in Figure 1. The method can be executed by a terminal or a server, or can be executed by a terminal and a server together. In the embodiment of the present application, the method is executed by a server as an example for illustration. As shown in Figure 3, the video tag recognition method is based on S210~S240 shown in Figure 2, and S230 is expanded to S310~S330, wherein the first multimodal feature includes a first video frame feature, and the second multimodal feature includes a second video frame feature, and the video frame feature is a feature extraction of a video frame of a video to be recognized; S310~S330 are described in detail as follows.

S310, performing feature conversion processing on the first video frame features to obtain a first visual text mark sequence, and performing feature conversion processing on the second video frame features corresponding to each reference video to obtain a second visual text mark sequence corresponding to each reference video.

In an embodiment of the present application, the video label is text-type information, and the video label and video frame features come from different forms and modes. In order to facilitate the subsequent construction of contextual learning information, the video frame features can be converted into a visual text tag sequence through feature conversion processing. The visual text tag sequence is a token sequence corresponding to the video frame features, wherein the process of performing feature conversion processing on the first video frame features and the second video frame features is the same.

In one example, the number of first video frame features includes multiple, and a first visual text mark sequence is performed on the first video frame features through feature conversion, including: performing feature fusion processing on the multiple first video frame features to obtain target video features; aligning the target video features to a preset text feature space through a pre-trained feature alignment module to obtain a first visual text mark sequence.

Among them, since there are multiple first video frame features, feature fusion processing is performed on the multiple first video frame features to obtain a target video feature, and the target video feature can better contain comprehensive video content; the feature fusion processing can be an average pooling processing or a weighted processing, wherein the weight of each first video frame feature can be determined according to the position of the video frame corresponding to the first video frame feature in the video to be identified, such as the closer the position of the video frame in the video to be identified is to the middle position, the greater the weight of the video frame feature corresponding to the video frame; the closer the position of the video frame in the video to be identified is to the start or end position, the smaller the weight of the video frame feature corresponding to the video frame.

In other embodiments of the present application, if there is only one first video frame feature, the first video frame feature can be enhanced to expand the video frame feature, and then feature fusion processing is performed based on the first video frame feature and the expanded video frame feature to obtain the target video feature.

In an embodiment of the present application, the feature alignment module is pre-trained, and the target video features are input into the feature alignment module to align the target video features to a preset text feature space, which can be the input text feature space of a pre-trained label generation model, that is, to align non-text tokens to the space of the label generation model input text tokens, thereby "translating" the non-text features into content that the label generation model can understand, so that the subsequent label generation model can identify video labels based on contextual learning information.

Similarly, feature fusion is performed on multiple second video frame features to obtain video features, and the video features are aligned to a preset text feature space through a pre-trained feature alignment module to obtain a second visual text tag sequence.

S320: Obtain first text information corresponding to the video to be identified and second text information corresponding to each reference video.

In an embodiment of the present application, the first text information can be obtained based on the accompanying text information of the video to be identified, such as the accompanying text information including the video title, video release time, video release platform, text content detected and identified from the video, such as subtitles, text obtained by converting human voice, etc.; the text type contained in the second text information corresponding to the reference video is the same as the text type contained in the first text information.

It is understandable that the accompanying text information of the video to be identified has more and more complex information content. In order to construct efficient and accurate context learning information, the accompanying text information can be screened to obtain the first text information.

In one example, obtaining first text information corresponding to a video to be identified includes: obtaining a video title of the video to be identified, and an identification text converted from audio information of the video to be identified; generating a context description of the video to be identified based on a video scene of the video to be identified; and generating the first text information based on the video title, the identification text, and the context description.

Among them, metadata can be parsed from the video file of the video to be identified to extract the video title. If the extracted video title includes special characters or formats, it needs to be cleaned and normalized, such as removing special characters, processing spaces, normalizing upper and lower cases, etc.; if the language of the extracted video title is different from that of the video tag, the extracted video title can also be converted into the language corresponding to the video tag.

It is understandable that the video to be identified includes audio information, which is not background audio but audio emitted by an object contained in the video to be identified, such as an audio conversation. The audio information can be converted into recognized text through automatic speech recognition technology.

The video title and recognition text are the basic text information attached to the video. In order to further add descriptive language to provide more contextual background and further enrich the contextual information brought by the video itself, the context description can be determined according to the video scene of the video to be recognized. Among them, the scene content in the video can be analyzed by object detection, such as determining the corresponding scene of the video by identifying the objects appearing in the video; further analyze the timing information of the video to be recognized, determine the scene change information of the video, so as to help understand the context of the video, and then generate a context description based on the scene and scene change information of the video. For example, if elements such as the beach and the sun are detected in the video, a context description similar to "on a sunny beach" can be generated.

In one example, the situation description can be further refined based on the recognition of keywords and voice emotions (such as cheerfulness, seriousness) in the text. For example, if laughter or cheering is detected in the audio, it can be inferred that the situation of the video may be a happy or celebratory scene.

After obtaining the video title, recognition text and situation description, the video title, recognition text and situation description can be spliced together to obtain the first text information, that is, meaningful text information is obtained from a video, and a first text information with context and contextual understanding is generated to provide a basis for subsequent label generation.

In one example, if there are repeated words in the video title, recognition text and situation description, the first text information is spliced after deleting the repeated words; in one example, if the information content of the first text information is small, such as the number of text words contained in the first text information is less than a preset threshold, the first text information can be enhanced by text data enhancement, and diversified text inputs can be generated through methods such as synonym replacement, sentence reconstruction, and text expansion.

The process of acquiring the second text information is the same as the process of acquiring the first text information, and will not be described in detail here.

S330: Construct context learning information according to the second visual text mark sequence, the second text information and the video label corresponding to each reference video, and the first visual text mark sequence and the first text information.

In an embodiment of the present application, the second visual text marker sequence, the second text information and the video label corresponding to each reference video, as well as the first visual text marker sequence and the first text information can be spliced together to obtain contextual learning information; wherein, the second visual text marker sequence and the second text information corresponding to the same reference video can be used as input, and the video label can be used as output, and the input and output can be spliced together to obtain an input-output learning example, and contextual information is provided through the learning example, and the learning examples corresponding to multiple reference videos are spliced with the first visual text marker sequence and the first text information of the video to be identified to obtain contextual learning information.

In one example, contextual learning information is constructed based on the second visual feature, second text information and video label corresponding to each reference video, as well as the first visual text label sequence and the first text information, including: constructing a learning example based on the first text label used to isolate different information under the same video, and the second visual text label sequence, the second text information and the video label corresponding to the same reference video to obtain multiple learning examples corresponding to multiple reference videos; splicing the multiple learning examples to obtain learning information, wherein the multiple learning examples in the learning information are separated by the second text label used to isolate information from different videos; constructing a recognition example based on the first text label, the first visual text label sequence and the first text information; and generating contextual learning information based on the learning information and the recognition example.

For example, the first text tag and the second text tag are special tokens, such as the first text tag is <pad>, which is used to isolate different information under the same video. The learning example corresponding to reference video 1, that is, the aforementioned input-output example, such as: second visual text tag sequence 1<pad> second text information 1<pad> video tag 1; the learning example corresponding to reference video 2, such as: second visual text tag sequence 2<pad> second text information 2<pad> video tag 2, and so on to obtain multiple learning examples corresponding to multiple reference videos. The second text tag is <eoc>, which is used to isolate information from different videos. Multiple learning examples are spliced to obtain learning information, and the learning information is: second visual text tag sequence 1<pad> second text information 1<pad> video tag 1<eoc> second visual text tag sequence 2<pad> second text information 2<pad> video tag 2<eoc>, and so on, where video tag 1 and the second visual text tag sequence are information from different videos, and are therefore segmented by <eoc>.

In an embodiment of the present application, a recognition example is constructed, and the recognition example is: a first visual text tag sequence <pad> first text information; then the learning information and the recognition example are spliced together to form contextual learning information with prompt information, and the prompt information can be a mapping relationship between the visual text tag sequence, the text information and the video tag; wherein, the learning information and the recognition example also belong to information of different videos, so the learning information and the recognition example are spliced together through the second text tag to obtain the contextual learning information as shown in Figure 4-1.

It can be understood that the markup format of the first text mark and the second text mark can be flexibly adjusted according to actual conditions, and <pad> and <eoc> are examples provided in the embodiments of the present application.

In other embodiments of the present application, when constructing contextual learning information, in addition to combining multiple reference videos similar to the video to be identified, negative example videos that are unrelated to the video to be identified can also be introduced for comparison. For example, the third visual text tag sequence, the third text information and the video label of the negative example video can be spliced together through the first text tag to obtain negative information, and then the learning information, the negative information and the recognition example can be spliced together to obtain the contextual learning information as shown in Figure 4-2.

It should be noted that for other detailed descriptions of S210 - S220 , S240 shown in FIG. 3 , please refer to S210 - S220 , S240 shown in FIG. 2 , which will not be repeated here.

In an embodiment of the present application, by first listing the multimodal features and video labels of a reference video to construct a learning example, then listing the multimodal features of a video to be identified to construct an identification example, and then splicing the learning example and the identification example, an input containing contextual information of similar videos is constructed, which can effectively utilize multimodal information for contextual learning, so as to facilitate the subsequent accurate generation of content labels for the video to be identified.

The embodiment of the present application provides another video tag recognition method, which can be applied to the implementation environment shown in Figure 1. The method can be executed by a terminal or a server, or by a terminal and a server. In the embodiment of the present application, the method is executed by a server as an example for explanation. As shown in Figure 5, the video tag recognition method is based on that shown in Figure 2, and S220 shown in Figure 2 is expanded to S510~S530. Among them, the first multimodal feature includes a target video feature and a first text feature, wherein the target video feature can be obtained by feature fusion of multiple first video frame features, and the feature dimension of the target video feature is the same as the feature dimension of the first text feature. S510~S530 are described in detail as follows.

S510, obtaining a pre-established video feature retrieval library and a text feature retrieval library, wherein the video feature retrieval library includes a mapping relationship between each candidate video and the video features corresponding to the candidate video, and the text feature retrieval library includes a mapping relationship between each candidate video and the text features corresponding to the candidate video.

In an embodiment of the present application, a video feature retrieval library and a text feature retrieval library are pre-established, wherein the video feature retrieval library A includes a mapping relationship between each candidate video and the video features corresponding to the candidate video; and the text feature retrieval library B includes a mapping relationship between each candidate video and the text features corresponding to the candidate video.

S520, searching the video feature retrieval library and the text feature retrieval library respectively according to the target video feature, and searching the video feature retrieval library and the text feature retrieval library respectively according to the first text feature, to obtain a plurality of target candidate videos similar to the video to be identified.

According to the target video feature and the first text feature, searches are performed in retrieval library A and retrieval library B respectively. There are four retrieval methods: 1) The target video feature searches retrieval library A to determine the candidate video set 1 corresponding to the video features similar to the target video features in retrieval library A; 2) The target video feature searches retrieval library B to determine the candidate video set 2 corresponding to the text features similar to the target video features in retrieval library B; 3) The first text feature searches retrieval library A to determine the candidate video set 3 corresponding to the video features similar to the first text feature in retrieval library A; 4) The first text feature searches retrieval library B to determine the candidate video set 4 corresponding to the text features similar to the first text feature in retrieval library B.

Among them, during video retrieval, the feature similarity between the target video feature and the video feature and the text feature is calculated, and the feature similarity between the first text feature and the video feature and the text feature is calculated, wherein the feature similarity can be calculated by the cosine similarity of the feature vector, the Euclidean distance, etc., and then multiple candidate videos with high similarity are selected as the candidate video set, such as selecting K candidate videos corresponding to the top-K feature similarities to obtain the candidate video set, wherein the number of candidate videos included in each candidate video set can be the same or different.

In one example, duplicate videos are deleted from candidate video set 1 to candidate video set 4, and the candidate videos can be sorted from high to low according to similarity, and a specified number of candidate videos are selected as multiple target candidate videos.

In another example, the videos in candidate video set 1 to candidate video set 4 may be directly used as target candidate videos.

S530: Select a reference video from the multiple target candidate videos according to the video similarities between the multiple target candidate videos and the video to be identified.

In one example, the video similarity between each target candidate video and the video to be identified may be calculated, and then multiple reference videos may be selected from the multiple target candidate videos according to the video similarity.

In another example, the multiple target candidate videos include the multiple target candidate videos retrieved by each retrieval method, i.e., the target candidate videos in candidate video set 1 to the target candidate videos in candidate video set 4. For the multiple target candidate videos retrieved by each retrieval method, the similarity between the video to be identified and each target candidate video is calculated; for each target candidate video, the average similarity between the target candidate video and the video to be identified in each retrieval method is calculated; and a reference video is selected from the multiple target candidate videos according to the average similarity.

For multiple target candidate videos in the candidate video set 1 in retrieval method 1), calculate the video similarity between each target candidate video and the video to be identified; similarly, for retrieval methods 2)~4), calculate the video similarity between each target candidate video and the video to be identified; then, for each target candidate video, if the target candidate video appears in multiple candidate video sets at the same time, calculate the average similarity between the target candidate video and the video to be identified in each candidate video set; for example, retrieval method 1) retrieves four target candidate videos a, b, c, and d, and their video similarities with the video to be identified are {a: 0.4, b: 0.2, c: 0.1, d: 0.7} respectively; retrieval method 2) retrieves four target candidate videos a, b, d, and e, and their video similarities with the video to be identified are {a: 0.3, b: 0.5, d: 0.6, e: 0.6} respectively; retrieval method 3) retrieves three target candidate videos a, b, and c, and their video similarities with the video to be identified are {a: 0.4, b: 0.5,c：0.3 }; Retrieval method 4) retrieves four target candidate videos a, b, e, and f, and their video similarities with the video to be identified are {a: 0.6, b: 0.2, e: 0.4, f: 0.2} respectively; for the target candidate video a, the average similarity between the target candidate video a and the video to be identified is 0.425; the average similarity of the target candidate video b is: 0.35; if a target candidate video does not appear in a retrieval method, then in this retrieval method, the similarity of the target candidate video is 0, such as the average similarity of the target candidate video c is (0.1+0+0.3+0)/4=0.1, and the average similarity of the target candidate video d is 0.425; the average similarity of the target candidate video e is 0.25; the average similarity of the target candidate video f is 0.05.

When selecting a reference video from multiple target candidate videos according to the average similarity, the average similarity can be sorted from high to low, and then the top-N videos with high similarity can be selected. For example, the target candidate videos a, b, and d can be selected as reference videos, where N is less than K.

In other embodiments of the present application, when calculating the average similarity of each target candidate video, it is not necessarily necessary to assign the same weight to the four retrieval methods. Weights can be dynamically assigned to each retrieval method. For example, the weights corresponding to retrieval method 1) and retrieval method 3) are greater than the weights corresponding to retrieval method 2) and retrieval method 4), and then the weighted average similarity of the target candidate videos is calculated, and the reference video is selected according to the weighted average similarity.

It should be noted that, for the detailed introduction of S210, S230-S240 shown in FIG. 5, please refer to S210, S230-S240 shown in FIG. 2, which will not be repeated here.

In the embodiment of the present application, video features and text features are used to perform cross-search in two different retrieval libraries, so that multimodal information can be used to the greatest extent to retrieve target candidate videos similar to the video to be identified; by calculating the average similarity of the target candidate video with the video to be identified in each retrieval method, the accuracy and reliability of selecting reference videos are improved, and then the top-N videos most similar to the video to be identified and their label results are obtained.

The embodiment of the present application also provides another video tag recognition method, which can be applied to the implementation environment shown in FIG1 . The method can be executed by a terminal or a server, or by both a terminal and a server. In the embodiment of the present application, the method is executed by a server as an example for explanation. As shown in FIG6 , based on FIG2 , S210 shown in FIG2 is expanded to S610~S640. Among them, S610~S640 are described in detail as follows.

S610: Obtain a video to be identified, extract multiple video frames from the video to be identified, and divide the multiple video frames into multiple segments.

S620: extract a target video frame from the multiple segments, and perform video feature extraction on the target video frame to obtain a first video frame feature.

S630: Obtain first text information corresponding to the video to be identified, and perform text feature extraction on the first text information to obtain a first text feature.

S640: Obtain a first multimodal feature according to the first video frame feature and the first text feature.

In an embodiment of the present application, multiple video frames can be extracted from the video to be identified according to the video length of the video to be identified. For example, one video frame is extracted from the video to be identified at preset time intervals, and the longer the video length of the video to be identified, the longer the preset time is; for example, if the video to be identified is a 30S video, one video frame is extracted every 1s; if the video to be identified is a 1min video, one video frame is extracted every 2s; multiple video frames can also be randomly extracted from the video to be identified, and the video to be identified can also be divided into three segments, and video frames are extracted from the three segments according to different proportions.

After extracting multiple video frames, the multiple video frames can be divided into multiple segments, wherein the multiple video frames can be first sorted by time or randomly sorted, and then the multiple segments are equally cut, and then target video frames are extracted from the multiple segments, such as randomly extracting 1 frame or multiple frames from each segment to obtain multiple target video frames, and video features are extracted from the target video frames through a pre-trained video feature encoder to obtain first video frame features.

To obtain the first text information corresponding to the video to be identified, please refer to the embodiment shown in FIG3 , which will not be described in detail here; the first text feature is extracted from the first text information by a pre-trained text feature encoder.

The first video frame feature and the first text feature are used as the first multimodal feature.

In one example, a text feature encoder and a video feature encoder are trained using a CLIP (Contrastive Language–Image Pre-training) method, so that text features extracted by the text feature encoder and video features extracted by the video feature encoder are close.

It should be noted that for other detailed descriptions of S220 to S240 shown in FIG. 6 , please refer to S220 to S240 shown in FIG. 2 , which will not be repeated here.

In an embodiment of the present application, multiple video frames are extracted from a video, and then the video frames are divided into segments, and target video frames are extracted from the segments for video feature extraction, so that the obtained first video frame features are representative while avoiding feature uniformity; and based on the characteristics of the CLIP model, the video text features extracted by the encoder are ideally close to each video frame feature, thereby making the first multimodal feature more accurate and reliable.

In one embodiment of the present application, another video tag recognition method is also provided. The video tag recognition method can be applied to the implementation environment shown in FIG. 1. The method can be executed by a terminal or a server, or by a terminal and a server together. In the embodiment of the present application, the method is executed by a server as an example for explanation. As shown in FIG. 7, the video tag recognition method is based on the method shown in FIG. 2 to FIG. 6, and the process of performing video tag recognition in S240 in FIG. 2 is expanded to S710~S730. The S710~S730 are described in detail as follows.

S710 , performing sequence feature conversion on the context learning information to obtain a context learning sequence, where the sequence feature is an input feature supported by the label generation model.

In an embodiment of the present application, the context learning information is a continuous long text sequence. In order to facilitate the understanding and processing of the label generation model, the context learning information needs to be processed by sequence feature conversion, and the continuous long text sequence is converted into a token sequence that the model can understand and process. The token is the basic input unit of the language model, which can be a single character, word or subword, so that the model can better understand and process the semantics and structure of the text.

In one example, the context learning information may be converted into sequence features by a tokenizer of a language model to obtain a context learning sequence.

S720. Input the context learning sequence into the label generation model. The label generation model is obtained by freezing the original model parameters of the preset language model and adjusting the newly added model parameters of the language model according to the sample context learning sequence. The newly added model parameters are related to the low-rank adaptive module introduced into the language model.

In an embodiment of the present application, the label generation model can be obtained by training a preset language model, and the language model is obtained by pre-training the model, so the language model can be applied to the field of natural language processing (NLP), such as machine translation, speech recognition, text generation, etc., wherein the language model includes but is not limited to a large language model (LLM), a multimodal large language model (MLLM), and other generative models; on this basis, in order to enable the language model to be optimized for the application scenario of the content label, the language model is fine-tuned to better fit the needs of the business scenario. Among them, the embodiment of the present application uses a low-rank adaptation module (LoRA) to efficiently fine-tune the parameters of the language model. The LoRA introduces new model parameters, such as a low-rank matrix, into the language model. The new model parameters are trainable, while the original model parameters of the language model remain frozen, and the new model parameters of the language model are adjusted according to the sample context learning sequence, and then the language model containing the new model parameters and the original model parameters is used as the label generation model.

S730: Obtain a target video label of the video to be identified output by the label generation model.

The label generation model can learn from analogies to identify the video labels of the video to be identified based on the contextual information provided by the learning examples in the contextual learning information.

It should be noted that for other detailed descriptions of S210 to S230 shown in FIG. 7 , please refer to S210 to S230 shown in FIG. 2 , which will not be repeated here.

In an embodiment of the present application, contextual learning information includes reference prompts for generating video labels based on multimodal features, which are converted into a language model to better understand and process token sequences, and then a label generation model based on the language model can refer to the prompts to quickly generate labels for the video to be identified, and the label generation model is efficiently fine-tuned based on the language model so that it can better adapt to the task requirements of video labels.

It is worth noting that in one embodiment of the present application, another video tag recognition method is also provided. The video tag recognition method can be applied to the implementation environment shown in FIG1. The method can be executed by a terminal or a server, or by a terminal and a server together. In the embodiment of the present application, the method is executed by a server as an example for explanation. As shown in FIG8, the video tag recognition method is based on the method shown in FIG7, and the training process of the tag generation model is added, namely, S810~S850. Among them, S810~S850 are described in detail as follows.

S810: Obtain first sample video features, first sample text information, and first sample video labels corresponding to the first sample video, and second sample video features, second sample text information, and second sample video labels corresponding to the second sample video.

In an embodiment of the present application, the model is trained by a training set, the training set includes multiple sample videos, each sample video carries a video label, the first sample video is a reference sample video in training, and the second sample video is a sample video to be identified in training; the first sample video and the second sample video are similar videos; the process of obtaining sample video features of the sample video can be seen in Figure 6, and the process of obtaining sample text information can be seen in Figure 3, which will not be repeated here.

S820. Perform feature alignment processing on the first sample video feature and the second sample video feature respectively according to the pre-trained initial feature alignment module to obtain a first sample visual marker sequence and a second sample visual marker sequence, and construct sample context learning information according to the first sample visual marker sequence, the first sample text information and the first sample video label, and the second sample visual marker sequence and the second sample text information.

In an embodiment of the present application, an initial feature alignment module is pre-trained, and the initial feature alignment module is capable of mapping video features to a space aligned with text features, so that visual information can be better understood and utilized by the language model. The first sample video features and the second sample video features are respectively subjected to feature alignment processing through the initial feature alignment module to map the first sample video features and the second sample video features to a space aligned with text features of the language model, and obtain a first sample visual marker sequence and a second sample visual marker sequence, respectively. Then, sample context learning information is constructed according to the first sample visual marker sequence, the first sample text information and the first sample video label, and the second sample visual marker sequence and the second sample text information. Please refer to Figure 3 for the construction process.

In one example, an initial feature alignment module can be trained through image and text description pairs in conjunction with a language model, and the training steps of the initial feature alignment module include: obtaining a sample image and a sample description text corresponding to the sample image; extracting features from the sample image to obtain sample visual features, and inputting the sample visual features into the module to be trained, so that the module to be trained aligns the sample visual features to the input text feature space of the language model; obtaining sample text features corresponding to the alignment description instructions, and inputting the sample text features and the target sample visual text features output by the feature alignment module into the language model, wherein the model parameters of the language model remain frozen; obtaining the sample prediction description text output by the language model, and training the module to be trained based on the sample description text and the sample prediction description text to obtain the initial feature alignment module.

Among them, the module to be trained can be a network with a fully connected (FC) layer, or a multi-layer perceptron (MLP) or other types of neural network structures, such as q-former; the sample description text is used to describe the image content of the sample image. As shown in Figure 9, the sample image is input into the video feature encoder, and the sample visual features are extracted by the video feature encoder. The sample visual features contain the main visual information of the sample image, such as objects, backgrounds, colors, etc. At the same time, an alignment description instruction is obtained, which is used to instruct the language model to generate a description text corresponding to the image. The alignment description instruction can be a text-type instruction, such as one of the following sentences: "Briefly describe the following picture.", "Provide a brief description of the given picture.", "Briefly explain the provided picture.", etc.; In order to facilitate the language model to process the alignment description instruction, the alignment description instruction is converted into a sample text feature through Text Tokenizer processing, and the sample text feature is input into the language model.

Since visual features and text features come from different modalities, directly inputting them into a language model usually does not work well. Therefore, the sample visual features need to be input into the module to be trained. The module to be trained maps the visual features to a space consistent with the text features through learning to obtain the target sample visual text features so that they can be aligned with the text features in the same or similar feature space.

It should be noted that when training the module to be trained, the model parameters of the language model remain frozen. After the sample text features and the target sample visual text features are input into the language model, the language model combines the target sample visual text features and the sample text features to generate a sample prediction description text. The parameters of the module to be trained can be adjusted according to the difference between the sample prediction description text and the sample description text corresponding to the sample image until the module to be trained converges to obtain the initial feature alignment module. Through training, the initial feature alignment module can effectively align the visual features to a space consistent with the text features, so that the language model can perform well under multimodal input.

S830 , performing sequence feature conversion on the sample context learning information to obtain a sample context learning sequence.

The sample context learning sequence is obtained by performing sequence feature conversion on the sample context learning information through the tokenizer of the language model.

S840. Introduce a low-rank adaptive module into the language model, so as to introduce new model parameters into the language model through the low-rank adaptive module.

S850: Keep the original model parameters of the language model frozen, and adjust the newly added model parameters of the language model according to the sample context learning sequence and the second sample video label to obtain a label generation model.

In an embodiment of the present application, a LoRA module is introduced into a language model, and the LoRA module introduces a low-rank matrix into certain layers of the model; during the forward propagation process, the LoRA module transforms the input data through a low-rank matrix, and then passes it to the next layer of the model; this transformation is equivalent to adding an extra layer of parameter adjustment on the basis of the language model, that is, introducing new model parameters into the language model without changing the parameters of the original model; during the training process, only the new model parameters are trained, and the original model parameters of the language model are kept frozen without updating the original model parameters; in this way, the model can be fine-tuned while keeping the original model parameters unchanged.

In one example, a sample context learning sequence is input into a language model that introduces a low-rank adaptive module, and a predicted sample label output by the language model is obtained; the language model can learn by analogy from the sample context learning sequence to generate a predicted sample label corresponding to a second sample video; and the second sample video corresponds to a second sample video label, so the newly added model parameters of the language model can be adjusted according to the difference between the predicted sample label and the second sample video label to obtain a label generation model; such as calculating a loss function according to the difference between the predicted sample label and the second sample video label, such as a contrast loss function, a mean square error loss function, etc., and then optimizing the newly added model parameters according to the loss function until the difference between the predicted sample label and the second sample video label is less than a preset threshold to obtain a label generation model.

It should be noted that while training the LoRA module, the initial feature alignment module can be further optimized, that is, the module parameters of the initial feature alignment module are adjusted according to the difference between the predicted sample label and the second sample video label. For example, the module parameters of the initial feature alignment module are further optimized according to the loss function until the difference between the predicted sample label and the second sample video label is less than a preset threshold, thereby obtaining the final feature alignment module.

In an embodiment of the present application, the feature alignment module is first preliminarily trained in combination with the language model, and the parameters of the language model are kept frozen. Through training, the feature alignment module learns to align visual features with text features, thereby realizing effective conversion and fusion between multimodal features. Then, LoRA is introduced into the language model. LoRA uses low-rank decomposition to achieve effective adjustment of the model output with minimal parameter changes, adapts to the requirements of the video labeling task, trains LoRA, and continues to train the adapter so that the model can generate content labels for the video to be identified based on the multimodal context instances, and the feature alignment of the feature alignment module is more accurate.

For ease of understanding, the embodiment of the present application also provides a video tag recognition method, as shown in Figure 10, the video enters the content processing link from the content production link, obtains the corresponding content features through human-machine collaboration, and enters the downstream content distribution link. The video tag recognition method proposed in the embodiment of the present application belongs to the machine tagging link.

The entire process of video tag recognition can be seen in Figure 11, including video multimodal feature extraction, similar video retrieval, context learning instruction construction and model recognition process, in which the LLM model is taken as an example. LLM can choose open source Chinese LLM, such as chinese-llama, BLOOM, etc.

The video multimodal feature extraction includes: extracting two modes from the original video, video frames and video text. Video frames: First, extract one frame per second from a video, and then divide it into M segments. In the training mode, randomly select one frame from each segment, and in the test mode, select the middle frame from each segment, and get a total of M frames; Video text: extract the title of the video and the text converted by the video ASR, and splice them into a sentence.

In the embodiment of the present application, the reason for not using the original audio features is: based on the business scenario, many background sounds in short videos are unified popular background music, which does not have good discriminability, and important spoken information has been extracted by ASR and merged into the video text information. The original audio signal is not used as one of the inputs here.

The embodiment of this application uses the video feature encoder (cv encoder) and text feature encoder (text encoder) trained by the CLIP method to extract video multimodal features. Specifically, the video frame has a total of M images, and the video frame features are calculated by the cv encoder, denoted as f = {f ^v ₁ , f ^v ₂ ,…, f ^v _M }; the video title and other texts such as ASR are spliced together, and the video text features are calculated by the text encoder, denoted as f ^t . Based on the characteristics of the CLIP model alignment features, ideally, the video text features are relatively close to each video frame feature.

Before performing similar video retrieval, a retrieval library needs to be constructed. For each video, the above method can be used to obtain the video frame feature F = {f ^v ₁ , f ^v ₂ ,…, f ^v _M } and the video text feature f ^t . In order to simplify the calculation process, the video feature F is average pooled to obtain a feature f ^v with the same dimension as the text feature. Each video in the retrieval library has a label annotation result. At this time, two retrieval libraries can be built: retrieval library A: {f ^v ==> video}, which uses the visual features of the video frame to build the library; retrieval library B: {f ^t ==> video}, which uses the video text features to build the library.

The above steps of constructing the retrieval library can be completed by offline calculation.

For the video to be identified, after extracting the visual features f ^v and text features f ^t , a cross search is performed on the search libraries A and B, and the top-K results of the 4-way search are returned. Based on the above introduction, the features of CLIP ideally align the visual and feature features. In addition to searching the search library A with visual features and searching the search library B with text features, you can also use the cross search of visual features searching the text feature search library B and text features searching the visual feature search library A. The videos in the above 4-way top-K results are respectively calculated with the video to be identified, and then the average similarity of the retrieved videos in the 4-way results is calculated (if a video is not in a certain top-K, the similarity is regarded as 0), and the top-N videos and corresponding label results are obtained according to the average similarity, which are used as subsequent context reference samples. Among them, K can be set to a value larger than N, such as K = 2 * N.

The embodiment of the present application draws on the basic idea of in-context learning (ICL), that is, learning from analogy, and extends it to multimodal application scenarios; in the embodiment of the present application, the top-N samples closest to the video content to be identified are recalled through the similar video retrieval method described above, and the input of LLM is constructed using the following template.

Specifically, the template first lists the multimodal features and corresponding label results of the top-N videos, and then gives the features of the video to be identified, allowing LLM to generate its content label. Among them, <pad> and <eoc> are special tokens. <pad> is used to separate different modalities and result token sequences within a video; <eoc> stands for end of chunk, which is used to separate token sequences of different videos. In the template, the video text i is the information spliced together from the title of the video and text such as ASR (here refers to the original text, not the text feature f ^t ); the video tag i is the label annotation result text corresponding to the video; in the template, [visual token_i] represents the token sequence of the visual feature, which is obtained here by converting the video frame feature f ^v through an adapter (feature alignment module) that needs to be trained; the entire input sequence generates a token sequence (i.e., the aforementioned context learning sequence) through the LLM tokenizer; in this way, through this method, an input similar to the context demonstration example is constructed for the LLM; see Figure 4-1.

The model recognition process is that LLM receives a token sequence (i.e., the aforementioned context learning sequence) and generates a label based on this sequence.

It should be pointed out that the embodiment of the present application is different from the traditional one that does not train LLM at all. Considering that the task of content labeling is somewhat subjective, proper fine-tuning of LLM can better fit the needs of business scenarios. Therefore, a LoRA is used to efficiently fine-tune the parameters of LLM. During the training process, all LLM parameters remain frozen, and only the LoRA and adapter modules are trained.

The LLM training approach consists of two key phases.

Phase 1: Train the adapter to align visual features. In this phase, only the adapter is trained without adding LoRA, and the image-text description pair is used for training.

At this stage, no other information is provided to the LLM, and only the visual token can be received, forcing the adapter to successfully "translate" the visual information to the LLM to generate the correct image description text, so that the adapter can be better trained. For details, see Figure 9.

Phase 2: Training LoRA. In this phase, LLM joins LoRA and trains the adapter and LoRA at the same time, allowing the model to generate label results for the video to be identified based on the context instance, as shown in Figure 11. In this phase, the input token sequence is generated according to the template according to the method described above, without the need for manual design of instructions.

In the embodiment of the present application, if during the training process, the LLM model generates label results for the video to be identified, the selected context example combination and template structure can be optimized according to the quality feedback of the generated labels. For example, if the quality of the generated labels for certain categories is low, the context example combinations related to these categories can be preferentially selected for secondary learning.

It can be understood that after the model training is completed, the label of the new video can be predicted; after the visual features and text features of the video are extracted, by retrieving similar videos and constructing a context-enhanced input token sequence according to the template, LLM can directly generate the corresponding content label results.

The video tag recognition method provided in this application utilizes the inherent strong reasoning ability of the large model and also takes into account the differences in standards under business characteristics and scenarios, and proposes a multimodal large model based on context enhancement. By combining the idea of context learning, a similar video retrieval is introduced, and the top-N recalled similar videos and corresponding label results of the video to be identified are input to the large model in the form of reference samples for reasoning together, thereby enhancing the features of the input model and achieving the training goals with fewer training samples.

Here, an apparatus embodiment of the present application is introduced, which can be used to execute the video tag identification method in the above-mentioned embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the above-mentioned embodiment of the video tag identification method of the present application.

An embodiment of the present application provides a video tag recognition device, as shown in FIG12 , the device includes:

The acquisition module 1210 is used to acquire a video to be identified, and perform multimodal feature extraction on the video to be identified to obtain a first multimodal feature, where the first multimodal feature includes multiple modal features corresponding to multiple modalities.

The retrieval module 1220 is used to retrieve a plurality of reference videos similar to the video to be identified according to the first multimodal feature, each of the reference videos carrying a video tag.

The construction module 1230 is used to construct context learning information according to the second multimodal features corresponding to each of the reference videos, the video tags carried by each of the reference videos, and the first multimodal features of the video to be identified.

The identification module 1240 is used to identify the video tag of the video to be identified according to the context learning information.

In one embodiment of the present application, based on the aforementioned scheme, the first multimodal feature includes a first video frame feature, and the second multimodal feature includes a second video frame feature; the construction module is further used to perform feature conversion processing on the first video frame feature to obtain a first visual text mark sequence, and perform feature conversion processing on the second video frame features corresponding to each of the reference videos to obtain a second visual text mark sequence corresponding to each of the reference videos; obtain the first text information corresponding to the video to be identified and the second text information corresponding to each of the reference videos; construct the context learning information according to the second visual text mark sequence corresponding to each of the reference videos, the second text information and the video label, as well as the first visual text mark sequence and the first text information.

In one embodiment of the present application, based on the aforementioned scheme, the number of the first video frame features includes multiple; the construction module is further used to perform feature fusion processing on the multiple first video frame features to obtain target video features; and the target video features are aligned to the preset text feature space through a pre-trained feature alignment module to obtain the first visual text tag sequence.

In one embodiment of the present application, based on the aforementioned scheme, the construction module is further used to construct a learning example based on a first text tag for isolating different information under the same video, and the second visual text tag sequence, the second text information and the video label corresponding to the same reference video, so as to obtain multiple learning examples corresponding to multiple reference videos; splicing the multiple learning examples to obtain learning information, wherein the multiple learning examples in the learning information are separated by the second text tag for isolating information of different videos; constructing a recognition example based on the first text tag, the first visual text tag sequence and the first text information; and generating the contextual learning information based on the learning information and the recognition example.

In one embodiment of the present application, based on the aforementioned scheme, the acquisition module is further used to obtain the video title of the video to be identified, and the recognition text converted from the audio information of the video to be identified; generate a context description of the video to be identified according to the video scene of the video to be identified; and generate the first text information according to the video title, the recognition text and the context description.

In one embodiment of the present application, based on the aforementioned scheme, the first multimodal feature includes a target video feature and a first text feature; the retrieval module is used to obtain a pre-established video feature retrieval library and a text feature retrieval library, the video feature retrieval library includes a mapping relationship between each candidate video and the video feature corresponding to the candidate video, and the text feature retrieval library includes a mapping relationship between each candidate video and the text feature corresponding to the candidate video; the video feature retrieval library and the text feature retrieval library are searched separately according to the target video feature, and the video feature retrieval library and the text feature retrieval library are searched separately according to the first text feature to obtain multiple target candidate videos similar to the video to be identified; the reference video is selected from the multiple target candidate videos according to the video similarity between the multiple target candidate videos and the video to be identified.

In one embodiment of the present application, based on the aforementioned scheme, the multiple target candidate videos include multiple target candidate videos retrieved by each retrieval method; the retrieval module is further used to calculate the similarity between the video to be identified and each target candidate video for the multiple target candidate videos retrieved by each retrieval method; for each target candidate video, calculate the average similarity between the target candidate video and the video to be identified in each retrieval method; and select the reference video from the multiple target candidate videos according to the average similarity.

In one embodiment of the present application, based on the aforementioned scheme, the acquisition module is further used to extract multiple video frames from the video to be identified, and divide the multiple video frames into multiple segments; extract a target video frame from the multiple segments, and perform video feature extraction on the target video frame to obtain a first video frame feature; obtain the first text information corresponding to the video to be identified, and perform text feature extraction on the first text information to obtain a first text feature; obtain the first multimodal feature based on the first video frame feature and the first text feature.

In one embodiment of the present application, based on the aforementioned scheme, recognition is further used to perform sequence feature conversion on the context learning information to obtain a context learning sequence, and the sequence feature is an input feature supported by a label generation model; the context learning sequence is input into the label generation model, and the label generation model is obtained by freezing the original model parameters of a preset language model and adjusting the newly added model parameters of the language model according to the sample context learning sequence, and the newly added model parameters are related to the low-rank adaptive module introduced into the language model; and the target video label of the video to be identified output by the label generation model is obtained.

In one embodiment of the present application, based on the aforementioned scheme, the device also includes a training module for obtaining a first sample video feature, a first sample text information and a first sample video label corresponding to the first sample video, and a second sample video feature, a second sample text information and a second sample video label corresponding to the second sample video; performing feature alignment processing on the first sample video feature and the second sample video feature according to the pre-trained initial feature alignment module, respectively, to obtain a first sample visual marker sequence and a second sample visual marker sequence, and constructing sample context learning information according to the first sample visual marker sequence, the first sample text information and the first sample video label, and the second sample visual marker sequence and the second sample text information; performing sequence feature conversion on the sample context learning information to obtain the sample context learning sequence; introducing the low-rank adaptive module into the language model to introduce the newly added model parameters into the language model through the low-rank adaptive module; keeping the original model parameters of the language model frozen, and adjusting the newly added model parameters of the language model according to the sample context learning sequence and the second sample video label to obtain the label generation model.

In one embodiment of the present application, based on the aforementioned scheme, the training module is also used to obtain a sample image and a sample description text corresponding to the sample image; perform feature extraction on the sample image to obtain a sample visual feature, and input the sample visual feature into the module to be trained so that the module to be trained aligns the sample visual feature to the input text feature space of the language model; obtain the sample text feature corresponding to the alignment description instruction, and input the sample text feature and the target sample visual text feature output by the feature alignment module into the language model, wherein the model parameters of the language model remain frozen; obtain the sample prediction description text output by the language model, and train the module to be trained according to the sample description text and the sample prediction description text to obtain the initial feature alignment module.

In one embodiment of the present application, based on the aforementioned scheme, the training module is further used to input the sample context learning sequence into the language model introduced into the low-rank adaptive module, and obtain the predicted sample label output by the language model; according to the difference between the predicted sample label and the second sample video label, the newly added model parameters of the language model are adjusted to obtain the label generation model; the training module is also used to adjust the module parameters of the initial feature alignment module according to the difference between the predicted sample label and the second sample video label.

It should be noted that the device provided in the above embodiment and the method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment and will not be repeated here.

The device provided in the above embodiment may be arranged in a terminal or in a server.

An embodiment of the present application also provides an electronic device, comprising one or more processors and a storage device, wherein the storage device is used to store one or more computer programs, and when the one or more computer programs are executed by one or more processors, the electronic device implements the above-mentioned video tag recognition method.

FIG. 13 shows a schematic diagram of the structure of a computer system of an electronic device suitable for implementing an embodiment of the present application.

It should be noted that the computer system 1300 of the electronic device shown in FIG. 13 is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

As shown in FIG. 13 , a computer system 1300 includes a processor (Central Processing Unit, CPU) 1301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1302 or a program loaded from a storage part 1308 to a random access memory (RAM) 1303, such as executing the method in the above embodiment. Various programs and data required for system operation are also stored in RAM 1303. CPU 1301, ROM 1302 and RAM 1303 are connected to each other via a bus 1304. An input/output (I/O) interface 1305 is also connected to the bus 1304.

In some embodiments, the following components are connected to the I/O interface 1305: an input section 1306 including a keyboard, a mouse, etc.; an output section 1307 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 1308 including a hard disk, etc.; and a communication section 1309 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1309 performs communication processing via a network such as the Internet. A drive 1310 is also connected to the I/O interface 1305 as needed. A removable medium 1311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1310 as needed so that a computer program read therefrom is installed into the storage section 1308 as needed.

In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 1309, and/or installed from a removable medium 1311. When the computer program is executed by the processor (CPU) 1301, various functions defined in the system of the present application are executed.

It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (Erasable Programmable Read Only Memory), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier, which carries a computer-readable computer program. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the devices, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and a computer program.

The units or modules involved in the embodiments described in this application may be implemented by software or hardware, and the units or modules described may also be set in a processor. The names of these units or modules do not, in some cases, constitute limitations on the units or modules themselves.

Another aspect of the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the video tag recognition method as described above is implemented. The computer-readable storage medium can be included in the electronic device described in the above embodiment, or it can exist independently without being assembled into the electronic device.

Another aspect of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the electronic device performs the video tag recognition method as described above in the above embodiments.

It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person of ordinary skill in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims (16)

1. A method for identifying a video tag, comprising:

Acquiring a video to be identified, and extracting multi-modal features of the video to be identified to obtain a first multi-modal feature, wherein the first multi-modal feature comprises a plurality of modal features respectively corresponding to a plurality of modes;

Searching a plurality of reference videos similar to the video to be identified according to the first multi-mode characteristics, wherein each reference video carries a video tag;

Building context learning information according to the second multi-mode features corresponding to the reference videos, the video tags carried by the reference videos and the first multi-mode features of the videos to be identified;

and identifying the video tag of the video to be identified according to the context learning information.

2. The method of claim 1, wherein the first multi-modal feature comprises a first video frame feature and the second multi-modal feature comprises a second video frame feature; the construction of the context learning information according to the second multi-modal feature corresponding to each reference video, the video tag carried by each reference video, and the first multi-modal feature of the video to be identified includes:

Performing feature conversion processing on the first video frame features to obtain a first visual text marking sequence, and performing feature conversion processing on the second video frame features corresponding to each reference video to obtain a second visual text marking sequence corresponding to each reference video;

Acquiring first text information corresponding to the video to be identified and second text information corresponding to each reference video;

And constructing the context learning information according to the second visual text marking sequence, the second text information and the video tag corresponding to each reference video, and the first visual text marking sequence and the first text information.

3. The method of claim 2, wherein the number of first video frame features comprises a plurality; performing feature conversion processing on the first video frame features to obtain a first visual text marking sequence, including:

Performing feature fusion processing on the plurality of first video frame features to obtain target video features;

And aligning the target video features to a preset text feature space through a pre-training feature alignment module to obtain the first visual text marking sequence.

4. The method of claim 2, wherein said constructing said contextual learning information from said second visual text label sequence, said second text information, and said video tag, and said first visual text label sequence and said first text information for each of said reference videos comprises:

According to a first text mark used for isolating different information under the same video, and the second visual text mark sequence, the second text information and the video tag corresponding to the same reference video, a learning example is built, so that a plurality of learning examples corresponding to a plurality of reference videos are obtained;

splicing the plurality of learning examples to obtain learning information, wherein the plurality of learning examples in the learning information are separated by a second text mark for isolating information of different videos;

Constructing an identification example according to the first text mark, the first visual text mark sequence and the first text information;

The contextual learning information is generated from the learning information and the recognition example.

5. The method according to claim 2, wherein the obtaining the first text information corresponding to the video to be identified and the second text information corresponding to each of the reference videos includes:

Acquiring a video title of the video to be identified and an identification text obtained by converting the audio information of the video to be identified;

Generating a situation description of the video to be identified according to the video scene of the video to be identified;

Generating the first text information according to the video title, the identification text and the context description.

6. The method of claim 1, wherein the first multimodal feature comprises a target video feature and a first text feature; retrieving a plurality of reference videos similar to the video to be identified according to the first multi-modal feature, including:

Acquiring a pre-established video feature retrieval library and a text feature retrieval library, wherein the video feature retrieval library comprises mapping relations of each candidate video and video features corresponding to the candidate videos, and the text feature retrieval library comprises mapping relations of each candidate video and text features corresponding to the candidate videos;

Searching the video feature searching library and the text feature searching library according to the target video features respectively, and searching the video feature searching library and the text feature searching library according to the first text features respectively to obtain a plurality of target candidate videos similar to the video to be identified;

and selecting the reference video from the target candidate videos according to the video similarity between the target candidate videos and the video to be identified.

7. The method of claim 6, wherein the plurality of target candidate videos includes a plurality of target candidate videos retrieved for each retrieval mode; the selecting the reference video from the target candidate videos according to the video similarity between the target candidate videos and the video to be identified comprises the following steps:

The multiple target candidate videos are retrieved according to each retrieval mode, and the similarity between the video to be identified and each target candidate video is calculated;

For each target candidate video, calculating the average similarity between the target candidate video and the video to be identified in each retrieval mode;

and selecting the reference video from the target candidate videos according to the average similarity.

8. The method according to claim 1, wherein the performing multi-modal feature extraction on the video to be identified to obtain a first multi-modal feature includes:

extracting a plurality of video frames from the video to be identified, and dividing the plurality of video frames into a plurality of fragments;

Extracting target video frames from the plurality of fragments, and extracting video features of the target video frames to obtain first video frame features;

Acquiring first text information corresponding to the video to be identified, and extracting text features of the first text information to obtain first text features;

and obtaining the first multi-mode feature according to the first video frame feature and the first text feature.

9. The method according to any one of claims 1 to 8, wherein said identifying the video tag of the video to be identified based on the contextual learning information comprises:

Performing sequence feature conversion on the context learning information to obtain a context learning sequence, wherein the sequence feature is an input feature supported by a label generation model;

Inputting the context learning sequence into the label generation model, wherein the label generation model is obtained by keeping original model parameters of a preset language model frozen and adjusting newly-added model parameters of the language model according to the sample context learning sequence, and the newly-added model parameters are related to a low-rank self-adaptive module introduced into the language model;

and acquiring a target video tag of the video to be identified, which is output by the tag generation model.

10. The method of claim 9, wherein the training step of the tag generation model comprises:

Acquiring a first sample video feature, first sample text information and a first sample video tag corresponding to a first sample video, and a second sample video feature, second sample text information and a second sample video tag corresponding to a second sample video;

Performing feature alignment processing on the first sample video feature and the second sample video feature according to a pre-trained initial feature alignment module to obtain a first sample visual marker sequence and a second sample visual marker sequence, and constructing sample context learning information according to the first sample visual marker sequence, the first sample text information and the first sample video tag, and the second sample visual marker sequence and the second sample text information;

performing sequence feature conversion on the sample context learning information to obtain a sample context learning sequence;

introducing the low-rank adaptive module into the language model to introduce the newly added model parameters into the language model through the low-rank adaptive module;

and keeping the original model parameters of the language model frozen, and adjusting the newly added model parameters of the language model according to the sample context learning sequence and the second sample video label to obtain the label generation model.

11. The method of claim 10, wherein prior to feature alignment processing of the video features according to the pre-trained initial feature alignment module, the method further comprises:

acquiring a sample image and a sample description text corresponding to the sample image;

Extracting features from the sample image to obtain sample visual features, and inputting the sample visual features to a module to be trained so that the module to be trained aligns the sample visual features to an input text feature space of the language model;

Acquiring sample text features corresponding to the alignment description instruction, and inputting the sample text features and target sample visual text features output by the feature alignment module into the language model, wherein model parameters of the language model are kept frozen;

And acquiring a sample prediction description text output by the language model, and training the module to be trained according to the sample description text and the sample prediction description text to obtain the initial feature alignment module.

12. The method of claim 10, wherein said adjusting the model parameters of the language model based on the sample context learning sequence to obtain the tag generation model comprises:

Inputting the sample context learning sequence into a language model which is introduced into the low-rank self-adaptive module, and acquiring a prediction sample label which is output by the language model;

According to the difference between the prediction sample label and the second sample video label, the newly added model parameters of the language model are adjusted to obtain the label generation model;

the method further comprises the steps of:

And adjusting the module parameters of the initial feature alignment module according to the difference between the prediction sample label and the second sample video label.

13. A video processing apparatus, comprising:

The system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring a video to be identified, and carrying out multi-modal feature extraction on the video to be identified to obtain a first multi-modal feature, wherein the first multi-modal feature comprises a plurality of modal features corresponding to a plurality of modes respectively;

The retrieval module is used for retrieving a plurality of reference videos similar to the video to be identified according to the first multi-mode characteristics, and each reference video carries a video tag;

The construction module is used for constructing context learning information according to the second multi-mode characteristics corresponding to each reference video, the video tags carried by each reference video and the first multi-mode characteristics of the video to be identified;

and the identification module is used for identifying the video tag of the video to be identified according to the context learning information.

14. An electronic device, comprising:

One or more processors;

Storage means for storing one or more programs that, when executed by the one or more processors, cause the electronic device to perform the method of any of claims 1-12.

15. A computer readable storage medium, having stored thereon a computer program which, when executed by a processor of an electronic device, causes the electronic device to perform the method of any of claims 1 to 12.

16. A computer program product, characterized in that it comprises a computer program stored in a computer readable storage medium, from which a processor of an electronic device reads and executes the computer program, causing the electronic device to perform the method of any one of claims 1 to 12.