KR20210114074A - Method, apparatus, device and medium for generating caption information of multimedia data - Google Patents

Abstract

Embodiments of the present disclosure provide a method, apparatus, device and medium for generating captioning information of multimedia data. The method includes extracting characteristic information of multimedia data to be processed, wherein the multimedia data includes a video or an image; and generating a text caption of the multimedia data based on the extracted characteristic information. According to the method provided in the embodiment of the present disclosure, the accuracy of the generated text caption of the multimedia data can be effectively improved.

Description

Method, apparatus, device and medium for generating caption information of multimedia data

The present disclosure relates to the field of computer technology, and in particular, the present disclosure relates to a method, an apparatus, an electronic device and a storage medium for generating captioning information of multimedia data.

In computer vision technology, video captioning or image captioning refers to outputting a text caption for a given video or image. For example, for a video of a child mopping the floor, the video caption can automatically output a text caption of the video: "The child is mopping the floor." The video captioning is the intersection of computer vision and natural language processing.

The existing video captioning method generally selects a frame from a video, extracts full graph features from the selected frame, performs decoding using these features, and generates a text caption of the video based on a maximum probability probability. Image captioning has a similar principle. It can be seen from the above description that the existing video captioning model generally adopts an encoder-decoder structure. The encoder is responsible for extracting the features of the video frame, and the decoder is responsible for decoding the features of the video frame and generating text captions. Although there are many methods of generating video captioning information, the accuracy of the generated video captioning information still needs to be optimized.

Embodiments of the present disclosure provide a method, an apparatus, an electronic device, and a storage medium for generating video captioning information for improving the accuracy of the generated video captioning information.

SUMMARY Embodiments of the present disclosure provide a method, an apparatus, an electronic device, and a storage medium for generating video captioning information for improving the accuracy of the generated video captioning information. The solution provided by the embodiment of the present disclosure is as follows.

According to a first aspect of the present disclosure, there is provided a method for generating captioning information of multimedia data, the method comprising: extracting characteristic information of multimedia data to be processed, the multimedia data including a video or an image; and generating a text caption of the multimedia data based on the extracted characteristic information.

According to a second aspect of the present disclosure, there is provided a device for generating caption information of multimedia data, the device comprising: a characteristic information extraction module for extracting characteristic information of multimedia data to be processed; included; and a captioning information generating module that generates a text caption of the multimedia data based on the extracted characteristic information. .

According to a third aspect of the present disclosure, there is provided an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, the processor performing the methods provided by the embodiments of the present disclosure and execute the computer program to do so.

According to a fourth aspect of the present disclosure, there is provided a computer-readable storage medium, wherein the storage medium stores a computer program that, when executed by a processor, performs the method provided by the embodiments of the present disclosure.

The beneficial effects achieved by the technical solutions according to the embodiments of the present disclosure will be described in detail in the following description of a specific implementation manner in combination with various alternative embodiments not described herein.

In order to more clearly describe the technical solutions in the embodiments of the present disclosure, the drawings used in describing the embodiments of the present disclosure are briefly described below.
1 is a schematic diagram of exemplary image captioning.
2 is a schematic diagram of example video captioning.
3 is a schematic diagram of an existing video captioning algorithm.
4 is a schematic diagram of a training process of an existing video captioning algorithm based on supervised learning.
5 is a schematic flowchart of a method for generating caption information of multimedia data according to an embodiment of the present disclosure.
6 is a schematic diagram of acquiring semantic features through a semantic prediction network according to an embodiment of the present disclosure;
7A is a schematic diagram of a spatial scene graph according to an embodiment of the present disclosure;
7B is a schematic diagram of a spatial scene graph according to another embodiment of the present disclosure;
8 is a principle diagram of acquiring relational features through a relational prediction network according to an embodiment of the present disclosure.
9 is a principle diagram of acquiring attribute features through an attribute prediction network according to an embodiment of the present disclosure.
10 is a schematic diagram of a spatio-temporal scene graph according to an embodiment of the present disclosure;
11 is a schematic diagram of a spatio-temporal scene graph according to another embodiment of the present disclosure.
12 is a schematic diagram of a feature selection network according to an embodiment of the present disclosure;
13A is a schematic structural diagram of a self-attention-based codec according to an embodiment of the present disclosure.
13B is a schematic structural diagram of a self-attention-based codec according to an embodiment of the present disclosure.
14, 15, and 16 are schematic diagrams of a method of generating video captioning information according to three embodiments of the present disclosure, respectively.
17A and 17B are principle diagrams of video captioning information acquisition according to two alternative embodiments of the present disclosure.
18 and 19 are schematic flowcharts of a method for generating video captioning information according to another two alternative embodiments of the present disclosure.
20 is a diagram illustrating a principle of acquiring video captioning information according to an embodiment of the present disclosure.
21 is a schematic flowchart of a method for generating image captioning information according to an embodiment of the present disclosure.
22 and 23 are schematic structural diagrams of a codec according to two alternative embodiments of the present disclosure.
24 is a schematic flowchart of a method for learning a multimedia data captioning model according to an embodiment of the present disclosure.
25 is a schematic diagram of a sample video with a video captioning label (ie, an original captioning label) according to an embodiment of the present disclosure.
26 is a principle diagram of a method for training a video captioning model according to an embodiment of the present disclosure.
27 is a schematic flowchart of a method for obtaining augmented multimedia data captioning information according to an embodiment of the present disclosure;
28A and 28B are schematic structural diagrams of two codecs according to two alternative embodiments of the present disclosure;
29 is a schematic flowchart of a method for training an image captioning model according to an embodiment of the present disclosure.
30 and 31 are principle diagrams of a method for generating video captioning information according to two embodiments of the present disclosure.
32 is a schematic structural diagram of an apparatus for generating caption information of multimedia data according to an embodiment of the present disclosure. and
33 is a schematic structural diagram of an electronic device applicable to an embodiment of the present disclosure.

Embodiments of the present disclosure are described in detail below. The embodiments described below with reference to the drawings are exemplary, and are only used to describe the present disclosure in order to provide a comprehensive understanding of the embodiments of the present disclosure as defined by the claims and their equivalents. Various specific details are included to aid understanding, but these examples and details are illustrative only and should not be construed as limiting the disclosure. Accordingly, those skilled in the art will recognize that changes and modifications may be made to the described embodiments without departing from the scope and spirit of the present disclosure. In addition, in the following description, some descriptions of well-known functions and structures may be omitted for clarity and conciseness.

One of ordinary skill in the art should understand that the singular forms "a", "an", "the" and "the" include plural referents unless the context clearly dictates otherwise. The expression "comprising" or "comprising" as used in the context of this disclosure means the presence of a feature, integer, step, operation, element and/or component, but includes one or more other features, integers, steps, operations, It should be further understood that this does not exclude the presence or addition of elements, components and/or combinations thereof. It should be understood that when an element is referred to as being “connected” or “coupled” to another element, it may be directly connected or coupled to the other component, or an intermediate element may be present. Also, as used herein, the terms “coupled” or “coupled” may include wireless connection or wireless coupling. As used herein, the phrase “and/or” includes all or any one and all combinations of one or more of the related listed items.

First, the method for generating captioning information of multimedia data provided in an embodiment of the present disclosure may be used to generate captioning information of a video including images of a multi-frame video, and subtitling the captioning information of the image. It should be noted that it can also be used to create The source of the image is not limited in the embodiments of the present disclosure. For example, it may be an acquired, downloaded, or received image, or it may be an image within a video, such as a key frame image or a designated frame image. That is, in an embodiment of the present disclosure, the generating method may be a method of generating captioning information of a video or a method of generating captioning information of an image.

In order to better understand and describe the solutions of the embodiments of the present disclosure, some techniques associated with the embodiments of the present disclosure are briefly described below.

In computer vision technology, video/image captioning refers to outputting text captions for a given video or image, and it is the intersection of computer vision and natural language processing. Compared to other computer vision tasks such as object detection and image segmentation, video/image captioning is a more challenging task. There is a need for a more comprehensive understanding of a video or image, as well as a need to express the content of the video or image in the form of natural language. As shown in Figure 1. When the image shown in FIG. 1 is given in FIG. 1 , a text caption "the boy is playing tennis" of the image may be automatically output. As shown in Figure 2. When a video containing the multi-frame video shown in FIG. 2 is given, a text caption of the video "The child is mopping the floor" may be automatically output.

At present, existing image captioning models generally adopt an encoder-decoder structure. The encoder is usually designed based on Convolutional Neural Networks (CNN) responsible for extracting features of an image, and the decoder is a Recurrent Neural Network (RNN) responsible for decoding features of the image to generate text captions. ) is usually designed based on

Similarly, existing video captioning models typically select a frame from a video, extract full-graph features of the selected frame using CNN, and then decode the features of every frame using RNN and Generate a text caption of the video based on a maximum probability probability. It can be seen that the existing video subtitle algorithm mostly uses an encoder-decoder structure. Since CNN is used to encode video frames and is responsible for extracting features of those video frames, it may also be referred to as an encoder or CNN encoder. Since the RNN is responsible for decoding a video frame and decoding features of the video frame and generating text captions, it may also be referred to as a decoder or RNN decoder. The RNN may use a Long Short-Term Memory (LSTM), which can be called a current LSTM decoder.

For example, a schematic diagram of an existing video captioning model is shown in FIG. 3 . As shown in FIG. 3 , the frames shown in FIG. 3 are selected from video (ellipses in the figures are omitted to indicate frames not shown), and each frame is individually processed by a CNN encoder to obtain a picture of each selected video frame. The features are extracted, and the extracted features are decoded by the LSTM decoder to generate the corresponding text caption "The man is putting pizza in the oven".

Although the prior art has been able to generate text captions for video or images, the prior art has at least the following technical problems.

(1) Existing decoders such as RNNs, which are cyclic structures, need to be trained step by step during training. Therefore, existing decoders have problems such as slow learning speed, low training efficiency, and difficulty in learning long-distance dependencies, resulting in problems such as insufficient decoding ability.

(2) Currently, in datasets commonly used in the field of video/image captioning, a training sample (ie, sample video or sample image) has a small amount of captioning information. For example, there are typically only 5 captioning labels in the sample image. It is often difficult to fully represent the information in the image using only five captioning labels, and due to the diversity of natural languages, the same meaning can be expressed in different ways. Therefore, poor diversity of captioning information of training samples is also a problem that hinders further development of the field.

(3) In the case of a video including a multi-frame image, the prior art does not consider intra-frame information. However, this information is very important to generate more accurate video captions. Therefore, there is a need to solve the problem of how to fully utilize intra-frame information.

(4) The prior art does not take into account the very important semantic information of the video or image in order to generate more accurate video captions.

(5) Existing video or image captioning models are generally based on supervised learning methods. For example, in the case of a video caption algorithm, each training video corresponds to one or more labeled video captions. 4 , for data labeled with video captions, the video data P in the data is input to a video captioning model K, so that the video captioning model K analyzes and processes the video data P to obtain a corresponding video generating a caption, and then calculating the value of the loss function Tmark(α) based on the labeled data and the video caption Q in the generated video caption, and learning of the video captioning model K is performed using the loss function Tmark( α) is guided by However, labeling the video with captions requires a lot of labor and time, thereby limiting the number of samples in the existing video captioning datasets, whereby the video trained based on the video captioning dataset The accuracy and precision of the captioning model will decrease.

(6) In the existing method of generating video captioning information or image captioning information, the length of the generated captioning information is not controllable and the user's application requirements for different lengths of captioning information in different application scenarios can't satisfy For example, when a user posts an image or video, long captioning information is needed to share more detailed information. For example, when a user drives a car, short captioning information is needed. However, existing technologies cannot meet the above requirements.

In order to solve at least one of the technical problems of the prior art described above, an embodiment of the present disclosure provides a method, an apparatus, an electronic device and a storage medium for generating caption information of multimedia data, wherein the multimedia data is video or It can be an image.

In order to make the object, technical solution and advantage of the present disclosure more clear, each of the technical solutions of the embodiments of the present disclosure and the alternative implementation manners of the present disclosure will be described in detail with reference to specific embodiments and drawings. The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

5 is a schematic flowchart of a method for generating caption information of multimedia data according to an embodiment of the present disclosure. As shown in the figure, the method may mainly include the following steps.

Step S101: The characteristic information of the multimedia data to be processed is extracted.

Step S102: A text caption of the multimedia data is generated based on the extracted characteristic information.

In an embodiment of the present disclosure, the multimedia data includes a video or an image. Based on the method provided by the embodiment of the present disclosure, video captioning information or image captioning information may be generated.

In an alternative embodiment of the present disclosure, extracting characteristic information of the multimedia data to be processed includes at least one of the following: Extracting local visual features of targets included in respective target regions of each image in the multimedia data to do; extracting semantic features of the multimedia data; when the multimedia data is a video, extracting spatial-temporal visual features of the multimedia data; extracting global visual features of the multimedia data; extracting attribute characteristics of the targets included in respective target regions of each image in the multimedia data; and extracting global attribute features of each image from the multimedia data.

That is, the characteristic information of the multimedia data may include at least one of a local visual characteristic, a semantic characteristic, a spatial-temporal visual characteristic, a global visual characteristic, a local attribute characteristic (ie, an attribute characteristic of a target), and a global attribute characteristic. have. The local visual feature is a visual feature of a target area within an image, ie, the target area is relatively local to the image to which the target area belongs.

In the case of an image, it may be each of the images mentioned above within the multimedia data. In the case of video, each image in the multimedia data may be each frame in the video or multiple selected frames in the video. For a given video, for example, selecting frames at the same interval from the given video, or using a keyframe algorithm in which frames can be selected via a neural network, multiple image frames (i.e., keyframes) from a given video. ) can be selected. In the description below, in the case of a video, each image of multimedia data will be described taking each frame selected in the video as an example.

The visual feature is a feature that can reflect pixel information in the image, and the attribute feature is a feature that can reflect attribute information of each target in the image. Therefore, the visual characteristics and attribute characteristics may be used to generate captioning information of the video or image, and the local visual characteristics may reflect information of respective target regions in each image more accurately and subdivided. Therefore, the local visual feature fully uses the inter-frame information of each image, which can create more accurate text captions of videos or images.

In practical applications, in addition to the visual characteristics and properties of the image, some other characteristics are also conducive to the captioning of multimedia data. For example, the spatial-temporal visual characteristics of a video may effectively reflect spatial and temporal dynamic changes of the video, and the semantic characteristics of a video or image reflect semantic information of content contained within the video or image. . Therefore, when generating a text caption in a video or image, spatial-temporal visual features of the video, semantic features of the video or image, etc. are also extracted through a neural network, and more various features are generated in the video caption information , to improve the expressive power of the generated video caption and improve the accuracy of captioning information.

Alternatively, for each image, a feature extraction network may be used to obtain several target regions and local features of each target regions (ie local visual features above, which may also be referred to as target features). . For the above attribute characteristics, it may be obtained through an attribute prediction network. For example, Faster R-CNN (Fast Region-Convolutional Neural Network) may be applied to each image to extract several target regions and local visual features (ie, local feature/regional feature) of each target region. . Faster R-CNN can be pre-trained on sample datasets (such as ImageNet or Visual Genome datasets).

The method for extracting target regions and local features of each target region by Faster Region-CNN is exemplary, the disclosure is not limited thereto, and the feature extraction network may be implemented by any other available neural network. It should be noted that there may be

For the spatio-temporal visual features of the video, they can be obtained using a spatio-temporal feature extraction network. For example, the spatial-temporal visual features of the video may be extracted using a 3D visual feature extraction model such as an Efficient Convolutional Network for Online Video Understanding (ECO) model or 3D-CNN. Of course, other spatio-temporal visual feature extraction networks may also be used. For semantic features, they can be extracted through a semantic prediction network.

In particular, a trained semantic prediction network can be used to acquire semantic features of an entire video or image. For example, a structural diagram of a semantic prediction network is shown in FIG. 6 . As shown in the figure, the semantic prediction network may include a CNN and a multi-classification structure (multi-classification shown in the figure). As shown as an example in FIG. 6 , video is acquired, using the network, selected video frames are input to a semantic prediction network, and video features are extracted from each frame using CNN, and the extracted video A multiple classification operation is performed on a feature through a multiple classification structure to obtain probabilities corresponding to a plurality of predefined semantic features of the video, and finally, according to the probabilities, the plurality of predefined meanings Semantic Features Outputs one or more predefined semantic features. As shown in FIG. 6 , based on the input frames (images including people and dogs shown in the figure), probabilities corresponding to various semantic features such as people, dogs and roads are semantic predictions. It may be obtained through a network, for example, as shown in the figure, the probability of including a person is 0.95 and the probability of including a dog is 0.8. Based on the predicted probabilities and preset semantic feature filtering rules, semantic features with probabilities greater than a set threshold may be output, or a set number of semantic features with higher probabilities may be output. .

In an alternative embodiment of the present disclosure, the characteristic information of the multimedia data includes local visual characteristics of the target included in respective target regions in each image of the multimedia data. Based on the extracted characteristic information, generating the text caption of the multimedia data includes: acquiring relational characteristics between targets based on local visual characteristics of each target in the image; building a scene graph of the image based on the local visual features and the relationship features between the targets; obtaining graph convolution features of the image based on a scene graph of the image; and generating a text caption of the multimedia data based on graph convolution characteristics of each image of the multimedia data.

A scene graph refers to a graph structure in which local visual characteristics, attribute characteristics (to be described in detail later), and relational characteristics of respective target regions in the image are expressed in a graph manner. The scene graph may include multiple nodes and multiple edges, where each node of the multiple nodes is a target feature (ie, a local visual feature above) or a property of a target (ie an object) included in the target area. A characteristic is specifically indicated, and each edge among the multiple edges indicates a characteristic of a relationship between nodes.

As an example, FIG. 7A shows a scene graph corresponding to a frame in an image. As shown in FIG. 7A , some nodes in the scene graph represent the local visual features of each target in the frame image, for example the nodes “People”, “Dogs” and “Skateboards” shown in FIG. 7A . denotes the feature vector of "human", the feature vector of "dog" and the feature vector of "skateboard"; Some nodes indicate the attribute characteristics of the target, for example, "blue" node indicates the attribute characteristic of the target "person" and "black" indicates the attribute characteristic of "dog"; Edges between nodes in the scene graph represent a relationship between two connected nodes, that is, a relationship between two targets. For example, the relationship between the node "person" and the node "dog" is "holding", the relationship between the "person" and "skateboard" is "skating", and so on. It can be seen from Fig. 7a that the scene graph can reflect the spatial relationship between targets in the image, which can also be referred to as a spatial scene graph.

In practical applications, for each image, a feature extraction network may be used to obtain, inter alia, several target regions and a local feature of each target region (ie, local visual features above). Relationship characteristics between targets may be obtained using a relationship prediction network. In particular, after obtaining local features of each target region of images through a feature extraction network, a pre-trained relation prediction network may be used to obtain relational characteristics between target regions from the extracted target regional features, Then, regional characteristics of each target area and their relational characteristics are expressed in a graph manner to obtain a scene graph for each frame.

The relationship prediction network is a classification network used to predict association relationships between target regions. A specific network structure of the relationship prediction network may be selected according to actual needs. As an alternative solution, the relation prediction network may include a fully connected layer, a feature concatenation layer and a softmax layer. The fully connected layer can be used not only to extract local visual features of the target region, but also to reduce the dimension of features to adapt to be processed in successive feature chain layers and softmax layers. The relationship prediction network may be trained based on the sample dataset. For example, the relationship prediction network may be trained based on a visual genome. The visual genome dataset is a commonly used relation and attribute learning dataset, and has a plurality of object attributes and relation labels. Thus, we can use visual genomic datasets to train relational prediction networks.

In particular, when a relational feature between target regions is obtained through the above relational prediction network, for regional features of at least two target regions in each target region, features can be extracted using a fully connected layer, , the extracted features corresponding to respective target regions extracted by the fully connected layer are input to the feature chain layer for chain features and then to the softmax layer. According to a probability corresponding to each relationship output by the softmax layer, relationship characteristics between at least two target regions are obtained.

As an example, FIG. 8 shows a schematic diagram of predicting relational features using a relational prediction network. As shown in FIG. 8 , local features of humans and dogs, which are targets in an image frame, are respectively input to the fully connected layer of the relationship prediction network. Features output from the fully connected layer perform feature chaining, and then input to the softmax layer to obtain corresponding probabilities of various predefined relational features between humans and dogs. Finally, one predefined relational feature is output according to the probabilities. For example, the probability that the relational feature is "on" is 0.12, the probability that the relational feature is "left" is 0.20, the probability that the relational feature is "holding" is 0.40, and so on; Afterwards, based on the probability of each relational feature, the finally output relational feature is “retained” with the highest probability. Of course, in practical application, for at least two target regions, the corresponding relational feature may be one, and as in the above example, the relational feature corresponding to the maximum probability may be selected.

In the method provided in an embodiment of the present disclosure, when obtaining a text caption of a video or image, it is a local visual feature (simply a local feature, a local feature) of respective target regions (also called target candidate region) of each image ) is implemented based on Since the local visual characteristics of each target area can reflect the information of the local areas in each image more accurately and subdivided, the method in the embodiment of the present disclosure makes full use of the intra-frame information of each image, thereby You can create more accurate text captions for videos or images.

In addition, the method in an embodiment of the present disclosure determines relational characteristics between target regions based on the local characteristic, and builds a spatial scene graph (for the image, which is a graph) of each image based on the relational characteristic. can do. The graph convolution features of each image may be obtained based on the spatial scene graph, so that a text caption of the video or image may be obtained based on the graph convolution characteristics of each image. The spatial scene graph can reflect well the targets in the image and the relationships between the targets, and the relationships between the targets are very useful for understanding and describing image content, so the spatial scene graph based graph convolution feature can improve the accuracy of text captions of videos or images.

For convenience of description, hereinafter, with respect to a target region, a node corresponding to a local visual feature of the target region may be referred to as a target node, and a local visual feature represented by the target node may be referred to as a target feature, and A node of an attribute characteristic of the target region may be referred to as an attribute node.

In an alternative embodiment of the present disclosure, the characteristic information of the multimedia data may include attribute characteristics of targets included in respective target regions of each image in the multimedia data; Here, the construction of the scene graph of the image based on the local visual features of each target and the relational features between the targets includes the local visual features of each target, the relational features between the targets, and the attribute feature of each target. and constructing a scene graph of the image based on

In this alternative solution, local visual features, attribute features and relational features corresponding to the respective target regions are used to build a scene graph in a graph manner, that is, the scene graph is composed of target nodes and attribute nodes. may include In this case, the nodes in the scene graph indicate local visual characteristics or attribute characteristics of each target region, that is, attributes and target characteristics of targets included in each target region. Corresponding edges are edges representing a relationship between targets (eg, an edge between “person” and “dog” in the spatial scene graph shown in FIG. 7A ), and edges representing a relationship between a target and attributes. (eg, an edge between “person” and “In blue” in the spatial scene graph shown in FIG. 7A ).

In practical application, in order to reduce connection redundancy, attribute nodes and target nodes within the same target region may also be combined, ie, for a target region, nodes representing the local visual characteristics and attribute characteristics of the target region The indicated nodes may be identical nodes. For example, for the scene graph shown in FIG. 7A , target nodes and attribute nodes of the same target are combined, such that “person” and “person” in FIG. 7A represent target characteristics and attribute characteristics of a person in the image, respectively. obtain the scene graph shown in FIG. 7b as "blue", so they can be combined into a node of "person in blue" as shown in FIG. 7b, which simultaneously combines the target characteristics and attribute characteristics of a person in the target area in question reflect Similarly, “dog” and “black” in FIG. 7A indicate target characteristics and attribute characteristics of the dog in the image, respectively, so they may be combined into a node of “black dog” as shown in FIG. 7B .

The attribute characteristics may be obtained based on local visual characteristics of the target area. This alternative solution also takes into account the attribute characteristics of the respective target regions in the image when building the scene graph. Since the attributes of each target in the image are also very helpful in describing the image content, a scene graph incorporating attribute characteristics can more accurately describe the video.

Alternatively, attribute features corresponding to respective target regions may be obtained from the extracted local visual features using an attribute prediction network. The attribute prediction network is a multi-classification network. The attribute prediction network may be trained based on sample data sets (such as a visual genomic dataset). In actual application, the specific network structure of the attribute prediction network may be selected according to actual needs.

In an alternative embodiment of the present disclosure, the attribute prediction network may include multiple attribute classifiers, where each classifier corresponds to one type of attribute prediction.

A specific division of the above attribute types can be configured according to actual needs. As an alternative method, the attribute type may specifically refer to a part of speech corresponding to the attribute. For example, the part-of-speech corresponding to the attribute may include a noun, a verb, an adjective, and other relatively rare types of attributes. Prediction of attribute features is performed using a classifier comprising a plurality of attribute types.

In the traditional way of building a spatial scene graph, the properties of objects (ie, targets) are not distinguished, and various properties are classified by one classifier, and thus the accuracy of the properties obtained is low. However, when a spatial scene graph is constructed based on the attribute characteristics predicted in this way, noun attributes (eg, clothes, glasses, etc.), verb attributes (eg, standing, walking, etc.), adjective attributes (eg, height It is possible to obtain more specific target attribute information, including large, fat, etc.) and relatively rare attributes (eg, blue whale, pony, etc.). Additionally, different classifiers can be used to obtain different types of attributes. In this way, the acquired properties are more accurate and the corresponding properties are more diversified to generate more accurate captioning information based on the predicted property characteristics. Additionally, in order to reduce redundancy, attribute nodes and target nodes may be combined to increase data processing efficiency.

As an alternative solution, the structure of the attribute prediction network may include a fully connected layer and a multi-classification layer. The fully connected layer can be used not only to extract the attribute features of the target region, but also can be used to reduce the dimension of the features for adaptation to be processed in subsequent multiple classification layers. Alternatively, the multiple classification layer may be implemented by multiple Sigmoids. When using an attribute prediction network to obtain attribute characteristics between various target regions, the input of the network is a local visual characteristic of the target region, and the output is one or more attribute characteristics among several predefined attribute characteristics.

As an example, FIG. 9 is a principle diagram of predicting attribute characteristics of a target region using an attribute prediction network. As shown in FIG. 9 , local visual features (local features shown in the figure) of the target “person” in the image frame are input to the fully connected layer of the attribute prediction network, and features output from the fully connected layer are input to a multiple classification layer to perform multiple classification operations to obtain probabilities respectively corresponding to various predefined attribute characteristics of a person, and finally, from among the various predefined attribute characteristics according to the probabilities Print some features. For example, a "blue" attribute feature with a probability of 0.92 and a "tall" attribute feature with a probability of 0.54 are output. In particular, in practical application, for a target area, it may have one or more attribute characteristics. For example, the attribute feature with the maximum probability may be output as the attribute feature of the target area; Alternatively, an attribute feature having a higher probability than a set threshold value or a set number of attribute features having a maximum output probability may be output as attribute features of the target region.

As an alternative solution of the present disclosure, the above steps of obtaining attribute characteristics corresponding to respective target regions may be used or omitted. That is, when building the scene graph, local visual features, relational features, and attribute features can be used. Alternatively, it may be built based on local visual features and relational features without using attribute features. When the step of obtaining an attribute feature is omitted, nodes of the constructed scene graph represent target features corresponding to target regions, ie, local attribute features. When constructing a scene graph based on relational characteristics and attribute characteristics, each node in the scene graph represents a target characteristic and/or attribute corresponding to a target area, and each edge is a relation between targets corresponding to the nodes. indicates

Referring to FIG. 7A as an example for illustration, as shown in FIG. 7A , after extracting a target feature, an attribute feature, and a relationship feature between a person and a dog from an image, a scene graph for the image may be constructed. In the scene graph shown in Figure 7a. Each block represents a target characteristic or attribute characteristic of the target area, and a line between blocks representing the target characteristic indicates a relationship between the targets, ie, a corresponding relationship characteristic. The ellipse block represents the characteristics of the relationship between the target regions, and the arrow (ie, the direction of the edge) is the relationship between the subject and the object. As shown in the scene graph of Figure 7. In the relationship between "person" and "dog", the direction of the edge between "person" and "dog" indicates that "person" is the subject and "dog" is the object. A line between the node representing the target feature and the node representing the attribute feature represents the attribute relationship between them. 7A , in the relationship between "person" and "blue", "blue" is an attribute of a person, and the direction of the arrow indicates the attribute relationship, that is, the attribute of "blue" indicates that the attribute belongs to a person. . The scene graph shown in FIG. 7A may clearly indicate each target included in the drawing, the relative positions of the targets, attributes, and behavioral relationships.

In an alternative embodiment of the present disclosure, if the multimedia data is a video, the images of the multimedia data are a plurality of frames selected from the video, and when the target areas of two adjacent frames contain the same target, Scene graphs have temporal edges between nodes corresponding to the same target (target nodes), ie, a plurality of edges include temporal edges.

To better utilize temporal information, this alternative solution further considers temporal information between two adjacent ones of the frames selected for the video, and temporal information is added to the scenegraph corresponding to each frame, - Acquire a temporal scene graph. Specifically, if targets corresponding to target regions between two adjacent frames are the same, temporal edges are added between target nodes of target regions including the same target in scene graphs of the two adjacent frames, and the temporal edge Scene graphs to which they are added may reflect both spatial and temporal relationships between the targets, which may be referred to as spatial-temporal scene graphs.

As an example, FIG. 10 shows a schematic diagram of a space-time scene graph. Temporal edges are added to the scene graph corresponding to each frame. In scene graphs corresponding to two adjacent frames, if target classes of target regions belonging to scene graphs of the two adjacent frames are the same, a temporal edge is added between the two target regions. For example, in scene graphs corresponding to the first frame and the second frame shown in FIG. 10 . The target classes of person, oven and pizza in the two frames of the scene graph are the same, and temporal edges are added between the corresponding target regions in the two frames of the scene graph, for example, two frames of the scene graph The temporal edges are added between people in fields, between ovens, and between pizzas, respectively.

Compared to the spatial scene graph, the spatial-temporal scene graph adds relationships between objects (ie, targets) in the temporal dimension, which can better describe the spatial-temporal information of the video. In addition, the spatial-temporal scene graph may further include behavior information of a target corresponding to a temporal edge (to be described below) in order to improve accuracy of behavior captioning.

In this alternative solution of the embodiment of the present disclosure, inter-frame temporal information is also considered, that is, when the scene graph is built, the spatial-temporal scene graph is obtained by combining temporal edges. In this solution, the correlation between inter-frame images is fully taken into account by combining temporal information because temporal edges are established between identical targets in adjacent frames. Therefore, when graph convolution features are extracted based on the scene graph, it is possible to better learn the continuity information of the target in different images, thereby making the best use of the intra-frame and inter-frame information of the frames. Based on this, better video captions can be obtained.

In an alternative embodiment of the present disclosure, the obtaining graph convolution features of the image based on the scene graph of the image comprises: encoding nodes and edges in the scene graph to obtain a target dimension of a feature vector ; and obtaining graph convolution features using a graph convolution network based on the obtained feature vector.

In practical application, if the dimensions of the obtained local visual characteristics, attribute characteristics and relational characteristics are the same, this step may or may not be performed.

Specifically, when nodes in the constructed scene graph represent target features or attribute characteristics corresponding to a target region, relational features and attribute features obtained in the target region may be encoded into target dimensions of a feature vector, the Later, a graph convolution network is applied to the encoded feature vectors to learn the relationship between adjacent nodes and edges in the scene graph, so that the graph-convolved features of each node included in the scene graph (i.e., graph convolution feature). Features learned based on the graph convolution network are obtained based on a graph structure (scene graph), and the graph-convolved features may include target features, properties, and relationship information.

When the nodes in the constructed scene graph represent only target features, the graph convolved features may include target features and relationship information. At this time, the attribute characteristics of each target area are not obtained from the extracted local characteristics of the target by using the attribute prediction network.

For example, using a fully connected matrix to encode all or some of the nodes (such as target nodes) and edges in the scene graph into the target dimension of the feature vector (a fixed dimension relative to the feature dimension of the input vector of the subsequent vector) can do. For example, when the dimension of the relation feature of the scene graph is 512 and the target dimension is 1024, a 512*1024 matrix may be applied to the relation feature of the scene graph, so that a dimension equal to the target dimension 1024 can be obtained let it be

After obtaining the target dimension of the feature vector, for each node of the obtained feature vector, a graph convolution formula can be used to obtain the graph convolution features of each node, the simplest weightless graph convolution formula is It is represented by Equation 1 below:

where v _i denotes the feature vector of node i in the scenegraph, i.e., the target feature vector or attribute feature vector, and N( _vi ) is adjacent to node i in the scenegraph (i.e. within the same frame in the image). represents the set of nodes, and v _j is the feature vector of node j adjacent to node i in the same frame (ie, same scenegraph) of the image. Neighbors of node i generally do not include node i itself. W is the learnable network weight parameter of the graph convolution network, v _j ⁽¹⁾ denotes the graph convolutional features (graph convolution features) of node i, and σ denotes the nonlinear activation function.

In practical application, since the relationship between different nodes in the scene graph is different in the importance of the image, that is, the importance of the relationship between different targets to the final captioning information of the multimedia data is different, the edges in the scene graph are weighted edges, so as another alternative, the graph convolution characteristics of each node can be obtained using Equation 2 below.

Here, v _j and N(vi _i ) have the same meaning as in Equation 1, and N(vi _i ) may include v _j itself; W and b represent the learnable weights and bias parameters of the graph convolution network; dir(vj, vj) denotes the direction of the edge, where dir(vj, vj) is either vi to vj (i.e., the direction of the edge is from node i to node j) or two from vj to vi. has a value of ; W _dir(vj,vj) has two corresponding results, where _{each dir(vj,vj)} corresponds to one result; label(v _j, v _j ) _{indicates the relationship between v i} and v _j , and for different relationships, it has different bias values; σ denotes the nonlinear activation function; And v _j ⁽¹⁾ represents the graph convolution feature corresponding to node i.

As another alternative, Equation 2 may be expanded to the following form as shown in Equation 3 below. In this alternative, weights may be used for the node itself. That is, the importance of different nodes is also different, and different two weights are used for neighboring nodes according to the membership relationship, that is, two weights are used for relationship characteristics.

Here, _{the meanings of v j} ⁽¹⁾ , σ, v _i , v _j , N(vi _i ) have the same meanings as the aforementioned corresponding parameters, and are not repeated here; W _s , W _(sub,obj) , W _(in,out) , W _a are learnable parameter weight matrices. Specifically, W _s is the parametric weight matrix of v _{i , and W (sub,obj)} is the parametric weight matrix of the adjacent node feature (i.e., v _j ), where (sub, obj) denotes the membership of the relationship , and it has two values _{, each indicating that v i} is the subject or object of the relationship. For example, in the scene graph shown in Figure 7a. In the relationship between "person" and "dog", "person" is the subject, and "dog" is the object. If v _j is a subject and v _i is an object, then W _obj is used (eg, "person" is the subject for "dog" in FIG. 7A), otherwise W _sub is used. W _{(in, out)} denotes the parameter weight matrix of the relationship, where (in, out) denotes the direction of the edge, and has two values, whether the edge is output from node i or input to node i It has two values, each representing For the edge between "person" and "dog" shown in Fig. 7A, for the "human" node, an edge is output from the node, and for the "dog" node, the edge is input to that node. If v _i is a subject and v _j is an object, then W _in is used, otherwise W _out is used; e ^r denotes a relational feature vector between two corresponding nodes, and it may have different values depending on the direction of the edges, and of course may have the same value. e ^r _{{(vi, vj),(vj,vi)}} denotes the feature vector of the relationship between node i and node j, and according to different directions of edges, i.e. according to different subject-object relationship, said feature Vectors can be the same or different. For example, in the relationship between "person" and "dog", it is "holding" for "person", but "held" for "dog". A _vi,vj denotes an attention layer, particularly an attention parameter matrix, and W _a denotes a weight of the attention layer.

In another alternative manner, for video, if the constructed scene graph is a spatio-temporal scene graph, for each frame of an image, inter-frame information may be further considered in addition to the intra-frame information. That is, scene convolution features of a frame image may be obtained based on a scene graph of the frame image and a scene graph of an adjacent frame image of the frame image. Specifically, all or some of the nodes and edges in the space-time scene graph are encoded to obtain a target dimension of a feature vector, and then the obtained feature vector is trained using a graph convolution network to perform graph convolution to acquire the solution features.

Alternatively, the graph convolution features may be obtained by the following equation (4).

Here, the parameters in the same Equation 4 as in Equation 3 have the same meaning. N _b (v _i ) denotes the set of nodes with the same classification as node i in the adjacent frame image of the current frame image, ie the set of identical targets in the adjacent frames; As in the example shown in Figure 10, for a "person" node in the second frame of the image, if that node is v _{i ,} then the set of "person" nodes in the first frame and/or the third frame visible in the figure. is N _b (v _i ). W _(pre,aft) represents a sequence relationship between a current frame to which node i belongs and a frame to which node j belongs in _{N b} (v _i ), which is a parameter weighting matrix of the same target in adjacent frames. W _{(pre, aft)} is two values indicating whether the current frame is before or after the adjacent frame, i.e., for the adjacent frame, whether the frame in which node i is located is the previous frame or the next frame in the time sequence, respectively. have v If _j is the previous frame, W _(pre) is used, otherwise W _(aft) is used.

In an alternative embodiment of the present disclosure, when constructing the spatio-temporal scene graph, the method may further include determining behavioral characteristics of the target corresponding to the temporal edge.

At this time, for each frame of the image, based on local visual features corresponding to target regions of the frame image, attribute features (alternative), relational features, and behavioral features of the target corresponding to a temporal edge, the spatial-temporal You can build a scene graph.

In other words, behavioral characteristics of each target node corresponding to a temporal edge in the scene graph may also be added. Alternatively, an object tracking method may be used to identify the same target between adjacent frames. For this same target, a pre-trained behavior classifier (behavior detector) is used to identify the target's behavioral classification (also called behavioral relationship) in the image, and the feature vector of the behavioral classification can be used as the target's behavioral features. .

As in the space-time scene graph shown in Fig. 11, the same targets included in each frame include "person", "pizza" and "oven", where the action corresponding to "person" is "open ( opening)", that is, in the spatio-temporal scene graph, the value of the temporal edge of the same target in an adjacent frame is "opening"; The action corresponding to "pizza" is "held"; The action corresponding to "oven" is "opened". Compared with the scene graph of FIG. 10 , the scene graph may further include behavior information corresponding to the same target included in adjacent frames. When generating captioning information based on the scene graph, more image detail information may be used, thereby further improving the accuracy of the generated captioning information.

Corresponding to this alternative solution, the following equation (5) may be used to compute the graph convolution characteristics of each node in the scene graph.

For a description of the same parameters as in Equation 4, reference may be made to the foregoing. W _T denotes a parameter weighting matrix of the behavioral relationship (ie, behavioral classification) of the same target in an adjacent frame, and e ^a _(vi,vj) denotes a behavioral classification (specifically, it may be a feature vector of a behavioral classification), that is, It represents the behavioral characteristics of the same target in adjacent frames. 11 , for the same target “person” included in adjacent frames, the corresponding behavioral relationship of the person is “opening”; For the scenegraph of the first frame of the image, the corresponding node "oven" (ie, node i) in this scenegraph and the node "oven" in the scenegraph of the second frame (ie, node j in the adjacent frame) Behavior classification is a feature vector of behavior “openends”; W _T is a learnable weight matrix, with different weight values for different behavior classifications. A _vi,vj is an attention parameter matrix that can assign the different weights described above to different targets. As shown in Fig. 7b, when updating the characteristics of the "dog" node, the "human" node has a closer relationship with the "dog" node than the "skateboard" node, and a higher weight is given to the "human" node. is granted

Equation 5 above can be understood more intuitively, that is, a graph convolution network is applied to features vi of different objects (ie, target nodes), such as the node “person” shown in FIG. 11 . Then, the updated features (i.e. graph convolution features) include features vi of the target itself, features of some related objects of the target (eg oven, pizza), and features of target nodes in adjacent frames. do.

The manner of obtaining graph convolution features provided by an embodiment of the present disclosure is: (1) when updating node features, the attention (ie, attention weight A _vi,vj ) is added, that is, updating the features It is different from the existing graph convolution feature extraction method in that different weights can be given to its neighboring nodes. As in the example above, when updating the characteristics of the "dog" node, the "human" node has a closer relationship with the "dog" node than the "skateboard" node, and the "human" node is given a higher weight do. (2) For two adjacent objects, the acquisition method is the existing graph convolution feature extraction method in that the features can be updated using different weight parameter matrices according to the time difference and the difference between the subject and the object. different from For example, in the relationship of a "person" node and a "skateboard" node, "person" is a subject and "skateboard" is an object. When updating the “person” node, the weight parameter matrix is W _sub , and when updating the “skateboard” node, the weight parameter matrix is W _obj .

In a practical application, a node representing an attribute feature in a scene graph and a node representing a target feature of the same target may be combined, that is, a node in the scene graph may be a node representing both the target feature and the attribute feature. Through the above alternative method of the present application, the graph convolution characteristics of each node in the scene graph of each image may be obtained through the graph convolution network. The convolutional features of the image are convolutional features of all nodes included in the scene graph of the image. Additionally, a node representing attribute characteristics in the scene graph and a node representing a target characteristic of the same target may be combined, ie, nodes in the scene graph may be target nodes and attribute nodes. In this case, graph convolution characteristics of each target node and each attribute node may be obtained. When obtaining graph convolution features of nodes through the above alternative methods, if one or more parameters in the above equation do not exist for a particular node, a pre-set value such as a zero vector or other pre-set feature vectors this can be used

In an alternative embodiment of the present disclosure, if the characteristic information of multimedia data includes at least two of a local visual characteristic, a semantic characteristic, a spatial-temporal visual characteristic, and a global characteristic, based on the extracted characteristic information, the multimedia data text The generating of the caption may include: determining a weight of each characteristic information; assigning a weight to each characteristic information based on the weight of each characteristic information; and generating a text caption of the multimedia data based on the weighted characteristic information.

In actual application, for different types of multimedia data (such as different videos and different images), the importance of the characteristic information of each classification is likely to be different, and since the characteristic information of different classifications may have different weights, Different features play different roles, and thus, the solution of an embodiment of the present disclosure may be adaptable to generating captioning information of different videos; That is, for different videos, different characteristic information may each play different roles.

Alternatively, a feature selection network may be used when determining a weight for each type of feature information. By training the feature selection network, the network can select specific characteristic information to generate captioning information of multimedia data for different multimedia data, i.e., for a given multimedia data, to different characteristic information through the characteristic selection network. It is possible to determine the respective weights for

As an example, FIG. 12 illustrates a schematic diagram of a feature selection network. As shown in FIG. 12 , each feature information in this example is a graph convolution feature (V _GCN ), a spatial-temporal visual feature (V _3d V ) and a semantic feature (V _SF ). In the above example, a specific implementation of the feature selection network can be expressed by the following equation (6).

Here, a _t represents the set of weight values output by the feature selection network at time t, that is, the weight of each feature information at time t, and E _{1: t-1} is the number of words from time 1 to time t-1. Represents embedding, that is, when the t-th word of video captioning information is decoded, the feature vectors of the first t-1 words have already been decoded, and W _3d , W _GCN , W _SF and We are the network are the parameter weight matrices of Each of the features is transformed and added to the same dimensions using parametric weight matrices; After passing through the nonlinear layer tanh, it _{is transformed into a 3*l vector using a} ^{W a T} parameter weight matrix, and then finally normalized to a softmax. Each dimension represents the weight of a different feature, the sum of which is 1. The intuitive meaning of this formula is to obtain an attention weight of each feature by performing an attention operation on each feature.

As in the example shown in FIG. 12 , the weights of the spatio-temporal visual feature, graph convolution feature, and semantic feature are 0.3, 0.2 and 0.5, respectively.

It can be understood that the above time point l, time point t-1, and time point t are all relative time concepts. They are the relative decoding times obtained by the decoder decoding the l-th word, the t-1 th word and the t-th word in the video captioning information when the decoder decodes and outputs the video captioning information. The weight of each characteristic information at each time other than time 1 may be obtained based on each characteristic information and words decoded before the current time.

The solution provided by embodiments of the present disclosure uses a variety of different features to represent video information. In addition to the spatio-temporal scene graph feature (ie, graph convolution feature), it can use spatio-temporal visual features and semantic features, namely three types of features. If graph convolution features are more related to relationships and properties between targets and spatial-temporal visual features are more related to temporal information, then semantic features are more related to the overall semantic information contained in the video. A feature selection network (eg, a feature selection gate) may select different features based on different videos. The output of the feature selection gate is a set of weight values a _t each representing the weights of the different features. For example, some videos are longer and have more temporal information, so the spatial-temporal visual feature weights higher. Some videos are shorter and contain more objects, so the relationship between objects and their properties is more important, and hence the weight of the graph convolution feature is higher.

After obtaining the weight of each characteristic information, as an alternative method in the subsequent processing based on each characteristic information, each characteristic information may be weighted using each weight, and the weighted characteristic will be used for subsequent processing. can A weight may be given to the aggregation of each characteristic information based on each weight to obtain an integrated characteristic, and the integrated characteristic will be used for subsequent processing. In the example shown in FIG. 12 . The integrated features (i.e., 0.3 * spatio-temporal visual features + 0.2 * graph convolution features + 0.5 * semantic features) can be used for subsequent processing, or by processing these features with weights separately Weighted features can be obtained, ie, 0.3 * spatio-temporal visual features, 0.2 * graph convolution features, and 0.5 * semantic features. By adaptively assigning different weights to different features, different types of features can exhibit different importance when generating text captions according to features of the multimedia data itself.

In another embodiment of the present disclosure, generating a text caption of multimedia data based on the extracted characteristic information includes: encoding the acquired characteristic information using a self-attention-based encoder; generating a text caption of multimedia data by inputting encoded characteristic information into a decoder, wherein when the multimedia data is an image, the self-attention-based encoder is a self-attention-based intra-frame encoder; When the multimedia data is a video, the self-attention-based encoder includes a self-attention-based intra-frame encoder and/or a self-attention-based inter-frame encoder.

In other words, the self-attention based intra-frame encoder can be used to encode the obtained characteristic information respectively (if weighting is to be performed, it is a weighted characteristic; if weighted aggregation is to be performed, it may be a weighted integrated feature), obtain deeper and more advanced feature information and input the encoded features into the decoder to generate a corresponding text caption; In the case of video, it may encode the obtained video features using a self-attention based inter-frame encoder, and input the encoded features to the decoder to generate a text caption of the video. The decoder may be a self-attention-based decoder. For example, in the case of an image, it may be an attention-based intra-frame decoder to better learn intra-frame information during decoding. In the case of video, it may be a self-attention-based intra-frame decoder and/or self-attention-based inter-frame decoder for better learning intra-frame information and/or inter-frame information during decoding, so that the video to obtain a more accurate text caption of

Each frame selected, for example, a text caption of a video obtained on the basis of graph convolution characteristics (which as can be understood may also be spatio-temporal visual characteristics and/or semantic characteristics and graph convolution characteristics) For , a vector relating to the graph convolution feature may be obtained. These feature vectors are input to a self-attention-based decoder to learn to obtain inter-frame information between frames.

When generating the text caption of the video based on the graph convolution features, the decoder outputs words that can be output at each moment according to decoder input and the graph convolution features and their output probability . As an example, the self-attention-based decoder may be implemented by a transformer decoder.

In the process of generating the text caption of the video, at a first instant the inputs of the decoder are a global feature and a start token, and the output of the self-attention based decoders is a set of probability values, each value representing the probability of a word, The word with the greatest probability is selected as output at the first instant. The input of the second instant is the global feature + the start token + the output of the first instant, and the word with the maximum probability is still chosen as the output. The input of the third instant is the global feature + the start token + the output of the first instant and the output of the second instant. This loop continues until the word with the greatest probability at a certain moment becomes a stop token, and then the loop is terminated, thereby obtaining a sentence sequence, i.e., the output of the self-attention based decoder is the last related to the video caption. It is a sequence of sentences.

For example, the vocabulary is a, b, c, d, e. The transformer decoder, from the vocabulary that can be output at a first instant, has an output probability of word a (eg 60%) and an output probability of word b (eg 40%), c, d, e (0%) of words can be printed out. At the second instant, the output probability of word c (eg 60%), the output probability of word d (eg 20%) and the output probability of word e (eg 20%), a, b(0%) ), etc. am. In this case, according to an embodiment of the present disclosure, the video captioning sentence may be generated by a greedy decoding algorithm, that is, the video captioning sentence is a word having the maximum output probability that can be output at each instant. are created by combining them in chronological order. However, the present disclosure is not limited thereto, and other decoding methods may be used to generate the video captioning sentences.

According to an embodiment of the present disclosure, a video caption sentence may be obtained by combining words having a maximum probability that can be output at every moment in chronological order until a probability of outputting a stop token is maximum. The role of the self-attention-based decoder is to learn inter-frame information. The self-attention-based decoder has a self-attention mechanism-based structure including a multi-head attention layer, a layer normalization layer, and a feed forward network layer. The self-attention-based decoder has advantages in that the training speed is faster, the number of parameters is smaller, and long-range dependencies can be easily learned compared to the decoder of the RNN structure.

As an example, FIG. 13A shows a schematic structural diagram of a self-attention-based codec model provided by an embodiment of the present disclosure. As shown in FIG. 13A , the self-attention-based codec model of the present disclosure is divided into two parts: a self-attention-based encoder and a self-attention-based decoder. Alternatively, the self-attention-based codec may be implemented as a converter codec. In this example, the characteristic information of the video is still described by taking the graph convolution characteristic as an example. As shown in FIG. 13A , the self-attention-based encoder of this example includes a multi-head attention layer, a feed forward network and a layer normalization layer. The self-attention-based decoder may include a masked multi-head attention layer, a multi-head attention layer, a layer normalization layer, and a feed forward network layer.

The encoder having a structure based on the self-attention mechanism shown in FIG. 13A may have a multi-block structure, and the structure of each block may be the same or different. The multi-block structure can be sequentially cascaded, that is, the output of the current block becomes the input of the next block. In an alternative solution, for example, the encoder may comprise 6 blocks with the same structure (one shown in FIG. 13a ), each block being a fully connected feed forward network and multi-head attention layer per location It mainly includes two parts of , and the feed forward network can be implemented by two linear prediction layers. The two linear prediction layers include ReLU activation operations. The multi-head attention layer and feed forward network in each block may correspond to respective layer normalization layers. Specifically, as shown in FIG. 13A , each block may include a multi-head attention layer, a layer normalization layer, a feed forward network layer, and a layer normalization layer in turn. Each block can be stacked to obtain an encoder. The input of the encoder is a graph convolution feature.

When encoding graph convolution features using a self-attention based encoder, encoder embedding may be performed on the graph convolution features. The purpose is to change the dimension of the feature information to be suitable for subsequent encoders to process. The characteristic information output after embedding is input to the encoder to be encoded.

The following uses the processing of the first block in the encoder as an example for illustrative description. The graph convolution feature is first processed in a multi-head attention layer, the output result is integrated with the output of encoder embedding (such as further processing), and layer normalization processing is performed on the integration result. The normalized result is processed through a feed-forward network, and then integrated with the output of the previous layer normalization layer (eg, additional processing), after which the layer normalization is performed again to obtain the output result of the first block. The output result of the first block is used as an input of the second block, and the encoding process is sequentially performed to obtain an output result of the encoder (ie, the output of the encoder in FIG. 13 ).

The decoder structure of the self-attention mechanism shown in FIG. 13A may also be a multi-block structure. The structure of each block may be the same or different. The structures of several blocks may be cascaded in sequence. For example, the decoder may include six blocks having the same structure, each block having a masked multi-head attention, multi-head self-attention corresponding to features, and three parts of the feed forward network. may include The multi-head attention and feed forward network in each block may each correspond to layer normalization layers. Specifically, the structure of each block may include a masked multi-head attention layer, a layer normalization layer, a multi-head attention layer, a layer normalization layer, a feed forward network layer and a layer normalization layer in turn, and each block is stacked to obtain a decoder. can get The decoder inputs in the figure are global feature information and word vectors. The feature vector of the target region extracted by the feature extraction network in each frame may be referred to as local features or local features. Thus, regional characteristics of several target areas can be obtained for this frame. Global features corresponding to a frame may be obtained by averaging local features, or global features may be obtained by other methods (such as weighting). In addition, predicted start tokens and word vectors may also be obtained during the iterative prediction process (in the case of the first prediction of iterative prediction, only the start token is obtained, and all word vectors to be input when training the model. can). Decoder embedding processing can be performed on the above decoder input (i.e., word vectors predicted during iterative prediction process, and global feature and start token), which adapts the dimension of feature information for subsequent decoder processing. to change it to Global characteristic information, start token, and word vector output after embedding processing may be input to the decoder for decoding processing.

The following uses the processing of the first block as an example for illustrative description. The global feature information, start token, and word vector output after the embedding process are processed by the masked multi-head attention layer, and the processed result is integrated with the output of decoder embedding (eg, addition processing), and then Layer normalization processing is performed. The normalized result and the result output by the encoder are processed together by the multi-head attention layer (if the encoder includes an inter-frame encoder, the output of that encoder is the result output by the inter-frame encoder; If the encoder has only an intra-frame encoder, the output of the encoder may be a result obtained by integrating the results output from the intra-frame encoders, for example, a result of concatenating the results output from the intra-frame encoders. can be), and then merged with the output of the previous layer normalization layer (eg, addition processing), and then layer normalization processing is performed. The normalized result is processed through a feed forward network and then integrated with the output of the previous layer normalization layer, and then subjected to layer normalization, and the processed result is the output of the first block. The output result of the first block is used as an input of the second block, and the decoding process is sequentially performed to obtain an output result of the decoder.

The output of the decoder is linearly transformed by the linear layer and then processed by the softmax layer, which is the word vectors that can be output at the current moment (ie the prediction of this iteration) and the output probabilities of words a and b are, of course, for outputting corresponding output probabilities such as words a and b. The decoder, the linear layer, and the softmax layer repeat the above iterative prediction process until the probability of outputting a static character is maximized, and the cap corresponding to the video according to the word vector obtained in each iteration Schinging information may be obtained.

The above examples can be understood as using the graph convolution feature as an example for illustrative purposes. In practical applications, in addition to the graph convolution characteristics, it may also include spatio-temporal visual characteristics and/or semantic characteristics of the video. In this case, the encoding process may encode each characteristic information, and the decoder may decode the characteristics obtained by integrating the encoded characteristics. In this case, the above feature selection network may be used to determine the weights of the encoded features, the encoded features are integrated based on weights as an output of the encoder, and the decoder is configured to determine the weights of the encoded features based on the integrated features. Get the text caption of the video. It is also possible to respectively process the encoded features by inputting them into different cross-attention layers of the decoder based on the weights, and the decoder obtains a text caption of the video.

In another embodiment of the present disclosure, generating a text caption from the multimedia data based on the extracted characteristic information includes: inputting the extracted characteristic information to a plurality of decoders, respectively; and generating a text caption of the multimedia data based on the decoding result of the decoders.

In order to provide decoding capability and improve expression capability of captioning information, in the solution of the embodiment of the present disclosure, when the encoded results are processed, a decoder-bank including a plurality of decoders is used to store the encoded results By individually decoding, the decoding capability of the decoder may be improved, and final text caption information may be obtained based on decoding results of the respective decoders. For example, a final output may be obtained by averaging decoding results of respective decoders. The decoder bank may include two or more decoders, and types of decoders included in the decoder bank are not limited in the embodiment of the present disclosure. For example, the decoder bank may include an LSTM-based decoder, a gate circulation unit-based decoder, a self-attention-based decoder, and the like, and outputs of each decoder may be averaged to obtain a final output result.

Through the test, when the number of decoders in the decoder bank increases from 2, the effect gets better, but when the number exceeds 4, the improvement of decoding performance is stable, and the increase in the number of decoders is the system Since it also increases the complexity of , in practical applications, the number of decoders needs to be selected according to a trade-off between performance and complexity. Alternatively, in general two or three decoders can be selected. For example, when used in an on-device system, two decoders may be selected, and when used in a cloud end, three or more decoders may be selected.

In connection with selecting each decoder in the decoder bank, it is possible to pre-train a number of decoder banks. The decoders included in different decoder banks may be different. The number of decoders in the decoder banks may be different. In practical application, among several decoder banks, the decoder bank that has the best effect on the verification set or the test set may be selected. When selecting a decoder bank, two aspects of decoding efficiency and decoding performance of the decoder bank can be considered.

When multiple decoders are used for their respective decoding, while training the decoder bank, in order to make the output results of the decoders close to ground-truth, a coherence loss is added as a constraint to the output result of the decoder. It is necessary to prevent the performance of different decoders of a decoder bank from being significantly different, so that the performance of the decoder bank is not inferior to that of a single decoder. It is assumed that the decoder bank has two decoders, the outputs of which are respectively two probability distributions p1 and p2. The coherence loss is defined as Equation 7 below:

Here, D _KL represents KL divergence.

In each of the alternative solutions in the embodiments of the present disclosure, while performing encoding or decoding, an attention-based neural network can be used because it can simultaneously derive the global dependence between these different inputs and target locations, So long-term dependencies can learn better, and this type of neural network allows for more efficient parallel computing while processing data. Additionally, for self-attention based encoders or decoders, in particular self-attention based decoders, multiple cross-attention layers may be stacked. This is because self-attention-based neural networks can learn association information between elements of the same feature vector well, and cross-attention (the core of which is multi-head attention) can learn association information between different feature vectors well. am. Therefore, by adding a cross-attention layer in a self-attention based decoder, the decoder can not only learn the associated features between elements of a feature vector well, but also the associated features between different features, so It can better handle many different types of features to obtain better captioning information.

As an example, FIG. 13B shows a schematic structural diagram of a self-attention-based decoder provided in an embodiment of the present disclosure. In this example, the encoder part may include a spatio-temporal visual encoder (a spatio-temporal feature extraction network) and a semantic encoder (a semantic prediction network). With these two encoders, the output may include the encoder output of spatio-temporal visual features and semantic features. When the decoder of FIG. 13A is used for decoding, the output of the semantic encoder is input to the semantic cross-attention layer, and the output of the temporal-spatial visual feature encoder is input to the temporal-spatial visual cross-attention layer. The masked self-attention layer can ensure that information from the previous instant to the next moment will not be received during training, and will mask the input at the next instant. The input of the masked self-attention layer may correspond to the decoder input shown in FIG. 13A including the start token, for example, it may be a feature vector processed by the decoder embedding layer.

In an alternative embodiment of the present disclosure, generating a text caption of multimedia data based on the extracted characteristic information includes: obtaining length information of a text caption to be generated; and generating a text caption of the video based on the length information and the extracted characteristic information.

In order to solve the problem that video captions or image captions of different lengths cannot be generated for users in the prior art, in the solution provided herein, it is possible to generate text captions of corresponding lengths by obtaining length information of text captions to be generated. and this is to meet the needs of various application scenarios. The length information is "long" (eg, the generated captioning information is more than 20 words), "medium" (eg, the generated captioning information is between 10-20 words), "short" (eg, generated The closed captioning information may be relative length information such as less than 10 words). The length information may be obtained from a user. For example, a prompt may be sent to the user requesting to generate long captioning information or short captioning information, and the user may grant corresponding commands according to the prompt. The length information may be obtained by analyzing the video. When the video is a real-time captured video, the video may be analyzed to determine a current application scenario, and different length information may be determined according to different application scenarios.

For this solution, while training the decoder, unlike the prior art, the start token of the decoder may be a start token containing length information. For each training sample, the start identifier may include a start token indicating a longer captioning needing to be generated or a shorter captioning needing to be generated and different start tokens corresponding to different sample captioning label information. have. When the decoder is trained based on the training sample, the decoder may learn a mapping relationship between the start token corresponding to the different length information and the corresponding length of the captioning information, and decoding will be performed based on the trained decoder. At this time, a start token corresponding to the length information may be used as a start token of decoding based on the obtained length information, thereby generating a video caption or image caption that meets the length requirement.

That is, in the solution of the embodiment of the present disclosure, during training, the length of the output captioning information is replaced by length information such as "short", "medium" or "long" for "BOS" (Begin of Sentence) in the existing method. to control In actual training, different length identifiers may be used for different length information. Specifically, when training outputs short captioning information, the start token is input as "short", the middle captioning information corresponds to a "medium" start token, and the long captioning information corresponds to a "long" start token. respond As such, the lengths of the sentences may correspond to “short”, “medium” and “long” respectively during training. During online use, according to different needs of the user, "short", "medium" or "long" may be input to obtain captioning information of different lengths.

Based on the methods in alternative embodiments of the present disclosure, intra-frame information of each frame or image in a video (semantic information of the video or image, spatial-temporal visual characteristics, etc. as well as the image It is possible to create more accurate text captions by analyzing in detail (such as objects, properties, and relationships) and making the most of the image information. From the foregoing description, it can be seen that, in actual applications, various different specific implementations may be selected according to actual application requirements, based on the method of generating video captioning information provided in the embodiments of the present disclosure.

Further, in the solution provided in the embodiment of the present disclosure, while extracting characteristic information of multimedia data, in addition to using a feature extraction network to extract features of each region of an image, relationships between features of regions You can add more encoders that learn (i.e. relational prediction networks). The encoder may be implemented by a self-attention-based encoder (eg, a transformer encoder), thereby improving the performance of feature encoding to increase the performance of obtaining video or image captioning information. Additionally, while obtaining captioning information in an embodiment of the present disclosure, it is possible to use a self-attention-based decoder (eg, a transformer decoder) without using a conventional RNN structure of the decoder. Compared to the conventional RNN, the self-attention-based decoder has advantages in that the training speed is fast, the number of parameters is small, and it is easy to learn long-distance dependencies.

The following uses videos as an example to describe a method for generating captioning information of multimedia data provided by embodiments of the present disclosure in combination with several alternative embodiments.

Example 1

14 shows a schematic flow diagram of a method for generating video captioning information provided by an alternative embodiment of the present disclosure; 14 , the method for generating video captioning information may include the following steps.

Step S301: You can select frames from the video.

Step S302: A scene graph is built separately for each of the frames.

Step S303: Using the graph convolution network, obtain graph convolution features of each frame based on the constructed scene graph.

Step S304: A text caption for the video is generated based on the obtained graph convolution feature.

Alternatively, after obtaining the graph convolution characteristics of each frame, a text caption of the video may be obtained based on the graph convolution characteristics. For example, the obtained graph convolution features may be input to a decoder, and a text caption for a given video may be obtained by decoding the obtained graph convolution features. Alternatively, a self-attention-based decoder may be used to generate a text caption of the video according to the graph convolution characteristics of each frame, but the present disclosure is not limited thereto.

Example 2

15 shows a schematic flow diagram of an alternative method for generating the video captioning information given in this example. As shown in FIG. 15 , this alternative implementation may include the following steps.

Step S1201: Frames can be selected from a given video, like 501 in FIG. 17A and 1001 in FIG. 17B.

Step S1202: Several target regions and their characteristics (eg, local features or local features) are obtained from each of the selected frames, as 502 in FIG. 17A and 1002 in FIG. 17B, using a feature extraction network . For each image frame, the Faster R-CNN algorithm can be used to extract target regions within each frame and features of the target regions.

Step S1203: As shown in the example of FIG. 8 , a relation prediction network is applied to the extracted regional features of each target region to obtain relational features between the target regions.

Step S1204: A scene graph for each image frame is built based on the obtained relationship features between the respective target regions, as shown in 503 in FIG. 17A .

Step S1205: A graph convolution network is used to obtain graph convolution features based on nodes and edges in the scene graph of each frame, as shown in 504 in FIG. 17A .

Step S1206: A text caption of the video is generated based on the graph convolution features. Alternatively, a self-attention based decoder may be used to learn inter-frame information of selected frames according to the obtained graph convolution characteristics to generate a text caption of a given video. For example, for each selected frame, a vector for the graph convolution feature may be obtained, and these feature vectors are trained to obtain inter-frame information between frames, as shown at 505 in FIG. 17A . to the self-attention-based decoder. A vector of graph convolutional features of each frame may be input to an inter-frame converter decoder, and based on the output of the decoder, a text caption of the video, i.e., “a man is putting pizza in the oven” is obtained. .

Example 3

16 shows a schematic flow diagram of an alternative method for generating the video captioning information given in this example. 15 and 16, the first three steps of this example (FIG. 16) are the same as in Example 2 (FIG. 15) above, and thus will not be repeated. The above example is different from Example 1 in that step S1304 is added to this example, that is, an attribute prediction network is applied to the extracted features of each target regions to obtain an attribute characteristic of each target regions. For example, a relational prediction network may be trained based on a visual genomic dataset, and then, as shown in the example of FIG. 9 , the attribute characteristics of each target region are obtained using the trained attribute prediction network. can be

Therefore, in step S1305, when constructing the scene graph of each frame, it may be concretely constructed based on the obtained attribute characteristics of each target area and the relational characteristics between each target area, and then step In S1306, a graph convolution feature of each frame is obtained according to the scene graph constructed based on the attribute feature and the relation feature.

It should be noted that the order of the steps S1303 and S1304 may be reversed or the steps S1303 and S1304 may be performed simultaneously.

After obtaining the graph convolution features of each frame, a text caption of the video may be generated according to the graph convolution features obtained through step S1307. A self-attention based decoder can be used to learn inter-frame information of selected frames to generate a text caption of a given video.

For example, as shown at 505 of FIG. 17A , the attention-based decoder (eg, the inter-converter decoder shown in the figure) may use inter-frame information to generate a text caption of the video according to graph convolution characteristics. can be used to learn

As another example, as shown in 1005 , 1006 , and 1007 of FIG. 17B , the obtained graph convolution features may be separately encoded, and the encoded features may be processed to obtain a target dimension of a feature vector. The cross link between the scene graph construction and graph convolution features in Fig. 17b indicates that the used scene graph may be a spatio-temporal scene graph, i.e., inter-frame information will be considered when building the scene graph. and, of course, indicates that the spatial scene graph can also be used. Specifically, an encoding operation on the graph convolution features of each frame may be separately performed using a self-attention-based intra-encoder (eg, the intra-frame converter encoder shown in FIG. 17B ). The function of the self-attention-based intra-encoder is to learn inter-frame information, that is, it is possible to additionally learn association information between objects in the frame using the self-attention mechanism. Alternatively, the structure of the self-attention-based intra-frame encoder is based on a self-attention mechanism. The structure includes a multi-head attention layer, a layer normalization layer and a feed forward network layer. Next, the output from the self-attention based intra-frame encoder is processed to obtain the target dimension of the feature vector for each frame.

For example, it is assumed that the dimension of the sequence output from the self-attention-based intra-decoder is T*C, where T represents the number of nodes in the scene graph, and C is the feature dimension of a feature vector corresponding to each node. , and the self-attention-based intra-frame encoder learns a relationship between an output sequence and other information using a self-attention mechanism and outputs the learned sequence. Here, the length of the output sequence is equal to T*C, which is the length of the input sequence. The output is averaged to obtain a 1*C dimension feature vector. In this way, a 1*C feature vector can be obtained for each frame.

An encoded feature vector may be obtained for each selected frame. These encoded feature vectors are input to a self-attention based intra-frame encoder (eg, the inter-transformer encoder shown in FIG. 17B ), and then re-encoded to obtain the target dimension of the feature vector.

Thereafter, based on the encoded characteristics, a self-attention based decoder (eg, the inter-frame converter decoder shown in FIG. 17B ) converts the inter-frame information to generate a text caption of the given video. used to learn The encoded features are input to a self-attention based inter-frame decoder to learn to obtain inter-frame information between frames, and a text caption of a given video is generated by learning the input features.

As another example, only the self-attention-based intra-frame encoder may be used to separately encode a graph convolution feature of each of the obtained frames, and input the encoded features to the decoder to generate a text caption of the video .. Alternatively, only the self-attention based inter-frame encoder may be used to encode the graph convolutional features of each of the obtained frames, inputting the encoded features into a decoder to provide a text caption of the video create That is, it may perform only operation 1005 of FIG. 17B or operation 1006 of FIG. 17 , or it may perform operation 1005 of FIG. 17B and operation 1006 of FIG. 17B together.

After performing a series of processing on a given video, a text caption of the given video can be implemented. As shown in FIG. 17B , a text caption “Man is putting pizza in the oven” may be created from the selected frames.

Example 4

18 shows a schematic flow diagram of an alternative method for generating the video captioning information given in this example. As shown in Fig. 18, comparing Figs. 18 and 16, it can be seen that this example is different from Example 3 in step S1505. In this example, when constructing the scene graph of each frame, temporal information is also considered. Specifically, it is possible to construct a spatial scene graph for each image based on the acquired attribute characteristics of each target region and the relational characteristics between each target region, and temporal information between the scene graphs for frames is added to obtain a spatio-temporal scene graph as in the example shown in FIG. 10 .

This example may be implemented based on example 1, that is, step S1504 may be omitted and the attribute characteristics of the target area may not be considered.

After obtaining the spatio-temporal scene graph of each frame, in step S1506, the graph convolution network is used to obtain the graph convolution characteristics of each frame, based on the nodes and edges in the spatio-temporal scene graph. . Then, in step S1507, a text caption of the given video is generated according to the obtained graph convolution characteristics. For example, a self-attention based encoder may be used to encode graph convolution features. According to the encoded graph convolution feature, a self-attention based decoder is used to learn inter-frame information to generate a text caption of the video.

Example 5

18 shows a schematic flow diagram of an alternative method for generating the video captioning information given in this example. Comparing FIGS. 19 and 16 , it can be seen that this example ( FIG. 19 ) is different from the above example 3 ( FIG. 16 ) in the following respects.

Step S1602: For each of the selected frames, feature extraction to obtain several target regions and features of each target region (ie, regional features or local features), and spatial-temporal visual features of the video network is used.

Compared with the steps of extracting local features in the above example, the features extracted by this step in this example may also include spatio-temporal visual features of the video. Alternatively, as shown in FIG. 20 , the spatio-temporal visual features may be obtained through a spatio-temporal feature extraction network.

Step S1603: Based on the selected frames, semantic features of the video are extracted through the semantic feature extraction network. As shown in FIG. 20 , semantic features may be obtained through a semantic prediction network based on frames.

Steps S1604 to S1607 correspond to obtaining relational features and attribute features in the previous example, building a scene graph (spatial scene graph or space-temporal scene graph), and extracting graph convolution features, and , which is not repeated here.

Step S1608: A text caption of the video is generated according to the graph convolution characteristics of each frame, the above spatial-temporal visual characteristics and semantic characteristics. Specifically, a plurality of decoders (decoder bank shown in FIG. 20 ) learn inter-frame information to generate a text caption of a video according to spatio-temporal visual features, semantic features and graph convolution features. can be used

Each decoder may be a self-attention based decoder or an RNN based decoder. Specifically, in order to learn to obtain inter-frame information between frames, spatial-temporal visual features, semantic features and graph convolutional features may be input to the decoder bank, and then The results are averaged to obtain the final decoding result. A text caption for a given video is generated by learning the input features.

The method for generating video captioning information provided in the present disclosure solves the problem of low accuracy of the existing video captioning algorithm due to ignoring intra-frame information, and proposes an improved video captioning technique. When the method provided herein is implemented, it may obtain features based on a graph convolution network, and obtain a text caption of a video based on a decoding output of a self-attention structure. Specifically, after obtaining a graph convolution feature, the graph convolution feature can be directly input to a self-attention-based decoder, so as to decode and output a text caption for a given video, which is the obtained graph An encoding operation may be further performed on the convolution, and the encoded features are input to a self-attention based codec for inter-frame encoding and decoding, thereby outputting a text caption of a given video.

The accuracy of captioning information generated by the present disclosure can be further improved by using a self-attention-based intra-encoder and a self-attention-based inter-frame encoder. Alternatively, it can separately encode the obtained graph convolution features of each frame using a self-attention-based intra-encoder, and integrate the encoded features to input to a decoder to generate a text caption of the video. have; Or it may use a self-attention based inter-frame encoder to encode the obtained graph convolution feature of each frame and input the encoded feature to the decoder to generate a text caption of the video. That is, an intra-frame encoder and an inter-frame encoder can be selectively used. The solution provided by the embodiments of the present disclosure may make full use of intra-frame information and/or inter-frame information, thereby generating more accurate text captions for a given video.

The above images are taken as examples below to illustrate some alternative implementations of generating image captioning information.

Example 6

21 is a schematic flowchart of a method for generating image captioning information according to an embodiment of the present disclosure. As shown in the figure, the method includes the following steps.

Step S10: characteristic information corresponding to the image is extracted.

Step S20: Captioning information corresponding to the image is obtained based on the extracted characteristic information.

The image may be obtained from a local storage or a local database as required, or may be received from an external data source (such as the Internet, a server, a database, etc.) via an input device or transmission medium.

Specifically, characteristic information corresponding to the image may be extracted through the feature extraction network. As an alternative method, local features of respective target regions may be extracted by a trained Faster R-CNN, for example, by averaging and pooling the feature graph of the Pool5 (ie, region of interest (RoI)) layer. ), the acquired feature vector may be selected as a feature.

After obtaining the characteristic information of the image, captioning information corresponding to the image may be obtained through decoders according to the extracted characteristic information. The specific structure of the decoder is not limited in the embodiment of the present disclosure. For example, the decoder may be implemented by a self-attention-based decoder (eg, a transformer decoder). Specifically, the decoder outputs words that can be output at each moment based on the extracted characteristic information and an input word vector (which may include a starting token and a word vector predicted during iterative prediction process) and the output of those words. You can output the probabilities (which can be normalized probabilities). As an alternative solution, the self-attention-based decoder may include a masked multi-head attention layer, a multi-head attention layer, a layer normalization layer and a feed forward network layer.

For example, the vocabulary is a, b, c, d, e. The decoder, from the vocabulary that can be output at a first instant, has an output probability of word a (eg 60%) and an output probability of word b (eg 40%), and a word with c, d and e (0%). can be printed out. At the second instant, the output probability of word c (eg 60%), the output probability of word d (eg 20%) and the output probability of word e (eg 20%), a, b(0%) ), etc. am. In this case, according to an embodiment of the present disclosure, the image captioning sentence may be generated by a greedy decoding algorithm, that is, the image captioning sentence is a word having the maximum output probability that can be output at each instant. are created by combining them in chronological order. Alternatively, according to another exemplary embodiment of the present disclosure, an image captioning sentence may be generated by the Monte Carlo sampling method, that is, based on the output probability of a word that may be output at each moment in Monte Carlo By performing sampling, image captioning sentences are generated.

Correspondingly, when generating captioning information of an image, by combining words having the maximum output probability that can be output at each moment in chronological order until the output probability of a static character is maximum, the image cap A shunting sentence may be obtained.

Alternatively, the step S10 may further include obtaining a global feature corresponding to the image.

Correspondingly, in step S20 , the step of obtaining the text caption of the image may be a step of obtaining the text caption of the image based on the obtained local features and global features.

In order to obtain a more accurate image caption, after obtaining local features of respective target regions of the image, global features of the image may be further obtained based on the local features, which is based on the local and global feature information. to obtain more accurate image captioning information.

Alternatively, the acquiring global features corresponding to the image includes: acquiring global features of the image based on local features of the image, or extracting global features based on the image through a feature extraction network may include.

Specifically, local characteristic information corresponding to each target candidate regions of the image may be extracted through the feature extraction network, and global characteristic information corresponding to the image may be obtained based on the local characteristic information. . Correspondingly, the decoder may be used to obtain captioning information corresponding to the image according to the local characteristic information and the global characteristic information. As an alternative method, the global characteristic information may be obtained by averaging local characteristic information corresponding to respective target candidate regions of the image. Alternatively, the global feature information may be obtained by applying a feature extraction network (eg, CNN) to the image, for example, it may be obtained by applying a feature extraction network (eg, CNN) to the respective layers (ie, respective channels) of the image. It can be obtained by extracting the feature maps of , through ResNet, averaging them, and pooling them.

In an alternative embodiment of the present disclosure, the local feature may include a local image feature and/or a local attribute feature, the global feature comprising a global image feature and/or a global attribute feature; Correspondingly, a global image feature corresponding to the image may be obtained based on the local image feature; and/or a global attribute characteristic corresponding to the image may be obtained based on a local attribute characteristic.

That is, the obtained local property information may include local text property information in addition to the local image property information. Therefore, when extracting local features through the feature extraction network, the feature extraction network may further include an attribute prediction network. The attribute prediction network may be a multi-label classification network. Alternatively, attribute prediction networks can be obtained using weakly-supervised training methods such as noisy-OR. In practical applications, attributes can be finely divided according to nouns, verbs, adjectives, relatively rare words and subjects. Each attribute is obtained based on a unique attribute prediction network (eg Multiple Instance Learning (MIL)). Finally, various attribute functions can be chained to obtain the final text attribute characteristics.

Alternatively, when the obtained local property information includes local image property information and local text property information, the obtained global property information may also include global image property information and global text property information. Similarly, the global image property information and global text property information may be obtained based on corresponding local property information, that is, global image property information corresponding to the image may be obtained based on local image property information, , global text property information corresponding to the image may be obtained based on local text property information. The global image property information and global text property information may also be extracted based on the image through a neural network. Of course, with respect to local property information and global property information, one of the local property information and the global property information may include image property information and text property information, and the other may include image property information or text property information. and can be set according to application requirements.

In an alternative embodiment of the present disclosure, obtaining the text caption of the image according to the local feature and the global feature comprises: individually acquiring each local feature to obtain the encoded local feature based on all extracted local features. encoding; and obtaining a text caption of the image based on the encoded local feature and global feature.

That is, after obtaining the local feature information of the image, the encoder may be used to encode the local feature information according to all the extracted local feature information to obtain the encoded local feature information. The encoder may be used to learn a relationship between local features of each target area region based on all extracted local feature information.

As an alternative solution, the encoder may be implemented by a self-attention based encoder, and the encoder may encode each local characteristic information based on all the extracted local characteristic information to obtain encoded local characteristic information. have. Correspondingly, when the decoder obtains the captioning information corresponding to the image according to the local feature information and the global feature information, the decoder is configured to: It is possible to output the words that can be output at each moment and the output probabilities of those words (which can be the probability that can be normalized) based on the predicted starting token and the word vector during the process) . The image captioning sentence may be obtained by combining words having the maximum output probability that can be output at each instant in chronological order until the output probability of the stop character becomes maximum.

As an alternative solution, the self-attention-based encoder may include a sequentially cascaded multi-head attention layer, a layer normalization layer and a feed forward network layer.

As an example, FIG. 22 shows a schematic structural diagram of a codec provided in an embodiment of the present disclosure. As shown in the figure, for the image to be processed (the image displayed in the lower right corner of the figure) to obtain input information of the encoder, the local features of each target region of the image are determined by a feature extraction network (eg, the Faster R-CNN shown in the figure) can be extracted. Specifically, the feature extraction network may partition the input image into several target candidate regions (ie, target region) and obtain a feature vector (ie, local feature) from each target candidate region, thereby in FIG. A plurality of feature vectors such as {v _{j } shown are obtained, and each vector in {v j} } contains one local feature information of one target candidate area (region corresponding to the rectangular block shown in the lower left of the figure). indicates.

Local feature embedding may be performed on the local feature information extracted by the feature extraction network, in order to change the dimension of the feature information to be suitable for subsequent encoder processing. The local characteristic information output after the embedding process is input to the encoder for encoding.

Alternatively, the encoder shown in FIG. 22 may have one or more block structures. When the encoder has a multi-block structure, the structure of each block may be the same or different. For example, it is assumed that the encoder may include six blocks having the same structure, and the six blocks are sequentially cascaded. Each block may mainly contain two parts per location: a fully connected feed forward network and a multi-head attention layer. Alternatively, the feed forward network may be implemented by two linear prediction layers. The two linear prediction layers may include ReLU activation operations. A multi-head attention layer and a feed forward network in each block may respectively correspond to layer normalization layers. As shown in FIG. 22 , each block in this example may include a multi-head attention layer, a layer normalization layer, a feed forward network layer, and a layer normalization layer in order. Each block can be stacked to obtain an encoder.

The following takes the processing of the first block as an example for explanation. The local feature information is first processed by the multi-head attention layer, and the output result is integrated with the output of local feature embedding (eg, additional processing) and then subjected to layer normalization processing. The normalized result is processed through a feed forward network and then integrated with the output of the previous layer normalization layer (eg, additional processing), and then subjected to layer normalization to obtain the output of the first block. The output result of the first block is used as an input of the second block, and the encoding process is sequentially performed, thereby obtaining an output result of the encoder (ie, the encoder output in FIG. 25 ).

와 같은 전역 특성 정보를 획득하기 위해 평균화될 수 있다. 추가로, 반복 예측 과정 동안에 예측된 시작 토큰과 단어 벡터가 또한 획득될 수 있으며 (반복 예측의 제1 예측이라면, 시작 토큰만이 획득된다), 그리고 상기 시작 토큰 및 상기 예측 벡터는 도 22에서 w 로 도시된다. 상기 모델을 트레이닝시킬 때에, 상기 샘플에 대응하는 모든 단어 벡터가 입력될 수 있다.Alternatively, as shown in FIG. 22 , the global feature information may be further obtained based on the local feature information extracted by the feature extraction network. For example, the local feature information is a feature vector shown in FIG. 22 .

may be averaged to obtain global characteristic information such as In addition, a predicted start token and a word vector may also be obtained during the iterative prediction process (if the first prediction of the iterative prediction, only a start token is obtained), and the start token and the prediction vector are w in FIG. is shown as When training the model, all word vectors corresponding to the sample may be input.

및 반복 예측 프로세스 그리고 상기 반복 예측 과정 내 예측된 단어 벡터 w에 대해, 디코더 임베딩이 수행될 수 있으며, 이는 특성 정보의 차원을 후속 디코더 처리에 적합하도록 변경하기 위한 것이다. 임베딩 이후 출력된 상기 전역 특성 정보, 시작 토큰 및 단어 벡터는 디코딩을 위해 상기 디코더에 입력될 수 있다.the global property information

and for the iterative prediction process and the word vector w predicted in the iterative prediction process, decoder embedding may be performed, for changing the dimension of characteristic information to be suitable for subsequent decoder processing. The global characteristic information, start token, and word vector output after embedding may be input to the decoder for decoding.

Alternatively, the decoder of FIG. 22 may have one or more blocks, and the structure of each block may be the same. For example, the decoder may include six blocks having the same structure. As an alternative structure, each block may mainly include a masked multi-head attention layer, a multi-head self-attention layer corresponding to the above feature, and a feed forward network. A multi-head attention layer and feed forward network in each block may correspond to respective layer normalization layers. The structure of each block may sequentially include a masked multi-head attention layer, a layer normalization layer, a multi-head attention layer, a layer normalization layer, a feed forward network layer, and a layer normalization layer. You can get a decoder by stacking each block.

, 시작 토큰 및 단어 벡터 w는 마스킹된 다중-헤드 어텐션 레이어에 의해 먼저 처리되고, 그 처리 결과는 인코더 임베딩(예: 추가 처리)의 출력과 통합된 다음 레이어 정규화 처리된다. 정규화된 결과 및 상기 인코더에 의해 출력된 결과는 다중-헤드 어텐션 레이어를 통해 함께 처리되며,그 후에 이전의 레이어 정규화 레이어의 출력과 통합(예: 추가 처리)되며, 그리고 그 후에 레이어 정규화 처리를 거친다. 상기 정규화된 결과는 피드 포워드 네트워크를 통해 처리된 다음 이전의 레이어 정규화 레이어의 출력과 통합되고, 그 후에 레이어 정규화 처리를 거치며, 최종적으로는 상기 처리된 결과는 상기 제1 블록의 출력이다. 상기 제1 블록의 출력 결과는 제2 블록의 입력으로 사용되며, 상기 디코더의 출력 결과를 획득하기 위해 상기 디코딩 과정이 차례로 수행된다.The following takes the processing of the first block as an example for explanation. the global property information

, start token and word vector w are first processed by the masked multi-head attention layer, and the processing result is integrated with the output of encoder embedding (eg, further processing) and then subjected to layer normalization. The normalized result and the result output by the encoder are processed together through a multi-head attention layer, then merged with the output of the previous layer normalization layer (eg, additional processing), and then subjected to layer normalization processing . The normalized result is processed through a feed-forward network, then integrated with the output of the previous layer normalization layer, and then subjected to layer normalization processing, and finally, the processed result is the output of the first block. The output result of the first block is used as an input of the second block, and the decoding process is sequentially performed to obtain an output result of the decoder.

The output result of the decoder is processed in the linear layer, and then processed in the softmax layer, which is processed at the current moment (i.e., this iterative prediction ) to output word vectors and corresponding output probabilities. The decoder, the linear layer, and the softmax layer repeat the above iterative prediction process until the stop character output probability is maximized. Captioning information corresponding to the input image may be obtained according to the word vector obtained in each repetition.

When the local feature includes a local image feature and local text attribute information, the used encoder may include an image feature encoder and an attribute feature encoder, two parts of the encoder are: local image feature information and local text attribute information Each is used to encode. Specifically, the image feature encoder may encode each local image property information according to all the extracted local image property information to obtain encoded local image property information, and the property feature encoder encodes the local text property information Thus, the encoded local text attribute may be obtained according to all the extracted local text attribute information. Accordingly, at this time, the decoder may be used to obtain captioning information corresponding to the image according to the encoded local image property information, the encoded local text property information, the global image property information, and the global text property information.

As another example, FIG. 23 shows a schematic structural diagram of a codec provided in an embodiment of the present disclosure. As can be seen from FIGS. 22 and 23 , the structure of the decoder of FIG. 23 may be similar to that of the decoder of FIG. 22 . In this example, the encoder may include an image feature encoder and an attribute feature encoder, and the structures of the two parts may be the same or different.

As shown in Fig. 23, for the image to be processed (the image shown in the lower left of the figure), several target candidate regions are shown in Fig. 22 through a feature extraction network (eg, Faster R-CNN shown in the figure). Local image feature vectors such as {v _{j } can be extracted, and each vector in {v j} } is a single target candidate region (a rectangular block marked in the image processed by Faster R-CNN shown in FIG. 23 ) represents the local image feature vector of the region corresponding to . It can also obtain multiple local text attribute vectors, such as _{{a j} } shown in Fig. 23, _{each vector in {a j} } is one target candidate area (in the rectangular block shown in the lower left of the image above) It represents the local text attribute vector of the corresponding region).

Local image feature embedding may be performed on the extracted local video feature information. The local image characteristic information output after embedding is performed is input to the image characteristic encoder for encoding. Region attribute attribute embedding may be performed on the extracted local text attribute information. Local text attribute information output after performing embedding is input to the attribute feature encoder for encoding.

In this example, the structures of the image feature encoder and the attribute feature encoder shown in FIG. 23 are described taking the structure of the encoder shown in FIG. 22 as an example. For example, they may contain 6 blocks each with the same structure. The structure of each block is shown in Fig. 23, and each block can be stacked to obtain an encoder. The processing flow of each block may refer to the description of the block structure of the encoder of FIG. 22 described above. When the image feature encoder and the attribute feature encoder perform feature encoding, the only difference is that the input of the image feature encoder is a feature obtained after local image feature embedding is performed on the local image feature information, and the attribute feature The input of the encoder is that the feature is obtained after the local attribute feature embedding has been performed on the local text attribute. After the encoding process, the output result of the encoder is obtained (ie, the output of the image feature encoder and the output of the attribute feature encoder in FIG. 23 ).

와 같은 전역 이미지 특성 정보를 획득하기 위해 평균화될 수 있으며, 그리고 상기 로컬 텍스트 속성 정보는 도 23 내 특징 벡터

와 같은 전역 텍스트 속성 정보를 획득하기 위해 평균화될 수 있다. 추가로, 반복 예측 과정 동안에 예측된 시작 토큰과 단어 벡터가 더 획득될 수 있다 (그것이 반복 예측의 제1 예측이면, 시작 토큰만이 획득된다). 상기 시작 토큰 및 상기 예측 단어 벡터는 도 23에서 w로 도시되며, 그리고 상기 모델을 트레이닝할 때에, 상기 샘플의 모든 단어 벡터들이 입력될 수 있다.In addition, the local image characteristic information is a feature vector in FIG.

may be averaged to obtain global image characteristic information such as

may be averaged to obtain global text attribute information such as Additionally, a predicted start token and a word vector may be further obtained during the iterative prediction process (if it is the first prediction of the iterative prediction, only the start token is obtained). The start token and the predictive word vector are shown as w in FIG. 23 , and when training the model, all word vectors of the sample can be input.

, 전역 텍스트 속성 정보

, 시작 토큰 및 반복 예측 과정에서 예측된 단어 벡터 w에 관하여 수행될 수 있으며, 이는 후속 디코더 처리에 적합하도록 상기 특성 정보의 차원을 변경하기 위한 것이다. 상기 전역 이미지 특성 정보, 전역 텍스트 속성 정보, 시작 토큰, 임베딩 후 출력되는 단어 벡터는 디코딩을 위해 상기 디코더에 입력될 수 있다.The encoder embedding is global image characteristic information.

, global text attribute information

, the start token and the word vector w predicted in the iterative prediction process, to change the dimension of the characteristic information to be suitable for subsequent decoder processing. The global image characteristic information, global text attribute information, start token, and word vector output after embedding may be input to the decoder for decoding.

As an alternative structure, the decoder structure shown in FIG. 23 may have one or more blocks. When the decoder has a multi-block structure, the structure of each block may be the same or different. For example, it may include six blocks of the same structure, the structure of each block being a masked multi-head attention layer, a multi-head attention layer corresponding to an image feature, and a multi-head attention corresponding to an attribute feature. It may include layers and feed forward networks. The multi-head attention and feed forward network in each block may each correspond to layer normalization layers. Specifically, as shown in FIG. 23, the structure of each block is a masked multi-head attention layer, a layer normalization layer, a multi-head attention layer, a layer normalization layer, a multi-head attention layer, a layer normalization layer, and a feed forward network layer. , and a layer normalization layer may be sequentially included. Each block can be stacked to obtain a decoder.

, 전역 텍스트 속성 정보

, 시작 토큰 및 반복 예측 프로세스에서 예측된 단어 벡터 w는 마스킹된 다중-헤드 어텐션 계층에 의해 먼저 처리되고, 그 처리 결과는 디코더 임베딩(예: 추가 처리)의 출력과 통합된 다음 레이어 정규화 처리를 거친다. 정규화된 결과 및 상기 이미지 특징 인코더에 의해 출력된 결과는 다중-헤드 어텐션 레이어를 통해 함께 처리되며, 그 후에 이전의 레이어 정규화 레이어의 출력과 통합(예: 추가 처리)되며, 그리고 그 후에 레이어 정규화 처리를 거친다. 정규화된 결과 및 상기 속성 특징 인코더에 의해 출력된 결과는 다중-헤드 어텐션 레이어를 통해 함께 처리되며, 그 후에 이전의 레이어 정규화 레이어의 출력과 통합(예: 추가 처리)되며, 그리고 그 후에 레이어 정규화 처리를 거친다. 상기 정규화된 결과는 피드 포워드 네트워크를 통해 처리 된 다음 이전의 레이어 정규화 레이어의 출력과 통합되고, 그 후에 레이어 정규화를 거친다. 상기 처리 결과는 상기 제1 블록의 출력 결과이다. 상기 제1 블록의 출력 결과는 제2 블록의 입력으로 사용되며, 상기 디코더의 출력 결과를 획득하기 위해 상기 디코딩 과정이 차례로 수행된다.The following takes the processing of the first block as an example for illustration: the global image property information

, global text attribute information

, the start token and the word vector w predicted in the iterative prediction process are first processed by the masked multi-head attention layer, and the processing result is integrated with the output of the decoder embedding (eg, further processing) and then subjected to layer normalization processing. . The normalized result and the result output by the image feature encoder are processed together through a multi-head attention layer, and then integrated (eg, further processed) with the output of the previous layer normalization layer, and then layer normalization processing go through The normalized result and the result output by the attribute feature encoder are processed together through a multi-head attention layer, and then integrated (eg, further processed) with the output of the previous layer normalization layer, and then layer normalization processing go through The normalized result is processed through a feed-forward network, then integrated with the output of the previous layer normalization layer, and then subjected to layer normalization. The processing result is an output result of the first block. The output result of the first block is used as an input of the second block, and the decoding process is sequentially performed to obtain an output result of the decoder.

The output result of the decoder is processed in the linear layer, and then processed in the softmax layer, which is processed at the current moment (i.e., this iterative prediction ) to output word vectors and corresponding output probabilities. The decoder, the linear layer, and the softmax layer repeat the above iterative prediction process until the output probability of the stop character is maximized. Captioning information corresponding to the above input image may be obtained according to the word vector obtained in each repetition.

From the description of the method for generating image captioning information in the above-described alternative embodiment, it is understood that the generating method can be specifically implemented by an image captioning model, that is, an image is attached to the image captioning model. can be input, and it can be seen that the text caption of the image can be obtained based on the model output. The specific neural network structure of the image captioning model is not limited in the embodiments of the present disclosure. For example, a codec network structure including a self-attention-based encoder and a self-attention-based decoder illustrated in FIG. 22 or 23 may be used, but is not limited thereto.

It can be understood that the solutions in the above examples are merely examples of some alternative manners of the present disclosure, and are not used to limit the solutions of the present disclosure. Additionally, the above examples are suitable for generating captioning information of the image and are also suitable for generating captioning information of the video. The generation of captioning information of the image is different from the generation of video captioning information in the following respects: When generating the captioning information of the image, since there is only one image, inter-frame information, that is, the temporal information described above There is no need to consider information between adjacent videos like edges and inter-frame encoders.

From the foregoing description of the method for generating captioning information of multimedia data provided in each alternative embodiment of the present disclosure, generating captioning information of multimedia data may be specifically implemented using a multimedia data captioning model. can be known For videos, the multimedia data captioning model is a video captioning model, and for images, the multimedia data captioning model is an image captioning model. The video captioning model and the image captioning model may be different models or may be the same model. That is, it can be a model suitable for generating image captions as well as for generating video captions. Alternatively, the video captioning model may be an RNN-based model or another network structure-based model such as a Transformer-based model. In actual application, the specific structure of the model may be set according to actual needs, which is not limited in the embodiments of the present disclosure.

For a video for which video captioning information is to be obtained, a video or frames selected from the video may be input to a video captioning model, and a text caption of the video may be obtained based on the input of the video captioning model. . The specific model structure of the video captioning model is not limited in the embodiments of the present disclosure. An original video captioning model can be trained using video samples, and the trained video captioning model is used to generate video captioning information.

Specifically, in an alternative embodiment of the present disclosure, the text caption of the multimedia data is obtained by using a multimedia data captioning model, and the multimedia data captioning model is obtained by training in the following manner: training sample , wherein the training samples include first sample multimedia data with caption labels; train an original captioning model based on the first sample multimedia data until the model loss function converges; Then, the trained captioning model is taken as the multimedia data captioning model.

for a video captioning model, the sample multimedia data is a sample video, and the caption label is a video caption label; It can be understood that for the image captioning model, the sample multimedia data is a sample image, and the caption label is a sample image caption label. The specific form of the model loss function may be set according to actual needs. For example, a model loss function commonly used when training a video captioning model or an image captioning model may be selected. During training, whether the value of the model loss function represents a difference between captioning information and captioning label information of multimedia data predicted by the model, or whether the predicted captioning information meets other preset termination conditions indicate whether Through continuous training, the multimedia information predicted by the model may be close to the captioning label information or meet other preset conditions.

In order to improve the accuracy of the generated captioning information of multimedia data, FIG. 24 illustrates a method of training a multimedia data captioning model provided in an alternative embodiment of the present disclosure. As shown in the figure, the training sample of the training method also includes a second sample of multimedia data without caption labels. The model loss function includes a first loss function and a second loss function. When training the original captioning model based on the first sample multimedia data, the method may include the following steps S201 to S203.

Step S201: A preset multimedia data captioning model is trained based on the first sample multimedia data to obtain a value of a first loss function, and the captioning model is trained based on the second sample multimedia data to obtain a second loss function Get the value of the function.

Specifically, in an embodiment of the present disclosure, the first sample multimedia data with captioning labels and the second sample multimedia data without captioning labels may be used to train a preset video captioning model together.

Sources of the first sample multimedia data and the second sample multimedia data are not limited in the embodiment of the present disclosure. Taking a video as an example, the original video caption corresponding to the first sample video data may be manually labeled by a descriptor as in the video shown in FIG. 25 , and the descriptor may say "a child is cleaning the ground" for the video. You can label the video caption . The second sample video data may be any acquired video without a video caption, for example, a video acquired from a video website or a video shot by a user. Specific forms of the first loss function and the second loss function are not limited in the embodiments of the present disclosure, and may be set according to actual application requirements.

Step S202: A model loss function value of the captioning model is obtained based on a value of a first loss function and a value of a second loss function.

Alternatively, for different loss functions, each function may have its own weight, so the importance of the different loss functions in the training process is different. For example, since the first multimedia data has an original captioning label and the second sample multimedia data does not have an original captioning label, the captioning of the first sample multimedia data (ie, the original captioning label) The label information is very accurate, and the weight of the first loss function may be greater than the weight of the second loss function. When different loss functions have their own weights, the final loss function of the multimedia data captioning model may be determined based on a weight corresponding to each loss function. For example, the final loss function may be a weighted sum (weighted sum) of respective loss functions.

That is, the step of obtaining a value of the model loss function (also referred to as a final loss function) based on the value of the first loss function and the value of the second loss function may include the following steps.

It can obtain the value of the corresponding target first loss function based on the preset weight of the first loss function, and obtain the value of the corresponding target second loss function based on the preset weight of the second loss function have.

The sum of the value of the target first loss function and the value of the target second loss function is taken as a value of the final loss function.

Specifically, the value of the final loss function can be calculated by the following Equation 8:

Here, ε is a hyperparameter. In this example, J _label (θ) is the first loss function and J _unlabel (θ) is the second loss function. The weight of the first loss function may be set to l and the weight of the second loss function is ε. As such, the product of the primary loss function and the corresponding weight is the target primary loss function, and the product of the secondary loss function and the corresponding weight is the target target secondary loss function. The sum of the target first loss function and the target second loss function is a final loss function.

Step S203: A captioning model is trained based on the value of the final loss function until the final loss function converges to obtain a trained multimedia data captioning model.

Specifically, after obtaining the final loss function of the video captioning model, the model parameters of the video captioning model are updated based on the final loss function until the final loss function converges based on the minimum value, so that the trained video Acquire a captioning model. A final loss function of the video captioning model is determined by the first loss function and the second loss function. The final loss function that converges on the basis of the minimum may be such a function that converges on the basis of the minimum, or it may be a first loss function and a second loss function that converge simultaneously on the basis of the minimum.

In an embodiment of the present disclosure, when the first sample multimedia data having a captioning label is received, a preset multimedia data captioning model is trained based on the first sample multimedia data and the captioning label, so that the first loss function get the value of When the second sample multimedia data without a caption label is received, the captioning model is trained based on the second sample multimedia data to obtain the second loss function value, and a final loss function of the multimedia data captioning model is obtained. Values are obtained based on the first loss function and the second loss function, and a multimedia data captioning model is trained based on the final loss function until the final loss function converges based on the minimum value, which To obtain a multimedia data captioning model.

Through the above manner, in addition to training the multimedia data captioning model using sample video data with captioning labels, this alternative embodiment of the present disclosure provides a captioning method for training the video captioning model. The sample multimedia data without labels can also be used simultaneously. In particular, when the amount of the sample multimedia data is large, the labor and time cost required for labeling the caption information on the sample multimedia data can be greatly reduced. In addition, as the amount of sample multimedia data increases, the accuracy and precision of the multimedia data captioning model is also improved. Additionally, the algorithm in an embodiment of the present disclosure may be applied to different models, such as the RNN-based model or the transformer-based model. This method is a general training method.

In an alternative embodiment of the present disclosure, in step S201, obtaining a value of the first loss function based on the first sample multimedia data includes: inputting the first sample multimedia data into a video captioning model to predict target captioning obtaining information; and obtaining a value of the first loss function based on the target captioning information and the corresponding captioning label.

The value of the first loss function represents a difference between the target captioning information obtained based on the model output and the corresponding labeled captioning information.

As an example, FIG. 26 shows a schematic diagram of a method for training a multimedia data captioning model provided in an embodiment of the present disclosure. This example is explained using a video as an example. The labeled data shown in the figure corresponds to video data in the first sample video data, and the unlabeled data shown in the figure corresponds to the video data in the second sample video data. The training method is described below with reference to FIG. 26 .

26 , the labeled video data V may be input to a video captioning model M, wherein the video data is analyzed and processed by the video captioning model to generate a corresponding target video caption, the Afterwards, the value of the first loss function is calculated based on the original video caption (corresponding to the label y in FIG. 26 ) and the target video caption in the first sample video data. In this example, the first loss function may be a cross entropy loss function, and the cross entropy loss function is shown in Equation (9).

(y_t|y_1:t-1,V)는 현재 순간에 상기 모델에 의해 예측된 단어가 대응하는 라벨링된 단어일 확률을 나타낸다. 상기 손실 함수의 의미는, 현재 시간(시점)의 입력이 현재 시간(시점) 이전의 각 시간에서 맞는(correct) 단어일 때, 현재 시간의 출력이 맞는 단어일 확률이 최대화된다는 것이다.where J _label (θ) denotes the cross-entropy loss, θ denotes the model parameters of the video captioning model, t denotes the current moment, T denotes the maximum moment, and y _t denotes the actual measurement ( ground-truth), y _1:t -1 represents the actual measurement corresponding to the time point 1 to t-1, V represents the video, and p _*?* represents the probability that the output word is actually measured . Specifically, p

(y _t| y _1:t-1 ,V) represents the probability that the word predicted by the model at the current moment is the corresponding labeled word. The meaning of the loss function is that when the input of the current time (time point) is a correct word at each time before the current time (time point), the probability that the output of the current time is a correct word is maximized.

For example, the video shown in FIG. 25 is analyzed and processed by the video captioning model. _{Assuming that the current time t = 2, the word y 0} output at the initial time t = 0 is "a", and the word y ₁ output at the time t = 1 is "child", the current time t = 2 and the time t _{It is assumed that when the word y 1} "child" output at = 1 is the correct word, _{the probability of the output y 2} output "is" is maximized.

A video caption obtained by analyzing and processing the video shown in FIG. 25 through the video captioning model is "a child is sweeping the ground", and then the video captioning model is "a child is sweeping the ground" and Assume training is based on the original video caption "a child is cleaning the ground".

In an actual application, an embodiment of the present disclosure may preset a lexicon, from which a word to be output at each moment is determined. _{The word y 0} output at the initial time point t = 0 may be determined based on the start token of the video. For example, for the video shown in FIG. 25 , the word y ₀ “a” output at time t = 0 is determined based on the start token of the video. Of course, in an actual application, other methods may be used to determine the first word of the video caption, which is not limited in the embodiment of the present disclosure.

In an alternative embodiment of the present disclosure, training a captioning model based on the second sample multimedia data to obtain a value of the second loss function comprises: the second sample multimedia data to obtain a third sample multimedia data. performing data augmentation on the sample multimedia data at least once; inputting the second sample multimedia data into the captioning model to obtain at least one multimedia captioning; determining a score of each multimedia caption based on the second sample multimedia data and the third sample multimedia data; and obtaining a value of the second loss function based on the scores of the multimedia captioning.

That is, when a model is trained based on the second sample multimedia data without a caption label, the sample multimedia data may be augmented to obtain the third sample multimedia data; Based on the third sample multimedia data and the second sample video data, a score of each captioning information obtained through a model based on the second sample multimedia data is determined, and a second score is determined based on the respective scores. Get the value of the loss function.

Alternatively, for example, the second sample multimedia data may be input into the captioning model to obtain first and second captioning information. Based on the second sample multimedia data and the third sample multimedia data, a first score of the first captioning information and a second score of the second captioning information are determined, and the value of the second loss function is the It is obtained based on the first score and the second score.

In the solution of the embodiment of the present disclosure, when the multimedia data captioning model is trained using sample multimedia data without a caption label, the second sample multimedia data is K(K) to obtain the third sample multimedia data. It goes through data augmentation in turn. The captioning model is trained based on the second sample multimedia data and the third sample multimedia data. Since the second sample multimedia data and the third sample multimedia data are the same or similar, the captioning information of the second sample multimedia data and the captioning information of the third sample multimedia data must also be the same or similar, so this way A second loss function may be calculated based on , and a captioning model is trained based on this function, thereby further improving the accuracy and precision of the captioning model.

In an alternative embodiment of the present disclosure, the step of inputting the second sample multimedia data into the multimedia data captioning model to obtain the corresponding first captioning information and the second captioning information specifically includes: the second sample multimedia data inputting to the captioning model, and determining first captioning information from an output result of the captioning model through a greedy algorithm; and/or inputting the second sample multimedia data to the captioning model, and determining video captioning information from an output result of the captioning model based on the probability sampling.

A greedy algorithm (also known as a greedy search) is about always making the best choice when solving a problem. That is, instead of considering the overall optimal solution, in a sense it creates a local optimal solution.

Alternatively, taking a video as an example, the first captioning information is a first video caption, and the second captioning information is a second video caption. The formula for obtaining the first video caption c _g through the greedy algorithm may be as shown in Equation 10

where J _label (θ) denotes the cross-entropy loss, θ denotes the model parameters of the video captioning model, t denotes the current moment, T denotes the maximum moment, and y _t denotes the actual measurement ( ground-truth), y _1:t -1 represents the actual measurement corresponding to the time point 1 to t-1, V represents the video, and p _θ represents the probability that the output word is actually measured. Specifically, p _θ (y _t| y _1:t-1 ,V) represents the probability that the word predicted by the model at the present moment is the corresponding labeled word. The meaning of the loss function is that when the input of the current time (time point) is a correct word at each time before the current time (time point), the probability that the output of the current time is a correct word is maximized.

For example, the video shown in FIG. 25 is analyzed and processed by the video captioning model. _{Assuming that the current time t = 2, the word y 0} output at the initial time t = 0 is "a", and the word y ₁ output at the time t = 1 is "child", the current time t = 2 and the time t _{It is assumed that when the word y 1} "child" output at = 1 is the correct word, _{the probability of the output y 2} output "is" is maximized.

A video caption obtained by analyzing and processing the video shown in FIG. 25 through the video captioning model is "a child is sweeping the ground", and then the video captioning model is "a child is sweeping the ground" and Assume training is based on the original video caption "a child is cleaning the ground".

In an actual application, an embodiment of the present disclosure may preset a lexicon, from which a word to be output at each moment is determined. _{The word y 0} output at the initial time point t = 0 may be determined based on the start token of the video. For example, for the video shown in FIG. 25 , the word y ₀ “a” output at time t = 0 is determined based on the start token of the video. Of course, in an actual application, other methods may be used to determine the first word of the video caption, which is not limited in the embodiment of the present disclosure.

In an alternative embodiment of the present disclosure, training a captioning model based on the second sample multimedia data to obtain a value of the second loss function comprises: the second sample multimedia data to obtain a third sample multimedia data. performing data augmentation on the sample multimedia data at least once; inputting the second sample multimedia data into the captioning model to obtain at least one multimedia captioning; determining a score of each multimedia caption based on the second sample multimedia data and the third sample multimedia data; and obtaining a value of the second loss function based on the scores of the multimedia captioning.

That is, when a model is trained based on the second sample multimedia data without a caption label, the sample multimedia data may be augmented to obtain the third sample multimedia data; Based on the third sample multimedia data and the second sample video data, a score of each captioning information obtained through a model based on the second sample multimedia data is determined, and a second score is determined based on the respective scores. Get the value of the loss function.

Alternatively, for example, the second sample multimedia data may be input into the captioning model to obtain first and second captioning information. Based on the second sample multimedia data and the third sample multimedia data, a first score of the first captioning information and a second score of the second captioning information are determined, and the value of the second loss function is the It is obtained based on the first score and the second score.

In the solution of the embodiment of the present disclosure, when the multimedia data captioning model is trained using sample multimedia data without a caption label, the second sample multimedia data is K(K) to obtain the third sample multimedia data. It goes through data augmentation in turn. The captioning model is trained based on the second sample multimedia data and the third sample multimedia data. Since the second sample multimedia data and the third sample multimedia data are the same or similar, the captioning information of the second sample multimedia data and the captioning information of the third sample multimedia data must also be the same or similar, so this way A second loss function may be calculated based on , and a captioning model is trained based on this function, thereby further improving the accuracy and precision of the captioning model.

In an alternative embodiment of the present disclosure, the step of inputting the second sample multimedia data into the multimedia data captioning model to obtain the corresponding first captioning information and the second captioning information specifically includes: the second sample multimedia data inputting to the captioning model, and determining first captioning information from an output result of the captioning model through a greedy algorithm; and/or inputting the second sample multimedia data to the captioning model, and determining video captioning information from an output result of the captioning model based on the probability sampling.

A greedy algorithm (also known as a greedy search) is about always making the best choice when solving a problem. That is, instead of considering the overall optimal solution, in a sense it creates a local optimal solution.

Alternatively, taking a video as an example, the first captioning information is a first video caption, and the second captioning information is a second video caption. A formula for obtaining the first video caption c _g through the greedy algorithm may be shown in Equation (10):

Here, c _g (t) (t = 1, 2,..., T) represents a word output at the current time point t; Alternatively, c _g (t)=(argmax _y∈Y (p _θ (c _g,1:t-1 ,V)))), where V denotes the second sample video data, and c _{g,1: t-1} represents a sequence of words output from the initial time point to the time point t-1, that is, words output before the current time point. In this case, c _g (t) indicates that the word having the maximum probability at the current time t is selected as the word output at the current time, and the output probabilities of each candidate word at the current time are the current time and each time before the video V. It is determined based on the words output from the fields, and the final output word at the current time is the word with the highest probability among the candidate words.

After the words output at each time point are obtained, the words output at each time point are sorted according to the output order to obtain _{the first video caption c g .}

In the case of the second video captioning information, the probability sampling refers to that each unit of a sample of the survey population is selected with the same probability, also referred to as Monte Carlo sampling or probability sampling. Probabilistic sampling is sampling in which samples are sampled based on probability theory and the principle of randomness, which causes every unit of a population to be sampled with a known non-zero probability. The probability that a population unit will be sampled is specified through the sampling design, and can be achieved through some randomization operation, although random samples generally do not completely match the population.

Alternatively, the formula for obtaining the second video caption c _s based on the probabilistic sampling can be shown as Equation (11):

Here, c _s (t)(t = 1,2,..., T) represents the word output ut at the current time point t; Alternatively, c _s (t)=(multinomial _y∈Y (p _θ (c _s,1:t-1 ,V)))), where c _s,1:t-1 is from the initial time to the time t- A sequence of words output up to 1, that is, words output before the current start, is represented, and V represents the second sample video data. At this time, c _s (t) is sampled according to the output probability of each word at the current time, and the sampling result is used as an output, and the output probability of each candidate word corresponding to the current time is the current time and video V before It is determined based on the words output at each time point, and the final output word at the current time point is a Monte Carlo sampled from the respective output probabilities.

Taking the example in FIG. 26 as an illustrative example, for the second sample video data, that is, unlabeled data shown in FIG. 26 , data enhancement is performed on a third sample (such as augmented video data V′ in FIG. 26 ). It may be performed K times to obtain video data. The data augmentation method may be to randomly remove frames in the video or to perform transformation such as rotation and cropping at each frame in the video. It may be another way for data augmentation. In actual application, all methods of performing data augmentation on video data are applicable to the embodiments of the present disclosure, and the embodiments of the present disclosure are not limited thereto. The second sample video data is input to the video captioning model M to obtain _{corresponding first video caption c g} and second video caption c _{s .} For example, the first video caption c _g may be obtained by Equation 2 above, and the second video caption c _s may be obtained by Equation 3 above.

Specifically, for example, by using a video captioning model to analyze and process videos having the same content shown in Fig. 25 but no original captioning information, words corresponding to 5 times (output results) are can be obtained as: "a (corresponding to c _g (1))", "child (corresponding to c _g (2))", "is (corresponding to c _g (3))", "cleaning (c _{equivalent to g} (4))", "sweeping (corresponding to c _g (4))", "organizing (corresponding to c _g (4))", "ground (corresponding to c _g (5))", where At the time c _g (4), the output probabilities of the three candidate words are 0.5 (cleaning), 0.2 sweeping), and 0.3 (organizing), respectively, and without considering the output probabilities of other words, the output probabilities are The maximum "cleaning" _{is taken as the output word at time c g} (4), and the final c _g generated based on the greedy algorithm is "A child is cleaning the ground".

As another example, following the example above, the _{three candidate words determined at time c g} (4) are “sweeping”, “organizing” and “cleaning”. The output probability of "sweeping" is 0.2, the output probability of "organizing" is 0.3, and the output probability of "cleaning" is 0.5, without considering the output probability of other words, three video captions can be generated. , which is " a child is sweeping the ground", each with an output probability of 0.2. “a child is organizing the ground” with an output probability of 0.3, “a child is cleaning the ground” with an output probability of 0.5, and the final c _s generated based on probability sampling can be one of three video captions. have.

That is, in the above situation, the second sample video data is input to a video captioning model and an output result is obtained. Assuming that the video caption is generated 10 times for the output result based on the greedy algorithm, the obtained video caption may be "a child is cleaning the ground" all 10 times; Assuming that video captions are generated 10 times for the output result based on probabilistic sampling, the video caption of " a child is sweeping the ground " twice, the video caption of " a child is organizing the ground " 3 times, and " It is possible to acquire the video caption of "a child is cleaning the ground" 5 times.

In an alternative embodiment of the present disclosure, the step of obtaining a first score of the first captioning information and a second score of the second captioning information based on the second sample multimedia data and the third sample multimedia data includes: Specifically, it may include:

The first captioning information is input to a captioning model having second sample multimedia data and third sample multimedia data, respectively, to obtain a first output probability distribution of the second sample multimedia data and a second output probability distribution of the third sample. and a first score of the first captioning information is obtained based on the first output probability distribution and the second output probability distribution.

The second captioning information is input to a captioning model having second sample multimedia data and third sample multimedia data, respectively, to obtain a third output probability distribution of the second sample multimedia data and a fourth output probability distribution of the third sample. and a second score of the second captioning information is obtained based on the third output probability distribution and the fourth output probability distribution.

Specifically, taking video as an example, for a first video caption, the first video caption is taken as a ground truth and input to the video captioning model together with the second sample video data, so that each moment of the first video caption Obtain a first output probability distribution of . At the same time, the first video caption is taken as a ground truth and input to a video captioning model together with third sample video data to obtain a second output probability distribution of each instant of the first video caption, which is the first output probability The KL divergence of the distribution and the second output probability distribution may be calculated, and the first score r _g may be obtained based on the KL divergences.

As an alternative solution, we can multiply the KL divergences by time domain weights and multiply by a negative number to get the first score rg of the first video caption, as specifically indicated in equation (12):

When W _t =T/t is the time domain weight, in order to reduce the effect of error accumulation, the first words in the first video caption are given a higher weight and the last words of the first video caption are given a lower weight. to give Since the augmented video V' may include K data, K r _g may be obtained. In this case, a final primary score r' _g may be obtained _{based on K r g .} For example, the K r _g may be averaged to obtain the first score r' _{g .} It can also be obtained by other methods such as weighted averaging, and videos V' obtained using different augmentation methods are given different weights.

Similarly, in the case of the second video caption, the second video caption is considered real-world and is input to the video captioning model together with the second sample video data, so that the third output probability of each instant of the second video caption is get the distribution. At the same time, as specifically shown in Equation (13), the second video caption is regarded as ground truth and is input to the video captioning model together with the third sample video data, to obtain a fourth output probability distribution of each instant of the second video caption , and thereafter it may compute the KL divergences of the third and fourth output probability distributions, and the KL divergences are multiplied by time domain weights, and then multiplied by a negative 1 to determine the second A second score r _s of the video caption is obtained.

Since V' contains K pieces of data, the first score r' _s can be obtained _{by averaging K r s .}

After obtaining r' _g and r' _s , a second loss function of the video captioning model can be calculated through both _{r' g} and r' _s.

In the example shown in FIG. 26 . After acquiring the first video caption c _g , the c _g and the second sample video data V are input to the video captioning model M, and the third of each moment of the second video caption is based on the model output. obtain an output probability distribution; For each augmented video V′ (one V′ is shown in FIG. 21 ), the c _g and video V′ are input to a video captioning model M, so that each of the second video captions corresponding to each V′ _{A fourth output probability distribution of the instant is obtained, and r g} corresponding to the third output probability distribution and each fourth output probability distribution is _{calculated through Equation 12 above, and r' g} by averaging or other methods (one KL divergence corresponding to _{c g} in the figure) may be obtained. Using the same calculation principle, r' _s (one KL divergence corresponding to _{c s} in the figure) can be obtained through Equation 13.

In an alternative embodiment of the present disclosure, obtaining the value of the second loss function based on the first score and the second score includes: using the difference between the first score and the second score as a reward value to do; and obtaining a second loss function of the captioning model based on the compensation value and second captioning information.

Specifically, the second loss function may be a policy gradient loss function, and the policy gradient loss function is shown in Equation (14):

Here, (r' _s -r' _g ) is a reward value, that is, the difference between the first score and the second score. ∇θ is the gradient for calculating θ. After obtaining the second loss function, the policy gradient is used to train a captioning model. If the words obtained by sampling are more accurate, the KL divergences of the third output probability distribution and the fourth output probability distribution will be smaller, and the reward will be larger, so the probability of outputting the word will depend on whether the model is updated. It can be seen from the above description that it becomes larger afterward. Conversely, if the words obtained by sampling are relatively poor, the KL divergences of the third output probability distribution and the fourth output probability distribution will be large and the reward will be small, and the probability of outputting the word is that the model It gets smaller after being updated.

Still taking video as an example, based on the difference between _{r' g} (KL divergence corresponding to _{c g} in the figure _{) and r' s (KL divergence corresponding to c s} in the figure), as in the example shown in Fig. 26 . to obtain a compensation value (compensation shown in the drawing), the value of the policy gradient loss can be calculated through Equation 14 above based on the compensation value, and the value of the final loss function is the first loss It is obtained based on the value of the function (ie, the value of the cross-entropy loss shown in FIG. 26) and the value of the second loss function (ie, the value of the policy gradient loss shown in FIG. 26).

In addition, it can be seen from the above description that only a type of captioning information can be generated based on the second sample multimedia data. In this case, the value of the fraudulent second loss function may be obtained based only on the captioning information. Taking Equation 14 above as an example, _{it is also possible to remove r' g} from Equation 14, so Equation 14 is rewritten as Equation 15 below:

That is, a corresponding score (eg, the second score in the above example) may be obtained based only on the second captioning information, and based on the second captioning information and the score, value can be obtained.

Currently, in commonly used datasets of video captioning or image captioning, there are generally fewer captioning labels of video or image. For example, there are typically only 5 captioning labels in a training sample image. It is often difficult to fully represent the information in the image with only five captioning labels. In order to increase the diversity of training sample captioning annotations, a method for obtaining multimedia data captioning is provided in an embodiment of the present disclosure. Based on the method, the captioning labels of the sample multimedia data may be subjected to data augmentation to obtain augmented captioning information for increasing the number of captions of the sample data, so that the sample with the augmented captioning information Training can be performed to obtain a better multimedia data captioning model based on the data.

Accordingly, in an alternative embodiment of the present disclosure, the captioning label of the first sample multimedia data includes at least one original captioning label of the first sample multimedia data, and an enhanced caption corresponding to the respective original captioning labels. Labels may be included.

27 is a schematic flowchart of a method for obtaining captioning of multimedia data provided in an embodiment of the present disclosure; As shown in the figure, the method may include the following steps.

Step S2501: Obtain at least one original captioning label corresponding to the multimedia data.

For the first sample multimedia data, that is, obtain original captioning labels of the first sample multimedia data.

The multimedia data may be sample data in a training image data set or a training video dataset obtained from a local storage or a local database, as required, or in a training video dataset received from an external data source via an input device or transmission medium. . Taking an image as an example, the training image may include N predetermined image captioning labels, where N may be a positive integer greater than or equal to 1 . For example, the image in this solution could be a training image in a training image dataset commonly used in the image captioning field (eg dataset MS-COCO). There are usually five image captioning labels for an image in commonly used training image datasets. The five image captioning labels for the same training images have different but similar meanings.

Step S2502: Enhanced captioning information corresponding to each original captioning label is respectively generated based on each original captioning label corresponding to the multimedia data.

Specifically, the generator may generate each of the augmented captioning information corresponding to each of the original captioning labels according to the respective original captioning labels corresponding to the multimedia data. The generator may be used to generate captioning sentences having similar meanings different from those of the original captioning labels. That is, when a sentence labeled with the original captioning is input to the generator, the generator is a caption sentence having a different and similar meaning from the sentence labeled with the original captioning based on the sentence labeled with the original captioning can create

The process of generating a sentence by the generator is a time sequence process. Alternatively, a decoding method may be used to generate the sentence. That is, at a first instant, the input word vector is a starting token, and the output is the first word with the maximum predicted output probability; At a second instant, the input is the start token and the output at the first instant, the output continues until the output word becomes a stop token, such as the second word with the greatest expected output probability.

The specific network structure of the generator is not limited in the embodiments of the present disclosure. As an alternative solution, the generator may be implemented using a self-attention based encoder and a self-attention based decoder.

As two examples, FIGS. 28A and 28B show a schematic diagram of a network structure of a generator provided in an embodiment of the present disclosure. The generator may include a self-attention-based encoder (eg, a transformer encoder) and a self-attention-based decoder (eg, a transformer decoder) as shown in FIGS. 28A and 28B . The encoder in the generator may include a multi-head attention layer, a layer normalization layer and a feed forward network layer used to encode the input original captioning labels. The decoder of the generator may include a masked multi-head attention layer, a multi-head attention layer, a layer normalization layer and a feed forward network layer, which are enhanced by decoding an encoded image captioning label or a video captioning label. It is used to obtain image captioning information or augmented video captioning information. For a detailed description of the structure of each part of the encoder and decoder shown in FIGS. 28A and 28B , reference is made to the preceding corresponding description of the encoder and decoder shown in FIG. 13, 22 or 23 .

It should be noted that the network structure of the generator in an embodiment of the present disclosure may include, but is not limited to, the structures shown in the above examples, and any other available encoders and decoders may be used to implement the generator. do.

To ensure the accuracy of the augmented image captioning information generated by the generator, the generator also needs to be trained. Alternatively, the generator may be obtained by training in the following manner.

The training dataset includes several training sample data and each training sample data is obtained a training dataset including N original captioning labels, where N is a positive integer greater than or equal to 1 .

The generator is trained based on original captioning labels of several training sample data in the training dataset, where the generator is used to generate captioning information that is different from, and has a similar meaning to, the original captioning labels.

Further, in order to enhance the effectiveness of the vigour, as an alternative solution, a discriminator may be introduced when training the generator, and the generator is trained in an adversarial training manner. Specifically, training the generator may include the following.

The generator and the discriminator are alternately trained until a similarity value of the captioning information generated by the generator for respective original captioning labels of each training sample data satisfies a preset condition, wherein the discriminator The group may be specifically used to determine the probability that the captioning information generated by the generator is a true original captioning label.

The specific network structure of the discriminator may be set according to actual needs. It can be appreciated that the discriminator also needs to be trained. When the trained discriminator determines that the caption sentence generated by the generator is most likely to be a true original captioning label (eg, exceeds a predetermined threshold), it determines that the caption sentence generated by the generator is This means that the trained discriminator is close to the captioning of the real sample (i.e. the true original captioning label) that can "trick" the discriminator. In this case, such captioning sentences can be used as augmented captioning information applied in the training process to increase sample diversity.

Specifically, during training, the generator and the discriminator are alternately trained until a similarity value of captioning information generated by the generator for respective original captioning labels of each training sample data satisfies a preset condition. In this case, the specific calculation method of the similarity value is not limited in the embodiments of the present disclosure.

As an alternative solution, the similarity value may be a CIDEr value. CIDEr is an evaluation metric commonly used to evaluate the performance of captioning. The higher the CIDEr value, the more similar the generated captioning sentence is to the actual original captioning label. The CIDEr metric treats each sentence as a "document" and can be expressed as a tf-idf vector. The cosine similarity between the reference (ie, true) captioning sentence and the generated captioning sentence is calculated and used as a score for generating a CIDEr value. Therefore, according to an exemplary embodiment of the present disclosure, the CIDEr value of the generated captioning sentence (image captioning sentence or video captioning sentence) is the original captioning label for generating the captioning sentence and the captioning sentence It may be calculated based on the similarity between the N original captioning labels of the training sample data. For example, taking the image as an example, when the CIDEr value of the image captioning sentence generated by the generator for each original captioning label of each training image meets a preset condition, this means that the generator is actually It is possible to generate image captioning sentences that are very similar to image captioning labels, meaning that training for the generator and discriminator is complete.

The preset condition is that a similarity value of the captioning information generated for respective original captioning labels of each training sample data reaches a predetermined threshold, or for each original captioning label of each training sample data. and reaching an average similarity value of the generated image captions to a predetermined threshold. The preset condition may be a system default, or may be set by a user according to necessity or experience. In addition, it may be determined whether training of the generator and the discriminator is completed according to the user's needs or experience. For example, when the generator and discriminator have been trained to some extent, the user can test the generator using a batch of training sample data and observe whether the generator's output is satisfactory. When the output of the generator is satisfactory, training of the generator and discriminator can be completed.

In an alternative embodiment of the present disclosure, alternatingly training the generator and discriminator comprises: training the discriminator with fixed generator parameters; and training the generator with the fixed trained discriminator parameters.

That is, when the generator and the discriminator are alternately trained, the discriminator is first trained with fixed generator parameters, and then the generator is trained with the fixed trained identifier parameters. For different training sample datasets, the above training process may be iteratively performed. For example, taking an image as an example, for the first set of training images, the discriminator and the generator are trained once based on the original parameters (ie, network structure parameters) of the generator and the discriminator. Then, for the second set of training images, the discriminator and generator are trained once again based on the parameters of the discriminator and generator trained on the first set of training images. Then, for the third set of training images, the discriminator and generator are trained once again based on the parameters of the discriminator and generator trained on the second set of training images, which each This is continued until the similarity value of the image captioning information generated by the generator for the image captioning labels of either satisfies a preset condition or the output result of the generator is satisfied after being tested by the user.

In an alternative embodiment of the present disclosure, the discriminator may be trained by the following operation.

The following operations (in this mode of operation the number of original captioning labels of the sample data is greater than 1, ie, N is greater than 1) are performed for each of the original captioning labels of each training sample data.

the original captioning label is paired with other N-1 original captioning labels of the training sample data, respectively, to generate N-1 first pairs; The original captioning label is input to the generator, which is for generating captioning information by the generator, and the generated captioning information is paired with the original captioning label to generate a second pair. Based on N-1 of the first pairs and the second pair, the discriminator can be trained using a cross entropy loss function, where the output of the discriminator is that each pair is two true original captioned labels. is the probability

That is, one original caption label (also referred to as a baseline label) is paired with each other N-1 original subtitle labels to obtain N-1 reference pairs (ie, sample pairs). Based on the baseline label, the generator may generate N-1 pieces of captioning information, and the baseline label is paired with N-1 pieces of generated captioning information to obtain prediction pairs, each Calculate the value of the loss function based on the corresponding sample pairs and the predicted pairs of ; And the network parameters of the discriminator are adjusted based on the value of the loss function until preset conditions are met. For example, in the case of an image, the output of the discriminator is that the probability that each prediction pair is two true image captioning labels (ie reference pairs) is greater than a set threshold.

In an alternative embodiment of the present disclosure (which may be referred to as scheme 1), when the parameters of the trained discriminator are fixed, the step of training the generator is for each original captioning labels for each training sample data. It may include performing the following operations. .

The original captioning label is input to the generator to generate captioning information through a greedy decoding method: and the following operations are performed on the generated captioning information.

a similarity value corresponding to the generated captioning information is calculated based on the generated captioning information and N original captioning labels of a corresponding training image; The generated captioning information is paired with original captioning labels to generate a second pair, and a probability value that the second pair is two original image captioning labels is obtained by using a trained discriminator, and the calculated the similarity value and the obtained probability value are weighted and summed to obtain a reward; And the parameters of the generator are adjusted according to the obtained reward.

In another alternative embodiment of the present disclosure (which may be referred to as scheme 2), when the parameters of the trained discriminator are fixed, the step of training the generator comprises: for each training sample data, each original caption It may include performing the following operations for the labels.

The original captioning label is input to the generator to generate first captioning information through a greedy decoding method.

The original captioning label is input to the generator to generate second captioning information through a Monte Carlo sampling method.

The following operations are performed on the generated first captioning information.

a first similarity value corresponding to the generated first captioning information is calculated based on the generated first captioning information and N original captioning labels of a corresponding training image; The generated first captioning information is paired with original captioning labels to generate a second pair, and a first probability value that the second pair is two true original captioning labels is obtained by using a trained discriminator, , a weight is applied to the calculated first similarity value and the obtained first probability value and summed to obtain a first reward.

The following operations are performed on the generated secondary captioning information.

a second similarity value corresponding to the generated second captioning information is calculated based on the generated second captioning information and N original captioning labels of a corresponding training image; The generated second captioning information is paired with original captioning labels to generate a second pair, and a second probability value that the second pair is two true original captioning labels is obtained by using a trained discriminator, , a weight is applied to the calculated second similarity value and the obtained second probability value and summed to obtain a second reward.

The parameters of the generator are adjusted according to the final compensation, which is the difference between the first compensation and the second compensation.

In practical applications, due to the discrete nature of text data, it is difficult to pass the gradient of the discriminator back to the generator. To solve this problem, as an alternative method, a policy gradient method may be adopted. The reward is calculated based on the caption sentence generated by the generator. The higher the reward, the better the currently generated captioning sentence is generated, and more parameters of the generator are adjusted in this direction. In the traditional method, the reward includes only the output of the discriminator, but the compensation of the above alternative embodiment provided herein may include two parts: the output of the discriminator and a similarity value (eg, CIDEr value), , both of them are weighted and summed to the final reward. By using more diverse data to determine a reward for adjusting the generator's parameters, the generator can effectively learn more information, and enhanced captioning information more similar to but different from the original captioning labels. , which provides better augmented image captioning information based on a trained generator and more and more for training a multimedia data captioning model based on sample data including the augmented captioning information. This is to provide a good data base.

In order to better understand and explain the training scheme of the generator provided in an embodiment of the present disclosure, the training scheme will be described in more detail with reference to FIGS. 28A and 28B . In this example, the multimedia data is an image. It can be understood that the principles of this example apply to video as well.

As an alternative example, as shown in FIG. 28A , for scheme 1 above, when training the generator, the following operations may be performed for each image captioning label of each training video. The image captioning label (eg, X1: T shown in the figure) is input to the generator to generate image captioning information based on the greedy decoding method; With respect to the generated image captioning information (eg, Y1: T), the following operations are performed. a similarity value corresponding to the generated image captioning information is calculated based on the generated image captioning information and N image captioning labels of a corresponding training image; The generated image captioning information is paired with image captioning labels to generate a second pair (eg, X1: T, Y1: T), and the probability value that the second pair is two true image captioning labels is obtained by using a trained discriminator, weighted and summed to the calculated similarity value and the obtained probability value to obtain a reward; And the parameters of the generator are adjusted according to the obtained reward.

Specifically, as shown in FIG. 28A , ^{the CIDEr value calculated for the image captioning sentence y b} generated by the generator according to the greedy decoding method and the image generated by the generator according to the greedy decoding method The probability values obtained by the discriminator for the caption sentence y ^b are weighted and summed to obtain the reward (y ^b ). The weighted and summed formula is the following Equation (16):

where r ( ^{corresponding to r(y b} ) in Fig. 26a) is the reward, τ is the weighting coefficient, D _φ is the probability value output by the discriminator, and C is the CIDEr value (corresponding to the CIDEr score in Fig. 26a) )am.

In the example shown in 28a, the structure of the discriminator may be a CNN-based structure, for example, it may include a convolutional layer, a maximum pooling layer, and the like. Specifically, for each pair of image captioning labels and corresponding generated image captioning information, that is, for each second pair, the pair may be processed by embedding to obtain a corresponding feature vector. A convolution layer using a variety of different convolution processing parameters is used to perform convolution processing on a feature vector, and each convolution result is pooled through a max pooling layer, and the pooling results are concatenated. , a probability value corresponding to each of the second pairs is obtained based on the concatenated vector prediction.

In addition, according to another alternative scheme, namely scheme 2 above, in order to further improve the effect, the generator may also be trained using a self-critical mechanism, ie, image captioning obtained by Monte Carlo sampling The compensation difference between the sentence and the image captioning sentence obtained by greedy decoding is used as the final compensation, and the parameters of the generator are adjusted according to the obtained guarantee.

^{In the example shown in FIG. 28B , the CIDEr value calculated for the image captioning sentence y b} generated by the generator according to greedy decoding and the image ^{captioning sentence y b} generated by the generator according to the greedy decoding A weight is given to the probability values obtained by the discriminator for , and summed to obtain the reward (y ^b ). The Probability values obtained by a determination for, based on the CIDEr value and a Monte Carlo sampling calculated for the image captioning text y ^s generated by the generator based on the Monte Carlo sampling generated by the generator image captioning text y ^s Weights are given and summed to obtain the reward y ^s . r(y ^s )-r(y ^b ) is used as the final compensation to adjust the parameters of the generator. The weighted value of probability for the generated based on the calculated CIDEr value and a Monte Carlo sampling image captioning text y ^s is given and summed compensation (y ^s) for an image generated based on Monte Carlo sampling captioning text y ^s A specific way of ^{obtaining refers to the description of obtaining r(y b} ) in FIG. 28A , which has a similar principle, and the description is not repeated here.

In another embodiment of the present disclosure, in order to avoid redundant information in the generated augmented captioning information, the method may further include the following.

When the repeated augmented captioning information is present, the original captioning label corresponding to the repeated augmented captioning information is re-entered into the generator, and based on the beam search method, the generator determines the size of the beam value It is used to regenerate the augmented captioning information by adjusting.

Taking the image as an example, after the training of the generator and the discriminator is completed, the trained generator is used to augment the corresponding image captioning labels according to the respective image captioning labels corresponding to the image. image captioning information can be generated. Such augmented image captioning is different from true image captioning labels, but such augmented image captioning may be repeated with each other.

In order to solve the problem that there may be repetition in the augmented captioning information, as an alternative method, a beam search method may be adopted to regenerate the augmented captioning information. The generator generates augmented captioning information based on the maximum probability, that is, the word with the maximum predicted probability is output at each instant (equivalent to beam value 1), and the beam search method is (2, 3, etc.) ) by changing the beam value, it is possible to adjust the generation results of the generator. For example, when there are two identical augmented captions, a true caption (ie, an original captioning label) corresponding to one of the augmented captioning information is input to the generator, the beam value is set to 2, and , and the generator is used to generate different augmented captioning information. For example, the generator may output two words with the top two probabilities at the first instant, assuming that they are {a} and {b}, respectively; Based on the two words {a} and {b} at the first instant, the two words having the maximum probability are output at the next instant, and these are {a, c}, {a, d}, {b, e}, { b, assuming that the method is; Then the two with the greatest probability assumed to be {a, c}, {b, e} are selected from these four sequences; The next moment proceeds similarly. As another example, when there are three identical augmented captions, an original captioning label corresponding to one of the augmented captioning information is input to the generator, the beam value is set to 2, and the generator is different Used to generate augmented captioning information. Additionally, a true caption corresponding to one of the augmented captioning information may be input to the generator and set a beam value to 3, the generator may further generate different augmented captioning information, etc. is used for As such, different beam sizes may be used to generate augmented captioning sentences and alter the generated results to solve the iteration problem.

After obtaining the augmented captioning information by using the method for obtaining the augmented captioning information provided in an embodiment of the present disclosure, original captioning labels of the multimedia data and corresponding enhanced captioning information may be used as caption label information of the multimedia data. The multimedia data samples containing more label information are used to train the initial multimedia data captioning model to obtain a better captioning model. Specifically, taking an image as an example, an image captioning model may be obtained by training in the following manners.

Training samples are obtained, each sample image in the training samples has corresponding label information, wherein the label information includes at least one image captioning label of the sample image and enhanced image captioning information corresponding to each image captioning label. include

The original image captioning model is trained based on each sample image until a preset training termination condition is satisfied to obtain a trained image captioning model, which is to obtain a trained image captioning model.

For each sample image, enhanced image captioning information corresponding to the sample image is obtained by using the method for obtaining image captioning provided in any alternative embodiment of the present disclosure. The specific network structure of the image captioning model is not limited in the embodiment of the present disclosure. For example, it may be an encoder- and decoder-based image captioning model. For example, it may be a network image captioning model based on the codec shown in FIG. 22 or FIG. 23 .

As an example, FIG. 29 shows a schematic flowchart of a method for training an image captioning model provided in an embodiment of the present disclosure. In this example, the image captioning model is an image captioning model based on an encoder and a decoder. As shown in the figure, the method may include the following steps.

Step S2701: For each training image (ie, a sample image) in the training image dataset, a first training is performed on the encoder and decoder using a cross entropy loss function.

Alternatively, the training image in this step may be a training image in a training image dataset, wherein the training image may include N predetermined image captioning labels, where N may be a positive integer greater than or equal to 1 . For example, the training image may be a training image having five image captioning labels in a training image dataset (eg, dataset MS-COCO) commonly used in the image captioning field.

Specifically, the captioning information corresponding to the training video may be obtained by referring to or based on various alternative methods in FIG. 24 based on the training image, and the augmented image captioning information corresponding to the training image is It can be obtained by referring to the method shown in FIG. 27 . Based on the obtained captioning information, video captioning labels of training images, and augmented image captioning information, the encoder and decoder are trained using a cross entropy loss function. For example, they can be trained by Equation 17 of the cross entropy loss function.

where J _xe (θ) denotes the loss, θ denotes the parameters of the transformer encoder 302 and the transformer decoder 303 , t denotes the current moment, T denotes the maximum instant, and y _t denotes the current instant denotes the word output of , y _1:t-1 denotes the ground-truth word at the previous moment, I denotes the current image, and p _θ denotes the probability that the output word is true. Here, the first image captioning sentence is a combination of words having the maximum output probability of each moment, and y _t at each moment may be obtained from the first image captioning sentence. Additionally, the ground truth word of each moment may be obtained from each of the image captioning label and the augmented image captioning of the training image.

Step S2702: When the training of the encoder and decoder based on the first training is completed, for each training image in the training image dataset, the encoder obtained through the first training by using a policy gradient and/or a self-critical mechanism and a second training is performed for the decoder.

Specifically, the policy gradient is used because the optimization target of the cross-entropy loss is different from the metric used to evaluate the captioning (eg CIDEr value). To solve this problem, the policy gradient is used to directly optimize the CIDEr value. The formula is the following Equation 18.

where J(θ) is the loss, θ is the parameters of the encoder and decoder, E is the expected value, y ^s is the sampled image captioning sentence, r(y ^s ) is the CIDEr value that is the compensation, y ^s ~ p _θ represents a set of image captioning sentences sampled by the existing network parameters.

The self-critical mechanism is configured such that the reward is set to the difference between the CIDEr value of the image captioning sentence obtained by Monte Carlo sampling and the CIDEr value of the image captioning sentence obtained by greedy decoding, that is, the It indicates that the effect of greedy decoding would be better used to limit the compensation. The above formula is the following Equation 19.

는 그리디 디코딩에 의해 획득된 이미지 캡셔닝 문장이며, y^s는 몬테카를로 샘플링에 의해 획득된 이미지 캡셔닝 문장이며, r은 계산된 CIDEr 값이며, ∇_θL(θ)은 손실의 그래디언트이며, 그리고 p_θ(y^s)는 y^s 을 샘플링할 때의 대응 확률이다.From here

is the image captioning sentence obtained by greedy decoding, y ^s is the image captioning sentence obtained by Monte Carlo sampling, r is the calculated CIDEr value, ∇ _θ L(θ) is the gradient of loss, and p _θ (y ^s ) is the corresponding probability when sampling ^{y s .}

Alternatively, when performing the second training, the first image captioning sentence may be obtained by using the greedy decoding with reference to the method of FIG. 24 or based on the method shown in FIG. 24 ; The second image captioning sentence may be obtained by using Monte Carlo sampling with reference to the method of Fig. 24 or based on the method shown in Fig. 24, and the first trained encoder by using a policy gradient and a self-critical mechanism. and a second training is performed for the decoder. Specifically, the CIDEr value of the first image captioning sentence may be calculated based on the similarity between the first image captioning sentence and N image captioning labels of the corresponding training image, and the CIDEr value of the second image captioning sentence The CIDEr value may be calculated based on the similarity between the second image captioning sentence and the N image captioning labels of the corresponding training image. It calculates the difference between the CIDEr value of the first image captioning sentence and the CIDEr value of the second image captioning sentence to obtain a reward, and according to the obtained reward, parameters of the first trained encoder and decoder parameters can adjust them.

It can be seen that the training method of the image captioning model provided in the embodiment of the present disclosure is only an alternative training method. As long as the augmented video caption obtained based on the method for obtaining augmented image captioning provided in the embodiment of the present disclosure is used as the label information of the sample image, it represents the amount and diversity of label data of the sample image. can increase An image captioning model is trained based on training data including the sample image, which can effectively improve the performance of the model. In addition, it is obvious to those skilled in the art that the solution applicable to images is also applicable to the video, and the principle is the same.

Additionally, to improve the expressive power of captioning labels of the training sample. In an alternative embodiment of the present disclosure, captioning data augmentation may be performed using a generative adversarial network, and samples for captioning data augmentation may be applied to training of a captioning model, thereby It increases the diversity of samples and further enhances the effectiveness of the video captioning model or image captioning model of the present disclosure.

The following describes a method for generating caption information of multimedia data provided by the present disclosure by combining with two schematic diagrams.

Taking video as an example, FIG. 30 shows a schematic flowchart of a method for generating video captioning information of this disclosure. As shown in the figure, for a given video, a frame of the video may be selected through a frame selection step. As shown in the figure, the regional encoder is a regional encoder for extracting local visual features of respective target regions in each frame of the selected frames. Alternatively, the regional encoder may include a regional feature extraction network, a relationship detector (ie, a relationship prediction network), an attribute detector (ie, a property prediction network), and a behavior detector (ie, a behavior classifier).

To obtain a trained video captioning model, before the model of the codec structure shown in this example is used for video captioning, as shown in FIG. 30 , it is subjected to semi-supervised learning (or semi-supervised) training) can be trained. It can be trained using labeled videos (ie, videos with captioning labels) and unlabeled videos (ie, videos without captioning labels). During training each video, multiple frames of video may be used. The unlabeled video may be augmented by data augmentation processing to obtain an augmented video. At least one video captions may be obtained for the unlabeled video. A score of each video caption may be obtained based on the augmented video to obtain a value of the second loss function. For a labeled video, a value of a first loss function may be obtained based on the target captioning information and corresponding label information of the video output by the model, wherein the value of the total loss function of the model is the first It is obtained based on the value of the loss function and the value of the second loss function, and the model training is guided based on this value until the overall loss function of the model converges to a minimum value.

In addition, for the part for acquiring the augmented captioning information shown in the figure, this part is used to obtain a greater amount of diversity label information when training the model. It may generate the augmented captioning information corresponding to the captioning label through the generator based on the original captioning label (ie, the true captioning information) of the labeled video. Both the original captioning label and the augmented captioning label are used as captioning label information of the sample video data during training, thereby increasing the amount and variety of captioning label information. More label information and captioning information predicted by the decoder of the model are used to guide model training, which can further improve the stability of the model and the accuracy of the generated captioning information.

In the example shown in Figure 30, when performing video processing based on a trained video captioning model, the regional encoder performs local visual features, relational features, attribute features, etc. of respective target regions in each of the frames of the video. can be extracted, and a scene graph for each frame can be constructed based on the extracted features. In this example, the scene graph may be a spatio-temporal scene graph incorporating temporal information, and corresponding updated features (ie, graph convolution features) may be obtained through a graph convolution network. Correspondingly, for the decoder part of the model, this example may use a self-attention based intra-decoder and a self-attention based inter-decoder. In the case of generating captioning information by performing encoding, a text caption of the video may be generated that better meets the user's requirements by obtaining information about the captioning information expected to be generated by the user. For example, while the user is driving, collect real-time video in front of the user's eyes and analyze the video to analyze the generated captioning information of the video when the user needs a prompt, such as when there is a potential risk. Thus, a reminder corresponding to the user may be provided or the captioning information may be reproduced to the user.

As another example, FIG. 31 shows a schematic flowchart of a method for generating video captioning information of this disclosure. The three-dimensional visual feature encoder (i.e., spatial-temporal feature extraction network), regional encoder, and semantic encoder (i.e., semantic prediction network) shown in the figure are the local visual features (local features shown in the figure), spatial -Encoders for extracting temporal visual features and semantic features of respective target regions in each frame of video. Based on the local visual characteristics, a spatio-temporal scene graph for each frame of the images can be built. The graph convolution feature (updated local feature shown in the figure) may be obtained through the graph convolution network. In this example, the 3D visual feature encoder may also be used to extract spatio-temporal visual features of a video, and the semantic encoder may be used to extract semantic features of the video. For the obtained spatial-temporal visual features, semantic features and graph convolution features, the feature selection network performs feature selection on various features, ie, determines the weight of each feature. In addition, an integrated feature can be obtained by performing weighted integration of features based on the weight of each feature. The decoder bank (a decoder bank composed of several decoders shown in the figure) performs decoding according to the integrated characteristics and length information of the desired captioning information, and the final captioning information (output captioning information shown in the figure) is It is finally obtained according to the results of the respective decoders. For example, decoding results of respective decoders may be averaged, and the final captioning information may be obtained based on the averaged result. Alternatively, the decoder bank may include a self-attention based intra-decoder, and length information of desired captioning information may be input to the decoder, and the decoder may control the length of the finally generated captioning information. let it do

Similarly, in order to obtain a trained video captioning model, before the model of the codec structure shown in this example is used for video captioning, semi-supervised if adversarial training is used for training. -supervised) learning (ie, semi-supervised training) manner. For details of the training process, reference is made to the description of a part corresponding to the description of model training of FIG. 30 above, and a description thereof will not be repeated here.

In addition, in actual application, after obtaining each characteristic information of a video through the encoding unit shown in FIGS. 30 and 31 , before decoding by the decoder, the self-attention-based intra-frame encoder and the self-attention-based inter - It is also possible to encode the extracted characteristic information using a frame encoder and input the encoded characteristics to the decoder.

Based on the same principle as the method for generating captioning information of multimedia data provided in the embodiment of the present disclosure, the present disclosure also provides an apparatus for generating captioning information of multimedia data. As shown in FIG. 32 . The apparatus 100 for generating captioning information may include a characteristic information extraction module 110 and a captioning information generation module 120 .

The characteristic information extraction module 110 is configured to extract characteristic information of multimedia data to be processed, wherein the multimedia data includes a video or an image.

The captioning information generating module 120 is configured to generate a text caption of the multimedia data based on the extracted characteristic information.

Alternatively, the captioning information generating module 120 is specifically configured to execute at least one of the following: extracting local visual features of targets included in respective target regions of each image in the multimedia data; extracting semantic features of the multimedia data; when the multimedia data is a video, extracting spatial-temporal visual features of the multimedia data; extracting global visual features of the multimedia data; extracting attribute characteristics of the targets included in respective target regions of each image in the multimedia data; and extracting global attribute features of each image from the multimedia data.

Additionally, the characteristic information includes local visual characteristics of targets included in respective target regions in each image of the multimedia data, and the captioning information generating module 120 specifically: obtain relationship characteristics between the targets based on local visual characteristics; build a scene graph of the image based on the local visual features and the relational features; obtain graph convolution features of the image based on a scene graph of the image; and generate a text caption of the multimedia data based on graph convolution characteristics of each image of the multimedia data.

Alternatively, the scene graph comprises a plurality of nodes and a plurality of edges, wherein one node represents a local visual feature of one target and each of the plurality of edges represents a relationship feature between two connected nodes.

Alternatively, the characteristic information includes attribute characteristics of targets included in respective target regions of each image in the multimedia data; When the captioning information generating module 120 builds the scene graph of the image, it specifically builds the scene graph of the image based on the local visual characteristics of each target, the relationship characteristics between the targets and the attribute characteristics of each target. , wherein a node of the scene graph represents a local visual feature or attribute feature of a target corresponding to the target region.

Alternatively, if the multimedia data is a video, the images of the multimedia data are a plurality of frames selected from the video, and if the target areas of two adjacent frames contain the same targets, the scene graphs of the adjacent two frames are on the same target It has temporal edges between corresponding nodes.

Alternatively, when the captioning information generating module 120 obtains the graph convolution features of the image according to the scene graph of the image, it: encodes the nodes and edges in the scene graph to obtain the target dimension of the feature vector obtain; It is used to obtain the graph convolution features using a graph convolution network based on the obtained feature vectors.

Alternatively, if the characteristic information of the multimedia data includes at least two of a local visual characteristic, a semantic characteristic, a spatial-temporal visual characteristic, and a global characteristic, the captioning information generating module 120 may include: a weight of each characteristic information to decide; assigning a weight to each characteristic information based on the weight of each characteristic information; And it may be used to generate a text caption of the multimedia data based on the weighted characteristic information.

Alternatively, the captioning information generating module 120 may: encode the acquired characteristic information using a self-attention-based encoder; input the encoded characteristic information to a decoder to generate a text caption of multimedia data, wherein if the multimedia data is an image, the self-attention-based encoder is a self-attention-based intra-frame encoder; If the multimedia data is video, the self-attention-based encoder includes a self-attention-based intra-frame encoder and/or a self-attention-based inter-frame encoder.

Alternatively, the captioning information generating module 120 may: input the extracted characteristic information to a plurality of decoders, respectively; and generate a text caption of the multimedia data based on a decoding result of the decoders.

Alternatively, the captioning information generating module 120 is configured to: obtain length information of the text subtitle to be generated; and generate a text caption of the video based on the length information and the extracted characteristic information.

Alternatively, the captioning information generating module 120 may specifically acquire text captions of multimedia data using a multimedia data captioning model, and the multimedia data captioning model is performed by a model training device performing training. obtained, wherein the model training apparatus comprises: a sample acquiring module configured to acquire training samples, the training samples comprising first sample multimedia data with caption labels; a model learning module, configured to train an original captioning model based on the first sample multimedia data until a model loss function converges; The trained captioning model is taken as a multimedia data captioning model.

alternatively, the training sample further comprises second sample multimedia data without caption label, and the model loss function comprises a first loss function and a second loss function; The model training module is configured to: train a preset captioning model based on the first sample multimedia data to obtain a value of a first loss function, train the captioning model based on the second sample multimedia data to obtain a second obtain the value of the loss function; obtain a value of a final loss function based on the value of the first loss function and the value of the second loss function; It can be used to train a captioning model based on the value of the final loss function until it converges.

Alternatively, when the model training module trains a captioning model based on the second sample multimedia data and obtains the value of the second loss function, it is specifically: to obtain the third sample multimedia data at least once perform data augmentation on the second sample multimedia data; input the second sample multimedia data into the captioning model to obtain at least one multimedia captioning; determine a score of each multimedia caption based on the second sample multimedia data and the third sample multimedia data; and may be used to obtain a value of the second loss function based on the scores of the multimedia captioning.

Alternatively, the captioning label of the first sample multimedia data includes at least one original captioning label of the first sample multimedia data and an augmented captioning label corresponding to each original captioning label; The augmented captioning label is obtained by generating an augmented image captioning label corresponding to each of the original captioning labels, respectively, based on each original captioning label of the first sample multimedia data.

Based on the same principle as the method and apparatus provided in the embodiments of the present disclosure, the embodiments of the present disclosure further provide an electronic device. The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the processor may perform the methods shown in any alternative embodiment of the present disclosure.

An embodiment of the present disclosure further provides a computer-readable storage medium. The storage medium stores a computer program. When the computer program is executed by a processor, it may perform the method shown in any alternative embodiment of the present disclosure.

As an example, FIG. 33 illustrates a schematic structural diagram of an electronic device applicable to an embodiment of the present disclosure. As shown in FIG. 33 , the electronic device 4000 shown in FIG. 33 includes a processor 4001 and a memory 4003 . The processor 4001 and the memory 4003 are connected via, for example, a bus 4002 . Alternatively, the electronic device 4000 may further include a transceiver 4004 . It should be noted that in an actual application, the number of the transceivers 4004 is not limited to one, and the structure of the electronic device 4000 does not limit the embodiment of the present disclosure.

The processor 4001 may be a central processing unit (CPU), general purpose processor, digital signal processor (DSP), application specific integrated circuit (ASIC), or field programmable gate array (FPGA) or other programmable logic device, transistor logic device, hardware components, or a combination thereof. It may implement or execute the various illustrative logical blocks, modules, and circuits described in connection with this disclosure. The processor 4001 may also be a combination realizing a computing function, for example, a combination including one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

The bus 4002 may include a path for transferring information between the aforementioned components. The bus 4002 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, and the like. For convenience of expression, the bus is indicated by only a bold line in FIG. 33, but that does not mean that there is only one bus or one type of bus.

The memory 4003 may be read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM), or other type of dynamic storage device capable of storing information and instructions. It may also include Electrically Erasable Programmable Read Only Memory (EEPROM), Compact Disc Read Only Memory (CD-ROM), or other types of optical disc storage; optical disk storage (including compressed optical disks, laser disks, optical disks, digital versatile disks, Blu-ray disks, etc.), magnetic disk storage media or other magnetic storage devices, or the desired program code in the form of instructions or data structures; It can be, but is not limited to, any other medium that can be used for carrying or storing and accessible by a computer.

The memory 4003 is configured to store application program code for executing the solution of the present disclosure, and the processor 4001 controls execution. The processor 4001 is configured to execute the application program code stored in the memory 4003 to implement the content shown in any one of the above-described method embodiments.

The methods and models (such as video captioning models, image captioning models, etc.) provided in alternative embodiments of this application can be used for any terminal that needs to generate video captioning information and image captioning information (either a user terminal or may be executed on a server, etc.). Alternatively, the terminal may have the following advantages.

(1) In a hardware system, the device has a central processing unit, a memory, input components and output components, that is, the device is often a microcomputer device with a communication function. In addition, it can have multiple input methods such as keyboard, mouse, touch screen, microphone, camera, etc., which can be adjusted as needed. At the same time in devices, the devices often have multiple output modes, such as receivers and display screens, which can also be adapted as required.

(2) in the software system, the device is Windows Mobile. You must have an operating system such as Symbian, Palm, Android, iOS, etc. At the same time, these operating systems are becoming more and more open, and personalized applications based on these open operating system platforms are appearing in an infinite stream such as address books, calendars, notepads, calculators and various games, which greatly meets the needs of personalized users. make it

(3) In terms of communication capability, the device has a flexible access method and high-bandwidth communication capability, and it can automatically adjust the selected communication method according to the selected service and environment, thereby facilitating the use of the user. . The device includes Global System for Mobile Communication (GSM), Wideband Code Division Multiple Access (WCDMA), Code Division Multiple Access (CDMA2000), and Time Division-Synchronous Code Division Multiple Access (Time Division Multiple Access). -TDSCDMA), Wi-Fi and WiMAX (Worldwide Interoperability for Microwave Access), etc., to adapt to various standard networks supporting not only voice service but also multiple wireless data service.

(4) In terms of function, the device pays more attention to humanization, personalization and multifunctionality. With the development of computer technology, the device moves from "device-centric" mode to "person-centric" mode, integrating embedded computing, control technology, artificial intelligence technology and biometric authentication technology, which completely achieves the person-centric purpose. reflect Due to the development of software technology, the configuration of the device can be adjusted according to the needs of the individual being more personalized. At the same time, the device itself integrates a lot of software and hardware, and its functions are becoming more and more powerful.

Also, the apparatus or method for generating caption information of multimedia data according to the disclosed embodiment may be provided as a computer program product. The computer program product may be traded as a commodity between a seller and a buyer.

The computer program product may include a software program and a computer-readable storage medium in which the software program is stored. For example, the computer program product may include a product in the form of a software program (eg, a downloadable app) distributed electronically through a manufacturer of an electronic device or an electronic market (eg, Google Play Store, AppStore). For electronic distribution, at least a portion of the software program may be stored on a storage medium or created temporarily. In this case, the storage medium may be a server of a manufacturer, a server of an electronic market, or a storage medium of a relay server temporarily storing a SW program.

The computer program product may include a storage medium of the server or a storage medium of the client device in a system including a server and a client device. Alternatively, if there is a third device (eg, a smartphone) that communicates with the server or client device, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include the S/W program itself transmitted from the server to the client device or the third device, or from the third device to the client device.

In this case, one of the server, the client device, and the third device may execute a computer program product to perform the method according to the disclosed embodiment. Alternatively, at least two of the server, client device, and third device may execute a computer program product for distributing and performing the method according to the disclosed embodiment.

For example, a server (eg, a cloud server or artificial intelligence server) may execute a computer program product stored on the server to control a client device that communicates with the server to perform a method according to the disclosed embodiments.

Although the steps in the flowcharts of the drawings are sequentially displayed according to the direction of the arrow, it should be understood that these steps are not necessarily performed in the order indicated by the arrow. Unless explicitly stated herein, the execution of these steps is not strictly limited, and they may be performed in a different order. Moreover, at least some of the steps in the flowcharts of the figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily performed simultaneously and may be performed at different times. The execution order is not necessarily performed in order, and at least some of other steps or sub-steps or other steps stages may be performed sequentially or alternately.

The above description is only part of the implementation of the present disclosure. It should be noted that various improvements and modifications may be made to those skilled in the art without departing from the principles of the present disclosure. Such improvements and modifications should also be considered to fall within the protection scope of the present application.

Claims (15)

How to create caption information of multimedia data
extracting characteristic information of multimedia data to be processed, the multimedia data including video or images; and
and generating a text caption of the multimedia data based on the extracted characteristic information. The method of claim 1, wherein extracting characteristic information of the multimedia data to be processed comprises:
extracting local visual features of targets included in respective target regions of each image in the multimedia data;
extracting semantic features of the multimedia data;
when the multimedia data is a video, extracting spatial-temporal visual features of the multimedia data;
extracting global visual features of the multimedia data;
extracting attribute characteristics of the targets included in respective target regions of each image in the multimedia data; and
at least one of extracting global attribute features of each image in the multimedia data. The method of claim 2, wherein the characteristic information includes local visual characteristics of the targets included in respective target regions in each image of the multimedia data, and a text caption of the multimedia data is generated based on the extracted characteristic information. The steps to create are:
obtaining relationship characteristics between targets based on local visual characteristics of each target in the image;
building a scene graph of the image based on the local visual features and the relational features;
obtaining graph convolution features of the image based on a scene graph of the image; and
generating a text caption of the multimedia data based on graph convolution characteristics of each image of the multimedia data. 4. The scene graph of claim 3, wherein the scene graph comprises a plurality of nodes and a plurality of edges, wherein one node represents a local visual feature of one target, and each of the plurality of edges is between two connected nodes. A method for representing a relationship characteristic. 4. The method of claim 3, wherein: the characteristic information includes attribute characteristics of the targets included in respective target regions of each image in the multimedia data;
Building a scene graph of the image based on the local visual features and the relational features comprises:
building a scene graph of the image based on local visual characteristics of each target, relationship characteristics between the targets, and attribute characteristics of each target, wherein one node in the scene graph is one target Indicative of local visual characteristics or attribute characteristics of 4. The method of claim 3,
When the multimedia data is a video, the images of the multimedia data are a plurality of frames selected from the video, and when target areas of two adjacent frames include the same targets, scene graphs of the adjacent two frames correspond to the same target having temporal edges between nodes that 4. The method of claim 3, wherein obtaining graph convolution features of the image based on a scene graph of the image comprises:
encoding nodes and edges in the scene graph to obtain a target dimension of a feature vector; and
obtaining the graph convolution features by using a graph convolution network based on the obtained feature vector. The method of claim 2, wherein when the characteristic information of the multimedia data includes at least two of the local visual characteristic, the semantic characteristic, the spatial-temporal visual characteristic, and the global characteristic, based on the extracted characteristic information The step of generating a text caption of the multimedia data by:
determining weights of each characteristic information;
assigning a weight to each characteristic information based on the weights of each characteristic information; and
and generating a text caption of the multimedia data based on the weighted characteristic information. The method of claim 2, wherein the generating of the text caption of the multimedia data based on the extracted characteristic information comprises:
encoding the acquired characteristic information using a self-attention-based encoder;
inputting the encoded characteristic information to a decoder to generate a text caption of the multimedia data;
when the multimedia data is an image, the self-attention-based based encoder is a self-attention based intra-frame encoder; When the multimedia data is video, the self-attention-based encoder comprises a self-attention-based intra-frame encoder and/or a self-attention-based inter-frame encoder. The method of claim 1, wherein the generating a text caption of the multimedia data based on the extracted characteristic information comprises:
inputting the extracted characteristic information to a plurality of decoders, respectively; and
and generating a text caption of the multimedia data based on the decoding result of the decoders. The method of claim 1 , wherein generating a text caption of the multimedia data based on the extracted characteristic information comprises:
obtaining length information of the text caption to be generated; and
and generating a text caption of the video based on the length information and the extracted characteristic information. The method of claim 1 , wherein the text caption of the multimedia data is generated through a multimedia data captioning model, and the multimedia data captioning model comprises:
obtaining training samples comprising first sample multimedia data having caption labels;
training an initial captioning model based on the first sample multimedia data until the model loss function converges; and taking the trained captioning model as the multimedia data captioning model. 13. The method of claim 12, wherein: the training samples further comprise second sample multimedia data without the captioning labels, the model loss function comprising a first loss function and a second loss function;
Training the initial captioning model based on the first sample multimedia data until the model loss function converges:
To obtain a value of the first loss function, train a preset captioning model based on the first sample multimedia data, and to obtain a value of the second loss function, use the captioning model as the second 2 training based on the sample multimedia data;
obtaining a value of a final loss function based on the value of the first loss function and the value of the second loss function; and
training the captioning model based on the value of the final loss function until the final loss function converges. A device for generating caption information of multimedia data:
a memory storing one or more instructions; and
a processor configured to execute one or more instructions stored in the memory, the processor comprising:
extracting characteristic information of multimedia data to be processed, the multimedia data including video or images;
An apparatus for generating a text caption of the multimedia data based on the extracted characteristic information. A computer program product comprising a non-transitory computer readable storage medium, the storage medium storing a computer program that, when executed by a processor, performs the method of claim 1 .

Images