CN114494980B - Diverse video comment generation method, system, device and storage medium - Google Patents

Abstract

The invention discloses a method, system, device and storage medium for generating diverse video comments. In view of the problem of one-sided and single comments generated by the current video comment generating model, from the perspective of emotional diversity, the weight of emotional categories is introduced for labeling. The idea of variational autoencoder, modeling and controlling the emotional latent vector to guide the generation of diverse video comments with controllable emotions, can achieve high-quality real-time video comment generation, and can enhance the user's communication experience.

Description

Diverse video comment generation method, system, device and storage medium

technical field

The present invention relates to the technical field of video comment generation, and in particular, to a method, system, device and storage medium for generating diverse video comments.

Background technique

With the development of the times, the video barrage system has successively landed on popular video platforms such as BiliBili, iQiyi, and Youku. The wide application of the bullet screen system creates a two-way communication mode between users and videos, which enhances users' real-time participation in the video viewing process. The real-time barrage of videos can provide richer viewpoints, attract users' attention and discussion, and enhance users' communication experience. Therefore, the realization of high-quality real-time video comment (barrage) generation has great application value.

Current real-time video comment generation methods mostly use traditional end-to-end models, combining video clips and adjacent bullet screens to generate real-time comments. However, following the logic of comment generation, for the same video clip, it is affected by the commenter's point of view, emotional inclination, and thinking mode, and the comments show diverse characteristics. Most of the current real-time video comment generation methods only optimize for the quality of comments, but ignore the diversity characteristics of comments, and only generate a single video comment. For the same video clip and adjacent comment input, the ground truth (labeling information) reference comments often contain multiple types. The model generates a single comment, which is not only not conducive to performance evaluation and model optimization, but also does not meet the logical characteristics of comments.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a method, system, device and storage medium for generating diverse video comments, which realizes the generation of diverse video comments with controllable emotional tendencies.

The purpose of this invention is to realize through the following technical solutions:

A diverse video comment generation method including:

Utilize the video frame image of the current moment and its several nearest neighboring video frame images to construct a video frame image set, extract the comments in the video frame image of the current moment as a reference comment, and extract the comments in all the nearest video frame images to form a comment text;

Extract visual features from the video frame image set, extract text features from the comment text in combination with the visual features, and combine the emotion category weights corresponding to the reference comments to generate emotional latent vectors and encode them as emotional latent vector encoding features ;

Interact the input vocabulary with the generated vocabulary of the previous time step, the visual feature, the text feature and the emotional latent vector coding feature in turn to obtain the vocabulary probability distribution of the current time step, and determine the current time according to the vocabulary probability distribution of the current time step. The generated vocabulary of the step is combined with the generated vocabulary of all time steps to form the video comment at the current moment; wherein, the input vocabulary is the vocabulary in the reference comment or the vocabulary in the generated vocabulary of the previous time step.

A diverse video comment generation system including:

The information acquisition unit is used to construct a video frame image set using the video frame image at the current moment and some of its nearest neighboring video frame images, extract the comments in the video frame image at the current moment as a reference comment, and extract all the most adjacent video frame images. The comments constitute the comment text;

a visual encoder for extracting visual features from the set of video frame images;

a text encoder for extracting text features from the review text in combination with the visual features;

The latent vector encoder is used to combine the emotion category weights corresponding to the reference comments to generate emotional latent vectors and encode them as emotional latent vector coding features;

The comment decoder is used to interact the input vocabulary with the generated vocabulary of the previous time step, the visual feature, the text feature and the emotional latent vector coding feature in turn, so as to obtain the vocabulary probability distribution of the current time step; wherein, the input The vocabulary of is the vocabulary in the reference review or the vocabulary in the generated vocabulary of the previous time step;

The video comment generating unit is used for determining the generated words at the current time step according to the word probability distribution of the current time step, and synthesizing the generated words at all time steps to form the video comment at the current moment.

A processing device, comprising: one or more processors; a memory for storing one or more programs;

Wherein, when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the aforementioned method.

A readable storage medium storing a computer program, characterized in that the aforementioned method is implemented when the computer program is executed by a processor.

It can be seen from the above technical solutions provided by the present invention that, in view of the problem of one-sided and single comments generated by the current video comment generation model, from the perspective of emotional diversity, the emotional category weight is introduced as emotional annotation, and the idea of variational autoencoder is used for reference. , modeling and controlling the emotional latent vector to guide the generation of emotionally controllable and diverse video comments, which can achieve high-quality real-time video comment generation and enhance the user's communication experience.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

1 is a flowchart of a method for generating diverse video comments provided by an embodiment of the present invention;

2 is a schematic diagram of the overall structure of a diversified video comment generation model provided by an embodiment of the present invention;

3 is a schematic diagram of a system for generating diverse video comments provided by an embodiment of the present invention;

FIG. 4 is a schematic diagram of a processing device according to an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present invention.

First a description of terms that may be used in this article:

The terms "comprising", "comprising", "containing", "having" or other descriptions with similar meanings should be construed as non-exclusive inclusions. For example: including certain technical characteristic elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, processes, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.), should be construed to include not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

The following describes in detail a method, system, device and storage medium for generating a variety of video comments provided by the present invention. Contents that are not described in detail in the embodiments of the present invention belong to the prior art known to those skilled in the art. If the specific conditions are not indicated in the examples of the present invention, it is carried out according to the conventional conditions in the art or the conditions suggested by the manufacturer.

Example 1

As shown in Figure 1, a method for generating diverse video comments mainly includes the following steps:

Step 1, utilize the video frame image of the current moment and its several nearest adjacent video frame images to construct a video frame image set, extract the comments in the video frame image of the current moment as a reference comment, and extract the comments in all the nearest video frame images to form Comment text.

Step 2, extracting visual features from the video frame image collection, extracting text features from the comment text in combination with the visual features, and combining the emotion category weights corresponding to the reference comments to generate an emotional latent vector, and encode it as emotional latent vector. Vector encoding features.

Step 3. Interact the input vocabulary with the generated vocabulary of the previous time step, the visual feature, the text feature and the emotional latent vector coding feature in turn, to obtain the vocabulary probability distribution of the current time step, according to the vocabulary probability distribution of the current time step. The generated vocabulary of the current time step is determined, and the generated vocabulary of all time steps is combined to form the video comment at the current moment; wherein, the input vocabulary is the vocabulary in the reference comment or the vocabulary in the generated vocabulary of the previous time step.

In the above solution of the embodiment of the present invention, the visual feature extraction from the video frame image set is implemented by a visual encoder; the textual feature extraction from the comment text in combination with the visual feature is implemented by a text encoder; The emotion category weights are generated, and the emotion latent vector is generated and encoded into the emotion latent vector. The encoding feature is realized by the latent vector encoder; the input vocabulary is sequentially combined with the generated vocabulary, the visual feature, the text feature and the emotional latent vector of the previous time step. The encoded features interact with each other, which is implemented by a comment decoder based on the obtained word probability distribution at the current time step. The above-mentioned visual encoder, text encoder, latent vector encoder and comment decoder constitute a diversified video comment generation model. Figure 2 shows the overall structure of the diversified video comment generation model. The working principle of each part of the model and the loss function during training are introduced in detail.

1. Visual encoder.

As shown in Figure 2, the Video Encoder mainly includes a Convolutional Neural Network (CNN) and the first Transformer model. The first Transformer model mainly includes a Multi-head Attention sublayer and a fully connected feedforward. Networks (Position-wise Feed-Forward Networks).

In this embodiment of the present invention, a visual feature is extracted from the video frame image set by using a visual encoder. Denote the set of video frame images as F= { F ₁ , F ₂ ,..., F _J }, where F _j represents the j -th video frame image, and each video frame image corresponds to a moment, j =1,2,... , J , J represent the number of video frame images; the video frame image set F is the video frame image set in which the specified video includes the video frame at the specified moment and its nearest neighbor moment, F ₁ is the video frame image at the current moment, and the subsequent F ₂ ,... , F _J is the J -1 video frame image at the closest moment to the video frame image at the current moment.

First, the features of each video frame image (image) are extracted through a convolutional neural network, which is expressed as:

V _j = CNN( F _j )

In the above formula, V _j represents the feature of the extracted j -th video frame image F _j . Exemplarily, the convolutional neural network can use a ResNet network.

Denote the feature set V = { V ₁ , V ₂ ,..., V _J } corresponding to the video frame image set, and encode the feature set V of the video frame image set F through the first Transformer model, which is expressed as:

W _j = FNN _F (MultiHead-Atten _F ( V _j , V, V ) )

In the above formula, MultiHead-Atten _F and FNN _F respectively represent the multi-head attention module and the fully connected feedforward network in the first Transformer model; W _j represents the visual feature of the jth video frame image obtained by encoding processing.

Finally, the visual features of the video frame image set _F are denoted as WF = { W 1 _, W ₂ ,..., W _J }.

Those skilled in the art can understand that ( V _j , V, V ) is the input information of the multi-head attention module, which in turn corresponds to the query (query matrix), key (key matrix) and value (value matrix) in the multi-head attention, That is, the query value takes V _j , and the key and value take V . The multi-head attention module involved in the subsequent related formulas also has similar meanings. Considering that the expressions in the formulas are all common in the field, they will not be repeated here.

Second, the text encoder.

In the embodiment of the present invention, the text encoder (Text Encoder) includes a first linear encoding layer and a second Transformer model, and the second Transformer model includes two multi-head attention modules and a fully connected feedforward network; Figure 2 The text encoder and the sex-encoding layers contained in other parts are denoted as Embedding, similarly, all multi-head attention modules are denoted as Multi-head Attention, and all fully connected feed-forward networks are denoted as FeedForward.

In this embodiment of the present invention, text features are extracted from the comment text (context) by using the text encoder in combination with the visual features.

First, linearly encode the comment text through the first linear encoding layer to obtain a corresponding set of word embedding vectors e = { e ₁ , e ₂ ,..., e M _} , where em represents the _mth word in the comment text The word embedding vector of each vocabulary, m =1,2,…, M , where M is the total number of review text vocabulary. In this embodiment of the present invention, the comment text may be a bullet screen text, which is generally a bullet screen near a specified time of the specified video. The adjacent range here can be set by those skilled in the art according to actual needs or experience. The larger the range, the greater the number of words. many. Referring to the introduction in the aforementioned visual encoder, the specified moment is the current moment, then F ₁ is the video frame at the current moment, then the comment text contains the comments in the video frame images F ₂ , . . . , F _J .

After that, the word embedding vector set e is processed through the first multi-head attention module MultiHead-Atten _e1 in the second Transformer model, and then through the second multi-head attention module MultiHead-Atten _e2 and the fully connected feedforward network FNN _e Interacting the processing results of the first multi-head attention module with the visual features to obtain text features, the processing process is expressed as:

em '= _MultiHead - _{Atten e1} ( em , e , e )

E _m = FNN _e (MultiHead-Atten _e2 ( e _m ' , W _F , W _F )

Among them, em ' represents the processing result of the word embedding vector em of the mth _word by the first multi-head attention module, WF _represents the visual feature, and _{Em represents} the text feature corresponding to the mth word .

Finally, the text feature of the review text is recorded as We = { E ₁ , E _{2 ,} ..., E M _} .

Third, the hidden vector encoder.

In the embodiment of the present invention, the latent vector encoder (Latent Vector Encoder) introduces the coding principle of the variational auto-encoder on the basis of the Transformer model. By training a mixture of Gaussian distribution, the sampling emotional latent vector z guides the generation of diverse comments . The latent vector encoder generates the emotional latent vector z according to the reference comment (comment), the text feature We and the emotional _category weight, and then encodes it into the emotional latent vector encoding feature; the main principles are as follows:

The probability distribution p(z| c , We) of the sentiment _latent vector can be modeled as a mixture Gaussian distribution model weighted with the sentiment category weight _ck , expressed as:

表示第k个高斯分布模型，

与

分别表示建模定义的高斯分布模型的均值与方差，I表示标准单位矩阵，W _e 表示文本特征；z表示情感隐向量。Among them, _ck represents the k -th emotional category weight, K represents the number of emotional category weights, c represents the set of emotional category weights, c={ c _k } _K ,

represents the kth Gaussian distribution model,

and

respectively represent the mean and variance of the Gaussian distribution model defined by the modeling, I represents the standard unit matrix, We represent the text feature; _z represents the emotional latent vector.

As shown in FIG. 2 , the latent vector encoder includes: two linear coding layers, a third Transformer model, a multi-layer perceptron (MLP) and a sampling layer (sample).

The two linear coding layers (Embedding) are respectively called the second linear coding layer and the third linear coding layer, which are located at the beginning and end of the hidden vector encoder. The reference comments are linearly encoded by the second linear encoding layer to obtain the corresponding word embedding vector set d= { d ₁ , d ₂ , …, d _L }, where d _l represents the word embedding of the lth word in the reference comments vector, l = 1,2,…, L , where L is the total number of words in the reference reviews.

Referring also to FIG. 2, the third Transformer model includes two multi-head attention modules and a fully connected feedforward network. The word embedding vector set d is processed through the first multi-head attention module MultiHead-Atten _z1 , and then processed through the first multi-head attention module MultiHead-Atten z1. The two multi-head attention modules MultiHead-Atten _z2 and the fully connected feedforward network FNN _z interact with the processing results of the first multi-head attention module and the text features to obtain an intermediate hidden vector set h , and the processing process is expressed as:

d _l '= MultiHead-Atten _z1 ( d _l , d , d)

h _l = FNN _z ( _MultiHead -Atten _z2 ( d _l ' , We , We ₎ )

Among them, d _l ' represents the processing result of the l -th layer of the first multi-head attention module on the word embedding vector d _l of the l -th word in the reference review; when l = 2,..., L , the first multi-head attention The input of the l -th layer of the force module also contains the intermediate hidden vector h _{l -1} corresponding to the l -1th word in the reference comment; h _l represents the output of the l -th layer of the second multi-head attention module through a fully connected feedforward network. The intermediate hidden vector obtained by FNN _z processing, the intermediate hidden vector h _l corresponds to the lth word in the reference comment; the final processing obtains the intermediate hidden vector set h = { h ₁ , h ₂ , …, h _L }, and h _L is called is the last hidden vector.

Those skilled in the art can understand that when l =2,..., L , the first multi-head attention module MultiHead-Atten _z1 in the process of calculating d _l ' includes two types of inputs, one is d _l and d, the other The class is h _{l -1} , that is, the first multi-head attention module MultiHead-Atten _z1 will combine the output of the second multi-head attention module MultiHead-Atten _z2 and the fully connected feedforward network FNN _z to the set of word embedding vectors d is processed, but since this mechanism is already covered in the multi-head attention module, it is not shown by the formula.

The last layer of latent vector h _L is encoded as the mean and variance of the Gaussian distribution model through the multi-layer perceptron, which is expressed as:

where MLP stands for Multilayer Perceptron.

与方差

，带入式子p(z|c, W _e )获得情感隐向量z的概率分布；通过采样层采样获得情感隐向量z，再通过第三线性编码层编码为情感隐向量编码特征W _z。Combining the sentiment category weights ck obtained from the reference reviews and the mean of the Gaussian distribution model encoded by the multilayer _perceptron

with variance

, and the probability distribution of the emotional latent vector _{z is obtained by entering the formula p(z|c, We); the emotional latent vector z is obtained by sampling the sampling layer, and then encoded into the emotional latent vector encoding feature W z} through _the third linear coding layer .

与

尽可能的接近于

与

，可以有效建模选定的隐向量空间与生成评论直接的映射关系，测试阶段通过选择不同的情感类别权重c _k ，实现多样化的评论生成。The above corresponds to the inference phase, where the model passes the control

and

as close as possible

and

, which can effectively model the direct mapping relationship between the selected latent _{vector space and the generated comments. In the testing stage, by selecting different sentiment category weights ck} , a variety of comments can be generated.

In the embodiment of the present invention, the ready-made database SnowNLP may be used to perform an Emotion Analysis (Emotion Analysis) evaluation on the reference comment s, and SnowNLP (s) outputs an evaluation value in the [0,1] interval, which is used to illustrate the diversity of the reference comment s The higher the score, the more positive sentiment the sentence has. Emotional tendencies can be divided into three directions: positive, objective, and negative ( K = 3 ), from which the weight _ck of the corresponding emotion category can be obtained:

c ₁ = 1for T2<SnowNLP(s)≤T1;

c ₂ = 1for T3<SnowNLP(s)≤T2;

c3 ₌ 1else .

The emotional category weight is c={ c _1, c _2, c ₃ }. The above formula shows the condition that the weight of different emotional categories is equal to 1 in the case of K=3. In the above formula, T1, T2, and T3 are all The set threshold value satisfies T1>T2>T3. Exemplarily, T1=1, T2=0.7, and T3=0.3 can be set.

与方差

。In the embodiment of the present invention, the symbol _k of the emotional category weight ck indicates the emotional category, and the corresponding emotional category k can be determined according to the current reference comment, the corresponding emotional category weight ck ₌ 1, and the rest emotional category weights are all 0 , therefore, when bringing into the formula p(z|c, We _e ), only the mean value of the Gaussian distribution model corresponding to the emotion category k is needed

with variance

.

Fourth, the comment decoder.

As shown in FIG. 2 , the comment decoder (Comment Decoder) mainly includes a fourth linear coding layer, a fourth Transformer model, a linear layer (not shown) and a softmax layer.

The fourth linear coding layer is used to encode the input vocabulary and obtain the word embedding vector corresponding to the input vocabulary, denoted as y'; the inference stage directly samples the reference comment vocabulary as the input vocabulary, and the test stage uses the previous time The vocabulary generated by the step is used as the input vocabulary, as shown in Figure 2, an example of the vocabulary in the reference comment is given as the input vocabulary.

Referring also to Fig. 2, the fourth Transformer model includes four multi-head attention modules and a fully connected feedforward network; the word embedding vector y' is generated by the first multi-head attention module MultiHead-Atten _o1 and the previous time step The vocabulary interacts, and the interaction results of the first multi-head attention module are interacted with the visual features through the second multi-head attention module MultiHead-Atten _o2 , and the second multi-head attention module MultiHead-Atten _o3 is used. The interaction results of the multi-head attention modules interact with the text features, and the interaction results of the third multi-head attention module interact with the emotional latent vector encoding features through the fourth multi-head attention module MultiHead-Atten _o4 . And output the final decoded features through the fully connected feedforward network FNN _o , and the processing process is expressed as:

y _-1 = MultiHead-Atten _o1 (y' , y , y)

y _-2 = _MultiHead - _{Atten o2} ( y _-1 , WF , WF )

y _-3 = _MultiHead - _{Atten o3} ( y _-2 , We , We )

s _t = FNN _o (MultiHead-Atten _o4 ( y _-3 , W _z , W _z ) )

Wherein , y , WF , We, and _Wz represent the generated _vocabulary , the visual feature, the text feature, and the emotional latent vector coding feature of the previous time step in turn; s _t represents the final decoding feature _.

The final decoded feature s _t is passed through the linear layer and the softmax layer to obtain the vocabulary probability distribution of the current time step, which is expressed as:

p(y _t |y ₀ ,…,y _{t −1} , W _z , W _I , W _e ) = Softmax( Ws _t )

Among them, y ₀ ,...,y _{t −1} represents the vocabulary generated from the initial time step 0 to the previous time step t-1, that is, the generated vocabulary y of the previous time step, y _t represents the vocabulary generated at the current time step, and W represents the linear layer parameters.

In the embodiment of the present invention, the multi-head attention mechanism involved in the multi-head attention module in each part of the diversified video comment generation model can refer to the conventional technology, and the correlation formula shows the relevant processing process, and the processing obtains various features and The principle of the intermediate hidden vector can be referred to the conventional technology, which is not repeated in the present invention; in addition, in the introduction of the above comment decoder, the generated vocabulary y of the previous time step needs to be used. In the actual calculation process, the generated vocabulary y of the previous time step needs to be used. It needs to be converted into the corresponding set of embedded vectors y , but considering that this principle is a conventional technology, and the expression mainly displays the required data, therefore, those skilled in the art can understand the expression expressed by the current display method meaning.

5. Loss function.

的对数似然函数实现，而本发明定义的是一个由情感隐向量z控制的生成模型：Traditional encoder-decoder models generate reviews by maximizing

The log-likelihood function of , and the present invention defines a generative model controlled by the emotional latent vector z:

={y ₀ , y ₁ ,…}，即为当前时刻的视频评论，它包含当前时刻所有时间步的生成词汇。Among them, generate comments

={ y ₀ , y ₁ ,… } is the video comment at the current moment, which contains the generated vocabulary of all time steps at the current moment.

的对数似然函数的一个变分下界（ELBO）：Since it is impossible to traverse all the emotional latent vectors z for integration, the video comment at the current moment can be obtained by drawing on the mathematical derivation in the variational autoencoder.

A variational lower bound (ELBO) of the log-likelihood function of :

))≥ E _z [log p(

|z)]− D _KL [q(z|

), p(z)] log ( p (

)) ≥ E _z [ log p (

| z)] − D _KL [ q (z |

) , p (z)]

)表示生成当前时刻的视频评论

的概率分布，p(

|z)表示条件为情感隐向量z时生成当前时刻的视频评论

的概率分布，p(z)为情感隐向量z的分布，E _z[.]表示求取关于情感隐向量z的数学期望，D _KL 表示计算KL距离（相对熵），q(z|

)对应隐向量编码器所获取的概率分布，用于近似评论解码器的后验概率分布p(z|

)。因此，本发明的优化目标为最大化对数似然函数的变分下界。where, p (

) means generating the video comment at the current moment

The probability distribution of , p (

| z) indicates that the video comment at the current moment is generated when the condition is the emotional latent vector z

The probability distribution of , p (z) is the distribution of the emotional latent vector z, E _z [ . ] represents the mathematical expectation about the emotional latent vector z, D _KL represents the calculation of the KL distance (relative entropy), q (z |

) corresponds to the probability distribution obtained by the latent vector encoder, which is used to approximate the posterior probability distribution p (z |

). Therefore, the optimization objective of the present invention is to maximize the variational lower bound of the log-likelihood function.

Since the present invention is also affected by the set of word embedding vectors e corresponding to the video frame image set F and the comment text, the objective function L can be transformed into:

|z, F, e )] − D _KL [q(z|

, F, e), p(z|F, e)] L = E _{q (z|F, e)} [log p (

|z, F, e )] − D _KL [q(z|

, F, e), p (z|F, e)]

|z, F, e )表示以情感隐向量z，视频帧图像集合F以及评论文本对应的词嵌入向量集合e作为条件时，生成当前时刻的视频评论

的概率分布；E _{q(z| F, e)}[.]表示求取关于q(z| F, e)的数学期望，q(z| F, e)表示以视频帧图像集合F以及评论文本对应的词嵌入向量e作为条件的情感隐向量z的概率分布；q(z|

, F, e)表示以当前时刻的视频评论

，视频帧图像F集合以及评论文本对应的词嵌入向量集合e作为条件时，情感隐向量z的概率分布；p(z|F, e)表示以视频帧图像集合F以及评论文本对应的词嵌入向量集合e作为条件时，情感隐向量z的概率分布。第一项与传统模型的对数似然函数类似，鼓励生成更高质量的评论，即为重构损失。第二项鼓励训练得到的情感隐向量z分布能够尽可能的接近于先验分布p(z|F, e)，先验概率设置为标准正态分布N(0,1)，避免模型在训练过程中趋于同化，丢失多样性。where, p (

|z, F, e ) means that the video comment at the current moment is generated when the emotional latent vector z, the video frame image set F and the word embedding vector set e corresponding to the comment text are used as conditions

The probability distribution of ; E _{q (z| F, e)} [.] means to find the mathematical expectation about q (z| F, e), q (z| F, e) means to use the video frame image set F and the comment text The corresponding word embedding vector e is used as the probability distribution of the conditional emotional latent vector z; q(z|

, F, e) represent the video comment at the current moment

, when the video frame image set F and the word embedding vector set e corresponding to the comment text are used as conditions, the probability distribution of the emotional latent vector z; p (z|F, e) represents the word embedding corresponding to the video frame image set F and the comment text The probability distribution of the emotional latent vector z when the vector set e is used as the condition. The first term is similar to the log-likelihood function of traditional models and encourages the generation of higher quality reviews, which is the reconstruction loss. The second item encourages the emotional latent vector z distribution obtained by training to be as close as possible to the prior distribution p(z|F, e), and the prior probability is set to the standard normal distribution N(0,1) to avoid the model training The process tends to assimilate and lose diversity.

)对应隐向量编码器所获取的情感隐向量z的概率分布，用于近似评论解码器的后验概率分布p(z|

)，但由于后验概率分布p(z|

)不能够计算得到，因此，使用了不同表达形式进行区分。In this embodiment of the present invention, p and q are both probability distributions, and two expressions are used to distinguish different probability distributions; as described above, q (z |

) corresponds to the probability distribution of the sentiment latent vector z obtained by the latent vector encoder, which is used to approximate the posterior probability distribution p (z |

), but due to the posterior probability distribution p (z |

) cannot be calculated, therefore, different expressions are used to distinguish.

The above solution of the embodiment of the present invention, aiming at the problem of one-sided and single comments generated by the current video comment generation model, from the perspective of emotional diversity, an emotional analysis module is introduced for labeling, and based on the Transformer model, the idea of a variational autoencoder is used for reference. , modeling and controlling the latent vector z of emotion to guide emotion-controllable and diverse video comment generation. It is worth noting that the introduced sentiment analysis module (ie Emotion Analysis in Figure 2) is independent of the entire model, so it can be replaced with other sentiment analyzers to achieve more fine-grained and controllable sentiment comment generation; or replaced with The theme analysis module realizes the generation of comments with controllable themes, etc. It can be seen that the solution provided by the present invention has strong promotion and application value.

Embodiment 2

The present invention also provides a system for generating diverse video comments, which is mainly implemented based on the method provided in the first embodiment. As shown in FIG. 3 , the system mainly includes:

The information acquisition unit is used to construct a video frame image set using the video frame image at the current moment and some of its nearest neighboring video frame images, extract the comments in the video frame image at the current moment as a reference comment, and extract all the most adjacent video frame images. The comments constitute the comment text;

a visual encoder for extracting visual features from the set of video frame images;

a text encoder for extracting text features from the review text in combination with the visual features;

The latent vector encoder is used to combine the emotion category weights corresponding to the reference comments to generate emotional latent vectors and encode them as emotional latent vector coding features;

The comment decoder is used to interact the input vocabulary with the generated vocabulary of the previous time step, the visual feature, the text feature and the emotional latent vector coding feature in turn, so as to obtain the vocabulary probability distribution of the current time step; wherein, the input The vocabulary of is the vocabulary in the reference review or the vocabulary in the generated vocabulary of the previous time step;

The video comment generating unit is used for determining the generated words at the current time step according to the word probability distribution of the current time step, and synthesizing the generated words at all time steps to form the video comment at the current moment.

Those skilled in the art can clearly understand that, for the convenience and conciseness of the description, only the division of the above-mentioned functional modules is used for illustration. In practical applications, the above-mentioned functions can be allocated to different functional modules as required. The internal structure of the system is divided into different functional modules to complete all or part of the functions described above.

It should be noted that the main principles of each part of the system have been described in detail in the previous Embodiment 1, so they will not be repeated.

Embodiment 3

The present invention also provides a processing device, as shown in FIG. 4 , which mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are described When executed by one or more processors, the one or more processors are caused to implement the methods provided by the foregoing embodiments.

Further, the processing device further includes at least one input device and at least one output device; in the processing device, the processor, the memory, the input device, and the output device are connected through a bus.

In this embodiment of the present invention, the specific types of the memory, the input device, and the output device are not limited; for example:

The input device can be a touch screen, an image capture device, a physical button or a mouse, etc.;

The output device can be a display terminal;

The memory may be random access memory (Random Access Memory, RAM), or may be non-volatile memory (non-volatile memory), such as disk memory.

Embodiment 4

The present invention also provides a readable storage medium storing a computer program, and when the computer program is executed by a processor, the methods provided by the foregoing embodiments are implemented.

In this embodiment of the present invention, the readable storage medium, as a computer-readable storage medium, may be provided in the aforementioned processing device, for example, as a memory in the processing device. In addition, the readable storage medium may also be a U disk, a removable hard disk, a read-only memory (Read-Only Memory, ROM), a magnetic disk, or an optical disk, and other mediums that can store program codes.

The above description is only a preferred embodiment of the present invention, but the protection scope of the present invention is not limited to this. Substitutions should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (5)

1. a method for generating diverse video comments, is characterized in that, comprising: Utilize the video frame image of the current moment and its several nearest neighboring video frame images to construct a video frame image set, extract the comments in the video frame image of the current moment as a reference comment, and extract the comments in all the nearest video frame images to form a comment text; Extract visual features from the video frame image set, extract text features from the comment text in combination with the visual features, and combine the emotion category weights corresponding to the reference comments to generate emotional latent vectors and encode them as emotional latent vector encoding features ; Interact the input vocabulary with the generated vocabulary of the previous time step, the visual feature, the text feature and the emotional latent vector coding feature in turn to obtain the vocabulary probability distribution of the current time step, and determine the current time according to the vocabulary probability distribution of the current time step. The generated vocabulary of the step, synthesizing the generated vocabulary of all time steps constitutes the video comment at the current moment; wherein, the input vocabulary is the vocabulary in the reference comment or the vocabulary in the generated vocabulary of the previous time step; Wherein, extracting visual features from the video frame image set is implemented by a visual encoder; extracting text features from the comment text in combination with the visual features is implemented by a text encoder; combined with the emotion category weights corresponding to the reference comments, generate emotion The hidden vector is encoded into the emotional latent vector encoding feature through the hidden vector encoder; the input vocabulary is interacted with the generated vocabulary, the visual feature, the text feature and the emotional latent vector encoding feature in the previous time step in turn to obtain the current The lexical probability distribution at time steps is implemented by the comment decoder; Extracting visual features from the set of video frame images includes: using a visual encoder comprising a convolutional neural network and a first Transformer model to extract visual features from the set of video frame images; Denote the set of video frame images as F= { F ₁ , F ₂ ,…, F _J }, where F _j represents the jth video frame image, j =1, 2,…, J , J represents the number of video frame images ; Each video frame image corresponds to _a moment, and the video frame image at the current moment is F ₁ , F ₂ , . _. . The features of each video frame image are extracted through a convolutional neural network, which is expressed as: V _j = CNN( F _j ) In the above formula, CNN represents a convolutional neural network, and V _j represents the feature of the extracted j -th video frame image F _j ; Denote the feature set V = { V ₁ , V ₂ ,..., V _J } corresponding to the video frame image set F, and encode the feature set V corresponding to the video frame image set F through the first Transformer model, which is expressed as: W _j = FNN _F (MultiHead-Atten _F ( V _j , V, V ) ) In the above formula, MultiHead-Atten _F and FNN _F respectively represent the multi-head attention module and the fully connected feedforward network in the first Transformer model; W _j represents the visual feature of the jth video frame image obtained by encoding processing; Denote the visual features of the video frame image set _F as WF = { W 1 _, W ₂ ,..., W _J }; The extracting text features from the review text in combination with the visual features includes: using a text encoder including a first linear encoding layer and a second Transformer model in combination with the visual features to extract text features from the review text; Wherein, the comment text is linearly encoded by the first linear encoding layer, and the corresponding word embedding vector set e = { e ₁ , e ₂ ,..., e M _} is obtained, where em represents the mth _in the comment text word embedding vector of each vocabulary, m =1,2,…, M , where M is the total number of comment text vocabulary; The second Transformer model includes two multi-head attention modules and a fully connected feedforward network. The first multi-head attention module MultiHead-Atten _e1 processes the word embedding vector set e, and then passes the second multi-head attention module. The module MultiHead-Atten _e2 interacts with the fully connected feed-forward network FNN _e to interact the processing results of the first multi-head attention module with the visual features to obtain text features. The processing process is expressed as: em '= _MultiHead - _{Atten e1} ( em , e , e ) E _m = FNN _e (MultiHead-Atten _e2 ( e _m ' , W _F , W _F ) ) Among them, em ' represents the processing result of the word embedding vector em of the mth _{vocabulary by} the first multi-head attention module, WF represents the visual feature, and _{Em represents} the text feature corresponding to the mth vocabulary ; Denote the text feature of the comment text as We = { E ₁ , E _{2 ,} ..., E M _} ; The generating the emotional latent vector in combination with the emotional category weights corresponding to the reference comments, and encoding them into emotional latent vector coding features includes: determining the emotional category weights by performing emotional analysis on the reference comments, and combining the reference comments, text, and text by using a hidden vector encoder. The feature and emotion category weight generate emotion latent vector, and encode it as emotion latent vector encoding feature; Among them, the probability distribution p(z| c , We _e ) of the emotional latent vector is modeled as a mixed Gaussian distribution model weighted by the emotional category weight _ck , which is expressed as:

Among them, _ck represents the k -th emotional category weight, K represents the number of emotional category weights, c represents the set of emotional category weights, c={ c _k } _K ,

represents the kth Gaussian distribution model,

and

respectively represent the mean and variance of the Gaussian distribution model defined by the modeling, I represents the standard unit matrix, We _e represents the text feature; z represents the emotional latent vector; The latent vector encoder includes: two linear coding layers, a third Transformer model, a multi-layer perceptron and a sampling layer; The two linear encoding layers are respectively called the second linear encoding layer and the third linear encoding layer. The reference comments are linearly encoded through the second linear encoding layer to obtain the corresponding word embedding vector set d= { d ₁ , d ₂ , … , d _L }, where d _l represents the word embedding vector of the lth word in the reference comment, l =1,2,…, L , L is the total number of words in the reference comment; The third Transformer model includes two multi-head attention modules and a fully connected feedforward network. The word embedding vector set d is processed through the first multi-head attention module MultiHead-Atten _z1 , and then the second multi-head attention module is used. The module MultiHead-Atten _z2 and the fully connected feedforward network FNN _z interact with the processing results of the first multi-head attention module and the text features to obtain the intermediate hidden vector set h , and the processing process is expressed as: d _l '= MultiHead-Atten _z1 ( d _l , d , d) h _l = FNN _z ( _MultiHead -Atten _z2 ( d _l ' , We , We ₎ ) Among them, d _l ' represents the processing result of the l -th layer of the first multi-head attention module on the word embedding vector d _l of the l -th word in the reference review; when l = 2,..., L , the first multi-head attention The input of the l -th layer of the force module also contains the intermediate hidden vector h _{l -1} corresponding to the l -1th word in the reference comment; h _l represents the output of the l -th layer of the second multi-head attention module through a fully connected feedforward network. The intermediate hidden vector obtained by FNN _z processing, the intermediate hidden vector h _l corresponds to the lth word in the reference comment; the final processing obtains the intermediate hidden vector set h = { h ₁ , h ₂ , …, h _L }, and h _L is called is the last hidden vector; The last layer of latent vector h _L is encoded as the mean and variance of the Gaussian distribution model through the multi-layer perceptron, which is expressed as:

Among them, MLP represents multi-layer perceptron; The sentiment category weight _ck of the reference comment and the mean and variance of the Gaussian distribution model encoded by the multilayer perceptron are brought into the formula p(z| c , _We ) to obtain the probability distribution of the emotional latent vector z; through the sampling layer Sampling to obtain an emotional latent vector z, and then encoding into the emotional latent vector coding feature W _z through the third linear coding layer; Interact the input vocabulary with the generated vocabulary of the previous time step, the visual feature, the text feature and the emotional latent vector coding feature in turn to obtain the vocabulary probability distribution of the current time step, and determine the current time according to the vocabulary probability distribution of the current time step. The generated vocabulary of the step is realized by the comment decoder; The comment decoder includes a fourth linear coding layer, a fourth Transformer model, a linear layer and a softmax layer; Among them, the fourth linear coding layer is used to encode the input vocabulary, and obtain the word embedding vector corresponding to the input vocabulary, denoted as y'; The fourth Transformer model includes four multi-head attention modules and a fully-connected feedforward network; the word embedding vector y' interacts with the generated vocabulary of the previous time step through the first multi-head attention module MultiHead-Atten _o1 . The second multi-head attention module _MultiHead -Atten _o2 interacts the interaction results of the first multi-head attention module with the visual features, and the second multi-head attention module The interaction result of the third multi-head attention module interacts with the text feature, and the interaction result of the third multi-head attention module interacts with the emotional latent vector encoding feature through the fourth multi-head attention module MultiHead-Atten _o4 . Feed the network FNN _o to output the final decoded features, and the processing process is expressed as: y _-1 = MultiHead-Atten _o1 (y' , y , y) y _-2 = _MultiHead - _{Atten o2} ( y _-1 , WF , WF ) y _-3 = _MultiHead - _{Atten o3} ( y _-2 , We , We ) s _t = FNN _o (MultiHead-Atten _o4 ( y _-3 , W _z , W _z ) ) Wherein , y , WF , We, and _Wz represent the generated _vocabulary , the visual feature, the text feature _, and the emotional latent vector coding feature of the previous time step in turn; s _t represents the final decoding feature; The final decoded feature s _t goes through the linear layer and the softmax layer in turn to obtain the word probability distribution at the current moment, which is expressed as: p(y _t |y ₀ ,…,y _{t −1} , W _z , W _F , W _e ) = Softmax( W s _t ) Among them, y ₀ ,...,y _{t −1} represents the vocabulary generated from the initial time step 0 to the previous time step t-1, that is, the generated vocabulary y in the previous time step, y _t represents the vocabulary generated at the current time step t, and W represents Parameters of the linear layer. 2. a kind of diversified video comment generation method according to claim 1, is characterized in that, described visual encoder, text encoder, latent vector encoder and comment decoder constitute a variety of video comment generation model, training stage , the objective function L of the diverse video review generation model is expressed as: L = E _{q (z|F, e)} [log p (

|z, F, e )] − D _KL [q(z|

, F, e), p (z|F, e)] Among them, z represents the emotional latent vector obtained by the latent vector encoder, F represents the video frame image set, e represents the word embedding vector set corresponding to the comment text,

Indicates the video comment at the current moment; p (

|z, F, e ) means that the video comment at the current moment is generated when the emotional latent vector z, the video frame image set F and the word embedding vector set e corresponding to the comment text are used as conditions

The probability distribution of ; E _{q (z| F, e)} [.] means to find the mathematical expectation about q (z| F, e), q (z| F, e) means to use the video frame image set F and the comment text The corresponding word embedding vector e is used as the probability distribution of the conditional emotional latent vector z; q(z|

, F, e) represent the video comment at the current moment

, when the video frame image set F and the word embedding vector set e corresponding to the comment text are used as conditions, the probability distribution of the emotional latent vector z; p (z|F, e) represents the word embedding corresponding to the video frame image set F and the comment text When the vector set e is used as the condition, the probability distribution of the emotional latent vector z, D _KL represents the calculation of the KL distance. 3. a variety of video comment generation system, is characterized in that, based on the method described in any one of claim 1～2, this system comprises: The information acquisition unit is used to construct a video frame image set using the video frame image at the current moment and some of its nearest neighboring video frame images, extract the comments in the video frame image at the current moment as a reference comment, and extract all the most adjacent video frame images. The comments constitute the comment text; a visual encoder for extracting visual features from the set of video frame images; a text encoder for extracting text features from the review text in combination with the visual features; The latent vector encoder is used to combine the emotion category weights corresponding to the reference comments to generate emotional latent vectors and encode them as emotional latent vector coding features; The comment decoder is used to interact the input vocabulary with the generated vocabulary of the previous time step, the visual feature, the text feature and the emotional latent vector coding feature in turn, so as to obtain the vocabulary probability distribution of the current time step; wherein, the input The vocabulary of is the vocabulary in the reference review or the vocabulary in the generated vocabulary of the previous time step; The video comment generating unit is used for determining the generated words at the current time step according to the word probability distribution of the current time step, and synthesizing the generated words at all time steps to form the video comment at the current moment. 4. A processing device, comprising: one or more processors; a memory for storing one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the method according to any one of claims 1-2. 5. A readable storage medium storing a computer program, wherein the method according to any one of claims 1 to 2 is implemented when the computer program is executed by a processor.

Images