CN105279495A - Video description method based on deep learning and text summarization - Google Patents

Abstract

The invention discloses a video description method based on deep learning and text summarization. The video description method comprises the following steps: through a traditional image data set, training a convolutional neural network model according to an image classification task; extracting a video frame sequence of a video, utilizing the convolutional neural network model to extract convolutional neural network characteristics to form a <video frame sequence, text description sequence> pair which is used as the input of a recurrent neural network model, and training to obtain the recurrent neural network model; describing the video frame sequence of the video to be described through the recurrent neural network model obtained by training to obtain description sequences; and through a method that graph-based vocabulary centrality is used as the significance of the text summarization, sorting the description sequences, and outputting a final description result of the video. An event which happens in one video and object attributes associated with the event are described through a natural language so as to achieve a purpose that video contents are described and summarized.

Description

A video description method based on deep learning and text summarization

technical field

The invention relates to the field of video description, in particular to a video description method based on deep learning and text summarization.

Background technique

Using natural language to describe a video is extremely important for both understanding the video and retrieving the video on the Web. At the same time, the language description of video is also a key research topic in the field of multimedia and computer vision. The so-called video description means that for a given video, by observing the content contained in it, the video features are obtained, and corresponding sentences are generated according to these contents. When people see a video, especially some action videos, they will have a certain degree of understanding of the video after watching the video, and can use language to tell what happened in the video. Example: Describe a video with a sentence like "A man is riding a motorcycle." However, in the face of a large number of videos, it takes a lot of time, manpower and financial resources to manually describe the videos one by one. It is very necessary to use computer technology to analyze video features and combine them with natural language processing methods to generate video descriptions. On the one hand, through the method of video description, people can understand videos more precisely from the perspective of semantics. On the other hand, in the field of video retrieval, when a user enters a textual description to retrieve the corresponding video, it is very difficult and has certain challenges.

In the past few years, a variety of video description methods have emerged. For example, by analyzing video features, objects in the video and the action relationship between objects can be identified. Then use a fixed language template: subject + verb + object, determine the subject and object from the recognized objects, and use the action relationship between objects as the predicate, and use this method to generate sentences to describe the video.

However, there are certain limitations in such a method. For example, the use of language templates to generate sentences can easily lead to a relatively fixed sentence structure, which is too single and lacks the color of human natural language expression. At the same time, different features are required to recognize objects and actions in the video, which makes the steps relatively cumbersome and requires a lot of time to train the video features. Not only that, the accuracy of recognition directly affects the quality of generated sentences. This step-by-step method needs to ensure high accuracy at each step, and it is difficult to implement.

Contents of the invention

The present invention provides a video description method based on deep learning and text summarization. The present invention uses natural language to describe the events that are happening in a video and the attributes of objects related to the events, so as to achieve the purpose of describing and summarizing the video content. See the description below for details:

A video description method based on deep learning and text summarization, characterized in that the video description method comprises the following steps:

Download videos from the Internet, and describe each video to form a <video, description> pair to form a text description training set;

Train the convolutional neural network model according to the image classification task through the existing image data set;

Extract video frame sequence to video, and utilize convolutional neural network model to extract convolutional neural network feature, form <video frame sequence, text description sequence> pair as the input of recurrent neural network model, train and obtain recursive neural network model;

Describe the video frame sequence of the video to be described by the recursive neural network model obtained through training, and obtain the description sequence;

By using graph-based lexical centrality as a saliency method for text summarization, the sequence of descriptions is sorted, and the final description result of the video is output.

The downloading of videos from the Internet, and describing each video to form a pair of <video, description> to form a text description training set is as follows:

The <video, description> pair is composed of the existing video collection and the sentence description corresponding to each video to form a text description training set.

The step of extracting a video frame sequence from the video, extracting convolutional neural network features using a convolutional neural network model, forming a <video frame sequence, text description sequence> pair as the input of a recurrent neural network model, and training to obtain a recurrent neural network model Specifically:

Use the parameters after training the convolutional neural network model to extract the convolutional neural network features of the image and the sentence description corresponding to the image for modeling to obtain the objective function;

Construct a recurrent neural network; model nonlinear functions through long and short-term memory networks;

Use gradient descent to optimize the objective function and obtain the trained long-short-term memory network parameters.

Describe the video frame sequence of the video to be described by the recursive neural network model obtained through training, and the steps of obtaining the description sequence are specifically:

Use the trained model parameters and use the convolutional neural network model to extract the convolutional neural network features of each image to obtain image features;

The image feature is used as input and the model parameters obtained by training are used to obtain a sentence description, so as to obtain a sentence description corresponding to the video.

The beneficial effect of the technical solution provided by the present invention is: each video is composed of a sequence of frames, and the convolutional neural network is used to extract the underlying features of each frame of the video. This method can effectively avoid the traditional use of deep learning to extract video features. A lot of noise will reduce the accuracy of the sentence generated in the later stage. Use the trained recurrent neural network to convert each frame of pictures into sentences to generate a collection of sentences. And use the method of automatic text summarization to calculate the centrality between sentences and select high-quality and representative sentences from the collection of sentences as video descriptions. This method can produce better video description effects and accuracy. Sex and sentence variety. At the same time, the method based on depth and text summarization can be effectively extended to the application of video retrieval, but this method is limited to the English description of video content.

Description of drawings

Fig. 1 is a flow chart of a video description method based on deep learning and text summarization;

The convolutional neural network model (CNN) schematic diagram that Fig. 2 present invention uses;

Among them, Cov represents the convolution kernel; ReLU represents the formula as max(0,x); Pool represents the Pooling operation; LRN represents the local corresponding normalization operation; Softmax represents the objective function.

The schematic diagram of the recursive neural network used in the present invention of Fig. 3;

Among them, t represents the input in the t state; h _t-1 represents the hidden state of the previous state; i is the input gate; f is the forgetgate; o is the output gate; c is the cell; m _t is the output after passing through an LSTM unit.

Figure 4(a) is the connection diagram after LexRank pruning;

Among them, S={S ₁ ,…,S ₁₀ } are 10 sentences generated by the recurrent neural network (RNN), and these 10 sentences are represented as 10 nodes in graph mode; the similarity between nodes is obtained by A straight line is used to represent and form a fully connected graph, and the thickness of the connection represents the size of the similarity.

Figure 4(b) is the initial fully connected graph of LexRank;

By setting the threshold, the connection lines with low similarity between nodes are removed, and the remaining connections between nodes, that is, the similarity between sentences is relatively high.

Fig. 5 is a schematic diagram of sentences generated after partial video frames are described.

Wherein, below each frame of image is the sentence generated after adopting the CNN-RNN model used in the present invention, and its arrow points to a summary of the video text description after the LexRank method as the text description of the video.

detailed description

In order to make the purpose, technical solution and advantages of the present invention clearer, the implementation manners of the present invention will be further described in detail below.

Based on the problems in the background technology, and the effect of using deep learning methods in images to describe images has been significantly improved, people are inspired by it and use deep learning methods in videos. The generated video descriptions are diverse. The accuracy and accuracy have been improved to a certain extent.

To this end, the embodiment of the present invention proposes a video description method based on deep learning and text summarization. First, this method extracts the visual features of each frame of the video through a convolutional neural network framework. Then, each video feature is used as input into the recurrent neural network framework, which can generate a sentence description for each visual feature, that is, each frame of the video. In this way, a collection of sentences is obtained. In order to obtain the most expressive and high-quality sentences as the description of the video, this method adopts the method of text summarization and sorts all sentences by calculating the similarity between sentences, so that Some wrong sentences and low-quality sentences are avoided as the final description of the video. The method of automatic text summarization can not only get a representative sentence, but also has certain correctness and reliability, thus improving the accuracy of video description. At the same time, this method also overcomes some technical difficulties faced by video retrieval.

Example 1

A video description method based on deep learning and text summary, see Figure 1, the method includes the following steps:

101: Download videos from the Internet, and describe each video (in English), form a <video, description> pair, and form a text description training set, wherein each video corresponds to multiple sentence descriptions, thereby forming a text description sequence;

102: Utilize the existing image data set to train a convolutional neural network (CNN) model according to the image classification task;

For example: ImageNet.

103: Extract the video frame sequence from the video, and use the convolutional neural network (CNN) model to extract CNN features, form a <video frame sequence, text description sequence> pair as the input of the recurrent neural network (RNN) model, and train the recurrent neural network (RNN) model;

104: Using the trained RNN model to describe the video frame sequence of the video to be described to obtain a description sequence;

105: Use graph-based lexical centrality as the method of text summary significance (LexRank) to rank the rationality of the description sequence, and select the most reasonable description as the final description of the video.

To sum up, the embodiment of the present invention realizes the description of the event occurring in a video and the object attributes related to the event through natural language through steps 101 to 105, so as to achieve the purpose of describing and summarizing the video content.

Example 2

201: Download images from the Internet, and describe each video to form a <video, description> pair to form a text description training set;

This step specifically includes:

(1) Download the Microsoft Research Video Description Dataset (MicrosoftResearchVideoDescriptionCorpus) from the Internet. This data set includes 1970 video segments collected from YouTube. The data set can be expressed as $VI\mathrm{D.}=\{{\mathrm{video}}_1,...,{\mathrm{video}}_{N_d}\},$ where _Nd is the video in the collection VID total.

(2) Each video will have multiple corresponding descriptions, and the sentence description of each video is Sentences={Sentence ₁ ,...,Sentence _N }, where N represents the sentence corresponding to each video (Sentence ₁ ,..., Sentence _N ) description number.

(3) The <video, description> pair is composed of the existing video collection VID and the sentence description Sentences corresponding to each video to form a text description training set.

202: Utilize the existing image data set, train a convolutional neural network (CNN) model according to the image classification task, and train CNN model parameters;

This step specifically includes:

(1) Construct the AlexNet[1]CNN model shown in Figure 2: This model includes 8 network layers, of which the first 5 layers are convolutional layers, and the last 3 layers are fully connected layers.

(2) Use Imagenet as the training set, sample each picture in the image data set to a picture of 256*256 size, $ImAG\mathrm{E.}=\{{\mathrm{image}}_1,...,{\mathrm{image}}_{N_m}\}$ As input, N _m is the picture The number of , according to the network layer set in Figure 2, the first layer can be expressed as:

F ₁ (IMAGE)=norm{pool[max(0,W ₁ *IMAGE+B ₁ )]}(1)

Among them, IMAGE represents the input image; W ₁ represents the convolution kernel parameter; B ₁ represents the bias; F ₁ (IMAGE) represents the output result after the first layer of network; Norm represents the normalization operation. In this network layer, the convolved image is processed through the linear correction function (max(0,x), x is W ₁ *IMAGE+B ₁ ), and then the mapping pool operation is performed, and local corresponding Normalization (LRN), the normalization method is:

${{\mathrm{<span\; class="notranslate">\; b</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}}_{\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}}<span\; class="notranslate">\; =</span>{{\mathrm{<span\; class="notranslate">\; a</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}}_{\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}}<span\; class="notranslate">\; /</span>{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; max</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>}{\overset{\mathrm{<span\; class="notranslate">\; min</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; no</span>}<span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>}{<span\; class="notranslate">\; \&Sigma;</span>}}{<span\; class="notranslate">\; (</span>{{\mathrm{<span\; class="notranslate">\; a</span>}}^{\mathrm{<span\; class="notranslate">\; j</span>}}}_{\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span>$

Among them, M is the number of feature maps after pooling; i is the i-th of M feature maps; n is the size of local normalization, that is, every n feature maps are normalized; a ⁱ _{x, y} means The value corresponding to the coordinate (x, y) in the i-th feature map; k is the bias; α, β are normalized parameters; b ⁱ _{x, y} are after local corresponding normalization (LRN) output result.

In AlexNet, k=2, n=5, α=10 ⁻⁴ , β=0.75.

Continuing to use this model, F ₁ (IMAGE) is used as the input of the second network layer. According to the second network layer, it can be expressed as:

F ₂ (IMAGE)=max(0,W ₂ *F ₁ (IMAGE)+B ₂ )(3)

Among them, W ₂ represents the convolution kernel parameter; B ₂ represents the bias; F ₂ (IMAGE) represents the output result after passing through the second layer network. The settings of the first layer and the second layer are the same, but the size of the mapping kernel kernel of the convolution layer and the pooling layer changes.

According to AlexNet's network settings, the remaining convolutional layers can be expressed in turn as:

F ₃ (IMAGE)=max(0,W ₃ *F ₂ (IMAGE)+B ₃ )(4)

F ₄ (IMAGE)=max(0,W ₄ *F ₃ (IMAGE)+B ₄ )(5)

F ₅ (IMAGE)=pool[max(0,W ₅ *F ₄ (IMAGE)+B ₅ )](6)

Among them, W ₃ , W ₄ , W ₅ and B ₃ , B ₄ , B ₅ are the convolution parameters and biases of each layer.

The last three layers are fully connected layers, which can be expressed in sequence according to the network layer settings in Figure 2:

F ₆ (IMAGE)=fc[F ₅ (IMAGE),θ ₁ ](7)

F ₇ (IMAGE)=fc[F ₆ (IMAGE),θ ₂ ](8)

F ₈ (IMAGE)=fc[F ₇ (IMAGE),θ ₃ ](9)

Among them, fc represents the fully connected layer, θ ₁ , θ ₂ , and θ ₃ represent the parameters of the three fully connected layers, and the feature F ₈ (IMAGE) of the last layer is input to a multi-class classifier with 1000 categories for classification.

(3) According to the current network, set a multivariate classifier, and its formula can be expressed as:

$\mathrm{<span\; class="notranslate">\; l</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; \&Theta;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; log</span>}\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; x</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; ;</span>\mathrm{<span\; class="notranslate">\; \&Theta;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 10</span>}<span\; class="notranslate">\; )</span>$

Among them, l(Θ) is the objective function, m is the category of the image in Imagenet, x ^(t) is the CNN feature extracted after each category passes through the Alexnet network, y ^(t) is the label corresponding to each image, Θ={ W _p , B _p , θ _q }, p=1,...,5, q=1, 2, 3 are parameters in each network layer respectively. The gradient descent method is used to optimize the parameters of the objective function, so as to obtain the parameter Θ set by the Alexnet network.

203: Extract the video frame sequence from the video, and use the convolutional neural network (CNN) model to extract CNN features, form a <video frame sequence, text description sequence> pair as the input of the recurrent neural network (RNN) model, and train the recurrent neural network (RNN) model;

The steps are specifically:

(1) According to step 201, use the parameters after training the CNN model to extract the CNN feature I of the image, and the sentence description S corresponding to the image to model, and its objective function is:

θ ^* = argmax∑logp(S|I; θ)

(11)

Among them, (S, I) represents the image-text pair in the training data; θ is the model parameter to be optimized; θ* is the optimized parameter;

The purpose of the training is to maximize the sum of the logarithmic probabilities of the sentences generated by all samples under the observation of the given input image I. The chain rule of conditional probability is used to calculate the probability p(S|I; θ), the expression is:

$\mathrm{<span\; class="notranslate">\; log</span>}\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; log</span>}\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; I</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span><span\; class="notranslate">\; ...</span><span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 12</span>}<span\; class="notranslate">\; )</span>$

Wherein, S ₀ , S ₁ ,...,S _t-1 , S _t represent words in the sentence. The unknown quantity p(S _t |I,S ₀ ,S ₁ ,...,S _t-1 ) in the formula is modeled using a recurrent neural network.

(2) Construct a recurrent neural network (RNN):

Under the condition of t-1 words, and represent these words as a fixed-length hidden state h _t until a new input x _t appears, and update the hidden state through the nonlinear function f, the expression is:

h _t+1 ＝f(h _t ,x _t )(13)

Among them, h _t+1 represents the next hidden state.

(3) For the nonlinear function f, model it by constructing a long-short-term memory network (LSTM) as shown in Figure 3;

Among them, it is the input gate input gate, f _t is the forget gate forgetgate, o _{t is} the output gate output gate, c is the cell cell, the update and output of each state can be expressed as:

i _t =σ(W _ix x _t +W _im m _t-1 )(14)

f _t = σ(W _fx x _t +W _fm m _t-1 )(15)

o _t = σ(W _ox x _t +W _om m _t-1 )(16)

p _t+1 ＝Softmax(m _t )(19)

in, Expressed as the product between gate values, the matrix W={W _ix ; W _im ; W _fx ; W _fm ; W _ox ; W _om ; W _cx ; W _ix ; W _cm } is the parameter to be trained, σ( ) is a Sigmoid function (for example: σ(W _ix x _t +W _im m _t-1 ), σ(W _fx x _t +W _fm m _t-1 ) is a Sigmoid function), h( ) is the hyperbolic tangent function (for example: h(W _cx x _t +W _cm m _t-1 ) is a hyperbolic tangent function). p _t+1 is the probability distribution of the next word after Softmax classification; m _t is the current state feature.

(4) Optimize the objective function (11) by gradient descent, and obtain the trained long-short-term memory network LSTM parameter W.

204: Using the trained RNN model to describe the video frame sequence of the video to be described, to obtain the description sequence, the steps for prediction are as follows:

(1) extraction test set ${\mathrm{VID}}^t=\{{{\mathrm{video}}^t}_1,...,{{\mathrm{video}}^t}_{N_t}\},$ N _t is the number of video in the test set, t is the video in the test set, and 10 frames of images are extracted for each video, which can be expressed as: ${\mathrm{image}}^t=\{{{\mathrm{image}}^t}_1,...,{{\mathrm{image}}^t}_{10}\}.$

(2) Use the trained model parameters Θ={W _i ,B _i ,θ _j }, i=1,...,5, ^j =1,2,3, and use the CNN model to extract each The CNN feature of the image, the image feature I ^t ={I ^t ₁ ,...,I ^t ₁₀ } is obtained.

(3) The image feature I ^t is used as an input and the model parameter W obtained from training is used to obtain the formula (12), and the sentence description S={S ₁ ,...,S _n } is obtained. Thus, the sentence description corresponding to the video is obtained.

205: Use the method of LexRank to sort the rationality of the description sequence, and select the most reasonable description as the final description of the video.

(1) Test the video feature sequence I ^t ={I ^t ₁ ,...,I ^t ₁₀ } by using the RNN model, and generate a corresponding sentence set S={S ₁ ,...S _i ,...,S _n }.

(2) Generate sentence features, sequentially scan all words in each sentence S _i in S in all sentence sets, where i=1,...,N _d , keep one for each different word, and form a vocabulary VOL represented by a word list ={w _i ,...,w _Nw }, where N _w is the total number of words in the vocabulary VOL. For each word w _i in the vocabulary VOL, sequentially scan each sentence S _j in the sentence set S, and count the number of occurrences n _ij of each word w _i in each sentence S _j , where j=1,..., N _s , N _s is the total number of sentences, and count the number of accompanying texts num(w _i ) containing word w _i in the set S; calculate the word frequency tf of each word w _i in each sentence S _j according to formula (20) (w _i , s _j ), where i=1,...,N _d , N _d is the total number of words in the vocabulary, j=1,...,N _s , N _s is the total number of all sentences S in the collection;

$\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; no</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; /</span>{<span\; class="notranslate">\; \&Sigma;</span>}_{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{{\mathrm{<span\; class="notranslate">\; N</span>}}_{\mathrm{<span\; class="notranslate">\; w</span>}}}{\mathrm{<span\; class="notranslate">\; no</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 20</span>}<span\; class="notranslate">\; )</span>$

Among them, n _kj is the number of occurrences of the kth word in the jth sentence.

For each word w _i in the vocabulary VOL, calculate its inverse document word frequency idf(w _i ) according to formula (21);

idf(w _i )=log(N _d /num(w _i ))(21)

Among them, N _d is the number of words in each sentence.

According to the vector space model, each sentence S _j in the set S is expressed as an N _w -dimensional vector, the i-th dimension corresponds to the word w _i in the vocabulary, and its value is tfidf(w _i ), the calculation formula is as follows:

tfidf(w _i )=tf(w _i ,s _j )×idf(w _i )(22)

(3) The cosine value between the two vectors S _i and S _j is used as the sentence similarity, and the calculation formula is as follows:

$\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; m</span>}\mathrm{<span\; class="notranslate">\; l</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{{<span\; class="notranslate">\; \&Sigma;</span>}_{\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}}{\mathrm{<span\; class="notranslate">\; tf</span>}}_{\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}{\mathrm{<span\; class="notranslate">\; tf</span>}}_{\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}}{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; idf</span>}}_{\mathrm{<span\; class="notranslate">\; w</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}{\sqrt{{<span\; class="notranslate">\; \&Sigma;</span>}_{{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; tf</span>}}_{{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}{\mathrm{<span\; class="notranslate">\; idf</span>}}_{{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; \&times;</span>\sqrt{{<span\; class="notranslate">\; \&Sigma;</span>}_{{\mathrm{<span\; class="notranslate">\; i</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; S</span>}\mathrm{<span\; class="notranslate">\; j</span>}}{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; tf</span>}}_{{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}}{\mathrm{<span\; class="notranslate">\; idf</span>}}_{{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; twenty\; three</span>}<span\; class="notranslate">\; )</span>$

in, is the word frequency of each word w in sentence S _i ; is the word frequency of each word w in the sentence S _j ; idf _w is the inverse document word frequency of each word; s _m is any word in the sentence S _i ; is the word frequency of word s _m in S _i ; is the inverse document word frequency of the word s _m ; s _n is any word in the sentence S _j ; is the word frequency of word s _n in S _j ; is the inverse document frequency of word s _n .

And form a fully connected undirected graph, as shown in Figure 4(a), each node u _i is a sentence S _i , and the edges between nodes are taken as sentence similarity.

(4) Set the threshold Degree, and delete all edges whose similarity is less than Degree, as shown in Figure 4(b).

(5) Calculate the LexRank score LR of each sentence node u _i , the initial score of each sentence node is: d/N, where N is the number of sentence nodes, d is the damping factor, and d is usually selected in [0.1,0.2] Between, the score LR is calculated according to the formula (4):

$\mathrm{<span\; class="notranslate">\; L</span>}\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; )</span>\underset{\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; d</span>}\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; \&lsqb;</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; \&rsqb;</span>}{<span\; class="notranslate">\; \&Sigma;</span>}\frac{\mathrm{<span\; class="notranslate">\; L</span>}\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; deg</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; +</span>\frac{\mathrm{<span\; class="notranslate">\; d</span>}}{\mathrm{<span\; class="notranslate">\; N</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; twenty\; four</span>}<span\; class="notranslate">\; )</span>$

Among them, deg(v) is the threshold of node v; LR(u) is the score of node u; LR(v) is the score of node v.

(6) Calculate the LR score of each sentence node, and sort them, and select the sentence with the highest score as the final description of the video.

To sum up, the embodiment of the present invention realizes the description of the event occurring in a video and the object attributes related to the event through natural language through steps 201 to 205, so as to achieve the purpose of describing and summarizing the video content.

Example 3

Here, two videos are selected as videos to be described, as shown in Figure 5, using the method based on deep learning and text summarization in the present invention to predict and output corresponding video descriptions:

(1) Use ImageNet as the training set, sample each picture in the data set to a picture of 256*256 size, $ImAG\mathrm{E.}=\{{\mathrm{image}}_1,...,{\mathrm{image}}_{N_m}\}$ As input, N _m is the number of pictures.

(2) Build the first convolutional layer, set the convolution kernel cov1 size to 11, stride to 4, select ReLU as max(0,x), perform pooling operation on the convolutional featuremap, and the kernel size is 3. The stride is 2, and the convolutional data is normalized using local corresponding normalization. In AlexNet, k=2, n=5, α=10 ⁻⁴ , β=0.75.

(3) Build the second convolutional layer, set the convolution kernel cov2 size to 5, stride to 1, select ReLU as max(0,x), perform pooling operation on the convolutional featuremap, and the kernel size is 3. The stride is 2, and the convolutional data is normalized using local corresponding normalization.

(4) Build the third convolutional layer, set the convolution kernel cov3 size to 3, stride to 1, and select ReLU as max(0,x).

(5) Build the fourth convolution layer, set the convolution kernel cov4 size to 3, stride to 1, and select ReLU to max(0,x).

(6) Build the fifth convolutional layer, set the convolution kernel cov5 size to 3, stride to 1, select ReLU as max(0,x), and perform pooling operation on the convolutional featuremap, the size of the kernel is 3, and the stride is 2.

(7) Build the sixth fully connected layer, set this layer as fc6, select ReLU as max(0,x), and perform dropout on the processed data.

(8) Build the seventh fully connected layer, set this layer as fc7, select ReLU as max(0,x), and perform dropout on the processed data.

(9) Build the eighth fully connected layer, set this layer as fc8, and add the Softmax classifier as the objective function.

(10) Establish a convolutional neural network (CNN) model by setting the above-mentioned eight-layer network layer.

(11) Training CNN model parameters.

(12) Data processing: 10 frames are evenly extracted from each video in the data set, and sampled to a size of 256*256. And input the image into the trained CNN model to obtain image features, and each frame of image randomly corresponds to the 5 sentences of the video as an image-text pair

(13) Construct a recurrent neural network (RNN) model.

Fig. 5 is the video text description result generated by the present invention. The picture part in the figure is the video frame extracted from the video, and the sentence corresponding to each frame of image is the result obtained after the video features pass through the language model. The lower part of the picture shows that after summarization, only the video features and sentences generated by image migration and the original description of the video are used.

In summary, the embodiment of the present invention converts the frame sequence of each video into a series of sentences through the convolutional neural network and the cyclic neural network, and selects high-quality and representative sentences from a large number of sentences through the method of text summarization. sex sentences. Users can use this method to obtain video descriptions, which have high accuracy and can be extended to video retrieval.

references

[1] KrizhevskyA, SutskeverI, HintonG. Image classification method based on deep convolutional neural network [J]. Neural Information Processing System Progress, 2012.

Those skilled in the art can understand that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the above-mentioned embodiments of the present invention are for description only, and do not represent the advantages and disadvantages of the embodiments.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection of the present invention. within range.

Claims (4)

1. A video description method based on deep learning and text summary, characterized in that, the video description method comprises the following steps: Download videos from the Internet, and describe each video to form a <video, description> pair to form a text description training set; Train the convolutional neural network model according to the image classification task through the existing image data set; Extract video frame sequence to video, and utilize convolutional neural network model to extract convolutional neural network feature, form <video frame sequence, text description sequence> pair as the input of recurrent neural network model, train and obtain recursive neural network model; Describe the video frame sequence of the video to be described by the recursive neural network model obtained through training, and obtain the description sequence; By using graph-based lexical centrality as a saliency method for text summarization, the sequence of descriptions is sorted, and the final description result of the video is output. 2. A kind of video description method based on deep learning and text summary according to claim 1, characterized in that, downloading videos from the Internet, and describing each video, forming <video, description> pair, constitute The text description training set is specifically: The <video, description> pair is composed of the existing video collection and the sentence description corresponding to each video to form a text description training set. 3. A kind of video description method based on deep learning and text summary according to claim 1, characterized in that, the video extracts a video frame sequence, and utilizes a convolutional neural network model to extract convolutional neural network features to form <Video frame sequence, text description sequence> For the input of the recurrent neural network model, the steps to train the recurrent neural network model are as follows: Use the parameters after training the convolutional neural network model to extract the convolutional neural network features of the image and the sentence description corresponding to the image for modeling to obtain the objective function; Construct a recurrent neural network; model nonlinear functions through long and short-term memory networks; Use gradient descent to optimize the objective function and obtain the trained long-short-term memory network parameters. 4. A kind of video description method based on deep learning and text summary according to claim 1, characterized in that, the recursive neural network model obtained through training describes the video frame sequence of the video to be described, and obtains the description sequence The specific steps are: Use the trained model parameters and use the convolutional neural network model to extract the convolutional neural network features of each image to obtain image features; The image feature is used as input and the model parameters obtained by training are used to obtain a sentence description, so as to obtain a sentence description corresponding to the video.