CN110288029B - Image description method based on Tri-LSTMs model - Google Patents

Abstract

The invention discloses an image description method based on the Tri‑LSTMs model, the steps of which are: generating a training set and mapping word vectors, building and training an RPN convolutional neural network and a Faster‑RCNN convolutional neural network, and extracting an image fully connected layer features, build and train Tri‑LSTMs models, and generate image descriptions. The invention combines a plurality of long-short-term memory network LSTMs, and simultaneously utilizes fully connected layer features of images and 300-dimensional GLOVE word vectors of words, effectively improves the diversity of generated subtitles, and generates more accurate image descriptions.

Description

Image description method based on Tri-LSTMs model

technical field

The invention belongs to the technical field of image processing, and further relates to an image description method based on a Tri-LSTMs model in the technical field of image description. The invention can be used to generate accurate and diverse sentences for a given image to describe the content of the image. Among them, Tri-LSTMs refers to the Tri-LSTMs model composed of three modules: semantic LSTM module, visual LSTM module and language LSTM module.

Background technique

Image description is given an image, generating sentences to describe the content of the image. The generated sentences must not only be fluent, but also be able to accurately describe the objects in the image and the attributes, positions, and relationships between objects. The generated image description can be used to find images that match the description content, which is convenient for image retrieval. In addition, converting the generated image descriptions into Braille can help blind people understand the image content.

Shenzhen University proposed an image description method based on the bag-of-words model in its patented technology "An image description method and system based on the bag-of-words model" (patent application number: 201410491596X, authorized announcement number: CN104299010B). This patented technology mainly solves the problems of information loss and low accuracy in traditional methods. The implementation steps of this patented technology are: (1) extract feature points from the image to be described; (2) calculate the distance set between the feature points and the visual words in the code book, and use the Gaussian membership function to use the distance Set to obtain the degree of membership set between the feature point and the visual word; (3) utilize the degree of membership set to count the degree of membership of the visual word used to describe each feature point to form a histogram vector, The histogram vector is used to describe the image to be described. Although this patented technology has improved the traditional image description technology and has higher description accuracy, the method still has the disadvantage that feature points need to be manually extracted, and the use of different extraction methods has a great impact on the results. Extraction The process is complicated, and the final image description diversity is insufficient.

Tianjin University has proposed a method based on the generation of structured text to image description in its patented technology "a generation method from structured text to image description" (patent application number: 2016108541692, authorized announcement number: CN106503055B). method. This patented technology mainly solves the problems of low accuracy and insufficient diversity of image descriptions generated by existing technologies. The implementation steps of this patented technology are: (1) Download pictures from the Internet to form a picture training set; (2) Perform lexical analysis on the descriptions corresponding to the images in the training set to construct structured text; (3) Utilize the existing neural network model, Extract the convolutional neural network features of the training set images, and use <image features, structured text> as input to construct a multi-task recognition model; (4) use the structured text extracted from the training set and the corresponding description as the input of the recurrent neural network , training to obtain the parameters of the recurrent neural network model; (5) input the convolutional neural network features of the image to be described, and obtain the predicted structured text through the multi-task recognition model; (6) input the predicted structured text, and obtain the predicted structured text through the recurrent neural network model image description. Although this patented technology has improved the problem of insufficient diversity of generated image descriptions, the method still has shortcomings in that it only uses image features and does not use other effective information to guide the decoding process, which affects the final generation The accuracy of the image description.

Oriol Vinyals et al. proposed an image description method based on the encoder-decoder model in their paper "Show and Tell: A Neural Image CaptionGenerator" (cvpr 2015 conference paper). This method is to first use the convolutional neural network (Convolutional Neural Network, CNN) to extract image features, and then send it to the long short-term memory network (Long Short-Term Memory, LSTM) to generate the description corresponding to the image. This method uses the encoder-decoder structure for the first time to solve the image description problem. However, the disadvantage of this method is that the model structure is too simple and the generated image description is not accurate.

In their paper "Show, Attend and Tell: Neural Image CaptionGeneration with Visual Attention" (cvpr 2015 conference paper), Kelvin Xu et al. proposed the combination of Long Short-Term Memory (LSTM) and attention mechanism. Image description method. This method assigns different weights to different positions of the image during the decoding process, thus giving different degrees of attention to objects at different positions. The method generates more accurate image descriptions and demonstrates the effectiveness of Long Short-Term Memory (LSTM) combined with attention mechanisms. However, the disadvantage of this method is that the single-layer long short-term memory network (Long Short-Term Memory, LSTM) undertakes multiple responsibilities such as sentence generation and image weight distribution at the same time, and the confusion of responsibilities leads to inaccurate image descriptions.

In their paper "Image Captioning with Semantic Attention" (cvpr 2016 conference paper), Quanzeng You et al. proposed an image description method that combines semantic attributes, image features and attention mechanisms at the same time. This method first selects the 1000 words with the highest frequency in the vocabulary as semantic attributes, and then introduces weighted semantic attributes into the input layer and output layer of the decoder. The method demonstrates the effectiveness of combining semantic attributes, image features and attention mechanisms simultaneously. However, the disadvantage of this method is that the difference between the descriptions corresponding to different images is too small, and the generated descriptions are rigid and templated.

Contents of the invention

The purpose of the present invention is to overcome the above-mentioned deficiencies in the prior art, and proposes an image description method based on the Tri-LSTMs model. The invention can effectively improve the accuracy and diversity of image description.

The technical ideas for realizing the present invention are: first, build and train the RPN convolutional neural network model and the faster-RCNN network model; then, build and train the Tri-LSTMs model; finally, use the pre-trained faster-RCNN network model to extract images Region, input the image region into the Tri-LSTMs model to generate an image description for the image.

The concrete steps that realize the object of the present invention are as follows:

(1) Generate a training set and map word vectors:

(1a) Select at least 80,000 samples from the image data set with image description to form the training set. Each selected sample is an image-description pair, and each image-description pair contains an image and five corresponding image description;

(1b) The image description of each sample in the training set is composed of several English words, and the frequency of occurrence of Chinese and English words in all image descriptions of all samples is counted and sorted by descending power. Select the first 1000 words, and each selected word Map to the corresponding 300-dimensional GLOVE word vector, and store it in the computer;

(2) Build the RPN convolutional neural network model and the faster-RCNN network model:

(2a) Build a RPN convolutional neural network model consisting of eight convolutional layers and a Softmax layer and set the parameters of each layer;

(2b) Build a faster-RCNN network model consisting of five convolutional layers, one ROIpooling layer, four fully connected layers and one Softmax layer and set the parameters of each layer;

(3) Training RPN convolutional neural network and fast-RCNN convolutional neural network:

Using the alternate training method, the RPN convolutional neural network and the fast-RCNN convolutional neural network are alternately trained to obtain the trained RPN convolutional neural network and the fast-RCNN convolutional neural network;

(4) Extract the fully connected layer features of each sample image in the training set:

(4a) Input each sample image in the training set into the trained RPN convolutional neural network in turn, and output the positions of all target rough selection frames and the types of targets in the frame in each sample image;

(4b) Input the image area in the rough selection frame of each target into the resnet101 network trained on the ImageNet database respectively, and store all the fully connected layer features of the network last layer fully connected layer output in the computer;

(5) Build the Tri-LSTMs model:

(5a) A long-term short-term memory network LSTM and an attention network are sequentially composed of a semantic LSTM module, and the long-term short-term memory network LSTM contains 1024 neurons;

(5b) A long-term short-term memory network LSTM and an attention network are sequentially composed of a visual LSTM module, and the long-term short-term memory network LSTM contains 1024 neurons;

(5c) A language LSTM module is composed of a long-term short-term memory network LSTM and a fully-connected layer in turn. The long-term short-term memory network LSTM contains 1024 neurons, and the number of neurons in the fully-connected layer is set to be included in all image descriptions in the training set. total number of words;

(5d) The semantic LSTM module, the visual LSTM module, and the language LSTM module are sequentially composed of a Tri-LSTMs model;

(6) Training Tri-LSTMs model:

(6a) At different moments, the words in different positions in the training sample image description are used as input, and the Tri-LSTMs model is trained from moment zero;

(6b) read all the fully connected layer features of the last layer fully connected layer output of the resnet101 network stored in the step (4b) computer, and use the average value of all fully connected layer features as a feature vector;

(6c) Add the feature vector and the word vector of the word mapping at the current moment in the image description, and input it into the long-term short-term memory network LSTM in the semantic LSTM module, and the long-term short-term memory network LSTM forwards the output hidden state;

(6d) read the 1000 300-dimensional GLOVE word vectors stored in the computer in step (1), input them into the attention network of the semantic LSTM module, and output the weighted GLOVE word vectors after the attention network conducts forward;

(6e) Add the hidden state of the semantic LSTM module at the current moment to the output of the attention network in the semantic LSTM module, and use the obtained sum vector as the output of the semantic LSTM module;

(6f) Input the sum vector output by the semantic LSTM module into the long-term short-term memory network LSTM in the visual LSTM module, and the long-term short-term memory network LSTM forwards the output hidden state;

(6g) Read all the fully connected layer features output by the last fully connected layer of the resnet101 network stored in the computer in step (4b), and input them into the attention network of the visual LSTM module. The attention network conducts forward, and the output is weighted The feature vector of the fully connected layer;

(6h) the hidden state of the visual LSTM module at the current moment and the output of the attention network in the visual LSTM module, and the obtained sum vector as the output of the visual LSTM module;

(6i) Input the sum vector of the output of the semantic LSTM module into the long-term short-term memory network LSTM in the language LSTM module, and the long-term short-term memory network LSTM forwards the output hidden state, inputs the hidden state into the fully connected layer, and outputs the next The probability vector of a word at a time;

(6j) Determine whether there is a word in the image description at the next moment, if so, perform step (6b) after calculating the cross entropy loss between the word probability vector and the word vector of the image description next moment, otherwise, perform step (6k);

(6k) Add the cross-entropy losses at all moments to obtain the total loss, use the BP algorithm to optimize all parameters in the model to minimize the total loss, stop training when the total loss converges, and obtain the trained Tri-LSTMs model;

(7) Generate image description:

(7a) Input a natural image into the pre-trained faster-RCNN, and output the target rough selection frame;

(7b) Input the image area in the rough selection frame of the target into the trained resnet101 network, and output the fully connected layer image features;

(7c) Input the fully connected layer image features into the Tri-LSTMs model to generate image descriptions.

Compared with the prior art, the present invention has the following advantages:

First, because the Tri-LSTMs model constructed by the present invention uses a combination of three long-term short-term memory networks LSTM, it overcomes the existing technology that only uses a single long-term short-term memory network LSTM to generate image descriptions, resulting in a model structure that is too simple to generate sufficiently accurate The shortcomings of the image description, so that the present invention can combine multiple long short-term memory networks LSTM, can effectively improve the accuracy of image description, and has the advantage of strong generalization ability.

Second, the present invention simultaneously utilizes the fully connected layer features of the image and the 300-dimensional GLOVE word vector of the word as the input of the Tri-LSTMs model, which overcomes the prior art that only uses the fully connected layer features of the image as the input of the model, and can utilize The effective information is too single, which leads to the problem of insufficient diversity of image descriptions generated by the image description method, so that the present invention has the advantage of generating more diverse image descriptions

Description of drawings

Fig. 1 is a flow chart of the present invention;

Fig. 2 is a structural diagram of the Tri-LSTMs model constructed in the present invention.

Fig. 3 is a simulation diagram of the present invention.

detailed description

The present invention will be further described below in conjunction with the accompanying drawings.

Referring to Fig. 1, the steps of the present invention are further described.

Step 1, generate a training set and map word vectors.

Select at least 80,000 samples from the image dataset with image description to form the training set. Each selected sample is an image-description pair, and each image-description pair contains an image and five corresponding image descriptions. Image description refers to the attributes, positions and relationships between objects in the image.

The image description of each sample in the training set is composed of multiple English words, and the frequency of occurrence of Chinese and English words in all image descriptions of all samples is counted and sorted by descending power. The first 1000 words are selected, and each selected word is mapped to the corresponding The 300-dimensional GLOVE word vector and store it in the computer.

Step 2, build the RPN convolutional neural network model and the faster-RCNN network model.

Build a RPN convolutional neural network model consisting of eight convolutional layers and a Softmax layer and set the parameters of each layer. The convolution kernel size of each layer is 3*3.

Build a faster-RCNN network model consisting of five convolutional layers, one ROIpooling layer, four fully connected layers and one Softmax layer and set the parameters of each layer. The size of the convolution kernel of each layer is 3*3.

Step 3, train RPN convolutional neural network and fast-RCNN convolutional neural network.

The alternate training method is used to alternately train the RPN convolutional neural network and the fast-RCNN convolutional neural network to obtain the trained RPN convolutional neural network and fast-RCNN convolutional neural network.

The steps of the alternate training method are as follows:

In the first step, a random value is selected for each parameter of the RPN convolutional neural network for random initialization.

The second step is to input the training sample image into the initialized RPN convolutional neural network, use the backpropagation BP algorithm to train the network, and adjust the parameters of the RPN convolutional neural network until all parameters converge to obtain the initial trained RPN Convolutional neural network.

In the third step, the training sample image is input into the trained RPN convolutional neural network, and the target rough selection box on the training sample image is output.

Step 4: Select a random value for each parameter of the fast-RCNN convolutional neural network and perform random initialization.

In step 5, input the training sample image and the target rough selection frame obtained in step 3 into the initialized fast-RCNN convolutional neural network, use the backpropagation BP algorithm to train the network, and adjust the fast-RCNN convolution Neural network parameters, until all parameters converge, the first trained fast-RCNN convolutional neural network is obtained.

Step 6, fix the parameters of the first five convolutional layers of the RPN convolutional neural network trained in step 2 of this step and the parameters of the fast-RCNN convolutional neural network trained in step 5 of this step, and set the training samples The image is input into the trained RPN convolutional neural network, and the backpropagation BP algorithm is used to fine-tune the unfixed parameters of the RPN convolutional neural network until it converges to obtain the final trained RPN convolutional neural network model.

In step 7, input the training sample image into the RPN convolutional neural network trained in step 6 of this step, and regain the target rough selection box on the sample image.

Step 8, fix the parameters of the first five convolutional layers of the fast-RCNN convolutional neural network trained in step 5 and the final trained RPN convolutional neural network parameters in step 6 of this step, and combine the training sample image with this step The target rough selection frame obtained in step 7 is input into the fast-RCNN convolutional neural network, and the unfixed parameters of the fast-RCNN convolutional neural network are fine-tuned using the backpropagation BP algorithm until it converges to obtain the final trained fast-RCNN Convolutional Neural Network.

Step 4, extract the features of the fully connected layer of the image.

The sample images in the training set are sequentially input into the trained RPN convolutional neural network, and the positions of all target rough selection boxes in each sample image and the types of targets in the boxes are output.

Input the image area in the rough selection frame of each target into the resnet101 network trained on the ImageNet database, and store all the fully connected layer features output by the last fully connected layer of the network into the computer.

Step 5, construct the Tri-LSTMs model.

A long short-term memory network LSTM and an attention network are sequentially composed of a semantic LSTM module. The long-term short-term memory network LSTM contains 1024 neurons.

A long-term short-term memory network LSTM and an attention network are sequentially composed of a visual LSTM module. The long-term short-term memory network LSTM contains 1024 neurons.

A long-term short-term memory network LSTM and a fully-connected layer are sequentially composed of a language LSTM module. The long-term short-term memory network LSTM contains 1024 neurons, and the number of neurons in the fully-connected layer is set to the total number of words contained in all image descriptions in the training set.

The Tri-LSTMs model is composed of the semantic LSTM module, visual LSTM module, and language LSTM module in turn, as shown in Figure 2.

Step 6, train the Tri-LSTMs model.

In the first step, at different moments, the words in different positions in the training sample image description are used as input, and the Tri-LSTMs model is trained from moment zero.

In step 2, read all the fully connected layer features output by the last fully connected layer of the resnet101 network stored in the computer in step 4, and use the average value of all fully connected layer features as the feature vector.

In the third step, the feature vector is added to the word vector of the word mapping at the current moment in the image description, and input to the long-term short-term memory network LSTM in the semantic LSTM module, and the long-term short-term memory network LSTM forwards and outputs the hidden state.

The forward conduction of the long short-term memory network LSTM is realized according to the following formula:

i _t ＝sigmoid(W _ix x _t +W _ih h _t-1 )

f _t ＝sigmoid(W _fx x _t +W _fh h _t-1 )

o _t ＝sigmoid(W _ox x _t +W _oh h _t-1 )

c _t ＝f _t ⊙c _t-1 +i _t ⊙tanh(W _cx x _t +W _ch h _t-1 )

h _t ＝o _t ⊙tanh(c _t )

e表示以自然常数e为底的指数操作，W_ix表示输入门的权重转移矩阵，x_t表示t时刻长短期记忆网络LSTM的输入，W_ih表示输入门所对应的隐藏态的权重转移矩阵，h_t-1表示t-1时刻长短期记忆网络LSTM的隐藏态，f_t表示t时刻长短期记忆网络LSTM的遗忘门，W_fx表示遗忘门的权重转移矩阵，W_fh表示遗忘门所对应的隐藏态的权重转移矩阵，o_t表示t时刻长短期记忆网络LSTM的输出门，W_ox表示输出门的权重转移矩阵，W_oh表示输出门所对应的隐藏态的权重转移矩阵，c_t表示t时刻长短期记忆网络LSTM的状态单元，⊙表示计算内积操作，c_t-1表示t-1时刻长短期记忆网络LSTM的状态单元，tanh表示激活函数

W_cx表示状态单元的权重转移矩阵，W_ch表示状态单元所对应的隐藏态的权重转移矩阵，h_t表示t时刻长短期记忆网络LSTM的隐藏态。Among them, i _t represents the input gate of the long-term short-term memory network LSTM at time t, and sigmoid represents the activation function

e represents the exponential operation with the natural constant e as the base, Wi _ix represents the weight transfer matrix of the input gate, x _t represents the input of the long-term short-term memory network LSTM at time t, and W _ih represents the weight transfer matrix of the hidden state corresponding to the input gate, h _t-1 represents the hidden state of the long-term short-term memory network LSTM at time t-1, f _t represents the forgetting gate of the long-term short-term memory network LSTM at time t, W _fx represents the weight transfer matrix of the forgetting gate, W _fh represents the corresponding value of the forgetting gate The weight transfer matrix of the hidden state, o _t represents the output gate of the long-term short-term memory network LSTM at time t, W _ox represents the weight transfer matrix of the output gate, W _oh represents the weight transfer matrix of the hidden state corresponding to the output gate, c _t represents t The state unit of the long-term short-term memory network LSTM at time, ⊙ represents the calculation inner product operation, c _t-1 represents the state unit of the long-term short-term memory network LSTM at time t-1, and tanh represents the activation function

W _cx represents the weight transfer matrix of the state unit, W _ch represents the weight transfer matrix of the hidden state corresponding to the state unit, and h _t represents the hidden state of the long-term short-term memory network LSTM at time t.

Step 4: Read 1,000 300-dimensional GLOVE word vectors stored in the computer in step 1, and input them into the attention network of the semantic LSTM module, and the attention network outputs weighted GLOVE word vectors after forward transmission.

The forward conduction of the attention network is realized according to the following formula:

a _i,t ＝tanh(W _s s _i +W _h h _t )

e表示以自然常数e为底的指数操作，W_s表示300维GLOVE词向量的权重转移矩阵，s_i表示输入的1000个300维GLOVE词向量中的第i个词向量，W_h表示语义LSTM模块中的长短期记忆网络LSTM输出的隐藏态的权重转移矩阵，h_t表示t时刻语义LSTM模块中的长短期记忆网络LSTM输出的隐藏态,

表示t时刻语义LSTM模块中注意力网络输出的特征向量，K表示300维GLOVE词向量的总数，∑表示求和操作，i表示词向量中每个向量的索引。Among them, a _{i, t} represents the weight value of the i-th vector among the 1000 300-dimensional GLOVE word vectors at time t, and tanh represents the activation function

e represents the exponential operation based on the natural constant e, W _s represents the weight transfer matrix of the 300-dimensional GLOVE word vector, s _i represents the i-th word vector among the 1000 300-dimensional GLOVE word vectors input, W _h represents the semantic LSTM The weight transfer matrix of the hidden state output by the long-term short-term memory network LSTM in the module, h _t represents the hidden state output by the long-term short-term memory network LSTM in the semantic LSTM module at time t,

Represents the feature vector output by the attention network in the semantic LSTM module at time t, K represents the total number of 300-dimensional GLOVE word vectors, ∑ represents the summation operation, and i represents the index of each vector in the word vector.

Step 5: Add the hidden state of the semantic LSTM module at the current moment to the output of the attention network in the semantic LSTM module, and the resulting sum vector is used as the output of the semantic LSTM module.

In the sixth step, the sum vector of the output of the semantic LSTM module is input into the long-term short-term memory network LSTM in the visual LSTM module, and the long-term short-term memory network LSTM forwards the output hidden state.

The forward conduction of the long short-term memory network LSTM is realized according to the following formula:

i _t ＝sigmoid(W _ix x _t +W _ih h _t-1 )

f _t ＝sigmoid(W _fx x _t +W _fh h _t-1 )

o _t ＝sigmoid(W _ox x _t +W _oh h _t-1 )

c _t ＝f _t ⊙c _t-1 +i _t ⊙tanh(W _cx x _t +W _ch h _t-1 )

h _t ＝o _t ⊙tanh(c _t )

e表示以自然常数e为底的指数操作，W_ix表示输入门的权重转移矩阵，x_t表示t时刻长短期记忆网络LSTM的输入，W_ih表示输入门所对应的隐藏态的权重转移矩阵，h_t-1表示t-1时刻长短期记忆网络LSTM的隐藏态，f_t表示t时刻长短期记忆网络LSTM的遗忘门，W_fx表示遗忘门的权重转移矩阵，W_fh表示遗忘门所对应的隐藏态的权重转移矩阵，o_t表示t时刻长短期记忆网络LSTM的输出门，W_ox表示输出门的权重转移矩阵，W_oh表示输出门所对应的隐藏态的权重转移矩阵，c_t表示t时刻长短期记忆网络LSTM的状态单元，⊙表示计算内积操作，c_t-1表示t-1时刻长短期记忆网络LSTM的状态单元，tanh表示激活函数

W_cx表示状态单元的权重转移矩阵，W_ch表示状态单元所对应的隐藏态的权重转移矩阵，h_t表示t时刻长短期记忆网络LSTM的隐藏态。Among them, i _t represents the input gate of the long-term short-term memory network LSTM at time t, and sigmoid represents the activation function

e represents the exponential operation with the natural constant e as the base, Wi _ix represents the weight transfer matrix of the input gate, x _t represents the input of the long-term short-term memory network LSTM at time t, and W _ih represents the weight transfer matrix of the hidden state corresponding to the input gate, h _t-1 represents the hidden state of the long-term short-term memory network LSTM at time t-1, f _t represents the forgetting gate of the long-term short-term memory network LSTM at time t, W _fx represents the weight transfer matrix of the forgetting gate, W _fh represents the corresponding value of the forgetting gate The weight transfer matrix of the hidden state, o _t represents the output gate of the long-term short-term memory network LSTM at time t, W _ox represents the weight transfer matrix of the output gate, W _oh represents the weight transfer matrix of the hidden state corresponding to the output gate, c _t represents t The state unit of the long-term short-term memory network LSTM at time, ⊙ represents the calculation inner product operation, c _t-1 represents the state unit of the long-term short-term memory network LSTM at time t-1, and tanh represents the activation function

W _cx represents the weight transfer matrix of the state unit, W _ch represents the weight transfer matrix of the hidden state corresponding to the state unit, and h _t represents the hidden state of the long-term short-term memory network LSTM at time t.

Step 7, read all the fully connected layer features output by the last fully connected layer of the resnet101 network stored in the computer in step 4, input them into the attention network of the visual LSTM module, the attention network conducts forward, and outputs the weighted Fully connected layer feature vector.

The forward conduction of the attention network is realized according to the following formula:

a _i,t ＝tanh(W _v v _i +W _h h _t )

e表示以自然常数e为底的指数操作，W_v表示全连接层特征的权重转移矩阵，v_i表示全部全连接层特征中第i个特征，W_h表示视觉LSTM模块中的长短期记忆网络LSTM的隐藏态的权重矩阵，h_t表示t时刻视觉LSTM模块中的长短期记忆网络LSTM输出的隐藏态，

表示t时刻视觉LSTM模块中注意力网络的输出，K表示全连接层特征向量的总数，∑表示求和操作，i表示特征向量中每个向量的索引。Among them, a _{i, t} represents the weight of the i-th feature in all fully connected layer features at time t, and tanh represents the activation function

e represents the exponential operation based on the natural constant e, W _v represents the weight transfer matrix of fully connected layer features, v _i represents the i-th feature in all fully connected layer features, W _h represents the long short-term memory network in the visual LSTM module The weight matrix of the hidden state of LSTM, h _t represents the hidden state output by the long short-term memory network LSTM in the visual LSTM module at time t,

Represents the output of the attention network in the visual LSTM module at time t, K represents the total number of feature vectors in the fully connected layer, ∑ represents the summation operation, and i represents the index of each vector in the feature vector.

Step 8, the hidden state of the visual LSTM module at the current moment and the output of the attention network in the visual LSTM module, and the obtained sum vector is used as the output of the visual LSTM module.

Step 9: Input the sum vector of the output of the semantic LSTM module into the long-term short-term memory network LSTM in the language LSTM module, and the long-term short-term memory network LSTM forwards the output hidden state, inputs the hidden state into the fully connected layer, and outputs A vector of probabilities for the words at the next moment.

The forward conduction of the long short-term memory network LSTM is realized according to the following formula:

i _t ＝sigmoid(W _ix x _t +W _ih h _t-1 )

f _t ＝sigmoid(W _fx x _t +W _fh h _t-1 )

o _t ＝sigmoid(W _ox x _t +W _oh h _t-1 )

c _t ＝f _t ⊙c _t-1 +i _t ⊙tanh(W _cx x _t +W _ch h _t-1 )

h _t ＝o _t ⊙tanh(c _t )

e表示以自然常数e为底的指数操作，W_ix表示输入门的权重转移矩阵，x_t表示t时刻长短期记忆网络LSTM的输入，W_ih表示输入门所对应的隐藏态的权重转移矩阵，h_t-1表示t-1时刻长短期记忆网络LSTM的隐藏态，f_t表示t时刻长短期记忆网络LSTM的遗忘门，W_fx表示遗忘门的权重转移矩阵，W_fh表示遗忘门所对应的隐藏态的权重转移矩阵，o_t表示t时刻长短期记忆网络LSTM的输出门，W_ox表示输出门的权重转移矩阵，W_oh表示输出门所对应的隐藏态的权重转移矩阵，c_t表示t时刻长短期记忆网络LSTM的状态单元，⊙表示计算内积操作，c_t-1表示t-1时刻长短期记忆网络LSTM的状态单元，tanh表示激活函数

W_cx表示状态单元的权重转移矩阵，W_ch表示状态单元所对应的隐藏态的权重转移矩阵，h_t表示t时刻长短期记忆网络LSTM的隐藏态。Among them, i _t represents the input gate of the long-term short-term memory network LSTM at time t, and sigmoid represents the activation function

e represents the exponential operation with the natural constant e as the base, Wi _ix represents the weight transfer matrix of the input gate, x _t represents the input of the long-term short-term memory network LSTM at time t, and W _ih represents the weight transfer matrix of the hidden state corresponding to the input gate, h _t-1 represents the hidden state of the long-term short-term memory network LSTM at time t-1, f _t represents the forgetting gate of the long-term short-term memory network LSTM at time t, W _fx represents the weight transfer matrix of the forgetting gate, W _fh represents the corresponding value of the forgetting gate The weight transfer matrix of the hidden state, o _t represents the output gate of the long-term short-term memory network LSTM at time t, W _ox represents the weight transfer matrix of the output gate, W _oh represents the weight transfer matrix of the hidden state corresponding to the output gate, c _t represents t The state unit of the long-term short-term memory network LSTM at time, ⊙ represents the calculation inner product operation, c _t-1 represents the state unit of the long-term short-term memory network LSTM at time t-1, and tanh represents the activation function

W _cx represents the weight transfer matrix of the state unit, W _ch represents the weight transfer matrix of the hidden state corresponding to the state unit, and h _t represents the hidden state of the long-term short-term memory network LSTM at time t.

Step 10, judge whether there are words in the image description at the next moment, if so, after calculating the cross-entropy loss between the word probability vector and the word vector of the image description at the next moment, then execute the second step of this step, otherwise, execute this step Step 11 of the steps.

The cross-entropy loss between the word probability vector and the image description word vector at the next moment is calculated according to the following formula:

Among them, loss represents the cross-entropy loss between the word probability vector and the word vector of the training set image description at the next moment, N represents the total number of words in the training set image description, ∑ represents the sum operation, and t represents the word in the training set image description The index of , log represents the logarithmic operation based on the natural constant e, P(s _t |I; θ) represents the average value of all fully connected layer features of the training set image input into the Tri-LSTMs model, and the output t Time word probability vector, I represents the average value of all fully connected layer features of the training set image, and θ represents all parameters of the Tri-LSTMs model.

Step 11: Add the cross-entropy losses at all times to obtain the total loss, use the BP algorithm to optimize all parameters in the model to minimize the total loss, stop training when the total loss converges, and obtain the trained Tri-LSTMs model.

Step 7, generate image description.

Input a natural image into the pre-trained faster-RCNN, and output the target rough selection box.

Input the image area in the target rough selection box into the trained resnet101 network, and output the fully connected layer image features.

Input the fully connected layer image features into the Tri-LSTMs model to generate image descriptions.

The effects of the present invention will be further described below in conjunction with simulation.

1. Simulation experiment conditions:

The hardware platform of the emulation experiment of the present invention is: Dell computer Intel (R) Core5 processor, main frequency 3.20GHz, internal memory 64GB;

The software platform of the simulation experiment of the present invention is: Python3.5, Tensorflow1.2 platform.

The data set used in the simulation experiment of the present invention is the COCO data set, which is a data set obtained by the Microsoft team that can be used for image description generation. The construction time of the data set is 2014. The training set of the data set and The test sets contain 123,287 and 40,775 images respectively.

2. Simulation content and result analysis:

The simulation experiment of the present invention is to adopt the present invention and two prior art (adaptive attention mechanism method, scst method), input 123287 training set samples of COCO data set respectively in the model that builds respectively to carry out training, test set The 40,775 images in the dataset were respectively input into the trained model to generate image descriptions for each test set image by the three methods.

In the simulation experiment, the two existing technologies adopted refer to:

The prior art adaptive attention mechanism method refers to the image description generation method proposed by Jiasen Lu et al. in the paper "Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning" (cvpr 2017 conference paper), referred to as adaptive Attention Mechanism Method.

The existing scst method refers to the image description generation method proposed by Jiasen Lu et al. in the paper "Self-critical Sequence Training for Image Captioning" (cvpr 2017 conference paper), referred to as the scst method.

In order to compare the pros and cons of the image descriptions generated by the three methods, four evaluation indicators (BLEU-4, METEOR, ROUGE-L, CIDER) are used to evaluate the image descriptions generated by the three methods on the COCO test set images. The index results are drawn into Table 1, where Net-1 represents the image description method based on the Tri-LSTMs model of the present invention, Net-2 represents the adaptive attention mechanism method, and Net-3 represents the scst method.

Table 1. Quantitative analysis table of the present invention and two prior art classification results in the simulation experiment

As can be seen from Table 1, compared with the method based on adaptive attention mechanism and the scst method, the network in the present invention has obtained higher scores on various evaluation indicators, so it performs better and can generate more accurate image description.

In order to describe the effect of the present invention more intuitively, two figures are randomly selected from the simulation results of the present invention on the COCO test set, as shown in Figure 3, where Figure 3(a) and Figure 3(b) are both COCO tests A collection of natural images and their corresponding image descriptions.

It can be seen from the simulation diagram in FIG. 3 that the image description generated by the present invention describes the contents of the image more accurately and specifically.

Claims (7)

1. An image description method based on Tri-LSTMs model is characterized in that a Tri-LSTMs model composed of a semantic LSTM module, a visual LSTM module and a language LSTM module is built, and sentence description image content is generated for any one natural image, and the method comprises the following steps:

(1) Generating a training set and mapping word vectors:

(1a) Selecting at least 80000 samples from an image data set with image descriptions to form a training set, wherein each selected sample is an image-description pair, and each image-description pair comprises one image and five corresponding image descriptions;

(1b) The image description of each sample in the training set consists of a plurality of English words, the frequency of occurrence of the English words in all the image descriptions of all the samples is counted, the English words are subjected to power reduction sorting, the first 1000 words are selected, each selected word is mapped into a corresponding 300-dimensional GLOVE word vector, and the vector is stored in a computer;

(2) Constructing an RPN convolutional neural network model and a master-RCNN network model:

(2a) Constructing an RPN convolutional neural network model consisting of eight convolutional layers and a Softmax layer, and setting parameters of each layer;

(2b) Building a master-RCNN network model consisting of five convolution layers, one ROIploling layer, four full-connection layers and one Softmax layer, and setting parameters of each layer;

(3) Training an RPN convolutional neural network and a fast-RCNN convolutional neural network:

performing alternate training on the RPN convolutional neural network and the fast-RCNN convolutional neural network by adopting an alternate training method to obtain a trained RPN convolutional neural network and a fast-RCNN convolutional neural network;

(4) Extracting the full connection layer characteristics of each sample image in the training set:

(4a) Sequentially inputting each sample image in the training set into a trained RPN convolutional neural network, and outputting the positions of all target rough selection frames in each sample image and the types of targets in the frames;

(4b) Respectively inputting the image area in each target rough selection frame into a resnet101 network trained on an ImageNet database, and storing all full-connection layer characteristics output by the last full-connection layer of the network into a computer;

(5) Constructing a Tri-LSTMs model:

(5a) Sequentially forming a long short-term memory network LSTM and an attention network into a semantic LSTM module, wherein the long short-term memory network LSTM comprises 1024 neurons;

(5b) Sequentially forming a long short-term memory network LSTM and an attention network into a visual LSTM module, wherein the long short-term memory network LSTM comprises 1024 neurons;

(5c) Sequentially forming a language LSTM module by a long-short term memory network LSTM and a full connection layer, wherein the long-short term memory network LSTM comprises 1024 neurons, and the number of the neurons of the full connection layer is set as the total number of words contained in all image descriptions in a training set;

(5d) Sequentially combining a semantic LSTM module, a visual LSTM module and a language LSTM module into a Tri-LSTMs model;

(6) Training Tri-LSTMs model:

(6a) At different moments, taking words at different positions in the image description of the training sample as input, and training a Tri-LSTMs model from zero moment;

(6b) Reading all full-connection layer characteristics output by the last full-connection layer of the resnet101 network stored in the computer in the step (4 b), and taking the average value of all full-connection layer characteristics as a characteristic vector;

(6c) Adding the feature vector and the word vector mapped by the word at the current moment in the image description, inputting the added feature vector into a long-short term memory network LSTM in a semantic LSTM module, and outputting a hidden state in a forward conduction manner by the long-short term memory network LSTM;

(6d) Reading 1000 300-dimensional GLOVE word vectors stored in the computer in the step (1), inputting the GLOVE word vectors into an attention network of a semantic LSTM module, and outputting the weighted GLOVE word vectors after the attention network conducts forwards;

(6e) Adding the hidden state of the semantic LSTM module at the current moment with the output of the attention network in the semantic LSTM module, and taking the obtained sum vector as the output of the semantic LSTM module;

(6f) Inputting the sum vector output by the semantic LSTM module into a long-short term memory network LSTM in the visual LSTM module, and conducting and outputting a hidden state in the forward direction of the long-short term memory network LSTM;

(6g) Reading all full-connection layer characteristics output by the last full-connection layer of the resnet101 network stored in the computer in the step (4 b), inputting the characteristics into an attention network of the visual LSTM module, conducting the attention network forwards, and outputting weighted full-connection layer characteristic vectors;

(6h) Taking the hidden state of the visual LSTM module at the current moment and the output of the attention network in the visual LSTM module, and taking the obtained sum vector as the output of the visual LSTM module;

(6i) Inputting the sum vector output by the semantic LSTM module into a long-short term memory network LSTM in the language LSTM module, outputting a hidden state by forward conduction of the long-short term memory network LSTM, inputting the hidden state into a full-connection layer, and outputting a probability vector of a word at the next moment;

(6j) Judging whether a word exists in the image description at the next moment, if so, calculating the cross entropy loss between the word probability vector and the word vector at the next moment of the image description, and then executing the step (6 b), otherwise, executing the step (6 k);

(6k) Adding the cross entropy losses at all times to obtain a total loss, optimizing all parameters in the model by using a BP algorithm to minimize the total loss, and stopping training when the total loss is converged to obtain a trained Tri-LSTMs model;

(7) Generating an image description:

(7a) Inputting a natural image into a pre-trained false-RCNN, and outputting a target rough selection frame;

(7b) Inputting the image area in the target rough selection frame into a trained resnet101 network, and outputting the image characteristics of the full connection layer;

(7c) And inputting the image characteristics of the full connection layer into a Tri-LSTMs model to generate image description.

2. The Tri-LSTMs model-based image description method of claim 1, wherein the image description in step (1 a) refers to the attributes, positions and relationships of objects in the image.

3. The image description method based on the Tri-LSTMs model of claim 1, wherein the step of the alternative training method in step (3) is as follows:

firstly, selecting a random value for each parameter of the RPN convolutional neural network, and carrying out random initialization;

secondly, inputting a training sample image into the initialized RPN convolutional neural network, training the network by using a back propagation BP algorithm, and adjusting parameters of the RPN convolutional neural network until all the parameters are converged to obtain the initially trained RPN convolutional neural network;

inputting the training sample image into the trained RPN convolutional neural network, and outputting a target rough selection frame on the training sample image;

fourthly, selecting a random value for each parameter of the fast-RCNN convolutional neural network, and carrying out random initialization;

fifthly, inputting the training sample image and the target rough selection box obtained in the third step into the initialized fast-RCNN convolutional neural network, training the network by using a back propagation BP algorithm, and adjusting the parameters of the fast-RCNN convolutional neural network until all the parameters are converged to obtain the initially trained fast-RCNN convolutional neural network;

fixing the parameters of the front five layers of convolutional layers of the RPN convolutional neural network trained in the second step and the parameters of the fast-RCNN convolutional neural network trained in the fifth step, inputting a training sample image into the trained RPN convolutional neural network, and finely adjusting the unfixed parameters of the RPN convolutional neural network by using a back propagation BP algorithm until the unfixed parameters are converged to obtain a finally trained RPN convolutional neural network model;

step seven, inputting the training sample image into the RPN convolutional neural network finally trained in the step six, and obtaining a target rough selection frame on the sample image again;

and eighthly, fixing parameters of the front five layers of convolutional layers of the fast-RCNN convolutional neural network trained in the fifth step and parameters of the RPN convolutional neural network trained finally in the sixth step, inputting the training sample image and the target rough selection box obtained again in the seventh step into the fast-RCNN convolutional neural network, finely adjusting unfixed parameters of the fast-RCNN convolutional neural network by using a back propagation BP algorithm until the unfixed parameters are converged, and obtaining the finally trained fast-RCNN convolutional neural network.

4. The image description method based on the Tri-LSTMs model of claim 1, wherein the long short term memory network LSTM forward conduction in step (6 c), step (6 f) and step (6 i) is implemented according to the following formula:

i _t ＝sigmoid(W _ix x _t +W _ih h _t-1 )

f _t ＝sigmoid(W _fx x _t +W _fh h _t-1 )

o _t ＝sigmoid(W _ox x _t +W _oh h _t-1 )

c _t ＝f _t ⊙c _t-1 +i _t ⊙tanh(W _cx x _t +W _ch h _t-1 )

h _t ＝o _t ⊙tanh(c _t )

wherein i _t An input gate of a long-term and short-term memory network LSTM at the time t is represented, sigmoid represents an activation function

e denotes an exponential operation with a natural constant e as base, W _ix Weight transfer matrix, x, representing input gates _t Representing the input, W, of the long-short term memory network LSTM at time t _ih Weight transfer matrix, h, representing hidden states corresponding to input gates _t-1 Representing the hidden state of the long-short term memory network LSTM at time t-1, f _t Forgetting gate, W, representing long-short term memory network LSTM at time t _fx Weight transfer matrix, W, representing forgetting gate _fh Weight transfer matrix, o, representing the hidden state corresponding to the forgetting gate _t Output gate, W, representing the long-short term memory network LSTM at time t _ox Weight transfer matrix, W, representing output gates _oh A weight transfer matrix representing the hidden state corresponding to the output gate, c _t Status cells indicating a long short term memory network LSTM at time t indicate a compute inner product operation, c _t-1 Representing the state unit of the long-short term memory network LSTM at the time t-1, and tanh representing the activation function

W _cx Weight transfer matrix, W, representing a state cell _ch Weight transfer matrix, h, representing hidden states corresponding to the state units _t Representing the hidden state of the long-short term memory network LSTM at time t.

5. The Tri-LSTMs model-based image description method of claim 1, wherein the attention network forward conduction in step (6 d) is implemented according to the following formula:

a _i,t ＝tanh(W _s s _i +W _h h _t )

wherein, a _i,t Representing the weight value of the ith vector in 1000 300-dimensional GLOVE word vectors at the time t, and tanh representing an activation function

e denotes an exponential operation with a natural constant e as base, W _s Weight transfer matrix, s, representing 300-dimensional GLOVE word vectors _i Represents the ith word vector, W, of the 1000 300-dimensional GLOVE word vectors input _h Weight transfer matrix, h, representing hidden states of long short term memory network LSTM outputs in semantic LSTM modules _t Represents the hidden state of the long-short term memory network LSTM output in the semantic LSTM module at the time t,

representing the feature vector output by the attention network in the semantic LSTM module at the time t, K representing the total number of 300-dimensional GLOVE word vectors, sigma representing the summation operation, and i representing the index of each vector in the word vectors.

6. The Tri-LSTMs model-based image description method of claim 1, wherein the attention network forward conduction in step (6 g) is implemented according to the following formula:

a _i,t ＝tanh(W _v v _i +W _h h _t )

wherein, a _i,t Represents the weight of the ith feature in all the fully-connected layer features at the moment t, and tanh represents the activation function

e denotes an exponential operation with a natural constant e as base, W _v Weight transfer matrix, v, representing the characteristics of the fully-connected layer _i Denotes the ith feature, W, of all fully-connected layer features _h Weight matrix, h, representing hidden states of long short term memory network LSTM in visual LSTM module _t Represents the hidden state of the long-short term memory network LSTM output in the vision LSTM module at the time t,

represents the output of the attention network in the visual LSTM module at time t, K represents the total number of fully connected layer feature vectors, Σ represents the summation operation, and i represents the index of each vector in the feature vectors.

7. The Tri-LSTMs model-based image description method of claim 1, wherein the cross entropy loss between the word probability vector in step (6 j) and the word vector at the next time of image description is calculated according to the following formula:

wherein loss represents the cross entropy loss between the word probability vector and the word vector at the next moment of the image description of the training set, N represents the total number of words in the image description of the training set, Σ represents the summation operation, t represents the index of the word in the image description of the training set, log represents the logarithm operation with the natural constant e as the base, and P(s) _t I; theta) represents the word probability vector at t moment output by inputting the average value of all the fully connected layer characteristics of the training set image into the Tri-LSTMs model, and the table IAnd (3) showing the average value of all the fully connected layer characteristics of the images of the training set, wherein theta represents all the parameters of the Tri-LSTMs model.

Images