CN110717431A - A fine-grained visual question answering method combined with multi-view attention mechanism - Google Patents

Abstract

The invention relates to a fine-grained visual question answering method combined with a multi-perspective attention mechanism, fully considers the guiding role of the specific semantics of the question, and proposes a multi-perspective attention model, which can effectively select the current task target (problem). Multiple salient target regions, learn from multiple perspectives to obtain the region information related to the answer in the image and the question text, extract the regional salient features in the image under the guidance of the question semantics, have more fine-grained feature expression, and analyze the image. There are multiple important semantic expression regions in the image, which has strong characterization ability, which increases the effectiveness and comprehensiveness of the multi-view attention model. Accuracy and Comprehensiveness of Semantic Understanding for Visual Question Answering. Using the method of the present invention to perform a visual question and answer task has the advantages of simple steps, high efficiency and high accuracy, and can be fully used in business and has a good market prospect.

Description

A fine-grained visual question answering method combined with multi-view attention mechanism

technical field

The invention relates to the technical field of computer vision and natural language processing, and more particularly, to a fine-grained visual question answering method combined with a multi-view attention mechanism.

Background technique

With the rapid development of computer vision and natural language processing, visual question answering system has become one of the more and more popular research fields of artificial intelligence. Visual question answering technology is an emerging topic, and its task is to combine the two disciplines of computer vision and natural language processing, taking a given image and a natural language question related to the image as input, and generating a natural language answer as output. Visual question answering is a key application direction in the field of artificial intelligence. By simulating real-world scenarios, visual question answering can help users with visual impairments to perform real-time human-computer interaction.

In essence, the visual question answering system is regarded as a classification task. The common practice is to extract picture and question features based on known pictures and questions, and then classify and obtain question and answer results by fusing picture features and question features. In recent years, visual question answering has attracted extensive attention in the fields of computer vision and natural language processing. Due to the relative complexity of visual question answering and the need for image and text processing, some existing methods still have a certain lack of accuracy and face greater challenges.

In practical applications, visual question answering systems often face the high dimensionality of images and the effects of noise, which can affect the algorithm's prediction of answers. Therefore, an effective visual question answering model can mine the structural features and semantically related parts of the image that are semantically consistent with the question for fine-grained prediction.

The visual attention model is to use the computer to simulate the human visual attention mechanism to obtain the part of an image that is most likely to attract people's attention, that is, the salient area of the image. In visual question answering, most of the methods using a single attention mechanism model often ignore the difference of the structural semantics of the image, and are somewhat insufficient for the situation where there are multiple important regions in the image, so the attention mechanism brought by such methods cannot be used. Avoided will affect visual question answering accuracy.

The study found that most of the existing visual question answering methods predict the semantic answer of the question through the question and the whole picture, but do not take into account the guiding role of the specific semantics of the question. Therefore, the image region features and question features learned by these models are in the The correlation in the semantic space is weak.

To sum up, in the prior art, effective visual question answering methods still need to be improved.

SUMMARY OF THE INVENTION

The purpose of the present invention is to overcome the deficiencies of the prior art and provide a fine-grained visual question answering method combined with a multi-view attention mechanism, which can effectively improve the accuracy and comprehensiveness of visual semantic information extraction, and reduce redundant data and noise data. Therefore, the fine-grained recognition ability of the visual question answering system and the judgment of complex problems are improved, and the accuracy of the visual question answering system and the interpretability of the model are improved to a certain extent.

The technical scheme of the present invention is as follows:

A fine-grained visual question answering method combined with multi-view attention mechanism, the steps are as follows:

1) Input image, extract image features; input question text, extract question features;

2) Input the image features and question features into the multi-view attention model, calculate the attention weight of the image, and perform a weighted operation on the image features of step 1) through the attention weight to obtain fine-grained image features;

3) Fusion of image fine-grained features and problem features to obtain fusion features;

4) Input the fused features into the classifier, and predict the answer.

Preferably, the multi-view attention model includes an upper-layer attention model and a lower-layer attention model, the single-attention weight is obtained through the upper-layer attention model, the salient attention weight is obtained through the lower-layer attention model, and the salient attention weight is obtained through the lower-layer attention model. It reflects that different target regions in the image correspond to different attention resources.

As a preference, the method for obtaining a single attention weight is as follows:

最后使用softmax函数归一化权值，得到单一注意力权重 Input image features and question features to the upper attention model, use a fully connected layer to project the data of image features and question features to the same dimensional space, and use the activation function ReLu to normalize the vector; then use Hada code product fusion, and then input in turn The two-layer fully connected layer processes the learned parameters and processes the learned parameters

Finally, use the softmax function to normalize the weights to get a single attention weight

为图像特征，

为问题特征，

为上层注意力模型待学习的权重参数，K为图像特征的空间区域个数，T为选取的问题特征长度，d为网络层中隐藏神经元的个数，h为该层设置的输出维度，ReLu是神经网络中的激活函数，其具体形式可以表达为f(x)＝max(0,x)。in,

is the image feature,

is the problem feature,

is the weight parameter to be learned by the upper attention model, K is the number of spatial regions of image features, T is the length of the selected problem feature, d is the number of hidden neurons in the network layer, h is the output dimension set by this layer, ReLu is an activation function in a neural network, and its specific form can be expressed as f(x)=max(0,x).

Preferably, the method for obtaining salient attention weights is as follows:

为下层注意力模型待学习的权重参数，

为获取的关联矩阵；Input image features and question features to the lower attention model, use a fully connected layer to project the data of image features and question features to the same dimensional space, and then calculate the correlation matrix C _i =ReLu(q _i ^T W _b V _i ) ;in,

is the weight parameter to be learned by the lower attention model,

is the obtained correlation matrix;

最后使用softmax函数归一化权值，输出显著性注意力权重

Multiply the correlation matrix as a feature with the problem feature, and fuse it with the input image feature. The fused parameter is

Finally, use the softmax function to normalize the weights and output the saliency attention weights

为下层注意力模型待学习的权重参数。in,

Weight parameters to be learned for the lower attention model.

Preferably, the attention weight of the image is calculated based on the single attention weight and the salient attention weight, as follows:

Among them, β ₁ and β ₂ are the weight ratio hyperparameters of the upper-layer attention model and the lower-layer attention model.

Preferably, in step 3), the fine-grained features of the image and the problem features are passed through the nonlinear layers f _v and f _q respectively, and the activation function ReLu is used in the nonlinear layers f _v and f _q to normalize the vectors; then the Hada code product fusion is used , get the fusion feature

最后，选取得分更高的输出；Preferably, in step 4), the fusion feature passes through the nonlinear layer f _o , and the activation function ReLu is used to normalize the vector in the pass through the nonlinear layer f _o ; then the linear mapping w _o is used to predict the candidate score of the answer

Finally, select the output with a higher score;

where σ is the sigmoid activation function and _wo is the weight parameter to be learned.

As an option, the sigmoid activation function normalizes the final score to the (0-1) interval, and the last stage acts as a logistic regression that predicts the correctness of each candidate answer with an objective function of

where z and k cover N candidate answers for M training questions, respectively, and s _zk is the real answer to the question.

Preferably, in step 1), the Faster-RCNN standard model is used to perform feature extraction on the input image I _i to obtain a deeply expressed image feature V _i =FasterRCNN(I _i ).

其中，x_t ⁽ⁱ⁾表示每个单词在词汇表中的位第t个单词；Preferably, in step 1), input the question text Q _i , first use spaces and punctuation to divide the question text Q _i into words, and then initialize by the pre-trained GloVe word embedding method to obtain the ith specified question sentence after encoding

Among them, x _t ⁽ⁱ⁾ represents the t-th word of each word in the vocabulary;

输入到LSTM网络中，取出最后一层的输出q_i作为

的特征表达，得到问题特征q_i。followed by

Input into the _LSTM network, take out the output qi of the last layer as

The feature expression of , the problem feature _qi is obtained.

The beneficial effects of the present invention are as follows:

The fine-grained visual question answering method combined with the multi-view attention mechanism described in the present invention proposes a multi-view attention model, which can effectively select multiple salient target areas related to the current task target (question), and extract the question semantics. The regional saliency feature in the guided image has more fine-grained feature expression, and has a strong ability to describe the situation that there are multiple important semantic expression regions in the image.

The invention fully considers the guiding role of the specific semantics of the question, learns and obtains the area information related to the answer in the image and the question text from multiple perspectives, increases the effectiveness and comprehensiveness of the multi-perspective attention model, and effectively strengthens the image area. Semantic correlation of salient features and question features to improve the accuracy and comprehensiveness of semantic understanding for visual question answering.

Using the method of the present invention to perform a visual question and answer task has the advantages of simple steps, high efficiency and high accuracy, and can be fully used in business and has a good market prospect.

Description of drawings

Fig. 1 is the schematic flow sheet of the present invention;

Figure 2 is a schematic diagram of a multi-view attention model;

Figure 3 is a heatmap for visualizing attention weights (a simple attention task);

Figure 4 is a heatmap for visualizing attention weights (tasks need to be highly focused on multiple locations in the image);

Fig. 5 is the comparative graph of the result obtained by the multi-view attention model of the present invention and the more advanced method at present;

Figure 6 is a graph of the loss function for final model performance training;

Figure 7 is a graph of training validation scores for final model performance training.

Detailed ways

The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

In order to solve the shortcomings of the prior art, the present invention provides a fine-grained visual question answering method combined with a multi-view attention mechanism. Visual question answering can be regarded as a multi-task classification problem, and each answer can be regarded as a classification category. In general visual question answering systems, the One-Hot method is used to encode the answer to obtain the One-Hot vector corresponding to each answer, forming an answer vector table. One-Hot encoding is the representation of categorical variables as binary vectors. This first requires mapping categorical values to integer values, then each integer value is represented as a binary vector, which is zero except for the integer's index, which is marked as 1.

As shown in Figure 1, the fine-grained visual question answering method combined with the multi-view attention mechanism described in the present invention has the following steps:

1) Input image, extract image features; input question text, extract question features;

2) Input the image features and question features into the multi-view attention model, calculate the attention weight of the image, and perform a weighted operation on the image features of step 1) through the attention weight to obtain fine-grained image features;

3) Fusion of image fine-grained features and problem features to obtain fusion features;

4) Input the fused features into the classifier, and predict the answer.

可以进一步表示成其中，

是Faster-RCNN提取的第k个区域特征，d为网络层中隐藏神经元的个数并同时表示输出维度。In this embodiment, in step 1), the Faster-RCNN standard model is used to perform feature extraction on the input image I _i to obtain image features V _i =FasterRCNN(I _i ). Let K be the number of spatial regions of the image feature, then the image feature

can be further expressed as in,

is the k-th regional feature extracted by Faster-RCNN, and d is the number of hidden neurons in the network layer and represents the output dimension at the same time.

的编码形式，其中，x_t ⁽ⁱ⁾表示每个单词在词汇表中的位第t个单词；In step 1), after entering the question text Q _i , first use spaces and punctuation to divide the question text Q _i into words, and then initialize it through the pre-trained GloVe word embedding method (Global Vectors for Word Representation) to obtain the i-th specified question. sentence

The encoded form of , where x _t ⁽ⁱ⁾ represents the t-th word of each word in the vocabulary;

输入到LSTM网络中，具体地，使用含有1280个隐藏单元的标准LSTM网络，取出最后一层的输出q_i作为

的特征表达，得到问题特征q_i。followed by

Input into the LSTM network, specifically, use a standard LSTM network with 1280 hidden units, and take out the output _qi of the last layer as

The feature expression of , the problem feature _qi is obtained.

Then, for the acquired image feature V _i and the encoded question feature qi _, the two kinds of features are input into the multi-view attention model, and the attention weight of the image is calculated.

The visual attention mechanism can essentially select the target area that is more critical to the current task goal from the image, so as to invest more attention resources in this area to obtain more detailed information of the target that needs attention, and suppress other useless information. information. In visual question answering tasks, semantic representations are diverse. In particular, there are some problems that require the model to understand the semantic representation between multiple target objects in an image. Therefore, the single visual attention model cannot effectively mine the correlation between different semantic objects and question semantics in the image;

In order to solve this problem, the present invention provides a multi-view attention model, which uses two different attention mechanisms to jointly learn important regions with different semantics that can be paid attention to in the problem, so as to obtain fine-grained attention features of images picture. Using the multi-view attention model to pay attention to the image attention weight obtained from the image, and using the weight to weight the image features to obtain the cumulative vector as the final image feature representation, that is, the fine-grained image feature, which can be better associated with the semantics of the question.

As shown in Figure 2, the multi-view attention model includes an upper-layer attention model and a lower-layer attention model. The single attention weight is obtained through the upper-layer attention model, and the salient attention weight is obtained through the lower-layer attention model. The attention weight reflects that different target regions in the image correspond to different attention resources.

Specifically, in the upper-layer attention model, the method for obtaining a single attention weight is as follows:

为图像特征，

为问题特征，

为上层注意力模型待学习的权重参数，K为图像特征的空间区域个数，T为选取的问题特征长度，d为网络层中隐藏神经元的个数，h为该层设置的输出维度，ReLu是神经网络中的激活函数，其具体形式可以表达为f(x)＝max(0,x)；Input image features and question features to the upper attention model, use a fully connected layer to project the data of image features and question features to the same dimensional space, and use the activation function ReLu to normalize the vector; then use the Hardmard product (Hardmard product) fusion , and then input the two fully connected layers in turn to process the learning parameters, and process the learned parameters in,

is the image feature,

is the problem feature,

is the weight parameter to be learned by the upper attention model, K is the number of spatial regions of image features, T is the length of the selected problem feature, d is the number of hidden neurons in the network layer, h is the output dimension set by this layer, ReLu is the activation function in the neural network, and its specific form can be expressed as f(x)=max(0,x);

Finally, use the softmax function to normalize the weights to get a single attention weight

为softmax权值，如果部分权值数值较大，其余部分势必权值较小。由于一幅图像常常含有多个不同语义，并且这些语义常常在不同区域进行视觉语义表达。单一注意力权重

常常会忽略一些具有重要语义的区域信息。为补充上层注意力模型的注意信息的缺失部分，本发明进一步提出了下层注意力模型。下层注意力模型同时兼顾图像与问题语义的关联性，达到问题引导多视角注意力模型的学习机制，增加特征细粒度挖掘能力，本发明通过计算图像与问题特征在语义的相似性来引导图像区域的注意力的学习。Consider a single attention weight

For softmax weights, if some of the weights have larger values, the rest will inevitably have smaller weights. Since an image often contains multiple different semantics, and these semantics are often expressed in different regions of the visual semantics. single attention weight

Some region information with important semantics is often ignored. To supplement the missing part of the attention information of the upper-layer attention model, the present invention further proposes a lower-layer attention model. The lower-level attention model also takes into account the relevance of the image and the question semantics, achieves the learning mechanism of the question-guided multi-view attention model, and increases the feature fine-grained mining ability. The present invention guides the image area by calculating the semantic similarity between the image and the question feature. attention learning.

Specifically, in the lower-layer attention model, the method of obtaining the salient attention weight is as follows:

为下层注意力模型待学习的权重参数，

为获取的关联矩阵；Input image features and question features to the lower attention model, use a fully connected layer to project the data of image features and question features to the same dimensional space, and then calculate the correlation matrix C _i =ReLu(q _i ^T W _b V _i ) ;in,

is the weight parameter to be learned by the lower attention model,

is the obtained correlation matrix;

最后使用softmax函数归一化权值，输出显著性注意力权重

其中，

为下层注意力模型待学习的权重参数，参数维度设置通上层注意力模型一致，K为图像特征的空间区域个数，T为选取的问题特征长度，d为网络层中隐藏神经元的个数，h为该层的输出维度，ReLu是神经网络中的激活函数。Multiply the correlation matrix as a feature with the problem feature, and fuse it with the input image feature. The fused parameter is

Finally, use the softmax function to normalize the weights and output the saliency attention weights

in,

is the weight parameter to be learned by the lower-layer attention model. The parameter dimension is set to be consistent with the upper-layer attention model. K is the number of spatial regions of image features, T is the length of the selected problem feature, and d is the number of hidden neurons in the network layer. , h is the output dimension of this layer, and ReLu is the activation function in the neural network.

The attention weight of the image is calculated based on the single attention weight and the salient attention weight, as follows:

Among them, β ₁ and β ₂ are the weight ratio parameters of the upper-layer attention model and the lower-layer attention model. In practical applications, the weights between the upper-layer attention model and the lower-layer attention model can be allocated by debugging parameters to achieve better results.

The image feature V _i can further represent the set form of K image space region features Further, the attention weight a _i is multiplied and weighted by the image features of each spatial region to obtain fine-grained features of the image.

In step 3), the fine-grained features of the image and the problem features are passed through the nonlinear layers f _v and f _q respectively, and the activation function ReLu is used to normalize the vectors in the nonlinear layers f _v and f _q ; feature

最后，选取得分更高的输出；Further, the visual question answering problem is a multi-label classification problem, and further, in step 4), the fusion features pass through the nonlinear layer f _o , and the activation function ReLu is used to normalize the vector in the passing through the nonlinear layer f _o ; and then use the linear mapping w _o to predict the candidate score of the answer

Finally, select the output with a higher score;

where σ is the sigmoid activation function and _wo is the weight parameter to be learned.

As an option, the sigmoid activation function normalizes the final score to the (0-1) interval, and the last stage acts as a logistic regression that predicts the correctness of each candidate answer with an objective function of

where the indices z and k cover the N candidate answers for the M training questions, respectively, and s _zk is the true answer to the question.

Compared with other common visual question answering using softmax classifier, the logistic regression classification used in the present invention is more effective. The sigmoid function uses a soft score (soft target) as the target result, which provides a richer training signal and can effectively capture the occasional uncertainty in the real answer.

In order to better observe how the attention model pays attention to the salient area of the image, after obtaining the attention map (a ^u , a ^b ) of the single attention weight and the salient attention weight, use matplotlib in python to draw the heatmap of the library The function visualizes the attention map as a matrix heatmap, as shown in Figure 3 and Figure 4.

Figure 3 and Figure 4 are the performance diagrams of the upper-layer attention model and the lower-layer attention model of the multi-view attention model for 2 different task images respectively, attention1 is the attention visual view of the upper-layer attention model, and attention2 is the lower-layer attention model. attention visuals. From the heat map of attention, it can be seen that the added lower-level attention model is able to learn different important regions of the input image. As can be seen from Figure 3, for a simple attention task, both the upper-layer attention model and the lower-layer attention model can find the correct position in the image. However, in Figure 4, it can be seen that when the task needs to be highly focused on multiple locations in the image, the lower-layer attention model pays attention to different parts from the upper-layer attention model, thereby improving the accuracy of the multi-view attention model Therefore, the multi-view attention model of the present invention has advantages over the prior art models.

Test dataset introduction: The VQA v2 dataset (Antol S, Agrawal A, Lu J, et al. Vqa:Visualquestion answering[C].Proceedings of the IEEE International Conference onComputer Vision.2015:2425-2433.) is a large-scale The visual question answering dataset in which all questions and answers are annotated by humans. There are 443,757 training questions, 214,354 validation questions and 447,793 test questions in the dataset. Each image is associated with three questions, for each question the annotators provide ten answers. In the standard visual question answering task, the questions in this dataset are often divided into three types: Yes/no, Number and other.

Further, in order to verify the validity of the present invention, the present invention was compared with the champion of the 2017 VQA Challenge (Anderson P, HeX, Buehler C, et al. Bottom-up and top-down attention for image captioning and visual question answering. arXiv preprint arXiv:1707.07998 , 2017.) The results are compared, as shown in Figure 5, the present invention replaces the original simple attention model with the multi-perspective attention model of the present invention on the basis of the reproduced thesis code, and the multi-perspective attention model of the present invention The final score of the force model is 64.35%, which is an improvement of about 1.2% compared with the paper in the evaluation of accuracy.

Some basic parameters in the experiment are designed as follows, the basic learning rate is set to α=0.0007, the random dropout rate after each LSTM layer is set to dropout=0.3, and the answer screening is set to N=3000. The hidden neurons of the fully connected layer are set to num_hid=1024, and the number of batch training is set to batch_size=512. The weight assignments of the single attention weight and the salient attention weight are β ₁ =0.7, β ₂ =0.3.

As shown in Figure 6, the loss function value (loss) of the model decreases with the increase of the training period and the convergence process; Performance.

Table 1 shows the comparison of the representative methods of the present invention in the test-dev case and the VQA task on the open standard data set VQA v2.

Table 1

In particular, the data is divided into three categories according to the question type for evaluation, and then the total evaluation result is calculated. The types of problems are whether Y/N is a problem, the number of Number problems, and Others other open problems. The scores in the table are the accuracy of the model's answering results for different types of questions. The larger the value, the higher the accuracy. It can be seen from the table that the multi-view attention model of the present invention achieves good effects on different tasks.

In particular, since the multi-view attention model of the present invention strengthens the fine-grained feature expression, the detection and recognition ability of objects is improved, and the evaluation of the type of Number has a good improvement compared with the previous method. The overall accuracy evaluation results of the model are even better than the results of most existing methods.

The above-mentioned embodiments are only used to illustrate the present invention, but not to limit the present invention. As long as it is in accordance with the technical essence of the present invention, changes, modifications, etc. to the above-described embodiments will fall within the scope of the claims of the present invention.

Claims (10)

1. A fine-grained visual question answering method in conjunction with a multi-view attention mechanism, characterized in that the steps are as follows: 1) Input image, extract image features; input question text, extract question features; 2) Input the image features and question features into the multi-view attention model, calculate the attention weight of the image, and perform a weighted operation on the image features of step 1) through the attention weight to obtain fine-grained image features; 3) Fusion of image fine-grained features and problem features to obtain fusion features; 4) Input the fused features into the classifier, and predict the answer. 2. The fine-grained visual question answering method combined with the multi-view attention mechanism according to claim 1, wherein the multi-view attention model comprises an upper-layer attention model and a lower-layer attention model, and the upper-layer attention model is The single attention weight is obtained, and the salient attention weight is obtained through the lower attention model, and the salient attention weight reflects that different target areas in the image correspond to different attention resources. 3. The fine-grained visual question answering method combining multi-view attention mechanism according to claim 2, is characterized in that, the method for obtaining single attention weight is as follows: Input image features and question features to the upper attention model, use a fully connected layer to project the data of image features and question features to the same dimensional space, and use the activation function ReLu to normalize the vector; then use Hada code product fusion, and then input in turn The two-layer fully connected layer processes the learned parameters and processes the learned parameters

Finally, use the softmax function to normalize the weights to get a single attention weight

in,

is the image feature, is the problem feature,

is the weight parameter to be learned by the upper attention model, K is the number of spatial regions of image features, T is the length of the selected problem feature, d is the number of hidden neurons in the network layer, h is the output dimension set by this layer, ReLu is an activation function in a neural network, and its specific form can be expressed as f(x)=max(0,x). 4. The fine-grained visual question answering method combining multi-view attention mechanism according to claim 2, is characterized in that, the method for obtaining salient attention weight is as follows: Input image features and question features to the lower attention model, use a fully connected layer to project the data of image features and question features to the same dimensional space, and then calculate the correlation matrix C _i =ReLu(q _i ^T W _b V _i ) ;in,

is the weight parameter to be learned by the lower attention model, is the obtained correlation matrix; Multiply the correlation matrix as a feature with the problem feature, and fuse it with the input image feature. The fused parameter is

Finally, use the softmax function to normalize the weights and output the saliency attention weights

in,

Weight parameters to be learned for the lower attention model. 5. The fine-grained visual question answering method combined with multi-view attention mechanism according to any one of claims 2, 3, and 4, wherein the attention weight of the image is calculated based on a single attention weight and a salient attention weight ,details as follows:

Among them, β ₁ and β ₂ are the weight ratio hyperparameters of the upper-layer attention model and the lower-layer attention model. 6. The fine-grained visual question answering method combined with multi-view attention mechanism according to claim 5, is characterized in that, in step 3), the fine-grained feature of the image and the question feature are respectively passed through the nonlinear layers f _v , f _q , The activation function ReLu is used to normalize the vector in the nonlinear layers f _v , f _q ; then the Hada code product fusion is used to obtain the fusion features

7. The fine-grained visual question answering method combined with multi-view attention mechanism according to claim 6, characterized in that, in step 4), the fusion feature passes through the nonlinear layer f _o , and uses the activation in the through nonlinear layer f _o The function ReLu normalizes the vector; the linear map w _o is then used to predict the candidate score of the answer

Finally, select the output with a higher score; where σ is the sigmoid activation function and _wo is the weight parameter to be learned. 8. The fine-grained visual question answering method combined with multi-view attention mechanism according to claim 7, characterized in that, the sigmoid activation function normalizes the final score to a (0-1) interval, and the last stage is used to predict each candidate answer Logistic regression for correctness with an objective function of

where z and k cover N candidate answers for M training questions, respectively, and s _zk is the real answer to the question. 9. the fine-grained visual question answering method in conjunction with multi-view attention mechanism according to claim 1, is characterized in that, in step 1), use Faster-RCNN standard model to carry out feature extraction to the image I _i of input, obtains deep expression The image features of _Vi = FasterRCNN(I _i ). 10. The fine-grained visual question answering method combined with multi-view attention mechanism according to claim 1, is characterized in that, in step 1), input question text Q _i , first use spaces and punctuation to divide question text Q _i into words, Then initialize it through the pre-trained GloVe word embedding method to obtain the encoded i-th specified question sentence

Among them, x _t ⁽ⁱ⁾ represents the t-th word of each word in the vocabulary; followed by Input into the _LSTM network, take out the output qi of the last layer as

The feature expression of , the problem feature _qi is obtained.

Images