CN115512191A - Question and answer combined image natural language description method - Google Patents

Abstract

A question-answer combined image natural language description method comprises three steps: firstly, extracting the features of an image target and an image background by using an image segmentation model to obtain different pixel-level classifications, and acquiring a segmentation feature map of the target and the background; step two, a problem generation module generates a relation characteristic diagram containing attention target information by constructing an implicit scene type representation, and generates a plurality of semantically related guide problems in a multi-granularity manner; and step three, the combined question-answering module introduces a loss function of contrast learning, performs combined multi-mode embedded representation on the relation characteristic graph and the guide question, and can generate a long text answer related to the question as a refined semantic description of the image content through training.

Description

A joint question-answering method for image natural language description

technical field

The invention belongs to the fields of computer vision and natural language processing.

Background technique

Image caption generation is a multimodal task across text and images, aiming to generate corresponding natural language descriptions from images. This task is very easy for humans, but very challenging for computers. With the popularity of deep learning, more and more people try to use neural networks to solve the image description problem of machines.

However, due to the diversity of natural language descriptions, it has different standard forms, and training with a specific data set can only obtain image content descriptions that conform to the distribution of the data set, which limits the scope of application. At the same time, most image description methods only passively generate monotonous sentences, and do not consider the connection between the scene in the image and the target of interest, so it is difficult to describe different, multi-scale targets at the same time, and it is easy to Turn a blind eye to potential connections. This kind of mechanical image description has a large deviation from human semantic cognition of images, and it is often impossible to understand each other when interacting with real humans.

Image description generation is generally divided into two sub-modules: image feature extraction and text generation. The common method of image feature extraction is to use the image recognition neural network model to extract the target, but this will lead to the loss of image information and the deviation of feature extraction. At the same time, short sentence generation based on feature extraction often only focuses on specific targets, and cannot generate multi-granularity descriptions from the rich semantic information of images, which is likely to cause a large loss of cross-modal information. Therefore, there is still a large gap between the machine's image description and human's natural cognition of images.

Contents of the invention

The present invention provides a joint question-and-answer image natural language description method in view of the deficiency of the background technology. The image content description is generated on the basis of the visual question-and-answer model. By designing the image segmentation module, the question generation module and the joint question-answer module, different The scale area generates multiple semantically related questions, using question-answer as the corresponding relationship, using guided questions with multi-grained features to generate a refined description of the image scene, this method can capture the possibility of implicit facts in the image, and generate more in line with A Natural Language Description of Human Semantic Cognition of Images.

The present invention adopts following technical scheme:

An image natural language description method based on joint question answering, characterized in that a refined description of image content is generated based on a visual question answering model. First, the image segmentation module obtains the segmentation feature map of image target and background category classification; second, the question generation module constructs an implicit scene type representation, and generates multi-granularity guidance questions centered on the target of interest; finally, the joint question answering module introduces contrastive learning A loss function for joint multimodal embedding representation of relational feature maps and bootstrapping problems. Based on visual question answering, this method generates a natural language description of the image content according to the question-answer correspondence.

A joint question-answering image natural language description method, including three steps:

Step 1, first use the image segmentation model to extract the features of the image target and the image background, obtain the division of different categories at the pixel level, and obtain the segmentation feature map of the target and the background;

Step 2, the question generation module generates a relational feature map containing the target information by constructing an implicit scene type representation, and generates several semantically related guiding questions at multiple granularities;

Step 3: The joint question answering module introduces the loss function of comparative learning, and performs joint multimodal embedding representation on the relationship feature map and the guiding question. After training, the model can generate long text answers related to the question as a refined semantic description of the image content. .

For step one, the present invention provides a preferred solution for extracting image features.

For step 2, the present invention discloses a problem generation model based on the LSTM model, which is characterized in that the segmentation feature map can be processed, and by constructing an implicit scene type representation, a relational feature map containing the target information of interest is first generated, Then centering on the target of interest, the relationship between the target of interest in the image and between the target of interest and the background is established on a multi-scale basis, and the multi-granularity guidance questions generated are used as a link in the subsequent joint question answering.

For step three, the present invention discloses a joint question answering model based on the BUTD (bottom-up, top-down) model, which is characterized in that a loss function of contrastive learning is introduced, which can combine relational feature maps and guiding questions to improve The cross-modal learning ability of the model enhances the model's understanding of the semantic connection between the image and the answer to the question, and generates a refined description of the image content.

specific,

Step 1: Image Segmentation

1.1 Utilize the publicly available image semantic segmentation dataset, in which batch images are labeled with pixel-level categories.

1.2 Use the deep learning method to train the image segmentation data set and construct the image segmentation neural network model. The task of image segmentation is to make dense predictions for the image. By marking different targets with specific colors, each pixel has its own target or closed area category.

1.3 Save the model weight of the trained image segmentation neural network. The network model can process the original image, distinguish different targets and backgrounds in the image, and finally output the segmentation feature map between targets and between targets and background.

Step 2: Question Generation

2.1 Process the publicly available visual problem generation data set, classify the problem categories in the data set, different problem categories look at the target and the relationship between them from multiple perspectives, multiple problem categories of the same image not only focus on different targets , also focus on image regions of different scales of the same object. At the same time, the answers and questions of the data set are merged to generate a complete natural language description.

2.2 Use the deep learning method to train the processed visual question generation data set, and construct the question generation neural network model. The question generation model constructs an implicit scene type representation, initially generates a relational feature map containing the information of the target of interest, and then centers on the target of concern, learns the correlation between the category of the question and the different granularity regions of the image, and generates and targets of concern on a multi-scale basis Context-dependent different questions.

Step 3: Joint Q&A

3.1 Integrate the image segmentation module, question generation module and joint question answering module, take the guiding question as the context, use the top-down attention mechanism for learning, introduce the loss function of contrastive learning, and perform joint multi-mode on the guiding question and the relationship feature map state embedded representation. According to the trained network, the candidate answer and its confidence are given to generate a natural language description of the image content.

Beneficial effects of the present invention:

1. Image segmentation (using other people's existing segmentation models) not only includes explicit target and background pixel-level information, but also includes subordination, juxtaposition and logical relationship between hidden objects, which can provide comprehensive and fine image feature information.

2. Compared with directly generating image descriptions in this design, the multi-granularity feature questions generated from images in step 2 can guide more detailed and specific multi-scale answers, and at the same time capture the possibility of implicit facts in images, providing The ability to describe simple events.

3. This design scheme provides a new method of image description, which uses multi-granularity feature questions as the guiding core of image content description generation, and uses the visual question answering model to construct the corresponding relationship between questions and answers (this application first proposed the use of visual question answering The model generates image descriptions), and the introduction of the loss function of contrastive learning improves the cross-modal learning ability of the model, and can provide natural language descriptions that are more in line with human perception of image semantics, which is conducive to improving the processing efficiency of human-computer interaction.

Description of drawings

Figure 1 Image Segmentation Model Diagram

Figure 2 The problem generation model designed by the present invention

Figure 3 joint question answering model diagram

Figure 4 Image description example of joint question answering

detailed description

Example

An example of image captioning for joint question answering is shown in Figure 4.

A method for segmented image content description based on visual question answering, comprising the following steps:

Step 1: Image Segmentation

1.1 In this embodiment, the dataset used is the Cityscapes dataset, which focuses on the visual understanding of complex urban street scenes, which contains a variety of stereoscopic video sequences recorded in street scenes from 50 different cities, With 20,000 frames of weak annotations and 5,000 frames of high-quality pixel-level annotations, a total of 30 categories of annotations including pedestrians, vehicles, roads, buildings, signal lights, and signal signs are provided.

1.2 In this embodiment, the deep learning method is used to train the image segmentation data set, and the Deeplab image segmentation network model of the encoder-decoder structure is constructed. The Deeplab series is a full convolution-based dilated convolution semantic segmentation model. The proposed dilated convolution can restore the feature map resolution that is continuously reduced in the convolution operation as much as possible without increasing the amount of parameters. The Deeplab v3+ model used in this embodiment is a segmentation network model with an encoder-decoder structure, which has high calculation speed and prediction accuracy. Among them, the encoder module is used to extract high-level semantic information in the image. The decoder module is used to restore the low-level spatial information, and obtain clear and complete category segmentation boundaries and the relationship between objects, objects and background. The network model is shown in Figure 1.

In the encoder structure stage, the Xception network combined with depthwise separable convolution is firstly used as the backbone network to extract the initial features of the input image. The Xception network uses residual connections and depthwise separable convolutions to downsample feature maps. The subsequent spatial pyramid pooling module with holes (ASPP, Atrous Spatial Pyramid Pooling) is used to further extract multi-scale feature information. The 1×1 convolution, global average pooling and expansion rate (indicated by Rate) in the module are 6 respectively. , 12, and 18 hole convolutions are combined into a vertical parallel structure, and then the multi-scale feature map is processed through 1×1 convolution to compress the number of channels to 256, and the resolution of the feature map is reduced to 1/16 of the original image, as an encoder The feature map output of the structure.

In the decoder structure stage, the feature map output by the encoder structure is subjected to 4 times bilinear interpolation upsampling processing, and the shallow feature map of the corresponding level on the backbone network is spliced after channel adjustment, and then two 3 The ×3 convolutional layer refines the feature map, and finally undergoes 4 times bilinear interpolation upsampling to obtain a segmentation prediction result with the same size as the original image with rich details and global information.

1.3 Save the model weight of the trained Deeplab v3+ image segmentation neural network. The network model can process the original image, distinguish different targets and backgrounds in the image, and finally output the segmentation feature map between targets and between targets and background. Pixel-level features of images are exhaustively extracted.

Step 2: Question generation

2.1 In this embodiment, the existing public visual problem generation data sets (including SQuAD data sets, etc.) are processed, and the problem categories in the data set are classified, and different problem categories look at the targets and their connections from multiple perspectives. , multiple problem categories of the same image not only focus on different objects, but also focus on image regions of different scales of the same object. At the same time, the answers and questions of the data set are merged to generate a complete natural language description.

The problem category includes object (Object), attribute (Attribute), relationship (Relationship), count (Counting), behavior (Behavior) and so on.

2.2 In this embodiment, a deep learning method is used to train the processed visual problem generation data set, and a neural network model for problem generation is constructed.

In order to generate vision-based questions directly from feature images, a residual-connected MLP (Multi-Layer Perceptron) is constructed, which takes a segmentation feature map containing category information as input, and outputs a relational feature map containing mainly target-of-interest information, using to guide subsequent question generation.

A typical MLP includes three layers: input layer, hidden layer, and output layer. Different layers of the MLP neural network are fully connected. The hidden layer is trained to construct image scene type representation weights. The calculation formula of each layer of neural network is as follows:

H＝XW _h +b _h

O＝HW _o +b _o

Y=σ(O)

Among them, X is the feature input, Y is the feature output, W _h and W _o are the weights of the hidden layer and the output layer respectively, b _h and b _o are the deviations of the hidden layer and the output layer respectively, σ represents the activation function, here Use the sigmoid function.

The segmentation feature map generates a relational feature map through the residual connection, and the channel representing the relational weight represents the salience of the target of interest. The closed target area of each segmentation feature map has a corresponding attention coefficient, and u represents the relational feature layer. Subsequently, the question generation model is based on LSTM (long-term short-term memory recurrent neural network), establishes the relationship between each target of interest and the surrounding closed range category in the relationship feature map in a multi-granularity and hierarchical manner, and predicts the highest priority at different scales. The question categories generate the associated bootstrap questions.

Use LSTM to learn the relationship between the focus target and the problem, so that the model can lock the relevant area of the focus target according to the problem for training. LSTM involves the transfer of short-term memory h and long-term memory C, including input gates, output gates and forget gates, including the following formulas (for prior art):

c ^t = z ^t ⊙i ^t +c ^t-1 ⊙f ^t

y ^t ＝(c ^t )⊙o ^t

Among them, x ^t is the input at the current moment, y ^t-1 is the short-term memory of the previous moment, c ^t-1 is the long-term memory of the previous moment, W _f , W _i , W _z and W _o are the corresponding input weights , R _f , R _i , R _z and R _o are the corresponding recursive weights, p _i represents the peephole weight matrix, which is used to update the particle size, b _f , b _i , b _z and b _o are the corresponding calculations Deviation, σ represents the sigmoid activation function.

The loss function of the problem generation neural network model is as follows:

是模型生成的问题向量，q_i是数据集中真实的问题向量，u表示预测的关系权重值，为正数。in,

is the question vector generated by the model, q _i is the real question vector in the data set, and u represents the predicted relationship weight value, which is a positive number.

The network model for question generation is shown in Figure 2. The relational feature map provides the overall context information of the image. By constructing an implicit scene type representation, the attention target can efficiently focus the problem generation on several targets, better limit the attention target of the image scene, and provide more It is an abstract target focus representation; centered on the focus target, at different levels of granularity, several questions of different scales are generated according to the predicted top priority category. These questions cover the important information of the core area in the image, making the question generation neural network The bootstrapping problem dealt with by the network model is more comprehensive.

Step 3: Joint Q&A

3.1 In this embodiment, the image segmentation module of step 1, the question generation module of step 2 and the joint question answering module of step 3 are integrated, and the guiding question is used as the context, and the top-down attention mechanism is used for learning, and the guiding question and the relationship feature Graph for joint multimodal embedding representation.

First, connect the question feature vector q generated by the question and the segmentation feature map v obtained by image segmentation as the context to guide the model to train the weights W _q and W _v of v and q;

f _q (q)=W _q q

f _v (v) = W _v v

Multimodal embedding of f _q (q) and f _v (v), computed using the Halma product:

p(y)=σ(h)

Among them, f _q represents the output of the problem flow, f _v represents the output of the visual flow, p(y) is the final output result, the problem feature vector q and the segmentation feature map v with a closer matching degree have higher scores, W _q and W _v represent the weight matrix, and σ represents the linear activation function. The loss function for the joint embedding is the contrastive loss:

where y _T represents the output of the correctly matched question feature vector q and segmentation feature map v.

The network model of joint question answering is shown in Figure 3. Integrating the image segmentation module, question generation module and joint question answering module, according to the trained network, it can give the answer prediction with the highest confidence. This method is based on visual question answering, and can make a fine description of the image content.

Claims (8)

1. a joint question-and-answer image natural language description method, is characterized in that, comprises three steps: Step 1, first use the image segmentation model to extract the features of the image target and the image background, obtain the division of different categories at the pixel level, and obtain the segmentation feature map of the target and the background; Step 2, the question generation module generates a relational feature map containing the target information by constructing an implicit scene type representation, and generates several semantically related guiding questions at multiple granularities; Step 3: The joint question answering module introduces the loss function of comparative learning, and performs joint multimodal embedding representation on the relationship feature map and the guiding question. After training, the model generates long text answers related to the question as a refined semantic description of the image content. 2. The description method as claimed in claim 1, characterized in that: For step 2, the problem generation model based on the LSTM model processes the segmentation feature map, and by constructing an implicit scene type representation, first generates a relational feature map containing the information of the target of interest, and then centers on the target of interest, multi-scale The relationship between the focus target of the image and the focus target and the background is accurately established, and the multi-granularity guidance questions generated are used as a part of the subsequent joint question answering. 3. The description method as claimed in claim 1, characterized in that: For step 3, the joint question answering model based on the BUTD model introduces a loss function of contrastive learning, joint relational feature maps and guiding questions, improves the model's cross-modal learning ability, and enhances the model's semantic connection between images and question answers understanding to generate a fine-grained description of image content. 4. The description method as claimed in claim 1, characterized in that: Step 1: Image Segmentation 1.1 Utilize the publicly available image semantic segmentation dataset, in which the batch images are labeled with pixel-level categories; 1.2 Use the deep learning method to train the image segmentation data set and construct the image segmentation neural network model; the task of image segmentation is to densely predict the image, and mark different objects with specific colors so that each pixel has its own type of target or closed area; 1.3 Save the model weight of the trained image segmentation neural network. The network model can process the original image, distinguish different targets and backgrounds in the image, and finally output the segmentation feature map between targets and between targets and background. 5. The description method as claimed in claim 1 or 2, characterized in that: Step 2: Question generation 2.1 Process the publicly available visual problem generation data set, classify the problem categories in the data set, different problem categories look at the target and the relationship between them from multiple perspectives, multiple problem categories of the same image not only focus on different targets , and also pay attention to the image areas of the same target and different scales; at the same time, the answers and questions of the data set are merged to generate a complete natural language description; 2.2 Use the deep learning method to train the processed visual question generation data set, and construct the question generation neural network model; the question generation model constructs an implicit scene type representation, initially generates a relational feature map containing the target information, and then uses the target target As the center, we learn the correlation between the question category and different granular regions of the image, and generate different questions related to the target context of attention at multiple scales. 6. The description method as claimed in claim 1 or 3, characterized in that: Step 3: Joint Q&A 3.1 Integrate the image segmentation module, question generation module and joint question answering module, take the guiding question as the context, use the top-down attention mechanism for learning, introduce the loss function of contrastive learning, and perform joint multi-mode on the guiding question and the relationship feature map State embedding representation; According to the trained network, candidate answers and their confidence are given to generate a natural language description of the image content. 7. The description method as claimed in claim 5, characterized in that: The loss function of the problem generation neural network model is as follows:

in,

is the question vector generated by the model, q _i is the real question vector in the data set, and u represents the predicted relationship weight value, which is a positive number. 8. The description method as claimed in claim 5, characterized in that: The loss function for the joint embedding is the contrastive loss:

where y _T represents the output of the correctly matched question feature vector q and segmentation feature map v.

Images