KR102039397B1 - Visual Question Answering Apparatus for Explaining Reasoning Process and Method Thereof - Google Patents

Abstract

According to the present exemplary embodiment, by adding a plurality of region maps to the visual question answer model, the selected regions and regions selected in the process of inferring the correct answer as well as the correct answer are output, and the plurality of region features are combined to the sentence generation model. The present invention provides a visual query answering apparatus and method capable of outputting a descriptive sentence about an object other than the correct answer with an image.

Description

Visual Question Answering Apparatus for Explaining Reasoning Process and Method Thereof}

The technical field to which this embodiment belongs is directed to an apparatus and method for visual query response.

The contents described in this section merely provide background information on the present embodiment and do not constitute a prior art.

Explainable Artificial Intelligence (EAI) is a technology that provides end users with a wealth of information to understand artificial intelligence. Traditional AI technology provides only the final answer, so the engineer or user can't figure out why the machine gave this answer, why not the other answers, when the prediction fails, and how to fix the error. There is.

Visual Question Answering (VQA) is a technique that infers correct answers by finding words from various visual semantic level questions about an image and finding important areas of the image. Visual question responses require questions at various visual semantic levels for the same image. For example, a question may ask for an image "what is the object", "what is the color of the object", "the number of objects", and the like. However, there is a problem that the existing visual question answer cannot explain how the answer is derived.

 [Thesis] 'Studying Visual Q & A using In-depth Learning', Journal of Korean Information Science Society, pp 989-991, 2017.12

Embodiments of the present invention output a selection of regions and regions selected in the process of inferring not only a correct answer but also a correct answer by adding a plurality of region maps to a VQA model, and combining the plurality of region maps. The main purpose of the present invention is to output a descriptive sentence about an object that is not a correct answer together with an image by applying it to a sentence generation model.

Still other objects of the present invention may be further considered without departing from the following detailed description and effects thereof.

According to an aspect of the present embodiment, in a visual question answering method by a computing device, at least one region map is added between a plurality of layers of a correct answer inference model, and a question vector associated with an image based on the correct answer inference model is provided. Inferring a correct answer, and answering a descriptive question for inferring the correct answer using the plurality of area maps.

According to another aspect of the present embodiment, a correct answer inference unit for adding at least one region map between a plurality of layers of a correct answer inference model and infers a correct answer with respect to a question vector associated with an image based on the correct answer inference model, and the plurality of Provided is a visual question answering apparatus including a correct answer explanation unit for answering a descriptive question for inferring the correct answer using an area map of.

According to another aspect of this embodiment, a computer program for visual query response, which is recorded on a non-transitory computer readable medium containing computer program instructions executable by a processor, the computer program instructions being a computing device. When executed by at least one processor of, adding at least one region map between the plurality of layers of the correct inference model and inferring the correct answer for a question vector associated with the image based on the correct inference model; and Providing a computer program for performing operations including using the plurality of region maps to answer descriptive questions for inferring the correct answer.

As described above, according to the embodiments of the present invention, in the process of inferring the correct answer, it is possible to output the selected regions and the order in which the regions are selected, and to output a descriptive sentence about an object other than the correct answer together with the image. It is effective for engineers or users to understand the machine learning process and fine tune the learning model.

Even if the effects are not explicitly mentioned herein, the effects described in the following specification and the tentative effects expected by the technical features of the present invention are treated as described in the specification of the present invention.

1 is a block diagram illustrating a visual query response device according to an embodiment of the present invention.
2 is a diagram illustrating an example of explaining a correct answer inference process according to a question type by a visual question answering apparatus according to an exemplary embodiment.
3 is a diagram illustrating a process of inferring a correct answer by a visual query response device according to an embodiment of the present invention.
4 is a diagram illustrating a correct answer inference model of the visual query response device according to an embodiment of the present invention.
5 and 6 are diagrams illustrating a gate circulation unit of a visual query response device according to an embodiment of the present invention.
7 is a flowchart illustrating a visual query response method according to another embodiment of the present invention.

Hereinafter, in the following description of the present invention, if it is determined that the subject matter of the present invention may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted and some embodiments of the present invention will be omitted. It will be described in detail with reference to the exemplary drawings.

1 is a block diagram illustrating a visual query response device, and FIG. 2 is a diagram illustrating a visual question answering device describing a correct answer inference process according to a question type.

As illustrated in FIG. 1, the visual query response apparatus 100 includes a correct answer inference unit 110 and a correct answer description unit 120. The apparatus 100 may omit some of the various components illustrated by way of example in FIG. 1 or further include other components.

The visual question answering apparatus 100 is a device for finding a word from various visual semantic level questions about an image, finding an important region of the image, and inferring a correct answer. The visual query response apparatus 100 outputs not only the correct answer but also the selected regions and the order in which the regions are selected in the process of inferring the correct answer by adding a plurality of region maps to the VQA model. The visual query response apparatus 100 combines a plurality of area maps and applies them to a sentence generation model to output a description sentence about an object that is not correct along with the image.

The visual query response apparatus 100 extracts a visual feature using a correct answer reasoning model having a network composed of a plurality of layers. The correct reasoning unit 110 extracts the visual feature from the image based on the feature extraction model 101 including N layers. Where N is a natural number. The feature extractor 110 may design and apply a CNN (Convolutional Neural Network) model that classifies an image by repeating convolution and pooling operations on an image input.

The correct answer inference unit 110 adds at least one region map between the plurality of layers of the correct answer inference model and infers the correct answer with respect to the question vector associated with the image based on the correct answer inference model. The correct answer inference unit 120 infers the correct answer from features of not only the highest pooling layer but also other higher pooling layers, and calculates the correct answer from the question vector through a softmax classifier in a manner of learning about a predetermined answer candidate.

The correct answer inference unit 110 extracts a visual feature corresponding to the NKth layer to the Nth layer, respectively, for the correct answer inference model including the N layers, and corresponds to the NKth layer to the Nth layer, respectively. Learn based on visual features and question vectors. K is a natural number smaller than N. The correct answer inference unit 110 generates K region maps for dividing an image into a plurality of pieces and predicting an area required to infer a correct answer to correspond to the N-Kth layer to the Nth layer, respectively.

The correct answer description unit 120 responds to a descriptive question for inferring a correct answer using at least one area map.

Referring to FIG. 2, the descriptive question is: (i) a first question (Q1) of where the correct answer is located in the image, (ii) a second area of the selected regions and the order in which the regions are selected in the course of deduction Question Q2, (iii) the third question Q3 regarding why the object region other than the correct answer in the image does not correspond to the correct answer, or a question about a combination thereof.

The visual question answering apparatus 200 or the correct answer description unit 120 displays the location 211 of the correct answer in the image by reflecting an area map on the visual feature corresponding to the final layer among the plurality of layers with respect to the first question.

The visual question answering apparatus 200 or the correct answer explaining unit 120 may include the region features 221, 222, and 223 respectively reflecting a plurality of region maps to a plurality of visual features corresponding to some layers among the plurality of layers with respect to the second question. ) In the image order.

The visual question answering apparatus 200 or the correct answer explaining unit 120 may include a gate recurrent unit that includes a region feature in which a plurality of region maps are respectively applied to a plurality of visual features corresponding to some layers among the plurality of layers with respect to the third question. Unit, GRU) to generate the answer sentence.

Hereinafter, a process of inferring a correct answer by the visual question answering apparatus will be described with reference to FIGS. 3 and 4, and a question generation model and a correct answer inference model will be described.

The visual query response apparatus 100 generates a question vector based on the question generation model 330. The question vector is generated by encoding a question based on a Recurrent Neural Network (RNN) of a Long Short Term Memory (LSTM) structure and includes semantic relationship information.

Here, RNN is a neural network useful for expressing and learning patterns that have a continuity like the sentence and the order before and after, and the LSTM model is used for encoding the sentence question. LSTM analyzes the question and generates the final question vector (Vq) given the question. The final question vector is sent to the layer of the correct answer inference model 310.

The RNN has a loop that connects the result of the hidden layer back into the input of the same hidden layer, and has a structure in which past data affects the future. In other words, the RNN forms a circular structure in which hidden nodes are connected to edges having directions.

The LSTM model adds cells with multiple gates connected to the hidden layer. The hidden layer has a memory block including an input gate, an output gate, and a forge gate. The forge gate is a gate for forgetting past information, and the input gate is a gate for storing present information. The gates each have an intensity and a direction. The cell acts as a conveyor belt, and the gradient can keep propagating relatively even over time.

The correct inference model 310 includes an image based on a convolutional neural network (CNN) that takes an image having a predetermined size as an input and outputs a visual feature.

The correct inference model 310 generates a feature map by extracting a feature through a convolution operator in one or more layers. Nodes of one or more layers are networked. The feature extraction model delivers the extracted features to other layers and integrates the features extracted through subsampling to reduce the spatial dimension. The layer may include a parameter, and the parameter of the layer includes a set of learnable filters. The parameters include weights and / or biases between nodes. The correct inference model 310 learns a plurality of parameters, some of which may be shared. The correct answer inference model 310 may include a Softmax function and a loss function.

Region features are generated using higher pooling layers among the pooling layers of the correct inference model 310. The region features are calculated from a plurality of region maps for predicting the region required to infer the correct answer based on the question vector, and have a feature to which the prediction region is applied. The region features may be calculated using a weighted sum between the prediction region and the visual features of the pooling layer.

The correct answer explanation model 320 may describe the reasoning process by displaying the attention results 301, 302, and 303 of each floor memorized in the correct answer reasoning process to the user.

Referring to FIG. 4, the correct answer inference model 310 memorizes and learns some of the N layers step by step, and sequentially decreases the size of a reception field from the NK th layer to the N th layer. do.

Soft Attention adds additional sub-recognition units such as Softmax Classifier to improve the accuracy of problem inference. The correct answer descriptor divides the image into m pieces (regions) (m1xm2) for attribute detection and detects the most advantageous region for the correct answer inference through the softmax classifier. That is, the candidate region is detected in the segmented image. The m-dimensional vector to be expressed on a 0 to 1 scale is called an area map P _I , and is calculated as in Equation 1. The size of the fragment and the size of the vector are exemplary only, and are not limited thereto. Suitable figures may be used depending on the design implemented.

The answer inference model starts learning with pre-trained initial values. Each layer segment of the correct inference model is called {F1, F2, ..., Fn-1, Fn}, and only Fn-2, Fn-1, Fn is set as a learnable layer and the rest is fixed. The attended features according to this are defined as in Equation 1.

는 행렬과 벡터의 곱 또는 크로네커 곱을 나타내고, W는 가중치를 의미한다. 영역 특징은 영역 맵과 시각 특징을 곱하여 산출된다.The area map P ^l is calculated using a sigmoid function on the Kronecker product of the visual feature F ^l and the question vector v _Q multiplied by the weight W.

Denotes a product of a matrix and a vector or a Kronecker product, and W denotes a weight. The region feature is calculated by multiplying the region map and the visual feature.

을 각 출력 attention으로 사용한다. The visual query response device 100

Is used for each output attention.

There are three types of questions that can be explained to the user. Possible questions can be divided into (i) where the answer is, (ii) what the reasoning process is, and (iii) why the area is not the correct answer, along with the picture input.

When the user asks the reasoning process which is the second question, the visual query answering apparatus 100 outputs the corresponding attentions in order.

Hereinafter, referring to FIGS. 5 and 6, an operation of generating an answer sentence by using the gate circulation unit will be described.

When the user enters a third question with a picture, the visual question answering device generates a reply sentence using a gated recurrent unit (GRU). The visual query answering apparatus generates words repeatedly through the GRU until the < end > mark appears. The t-th word is output through Equation 2.

을 concatenation해서 사용하므로, 이들을 활용한 설명에 관한 문장 출력이 가능하다.For the first word ₁ , a feature vector obtained by passing through an image learning model (eg, CNN, etc.) is used, and word _t is a word vector corresponding to the word to be output t. Collect these words until you see the <end> mark and combine them into the answer sentence. That is, attention feature that was previously learned in GRU expression

Can be used by concatenation, so that the sentence can be printed about the description using them.

In FIG. 6, each GRU cell is shown, r _t serves as a weight to determine how much memory information is stored in the GRU, and z _t serves as a weight to determine how much to limit the input and output in the GRU. do. That is, each attended feature may be additionally inputted to the GRU and concatenation to generate a sentence in consideration of them.

The visual question answering apparatus infers the last correct answer using Equation (3).

는 아다마르 곱(Hadamard Product)을 나타낸다.

는 성분 곱(Element-Wise Production)을 의미한다. The inference vector u is computed using the hyperbolic tangent function for the Hadamard product of the region feature corresponding to the final layer and the question vector v _Q multiplied by the weight.

Represents the Hadamard Product.

Means element-wise production.

The correct answer inference model obtains the final correct answer from the vector u through the Softmax classifier. Softmax classifier is a linear classifier commonly used in neural networks. It is a classifier that calculates the loss using Softmax function. Set a plurality of words to be correct candidates in advance, and learn to extract the correct answers from them.

Although components included in the visual query response apparatus are separately illustrated in FIGS. 1 and 2, the plurality of components may be combined with each other and implemented as at least one module. The components are connected to the communication path connecting the software module or the hardware module inside the device and operate organically with each other. These components communicate using one or more communication buses or signal lines.

The visual query answering apparatus may be implemented in logic circuitry by hardware, firmware, software, or a combination thereof, or may be implemented using a general purpose or special purpose computer. The device may be implemented using a hardwired device, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or the like. In addition, the device may be implemented as a System on Chip (SoC) including one or more processors and controllers.

The visual query response apparatus may be mounted in a computing device provided with hardware elements in the form of software, hardware, or a combination thereof. The computing device includes various or all communication devices such as a communication modem for performing communication with various devices or wired and wireless communication networks, a memory for storing data for executing a program, a microprocessor for executing and operating a program, and the like. It can mean a device.

7 is a flowchart illustrating a visual query response method according to another embodiment of the present invention. The visual question answering method may be performed by a computing device or a visual question answering device, and a detailed description of the operation performed by the visual question answering device and a duplicate description thereof will be omitted.

In operation S710, the computing device adds at least one region map between the plurality of layers of the correct answer inference model, and infers the correct answer with respect to the question vector associated with the image based on the correct answer inference model.

The correct inference model includes N (N is a natural number) layers, and extracts visual features corresponding to N-K (K is a natural number less than N) th layer to the N th layer, respectively. The correct answer inference model learns based on the visual feature and the question vector corresponding to the N-K th layer to the N th layer, respectively. The correct inference model generates K region maps for dividing the image into a plurality of pieces and predicting an area necessary to infer the correct answer to correspond to the N-Kth layer to the Nth layer, respectively.

The corrective reasoning model performs a memory-learning of some of the N layers step by step, and sequentially decreases the size of the reception field from the N-K th layer to the N th layer.

The question vector is generated by encoding a question based on a Recurrent Neural Network (RNN) of a Long Short Term Memory (LSTM) structure and includes semantic relationship information.

In operation S720, the computing device responds to the descriptive question for inferring the correct answer using the at least one area map.

The descriptive question is (i) a first question about where the correct answer is located in the image, (ii) a second question about the regions selected in the process of deriving the correct answer and the order in which the regions are selected, and (iii) the correct answer in the image. This question relates to a third question, or a combination, of why a non-object region is not the correct answer.

In response to the question that can be described (S720), the location of the correct answer is displayed in the image by reflecting the area map on the visual feature corresponding to the final layer among the plurality of layers.

Responding to the descriptive question (S720) displays area features in which the plurality of area maps are respectively reflected in the plurality of visual features corresponding to some of the plurality of layers with respect to the second question in the image.

In response to the question that can be described (S720), the gate features include a gated recurrent unit (GRU) for region features in which a plurality of region maps are respectively applied to a plurality of visual features corresponding to some layers among the plurality of layers. Type in to generate the answer sentence.

According to the present embodiments, the output area of each layer is a large area without using additional learning materials since the generation of the areas is sequentially performed using a large reception area from a lower layer. It is possible to make natural inference that narrows the attention to the narrow area at. By showing the attention result P ^l of each layer to the user, it can be explained to the user in a way that explains the reasoning process. Answer The reasoning model remembers the reasoning process step by step, so even if a question is entered along with a picture, there is an advantage that can be explained to the user based on that step.

Although each process is described as being sequentially executed in FIG. 7, this is merely an example, and a person skilled in the art may change the order described in FIG. 7 without departing from the essential characteristics of the exemplary embodiment of the present invention. It may be possible to apply various modifications and variations, or to execute one or more processes in parallel or to add other processes.

The operations according to the embodiments may be implemented in the form of program instructions that may be executed by various computer means and may be recorded in a computer readable medium. Computer-readable media refers to any medium that participates in providing instructions to a processor for execution. Computer-readable media can include program instructions, data files, data structures, or a combination thereof. For example, there may be a magnetic medium, an optical recording medium, a memory, and the like. The computer program may be distributed over networked computer systems so that the computer readable code is stored and executed in a distributed fashion. Functional programs, codes, and code segments for implementing the present embodiment may be easily inferred by programmers in the art to which the present embodiment belongs.

The present embodiments are for describing the technical idea of the present embodiment, and the scope of the technical idea of the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present embodiment.

100, 200: visual question and answer device
110, 310: Answer Inference
120, 320: Answer explanation
330: question generation model

Claims (15)

In the visual question answering method by the visual question answering device,
Adding at least one region map between the plurality of layers of the correct answer inference model by the correct reasoning unit, and inferring the correct answer with respect to the question vector associated with the image based on the correct answer reasoning model; And
Answering, by a correct answer descriptor, a descriptive question for inferring the correct answer using the at least one area map,
The descriptive question may be (i) a first question where the correct answer is located in the image, (ii) a second question about the regions selected in the process of deriving the correct answer and the order in which the regions are selected, (iii And a third question regarding a reason why the object region other than the correct answer in the image does not correspond to the correct answer, or a combination thereof. The method of claim 1,
The correct answer inference model includes N layers (where N is a natural number), and extracts visual features corresponding to layers NK (the K is a natural number less than N) to Nth layers, respectively. The image is divided into a plurality of pieces by learning based on the visual feature and the question vector respectively corresponding to the layer to the Nth layer, and the correct answer is arranged to correspond to the NKth to Nth layers, respectively. And generating K region maps for predicting the regions required for inference. The method of claim 2,
The correct answer inference model is a memory learning step by step of some layers of the N layer, characterized in that to sequentially reduce the size of the (Reception Field) from the NK th layer to the N th layer How to answer visual questions. The method of claim 1,
The question vector is generated by encoding a question based on a recurrent neural network (RNN) of a long short term memory (LSTM) structure and includes semantic relationship information. delete The method of claim 1,
Responding to the descriptive question may include:
And displaying the location of the correct answer in the image by reflecting the area map on a visual feature corresponding to a final layer among the plurality of layers with respect to the first question. The method of claim 1,
Responding to the descriptive question may include:
And answering the second question in order to display, in the image, region features reflecting a plurality of region maps in a plurality of visual features corresponding to some layers of the plurality of layers. The method of claim 1,
Responding to the descriptive question may include:
Generating an answer sentence by inputting region features reflecting a plurality of region maps to a plurality of visual features corresponding to some layers of the plurality of layers, respectively, to the gated recurrent unit (GRU) for the third question. Visual question and answer method characterized. A correct answer inference unit that adds at least one region map between a plurality of layers of a correct inference model and infers a correct answer to a question vector associated with an image based on the correct inference model; And
A correct answer descriptor for answering an explanatory question for inferring the correct answer using the at least one area map,
The descriptive question may be (i) a first question where the correct answer is located in the image, (ii) a second question about the regions selected in the process of deriving the correct answer and the order in which the regions are selected, (iii And a third question regarding a reason why the object region other than the correct answer in the image does not correspond to the correct answer, or a combination thereof. The method of claim 9,
The correct answer inference model includes N layers (where N is a natural number), and extracts visual features corresponding to layers NK (the K is a natural number less than N) to Nth layers, respectively. The image is divided into a plurality of pieces by learning based on the visual feature and the question vector respectively corresponding to the layer to the Nth layer, and the correct answer is arranged to correspond to the NKth to Nth layers, respectively. And generating K area maps for predicting an area required for inference. delete The method of claim 9,
The correct answer explanation section,
And answering the first question to display the location of the correct answer in the image by reflecting the area map on a visual feature corresponding to a final layer among the plurality of layers. The method of claim 9,
The correct answer explanation section,
And answering the second question in order to display, in the image, region features reflecting a plurality of region maps on a plurality of visual features corresponding to some layers of the plurality of layers. The method of claim 9,
The correct answer explanation section,
Generating an answer sentence by inputting region features reflecting a plurality of region maps to a plurality of visual features corresponding to some layers of the plurality of layers, respectively, to the gated recurrent unit (GRU) for the third question. Visual question answering device characterized in. A computer program recorded on a non-transitory computer readable medium containing computer program instructions executable by a processor, wherein the computer program instructions are executed by at least one processor of the computing device. If the,
Adding at least one region map between the plurality of layers of the correct inference model and inferring the correct answer for the question vector associated with the image based on the correct inference model; And
Perform operations including using the at least one area map to respond to a descriptive question for inferring the correct answer,
The descriptive question may be (i) a first question where the correct answer is located in the image, (ii) a second question about the regions selected in the process of deriving the correct answer and the order in which the regions are selected, (iii 3) a third question regarding why an object region other than the correct answer in the image does not correspond to the correct answer, or a combination thereof.

Images