JP2020149685A - Visual question answering model, electronic device, and storage medium - Google Patents

Abstract

To provide a visual question answering model, an electronic device, and a storage medium, which enable visual question answering by combining image information and text question information.SOLUTION: A visual question answering model is provided, including: a text encoder configured to perform pooling on a word vector sequence of an entered question text to extract a semantic representation vector of the question text; and an image encoder configured to extract an image feature of a given image in combination with the semantic representation vector.SELECTED DRAWING: Figure 1

Description

Examples of the present invention relate to the field of artificial intelligence technology, and more specifically to visual question answering models, electronic devices, and storage media.

Visual Question Answering (hereinafter abbreviated as VQA) is a typical application for multi-modality fusion. For example, for a given image, when a batter dressed in red is asked the related question "what color shirt is the batter waering", the VQA system combines the image information with the text question information. , The answer needs to be predicted as "red". In this process, the semantic features of the image and the text are mainly extracted, and the extracted image and the text are fused for the two modality features. Therefore, the coding part of the VQA-related model is mainly the text encoder and the image. It consists of an encoder.

However, because both the image encoder and the text encoder need to be used at the same time, the VQA model often contains a lot of parameters that need to be trained, so the training time of the model is very long. Become. Therefore, whether to simplify the model from the engineering point of view and improve the training efficiency of the model without significantly reducing the accuracy of the model is a technical problem that needs to be solved at present.

An embodiment of the present invention achieves to improve the training efficiency of a visual question answering model by simplifying the model from an engineering point of view without significantly reducing the accuracy of the visual question answering model. Provides equipment and storage media.

In the first aspect, an embodiment of the present invention combines a text encoder for pooling a word vector sequence of input question text to extract a semantic expression vector of the question text and the semantic expression vector. Provided is a visual question-and-answer model including an image encoder for extracting image features of a predetermined image.

In a second aspect, embodiments of the present invention further provide an electronic device, wherein the electronic device comprises one or more processors and a memory for storing one or more programs. When one or more programs are executed by the one or more processors, the one or more processors execute the visual question-and-answer model described in any of the embodiments of the present invention.

In a third aspect, an embodiment of the present invention provides a computer-readable storage medium in which a computer program is stored, and if the program is executed by a processor, it is described in any of the embodiments of the present invention. Visual question answering model is executed.

The embodiments of the present invention provide a visual question answering model, electronic devices, and storage media. The visual question answering model achieves the purpose of simplifying the visual question answering model by encoding the text vector in a pooling process, and trains in the visual question answering model by a simple coding method called pooling process. It reduces the number of parameters that need to be done, effectively improves the training efficiency of the visual question answering model, and is beneficial for engineering use.

It is a schematic block diagram of the visual question answering model which concerns on Example 1 of this invention. It is a schematic block diagram of another visual question answering model which concerns on Example 2 of this invention. It is a schematic block diagram of the electronic device which concerns on Example 3 of this invention.

Hereinafter, the present invention will be described in more detail with reference to the drawings and examples. It should be understood that the specific examples described herein are merely for the purpose of interpreting the present invention and are not intended to limit the present invention. For the sake of brevity, only some, but not all, of the configurations relating to the present invention are shown in the drawings.

Example 1
FIG. 1 is a visual question answering model according to the first embodiment of the present invention. This embodiment improves the training efficiency of the visual question-and-answer model by simplifying the visual question-and-answer model, which can be run on an electronic device such as a computer terminal or server.

As shown in FIG. 1, the visual question answering model according to the embodiment of the present invention is a text encoder for pooling the word vector sequence of the input question text and extracting the semantic expression vector of the question text. May be provided.

Here, the question text needs to be pre-processed before it can be encoded. As an example, the question text is processed by the word2vec model or the grow model to obtain the word vector sequence corresponding to the question text. To encode the question text, the word vector sequence corresponding to the question text can be input to the text encoder, and the word vector sequence of the question text can be pooled by the text encoder to extract the semantic expression vector of the question text. it can. In the prior art, an LSTM (Long Short-Term Memory, long / short-term memory) model or a Bi-LSTM (Bi-directional Long Short-Term Memory, bidirectional LSTM) model is used as a text encoder. In, the pooling process is used instead of the LSTM model or Bi-LSTM model as the text encoder, which simplifies the visual question-and-answer model.

In this embodiment, the pooling process is a max Pooling process, and the maximize pooling process is represented by the following equation.
f (w1, w2, ..., wk) = max ([w1, w2, ..., wk], dim = 1)

Where f represents the maximized pooling processing function, k is the number of word vectors contained in the question text, and wi is obtained by processing the question text using a pre-trained word vector model. Is the i-th word vector obtained, i is a natural number in [1, k], and max ([w1, w2, ..., wk], dim = 1) is the word vector w1, w2, .. It represents the maximum value of the dimension corresponding to each word vector in., Wk, and dim = 1 indicates a dimension, that is, it represents taking a value for each row in a predetermined two-dimensional matrix.

であり、上記の式によって当該単語ベクトルシーケンスを最大化プーリング処理して、

を得るため、

は、当該質問テキストのセマンティック表現ベクトルである。したがって、最大化プーリング処理することにより、視覚的質問応答モデルにおけるトレーニングする必要があるパラメータの数を削減し、視覚的質問応答モデルのトレーニング効率を向上させることができる。 As an example, a word vector sequence of one question text

The word vector sequence is maximized and pooled by the above equation.

To get

Is the semantic representation vector of the question text. Therefore, the maximized pooling process can reduce the number of parameters that need to be trained in the visual question answering model and improve the training efficiency of the visual question answering model.

In addition, the image encoder in the visual question answering model of the embodiment of the present invention is used in combination with a semantic expression vector to extract image features of a predetermined image.

The image contains background and rich content, so a visual attention mechanism (figure) to ensure that the machine pays attention to the image content related to the question and to improve the accuracy of answering the question. Attention in 1) can be adopted. Through the attachment mechanism, the image encoder narrows down the image content most related to the semantic expression vector by combining the semantic expression vectors corresponding to the question text acquired by the text encoder, and extracts the image features of the image content. The image feature vector can be obtained. A convolutional neural network model such as the Faster RCNN model can be adopted.

Further, as shown in FIG. 1, the visual question-and-answer model further comprises a feature fusion for fusing features of different modality, and in this embodiment, the feature fusion is output by an image encoder. The resulting image feature vector and the semantic representation vector output by the text encoder are fused. As an example, the image feature vector and the semantic representation vector can be fused by the dot product.

The visual question-and-answer model further comprises a classifier, which numerically processes the vector output by the feature fusion device by a softmax function (also called a normalized exponential function) to provide different answers. The relative probability between them is acquired, and the answer corresponding to the maximum relative probability is output.

Regarding the above visual question answering model, in one specific embodiment, the dataset Visual Genome released by Stanford University Centers for Artificial Intelligence is used as training sample data and validation data, and the training sample data and validation data are 2: 1. The visual question answering model can be trained and validated in proportion and randomly distributed. Table 1 shows specific data statistics for the dataset. Each image contains a fixed number of questions and a given answer is artificially labeled.

The above data will be used to train and validate the visual question answering model for this example. Specifically, the visual question answering model can be executed in the P40 cluster, and Table 2 shows the environment configuration of the P40 cluster and the basic parameters of the model. For comparison, we also trained and validated a prior art visual question answering model using LSTM or Bi-LSTM as a text encoder, and the results are shown in Table 3.

From the verification results shown in Table 3, the visual question-and-answer model of the embodiment of the present invention that employs the maximized pooling process as the text encoder is a conventional visual question-and-answer model that employs LSTM or Bi-LSTM as the text encoder. It can be seen that the prediction accuracy is reduced by about 0.5% as compared with the model, but the execution time of the model is shortened by up to 3 hours, and the training efficiency is greatly improved.

In the embodiment of the present invention, the visual question answering model encodes a text vector by a pooling process to achieve the purpose of simplifying the visual question answering model, and visually by a simple encoding method called pooling. It is possible to effectively improve the training efficiency of the visual question answering model without significantly reducing the prediction accuracy of the question answering model, which is useful for engineering use.

Example 2
FIG. 2 is a schematic configuration diagram of another visual question answering model according to this embodiment. As shown in FIG. 2, the visual question answering model includes a text encoder for pooling a word vector sequence of input question text to extract a semantic representation vector of the question text.

ただし、ｐは平均化プーリング処理関数を表し、ｋは前記質問テキストに含まれる単語ベクトルの数であり、ｗｉは予めトレーニングされた単語ベクトルモデルを利用して前記質問テキストを処理することによって得られたｉ番目の単語ベクトルであり、ｉは[１，ｋ]内の自然数であり、

は、単語ベクトルｗ１，ｗ２，...，ｗｋにおける各単語ベクトルに対応する次元の値の合計を表す。 Here, the pooling process is an averaging pooling process, and the averaging pooling process (avgPooling) can be expressed by the following equation.

Where p represents the averaging pooling processing function, k is the number of word vectors contained in the question text, and wi is obtained by processing the question text using a pre-trained word vector model. Is the i-th word vector, i is a natural number in [1, k],

Represents the sum of the dimensional values corresponding to each word vector in the word vectors w1, w2, ..., Wk.

であり、上記の式によって当該単語ベクトルシーケンスを平均化プーリング処理して

を得るので、

は当該質問テキストのセマンティック表現ベクトルである。したがって、平均化プーリング処理によって、視覚的質問応答モデルにおけるトレーニングする必要があるパラメータの数を減らし、視覚的質問応答モデルのトレーニング効率を向上させることができる。 As an example, a word vector sequence of one question text

The word vector sequence is averaged and pooled by the above formula.

Because you get

Is the semantic representation vector of the question text. Therefore, the averaging pooling process can reduce the number of parameters that need to be trained in the visual question answering model and improve the training efficiency of the visual question answering model.

In addition, the image encoder in the visual question answering model of the embodiment of the present invention is used in combination with a semantic expression vector to extract image features of a predetermined image.

Further, the visual question answering model further comprises a feature fusion device and a classifier, the details of the feature fusion device and the classifier will be referred to the above-described embodiment, and the details will not be described again here.

For the visual question answering model of this example, the Visual Genome dataset of the above-described example was trained and verified in the P40 cluster described in the above example, and at the same time, the visual sense of the prior art using LSTM or Bi-LSTM as a text encoder. The question answering model was trained and validated, and the results are shown in Table 4.

From Table 4, the visual question-and-answer model of the embodiment of the present invention that employs the averaging pooling process as the text encoder is compared with the conventional visual question-and-answer model that employs LSTM or Bi-LSTM as the text encoder. It can be seen that the prediction accuracy is reduced by about 0.4%, but the model execution time is shortened by up to 2.4 hours, and the training efficiency is greatly improved.

In the embodiment of the present invention, the visual question answering model encodes a text vector by an averaging pooling process to achieve the purpose of simplifying the visual question answering model, and a simple encoding method called averaging pooling process. As a result, it is possible to effectively improve the training efficiency of the visual question answering model without significantly reducing the prediction accuracy of the visual question answering model, which is useful for engineering use.

Example 3
FIG. 3 is a schematic configuration diagram of an electronic device according to a third embodiment of the present invention. FIG. 3 shows a block diagram of an exemplary electronic device 12 suitable for realizing the embodiment of the present invention. The electronic device 12 shown in FIG. 3 is merely an example and should not limit the functionality and scope of use of the embodiments of the present application.

As shown in FIG. 3, the electronic device 12 is shown in the form of a general-purpose computing device. The components of the electronic device 12 include one or more processors or processors 16, a memory 28, and a bus 18 connecting components of different systems (including the memory 28 and the processor 126). Not limited.

Bus 18 represents one or more of several types of bus structures, one of which is a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a variety of bus structures. Includes the local bus to use. For example, these architectures include Industry Standard Architecture (ISA) Bus, Micro Channel Architecture (MCA) Bus, Extended ISA Bus, Video Electronics Standards Association (VESA) Local Bus, and Peripheral Component Interconnect (PCI) Bus. Not limited to these.

The electronic device 12 usually includes a plurality of types of computer system readable media. These media may be any usable medium accessible by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.

The memory 28 may include computer system readable media in the form of volatile memory such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 can be used to read and write to a non-removable, non-volatile magnetic medium (not shown in FIG. 3, usually referred to as a "hard disk driver"). Although not shown in FIG. 3, a magnetic disk driver for reading and writing to a removable non-volatile magnetic disk (eg, "floppy disk") and a removable non-volatile optical disk (eg, CD-ROM, DVD). -An optical disk driver for reading and writing to (ROM or other optical medium) can be provided. In these cases, each driver can be connected to bus 18 via one or more data media interfaces. The memory 28 may include at least one program product having a set (eg, at least one) of program modules configured to perform the functions described in each embodiment of the present disclosure.

A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28, such program modules 42 as operating systems, one or more application programs, etc. Includes, but is not limited to, program modules and program data. Each or some combination of these examples may include the realization of a networking environment. Program module 42 typically performs the functions and / or methods of the embodiments described in the present disclosure.

The electronic device 12 is one that can communicate with one or more external devices 200 (eg, keyboard, pointing device, display 24, etc.) and also allows the user to interact with the electronic device 12. Or any device that can communicate with multiple devices and / or allow the electronic device 12 to communicate with one or more other computing devices (eg, network cards, modems, etc.). You can also communicate with. Such communication can be done via the input / output (I / O) interface 22. The electronic device 12 also communicates with one or more networks (eg, a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 20. be able to. As shown in the figure, the network adapter 20 communicates with other modules of the electronic device 12 via the bus 18. Other hardware and other hardware not shown in the figure, including, but not limited to, microcodes, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drivers, and data backup storage systems. / Or the software module can be used in combination with the electronic device 12.

The processor 16 executes various functional applications and data processing by executing a program stored in the memory 28, realizes, for example, a visual question answering model according to the above-described embodiment, and obtains the visual question. The response model pools the word vector sequence of the input question text and extracts the image features of the predetermined image in combination with the text encoder for extracting the semantic expression vector of the question text and the semantic expression vector. It is equipped with an image encoder for.

Example 4
Example 4 of the present invention provides a computer-readable storage medium, which stores a visual question-and-answer model according to an embodiment of the present invention and is executed by a computer processor. The visual question answering model pools a word vector sequence of input question text and combines it with a text encoder for extracting a semantic expression vector of the question text and an image of a predetermined image. It includes an image encoder for extracting features.

Of course, the computer-readable storage medium provided in the embodiments of the present invention can also carry out the visual question answering model provided in any of the embodiments of the present invention.

As the computer storage medium of the embodiment of the present invention, any combination of one or more computer-readable media can be used. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, but is not limited to, for example, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any combination of the above. More specific examples (non-exhaustive lists) of computer-readable storage media are electrical connections with one or more leads, portable computer disks, hard disks, random access memory (RAM), and read-only memory. Includes (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the above. .. As used herein, a computer-readable storage medium can be any tangible medium that can contain or store programs that can be used by, or combined with, instruction execution systems, devices, or devices.

Computer-readable signal media can include data signals propagating in the baseband or as part of a carrier wave, including computer-readable program code. The propagating data signal can adopt a variety of formats, including but not limited to electromagnetic signals, optical signals or any suitable combination of the above. Further, the computer-readable signal medium may be any computer-readable medium other than the computer-readable storage medium, which is used by the instruction execution system, device or device. Alternatively, a program used in combination with them can be transmitted, propagated or transmitted.

The program code contained in the computer-readable medium can be transmitted by any suitable medium including, but not limited to, wireless, wired, optical cable, RF, etc., or any suitable combination described above.

Computer programming code for performing the operations of the present invention can be created in one or more programming languages or a combination thereof, the programming languages being project-oriented such as Java®, Smalltalk, C ++. Includes programming languages, further including traditional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computer, partially on the user computer, as a stand-alone software package, or partially on the user computer. It may run on a remote computer, or it may run entirely on a remote computer or server. For remote computers, the remote computer can connect to the user's computer or to an external computer (eg, the Internet) over any type of network, including local area networks (LANs) or wide area networks (WANs). Connect via the Internet using a service provider).

It should be noted that the above are merely preferred embodiments of the present invention and technical principles applied thereto. Those skilled in the art will appreciate that the invention is not limited to the particular embodiments described herein and that various modifications, readjustments, and replacements can be made without departing from the scope of the invention. it can. Therefore, although the present invention has been described in detail with reference to the above examples, the present invention is not limited to the above examples, and equivalent examples can be included without departing from the spirit of the present invention. The scope of the present invention is determined by the scope of claims.

Claims (6)

A text encoder for pooling the word vector sequence of the input question text to extract the semantic representation vector of the question text,
An image encoder for extracting image features of a predetermined image in combination with the semantic expression vector,
A visual question answering model with. Specifically, the text encoder
The model according to claim 1, wherein a semantic expression vector of the question text is extracted by maximizing pooling or averaging the word vector sequence of the input question text. The model according to claim 2, wherein the maximized pooling process is represented by the following equation.
f (w1, w2, ..., wk) = max ([w1, w2, ..., wk], dim = 1)
Where f represents the maximized pooling processing function, k is the number of word vectors contained in the question text, and wi is obtained by processing the question text using a pre-trained word vector model. Is the i-th word vector obtained, i is a natural number in [1, k], and max ([w1, w2, ..., wk], dim = 1) is the word vector w1, w2, .. ., Represents the maximum value of the dimension corresponding to each word vector in wk. The model according to claim 2, wherein the average pooling process is represented by the following formula.

Where p represents the averaging pooling processing function, k is the number of word vectors contained in the question text, and wi is obtained by processing the question text using a pre-trained word vector model. Is the i-th word vector, i is a natural number in [1, k],

Represents the sum of the dimensional values corresponding to each word vector in the word vectors w1, w2, ..., Wk. With one or more processors
An electronic device comprising a memory for storing one or more programs.
When the one or more programs are executed by the one or more processors, the one or more processors execute the visual question-and-answer model according to any one of claims 1 to 4. Electronics. A computer-readable storage medium that stores computer programs
A computer-readable storage medium on which the visual question answering model according to any one of claims 1 to 4 is executed when the program is executed by a processor.

Images