JP7426919B2 - Program, device and method for estimating causal terms from images - Google Patents

Description

The present invention relates to a technique for estimating causal words consisting of a cause word and a result word for an image. This technology can be applied to applications in which a user viewing a video and a dialogue agent can have a natural dialogue.

Text-based systems are commonly used as interaction systems with users. The terminal functions as a user interface and transmits the user's speech to the dialogue system. The dialogue system estimates a response sentence that will be a natural dialogue for the uttered sentence, and sends the response sentence back to the terminal. Then, the terminal replies to the user with the response sentence in voice or text. Examples of such dialogue systems include "Siri (registered trademark)" and "Shabette Concierge (registered trademark)."

In recent years, there have been high expectations for dialogue system technology that responds to multimodal information (videos, images, captions, subtitles, audio, natural language text, etc.) surrounding the user. According to this technology, it is possible to estimate a natural response sentence to a user's utterance according to various multimodal information surrounding the user. In particular, by using AI (Artificial Intelligence), it is possible to have natural dialogues depending on the surrounding situation, such as TV programs, movies, and online videos, and it is expected that this will increase the user's desire for dialogue.

2. Description of the Related Art Conventionally, there is a technology for an interaction system in which a user and a robot interact based on the content of a video that the user is viewing (for example, see Non-Patent Document 1). According to this technology, by inputting a video with audio and subtitles, it is possible to respond to a user's question with a response text that corresponds to the video that the user is viewing.

There is also a technology that learns feature vectors of audio-accompanied video and subtitles and generates a response to the previous question (see, for example, Non-Patent Document 2). According to this technology, the dialogue system can perform fine tuning using the trained learning model GPT-2 (registered trademark) and improve the dialogue accuracy of response sentences according to multimodal information.

Furthermore, there is a technology that generates a response sentence that has a causal relationship with an uttered sentence and realizes a natural dialogue (for example, see Non-Patent Document 3). According to this technique, a dictionary of word pairs having a causal relationship is constructed in advance, and a response sentence having a causal relationship with an uttered sentence is preferentially selected (reranking response generation). Specifically, the words in the user's uttered sentence and the words in the response sentence are paired and compared against a word pair dictionary. When a match is found, it is determined that there is a causal relationship, and this response sentence is selected preferentially.
For example, it is possible to extract and learn causal words from the following sentences:
"As the yen has weakened, we can expect Japan's economy to improve from a trade perspective."
: Causal relationship word {(yen becomes weaker) -> (economy rises)}

Furthermore, there is also a technique for creating a learning model using only dialogue data (utterances and response sentences) that have causal relationships from a large-scale dialogue corpus (for example, see Non-Patent Document 4).

Hung Le, Doyen Sahoo, Nancy F. Chen, Steven C.H. Hoi, “Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systems” (2019), [online], [Retrieved October 11, 2020] , Internet <URL: https://arxiv.org/abs/1907.01166> Hung Le, Steven C.H. Hoi, “Video-Grounded Dialogues with Pretrained Generation Language Models” (2020), [online], [Retrieved October 11, 2020], Internet <URL: https://www.aclweb. org/anthology/2020.acl-main.518/＞ Shohei Tanaka, Koichiro Yoshino, Katsuhito Sudoh, and Satoshi Nakamura. "Conversational Response Re-ranking Based on Event Causality and Role Factored Tensor Event Embedding" 1st Workshop NLP for Conversational AI, ACL 2019 Workshop (ConvAI). Shota Sato, Kentaro Inui. “Chat response learning using data sampling based on causal relationships” Proceedings of the 24th Annual Conference on Language Processing (March 2018) “Explanation of Transformer paper, a major premise of the deep learning world!”, [online], [Retrieved October 11, 2020], Internet <URL: https://qiita.com/omiita/items/07e69aef6c156d23c538>

However, according to the technologies described in Non-Patent Documents 1 and 2 mentioned above, although multimodal information such as video with audio and subtitles is used, a response sentence is generated according to the user's previous utterance. It's just that. Therefore, it is difficult to generate natural dialogue, such as small talk, on open domain topics other than utterances and response sentences. This is, after all, nothing more than a relationship with the answer (response) to the user's previous question (utterance).

Furthermore, according to the technique described in the above-mentioned Non-Patent Document 3, during learning, only word pairs are compared, and therefore no consideration is given to the feature amounts of the context other than the word pairs. According to the causal relationship term in the above example {(the yen weakens) -> (the economy rises)}, there are no restrictive features such as "trade perspective" or "Japan" included. becomes. In addition, during operation, there is a problem that reranking cannot be achieved unless pre-learned causal relation words are completely matched against sentences uttered by an actual user.

Furthermore, according to the technology described in Non-Patent Document 4, since a learning model is created using only dialogue data (utterances and response sentences) that have a causal relationship, the learning model is too dependent on dialogue data as training data. Put it away. This also has the problem of lack of diversity and versatility as responses to user utterances in the training data.

On the other hand, the inventors of the present application believe that it is possible to extract the linguistic features of a video by using videos with subtitles as multimodal information and learning to associate subtitles with videos. I wondered if there was. Based on this, we thought that when multiple response sentence candidates can be estimated for a user's utterance, it would be possible to select a response sentence that corresponds to the language characteristics of the video.
To achieve this, at least if it is possible to infer the causal words (cause and result words) from the image (frame in the video), it is possible to respond with a response sentence that corresponds to the causal word. I thought that might be the case.

SUMMARY OF THE INVENTION Therefore, an object of the present invention is to provide a program, a device, and a method that can estimate causal words (cause and result words) from images. The purpose is to make the interaction with the user as natural as possible by estimating causal relation words from multimodal information around the user and responding to the user's utterances with a response sentence according to the causal relation.

According to the present invention, there is provided a program that causes a computer to function to estimate causal relation words of a cause word and a result word from an image,
As training data, images are associated with subtitles associated with the images.
Regarding the training stage,
a causal relationship learning engine that inputs feature vectors of subtitle sentences and estimates cause and effect words from subtitle sentences that are estimated to have a causal relationship;
Function as an image learning engine that inputs image feature vectors and trains to output cause and effect words estimated by the causal relationship learning engine,
Regarding the estimation stage,
The image learning engine is characterized by inputting images as target data and causing a computer to function so as to output cause words and result words.

According to another embodiment of the program of the present invention,
Regarding the training stage,
The image learning engine is composed of a generative adversarial network,
a generator inputting a feature vector of an image;
It is also preferable that the computer be trained as a discriminator that receives the cause and effect words output from the generator and the cause and effect words output from the causality estimation means.

According to another embodiment of the program of the present invention,
The generator is based on Transformer,
Preferably, the computer functions such that the discriminator is based on a convolutional neural network of the classification type.

According to another embodiment of the program of the present invention,
The causal learning engine is
During training,
Conjunctive particles that connect sentences before and after sentences in a causal relationship are registered in advance, and a subtitle sentence selection means that inputs subtitle sentences from teacher data and selects subtitle sentences that include conjunctive particles, and sends the selected subtitle sentences to the input layer. It is also preferable for the computer to function as a causal relationship word estimating means that is trained as a multi-task deep learning model so that the cause word is input, the cause word is output from the first output layer, and the result word is output from the second output layer.

According to another embodiment of the program of the present invention,
The causal relationship term estimation means is
an input layer;
an embedded layer;
a first recursive network layer, a first identification layer, and a first output layer branched from the embedding layer;
It is also preferable that the computer function as a second recursive network layer, a second identification layer, and a second output layer branched from the embedding layer.

According to another embodiment of the program of the present invention,
It is also preferred that the feature vectors act as if they were generated by a distributed representation generation algorithm.

According to another embodiment of the program of the present invention,
During training,
The training data consists of a video and a series of dialogue history including multiple sets of utterances and response sentences between the people viewing the video,
multimodal information extraction means for extracting an image from the video for each set of uttered sentences and response sentences in the dialogue history;
Functions as a response sentence estimation engine that trains each set of utterance sentences and response sentences to input the utterance sentences to the encoder side and output the response sentences from the decoder side,
When estimating,
The response sentence estimation engine inputs the user's utterance, outputs multiple candidate response sentences,
It is also preferable that the computer function as a response sentence reranking means that selects response sentences that include or are similar to the result word output by the image learning engine from among a plurality of candidate response sentences.

According to another embodiment of the program of the present invention,
It is also preferable that the computer functions as the response sentence estimation engine using Seq2Seq, which is capable of extracting features between a general-purpose utterance sentence and a response sentence.

According to the present invention, there is provided an estimation device for estimating a causal relation word between a cause word and a result word from an image,
As training data, images are associated with subtitles associated with the images.
Regarding the training stage,
a causal relationship learning engine that inputs feature vectors of subtitle sentences and estimates cause and effect words from subtitle sentences that are estimated to have a causal relationship;
and an image learning engine that trains to input image feature vectors and output cause and effect words estimated by the causal relationship learning engine,
Regarding the estimation stage,
The image learning engine is characterized by inputting images as target data and outputting cause words and result words.

According to the present invention, there is provided an estimation method of a device for estimating causal relation words of a cause word and a result word from an image, comprising:
As training data, images are associated with subtitles associated with the images.
The device is
Regarding the training stage,
a causal relationship learning engine that inputs feature vectors of subtitle sentences and estimates cause and effect words from subtitle sentences that are estimated to have a causal relationship;
and an image learning engine that trains to input image feature vectors and output cause and effect words estimated by the causal relationship learning engine,
Regarding the estimation stage,
The image learning engine is characterized by inputting images as target data and outputting cause words and result words.

According to the program, device, and method of the present invention, causal words (cause word and result word) can be estimated from an image. Then, by estimating a causal relationship word from multimodal information around the user and responding to the user's utterance with a response sentence that corresponds to the causal relationship, it is possible to interact with the user as naturally as possible.

FIG. 2 is a functional configuration diagram of an estimation device during training for estimating causal relation words from images. It is an explanatory diagram of a subtitle sentence selection part. FIG. 3 is an explanatory diagram of a causal relationship word estimation unit. It is a functional block diagram of the estimation device at the time of estimation which estimates a causal relationship word from an image. FIG. 2 is a functional configuration diagram of an estimation device during training that estimates causal words from videos with subtitles as teacher data. FIG. 2 is a functional configuration diagram of an estimation device at the time of estimating a causal relationship word from a video. FIG. 3 is a functional configuration diagram of an estimation device during estimation that responds with a response sentence to an uttered sentence according to a video. 8 is an explanatory diagram showing the flow of text as a specific example in FIG. 7. FIG. FIG. 2 is an explanatory diagram of a response sentence estimation engine.

Hereinafter, embodiments of the present invention will be described in detail using the drawings.

According to the drawings, for the sake of explanation, they are classified as follows.
First, FIGS. 1 to 4 will explain an estimation device that estimates causal terms from images. 1 to 3 will be described as <during training>, and FIG. 4 will be described as <during estimation>.
Next, FIG. 5 and FIG. 6 will explain an estimation device that estimates a causal relation word from a video. FIG. 5 will be described as <during training>, and FIG. 6 will be described as <during estimation>.
Furthermore, FIGS. 7 to 9 explain an estimation device that responds with a response sentence to an uttered sentence according to a video.

<<During training> in the estimation device that estimates causal terms from images>
FIG. 1 is a functional configuration diagram of an estimation device during training that estimates causal relation words from images.

According to FIG. 1, the estimation device 1 includes an image feature vector generation unit 101, a language feature vector generation unit 102, a causal relationship learning engine 11, and an image learning engine 12 during training. These functional components are realized by executing a program that causes a computer installed in the device to function. Furthermore, the processing flow of these functional components can also be understood as a training method for the estimation device.
Furthermore, images and subtitle sentences are input to the estimation device 1 as teacher data. Teacher data must be input externally during training.

[Image feature vector generation unit 101]
The image feature vector generation unit 101 receives an image and generates an image feature vector (random vector in latent space) from the image. The image feature vector is output to the image learning engine 12.
Specifically, the image feature vector is replaced with a high-dimensional vector by applying a distributed representation generation (embedding) algorithm such as VisualBERT (registered trademark).

[Language feature vector generation unit 102 (for subtitle text)]
The language feature vector generation unit 102 inputs a subtitle sentence associated with an image, analyzes it into morphemes through morpheme analysis, and generates a language feature vector (random vector in latent space) for each morpheme. The linguistic feature vector is output to the causal relationship learning engine 11.
Specifically, the language feature vector is also replaced with a high-dimensional vector by applying a distributed representation generation algorithm such as BERT (registered trademark) or GPT-2 (registered trademark).

BERT (Bidirectional Encoder Representations from Transformers) is an encoded representation of bidirectional learning using the Transformer architecture (for example, see Non-Patent Document 3), and is a natural language processing model of Google (registered trademark). For videos and images, there are VideoBERT and VisualBERT. BERT is a Seq2seq-based pre-learning model that learns by processing unlabeled feature vectors (distributed representations) with a Transformer. It not only predicts the next word in a sequence of sentences, but also bidirectionally predicts words that are masked from the surrounding context. In this way, context information corresponding to the word is learned.
Additionally, GPT-2 (Generative Pre-Training 2) is based on Open AI and uses pixels to learn instead of natural language. As a result, it is possible to predict the second half of the video (or the entire image) that humans would intuitively think of from the first half of the video (or some images) sequence.

[Causal relationship learning engine 11]
The causal relationship learning engine 11 inputs feature vectors of subtitle sentences as training data from the linguistic feature vector generation unit 102, and estimates cause words and result words (causal relationship words) from the subtitle sentences estimated to have a causal relationship. do.
According to FIG. 1, the causal relationship learning engine 11 includes a subtitle sentence selection section 111 and a causal relationship word estimation section 112.

[Subtitle sentence selection unit 111]
The subtitle sentence selection unit 111 has registered in advance conjunctive particles that connect the sentences before and after the sentences in a causal relationship. Then, the subtitle sentence selection unit 111 inputs the subtitle sentences of the teacher data and selects subtitle sentences that include conjunctive particles.

FIG. 2 is an explanatory diagram of the subtitle sentence selection section.

According to FIG. 2, the subtitle sentence selection unit 111 is a classification type neural network, which is trained in advance using corpus data. Here, the process is divided into the time of training the corpus data and the time of estimating the subtitle sentence classification determination.

(When training on corpus data)
The subtitle sentence selection unit 111 inputs corpus data and constructs a deep learning model that comprehensively extracts expression features of the entire sentence that have a causal relationship.
Corpus data is a large-scale "corpus" that is a large-scale collection of structured natural language sentences on the Internet. This is, for example, an encyclopedia such as Wikipedia (registered trademark), and is a group of sentences that are valid as natural language. Of course, it may also be a group of sentences of natural language knowledge content on a website.

From a group of sentences included in a large-scale corpus, training sentences containing "conjunctive particles" registered in the conjunctive particle table are selected. Sentences that include a conjunctive particle often include a cause word and a result word that are in a causal relationship with the conjunctive particle in between.
The conjunctive particle table is a register of conjunctive particles that connect the preceding and following sentences in a causal relationship. A "conjunctive particle" is a particle that establishes a causal relationship between a preceding sentence and a subsequent sentence, and serves as a clue to the causal relationship.
For example, there are particles such as:
"for,""from,""by,""by,"
"In the background of...""In response to...""As a result of...""In the wake of..."
"The effect of...""The cause of...""If you do...""If you do..."
"If we don't...""In accordance with...""Reflecting..."

It is assumed that the subtitle sentence selection unit 111 receives, for example, the following corpus data.
Corpus data: “I cut my hand and it bled.”
This corpus data is determined to include "tame" as a conjunctive particle, and the following conjunctive particle is deleted and concatenated to create the following sentence.
Causal words: “I cut my hand, there was blood”
After the causal relationship is established, data preprocessing, convolution layer, pooling layer, fully connected layer, and discrimination layer are used to train the data as having a causal relationship.

(When estimating the classification judgment of subtitle sentences)
When the subtitle sentence selection unit 111 receives a subtitle sentence with unknown causal relationship, it classifies it into morphemes through morphological analysis and outputs a classification result indicating whether there is a causal relationship.
It is assumed that the subtitle sentence selection unit 111 receives subtitle sentences with the following feature vectors from the language feature vector generation unit 102.
Subtitle: "I cut my hand and it bled"
This subtitle sentence is determined to have a "causal relationship" by data preprocessing, convolution layer, pooling layer, fully connected layer, and discrimination layer.
Then, the subtitle sentence selection unit 111 outputs only subtitle sentences estimated to have a causal relationship to the causal relationship word estimation unit 112.
Subtitle: "I cut my hand and it bled"

[Causal relation word estimation unit 112]
The causal relation word estimation unit 112 inputs the selected subtitle sentences to the input layer, and uses a multi-task deep learning model so that the cause words are output from the first output layer and the result words are output from the second output layer. Learn as.
The causal relationship word estimating unit 112 inputs only subtitle sentences estimated to have a causal relationship from the subtitle sentence selection unit 111, and outputs each of the cause word and result word (causal relationship word) in the subtitle sentence.

FIG. 3 is an explanatory diagram of the causal relationship word estimation unit.

According to FIG. 3, the causal relationship word estimating unit 112 is configured to branch into two series as follows.
{(Cause word)->(Result word)}
-> 1st recursive network layer -> 1st discrimination layer -> Cause word output layer
-> 2nd recursive network layer -> 2nd identification layer -> Result word output layer

The first recurrent network layer and the second recurrent network layer are the same RNN (Recurrent Neural Network). RNN is a model that can learn time dependence by inputting time-series data of learning sentences as they are. As the RNN, for example, LSTM (Long Short-Term Memory) or GRU (Gated Recurrent Unit) can be used. LSTM consists of multiple blocks lined up, each block consisting of a cell that keeps errors inside to prevent gradient disappearance, and input gates, output gates, and forgetting gates that retain and erase necessary information at the necessary timing. has been done. GRU is a simplified version of LSTM and consists of a reset gate and an update gate.
The first discrimination layer and the second discrimination layer are the same discriminator, the first discrimination layer discriminating cause words, and the second discrimination layer discriminating result words.
Finally, the cause word output layer outputs the cause word, and the result word output layer outputs the result word.

[Image learning engine 12]
The image learning engine 12 is trained to input image feature vectors and output the cause and effect words estimated by the causal relationship learning engine 11.
The image learning engine 12 is configured by a generative adversarial network such as a GAN (Generative adversarial network), and includes a generator and a discriminator.
The generator 121 inputs a feature vector of an image and outputs a cause word and a result word. The generator is based on Transformer.
The discriminator 122 inputs the cause word and result word output from the generator 121 and the cause word and result word output from the causal relation word estimation unit 112. The classifier is based on a classification type convolutional neural network. The discriminator 122 predicts and outputs whether the cause word and result word output from the generator 121 are genuine or fake. The prediction result of the discriminator 122 is fed back to the generator 121. As a result, the generator 121 learns to generate fake cause words and result words that the classifier 122 makes mistakes, and the classifier 122 learns to generate fake cause words and result words that the classifier 122 makes mistakes. I will continue to learn.

<<At the time of estimation> in the estimation device that estimates causal terms from images>
FIG. 4 is a functional configuration diagram of an estimation device for estimating a causal relation word from an image.

According to FIG. 4, it consists of an image feature vector generation unit 101 and a generator 121 of the image learning engine 12.
The image feature vector generation unit 101 inputs an arbitrary image and generates an image feature vector from the image. The image feature vector is output to the generator 121 of the image learning engine 12.
The generator 121 inputs an image of a feature vector and outputs a cause word and a result word (causal relationship word).
According to FIG. 4, the following causal relation words are output from an arbitrary image.
Causal words “cut your hand, bleed”

<Estimation device that estimates causality words from videos <During training>>
FIG. 5 is a functional configuration diagram of an estimation device during training that estimates causal words from videos with subtitles as teacher data.

The estimation device 1 realizes a natural dialogue with a user, and generates a response sentence to the user's utterance.
According to FIG. 5, an estimation device 1 having a server function communicates with a terminal 2 having a user interface function. As an input/output device for the user, the terminal 2 may acquire the user's voice through a microphone and speak to the user through a speaker, or may input text-based utterances from the user and display a response sentence. It may be something.
Note that the voice recognition function may be installed in the estimation device 1 or may be installed in the terminal 2.

According to FIG. 5, compared to FIG. 1, it further includes a multimodal information extraction unit 103, and inputs teacher data during training. Here, the teacher data is "video with subtitles" as a large amount of multimodal information recorded in the past.

[Multimodal information extraction unit 103]
The multimodal information extraction unit 103 has an image extraction function and a subtitle sentence extraction function for multimodal information.
The multimodal information extraction unit 103 extracts one sampling image from a series of videos of teacher data for each subtitle sentence. For example, for one subtitle sentence, an image that is one frame at that point in the video is extracted.
The image extracted from the video-attached subtitle text is output to the image feature vector generation unit 101.
Further, the subtitle sentence extracted from the video-attached subtitle sentence is output to the language feature vector generation unit 102.

<Estimation device for estimating causal relation words from video <Estimation time>>
FIG. 6 is a functional configuration diagram of an estimation device for estimating a causal relation word from a video.

According to FIG. 6, the estimation device 1 communicates with a terminal 2 serving as a user interface function. The terminal 2 is equipped with a device that can acquire multimodal information around the user. It is equipped with at least a camera (or a connection interface to a TV or display) that can capture the video the user is viewing. Such a terminal 2 may be a general smartphone or a robot such as "SOTA (registered trademark)" or "Unibo (registered trademark)". Alternatively, it may be a smart speaker such as "Google Home (registered trademark)" or "Amazon Echo (registered trademark)" equipped with a camera.

According to FIG. 6, compared to FIG. 4, it further includes a multimodal information extraction unit 103, and receives a video to be estimated from the terminal 2. This video is input to the multimodal information extraction unit 103, and an image is extracted. The image is input to the image feature vector generation unit 101 in the same manner as in FIG. 4 described above.

That is, during the training shown in FIG. 5 described above, the subtitled video of the teacher data is processed, whereas during the estimation shown in FIG. 6, the estimation target image received in real time via the communication interface is processed.

<Estimation device that responds with a response sentence to the uttered sentence according to the video>
FIG. 7 is a functional configuration diagram of an estimation device at the time of estimation, which responds with a response sentence to an uttered sentence according to a video.

According to FIG. 7, compared to FIG. 6 described above, it further includes a language feature vector generation section 102, a response sentence estimation engine 13, a response sentence reranking section 14, and a language conversion section 15. These functional components are realized by executing a program that causes a computer installed in the device to function. Further, the processing flow of these functional components can also be understood as an estimation method of the estimation device.

According to FIG. 7, compared to FIG. 6, a video as multimodal information around the user and a sentence uttered by the user are received from the terminal 2, and a response sentence is sent back to the user.
The terminal 2 is equipped with a microphone that picks up the user's uttered voice and a speaker that outputs a response voice to the user. Furthermore, the terminal 2 may be equipped with a voice recognition function that converts the user's uttered voice into a text-based utterance, and a voice conversion function that converts a text-based response sentence into voice directed to the user. However, it may be installed in the estimation device 1. Of course, the uttered sentence and the response sentence are not limited to voice, and may be based on key input and display display.
Furthermore, the multimodal information transmitted from the terminal 2 may be information in which the user's uttered voice is mixed with the video. In that case, the multimodal information extraction unit 103 may separate and extract the video and the utterance.

FIG. 8 is an explanatory diagram showing the flow of text as a specific example in FIG.

According to FIG. 8, character X, who is a dialogue agent of estimation device 1, and user Y are having a dialogue. At this time, the video that user Y is viewing and the sentence uttered by user Y are transmitted from the terminal 2 to the estimation device 1. Furthermore, the estimation device 1 transmits a response sentence to which the character X should respond to the terminal 2. This allows the user Y to interact with the character X displayed on the terminal 2 through voice.

The video is multimodal information that is shared by user Y and character X. Furthermore, the dialogue between user Y and character X becomes a "set of utterances and response sentences" between the people who are viewing the video together.

[Language feature vector generation unit 102 (for utterances)]
The language feature vector generation unit 102 inputs a user's utterance, analyzes it into morphemes through morpheme analysis, and generates a language feature vector for each morpheme. Its function is similar to that in FIG. The language feature vector is output to the response sentence estimation engine 13.
According to FIG. 8, the following utterances are input to the language feature vector generation unit 102.
User Y's utterance: “I'm not good at it, so I'm definitely going to cut off.”

[Response sentence estimation engine 13]
During training, the response sentence estimation engine 13 is trained to input utterances to the encoder side and output response sentences from the decoder side for each dialogue corpus (a set of utterance sentences and response sentences) as training data. It is. Here, the characteristics of the relationship between the uttered sentence and the response sentence are learned in a general manner, regardless of the causal relationship.

FIG. 9 is an explanatory diagram of the response sentence estimation engine.

The response sentence estimation engine 13 may be a general-purpose Seq2Seq that can extract features between an uttered sentence and a response sentence, or may be an improved model such as seq2seq+attention or transform.
seq2seq is a neural network that inputs a morpheme string and learns replacement rules that output another morpheme string. In this way, multiple response sentences are learned for each uttered sentence. Of course, for example, LSTM (Long Short-Term Memory), which is a type of RNN (Recurrent Neural Network) capable of learning character string dependencies, may also be used.

As a result, when a user's utterance is input to the encoder during estimation, the response sentence estimation engine 13 outputs a plurality of candidate response sentences from the decoder. The plurality of candidate response sentences are output to the response sentence reranking unit 14.

According to FIG. 8, it is assumed that the response sentence estimation engine 13 receives the following uttered sentence from the response sentence reranking unit 14.
User Y's utterance: “I'm not good at it, so I'm definitely going to cut off.”
In response, the response sentence estimation engine 13 outputs the following plural response sentences.
Candidate response sentence 1: “It’s okay.”
Candidate response sentence 2: “Peel off the white skin too.”
Candidate response sentence 3: “I’m bleeding.”
Candidate response sentence 4: “Be careful.”
Candidate response sentence 5: “I’m not good at it.”

[Response sentence reranking unit 14]
The response sentence reranking unit 14 selects a response sentence that includes or is similar to the result word output by the image learning engine 12 from among the plurality of candidate response sentences output from the response sentence estimation engine 13. Since the words are converted into feature vectors, it is also possible to compare the degree of similarity.
The selected response sentence is output to the language conversion section 15.

According to FIG. 8, the response sentence reranking unit 14 selects the following response sentences from among a plurality of candidate response sentences by inputting causal relation words (cut your hand -> blood comes out) from the image learning engine 12. Select the response sentence.
Candidate response sentence 3: “I’m bleeding.”

[Language conversion unit 15]
The language conversion unit 15 has a function opposite to that of the language feature vector generation unit 102 described above, and converts the response sentence feature vector output from the response sentence reranking unit 14 into the text of the response sentence. The converted response sentence is sent to the terminal 2 via the communication interface.

As described above in detail, according to the program, device, and method of the present invention, causal words (cause word and result word) can be estimated from an image. Then, by estimating a causal relationship word from multimodal information around the user and responding to the user's utterance with a response sentence that corresponds to the causal relationship, it is possible to interact with the user as naturally as possible.
In particular, according to the present invention, by using a response sentence estimation engine that has learned the characteristics of the context relationship of uttered sentences and response sentences in a general and exhaustive manner, it is possible to select causal sentences from among a plurality of candidate response sentences. You can respond with a related response sentence.

Regarding the various embodiments of the present invention described above, various changes, modifications, and omissions within the scope of the technical idea and viewpoint of the present invention can be easily made by those skilled in the art. The above description is merely an example and is not intended to be limiting in any way. The invention is limited only by the claims and their equivalents.

1 Estimation device 101 Image feature vector generation unit 102 Linguistic feature vector generation unit 103 Multimodal information extraction unit 11 Causal relationship learning engine 111 Subtitle sentence selection unit 112 Causal relationship word estimation unit 12 Image learning engine 121 Generator 122 Discriminator 13 Response sentence Estimation engine 14 Response sentence reranking unit 15 Language conversion unit 2 Terminal

Claims (10)

A program that causes a computer to function to estimate causal relations between a cause word and a result word from an image, the program comprising:
As training data, images are associated with subtitles associated with the images.
Regarding the training stage,
a causal relationship learning engine that inputs feature vectors of subtitle sentences and estimates cause and effect words from subtitle sentences that are estimated to have a causal relationship;
Function as an image learning engine that inputs image feature vectors and trains to output cause and effect words estimated by the causal relationship learning engine,
Regarding the estimation stage,
The image learning engine is a program that operates a computer to input images as target data and output cause words and result words. Regarding the training stage,
The image learning engine is composed of a generative adversarial network,
a generator inputting a feature vector of an image;
Claim 1, characterized in that the computer is operated to train as a discriminator that receives cause and effect words output from the generator and cause and effect words output from the causal relationship estimation means. Programs listed. The generator is based on Transformer,
3. The program according to claim 2, characterized in that the discriminator causes the computer to function as if it were based on a classification type convolutional neural network. The causal learning engine is
During training,
Conjunctive particles that connect sentences before and after sentences in a causal relationship are registered in advance, and a subtitle sentence selection means that inputs subtitle sentences from teacher data and selects subtitle sentences that include conjunctive particles, and sends the selected subtitle sentences to the input layer. The computer functions as a causal relationship word estimating means that is trained as a multi-task deep learning model such that a cause word is inputted, a cause word is output from the first output layer, and a result word is output from the second output layer. The program according to any one of claims 1 to 3. The causal relationship term estimation means is
an input layer;
an embedded layer;
a first recursive network layer, a first identification layer, and a first output layer branched from the embedding layer;
5. The program according to claim 4, causing a computer to function as a second recursive network layer, a second identification layer, and a second output layer branched from the embedding layer. 6. The program according to claim 1, wherein the program causes the feature vector to function as if it were generated by a distributed representation generation algorithm. During training,
The training data consists of a video and a series of dialogue history including multiple sets of utterances and response sentences between the people viewing the video,
multimodal information extraction means for extracting an image from the video for each set of uttered sentences and response sentences in the dialogue history;
Functions as a response sentence estimation engine that trains each set of utterance sentences and response sentences to input the utterance sentences to the encoder side and output the response sentences from the decoder side,
When estimating,
The response sentence estimation engine inputs the user's utterance, outputs multiple candidate response sentences,
Claims 1 to 6, characterized in that the computer functions as a response sentence reranking means for selecting a response sentence that includes or is similar to the result word output by the image learning engine from among a plurality of candidate response sentences. The program described in any one of the above. 8. The program according to claim 7, wherein the response sentence estimation engine causes the computer to function as Seq2Seq capable of extracting features between a general-purpose utterance sentence and a response sentence. An estimation device for estimating causal relation words of a cause word and a result word from an image,
As training data, images are associated with subtitles associated with the images.
Regarding the training stage,
a causal relationship learning engine that inputs feature vectors of subtitle sentences and estimates cause and effect words from subtitle sentences that are estimated to have a causal relationship;
and an image learning engine that trains to input image feature vectors and output cause and effect words estimated by the causal relationship learning engine,
Regarding the estimation stage,
The image learning engine is an estimation device characterized in that it inputs an image as target data and outputs cause words and result words. An estimation method of a device for estimating causal relation words of a cause word and a result word from an image, the method comprising:
As training data, images are associated with subtitles associated with the images.
The device is
Regarding the training stage,
a causal relationship learning engine that inputs feature vectors of subtitle sentences and estimates cause and effect words from subtitle sentences that are estimated to have a causal relationship;
and an image learning engine that trains to input image feature vectors and output cause and effect words estimated by the causal relationship learning engine,
Regarding the estimation stage,
An estimation method for an apparatus characterized in that the image learning engine inputs an image as target data and outputs a cause word and a result word.

Images