JP2022135734A - Program, device, and method for interacting in small-talk style by using multi-modal knowledge graph - Google Patents

Abstract

To provide a device, a method, and a program for interacting in a small-talk style by using a multi-modal knowledge graph.SOLUTION: An interaction device has: an utterance feature vector generation unit that generates an utterance feature vector from an utterance sentence from teacher data during training between entity words associated with an entity object image by using a knowledge graph linked by relational words; a knowledge graph searching unit that detects one or more utterance entity words included in the utterance sentence from the teacher data and searches for the relational words and the entity words from the utterance entity words by using the knowledge graph; a knowledge feature vector generation unit that generates a knowledge feature vector from the entity words, the entity object image, and the relational words; a coupling layer that couples the utterance feature vector and the knowledge feature vector to generate a coupled utterance feature vector; a response feature vector generation unit that generates a response feature vector from a response sentence of the teacher data corresponding to the utterance sentence of the teacher data and a response object image; and an encoder-decoder that receives input of the coupled utterance feature vector and outputs the response feature vector.SELECTED DRAWING: Figure 1

Description

The present invention relates to technology of a dialogue agent that realizes natural dialogue with a user.

Text-based systems are generally used for interactive systems with users. The terminal functions as a user interface and transmits the user's spoken voice to the dialogue system. The dialogue system generates a response sentence that becomes a natural dialogue for the utterance sentence, and returns the response sentence to the terminal. Then, the terminal replies the response sentence to the user by voice or text. Examples of such dialogue systems include "Siri (registered trademark)" and "Talking Concier (registered trademark)."

In contrast, multimodal dialogue systems are expected. The dialog system is capable of interacting with a user in multiple modes of communication such as text, voice and images. In particular, chat dialogue systems using AI (Artificial Intelligence) are expected to be able to respond with natural response sentences in response to multimodal information, and to increase user motivation for dialogue.

There is also technology for dialogue systems that make use of knowledge graphs in order to realize exchanges of dialogues that include a wealth of knowledge. A "knowledge graph" is a graph created by describing the relationships between entities. That is, it is created by setting the entity words as "nodes" and the related words between the entity words as "links".

Conventionally, there is a technique of automatically generating a response sentence in a text-based dialogue in consideration of the concept transition of a knowledge graph (see Non-Patent Document 1, for example).
There is also a technique for constructing a knowledge graph using multi-domain topics (movies, music, travel) (see Non-Patent Document 2, for example). According to this technique, a conversational dialogue corpus KdConv is used to generate a response sentence that integrates knowledge.
Furthermore, there is also a technique for generating dialogue response sentences using a knowledge graph for a specific task (see, for example, Non-Patent Document 3). According to this technology, the service center of the online mall generates response sentences for multimodal dialogue between the user and the sales operator using a product knowledge graph consisting of dialogue sentences and product photographs.
Furthermore, there is also a technique of generating a main concept from a user's utterance sentence, referring to both a task knowledge base and a general knowledge base, and generating a response sentence (see Patent Document 1, for example).

JP 2017-224204 A

Houyu Zhang, Zhenghao Liu, Chenyan Xiong, Zhiyuan Liu, “Grounded conversation Generation as Guided Traverses in Commonsense Knowledge Graphs” (2020), [online], [searched on February 21, 2021], Internet <URL: https: //arxiv.org/pdf/1911.02707.pdf> Hao Zhou, Chujie Zheng, Kaili Huang, Minlie Huang, Xiaoyan Zhu, “KdConv: A Chinese Multi-domain Dialogue Dataset Towards Multi-turn Knowledge-driven Conversation” (2020), [online], [February 21, 2021 day search], Internet <URL: https://www.aclweb.org/anthology/2020.acl-main.635.pdf> Lizi Liao, Yunshan Ma, Xiangnan He, Richang Hong, Tat-Seng Chua, “Knowledge-aware Multimodal Dialogue Systems” (2020), [online], [searched on February 21, 2021], Internet <URL: https ://nextcenter.org/wp-content/uploads/2020/04/Knowledge-Aware.pdf＞

According to the technique described in Non-Patent Document 1, the concept transition of the knowledge graph is connected with limited vocabularies such as book, bag, hope, based, and future. For this reason, related topics (related explanations) as knowledge are not described, and even if such a knowledge graph is applied, it is not possible to have a casual conversation involving knowledge.
According to the technology described in Non-Patent Document 2, since the knowledge graph is limited to a text base, it is not possible to have a multimodal chat dialogue including images.
According to the technique described in Non-Patent Document 3, a knowledge graph is applied to a predetermined task such as sales of goods and services, and multimodal chat dialogue based on abundant knowledge cannot be conducted.
The technique described in Patent Document 1 is a rule-based response generation method, and does not automatically generate response sentences from a large amount of teacher data. Moreover, both the task knowledge base and the general knowledge base consist only of words such as soda, code, tea, hot, and soup, and do not even describe related topics.

On the other hand, the inventors of the present application have considered whether it is possible to have a multimodal chat dialogue including images by constructing a knowledge graph including related topics and related images.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide a program, apparatus, and method for casually interacting using a multimodal knowledge graph.

According to the present invention, in a program that causes a computer to interact with a user,
As teacher data,
a dialogue history including a plurality of sets of utterance sentences, response sentences, and response target images;
Using a multimodal knowledge graph in which entity words associated with entity target images are linked by relational words,
during training,
an utterance feature vector generation means for generating an utterance feature vector from an utterance sentence of teacher data;
knowledge graph search means for detecting one or more utterance entity words contained in an utterance sentence of training data and searching for entity words linked from the utterance entity words by related terms using a knowledge graph;
knowledge feature vector generation means for generating a knowledge feature vector from the entity word, the entity target image, and the related term;
a connection layer that combines the utterance feature vector and the knowledge feature vector to generate a combined utterance feature vector;
response feature vector generation means for generating a response feature vector from a response sentence of teacher data corresponding to an utterance sentence of teacher data and a response target image;
It is characterized by having a computer act as an encoder-decoder trained to input a combined utterance feature vector and output a response feature vector.

According to another embodiment of the program of the present invention,
It is also preferable that the knowledge graph accumulating means searches for images from a search site using the entity words and related words in the knowledge graph as keys, and causes the computer to function such that the retrieved images are associated with the entity words. .

According to another embodiment of the program of the present invention,
As the dialogue
Enter the utterance sentence that will be the target data,
Utterance feature vector generation means generates an utterance feature vector from the utterance sentence of the target data,
The knowledge graph search means detects one or more utterance entity words included in the utterance sentence of the target data, and uses the knowledge graph to retrieve entity words and entity target images linked from the utterance entity words by related words,
knowledge feature vector generating means generates a knowledge feature vector from the entity words, entity target images, and related words retrieved by the knowledge graph retrieval means;
an encoder decoder inputs a combined utterance feature vector and outputs a response feature vector;
It is also preferable that the response feature vector generating means causes the computer to function so as to input the response feature vector and output the response sentence and the response target image.

According to another embodiment of the program of the present invention,
It is also preferable to make the computer function such that the utterance target image is associated with the utterance sentence.

According to another embodiment of the program of the present invention,
Preferably, the knowledge graph search means causes the computer to search for entity words and entity target images linked by relation terms from the utterance entity word at one or more predetermined hop counts using the knowledge graph.

According to another embodiment of the program of the present invention,
The encoder-decoder may cause the computer to train so as to minimize the loss between the response feature vector output from the encoder-decoder and the response feature vector generated by the response sentence feature vector generating means. preferable.

According to another embodiment of the program of the present invention,
It is also preferable that the knowledge feature vector generating means causes the computer to function as if it were a GNN (Graph Neural Network).

According to the present invention, in an interactive device that interacts with a user,
As teacher data,
a dialogue history including a plurality of sets of utterance sentences, response sentences, and response target images;
Using a multimodal knowledge graph in which entity words associated with entity target images are linked by relational words,
during training,
an utterance feature vector generation means for generating an utterance feature vector from an utterance sentence of training data;
knowledge graph search means for detecting one or more utterance entity words contained in an utterance sentence of training data and searching for entity words linked from the utterance entity words by related terms using a knowledge graph;
knowledge feature vector generation means for generating a knowledge feature vector from the entity word, the entity target image, and the related term;
a connection layer that combines the utterance feature vector and the knowledge feature vector to generate a combined utterance feature vector;
response feature vector generation means for generating a response feature vector from a response sentence of teacher data corresponding to an utterance sentence of teacher data and a response target image;
an encoder-decoder trained to input the combined utterance feature vector and output a response feature vector.

According to the present invention, in an interaction method for a device that interacts with a user,
As teacher data,
a dialogue history including a plurality of sets of utterance sentences, response sentences, and response target images;
Using a multimodal knowledge graph in which entity words associated with entity target images are linked by relational words,
During training, the device
a first step of generating an utterance feature vector from an utterance sentence of teacher data;
a second step of detecting one or more utterance entity words contained in an utterance sentence of training data, and searching for entity words linked by related terms from the utterance entity words using a knowledge graph;
a third step of generating a knowledge feature vector from the entity words, entity target images and related terms;
a fourth step of combining the utterance feature vector and the knowledge feature vector to generate a combined utterance feature vector;
a fifth step of generating a response feature vector from the response target image and the response sentence of the teacher data corresponding to the utterance sentence of the teacher data;
and a sixth step of training the encoder-decoder to input the combined utterance feature vector and output the response feature vector.

According to the program, device, and method of the present invention, it is possible to have a casual conversation using a multimodal knowledge graph.

FIG. 4 is a functional configuration diagram during training in the dialogue device of the present invention; FIG. 10 is an explanatory diagram showing a dialogue history as teacher data during training; FIG. 10 is a first explanatory diagram showing a knowledge graph as teacher data during training; FIG. 11 is a second explanatory diagram showing a knowledge graph as teacher data during training; FIG. 4 is an explanatory diagram showing feature vector training in the dialog apparatus of the present invention; FIG. 3 is a functional configuration diagram during dialogue in the dialogue device of the present invention; FIG. 4 is an explanatory diagram showing a first dialogue example in the present invention; FIG. 10 is an explanatory diagram showing a second dialogue example in the present invention;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a functional configuration diagram during training in the interactive device of the present invention.

According to FIG. 1, the dialogue apparatus 1 can realize natural dialogue such as chatting with the user using a multimodal knowledge graph. The dialogue device 1 is equipped with a plurality of machine learning engines, and is divided into <during training> and <during dialogue>. Further, the interactive device 1 builds a learning model using <teacher data> during training of the machine learning engine.

The dialogue device 1 of the present invention is to cross-modally train both text and images using a distributed representation generator (embedder) and an encoder-decoder (encoder-decoder) in a deep learning model. This makes it possible to generate a multimodal response sentence and response target image for the utterance sentence (and utterance target image).

<Teacher data>
According to FIG. 1, the dialogue device 1 has a dialogue history accumulation unit 100 and a knowledge graph accumulation unit 101 as teacher data.

[Dialogue history accumulation unit 100]
The dialogue history accumulation unit 100 accumulates, as teacher data, "dialogue histories" including at least a plurality of sets of "utterance sentences", "response sentences", and "response target images". Here, the utterance sentence may also be associated with the "utterance target image". That is, the dialogue history consists of multimodal information in which images are also exchanged between users.
A dialogue history is a series of dialogue sentences that have been exchanged between users in large quantities in the past. According to the present invention, at least a response sentence may be associated with a "response target image", and an utterance sentence may be associated with an "utterance target image".
Of course, the dialogue history accumulation unit 100 does not need to be stored by the dialogue apparatus 1 itself, but must be input from the outside during training.

FIG. 2 is an explanatory diagram showing a dialogue history as teacher data during training.

According to FIG. 2, users A and B are exchanging dialogue sentences using images. The image here may be an image clipped from the video being viewed during the dialogue, an image captured by a camera, or a quoted image searched on the Internet.
According to FIG. 2, the dialogue is as follows.
・・・・・・・・・・・・・・・・・・・・
User B: What kind of TV shows do you like?
User A: Do you like wild animals more than dogs and cats?
User B: Are you a lion? (lion image)
User A: No, it's cute to look like an elephant parent and child (image of an elephant parent and child)
・・・・・・・・・・・・・・・・・・・・
According to the present invention, a dialogue history, which is an exchange of multimodal information including not only text but also images, between users, is used as training data.

[Knowledge graph accumulation unit 101]
The knowledge graph accumulation unit 101 accumulates a multimodal “knowledge graph” in which entity words associated with entity target images are linked by relational terms. Also, the entity word may be associated with a sentence as a related topic.

As a general knowledge graph, there is a casual dialogue corpus called “KdConv” described in Non-Patent Document 2. However, KdConv does not associate images with entity words, and is not multimodal.
In contrast, the knowledge graph of the present invention is constructed as a multimodal one by associating images with entity words.

The knowledge graph accumulating unit 101 may retrieve images from a search site using entity words and related words in the knowledge graph as keys, and associate the retrieved images with the entity words. For example, an image obtained by searching a conversational dialogue corpus such as KdConv using an entity word and a related word as keys can be associated with the entity word.

Wikipedia (registered trademark), for example, can also be used as a large-scale knowledge graph in other embodiments. Search Wikipedia and link a part or abstract of the obtained text as a node.

FIG. 3 is a first explanatory diagram showing a knowledge graph as teacher data during training.
According to FIG. 3, a plurality of entity words linked by relational words are represented as seen from the entity word "elephant". "Elephant" is linked to entity words (including related topics) by the related words "total length", "feature", "origin", "creation", and "attribute".

FIG. 4 is a second explanatory diagram showing a knowledge graph as teacher data during training.
According to FIG. 4, a plurality of entity words linked by relational words are represented as seen from the entity word "the road leading to heaven". In "Road to Heaven", entity words (including related topics) are linked to each other by the related words "total length", "characteristics", "origin", "nearby sightseeing spots", and "location".

<during training>
According to FIG. 1, the dialogue device 1 includes an utterance feature vector generator 11, a knowledge graph searcher 12, a knowledge feature vector generator 13, a joint layer 14, a response feature vector generator 15, an encoder decoder 16 and These functional components are implemented by executing a program that causes a computer installed in the device to function. In addition, the flow of processing of these functional components can also be understood as a training method for the interactive device.

FIG. 5 is an explanatory diagram showing feature vector training in the dialog system of the present invention.

[Utterance Feature Vector Generation Unit 11]
The utterance feature vector generation unit 11 generates an utterance feature vector from an utterance sentence of teacher data. The utterance feature vector generation unit 11 receives an utterance sentence and an utterance target image added thereto, and generates an utterance feature vector from each of them. The generated utterance feature vector is input to the coupling layer 14 .

The utterance feature vector for the utterance sentence is specifically obtained by applying a distributed representation generation algorithm (embedding) such as BERT (registered trademark) or GPT-2 (registered trademark) and replacing it with a high-dimensional vector. Further, the utterance feature vector for the utterance target image is specifically obtained by applying VisualBERT (registered trademark).

According to the dialogue history of FIG. 2 described above, the utterance feature vector generation unit 11 generates the utterance sentence of user A in the dialogue history, "It's cute like an elephant parent and child, isn't it?" input and generate an utterance feature vector.

[Knowledge graph search unit 12]
The knowledge graph search unit 12 detects one or more utterance entity words included in an utterance sentence of teacher data, and uses the knowledge graph to retrieve entity words linked from the utterance entity words by related terms. The retrieved entity words, entity target images, and related terms are output to the knowledge feature vector generation unit 13 .
Here, it is also possible to retrieve only the entity word linked from the uttered entity word by one hop (predetermined number of hops) of the related word. Other entity words within one hop can be said to be knowledge that is highly related to the utterance entity word.

According to FIG. 2 described above, the knowledge graph search unit 12 generates the utterance entity word "elephant" from the utterance sentence of the user A in the dialogue history, "It's cute that it looks like an elephant parent and child." Then, the knowledge graph search unit 12 uses the knowledge graph storage unit 101 to search for other entity words linked from the utterance entity word "elephant" by the related words. At this time, only entity words linked by relational words in one hop from the uttered entity word are retrieved.

[Knowledge Feature Vector Generation Unit 13]
The knowledge feature vector generation unit 13 generates a knowledge feature vector from the entity word, entity target image, and related term searched by the knowledge graph search unit 12 . A knowledge feature vector is input to the coupling layer 14 .

The knowledge feature vector generator 13 may be a GNN (Graph Neural Network).
A CNN (Convolutional Neural Network), for example, convolves information from eight directions, up, down, left, and right diagonally of an image. be.

[Binding layer 14]
A joint layer 14 combines the utterance feature vector and the knowledge feature vector to generate a combined utterance feature vector. The generated combined utterance feature vector is input to the encoder side of encoder decoder 16 .

[Response feature vector generator 15]
The response feature vector generation unit 15 generates a response feature vector from the response sentence of the teacher data corresponding to the utterance sentence of the teacher data and the response target image.
The response feature vector generation unit 15 receives a response sentence and a response speech target image, and generates a response feature vector from each of them. The generated response feature vector is input to the decoder side of the encoder decoder 16 .
Similar to response feature vectors for response sentences, specifically, distributed representation generation algorithms (embedding) such as BERT (registered trademark) and GPT-2 (registered trademark) are applied and replaced with high-dimensional vectors. . Moreover, the response feature vector for the response target image is specifically obtained by applying VisualBERT (registered trademark).

BERT (Bidirectional Encoder Representations from Transformers) is an encoded representation of bidirectional learning by the Transformer architecture, and is a natural language processing model of Google (registered trademark). For images there is VisualBERT. BERT is a Seq2seq-based pre-trained model that is learned by processing unlabeled feature vectors (distributed representation) with Transformer. It not only predicts the next word in a sequence of sentences, but also predicts words that are masked from the surrounding context in both directions. This learns the contextual information corresponding to the word.
In addition, GPT-2 (Generative Pre-Training 2) is based on Open AI, and by learning with pixels instead of natural language, from the first half of the image (or part of the image) sequence, human The second half of the image (or the whole image) can be predicted as intuitively.

Here, the combined speech feature vector (speech feature vector and knowledge feature vector) and the response feature vector potentially contain "attention". The attention mechanism is designed so that natural language processing does not give excessive priority to natural sentences. As a result, words and phrases to be emphasized are specified, and appropriate natural language processing in the encoder/decoder 16 is enabled.

[Encoder decoder 16]
The encoder-decoder 16 is trained to input the combined utterance feature vector and output the response feature vector.
For the encoder decoder 16, the encoder inputs the utterance sentence (and the utterance target image) and the joint utterance feature vector based on the knowledge graph, and outputs the latent vector. On the one hand, the decoder inputs the latent vector output from the encoder and outputs the response feature vector.
At this time, the encoder/decoder 16 trains so that the loss between the response feature vector output from the encoder/decoder 16 and the response feature vector generated by the response sentence feature vector generator 15 is minimized.

The encoder decoder 16 may be Transformer based. As previously described, encoder-decoder 16 cross-modally trains unlabeled language-based feature vectors and image-based feature vectors. This means training to associate the utterance sentence (and utterance target image), the response sentence and response target image, and the knowledge graph.

<during dialogue>
FIG. 6 is a functional configuration diagram during dialogue in the dialogue device of the present invention.
According to FIG. 6, the functional configuration during dialogue in the dialog apparatus 1 is the same as the functional configuration during training described above with reference to FIG.

The dialogue device 1 further has a communication interface 102 , receives an utterance sentence (and an utterance target image) from the terminal 2 serving as a user interface, and transmits a response sentence and a response target image to the terminal 2 .
The communication interface 102 has a speech recognition function of user's uttered voice and a speech synthesis function of a response sentence to the user. The voice recognition function converts the user's uttered voice acquired by the microphone of the terminal 2 into a text-based uttered sentence. The speech synthesis function converts the generated response sentence into a speech signal. A history of a set of these utterance sentences and response sentences is a dialogue sentence history.
Note that the voice recognition function and voice synthesis function may be installed in the terminal 2 . In that case, it receives a text-based “utterance sentence” from the terminal 2 and transmits a “response sentence” to the terminal 2 .

During training in FIG. 1 described above, teacher data is processed, while during dialogue in FIG. 6, target data received in real time by communication interface 102 is processed.

The terminal 2 is equipped with a device capable of acquiring multimodal information from a user and displaying the multimodal information to the user. At least, it is equipped with a display for displaying an image to the user, a microphone capable of picking up a voice uttered by the user, and a camera capable of capturing an image that the user is viewing. Examples of such terminals 2 include robots (hereinafter referred to as "terminals") such as "SOTA (registered trademark)" and "Unibo (registered trademark)". It may also be a tablet such as "Google Home (registered trademark)" or "Amazon Echo (registered trademark)" equipped with a display, a microphone and a camera.

According to FIG. 6 , the utterance feature vector generation unit 11 inputs a user's utterance sentence (and an utterance target image) and outputs the utterance feature vector to the encoder/decoder 16 .
The utterance feature vector generation unit 11 generates an utterance feature vector from the utterance sentence of the target data, and inputs the utterance feature vector to the coupling layer 14 .
The knowledge graph search unit 12 detects one or more utterance entity words included in the utterance sentence of the target data, and uses the knowledge graph to retrieve entity words linked from the utterance entity words by related terms.
The knowledge feature vector generation unit 13 generates a knowledge feature vector from the entity words and related words searched by the knowledge graph search unit 12 and inputs the knowledge feature vector to the connection layer 14 .
The connection layer 14 combines the utterance feature vector and the knowledge feature vector to generate a combined utterance feature vector, and outputs the combined utterance feature vector to the encoder side of the encoder decoder 16 .
The encoder/decoder 16 receives the combined utterance feature vector and outputs the response feature vector to the response feature vector generator 15 .
The response feature vector generation unit 15 inputs a response feature vector, generates a response sentence and a response target image, and transmits them to the terminal 2 from the communication interface 102 .

FIG. 7 is an explanatory diagram showing a first dialogue example in the present invention.

According to FIG. 7, it is assumed that the user is chatting with the interactive device 1 while watching television. At this time, the interactive device 1 may recognize the television image that the user is watching.
For example, we have the following dialogue:
・・・・・・・・・・・・・・・・・・・・
S: I'm doing a wildlife program right now. (wildlife footage, e.g. television)
U: This elephant is cute.
S: Parent and child.
U: By the way, what is the origin of the elephant?
S: It is said that it is a hieroglyph in the shape of an elephant that also lived in ancient China. (picture of hieroglyphs)
U: Hmm~

According to FIG. 7, the interactive device 1 receives the utterance sentence "Origin of the elephant" from the user. Then, the dialogue device 1 generates the response sentence "The figure of an elephant that lived in ancient China" from the utterance feature vector generated from the utterance sentence and the knowledge feature vector for the knowledge graph including "elephant" and "origin". is a hieroglyph in the shape of '' and the response target image ``hieroglyph'' can be output. Hieroglyphs and their images are knowledge that cannot be obtained from the history of past conversations between users. Such knowledge can also be used for casual conversation.

FIG. 8 is an explanatory diagram showing a second dialogue example in the present invention.

According to FIG. 8, for example, it is assumed that the user is having a casual conversation with the dialogue device 1 while driving a car. At this time, it is assumed that the user's line of sight is captured by the camera of the terminal 2 and that image is transmitted to the dialogue device 1 as an image to be spoken.
For example, we have the following dialogue:
・・・・・・・・・・・・・・・・・・・・
U: Why is this road called "The Road to Heaven"? (image of line of sight)
S: Because it "seems to continue to the horizon." (image of the road)

According to FIG. 8, the interactive device 1 receives the utterance sentence "the road leading to heaven" from the user and the utterance target image. Then, the dialogue apparatus 1 uses the utterance feature vector generated from the utterance sentence and the utterance target image, and the knowledge feature vector for the knowledge graph including "the road leading to heaven" to generate the response sentence "It looks like it continues to the horizon." A response target image (a road leading to heaven) can be output. The origins and images of the road leading to heaven are knowledge that cannot be obtained from the history of conversations between users in the past. Such knowledge can also be used for casual conversation.

As described in detail above, according to the interactive program, apparatus, and method of the present invention, it is possible to have casual conversation using a multimodal knowledge graph.

According to Non-Patent Documents 1, 2, and 4 as conventional techniques, there is a problem that "a multimodal chat dialogue including images cannot be developed." In contrast, according to the present invention, by using a knowledge graph including images, not only text, but also multimodal chat dialogue can be realized.
In addition, according to Non-Patent Documents 1 and 3 as prior art, there is a problem that "it is limited to task-oriented dialogue". In contrast, according to the present invention, since response sentences and response target images are generated by a deep learning model, natural chat conversations can be realized without being limited to specific tasks.
Furthermore, according to Non-Patent Documents 1 and 4 as prior art, there is a problem that they are "limited to conceptual knowledge graphs." In contrast, according to the present invention, a knowledge graph based on topics/topics is constructed, and all information and images highly relevant to the topics/topics are stored in the knowledge graph. Using such knowledge graphs, we can expect to generate responses in multimodal conversations about topics. As a result, it is possible to realize chat conversations containing a wealth of knowledge.

In addition, as a result, for example, "user support and business contact can be made through chatty dialogue using a multimodal knowledge graph", so Goal 8 of the Sustainable Development Goals (SDGs) led by the United Nations Promote inclusive and sustainable economic growth, employment and decent work for all.”

For the various embodiments of the present invention described above, various changes, modifications and omissions within the spirit and scope of the present invention can be easily made by those skilled in the art. The foregoing description is exemplary only and is not intended to be limiting. The invention is to be limited only as limited by the claims and the equivalents thereof.

1 Dialogue Apparatus 100 Dialogue History Accumulator 101 Knowledge Graph Accumulator 102 Communication Interface 11 Utterance Feature Vector Generating Unit 12 Knowledge Graph Searching Unit 13 Knowledge Feature Vector Generating Unit 14 Connection Layer 15 Response Feature Vector Generating Unit 16 Encoder Decoder 2 Terminal

Claims (9)

In a program that causes a computer to interact with a user,
As teacher data,
a dialogue history including a plurality of sets of utterance sentences, response sentences, and response target images;
Using a multimodal knowledge graph in which entity words associated with entity target images are linked by relational words,
during training,
an utterance feature vector generation means for generating an utterance feature vector from an utterance sentence of training data;
knowledge graph search means for detecting one or more utterance entity words contained in an utterance sentence of training data and searching for entity words linked from the utterance entity words by related terms using a knowledge graph;
knowledge feature vector generation means for generating a knowledge feature vector from the entity word, the entity target image, and the related term;
a connection layer that combines the utterance feature vector and the knowledge feature vector to generate a combined utterance feature vector;
response feature vector generation means for generating a response feature vector from a response sentence of teacher data corresponding to an utterance sentence of teacher data and a response target image;
A program characterized by causing a computer to act as an encoder-decoder trained to input a combined speech feature vector and output a response feature vector. The knowledge graph accumulating means is characterized by retrieving images from a search site using entity words and related words in the knowledge graph as keys, and causing the computer to function as if the retrieved images were associated with the entity words. The program according to claim 1, wherein: As the dialogue
Enter the utterance sentence that will be the target data,
Utterance feature vector generation means generates an utterance feature vector from the utterance sentence of the target data,
The knowledge graph search means detects one or more utterance entity words included in the utterance sentence of the target data, and uses the knowledge graph to retrieve entity words and entity target images linked from the utterance entity words by related words,
knowledge feature vector generating means generates a knowledge feature vector from the entity words, entity target images, and related words retrieved by the knowledge graph retrieval means;
an encoder decoder inputs a combined utterance feature vector and outputs a response feature vector;
3. The program according to claim 1, wherein the response feature vector generating means inputs the response feature vector and causes the computer to output the response sentence and the response target image. 4. The program according to any one of claims 1 to 3, causing a computer to function such that an utterance target image is associated with an utterance sentence. The knowledge graph retrieving means uses the knowledge graph to make the computer function to retrieve the entity words and the entity target images linked by the relational words from the utterance entity word at one or more predetermined number of hops. A program according to any one of claims 1 to 4. The encoder-decoder causes the computer to train so as to minimize the loss between the response feature vector output from the encoder-decoder and the response feature vector generated by the response sentence feature vector generating means. 6. A program according to any one of claims 1 to 5. 7. The program according to any one of claims 1 to 6, wherein the knowledge feature vector generating means causes the computer to function as if it were a GNN (Graph Neural Network). In an interactive device that interacts with a user,
As teacher data,
a dialogue history including a plurality of sets of utterance sentences, response sentences, and response target images;
Using a multimodal knowledge graph in which entity words associated with entity target images are linked by relational words,
during training,
an utterance feature vector generation means for generating an utterance feature vector from an utterance sentence of teacher data;
knowledge graph search means for detecting one or more utterance entity words contained in an utterance sentence of training data and searching for entity words linked from the utterance entity words by related terms using a knowledge graph;
knowledge feature vector generation means for generating a knowledge feature vector from the entity word, the entity target image, and the related term;
a connection layer that combines the utterance feature vector and the knowledge feature vector to generate a combined utterance feature vector;
response feature vector generation means for generating a response feature vector from a response sentence of teacher data corresponding to an utterance sentence of teacher data and a response target image;
and an encoder-decoder trained to input a combined utterance feature vector and output a response feature vector. In a method for interacting with a device that interacts with a user,
As teacher data,
a dialogue history including a plurality of sets of utterance sentences, response sentences, and response target images;
Using a multimodal knowledge graph in which entity words associated with entity target images are linked by relational words,
During training, the device
a first step of generating an utterance feature vector from an utterance sentence of training data;
a second step of detecting one or more utterance entity words included in an utterance sentence of training data, and searching for entity words linked by related terms from the utterance entity words using a knowledge graph;
a third step of generating a knowledge feature vector from the entity words, entity target images and related terms;
a fourth step of combining the utterance feature vector and the knowledge feature vector to generate a combined utterance feature vector;
a fifth step of generating a response feature vector from the response target image and the response sentence of the teacher data corresponding to the utterance sentence of the teacher data;
and a sixth step of training an encoder-decoder to input a combined speech feature vector and output a response feature vector.

Images