JP7058588B2 - Conversation system and conversation program - Google Patents

Description

The present invention relates to conversation systems and conversation programs, and in particular relates to image-related utterances shared with users.

Conventionally, a method of extracting items related to a subject projected on an image as a keyword has been known. For example, Patent Document 1 discloses an information processing apparatus for acquiring a name and an explanation of a subject included in an image such as a photograph. In this information processing device, information related to a predetermined image is acquired as image-related information, and a keyword is generated based on this. Further, Patent Document 2 discloses an information retrieval device that extracts information related to an image from an image. In this information retrieval device, first, the feature data of the image data is extracted from the image data to be searched. Next, the database is searched using this feature data as a search key. This database stores the feature data of the image and the keywords associated with it. This identifies feature data that is the same as or similar to the search key, and extracts keywords associated with it.

Further, there is also known a method of generating a response sentence to a user's utterance by referring to a keyword related to a subject projected on an image captured by a camera. For example, Patent Document 3 discloses an information processing apparatus that generates a response sentence by referring to information related to an object that the user is looking at. Specifically, first, the user's utterance is acquired by the microphone, and the object seen by the robot is imaged by the CCD camera, which is the "eye". Next, the dictionary information of the words related to the imaged object is referred to, and speech recognition and language analysis are performed. Then, among the information stored in the knowledge database, the information related to the captured object is referred to, and the response sentence corresponding to the language analysis result (result of meaning understanding) is generated. As a result, it is possible to generate a response sentence to the user's utterance accurately and at high speed.

Japanese Unexamined Patent Publication No. 2013-101450 Japanese Unexamined Patent Publication No. 2002-297648 Japanese Unexamined Patent Publication No. 2001-188780

By the way, the conventional conversation system passively responds to a user's question, but spontaneously speaks from the system side without the user's question, in other words, the topic is active from the system side. There is almost nothing to present. Therefore, when the user's own utterance is poor, the conversation tends to be interrupted, and there is a problem that it is difficult to establish the flow as a conversation. This point does not differ even when a response sentence to the user's utterance is generated by using the keyword acquired from the image captured by the camera. In addition, the above-mentioned Patent Document 3 uses the keyword acquired from the image captured by the camera to improve the accuracy and response of the response sentence, and does not use it to give continuity to the flow of conversation. not.

The present invention has been made in view of such circumstances, and an object of the present invention is to be continuous with the flow of conversation in a conversation system in which a conversation is performed using keywords related to a subject acquired from an image shared with a user. It is to have sex.

In order to solve such a problem, the first invention provides a conversation system having a keyword generation unit and a conversation processing unit and having a conversation with a user. The keyword generation unit receives a shared image shared with the user as an input, and generates a plurality of keywords related to the subject projected on the shared image. The conversation processing unit selects a first keyword from a plurality of keywords generated by the keyword generation unit, and spontaneously utters a conversation sentence using the first keyword. Further, when it is determined that the user's response is interrupted in the conversation starting from the utterance of the conversation sentence using the first keyword, the conversation processing unit selects from a plurality of keywords generated by the keyword generation unit. A second keyword different from the first keyword is selected, and a conversational sentence using the second keyword is spontaneously spoken.

Here, in the first invention, a keyword management table for managing the usage state of the keyword in the conversation with the user may be further provided. In this case, the keyword generation unit newly registers the generated plurality of keywords in the keyword management table. Further, the conversation processing unit updates the usage state of the first keyword in the keyword management table according to the selection of the first keyword, and the second keyword management table according to the selection of the second keyword. Update the usage status of the keyword.

In the first invention, the conversation processing unit interrupts the user's response when the user's no response continues for a predetermined time and the user's response content satisfies at least one of the predetermined conditions. It may be determined that the product has been used. In addition, the conversation processing unit utters the first conversation sentence by inserting the first keyword into the first conversation sentence template selected according to the attribute of the first keyword, and the second conversation sentence. The second conversation sentence may be uttered by inserting the second keyword into the second conversation sentence template selected according to the attribute of the keyword.

The second invention provides a conversation program for having a conversation with a user by causing a computer to execute a process having the first to fourth steps. In the first step, a shared image shared with the user is input, and a plurality of keywords related to the subject projected on the shared image are generated. In the second step, the first keyword is selected from a plurality of keywords, and a conversational sentence using the first keyword is spontaneously uttered. In the third step, it is determined whether or not the user's response is interrupted in the conversation starting from the utterance of the conversation sentence using the first keyword. In the fourth step, when it is determined that the user's response is interrupted, a second keyword different from the first keyword is selected from a plurality of keywords, and a conversation sentence using the second keyword is spontaneously generated. Speak to the target.

Here, in the second invention, there may be further provided a fifth step of newly registering a plurality of keywords in the keyword management table that manages the usage state of the keywords in the conversation with the user. In this case, the second step includes a step of updating the usage state of the first keyword in the keyword management table. Further, the fourth step includes a step of updating the usage state of the second keyword in the keyword management table.

In the second aspect of the invention, the third step is that the user's response satisfies at least one of the condition that the user's response continues for a predetermined time and the user's response content satisfies the predetermined condition. It may be determined that it is interrupted. Further, the second step includes a step of uttering a first conversation sentence by inserting the first keyword into the first conversation sentence template selected according to the attribute of the first keyword. , The fourth step described above includes a step of uttering a second conversation sentence by inserting the second keyword into the second conversation sentence template selected according to the attribute of the second keyword. You may.

According to the present invention, utterances from the system side are spontaneously and continuously performed by selectively using a plurality of keywords related to the shared image. In a conversation that starts with an utterance using the first keyword, if the user's response is interrupted, an utterance using a second keyword different from the first keyword is spontaneously made to the user. Topic is presented. Since these keywords relate to shared images that the user recognizes and shares, even if a new topic is presented within the range, the user does not feel abrupt. As a result, it is possible to give continuity to the conversation in a natural flow.

In particular, if the usage status of a keyword in a conversation with a user is managed in a keyword management table, it is possible to avoid a situation in which a topic related to the same keyword is repeated immediately after a topic related to a certain keyword is interrupted, that is, duplication of topics is possible. ..

Conversation system block configuration diagram Explanatory diagram of keyword management table as an example Flowchart showing the procedure of conversation processing Diagram showing an example of keywords generated from a shared image Figure showing an example of initial registration of keyword management table Diagram showing an example of the flow of conversation with the user Explanatory diagram of the keyword management table according to the first modification Explanatory diagram of the keyword management table according to the second modification

FIG. 1 is a block configuration diagram of a conversation system according to the present embodiment. The conversation system 1 is mounted on an interactive robot toy, a smartphone in which a dialogue application is installed, a smart speaker, a personal computer (PC), and the like, and has a conversation with a user who is the other party of the conversation. In this conversation, the conversation system 1 responds to a question from the user at any time, and voluntarily and actively presents a topic related to the "shared image" to the user. In the present specification, the "shared image" means an image shared with a user who is a conversation partner and grasped by the user himself / herself. For example, in a toy such as an interactive robot, an image of the surroundings of the robot captured by a camera mounted as the eyes of the robot, an image captured by the built-in camera of the smartphone, and displayed on the display of a smartphone or a personal computer (PC). The image etc. can be mentioned. All the subjects projected in these images can be a topic shared with the user because they are grasped by the user himself / herself.

The conversation system 1 outputs utterances corresponding to the utterances of the user collected by the microphone 2 and the contents of the specific shared image. The user's utterance may be acquired not only as voice information collected by the microphone 2, but also character information input by a keyboard or the like, such as a LINE bot, may be acquired as the user's utterance. Further, as a means for acquiring a shared image, for example, a camera that captures an image by a user's operation can be used. In addition to this, an image selected by the user among the images stored in advance in the conversation system 1, such as displaying an image (image file) stored in advance in the toy, or the user is actually viewing the image. Things may be used as shared images.

The conversation system 1 is mainly composed of a keyword generation unit 3 and a conversation processing unit 4 in addition to the microphone 2 that collects the user's utterances. Further, the conversation system 1 includes a keyword management table 5, an attribute table 6, a conversation sentence template storage unit 7, and the like as storage units for storing necessary information.

The keyword generation unit 3 inputs a shared image when talking with the user, and generates and outputs items related to the subject projected on the shared image as a keyword (text). For example, it seems that the keyword "Mt. Fuji" can be obtained from a shared image with Mt. Fuji as the subject. The keyword extraction method itself is known, and any method can be used. For example, the methods described in Patent Document 1 and Patent Document 2 described above may be used, or an object detection algorithm based on deep learning such as YOLO (You Only Look Once) or SSD (Single Shot MultiBox Detector) may be used. good. The plurality of keywords generated by the keyword generation unit 3 are output to the conversation processing unit 4 and newly registered in the keyword management table 5.

The keyword management table 5 manages the usage state of the keyword in the conversation with the user. FIG. 2 is an explanatory diagram of the keyword management table 5 as an example. The keyword management table 5 manages the usage status of the plurality of keywords A to D in a conversation with the user as a status. There are two statuses, "unused", which indicates a state that has not been used in the conversation, and "used", which indicates a state that has already been used in the conversation. At the time of new registration in the keyword management table 5, all the statuses of the keywords A to D are set to "unused".

The conversation processing unit 4 is provided with a text reading upper part 4a (TTS) that reads out a text sentence by voice, analyzes the user's voice acquired by the microphone 2 and responds to the content, like an existing smart speaker or the like. Speak the spoken sentence (response sentence). In addition to passive utterances in response to user utterances, the conversation processing unit 4 also performs spontaneous utterances that actively present new topics. This spontaneous utterance is a system-driven push-type utterance, and unlike passive utterance (response), it does not depend on the user's utterance. As a result, various topics are presented to the user within the range of matters related to the shared image shared with the user.

This spontaneous utterance is performed by referring to the keyword management table 5. Specifically, first, from a plurality of keywords A to D generated by the keyword generation unit 3, one having a status of "unused" in the keyword management table 5 (for example, keyword A) is selected. The keyword management table 5 is updated according to the selection of the keyword A, and the status of the selected keyword A is changed from "unused" to "used". Next, the attribute of the keyword A is specified by referring to the attribute table 6. Next, a conversation sentence template corresponding to the attribute of the keyword A is selected from a large number of conversation sentence templates stored in the conversation sentence template storage unit 7. When there are a plurality of conversational sentence templates corresponding to the attribute of the keyword A, one conversational sentence template is selected based on a predetermined selection rule or at random. Then, by inserting the keyword A in a blank place in the selected conversation sentence template, a conversation sentence with the keyword A as a topic is generated and spoken by the text reading upper part 5a.

Further, the conversation processing unit 4 determines whether or not the user's response is interrupted in the conversation starting from the utterance of the conversation sentence using the keyword A. When it is determined that the user's response is interrupted, a new topic different from the most recently used keyword A, that is, a new topic using any of the keywords B to D is presented to the user. Specifically, first, from the plurality of keywords A to D generated by the keyword generation unit 3, the one whose status in the keyword management table 5 is "unused" at that time (for example, keyword B) is selected. Will be done. The keyword management table 5 is updated according to the selection of the keyword B, and the status of the keyword B is changed from "unused" to "used". Next, the attribute of the keyword B is specified by referring to the attribute table 6. Next, one of the conversational sentence templates corresponding to the attribute of the keyword B is selected from a large number of conversational sentence templates stored in the conversational sentence template storage unit 7. Then, by inserting the keyword B in the blank part of the selected conversation sentence template, a conversation sentence with the keyword B as a topic is generated and spoken by the text reading upper part 5a.

FIG. 3 is a flowchart showing the procedure of conversation processing. The conversation system shown in FIG. 1 can be realized by causing a computer (microcomputer) to execute a computer program (including an application) installed in advance. Hereinafter, as shown in FIG. 4, a case where an image obtained by capturing a view of Mt. Fuji with a built-in camera as a shared image in a smartphone on which a conversation application is installed will be described as an example.

First, in step 1, a shared image is input to the conversation system 1. The shared image may be specified according to the explicit instruction of the user, or may be automatically specified by the system side.

Next, in step 2, items related to the subject projected on the shared image input in step 1 are generated and output as a plurality of keywords. For example, as shown in FIG. 4, seven keywords such as "Mt. Fuji", "mountain", "snow", "sky", "blue", "white", and "volcano" are acquired from the shared image of Mt. Fuji. To. Depending on the keyword extraction algorithm, the accuracy (accuracy) is output together with the output of each keyword.

In step 3, the plurality of keywords acquired in step 2 are newly registered in the keyword management table 5. For example, as shown in Fig. 5, the status is set to "unused" for seven keywords such as "Mt. Fuji", "mountain", "snow", "sky", "blue", "white", and "volcano". After that, it is registered in the keyword management table 5.

In step 4, one of the keywords whose status is "unused" is selected from the plurality of keywords generated in step 2. The keywords may be selected randomly, or may be selected based on a predetermined selection rule, for example, in descending order of accuracy. Then, the status management table 5 is updated, and the status of the selected keyword is changed from "unused" to "used". For example, when "Mt. Fuji" is selected from the above seven keywords, the status of "Mt. Fuji" is changed from "unused" to "used".

In step 5, a spontaneous conversational sentence is uttered using the keyword selected in step 4 (spontaneous utterance). Explaining the case where "Mt. Fuji" is selected as the keyword, "mountain" is specified as the attribute of "Mt. Fuji" with reference to the attribute table 6. Next, from the conversation sentence template storage unit 7, the conversation sentence template "○○ is beautiful" is selected as the conversation sentence template corresponding to the attribute "mountain". Then, by inserting the keyword "Mt. Fuji" into the "○○" of the selected conversation sentence template, the conversation sentence "Mt. Fuji is beautiful" is uttered, and the topic of "Mt. Fuji" is raised to the user.

In step 6, it is determined whether or not the conversation is interrupted in the conversation starting from the spontaneous utterance in step 5. This determination condition can be arbitrarily set, but for example, when the state in which there is no response from the user continues for a predetermined time, it may be determined that the conversation is interrupted. This is because it can be considered that the user is not interested in this topic because there is no response from the user. Further, although there is a response from the user, if the content satisfies a predetermined condition, for example, if the user has a negative response or a negative affirmation, it may be determined that the conversation is interrupted. At that time, the intonation of the user's voice may be taken into consideration.

If the determination result in step 6 is negative, that is, if it is determined that the conversation is not interrupted due to the user's response, the process proceeds to step 7, and a passive conversation sentence corresponding to the user's response is uttered. (Passive utterance). The utterance of this passive conversational sentence is repeated until it is determined that the conversation is interrupted (steps 6 and 7). As a result, the conversation with the user starting from the spontaneous utterance "Mt. Fuji is beautiful" by the system will be continued.

On the other hand, if the determination result in step 6 is affirmative, that is, if it is determined that the conversation is interrupted, the process returns to step 4. Then, a new keyword whose status is "unused" is selected (step 4), and a spontaneous conversation sentence is uttered using this keyword (step 5). As a result, the subsequent conversation will shift from the topic of "Mt. Fuji" to another topic (for example, "volcano").

The series of exchanges with the user as described above is continued until it is determined by another routine that the conversation has ended.

FIG. 6 is a diagram showing an example of the flow of conversation with the user. First, as a spontaneous utterance on the conversation system 1 side, "Mt. Fuji is beautiful" with the keyword "Mt. Fuji" is spoken. Then, the response on the user side starting from this and the passive utterance on the conversation system 1 side are repeated. After that, when the user's response (negative affirmation) saying "That's right" is spoken, the interruption condition is satisfied and the topic of "Mt. Fuji" ends.

With the end of this topic, "But volcanoes are scary, aren't they?" With the keyword "volcano", which is different from "Mt. Fuji", as a spontaneous utterance on the conversation system 1 side. After that, when the user's response (negative response) saying "I'm not scared" is spoken, the interruption condition is satisfied and the topic of "volcano" ends.

With the end of this topic, as a spontaneous utterance on the conversation system 1 side, "Let's go up the mountain" with the keyword "mountain" different from "volcano" is spoken. After that, when the user response (negative affirmation) of "Like" is uttered, the interruption condition is satisfied and the topic of "Mountain" ends.

With the end of this topic, as a spontaneous utterance on the conversation system 1 side, "I like a snow-free day" with the keyword "snow", which is different from "mountain", was spoken, and communication with users has continued since then. Will be done. Even if the topic shifts in the order of "Mt. Fuji", "volcano", "mountain", and "snow" in the situation where the captured image of Mt. Fuji is shared with the user, there is no unnatural flow in the conversation and the user. It doesn't make you feel abrupt.

As described above, according to the present embodiment, utterances from the system side are spontaneously and continuously performed by selectively using a plurality of keywords related to the shared image. In a conversation that starts with an utterance using a certain keyword, if the user's response is interrupted, an utterance using another keyword different from this keyword is spontaneously performed, and a new topic is presented to the user. To. Since these keywords are related to the shared image shared with the user, even if a new topic is presented within the range, the user does not feel abrupt. As a result, it is possible to give continuity to the conversation in a natural flow.

Further, according to the present embodiment, by managing the usage state of the keyword in the conversation with the user by using the keyword management table 5, for example, the topic of "Mt. Fuji" is repeated immediately after the topic of "Mt. Fuji" is interrupted. It is possible to avoid such a situation. As a result, it is possible to give continuity to the conversation without giving the user a sense of discomfort.

In the above-described embodiment, the presence / absence (unused / used) of each keyword is managed as the status of the keyword management table 5, but as shown in FIG. 7, for example, 0 times and 1 time. You may manage the number of times each keyword is used, such as once or twice. Further, as shown in FIG. 8, only what is the keyword currently the theme may be managed by the flag. For example, in a conversation starting from a spontaneous utterance using the keyword A, when the user's response is interrupted, the next spontaneous utterance from the status (usage state) in the figure is something other than the current keyword A, that is, One of the keywords B to D is selected. As a result, as in the above-described embodiment, it is possible to avoid a situation in which the immediately preceding topic is repeated when shifting to a new topic.

1 Conversation system 2 Microphone 3 Keyword generation unit 4 Conversation processing unit 4a Text reading upper part 5 Keyword management table 6 Attribute table 7 Conversation sentence template storage unit

Claims (8)

In a conversation system that talks with the user
A keyword generation unit that generates a plurality of keywords related to the subject projected on the shared image by inputting a shared image shared with the user, and a keyword generation unit.
A first keyword is selected from a plurality of keywords generated by the keyword generation unit, a conversation sentence using the first keyword is spontaneously spoken, and a conversation sentence using the first keyword is spoken. When it is determined that the user's response is interrupted in the conversation starting from the utterance of the above, a second keyword different from the first keyword is selected from the plurality of keywords generated by the keyword generation unit. , A conversation system characterized by having a conversation processing unit that spontaneously utters a conversation sentence using the second keyword. It also has a keyword management table that manages the usage status of keywords in conversations with users.
The keyword generation unit newly registers the generated plurality of keywords in the keyword management table, and then registers the generated keywords in the keyword management table.
The conversation processing unit updates the usage state of the first keyword in the keyword management table according to the selection of the first keyword, and the keyword management table according to the selection of the second keyword. The conversation system according to claim 1, wherein the usage state of the second keyword is updated. The conversation processing unit determines that the user's response is interrupted when the user's no response continues for a predetermined time and the user's response content satisfies at least one of the predetermined conditions. The conversation system according to claim 1 or 2. The conversation processing unit utters the first conversation sentence by inserting the first keyword into the first conversation sentence template selected according to the attribute of the first keyword, and at the same time, the first conversation sentence. Any of claims 1 to 3, wherein the second conversation sentence is uttered by inserting the second keyword into the second conversation sentence template selected according to the attribute of the second keyword. The conversation system described in. In a conversation program that talks to the user
The first step of generating a plurality of keywords related to the subject projected on the shared image by inputting the shared image shared with the user, and
A second step of selecting a first keyword from the plurality of keywords and spontaneously uttering a conversational sentence using the first keyword.
In the conversation starting from the utterance of the conversation sentence using the first keyword, the third step of determining whether or not the user's response is interrupted, and
When it is determined that the user's response is interrupted, a second keyword different from the first keyword is selected from the plurality of keywords, and a conversation sentence using the second keyword is spontaneously spoken. A conversation program comprising causing a computer to execute a process having a fourth step. It further has a fifth step of newly registering the plurality of keywords in the keyword management table that manages the usage state of the keywords in the conversation with the user.
The second step includes updating the usage status of the first keyword in the keyword management table.
The conversation program according to claim 5, wherein the fourth step includes a step of updating the usage state of the second keyword in the keyword management table. In the third step, it is determined that the user's response is interrupted when the user's no response continues for a predetermined time and the user's response content satisfies at least one of the predetermined conditions. The conversation program according to claim 5 or 6. The second step is a step of uttering the first conversation sentence by inserting the first keyword into the first conversation sentence template selected according to the attribute of the first keyword. Including,
The fourth step is a step of uttering the second conversation sentence by inserting the second keyword into the second conversation sentence template selected according to the attribute of the second keyword. The conversation program according to any one of claims 5 to 7, wherein the conversation program comprises.

Images