JP2017054064A - Interactive device and interactive program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an interactive device and interactive program, capable of appropriately grasping user's intention.SOLUTION: An interactive device 100 is for providing dialogue with a user. The interactive device 100 includes a voice acquisition unit 131 configured to acquire voice of the user, an image acquisition unit 132 configured to acquire an image, and an identification unit 135 configured to identify a target object included in the image acquired by the image acquisition unit 132 based on the voice of the user acquired by the voice acquisition unit 131.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a dialogue apparatus and a dialogue program.

  A conventional dialogue apparatus recognizes a user's voice, grasps the intention of the user, and performs a dialogue with the user (for example, see Patent Document 1 below).

JP 2002-182896 A

  In a conventional dialog device that performs a dialog only by user's voice recognition, the dialog device may not be able to accurately grasp the user's intention. For example, assume that the user wants to ask the dialog device about the height of a certain mountain. If the user knows knowledge about the mountain, for example, the mountain name “A”, the user can pronounce the mountain name. For this reason, if the user utters the voice of a question such as “What is the height of A?”, The content of the question is clear only by the voice, so that the dialogue device can grasp the user's intention and perform a dialogue. it can. On the other hand, if the user does not know the name of the mountain, the user cannot pronounce the name of the mountain. For this reason, the user cannot utter an appropriate voice for asking the height of the mountain only by voice. In that case, the dialog device cannot grasp the user's intention and cannot perform a dialog.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide an interactive device and an interactive program capable of more appropriately grasping a user's intention.

  An interactive apparatus according to an aspect of the present invention is an interactive apparatus for performing a dialog with a user, and is acquired by a voice acquisition unit that acquires a user's voice, an image acquisition unit that acquires an image, and a voice acquisition unit. Specifying means for specifying an object included in the image acquired by the image acquisition means based on the user's voice.

  Further, an interactive program according to an aspect of the present invention includes a computer provided in an interactive apparatus for performing an interaction with a user, an audio acquisition unit that acquires the user's audio, an image acquisition unit that acquires an image, Based on the user's voice acquired by the voice acquisition means, it is made to function as a specifying means for specifying an object included in the image acquired by the image acquisition means.

  According to the interactive apparatus or the interactive program, the object included in the image acquired by the image acquisition unit is specified based on the user's voice acquired by the audio acquisition unit. Thus, for example, even when the user does not have knowledge about the object, if the user intends to specify (specify) that the object is included in the image, the sound is based on the sound. The object intended by the user is specified. Therefore, the user's intention can be grasped more appropriately in the dialogue.

  The dialogue apparatus further includes a recognition unit that executes a recognition mode for recognizing the user's voice acquired by the voice acquisition unit, the specifying unit specifies the name of the object, and the recognition mode is the voice acquisition unit. The user's voice is recognized based on the first recognition mode for recognizing the user's voice acquired by the user, the user's voice acquired by the voice acquisition means, and the name of the object specified by the specifying means. And a second recognition mode. When the first recognition mode is executed, the user's voice is recognized, and a dialogue with the user is performed in the same manner as a conventional dialogue device. On the other hand, when the second recognition mode is executed, the user's voice is recognized based on the user's voice and the name of the identified object, and a dialogue with the user is performed. By executing such a second recognition mode, it is possible to interact with the user while more appropriately grasping the user's intention.

  The identification unit identifies a name of an object that can be replaced with a part of the user's voice acquired by the voice acquisition unit, and corresponds to the user's voice acquired by the voice acquisition unit in the second recognition mode. After a part of the data to be replaced is replaced with data corresponding to the name of the object specified by the specifying unit, the user's voice may be recognized based on the replaced data. Thus, for example, even when there are a plurality of candidates for the name of an object, the object name that can be replaced with a part of the user's voice, that is, an object suitable for the context (flow of conversation) Is specified. By replacing the data corresponding to the name of the object identified in this way with a part of the data corresponding to the user's voice, the user's intention is more appropriate along the flow of the conversation. Can grasp.

  When the recognizing unit recognizes that the user's voice acquired by the voice acquiring unit includes a predetermined voice while executing the first recognition mode, the recognizing unit changes the recognition mode to be executed to the first recognition mode. May be switched to the second recognition mode. For example, by setting the predetermined voice to a voice that means that the target object is included in the image, the second recognition mode is executed at an appropriate timing, and the user's intention is more appropriate. To be able to grasp.

  The specifying unit may specify the target based on the position information of the target in the image and the user's voice acquired by the voice acquisition unit. Thereby, even when a plurality of objects are included in the image, the object intended by the user can be specified based on the user's voice.

  The interactive apparatus may further include an output unit that outputs an image, and the image acquisition unit may acquire the image output by the output unit. Thereby, for example, even when a dialogue between the dialogue apparatus and the user is performed using the image or video output by the output means as a topic, as described above, the object intended by the user is specified, and the user Can better understand the intentions.

  According to the present invention, a user's intention can be grasped more appropriately.

It is a figure which shows the functional block of a dialogue apparatus. 4 is a diagram illustrating an example of a data table stored in a storage unit 140. FIG. It is a figure which shows the hardware constitutions of a dialogue apparatus. It is a figure which shows the structure of a dialogue program. It is a 1st flowchart which shows an example of the process performed in a dialogue apparatus. It is a 2nd flowchart which shows an example of the process performed in a dialogue apparatus. It is a figure which shows the functional block of the dialogue apparatus which concerns on a modification.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant descriptions are omitted.

  The interactive device according to the embodiment is a device that performs a dialog with a user. The interactive device may be realized as a mobile terminal device such as a smartphone or a stationary terminal, or may be realized as a robot imitating a human outer shape.

  FIG. 1 is a diagram illustrating functional blocks of the interactive apparatus according to the embodiment. As illustrated in FIG. 1, the dialogue apparatus 100 includes an input unit 110, an output unit 120, a control unit 130, a storage unit 140, and a communication unit 150.

  The input unit 110 and the output unit 120 are portions (input / output interfaces) for exchanging information with the outside of the interactive apparatus 100, mainly with the user. The input unit 110 accepts input of voice around the interactive apparatus 100 including the user's voice (hereinafter, also simply referred to as “ambient voice”), and images and video around the interactive apparatus 100 including the user ( Hereinafter, the input of “sometimes referred to as“ ambient image ”” is accepted. The output unit 120 outputs various images and videos, and outputs various sounds.

  Specifically, the input unit 110 includes a sound collection unit 111 and an imaging unit 112. The sound collection unit 111 is a part that receives an input of ambient sound. The sound collection unit 111 is constituted by a microphone, for example. The sound collection unit 111 may be configured by a microphone array in which a plurality of microphones are arranged so as to have directivity, for example. The imaging unit 112 is a part that receives an input of a surrounding image. The imaging unit 112 is configured with a camera, for example. The imaging unit 112 is configured by a plurality of cameras, for example, so that the distance from the object can be grasped. Note that the input unit 110 may further include an element such as an operation button for the user to operate the interactive device 100, for example.

  The output unit 120 includes a sound generation unit 121 and a display unit 122. The sound generator 121 is a part that outputs sound. The sound generation unit 121 is configured by a speaker, for example. The sound generation unit 121 may be configured by an array speaker in which a plurality of speakers are arranged so as to have directivity, for example. The display unit 122 is a part (output unit) that outputs images and videos. The display unit 122 is configured by a display, for example. The display may be configured with a touch panel. In this case, the touch panel also functions as an element for the user to operate the interactive device 100.

  The display unit 122 displays contents such as images and videos that are distributed via a communication network such as the Internet or received as a television broadcast. The dialogue between the dialogue apparatus 100 and the user may be performed using the images and videos displayed by the display unit 122 as topics.

  The control unit 130 is a part that performs overall control of the interactive device 100 by controlling each element of the interactive device 100. The control unit 130 includes a voice acquisition unit 131, an image acquisition unit 132, a voice recognition unit 133, an image recognition unit 134, and a specification unit 135.

  The sound acquisition unit 131 is a part that acquires the ambient sound input to the sound collection unit 111 of the input unit 110. That is, the sound acquisition unit 131 and the sound collection unit 111 function as a sound acquisition unit that acquires ambient sound. Hereinafter, unless otherwise specified, the voice acquisition unit is simply referred to as the voice acquisition unit 131.

  The image acquisition unit 132 is a part that acquires the surrounding image input to the imaging unit 112 of the input unit 110. That is, the image acquisition unit 132 and the imaging unit 112 function as an image acquisition unit that acquires a surrounding image. Hereinafter, the image acquisition unit will be referred to as the image acquisition unit 132 unless otherwise specified.

  The image acquisition unit 132 acquires various images and videos displayed by the display unit 122 described above as surrounding images. That is, the image acquisition unit 132 is also a part that acquires the image output by the imaging unit 112.

  The voice recognition unit 133 is a part (recognition means) that executes voice recognition processing for recognizing ambient voice, particularly user voice. The voice recognition unit 133 performs a voice recognition process on the surrounding voice acquired by the voice acquisition unit 131. The speech recognition process is realized by various methods including, for example, a method using a prepared acoustic model and language model. The language model may be a model including a language of a specialized dictionary so that it can correspond to various specialized fields.

  The voice recognition process executed by the voice recognition unit 133 includes a process of analyzing the waveform of the user's voice and outputting the utterance content as result data. The result data is data corresponding to the user's voice, and is, for example, character string data (text data).

  In the speech recognition process, the first recognition mode and the second recognition mode are executed. In the first recognition mode, the user's voice acquired by the voice acquisition unit 131 is recognized as it is. On the other hand, in the second recognition mode, the user's voice is recognized based on the user's voice acquired by the voice acquisition unit 131 and the name of the target specified by the specifying unit 135 described later. More specifically, in the second recognition mode, a part of data (text data or the like) corresponding to the user's voice acquired by the voice acquisition unit 131 corresponds to the name of the object specified by the specification unit 135. After being replaced with data to be performed (text data or the like), the user's voice is recognized (decoded) based on the replaced data.

  The image recognition unit 134 is a part for recognizing the surrounding image acquired by the image acquisition unit 132. The image recognition unit 134 performs an image recognition process on the surrounding image acquired by the sound acquisition unit 131. The image recognition processing technique is not particularly limited as long as it is a commonly used image recognition processing technique. For example, various techniques such as open CV (Open Source Computer Vision Library) are used.

  In addition, the image recognition unit 134 estimates the distance from the imaging unit 112 to the target object included in the surrounding image acquired by the image acquisition unit 132. For example, when the imaging unit 112 is configured by a plurality of cameras as described above, the image recognition unit 134 can measure the distance to the object using the triangulation method.

  The specifying unit 135 is a part (specifying unit) that specifies an object included in the surrounding image acquired by the image acquiring unit 132 based on the surrounding sound acquired by the sound acquiring unit 131, particularly the user's voice. The target object is an object that the user is trying to specify (designate) using sound among the objects included in the surrounding image.

  For example, when the user tries to ask the dialogue apparatus 100 the height of a certain mountain, the user does not know the name “A” of the mountain, but has a photograph of the mountain (or a photograph of the mountain). Is present near the user). At this time, the user presents the photograph to the dialogue apparatus 100 and utters a voice of a question “What is the height of this mountain?”. Then, based on the user's voice “What is the height of this mountain?” Acquired by the voice acquisition unit 131, the specifying unit 135 targets the mountain reflected in the photograph acquired by the image acquisition unit 132. Identify as a thing. For example, the specifying unit 135 specifies the name of the object, in this example, the name “A” of the mountain.

  Specifically, first, the specifying unit 135 specifies an indicator (indicator) or an indicator pronoun included in the user's voice. In the above example, “this” included in the user's voice “What is the height of this mountain?” Is specified as an indicator.

  Further, the specifying unit 135 specifies an image of the object included in the surrounding image based on the specified directive or indicating pronoun. In the above-described example, the mountain image shown in the photograph is specified as the image of the object corresponding to the indicator “this” and the noun “mountain” following the indicator, that is, “this mountain”.

  Then, the specifying unit 135 specifies the name of the target object based on the specified target image. In the above example, the name of the mountain shown in the photograph is specified as “A”. The specification of the name of the mountain may be performed, for example, by referring to various information stored in the storage unit 140 described later, or information is acquired from the outside of the interactive apparatus 100 via the communication unit 150. May be performed.

  Here, the specifying unit 135 may specify the target based on the position information in the surrounding image of the target and the user's voice acquired by the voice acquisition unit 131. In the above-described example, from the meaning of the directive “this”, the object intended by the user may be a mountain that is present in the vicinity of the surrounding image, that is, at a short distance from the interactive device 100. Can be estimated. Based on the position information estimated in this way (in this case, “short distance”), the object intended by the user in the surrounding image can be appropriately identified.

  On the other hand, when a picture of a mountain is placed at a position far behind the user as viewed from the interactive device 100, the user may utter a voice of a question such as “What is the height of that mountain?”. In that case, from the meaning of the directive “that”, it can be estimated that the object intended by the user is a mountain that exists behind the surrounding image, that is, at a position far from the interactive device 100.

  When the user utters a voice of a question such as “What is the height of the mountain?”, From the meaning of the directive “that”, the object intended by the user is a medium distance from the interactive device 100. It can be estimated that the mountain exists at the position of.

  In the above, the example in which the specifying unit 135 specifies the image of the target object included in the surrounding image based on the directives such as “this”, “that”, and “that” has been described. Instead of the lyrics, the image of the object included in the surrounding image may be specified based on the indicating pronoun. For example, in the above-described example, it is also assumed that the user utters a voice of a question such as “How tall is this” while showing a picture of a mountain. In this case, the specifying unit 135 indicates that the object intended by the user is an object that is present in the foreground in the surrounding image, that is, at a position at a short distance from the interactive device 100, from the meaning of the pronoun pronoun “this”. It can be estimated that there is. Even on the basis of the position information estimated in this way, the object intended by the user in the surrounding image can be appropriately identified.

  The storage unit 140 is a part that stores various information necessary for processing executed by the control unit 130. The storage unit 140 stores, for example, the above-described acoustic model and language model. In addition, the storage unit 140 stores a program (interactive program) for causing the interactive apparatus 100 to execute processing necessary for the interactive apparatus 100 to interact with the user.

  In addition, the storage unit 140 stores various information necessary for the specifying unit 135 to specify the name of the object included in the surrounding image. For example, a database storing images and names of various objects is stored in the storage unit 140, and by comparing an object image included in the surrounding image with an object image stored in the database. The name of the object can be specified.

  In addition, the storage unit 140 stores a data table in which the specifier or the pronoun by the specifying unit 135 described above and the positional information in the surrounding image of the target object are described in association with each other.

  FIG. 2 is a diagram illustrating an example of a data table stored in the storage unit 140. As shown in FIG. 2, this data table describes “keyword”, “distance from device”, and “presence / absence of noun following keyword” in association with each other.

  The “keyword” is, for example, a demonstrative pronoun or a directive. In FIG. 2, “this”, “it”, and “that” are illustrated as the pronouns, and “this”, “that”, and “that” are illustrated as the directives.

  The “distance from the device” is a distance from the interactive device 100 (more specifically, the imaging unit 112) to the target, which is assumed in accordance with each keyword, that is, each directive pronoun or each directive. In FIG. 2, the distance from the device is classified into three types of “short distance”, “medium distance”, and “far distance”.

  For example, when the distance from the interactive apparatus 100 to the object is less than the first predetermined distance, the distance is set to “short distance”. When the distance from the interactive apparatus 100 to the object is not less than the first predetermined distance and less than the second predetermined distance, the distance is set to “medium distance”. When the distance from the interactive apparatus 100 to the object is equal to or greater than the second predetermined distance, the distance is set to “far”. The second predetermined distance is greater than the first predetermined distance.

  “Presence / absence of noun following keyword” indicates whether each keyword is a keyword that can be followed by a noun. Specifically, when the keyword is a directive pronoun such as “this”, “it”, or “that”, the presence or absence of the noun following the keyword is set to “none”. On the other hand, if the keyword is an indicator such as “this”, “that”, “that”, the presence or absence of the noun following the keyword is “present”.

  Referring to the data table shown in FIG. 2, when a predetermined voice (keyword) is included in the user's voice, based on the keyword, position information regarding the object intended by the user, that is, an interactive device The distance from 100 can be estimated. Also, by determining whether there is a noun following the keyword, for example, if there is a noun following the keyword, the target object intended by the user can be grasped more appropriately by referring to the subsequent noun. The possibility increases.

  Returning to FIG. 1 again, the communication unit 150 is a part that communicates with the outside of the interactive apparatus 100. By the communication unit 150, for example, the above-described various information stored in the storage unit 140 may be additionally acquired or updated. In addition, by accessing the Internet or the like using the communication unit 150, a search process for specifying the name of the object from the image of the object included in the surrounding image may be executed.

  Next, the hardware configuration of the interactive apparatus 100 will be described with reference to FIG. As shown in FIG. 3, the interactive apparatus 100 physically has one or more CPUs (Central Processing Units) 21, RAMs (Random Access Memory) 22, ROMs (Read Only Memory) 23, and imaging such as a camera. Device 24, communication module 26 that is a data transmission / reception device, auxiliary storage device 27 such as a semiconductor memory, input device 28 that receives an input of a user operation such as an operation panel (including operation buttons) or a touch panel, an output device 29 such as a display, Moreover, it may be configured as a computer including a reading device 2A such as a CD-ROM drive device. The function of the interactive device 100 in FIG. 1 is controlled by the CPU 21 by, for example, reading one or a plurality of programs stored in a storage medium M such as a CD-ROM with a reading device 2A and taking it into hardware such as a RAM 22. This is realized by operating the imaging device 24, the communication module 26, the input device 28, and the output device 29 under the above, and reading and writing data in the RAM 22 and the auxiliary storage device 27.

  FIG. 4 shows a module of an interactive program for causing a computer to function as the interactive apparatus 100. As shown in FIG. 4, the dialogue program P100 includes a voice acquisition module P101, an image acquisition module P102, a voice recognition module P103, an image recognition module P104, and a specific module P105. Each module implements the functions of the voice acquisition unit 131, the image acquisition unit 132, the voice recognition unit 133, the image recognition unit 134, and the specifying unit 135 described above with reference to FIG.

  The interactive program is provided by being stored in a storage medium, for example. The storage medium may be a flexible disk, CD-ROM, USB memory, DVD, semiconductor memory, or the like.

  Next, the operation of the interactive device 100 (the interactive method executed by the interactive device 100) will be described using FIG. 5 and FIG.

  FIG. 5 is a flowchart illustrating an example of processing executed in the interactive apparatus 100. The processing of this flowchart is executed in response to the start of dialogue between the dialogue apparatus 100 and the user. Unless otherwise specified, each process can be executed by the control unit 130 (that is, regardless of which element is included in the control unit 130).

  First, the interactive apparatus 100 executes speech recognition using the recognition mode as the first recognition mode (step S1). Specifically, the voice recognition unit 133 executes the first recognition mode and performs a dialog with the user. In addition, the process performed in following step S2, S3 is a process performed in 1st recognition mode.

  Next, the dialogue apparatus 100 executes an ambiguous input determination process (step S2). That is, the dialogue apparatus 100 determines whether or not there is an ambiguous input (step S3). This process is executed by the voice recognition unit 133. For example, if a keyword such as a directive or a pronoun is included in the user's voice, the user's voice alone does not clarify the intention of the user's utterance. It is judged that there was. When there is an ambiguous input (step S3: YES), the dialogue apparatus 100 proceeds with the process to step S4. When that is not right (step S3: NO), the dialogue apparatus 100 returns a process to step S1 again.

  Even if a directive is included, the nouns that follow the directives “that” and “that”, such as “that night” and “that day” cannot be represented by images or videos. It may be determined that there is no ambiguous input (step S3: NO).

  In step S4, the interactive apparatus 100 switches the recognition mode to the second recognition mode. Specifically, the voice recognition unit 133 switches the recognition mode executed in the voice recognition process from the first recognition mode to the second recognition mode. Therefore, the processing executed in the following steps S5 to S7 described later is processing executed in the second recognition mode.

  Specifically, the interactive device 100 executes a designated distance estimation process based on the voice recognition result (step S5). This process is executed by the specifying unit 135, for example. Specifically, the data table as shown in FIG. 2 described above stored in the storage unit 140 is referred to, and the distance from the interactive device 100 intended by the user to the target is estimated. The

  Further, the dialogue apparatus 100 executes designated noun identification processing based on the voice recognition result (step S6). This process is executed by the specifying unit 135, for example. Specifically, when a directive is included in the user's voice, a noun following the directive is specified and retained as a designated noun.

  After the processes of step S5 and step S6 are completed, the interactive device 100 executes an image recognition process (step S7).

  FIG. 6 is a flowchart showing details of processing executed in the image recognition processing (step S7 in FIG. 5).

  First, the interactive device 100 executes image recognition / distance measurement processing (step S11). Specifically, the image recognition unit 134 identifies the number (n) of objects included in the surrounding image acquired by the image acquisition unit 132 and recognizes the image of each object. Further, the image recognition unit 134 measures (estimates) the distance from the interactive apparatus 100 to each object.

  In the next steps S12 to S17, the interactive apparatus 100 evaluates which of the n objects is likely to correspond to the object intended by the user. Specifically, the interactive device 100 sets the initial value of the variable i to 0 (step S12), increases i by 1 (step S16), and until i becomes n or more (step S17: NO), The following steps S13 to S15 are repeatedly executed for the i-th object.

  First, the interactive apparatus 100 calculates the number of distance points (step S13). Specifically, the specifying unit 135 sets the distance from the interactive device 100 to the object estimated in the previous step S11 and the target from the interactive device 100 intended by the user estimated in the previous step S5 (FIG. 5). The distance score is calculated based on the distance to the object. For example, the distance points are calculated to be larger as the distance between the two is closer.

  Next, the dialogue apparatus 100 determines whether or not the designated noun word is held (step S14). Specifically, if there is a noun following the directive specified and retained in the previous step S6 (FIG. 5), it is determined that the designated noun word is retained. When the word of the designated noun is held (step S14: YES), the dialogue apparatus 100 proceeds with the process to step S15. Otherwise (step S14: NO), the interaction device 100 skips step S15 and proceeds to step S16.

  In step S15, the dialogue apparatus 100 calculates a score using the recognized image name and the retained noun (step S15). This process is executed by the specifying unit 135. For example, if the type of the object identified by the image recognition unit 134 (such as “mountain”) matches the noun (such as “mountain”) following the directive, it is calculated in the previous step S13. The distance score is increased. Conversely, if the name of the object specified by the image recognition unit 134 and the noun following the directive do not match, the distance score is maintained or reduced.

  After the processes in steps S13 to S15 described above have been executed for n objects (step S17: YES), the interactive apparatus 100 proceeds with the process to step S18.

  In step S18, the interactive apparatus 100 determines one target based on the total score. Specifically, the specifying unit 135 specifies the object having the highest score calculated in the previous step S13 and / or step S15 among the n objects as the object intended (designated) by the user. . That is, in step S18, an object that is likely to be replaced with a part of the user's voice from the conversation context such as the distance from the interactive device 100 to the object and the presence or absence of a noun and a noun following the pronoun. The name of the object is specified.

  Specifically, from the conversation context such as the presence of a noun that follows the directive and the pronoun, the name of the object that is likely to be replaced with a part of the user's voice is as follows. Can be identified. As an example, it is assumed that a certain graphic is included in the surrounding image and the graphic is a graphic indicating a certain fruit and a graphic indicating a logo of a certain company. In this case, when the user utters a voice saying “I want to eat this”, there is no noun that follows from the indicating pronoun “this”. For this reason, the said figure is simply recognized as a figure (image) which shows a fruit, and the name of the said fruit is specified as a name of the target object which the user intends. On the other hand, when the user utters “Who is the president here?”, There is a noun “president” following the directive “here”. For this reason, the figure is not simply a figure showing fruit, but is recognized as a figure (image) showing a word related to the noun “president”, for example, “company”, “company”, etc. Is specified as the name of the object intended by the user. For example, in this way, the names of objects that are likely to be replaced with a part of the user's voice are specified from the conversation context.

  Thereafter, the interactive device 100 executes a speech recognition result correction process (step S19). This process is executed by the voice recognition unit 133. Specifically, the indicator and subsequent noun or indicator pronoun included in the data (text data or the like) corresponding to the user voice is replaced with data corresponding to the name of the object specified in the previous step S18. Then, based on the replaced data, the user's voice is recognized (the contents are decoded).

  After the process of step S19 is completed, the dialogue apparatus 100 returns the process to step S1 again (FIG. 5).

  Next, the function and effect of the interactive device 100 will be described. According to the interactive apparatus 100, the specifying unit 135 specifies an object included in the surrounding image acquired by the image acquiring unit 132 based on the user's voice acquired by the voice acquiring unit 131 (step S7). Thereby, for example, even when the user does not have knowledge about the target object, the voice that intends (designates) that the target object is included in the surrounding image, for example, an indicator, an indicating pronoun is included. If a voice is uttered, an object intended by the user is specified based on the voice. Therefore, the user's intention can be grasped more appropriately in the dialogue.

  Specifically, the voice recognition unit 133 first executes the first recognition mode as voice recognition processing, and recognizes the user's voice as it is. Thereby, the dialog with the user is performed as in the conventional dialog device (steps S1, S2, S3: NO). Furthermore, the voice recognition unit 133 executes the second recognition mode, and recognizes the user's voice based on the user's voice and the name of the identified object (steps S4 to S7). By executing the second recognition mode, it is possible to interact with the user while more appropriately grasping the user's intention.

  Further, the specifying unit 135 specifies the name of an object that can be replaced with a part of the user's voice acquired by the voice acquisition unit 131 (step S18). Then, after the voice recognition unit 133 replaces a part of the data corresponding to the user's voice with the data corresponding to the name of the target specified by the specifying unit 135, the user recognizes the user based on the replaced data. The voice is recognized (step S19). Thereby, for example, even when a plurality of objects are included in the surrounding image and there are a plurality of candidates for the name of the object intended (designated) by the user, it is replaced with a part of the user's voice. The name of the object that can be used, that is, the name of the object suitable for the context (flow of conversation) is specified. By replacing the data corresponding to the name of the object identified in this way with a part of the data corresponding to the user's voice, the user's intention is more appropriate along the flow of the conversation. Can grasp.

  Further, when the voice recognition unit 133 recognizes that the user's voice includes a keyword such as a directive or a pronoun when the first recognition mode is being executed, the voice recognition unit 133 sets the recognition mode to be executed to the first. The recognition mode is switched to the second recognition mode (step S3: YES, step S4). As a result, the second recognition mode is executed at an appropriate timing when the name of the object needs to be specified based on the directive and the pronoun, and the user's intention can be grasped more appropriately. .

  Further, the specifying unit 135 specifies the target object based on the position information of the target object in the surrounding image, more specifically, based on the distance from the interactive device 100 to the target object and the user's voice (Steps S13 to S13). S15, S18). Thereby, even when a plurality of objects are included in the surrounding image, the object intended (designated) by the user can be specified based on the user's voice.

  The surrounding image may be an image or video displayed by the display unit 122. Thereby, for example, a conversation with the user can be performed using the image and video displayed on the display unit 122 as a topic. For example, it is assumed that a video of a certain mountain landscape received as a television broadcast is displayed on the display unit 122 during a dialogue with the user. For example, when the user utters a voice such as “Where is this place?”, The user's intention is appropriately determined by specifying the name of the mountain displayed by the display unit 122 according to the indicating pronoun “here”. Dialogue can be conducted while grasping.

  Each function of the interactive apparatus 100 described above can also be realized, for example, by executing an interactive program on a computer.

  Although one embodiment of the present invention has been described above, the present invention is not limited to the above embodiment.

  FIG. 7 is a diagram showing functional blocks of an interactive apparatus according to a modification. The dialogue apparatus 100 </ b> A constitutes a dialogue system 1 that performs dialogue with the user in cooperation with the server 200. In this modification, the dialogue system 1 corresponds to the dialogue device according to the present invention.

  As illustrated in FIG. 7, the server 200 includes a control unit 230, a storage unit 240, and a communication unit 250.

  The control unit 230 includes a voice acquisition unit 231, an image acquisition unit 232, a voice recognition unit 233, an image recognition unit 234, and a specification unit 235. Each of these elements has the same functions as the voice acquisition unit 131, the image acquisition unit 132, the voice recognition unit 133, the image recognition unit 134, and the specification unit 135 described above with reference to FIG.

  The storage unit 240 has the same function as the storage unit 140 described above with reference to FIG. That is, the storage unit 240 is a part that stores various information necessary for processing executed by the control unit 230. For example, an acoustic model, a language model, a dialogue program, a database that stores images and names of various objects, A data table as shown in FIG. 2 is stored.

  The communication unit 250 is a part that communicates with the communication unit 150 of the interactive apparatus 100A. The communication unit 250 enables communication between the dialogue apparatus 100A and the server 200. By accessing the Internet or the like using the communication unit 250, a search process for specifying the name of the object from the image of the object included in the surrounding image may be executed.

  With the above configuration, the dialogue system 1 enables dialogue with the user through cooperation between the dialogue apparatus 100 </ b> A and the server 200. That is, in the interactive system 1, a part of the processing (particularly, processing executed by the control unit 130) executed for performing a dialog with the user in the interactive device 100 (FIG. 1) is executed by the server 200. Therefore, according to the interactive device 100A, the processing load on the interactive device can be reduced as compared with the interactive device 100.

  Specifically, the interaction device 100A may include a control unit 130A and a storage unit 140A in place of the control unit 130 and the storage unit 140, as compared with the interaction device 100.

  The control unit 130A is a part that performs overall control of the interactive apparatus 100A. However, as compared with the control unit 130, the control unit 130A does not necessarily include the voice acquisition unit 131, the image acquisition unit 132, the voice recognition unit 133, the image recognition unit 134, and the specification unit 135. The configuration of the unit 130A can be simplified as compared with the control unit 130.

  The storage unit 140A is a part that stores various information necessary for processing executed by the control unit 130A. However, storage of data that overlaps the storage unit 240 of the server 200 is not essential in the storage unit 140A. Accordingly, the configuration can be simplified by making the storage capacity of the storage unit 140A smaller than the storage capacity of the storage unit 140.

  DESCRIPTION OF SYMBOLS 100,100A ... Dialogue device, 110 ... Input part, 111 ... Sound collecting part (sound acquisition means), 112 ... Imaging part (image acquisition means), 120 ... Output part, 121 ... Sound generation part, 122 ... Display part (output means) , 130, 130A, 230 ... control unit, 131, 231 ... voice acquisition unit (sound acquisition unit), 132, 232 ... image acquisition unit (image acquisition unit), 133, 233 ... voice recognition unit (recognition unit), 134 , 234 ... Image recognition unit, 135, 235 ... Identification unit (identification means), 140, 140A, 240 ... Storage unit, 150, 250 ... Communication unit.

Claims (7)

An interactive device for interacting with a user,
Voice acquisition means for acquiring the user's voice;
Image acquisition means for acquiring images;
A specifying unit for specifying an object included in the image acquired by the image acquiring unit based on the voice of the user acquired by the audio acquiring unit;
An interactive apparatus comprising: Recognizing means for executing a recognition mode for recognizing the voice of the user acquired by the sound acquiring means;
The specifying means specifies the name of the object,
The recognition mode is
A first recognition mode for recognizing the user's voice acquired by the voice acquisition means;
A second recognition mode for recognizing the user's voice based on the voice of the user acquired by the voice acquisition means and the name of the object specified by the specifying means;
The interactive apparatus according to claim 1. The specifying means specifies the name of the object that can be replaced with a part of the user's voice acquired by the voice acquisition means,
In the second recognition mode, after a part of the data corresponding to the user's voice acquired by the voice acquisition unit is replaced with data corresponding to the name of the object specified by the specifying unit, The user's voice is recognized based on the replaced data.
The interactive apparatus according to claim 2. When the recognition unit recognizes that the user's voice acquired by the voice acquisition unit includes a predetermined voice while executing the first recognition mode, the recognition unit executes the recognition mode to be executed. Switching from the first recognition mode to the second recognition mode,
The interactive apparatus according to claim 2 or 3. The specifying unit specifies the target based on position information of the target in the image and a user's voice acquired by the voice acquisition unit.
The interactive apparatus according to any one of claims 1 to 4. Output means for outputting an image,
The image acquisition means acquires the image output by the output means;
The interactive apparatus according to any one of claims 1 to 5. A computer provided in a dialog device for performing a dialog with a user,
Voice acquisition means for acquiring the user's voice;
Image acquisition means for acquiring images;
A specifying unit for specifying an object included in the image acquired by the image acquiring unit based on the voice of the user acquired by the audio acquiring unit;
Interactive program to function as

Images