JP6562790B2 - Dialogue device and dialogue program - Google Patents

Description

  The present invention relates to a dialog device and a dialog program for performing a dialog with a user.

  2. Description of the Related Art Conventionally, as described in Patent Document 1, for example, an interactive device that performs an interaction with a user has been proposed. The dialogue between the user and the dialogue device may be started in a state where the user does not perform a touch operation or the like (hereinafter referred to as “hands-free state”). In this case, the dialogue apparatus starts a dialogue in response to, for example, detection of the user's voice (utterance).

JP 2002-182896 A

  An interactive apparatus may be placed in a noise environment in which noises other than the user's voice, for example, conversations of people other than the user, noise, and noise such as television voices are present. In a noisy environment, there is a possibility that the conversation apparatus may misrecognize the noise as the user's voice and start the conversation. There is also a possibility that the conversation is terminated by mistake due to the recognition error.

  The present invention has been made in view of the above problems, and an object thereof is to provide an interactive apparatus and an interactive program with improved noise tolerance.

  An interactive apparatus according to an aspect of the present invention is an interactive apparatus for performing an interaction with a user, and an acquisition unit that acquires an image of the user and an interaction with the user are started or ended. And determining means for determining the timing for starting the conversation with the user in the hands-free state or the timing for ending the conversation with the user based on the image acquired by the acquiring means.

  In addition, a program according to one embodiment of the present invention provides a computer provided in an interaction device for performing an interaction with a user, an acquisition unit that acquires an image of the user, and a user when starting an interaction with the user or When ending the user's dialogue, based on the image acquired by the acquisition unit, the function to serve as a determination unit that determines the timing for starting the conversation with the user in the hands-free state or the timing for ending the dialogue with the user .

  According to the above-described dialog device or program, the timing for starting the dialog with the user or the timing for ending the dialog with the user is determined based on the user image. Based on the user's image, the timing for starting the dialogue with the user is appropriately determined without being affected by noises such as conversations, noises, and TV voices of persons other than the user. Therefore, it is possible to improve the noise tolerance of the dialog device as compared with the conventional case where the dialog is started in response to detection of the user's voice.

  The determining unit may detect that the user's line of sight is continuously directed to the dialogue apparatus for a predetermined time or more, and may determine the timing of the detection as a timing for starting the dialogue with the user. For example, in a situation where an interactive device is mounted on the face part of a robot that imitates the outline of a human being, when the user's line of sight continues to be directed to the interactive device for a predetermined time or longer, the user interacts. It is likely that you have the intention to start. For this reason, according to the said structure, the timing which starts the dialog with a user can be determined appropriately.

  The determining means may determine the timing when the user's mouth is opened as the timing for starting the dialogue with the user. If the user's mouth opens, the user is likely to start a conversation. For this reason, according to the said structure, the timing which starts the dialog with a user can be determined appropriately.

  The determination unit may detect that the user's mouth is continuously closed for a predetermined time or more, and may determine the timing of the detection as the timing of ending the dialogue with the user. When the user's mouth is continuously closed for a predetermined time or more, there is a high possibility that the user has an intention to end the dialogue. For this reason, according to the said structure, the timing which complete | finishes a dialog with a user can be determined appropriately.

  The acquisition unit further acquires the user's voice, and the dialogue apparatus performs a voice recognition process executed for recognizing the user's voice acquired by the acquisition unit in the dialogue based on the user image acquired by the acquisition unit. And selecting means for selecting from a plurality of predetermined voice recognition processes. Thereby, an appropriate voice recognition process corresponding to the user is selected and executed in the dialogue. As a result, the recognition accuracy of the user's voice in the dialogue can be improved.

  The acquisition unit may include a sound collection unit and an adjustment unit that adjusts the directivity of the sound collection unit toward the user's face based on the image acquired by the acquisition unit. Thereby, the recognition accuracy of the user's voice can be improved even in a noisy environment in which there are conversations, noises, and TV voices of persons other than the user.

  An acquisition unit is configured to capture an image capturing unit that captures a surrounding image of the interactive device, and when a plurality of persons are included in the surrounding image captured by the image capturing unit, a person whose line of sight is directed to the interactive device is selected from the plurality of persons. The adjusting unit may further adjust the directivity of the sound collecting unit toward the face of the user specified by the specifying unit. In the case where there are a plurality of persons, it is highly likely that a person who is looking at the conversation device among the plurality of persons is a user who has an intention to start a conversation. According to the above configuration, a user who has a high possibility of starting such a dialogue is identified, and the directivity of the sound collection unit is adjusted toward the face of the user. Therefore, even when there are a plurality of persons, it is possible to improve the recognition accuracy of the voice of the user who is the subject of the conversation.

  Alternatively, the acquisition unit may further include a specifying unit that specifies a user from a plurality of persons based on face information included in the surrounding image when the acquired surrounding image includes a plurality of persons. Accordingly, it is possible to use image authentication information such as personal facial features, specify a user who is about to start a dialogue, and limit the number of users.

  Alternatively, the dialogue apparatus performs a plurality of predetermined voice recognition processes based on a use history of the voice recognition process performed by the user, based on a use history of the voice recognition process performed by the user. Selecting means for selecting from the above may be further provided. Thereby, the recognition accuracy of the user can be improved by selecting and executing the speech recognition process suitable for the user. In addition, it is possible to eliminate the need for the user to manually select a language.

  According to the present invention, an interactive apparatus and an interactive program with improved noise tolerance are provided.

It is a figure which shows the functional block of a dialogue apparatus. It is a figure which shows the hardware constitutions of a dialogue apparatus. It is a figure which shows the structure of a dialogue program. It is a 1st flowchart which shows an example of the process performed in a dialogue apparatus. It is a 2nd flowchart which shows an example of the process performed in a dialogue apparatus. It is a 3rd flowchart which shows an example of the process performed in a dialogue apparatus. It is a figure which shows the functional block of the dialogue apparatus which concerns on a modification.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant descriptions are omitted.

  The interactive device according to the embodiment is a device that performs a dialog with a user. The interactive device may be realized as a mobile terminal device such as a smartphone or a stationary terminal, or may be realized as a robot imitating a human outer shape. The user can interact with the interaction device in a hands-free state. The hands-free state means a state where the user is not in contact with physical elements (operation buttons, touch panel, etc.) for operating the interactive device. Even when the user is in contact with something other than the above elements, if the user is not in contact with the above elements, it is in a hands-free state. Note that the dialogue apparatus according to the embodiment appropriately determines the start and end timing of the dialogue with the user when the user is in the hands-free state. Therefore, the user is not necessarily in the hands-free state during the conversation.

  FIG. 1 is a diagram illustrating functional blocks of the interactive apparatus according to the embodiment. As illustrated in FIG. 1, the dialogue apparatus 100 includes an input unit 110, an output unit 120, a control unit 130, a storage unit 140, and a communication unit 150.

  The input unit 110 and the output unit 120 are portions (input / output interfaces) for exchanging information with the outside of the interactive apparatus 100, mainly with the user. The input unit 110 accepts input of voice around the interactive apparatus 100 including the user's voice (hereinafter, also simply referred to as “ambient voice”), and images and video around the interactive apparatus 100 including the user ( Hereinafter, the input of “sometimes referred to as“ ambient image ”” is accepted. The output unit 120 outputs various images and videos, and outputs various sounds.

  Specifically, the input unit 110 includes a sound collection unit 111 and an imaging unit 112. The sound collection unit 111 is a part that receives an input of ambient sound. The sound collection unit 111 is constituted by a microphone, for example. The sound collection unit 111 may be configured by a microphone array in which a plurality of microphones are arranged so as to have directivity, for example. The imaging unit 112 is a part that receives an input of a surrounding image. The imaging unit 112 is configured with a camera, for example. The imaging unit 112 may be configured by a plurality of cameras so that the distance to the imaging target can be grasped, for example. Note that the input unit 110 may further include an element such as an operation button for the user to operate the interactive device 100, for example.

  The output unit 120 includes a sound generation unit 121 and a display unit 122. The sound generator 121 is a part that outputs sound. The sound generation unit 121 is configured by a speaker, for example. The sound generation unit 121 may be configured by an array speaker in which a plurality of speakers are arranged so as to have directivity, for example. The display unit 122 is a part that outputs images and videos. The display unit 122 is configured by a display, for example. The display may be configured with a touch panel. In this case, the touch panel also functions as an element for the user to operate the interactive device 100.

  The control unit 130 is a part that performs overall control of the interactive device 100 by controlling each element of the interactive device 100. The control unit 130 includes an acquisition unit 131, a determination unit 132, a voice recognition unit 133, a selection unit 134, an image recognition unit 135, a specification unit 136, and an adjustment unit 137.

  The acquisition unit 131 is a part that acquires the surrounding sound and the surrounding image input to the input unit 110. That is, the acquisition unit 131 and the input unit 110 (including the sound collection unit 111 and the imaging unit 112) function as an acquisition unit that acquires ambient sound and a surrounding image. The specifying unit 136 described later can also constitute part of the acquisition unit. Hereinafter, unless otherwise specified, the acquisition unit is simply referred to as the acquisition unit 131 and described.

  The determination unit 132 is a part (determination unit) that determines the timing for starting or ending the dialog with the user. In particular, the determination unit 132 determines the timing for starting or ending the conversation with the user in the hands-free state, based on the surrounding sound and the surrounding image acquired by the acquiring unit 131. Details of timing determination by the determination unit 132 will be described later.

  The voice recognition unit 133 is a part that executes voice recognition processing for recognizing ambient voice, particularly user voice. The voice recognition unit 133 performs a voice recognition process on the ambient voice acquired by the acquisition unit 131. The voice recognition processing is realized by various known methods including methods using acoustic models and language models prepared in advance. The language model may be a model including a language of a specialized dictionary so that it can correspond to various specialized fields. A plurality of acoustic models and language models may be prepared. In that case, a plurality of speech recognition processes corresponding to the combination of the acoustic model and the language model are prepared in advance, and the speech recognition unit 133 may execute the speech recognition process selected by the selection unit 134 described later.

  The selection unit 134 is a part (selection unit) that selects a speech recognition process to be executed by the speech recognition unit 133 when a plurality of speech recognition processes are prepared. The selection unit 134 selects a voice recognition process to be executed by the voice recognition unit 133 based on the surrounding voice and the surrounding image acquired by the acquisition unit 131.

  For example, the selection unit 134 may estimate a race based on a surrounding image or the like based on a person, in particular, a user's hair color or face, and may select a speech recognition process suitable for the user of the estimated race. . For example, an acoustic model corresponding to a language used by an estimated race user and a speech recognition process corresponding to a combination of language models are selected.

  The selection unit 134 may estimate the gender of the person, particularly the user, based on the surrounding image and the like, and may select a speech recognition process suitable for the estimated gender user. Since gender relates to the voice frequency band, it can be said that the selection unit 134 estimates the voice frequency band of the user. For example, a speech recognition process corresponding to a combination of an acoustic model corresponding to a user of an estimated gender (voice frequency band) and a language model corresponding to spoken language (tone, expression) corresponding to gender is selected.

  Further, the selection unit 134 may select a speech recognition process suitable for the user based on the use history of the speech recognition process by the user. For example, a speech recognition process corresponding to a combination of an acoustic model and a language model with a history used by the user in the past is selected.

  The image recognition unit 135 is a part for recognizing surrounding images, particularly a user image or video. The image recognition unit 135 performs image recognition processing on the surrounding image acquired by the acquisition unit 131. The image recognition processing is realized by various known methods such as open CV (Open Source Computer Vision Library).

  The specifying unit 136 is a part for specifying a user who is an object of dialogue when a plurality of users are included in the surrounding image. For example, the specifying unit 136 directs the line of sight to the interactive device 100 among a plurality of users based on the surrounding image acquired by the imaging unit 112, more specifically, the recognition result of the image recognizing unit 135 for the surrounding image. Identify who is.

  The adjustment unit 137 is a part that adjusts the directivity of the sound collection unit 111. When the sound collection unit 111 is configured by a microphone array, the directivity is adjusted by phase control, for example. Even when the sound collection unit 111 is configured by a single microphone, the directivity can be adjusted by, for example, physically changing the direction of the microphone. The adjustment unit 137 adjusts the directivity of the sound collection unit 111, for example, toward the user's face based on the surrounding image acquired by the acquisition unit 131, more specifically, the recognition result of the image recognition unit 135. To adjust the directivity toward the user's face is to adjust the directivity so that sounds generated in and near the user's face are more easily collected than sounds generated in other parts. .

  Note that the control unit 130 also performs processing for generating various information output by the output unit 120 to the user, for example, data such as sound by the sound generation unit 121, images and video by the display unit 122, for example, in a dialog. Can be executed.

  With the configuration described above, the control unit 130 executes various processes necessary for the interaction apparatus 100 to interact with the user. Details of processing executed by the control unit 130 will be described later with reference to FIGS. 4 to 6.

  The storage unit 140 is a part that stores various information necessary for processing executed by the control unit 130. The storage unit 140 stores, for example, the above-described acoustic model, language model, and usage history. In addition, the storage unit 140 stores a program (interactive program) for causing the interactive apparatus 100 to execute processing necessary for the interactive apparatus 100 to interact with the user.

  In addition, the storage unit 140 stores data (user data) of users (users) who are permitted to use the interactive device 100. The user data may include user user feature data. The feature data may be, for example, data indicating features related to the user's face or data indicating features related to the user's voice. Further, the user data may include data necessary for estimating the race of the user or estimating the gender of the user. The user data in this case may be, for example, data associating race with hair color, facial features, etc., or data associating gender with hair color, facial features, etc. It's okay.

  The communication unit 150 is a part that communicates with the outside of the dialogue apparatus 100. For example, the above-described acoustic model, language model, dialogue program, user data, and the like may be additionally acquired or updated by the communication unit 150.

  Here, the hardware configuration of the interactive apparatus 100 will be described with reference to FIG. As shown in FIG. 2, the interactive apparatus 100 physically has one or more CPUs (Central Processing Units) 21, RAMs (Random Access Memory) 22, ROMs (Read Only Memory) 23, and imaging such as a camera. Device 24, communication module 26 that is a data transmission / reception device, auxiliary storage device 27 such as a semiconductor memory, input device 28 that receives an input of a user operation such as an operation panel (including operation buttons) or a touch panel, an output device 29 such as a display, Moreover, it can be configured as a computer including a reading device 2A such as a CD-ROM drive device. The function of the interactive apparatus 100 in FIG. 1 is controlled by the CPU 21 by, for example, reading one or a plurality of programs stored in a storage medium M such as a CD-ROM with a reading apparatus 2A and taking them into hardware such as a RAM 22. This is realized by operating the imaging device 24, the communication module 26, the input device 28, and the output device 29 under the above, and reading and writing data in the RAM 22 and the auxiliary storage device 27.

  Further, FIG. 3 shows a module of an interactive program for causing a computer to function as the interactive apparatus 100. As shown in FIG. 3, the dialogue program P100 includes an acquisition module P101, a determination module P102, a voice recognition module P103, a selection module P104, an image recognition module P105, a specific module P106, and an adjustment module P107. The functions of the acquisition unit 131, the determination unit 132, the voice recognition unit 133, the selection unit 134, the image recognition unit 135, the specification unit 136, and the adjustment unit 137 described above with reference to FIG. .

  The interactive program is provided by being stored in a storage medium, for example. The storage medium may be a flexible disk, CD-ROM, USB memory, DVD, semiconductor memory, or the like.

  Next, the operation of the interactive device 100 (the interactive method executed by the interactive device 100) will be described with reference to FIGS.

  4 and 5 are flowcharts illustrating an example of processing executed in the interactive apparatus 100. The process of this flowchart is executed when the interactive apparatus 100 starts or ends a dialog with the user. As a premise, the user is assumed to be in a hands-free state at least at the start or end of the dialogue. Unless otherwise specified, each process can be executed by the control unit 130 (that is, regardless of which element is included in the control unit 130).

  First, the interactive device 100 creates a user list during utterance (step S1). The utterance user list is a list of users who are making utterances to interact with the interaction apparatus 100 in the process of this flowchart. The talking user list may be stored in the storage unit 140, for example. Note that at the time when the talking user list is created in step S1, the user is not included in the talking user list, and in step S37 described later, the user is added to the talking user list. Since the process of the flowchart loops, in the second and subsequent flows, the user can be included in the talking user list.

  Next, the interactive device 100 determines the number of people n by face detection (step S2). For example, the image recognition unit 135 recognizes the surrounding image acquired by the acquisition unit 131. Then, for example, the specifying unit 136 detects the faces of persons included in the surrounding image, and determines the number of detected faces as the number of persons n.

  In the next steps S <b> 3 to S <b> 7, the dialogue apparatus 100 specifies the line of sight of each of n persons. Specifically, the interactive apparatus 100 sets the initial value of the variable i to 0 (step S3), increases i by 1 (step S6), and until i becomes n or more (step S7: NO), The following steps S4 and S5 are repeatedly executed for the i-th user.

  That is, the dialogue apparatus 100 identifies an individual by recognizing a face (step S4), and identifies a viewing direction by recognizing a line of sight (step S5). Specifically, the specifying unit 136 specifies a person in the surrounding image based on the recognition result of the image recognition unit 135, and specifies the direction of the line of sight of the specified person (individual).

  After the processes of step S4 and step S5 are completed for each of the n persons (step S7: YES), the dialogue apparatus 100 advances the process to step S8.

  In step S <b> 8, the dialogue apparatus 100 determines whether there is a user whose face is not authenticated in the utterance user list. For example, when a person other than the person specified in the previous step S4 is included in the utterance user list, it may be determined that there is a user whose face is not authenticated in the utterance user list. When there is a user whose face is not authenticated in the utterance user list (step S8: YES), the dialogue apparatus 100 proceeds with the process to step S9. When that is not right (step S8: NO), the dialogue apparatus 100 advances a process to step S10.

  In step S9, the dialogue apparatus 100 deletes the user from the uttering user list and ends the speech recognition (step S9). Specifically, the user who is determined to be a user whose face is not authenticated in the utterance user list in the previous step S8 is deleted from the utterance user list. In addition, the voice recognition unit 133 ends the voice recognition process. Note that the process of step S9 is a process executed when there is a user whose face is not authenticated in the talking user list in step S8. Therefore, the process of step S9 can be executed when a user is added to the utterance user list in step S37, which will be described later, and after the voice recognition process is started, the process of the flowchart loops to reach step S8 again. It is processing.

  In the next steps S <b> 10 to S <b> 24, the dialogue apparatus 100 specifies the number of speaking users among n persons. The uttering user is a user who is considered to have uttered a voice to start a dialogue with the dialogue apparatus 100. Specifically, the interactive apparatus 100 sets the initial values of the variable j and the variable m to 0 (steps S10 and S11), and increases j by 1 (step S23) until j becomes n or more (steps). S24: NO), the following steps S12 to S22 are repeated for the j-th user.

  That is, first, the dialogue apparatus 100 determines whether or not the user is present in the talking user list (step S12). For example, when the j-th person is included in the talking user list, it may be determined that the user is in the talking user list. If the user is in the utterance user list (step S12: YES), the dialogue apparatus 100 advances the process to step S13. When that is not right (step S12: NO), the dialogue apparatus 100 advances a process to step S15.

  In step S13, the dialogue apparatus 100 determines whether or not the mouth is kept closed. For example, the determination unit 132 executes this process based on the recognition result of the image recognition unit 135. For example, it may be determined that the mouth is kept closed when the person's mouth is closed for a predetermined time or longer. When the mouth is kept closed (step S13: YES), the dialogue apparatus 100 proceeds with the process to step S14. When that is not right (step S13: NO), the dialogue apparatus 100 advances a process to step S20.

  In step S14, the interactive device 100 deletes the user from the uttering user list and ends the speech recognition (step S14). Specifically, the jth person is deleted from the speaking user list. In addition, the voice recognition unit 133 ends the voice recognition. Thereby, the dialog with the user is completed. After the process of step S14 is completed, the interactive device 100 proceeds with the process to step S23.

  In step S15, the dialogue apparatus 100 determines whether or not the line of sight faces a predetermined direction. This process is executed by the determining unit 132 or the specifying unit 136 based on the line-of-sight direction specified in step S5 described above, for example. The predetermined direction may be a direction toward the interactive apparatus 100. For example, when the direction of the line of sight is continuously directed to the interactive apparatus 100 for a predetermined time or more, it may be determined that the direction of the line of sight of the person is the predetermined direction. The predetermined time may be about several seconds. When the line of sight is directed in a predetermined direction (step S15: YES), the dialogue apparatus 100 proceeds with the process to step S16. When that is not right (step S15: NO), the dialogue apparatus 100 determines that the person is a user who has no intention to speak (step S17), and proceeds to step S23.

  In step S <b> 16, the dialogue apparatus 100 determines whether or not the person is a usage-permitted user. For example, the determining unit 132 or the specifying unit 136 determines whether or not the user is a user by collating the recognition result of the image recognition unit 135 with the user data stored in the storage unit 140. If the person is a use-permitted user (step S16: YES), the dialogue apparatus 100 proceeds with the process to step S18. Otherwise (step S16: NO), the dialogue apparatus 100 determines that the person is a non-permitted user (step S19), and proceeds to step S23.

  In step S18, the dialogue apparatus 100 determines whether or not the mouth has started to move. For example, the determination unit 132 executes this process based on the recognition result of the image recognition unit 135. For example, when the person's mouth changes from a closed state to an open state, it may be determined that the person's mouth has started to move. When the mouth starts to move (step S18: YES), more specifically, the line of sight of the person is facing a predetermined direction (step S15: YES), and the person is a use-permitted user (step S16: YES). When the person's mouth starts to move (step S18: YES), the dialogue apparatus 100 determines that the person is a speaking user (step S20), increases the variable m by 1 (step S21), and proceeds to step S23. Proceed with the process. That is, the variable m is a value indicating the number of speaking users. On the other hand, when the person's mouth has not started to move (step S18: NO), the dialogue apparatus 100 advances the process to step S22.

  In step S22, the dialogue apparatus 100 determines whether or not a speech utterance has been detected (step S22). This process is executed in the same manner as a conventional dialogue apparatus using the functions of the sound collection unit 111, the acquisition unit 131, and the voice recognition unit 133, for example. When a speech utterance is detected (step S22: YES), the dialogue apparatus 100 proceeds with the process to step S20 described above. When that is not right (step S22: NO), the dialogue apparatus 100 advances a process to step S23.

  After the processes in steps S12 to S22 are completed for each of n persons (step S24: YES), the dialogue apparatus 100 executes a dialogue start process (step S25).

  FIG. 6 is a flowchart showing an example of processing executed in the dialogue start processing (step S25 in FIG. 5).

  In the next steps S31 to S39, the dialogue apparatus 100 performs speech recognition and the like after performing microphone adjustment and optimization of an acoustic model and a language model for each of m speech users. Specifically, the interactive apparatus 100 sets the initial value of the variable k to 0 (step S31), increases k by 1 (step S38), and until k becomes m or more (step S39: NO), The following steps S32 to S37 are executed for the k-th utterance user.

  That is, first, the dialogue apparatus 100 derives the microphone direction from the face position of the video (or image) (step S32), and controls the microphone direction (step S33). Specifically, the adjustment unit 137 adjusts the directivity of the sound collection unit 111 toward the uttering user's face based on the recognition result of the image recognition unit 135. In the case where the sound collection unit 111 is a microphone array, the directivity may be adjusted in each of the directions of all the utterance users so that the voices of all the utterance users can be recognized simultaneously. When the sound collection unit 111 is a single microphone, for example, the directivity of the microphone may be adjusted with respect to the user who speaks in the first loop (k = 0).

  Next, the dialogue apparatus 100 determines whether or not the user state is already speaking (step S34). For example, if voice recognition has been started in step S37 in the previous loop and a voice utterance has been detected, it may be determined that the user state is already speaking. When the user state is already speaking (step S34: YES), the dialogue apparatus 100 continues the voice recognition (step S35), and advances the process to step S38. When that is not right (step S34: NO), the dialogue apparatus 100 advances the process to step S36.

  In step S36, the dialogue apparatus 100 reads user data and determines an acoustic model and a language model. Specifically, the selection unit 134 matches the user image (face image) recognized in the previous step S4 (FIG. 4) with the user data stored in the storage unit 140, thereby allowing the user to speak. Determine suitable acoustic and language models. The selection unit 134 may determine an acoustic model and a language model suitable for the uttering user based on the usage history stored in the storage unit 140.

  Then, the dialogue apparatus 100 adds to the talking user list and starts speech recognition (step S37). Specifically, the k-th utterance user is added to the utterance user list. In addition, the voice recognition unit 133 starts a voice recognition process. After that, if k becomes m or more through the process of step S38 (step S39: YES), the interactive device 100 returns the process to step S2 (FIG. 4) again.

  Next, the function and effect of the interactive device 100 will be described. According to the interaction device 100, the timing for starting or ending the interaction with the user is determined based on the user's image (or video). Based on the user's image, the timing for starting the dialogue with the user is appropriately determined without being affected by noises such as conversations, noises, and TV voices of persons other than the user. Therefore, it is possible to improve the noise tolerance of the dialog device as compared with the conventional case where the dialog is started in response to detection of the user's voice.

  Specifically, the determination unit 132 detects that the user's line of sight has been continuously directed to the dialogue apparatus 100 for a predetermined time (step S15: YES), and starts a dialogue with the user at the detection timing. The timing is determined (step S25). When the user's line of sight is continuously directed to the dialogue apparatus 100 for a predetermined time or more, there is a high possibility that the user has an intention to start the dialogue. For this reason, according to the said process of the determination part 132, the timing which starts the dialog with a user can be determined appropriately.

  Moreover, the determination part 132 determines the timing which a user's mouth opened as a timing which starts a dialog with a user (step S18: YES, step S25). If the user's mouth opens, the user is likely to start a conversation. For this reason, according to the said process of the determination part 132, the timing which starts the dialog with a user can be determined appropriately.

  Further, the determination unit 132 detects that the user's mouth is continuously closed for a predetermined time or more, and determines the timing of the detection as the timing of ending the dialogue with the user (step S13: YES, step S14). ). When the user's mouth is continuously closed for a predetermined time or more, there is a high possibility that the user has an intention to end the dialogue. For this reason, according to the said process of the determination part 132, the timing which complete | finishes a dialog with a user can be determined appropriately.

  In addition, the voice recognition processing executed for the selection unit 134 to recognize the user's voice in the dialogue is based on a plurality of predetermined voice recognition processes based on the user's image or video acquired by the acquisition unit 131. (Step S36). Thereby, an appropriate voice recognition process corresponding to the user is selected and executed in the dialogue. As a result, the recognition accuracy of the user's voice in the dialogue can be improved.

  For example, the selection unit 134 estimates the race of the user and selects a speech recognition process suitable for the user of the estimated race. Thereby, an appropriate speech recognition process according to the race of the user is selected. For example, the accuracy of speech recognition can be improved by selecting an acoustic model corresponding to the language used by the estimated race user and a speech recognition process corresponding to the combination of language models. In addition, it is possible to eliminate the need for the user to manually select a language.

  For example, the selection unit 134 estimates the gender of the user and selects a speech recognition process suitable for the estimated gender user. Since gender relates to, for example, a voice frequency band, the accuracy of voice recognition can be improved by selecting a voice recognition process using an acoustic model suitable for the estimated user's voice frequency band of gender. Moreover, the accuracy of speech recognition can be improved by selecting speech recognition processing using a language model corresponding to the spoken language (tone, expression) corresponding to the estimated gender. Of course, the accuracy of speech recognition can be further improved by selecting speech recognition processing corresponding to the combination of the acoustic model and the language model.

  Further, the adjustment unit 137 adjusts the directivity of the sound collection unit 111 toward the user's face (steps S32 and S33). Thereby, the recognition accuracy of the user's voice can be improved even in a noisy environment in which there are conversations, noises, and TV voices of persons other than the user.

  In addition, when a plurality of persons are included in the surrounding image captured by the imaging unit 112, the specifying unit 136 specifies, as a user, a person whose line of sight is directed to the interactive device 100 among the plurality of persons (Step S15). : YES, step S20). Then, the adjusting unit 137 adjusts the directivity of the sound collecting unit 111 toward the user's face specified by the specifying unit 136 (steps S32 and S33). When there are a plurality of persons, it is highly likely that a person who is looking at the conversation apparatus 100 among the plurality of persons is a user who has an intention to start a conversation. According to the above processing of the specifying unit 136 and the adjusting unit 137, a user who has a high possibility of starting such a dialogue is specified, and the sound collection unit is directed toward the user's face. Sex is adjusted. Therefore, even when there are a plurality of persons, it is possible to improve the recognition accuracy of the voice of the user who is the subject of the conversation.

  In addition, the specifying unit 136 specifies the user who uses the user's face included in the surrounding image or the user's voice included in the surrounding sound and the user data stored in the storage unit 140 ( Step S16: YES). Thereby, it is possible to specify the uttering user by using the image authentication information such as personal facial features (step S16: YES, steps S20 and S21).

  Note that iris recognition may be used instead of authentication based on facial features and the like. In that case, the process (step S15) for specifying the line of sight may be omitted.

  In addition, the selection unit 134 refers to the history information stored in the storage unit 140 and selects an acoustic model and a language model having a usage history, so that the sound corresponding to the combination of the acoustic model and the language model suitable for the user is selected. A recognition process is selected (step S36). Also by this, the recognition accuracy of the user's voice can be improved. In addition, it is possible to eliminate the need for the user to manually select a language.

  Each function of the interactive apparatus 100 described above can also be realized, for example, by executing an interactive program on a computer.

  Although one embodiment of the present invention has been described above, the present invention is not limited to the above embodiment.

  FIG. 7 is a diagram illustrating functional blocks of the interactive apparatus according to the modification. The dialogue apparatus 100 </ b> A constitutes a dialogue system 1 that performs dialogue with the user in cooperation with the server 200. In this modification, the dialogue system 1 corresponds to the dialogue device according to the present invention.

  As illustrated in FIG. 7, the server 200 includes a control unit 230, a storage unit 240, and a communication unit 250.

  The control unit 230 includes an acquisition unit 231, a determination unit 232, a voice recognition unit 233, a selection unit 234, an image recognition unit 235, a specification unit 236, and an adjustment unit 237. Each of these elements has the same functions as those of the acquisition unit 131, the determination unit 132, the voice recognition unit 133, the selection unit 134, the image recognition unit 135, the specifying unit 136, and the adjustment unit 137 described above with reference to FIG. Have.

  The storage unit 240 has the same function as the storage unit 140 described above with reference to FIG. In other words, the storage unit 240 is a part that stores various information necessary for processing executed by the control unit 230, and stores, for example, an acoustic model, a language model, a dialogue program, user data, and a usage history.

  The communication unit 250 is a part that communicates with the communication unit 150 of the interactive apparatus 100A. The communication unit 250 enables communication between the dialogue apparatus 100A and the server 200.

  With the above configuration, the dialogue system 1 enables dialogue with the user through cooperation between the dialogue apparatus 100 </ b> A and the server 200. That is, in the interactive system 1, a part of the processing (particularly, processing executed by the control unit 130) executed for performing a dialog with the user in the interactive device 100 (FIG. 1) is executed by the server 200. Therefore, according to the interactive device 100A, the processing load on the interactive device can be reduced as compared with the interactive device 100.

  Specifically, the interaction device 100A may include a control unit 130A and a storage unit 140A in place of the control unit 130 and the storage unit 140, as compared with the interaction device 100.

  The control unit 130A is a part that performs overall control of the interactive apparatus 100A. However, it is essential that the control unit 130A includes the acquisition unit 131, the determination unit 132, the voice recognition unit 133, the selection unit 134, the image recognition unit 135, the specifying unit 136, and the adjustment unit 137, as compared with the control unit 130. Accordingly, the configuration of the control unit 130 </ b> A can be simplified as compared with the control unit 130.

  The storage unit 140A is a part that stores various information necessary for processing executed by the control unit 130A. However, storage of data that overlaps the storage unit 240 of the server 200 is not essential in the storage unit 140A. Accordingly, the configuration can be simplified by making the storage capacity of the storage unit 140A smaller than the storage capacity of the storage unit 140.

  DESCRIPTION OF SYMBOLS 100,100A ... Dialogue device, 110 ... Input part (acquisition means), 111 ... Sound collection part (acquisition means), 112 ... Imaging part (acquisition means), 120 ... Output part, 121 ... Sound generation part, 122 ... Display part, 130, 130A, 230 ... control unit, 131, 231 ... acquisition unit (acquisition unit), 132, 232 ... determination unit (determination unit), 133, 233 ... voice recognition unit, 134, 234 ... selection unit (selection unit), 135, 235 ... image recognition unit, 136, 236 ... identification unit (acquisition means), 137, 237 ... adjustment unit, 140, 140A, 240 ... storage unit, 150 ... communication unit, 200 ... server.

Claims (9)

An interactive device for interacting with a user,
Obtaining means for obtaining an image of the user;
When starting a dialog with the user or when ending the dialog with the user, based on the image acquired by the acquisition means, the timing for starting the dialog with the user in a hands-free state or the user Determining means for determining when to end the dialogue with
Equipped with a,
The start of the dialog includes starting a voice recognition process for recognizing the voice of the user, and the end of the dialog includes ending the voice recognition process,
The dialog device detects that the mouth of the user is continuously closed for a predetermined time or more, and determines the timing of the detection as a timing to end the dialog with the user . The determining means detects that the user's line of sight has been continuously directed to the interactive device for a predetermined time or more, and determines the timing of the detection as a timing for starting a conversation with the user.
The interactive apparatus according to claim 1. The determining means determines the timing when the user's mouth is opened as the timing for starting a dialogue with the user.
The interactive apparatus according to claim 1 or 2. The acquisition means further acquires the user's voice,
The dialogue apparatus performs a voice recognition process executed for recognizing the voice of the user acquired by the acquisition unit in the dialogue based on the image of the user acquired by the acquisition unit. Selecting means for selecting from a plurality of voice recognition processes;
The dialogue apparatus according to any one of claims 1 to 3 . The acquisition means includes
The sound collection section;
An adjustment unit that adjusts the directivity of the sound collection unit toward the user's face based on the image acquired by the acquisition unit;
including,
The interactive apparatus according to any one of claims 1 to 4 . The acquisition means includes
An imaging unit that captures an image around the interactive device;
A plurality of persons included in the surrounding image captured by the image capturing unit; a specific unit that identifies a person who is looking at the interactive device as the user among the plurality of persons;
Further including
The adjusting unit adjusts the directivity of the sound collecting unit toward the face of the user specified by the specifying unit;
The interactive apparatus according to claim 5 . The acquisition means includes
A specifying unit that specifies the user from the plurality of persons based on face information included in the surrounding image when the acquired surrounding image includes a plurality of persons;
Further including
The interactive apparatus according to any one of claims 1 to 5 . A voice recognition process executed for recognizing the user's voice acquired by the acquisition means in the dialogue is selected from a plurality of predetermined voice recognition processes based on a use history of the voice recognition process by the user. Further comprising selection means for
The dialogue apparatus according to any one of claims 1 to 3 . A computer provided in a dialog device for performing a dialog with a user,
Obtaining means for obtaining an image of the user;
When starting a dialog with the user or when ending the dialog with the user, based on the image acquired by the acquisition means, the timing for starting the dialog with the user in a hands-free state or the user Determining means for determining when to end the dialogue with
An interactive program to function as,
The start of the dialog includes starting a voice recognition process for recognizing the voice of the user, and the end of the dialog includes ending the voice recognition process,
The said determination means detects that the said user's mouth was closed continuously more than predetermined time, and determines the timing of the said detection as a timing which complete | finishes the dialog with the said user .

Images