JP6819672B2 - Information processing equipment, information processing methods, and programs - Google Patents

Description

The present disclosure relates to information processing devices, information processing methods, and programs.

In recent years, various devices have been proposed that recognize a user's utterance or action and execute a process corresponding to the recognized input. Further, there is known a technique of giving feedback to an input from a user to improve the recognition accuracy of the input. As the above-mentioned technique, for example, there is an information processing apparatus described in Patent Document 1.

Japanese Unexamined Patent Publication No. 2011-209787

However, in the information processing apparatus described in Patent Document 1, the feedback method for the input is fixed, and it is difficult to respond according to the user's situation.

Therefore, this disclosure proposes an information processing device, an information processing method, and a program capable of realizing a more natural dialogue between a user and a system.

According to the present disclosure, a control unit that selects a feedback mode for the user's utterance mode from a plurality of modes according to information related to the user's utterance recognition is provided, and the plurality of modes provide implicit feedback. An information processing apparatus is provided that includes a first mode in which the above is performed and a second mode in which explicit feedback is performed.

As described above, according to the present disclosure, it is possible to realize a more natural dialogue between the user and the system. It should be noted that the above effects are not necessarily limited, and together with or in place of the above effects, any of the effects shown herein, or any other effect that can be grasped from this specification. May be played.

It is a figure which shows the system configuration example which concerns on this disclosure. It is a functional block diagram of the input / output terminal which concerns on this disclosure. It is a functional block diagram of the information processing apparatus which concerns on this disclosure. It is a figure which shows the example of the dialogue between an input / output terminal and a user which concerns on embodiment of this disclosure. It is a figure which shows the flow of control of the implicit feedback which concerns on this embodiment. It is a figure for demonstrating the volume level of the input / output terminal which concerns on this embodiment. It is a figure which shows the flow of feedback control based on the number of recognition trials which concerns on this embodiment. It is a figure which shows the flow of feedback control based on the degree of change of the utterance mode which concerns on this embodiment. It is a figure which shows the flow of the feedback control with respect to the utterance position and the utterance direction which concerns on the same embodiment. It is a figure for demonstrating the flow of the control of the explicit feedback which added the improvement reason which concerns on the same actual embodiment. It is a figure which shows the flow of the additional control of feedback by the visual information which concerns on the same embodiment. This is an example of feedback using the indicator according to the same embodiment. This is an example of feedback using the indicator according to the same embodiment. This is an example of feedback using an avatar according to the same embodiment. This is an example of feedback using an avatar according to the same embodiment. This is an example of feedback using an avatar according to the same embodiment. This is an example of feedback using a graphic according to the same embodiment. This is an example of feedback using a graphic according to the same embodiment. This is an example of feedback using a graphic according to the same embodiment. This is an example of feedback using an avatar according to the same embodiment. This is an example of feedback using an avatar according to the same embodiment. This is an example of feedback using an avatar according to the same embodiment. This is an example of feedback using a graphic according to the same embodiment. This is an example of feedback using a graphic according to the same embodiment. It is a figure which shows the hardware configuration example of an input / output terminal and an information processing apparatus.

Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, so that duplicate description will be omitted.

The explanations will be given in the following order.
1. 1. Control of feedback according to the present disclosure 1.1. Feedback in speech recognition technology 1.2. System configuration example according to the present disclosure 1.3. Input / output terminal 10 according to the present disclosure
1.4. Information processing device 30 according to the present disclosure
2. 2. Embodiment 2.1. Feedback mode 2.2. Examples of implied feedback 2.3. Switching modes related to feedback 2.4. Explicit feedback with reasons for improvement 2.5. Additional control of feedback by visual information 2.6. Example of feedback by visual information 3. Hardware configuration example of the input / output terminal 10 and the information processing device 30 4. Summary

<1. Control of feedback relating to this disclosure>
<< 1.1. Feedback in speech recognition technology >>
In recent years, various devices using voice recognition technology have been provided. Devices using voice recognition technology are widely used in information processing devices such as PCs (Personal Computers) and smartphones, as well as in home appliances and in-vehicle devices, which are close to general consumers. In addition, devices using voice recognition technology are expected to be used in the future as an entity that provides services to customers in place of people in commercial facilities and public facilities.

In a device as described above, for example, it is possible to recognize a user's utterance and execute application processing based on the recognized utterance content. In addition, the above-mentioned device may have a function of giving feedback to the user according to the recognition result of the user's utterance. The feedback is provided, for example, by displaying a text message "Voice is low" on the display of the device when the device cannot recognize the user's utterance. By confirming the message, the user can perceive that the uttered content has not been recognized by the device and take the next action. In this way, the device using the voice recognition technology can request the user to improve the utterance by giving feedback on the utterance. At this time, the clearer the content of the feedback by the device, the higher the possibility that the user's utterance can be improved.

On the other hand, the feedback provided by the device greatly affects the user's impression of the device. For example, if the feedback provided by the device is always clear, the user may have the impression that the device is "cold" or "hard". This is for the user to compare the actual human interaction with the device interaction. Since the dialogue between humans changes according to the situation, the user feels unnatural for a device that always gives constant feedback. In addition, the above impression may lead to the evaluation that the technical level of the device is low. Furthermore, when the feedback from the device is clear, it is assumed that some users feel that they are instructed by the device and feel disgusted with the device.

The information processing device, information processing method, and program according to the present disclosure are conceived by paying attention to the above points, and a feedback mode for the user's utterance mode is set according to the information related to the user's utterance recognition. One of the features is to select from multiple modes. In addition, the plurality of modes described above include a first mode in which implicit feedback is performed and a second mode in which explicit feedback is performed. In the following description, the effects of the information processing device, the information processing method, and the program according to the present disclosure will be described with reference to the features.

<< 1.2. System configuration example related to this disclosure >>
First, a configuration example of the information processing system according to the present disclosure will be described with reference to FIG. Referring to FIG. 1, the information processing system according to the present disclosure includes an input / output terminal 10 and an information processing device 30. Further, the input / output terminal 10 and the information processing device 30 are connected so as to be able to communicate with each other via the network 20.

Here, the input / output terminal 10 may be a terminal that collects the user's utterance and presents the processing result of the application based on the utterance to the user. Further, the input / output terminal 10 may have a function of giving feedback to the user's utterance. FIG. 1 shows an example in which the input / output terminal 10 gives feedback by voice output such as "May I help you?" To the utterance "Hello." Of the user P1.

Further, the information processing device 30 according to the present disclosure has a function of controlling feedback executed by the input / output terminal 10 with respect to the utterance based on the utterance of the user P1 picked up by the input / output terminal 10. The information processing device 30 acquires the utterance of the user P1 picked up by the input / output terminal 10 via the network 20, and sets the feedback mode for the utterance mode from the plurality of modes according to the information based on the utterance. You may choose. In addition, the plurality of modes described above may include a first mode in which implicit feedback is provided and a second mode in which explicit feedback is provided.

Further, the network 20 may include various LANs (Local Area Network) including Ethernet (registered trademark), WAN (Wide Area Network) and the like. The network 20 may be a dedicated line network such as IP-VPN (Internt Protocol-Virtual Private Network).

In the following description, a system configuration including an input / output terminal 10 and an information processing device 30 will be described as an example, but the configuration of the information system according to the present disclosure is not limited to such an example. For example, the information processing device 30 may have a function of picking up a user's utterance and executing feedback on the utterance. When the information processing device 30 has the above-mentioned input / output function, the information processing device 30 can play a role as an input / output terminal 10 at the same time as controlling feedback.

Further, the function of the application processed based on the result of voice recognition may be executed by the input / output terminal 10 or the information processing device 30. The processing of the application based on the user's utterance can be appropriately changed depending on the specifications of the application, the input / output terminal 10, and the information processing device 30.

<< 1.3. Input / output terminal 10 according to the present disclosure >>
Next, the input / output terminal 10 according to the present disclosure will be described in detail. As described above, the input / output terminal 10 according to the present disclosure has a function of collecting the utterance of the user. Further, the input / output terminal 10 has a function of presenting to the user feedback information controlled by the information processing apparatus 30 according to the information related to the user's utterance recognition.

The input / output terminal 10 according to the present disclosure can be realized as various devices having the above functions. The input / output terminal 10 according to the present disclosure may be, for example, a dedicated agent that executes various processes based on the utterance of the user. The agent may include an interactive robot, a vending machine, a voice guidance device, and the like. Further, the input / output terminal 10 according to the present disclosure may be an information processing terminal such as a PC (Personal Computer), a tablet, or a smartphone. Further, the input / output terminal 10 may be a device used by being incorporated in a building or a vehicle. The input / output terminal 10 according to the present disclosure can be widely applied to various devices to which the voice recognition function is applied.

Hereinafter, a functional configuration example of the input / output terminal 10 according to the present disclosure will be described with reference to FIG. Referring to FIG. 2, the input / output terminal 10 according to the present disclosure includes a voice input unit 110, a sensor unit 120, a voice output unit 130, a display unit 140, a terminal control unit 150, and a server communication unit 160.

(Voice input unit 110)
The voice input unit 110 has a function of collecting user's utterances and environmental sounds. The voice input unit 110 may include a microphone that converts the user's utterance and environmental sound into an electric signal. Further, the voice input unit 110 may include a microphone array having a directivity for collecting sound in a specific direction. With the microphone array as described above, the voice input unit 110 can also collect the user's utterance separately from the environmental sound. Further, the voice input unit 110 may include a plurality of microphones and microphone arrays. With this configuration, it is possible to detect the position, orientation, movement, etc. of the sound source with higher accuracy.

(Sensor unit 120)
The sensor unit 120 has a function of detecting various information about an object including a user. The sensor unit 120 may include a plurality of sensors for detecting the above information. The sensor unit 120 may include an image sensor for detecting the user's movement, an infrared sensor, a temperature sensor, and the like. Further, the sensor unit 120 may have a function of performing image recognition based on the captured image. For example, the sensor unit 120 can identify the user who is speaking by detecting the movement of the user's mouth.

(Audio output unit 130)
The voice output unit 130 has a function of converting an electric signal into sound and outputting it. Specifically, the voice output unit 130 has a function of giving feedback to the user by voice output based on the feedback information controlled by the information processing device 30. The audio output unit 130 may include a speaker having the above functions. Further, the speaker included in the audio output unit 130 may have a function of realizing audio output having directivity in a specific direction or distance. By including the speaker having the function, the voice output unit 130 can perform voice output according to the position of the user detected by the sensor unit 120, for example. Further, the audio output unit 130 may include a plurality of speakers. When the audio output unit 130 includes a plurality of speakers, it is possible to execute feedback according to the position of the user by controlling the speakers that output feedback. The details of the function will be described later.

Further, the voice output unit 130 may have a function of performing voice synthesis based on the feedback information controlled by the information processing device 30. For example, the voice output unit 130 may perform text reading (TTS: Text To Speech) based on the text information acquired from the information processing apparatus 30.

(Display unit 140)
The display unit 140 has a function of giving feedback to the user by visual information based on the feedback information controlled by the information processing device 30. The function may be realized by, for example, a CRT (Cathode Ray Tube) display device, a liquid crystal display (LCD: Liquid Crystal Display) device, or an OLED (Organic Light Emitting Mode) device. Further, the display unit 140 may have a function as an operation unit that accepts a user's operation. The function as an operation unit can be realized by, for example, a touch panel.

(Terminal control unit 150)
The terminal control unit 150 has a function of controlling each configuration of the input / output terminal 10 described above. The terminal control unit 150 may have, for example, a function of acquiring various information detected by the voice input unit 110 and the sensor unit 120 and transmitting the information to the information processing device 30 via the server communication unit 160 described later. Further, the terminal control unit 150 may acquire information related to feedback from the information processing device 30 via the server communication unit 160, and control the voice output unit 130 and the display unit 140 based on the information. Further, when the input / output terminal 10 is provided with an application that executes processing based on the user's utterance, the terminal control unit 150 can control the processing of the application.

The input / output terminal 10 according to the present disclosure has been described above. In the above description, an example is taken in the case where the input / output terminal 10 has both a function of accepting an input such as a user's utterance and a function of presenting feedback to the user according to information related to the user's utterance recognition. explained. On the other hand, the system configuration according to the present disclosure is not limited to such an example. The system according to the present disclosure may separately include an input terminal having an input function and an output terminal that presents the above feedback. In this case, the input terminal may include, for example, the functions of the voice input unit 110, the sensor unit 120, and the terminal control unit 150 described above. Further, the output terminal may include, for example, the functions of the audio output unit 130, the display unit 140, and the terminal control unit 150 described above. The system configuration according to the present disclosure can be flexibly modified.

(Server communication unit 160)
The server communication unit 160 has a function of performing information communication with the information processing device 30 via the network 20. Specifically, the server communication unit 160 transmits the information acquired by the voice input unit 110 and the sensor unit 120 to the information processing device 30 based on the control of the terminal control unit 150. Further, the server communication unit 160 delivers the feedback information acquired from the information processing device 30 to the terminal control unit 150.

<< 1.4. Information processing device 30 according to the present disclosure >>
Next, the information processing device 30 according to the present disclosure will be described in detail. The information processing device 30 according to the present disclosure has a function of controlling the feedback executed by the input / output terminal 10 with respect to the user's utterance mode according to the information related to the user's utterance recognition collected by the input / output terminal 10. .. The information processing device 30 can select a feedback mode for the user's utterance mode from a plurality of modes according to the information related to the user's utterance recognition. The plurality of modes described above may include a first mode in which implicit feedback is provided and a second mode in which explicit feedback is provided. In addition, the above-mentioned utterance mode may include utterance volume, utterance speed, utterance pitch, pronunciation clarity, utterance position, utterance direction, utterance content, and environmental sound.

The information processing device 30 according to the present disclosure may be a server having the above functions. Further, the information processing device 30 may be various agents, PCs, tablets, or smartphones that detect the user's utterance and execute feedback.

Hereinafter, a functional configuration example of the information processing apparatus 30 according to the present disclosure will be described with reference to FIG. Referring to FIG. 3, the information processing device 30 according to the present disclosure includes a terminal communication unit 310, a voice analysis unit 320, a voice recognition unit 330, a state storage unit 340, a position detection unit 350, and an output control unit 360.

(Terminal communication unit 310)
The terminal communication unit 310 has a function of performing information communication with the input / output terminal 10 via the network 20. Specifically, the terminal communication unit 310 delivers various information acquired from the input / output terminal 10 to the voice analysis unit 320, the voice recognition unit 330, and the position detection unit 350. Further, the terminal communication unit 310 has a function of acquiring feedback information controlled by the output control unit 360 and transmitting it to the input / output terminal 10. When the information processing device 30 controls a plurality of input / output terminals 10, the terminal communication unit 310 may perform information communication with the plurality of input / output terminals 10 via the network 20.

(Voice analysis unit 320)
The voice analysis unit 320 has a function of acquiring information collected by the input / output terminal 10 and analyzing the information. The voice analysis unit 320 can analyze information on the user's utterance mode, including, for example, the user's utterance volume, utterance speed, utterance pitch, or pronunciation clarity. In addition, the user's utterance mode may include environmental sounds collected in association with the user's utterance. Further, the voice analysis unit 320 may have a function of separating the user's utterance and the environmental sound from the information collected by the input / output terminal 10. Separation of the user's utterance and the environmental sound may be performed based on the information of the frequency band related to the human voice, or may be realized by VAD (Voice Activity Detection) technology or the like. Further, when the state storage unit 340 described later stores personal information regarding the voice of a predetermined user, the voice analysis unit 320 can also use the information to separate the user's utterance and the environmental sound. ..

Further, the voice analysis unit 320 may have a function of analyzing the utterance of the user picked up by the input / output terminal 10 and identifying the user. The user may be specified by comparing the analysis result of the user's utterance with the user's voiceprint information stored in the state storage unit 340 described later.

Further, the voice analysis unit 320 may have a function of analyzing the utterance of the user picked up by the input / output terminal 10 and estimating the emotion of the user. The estimation of the user's emotion may be performed by, for example, analysis of prosody, amplitude, stress, and the like.

(Voice recognition unit 330)
The voice recognition unit 330 has a function of recognizing a user's utterance based on the voice picked up by the input / output terminal 10 or the user's voice separated by the voice analysis unit 320. Specifically, the voice recognition unit 330 may have a function of converting the acquired voice information into phonemes and converting it into text. Since various methods may be used for voice recognition by the voice recognition unit 330, detailed description thereof will be omitted.

(State storage unit 340)
The state storage unit 340 has a function of storing the processing results of the voice analysis unit 320 and the voice recognition unit 330. The state storage unit 340 can store, for example, information related to the user's utterance mode analyzed by the voice analysis unit 320 and the result of voice recognition by the voice recognition unit 330. Further, the state storage unit 340 may store the user's attribute information including the characteristics related to the user's voice, the tendency of the utterance mode, and the like.

(Position detector 350)
The position detection unit 350 has a function of estimating the utterance position and the utterance direction of the user based on the information acquired by the input / output terminal 10. The position detection unit 350 uses the user's utterance position and utterance based on the voice information collected by the voice input unit 110 of the input / output terminal 10 and the information collected from various sensors including the image information acquired by the sensor unit 120. The direction can be estimated. Further, the position detection unit 350 may estimate the position of a person or an object other than the user who is speaking based on the above information.

(Output control unit 360)
The output control unit 360 has a function of acquiring various information from the voice analysis unit 320, the voice recognition unit 330, the state storage unit 340, and the position detection unit 350, and controlling the feedback to the user's utterance. The output control unit 360 selects a feedback mode for the user's utterance mode from a plurality of modes based on the above information. The plurality of modes include a first mode in which implicit feedback is provided and a second mode in which explicit feedback is provided. Further, the output control unit 360 may generate feedback information by voice or visual information performed by the input / output terminal 10 and transmit it to the input / output terminal 10 via the terminal communication unit 310. The output control unit 360 may generate the above feedback information by searching the output DB 370, which will be described later, for feedback information based on the conditions. The details of the feedback control by the output control unit 360 will be described later.

(Output DB370)
The output DB 370 may be a database that stores feedback information by voice or visual information performed by the input / output terminal 10. The output DB 370 may store, for example, voice information related to feedback, or may store text information for voice output by the voice synthesis function of the input / output terminal 10. Further, the output DB 370 may store image information and text information related to feedback by visual information performed by the input / output terminal 10.

<2. Embodiment>
<< 2.1. Feedback mode >>
The control of feedback according to the present disclosure has been outlined above. Subsequently, the control of feedback according to the embodiment of the present disclosure will be described in detail. The information processing device 30 according to the present embodiment can select a feedback mode for the user's utterance mode from a plurality of modes according to the information related to the user's utterance recognition.

(Implicit feedback)
Implicit feedback is feedback that includes indirect improvement methods for the user's utterance mode. That is, in the implicit feedback, the feedback is provided by changing the output mode of the input / output terminal 10 without directly presenting the user a method of improving the utterance mode. Here, the implicit feedback according to the present embodiment may be defined as feedback in an utterance mode with higher recognition accuracy than the user's utterance mode. The above recognition accuracy may be the recognition accuracy of the user's utterance by the input / output terminal 10. In other words, in the first mode in which the implicit feedback is performed, the feedback is performed according to the utterance mode expected from the user.

For example, when the user's utterance volume is low, in the first mode, fordback may be performed by voice output having a volume higher than the user's utterance volume. Further, for example, when the user's utterance speed is too fast, in the first mode, feedback by voice output may be performed at a speed slower than the user's utterance speed. Further, in the first mode according to the present embodiment, feedback in the opposite direction to the above can be performed. That is, when the user's utterance volume is high, in the first mode, fordback may be performed by voice output having a volume lower than that of the user's utterance volume. Further, for example, when the user's utterance speed is too slow, in the first mode, feedback by voice output may be performed at a speed faster than the user's utterance speed.

FIG. 4 is a diagram showing an example of dialogue between the user and the input / output terminal 10 using implicit feedback. In FIG. 4, the horizontal axis shows the passage of time, and the vertical axis shows the loudness of the utterance volume. Here, the unit of time lapse related to the horizontal axis may be, for example, milliseconds (msec). Further, the unit of the utterance volume related to the vertical axis may be, for example, a decibel (dB). In the example shown in FIG. 4, the user first makes an utterance Uv1. Here, the utterance content of the utterance Uv1 may be, for example, an inquiry such as "What are your plans for tomorrow?". Further, the utterance volume of the utterance Uv1 in this example may be, for example, 45 dB. However, since the utterance Uv1 is uttered at a volume lower than the recognizable volume, it is difficult for the information processing device 30 to recognize the utterance Uv1. Therefore, the information processing device 30 causes the input / output terminal 10 to output the system output Sv1 at a recognizable volume as implicit feedback. Here, the utterance content of the system output Sv1 may be, for example, re-listening to "Do you want it?". Further, the system output Sv1 at this time may be, for example, 60 dB.

Subsequently, the user who receives the system output Sv1 as implicit feedback makes another inquiry by the utterance Uv2. The utterance content of the utterance Uv2 may be, for example, an inquiry such as "Are you planning tomorrow?" Further, the utterance volume of the utterance Uv2 in this example may be, for example, 45 dB. That is, in the example shown in FIG. 4, the user's utterance Uv2 is uttered at a louder volume than the utterance Uv1 by receiving the system output Sv1 which is an implicit feedback, and is higher than the recognizable volume. Therefore, the utterance Uv2 is recognized by the information processing device 30, and the input / output terminal 10 outputs the system output Sv2 as the execution result of the application based on the recognized utterance Uv2. Here, the system output Sv2 may be, for example, a search result regarding the user's schedule such as "I have a schedule of going to the hospital at noon tomorrow." The system output Sv2 at this time may be performed at a volume equivalent to, for example, the system output Sv1.

In the example shown in FIG. 4, the user speaks first, but when the input / output terminal 10 outputs voice first, the system output at a volume recognizable by the input / output terminal 10 is started. It is also possible to induce the user's utterance volume by exemplifying the above.

As described above, in the first mode in which the implicit feedback is performed, the feedback that encourages the user to improve the speech mode is performed by exemplifying the mode with high recognition accuracy expected from the user.

(Explicit feedback)
On the other hand, the explicit feedback may be feedback indicating a direct improvement method for the user's utterance mode. That is, in the explicit feedback, unlike the implicit feedback in which the mode of output by the input / output terminal 10 is changed, the improvement method for improving the recognition accuracy by the input / output terminal 10 may be directly shown to the user. .. Therefore, in the second mode in which explicit feedback is performed, a method for improving the utterance that the user can take is specifically presented in order to recognize the utterance of the user. For example, when the user's utterance volume is low, in the second mode, a voice output such as "Please speak louder" may be performed. Further, for example, when the user's utterance speed is too fast, a voice output such as "Please speak more slowly" may be performed in the second mode. As described above, in the second mode in which explicit feedback is provided, feedback is provided to encourage the user to improve his or her speech mode by clearly indicating the improvement means that the user can take.

The information processing device 30 according to the present embodiment is characterized in that the first mode or the second mode described above is selected according to the information related to the user's utterance recognition including the user's utterance mode. Make one. As mentioned above, explicit feedback can spoil the user's impression of the device, and can be overused to make the dialogue unnatural. On the other hand, although implicit feedback can realize a more natural dialogue closer to human beings than explicit feedback, it is expected that the effect of improving the speech mode is lower than that of explicit feedback.

Based on the above points, the information processing apparatus 30 according to the present embodiment is based on the first mode in which implicit feedback is performed, and is set in the second mode in which explicit feedback is performed depending on the situation. Controls switching. By performing the control by the information processing device 30, it is possible for the user to improve the recognition accuracy of the utterance while realizing a more natural dialogue with the user.

<< 2.2. Examples of implied feedback >>
Next, a specific example of implicit feedback according to the present embodiment will be described. In this embodiment, various implicit feedbacks may be given depending on the speech mode of the user. FIG. 5 is a flowchart showing a flow of implicit feedback control by the output control unit 360 according to the present embodiment.

Referring to FIG. 5, first, the output control unit 360 determines whether or not the user's utterance can be recognized (S1101). At this time, the output control unit 360 may acquire the recognition result from the voice recognition unit 330 and make a determination. When the output control unit 360 determines in step S1101 that the user's utterance is recognized (S1101: Yes), the output control unit 360 ends the control related to the implicit feedback and shifts to the processing of the application based on the recognized voice.

On the other hand, in step S1101, when the output control unit 360 determines that the user's utterance is not recognized (S1101: No), the output control unit 360 subsequently determines the user's utterance volume and the volume of the environmental sound. Compare (S1102). At this time, the output control unit 360 may make a determination based on the analysis result by the voice analysis unit 320. When the output control unit 360 determines in step 1102 that the volume of the environmental sound exceeds the utterance volume of the user (S1102: Yes), the output control unit 360 generates feedback information proposing the environmental adjustment (S1106).

The feedback information that proposes the above environmental adjustment may be, for example, a command to output a voice saying "the surrounding sound is loud". As described above, the implicit feedback according to the present embodiment may include feedback in a manner with high recognition accuracy expected from the user, as well as feedback that gives the user awareness of the improvement in speech.

On the other hand, in step S1102, when the output control unit 360 determines that the user's utterance volume is louder than the volume of the environmental sound (S1102: No), the output control unit 360 subsequently recognizes the user's utterance volume. It is determined whether the size is sufficient for this (S1103). That is, the output control unit 360 determines whether or not the cause of the failure to recognize the user's utterance is the user's utterance volume.

Here, when the output control unit 360 determines that the user's utterance volume is not sufficient (S1103: No), the output control unit 360 generates feedback information that outputs voice at a volume higher than the user's utterance volume. (S1107). The feedback information may be, for example, a command for re-listening at a volume higher than the user's utterance volume.

On the other hand, when the output control unit 360 determines in step S1103 that the user's utterance volume is sufficient (S1103: Yes), the output control unit 360 subsequently determines whether or not the user's utterance can be phonemeized. Is determined (S1104). That is, the output control unit 360 determines whether or not the cause of the failure to recognize the user's utterance is the user's utterance speed or the clarity of pronunciation.

Here, when the output control unit 360 determines that the user's utterance cannot be phonemed (S1104: No), the output control unit 360 outputs feedback information at a speed slower than the user's utterance speed. Generate (S1108). The feedback information may be, for example, an instruction for re-listening at a speed slower than the user's utterance speed. In addition, the feedback information may be a command to output a voice that clearly pronounces changes in pitch, voice quality, and the like.

On the other hand, in step S1104, when the output control unit 360 determines that the user's utterance has been converted into phonemes (S1104: Yes), the output control unit 360 subsequently determines whether or not the user's utterance can be converted into text. Is determined (S1105). That is, the output control unit 360 determines whether or not the phonemeized information based on the user's utterance is recognized as a word.

Here, when the output control unit 360 determines that the user's utterance is not even recognized as a word (S1105: No), the output control unit 360 uses the phonemeized sound to output voice feedback information. Is generated (S1109). The feedback information may be, for example, a command for re-listening to "Are you OO?" Using the information of the phonemeized sound.

The example of the implicit feedback according to the present embodiment has been described above. As described above, the information processing device 30 according to the present embodiment can cause the input / output terminal 10 to execute various implicit feedbacks according to the information related to the user's utterance recognition. The control by the information processing device 30 makes it possible to realize a more natural dialogue close to that of human beings.

<< 2.3. Select mode for feedback >>
Next, the mode selection related to the feedback by the output control unit 360 of the present embodiment will be described. As described above, the information processing apparatus 30 according to the present embodiment can select the mode of feedback for the user's utterance mode according to the information related to the user's utterance recognition. Here, the information related to the user's utterance recognition may include, for example, user information, content information, environment information, and device information.

The above-mentioned user information is information about a user, and may be, for example, a user's utterance mode, utterance content, attribute information, emotional information, or the like. The output control unit 360 according to the present embodiment can select a feedback mode for the user's utterance mode according to the user information.

Further, the above-mentioned content information may be information related to an application that recognizes a user's utterance and performs processing. Here, the content information may include, for example, information regarding the type and specifications of the application. The output control unit 360 according to the present embodiment can select a mode of feedback for the user's utterance mode according to the content information. For example, the output control unit 360 may select the first mode in an application whose main purpose is conversation with a user, and may select a second mode in an application whose main purpose is information retrieval.

Further, the above-mentioned environmental information may be information related to the surrounding environment of the user and the input / output terminal 10. The environmental information may include, for example, information on detecting a person other than the user, information on environmental sounds, and the like. The output control unit 360 according to the present embodiment can select a feedback mode for the user's utterance mode according to the environmental information.

Further, the above device information may be information regarding the type and specifications of the input / output terminal 10. The output control unit 360 according to the present embodiment can select a feedback mode for the user's utterance mode according to the device information. The output control unit 360 selects, for example, a first mode when the input / output terminal 10 is an agent whose main purpose is to talk with a user, and the input / output terminal 10 is a device that is used in a business or mechanical manner. The second mode can also be selected when.

The information related to the user's utterance recognition has been described above. As described above, the information processing apparatus 30 according to the present embodiment is based on the first mode in which implicit feedback is performed, and switches to the second mode in which explicit feedback is performed depending on the situation. You can control it. Hereinafter, the conditions under which the output control unit 360 switches the mode related to feedback will be specifically described.

(Selection of mode based on the number of recognition attempts)
First, mode selection based on the number of recognition trials will be described. The output control unit 360 according to the present embodiment can select a second mode in which explicit feedback is performed based on the fact that the user's utterance is not recognized within a predetermined number of times. The above-mentioned predetermined number of times may be defined in various ways according to the specifications of the system or application. The predetermined number of times according to the present embodiment may be, for example, the number of times that the input related to the user's utterance is detected but not recognized (the number of times of authentication failure). Further, the predetermined number of times may be the number of times that the input waiting state related to recognition has timed out (timeout number). Further, the predetermined number of times may be the number of utterances of the user (number of utterances). Furthermore, the predetermined number of times may be the total number of times of the above-mentioned examples. Hereinafter, the above control will be described in detail with reference to FIGS. 6 and 7. In the following description, the case of determining the utterance volume of the user will be described as an example.

First, the volume level of the audio output according to the present embodiment will be described with reference to FIG. FIG. 6 is a diagram for explaining the volume level of the audio output by the audio output unit 130 of the input / output terminal 10. In the example shown in FIG. 6, the volume level of the audio output is defined in three stages of levels 1 to 3, and the higher the level value, the louder the volume may be. Here, level 1 may be an initial setting value in the audio output by the audio output unit 130. The volume range at level 1 may be, for example, 0 dB to 50 dB. When the user's utterance is normally recognized, the input / output terminal 10 may make a subsequent response at a level 1 volume. Further, level 2 may be defined as a volume one step louder than level 1. The volume range at level 2 may be, for example, 51 dB to 70 dB. Also, level 3 is one step louder than level 2 and may be defined as the maximum volume in implicit feedback. The volume range at level 3 may be defined as, for example, 71 dB to 100 dB, or 71 dB or more.

Next, with reference to FIG. 7, selection of the second mode based on the number of recognition trials will be described. FIG. 7 is a flowchart showing a control flow of the output control unit 360 based on the number of recognition trials. First, the output control unit 360 determines whether or not the user's utterance volume acquired from the input / output terminal 10 is insufficient as the volume for recognition (S1201).

Here, when the user's utterance volume is sufficient for recognition (S1201: No), the output control unit 360 ends the process of determining the user's utterance volume. On the other hand, when the user's utterance volume is insufficient (S1201: Yes), the output control unit 360 generates feedback information for executing the implicit feedback by the level 2 volume shown in FIG. 6 (S1202). .. That is, the output control unit 360 causes the input / output terminal 10 to execute implicit feedback with a volume one step higher than the user's utterance volume.

When the user's utterance is acquired again after the implicit feedback in step S1202 is executed, the output control unit 360 again checks whether or not the acquired user's utterance volume is insufficient as the volume for recognition. The determination is made (S1203). Here, when the user's utterance volume is sufficient for recognition (S1203: No), the output control unit 360 ends the process of determining the user's utterance volume.

On the other hand, when the user's utterance volume is insufficient again (S1203: Yes), the output control unit 360 generates feedback information for executing the implicit feedback by the level 3 volume shown in FIG. 6 (S1204). ). That is, the output control unit 360 causes the input / output terminal 10 to execute implicit feedback at the set maximum volume.

When the user's utterance is acquired again after the implicit feedback in step S1204 is executed, the output control unit 360 again checks whether or not the acquired user's utterance volume is insufficient as the volume for recognition. The determination is made (S1205). Here, when the user's utterance volume is sufficient for recognition (S1205: No), the output control unit 360 ends the process of determining the user's utterance volume.

On the other hand, when the user's utterance volume is insufficient again (S1205: Yes), the output control unit 360 generates feedback information for executing explicit feedback (S1206). That is, the output control unit 360 determines that the implicit feedback does not lead to the improvement of the user's utterance mode, and causes the input / output terminal 10 to execute the explicit feedback. The input / output terminal 10 may output a voice such as "Please speak in a louder voice" to the user according to the control by the output control unit 360.

As described above, the output control unit 360 according to the present embodiment can select a second mode in which explicit feedback is performed based on the number of recognition trials. The above control by the output control unit 360 makes it possible to improve the accuracy of recognition when the user's utterance is not recognized within a predetermined number of times, while being based on a natural dialogue by implicit feedback.

In the above description, the volume level is defined in three stages, and the case where the number of recognition trials is three is described as an example. However, the selection of the second mode based on the number of recognition trials according to the present embodiment is not limited to such an example. The volume level and the number of recognition trials may be changed as appropriate. Further, in the above description, the case of determining the utterance volume of the user among the utterance modes of the user has been described as an example. However, the selection of the second mode based on the number of recognition trials according to the present embodiment is not limited to such an example. The output control unit 360 can also determine the user's utterance speed and utterance clarity and select the second mode.

(Selection of mode based on the degree of change in utterance mode)
Next, the mode selection based on the degree of change in the user's utterance mode will be described. The output control unit 360 according to the present embodiment can select the second mode based on the fact that no improvement is observed in the utterance mode of the user who receives the implicit feedback. Hereinafter, the above control will be described in detail with reference to FIG. In the following description, the case of determining the utterance volume of the user will be described as an example.

First, the output control unit 360 determines whether or not the user's utterance volume acquired from the input / output terminal 10 is insufficient as the volume for recognition (S1301). When the user's utterance volume is sufficient for recognition (S1301: No), the output control unit 360 ends the process of determining the user's utterance volume. On the other hand, when the user's utterance volume is insufficient (S1301: Yes), the output control unit 360 generates feedback information for executing the implicit feedback by the level 2 volume shown in FIG. 6 (S1302). ..

When the user's utterance is acquired again after the implicit feedback in step S1302 is executed, the output control unit 360 compares the acquired user's utterance volume with the previous utterance volume and the degree of change in the utterance volume. Is determined (S1303). At this time, the output control unit 360 can make the above determination by acquiring the analysis result of the user's previous utterance mode stored in the state storage unit 340.

Here, when the user's utterance volume has changed sufficiently to a recognizable level (S1303: sufficient change), the output control unit 360 ends the process of determining the user's utterance volume. On the other hand, in step S1303, when the user's utterance volume changes but does not reach a recognizable level (S1303: insufficient change), the output control unit 360 depends on the level 3 volume shown in FIG. Generate feedback information for performing implicit feedback (S1305). Further, in step S1303, when there is no change in the user's utterance volume or the utterance volume is low (S1303: no change), the output control unit 360 uses implicit feedback to improve the user's utterance mode. It is determined that the connection cannot be established, and the input / output terminal 10 is made to execute explicit feedback.

As described above, the output control unit 360 according to the present embodiment can select a second mode in which explicit feedback is performed based on the degree of change in the user's utterance mode. The above control by the output control unit 360 makes it possible to improve the accuracy of recognition even when the user does not respond to the implicit feedback.

In the above description, the case of determining the user's utterance volume among the user's utterance modes has been described as an example. However, the selection of the second mode based on the degree of change in the user's utterance mode according to the present embodiment is not limited to the above example. The output control unit 360 can also determine the user's utterance speed and utterance clarity and select the second mode.

(Selection of mode based on utterance position or utterance direction)
Next, the mode selection based on the user's utterance position or utterance direction will be described. The output control unit 360 according to the present embodiment can select the second mode based on the fact that no improvement is observed in the utterance position or the utterance direction of the user who has received the implicit feedback. As described above, the utterance mode of the user according to the present embodiment may include the utterance position and the utterance direction of the user.

FIG. 9 is a flowchart showing a control flow of the output control unit 360 based on the user's utterance position or utterance direction. First, the output control unit 360 determines whether or not the user's utterance volume acquired from the input / output terminal 10 is insufficient as the volume for recognition (S1401). Here, when the user's utterance volume is sufficient for recognition (S1401: No), the output control unit 360 ends the process of determining the user's utterance position and utterance direction.

On the other hand, when the user's utterance volume is insufficient (S1401: Yes), the output control unit 360 subsequently determines whether or not the user's utterance position is appropriate (S1402). That is, the output control unit 360 determines whether or not the insufficient utterance volume is due to the utterance position. At this time, the output control unit 360 can make the above determination based on the information of the user's utterance position estimated by the position detection unit 350.

When the output control unit 360 determines in step S1402 that the user's utterance position is not appropriate (S1402: No), the output control unit 360 generates feedback information for executing implicit feedback on the user's utterance position. (S1404). The feedback information may be, for example, a command to output the voice "The voice seems to be far away" to the input / output terminal 10.

On the other hand, when the output control unit 360 determines in step S1402 that the user's utterance position is appropriate (S1402: Yes), the output control unit 360 subsequently determines whether or not the user's utterance direction is appropriate. (S1403). That is, the output control unit 360 determines whether or not the insufficient utterance volume is caused by the utterance direction. At this time, the output control unit 360 can make the above determination based on the information of the user's utterance direction estimated by the position detection unit 350.

When the output control unit 360 determines in step S1403 that the user's utterance direction is appropriate (S1403: Yes), the output control unit 360 ends the process of determining the user's utterance position and utterance direction.

On the other hand, when the output control unit 360 determines in step S1403 that the user's utterance direction is not appropriate (S1403: No), the output control unit 360 causes feedback information for executing implicit feedback on the user's utterance direction. Is generated (S1405). The feedback information may be, for example, a command to output a voice such as "Are you talking to me?" To the input / output terminal 10. Further, the feedback information generated in step S1405 may be a designation relating to a speaker that outputs audio. For example, when the voice output unit 130 of the input / output terminal 10 includes a plurality of speakers, the user can be made aware of the utterance direction by limiting the speakers that output implicit feedback. When the audio output unit 130 includes a microphone array, the feedback information may include information for setting the directivity of the microphone array.

When the information related to the user's utterance recognition is acquired again after the implicit feedback is executed in step S1404 or step S1405, the output control unit 360 determines whether or not the user's utterance position or utterance direction has been improved. (S1406). Here, when the output control unit 360 determines that the user's utterance position or utterance direction has been improved (S1406: Yes), the output control unit 360 ends the process of determining the utterance position and the utterance direction.

On the other hand, when the output control unit 360 determines that the user's utterance position or utterance direction has not been improved (S1406: No), the output control unit 360 causes the user to execute explicit feedback on the user's utterance position or utterance direction. Generate feedback information. The feedback information may be, for example, a command to output voices such as "Please approach the microphone" and "Speak in the direction of the microphone" to the input / output terminal 10.

As described above, the output control unit 360 according to the present embodiment can select a second mode in which explicit feedback is performed based on the user's utterance position or utterance direction. Although the case where the user's utterance volume is insufficient has been described above, the selection of the second mode based on the user's utterance position or utterance direction may be controlled in consideration of the volume of the environmental sound. .. For example, it is assumed that the input / output terminal 10 is an agent incorporated in a building, and a voice input unit 110 and a voice output unit 130 are provided in a plurality of rooms in the building. At this time, if the environmental sound of the room in which the user has spoken is loud and it is difficult to recognize the user's utterance, the information processing device 30 may generate feedback information for guiding the user to another room. .. That is, the information processing device 30 can guide the user so that the utterance is performed on a microphone different from the microphone that has detected the utterance of the user. In this way, the output control unit 360 controls the feedback based on the user's utterance position or utterance direction, so that various feedbacks according to the specifications of the input / output terminal 10 can be realized.

(Selection of mode based on analysis of utterance mode)
Next, the mode selection based on the analysis of the user's utterance mode will be described. The output control unit 360 according to the present embodiment can control the feedback mode based on the analysis result of the user's utterance mode. The above utterance modes may include utterance volume, utterance speed, pitch of utterance, clarity of pronunciation, utterance position, utterance direction, utterance content, and environmental sound.

-Mode selection based on user attribute information-
First, the mode selection based on the user's attribute information will be described. The output control unit 360 according to the present embodiment can control the feedback mode based on the attribute information of the user. The user attribute information may be information obtained by the voice analysis unit 320 analyzing the user's utterance mode, or information obtained from the result of voice recognition by the voice recognition unit 330. In addition, the attribute information of the user may include profile information such as the user's gender and age, and information such as the language used and the tendency of the utterance mode.

The output control unit 360 may select the feedback mode based on the tendency of the user's utterance mode. For example, when the user identified by the analysis of the voice analysis unit 320 tends to have a low utterance volume, the output control unit 360 may preferentially select the second mode. In this way, by selecting the mode based on the user's utterance tendency, the output control unit 360 can be expected to shorten the time until the user's utterance is recognized.

Further, the output control unit 360 may select the feedback mode based on the user's setting regarding the mode. When the user can set the feedback mode for the utterance, the output control unit 360 can set the feedback mode according to the user's setting specified by the analysis of the voice analysis unit 320.

Further, the output control unit 360 may select the feedback mode based on the statistical information obtained from the attribute information of a plurality of users. The output control unit 360 may acquire the tendency of the utterance mode of the user group having the attribute by using the specific attribute information as a key, and select the mode based on the tendency. The above control is particularly effective in an environment in which the input / output terminal 10 is used by an unspecified number of users.

-Mode selection based on user emotions-
Next, the mode selection based on the user's emotion will be described. The output control unit 360 according to the present embodiment can control the feedback mode based on the user's emotions. The user's emotion may be information obtained by the voice analysis unit 320 analyzing the user's utterance mode.

The output control unit 360 may select the first mode, for example, based on the presumption that the user is in an excited state. As mentioned above, some users may be disgusted with explicit feedback. Therefore, when the user is presumed to be in an excited state, the output control unit 360 can reduce the risk of damaging the user's emotions by causing the input / output terminal 10 to provide implicit feedback. ..

-Selection of mode based on utterance content-
Next, the mode selection based on the utterance content of the user will be described. The output control unit 360 according to the present embodiment can control the feedback mode based on the content of the user's utterance. The content of the user's utterance may be information obtained from the result of voice recognition by the voice recognition unit 330.

The output control unit 360 may select the second mode, for example, based on the presumed content of the user's utterance to include privacy information. In the suggestive feedback according to the present embodiment, there is a possibility that re-listening is performed using elements that can be phonemed or texted. Therefore, in order to prevent the privacy information from being leaked to a person other than the user, the output control unit 360 can set the second mode. Further, by performing the above control by the output control unit 360, the recognition accuracy of the utterance related to the privacy information can be improved, and the dialogue can be completed in a shorter time.

-Selection of mode based on environmental information-
Next, the mode selection based on the environmental information will be described. The output control unit 360 according to the present embodiment can control the feedback mode based on the detection of the presence of a third party around the user. The detection of the third party may be information obtained from the detection result by the position detection unit 350 or information obtained from the result of voice recognition by the voice recognition unit 330.

The output control unit 360 may select the first mode based on the detection of the presence of a third party around the user, for example. As mentioned above, some users may feel that explicit feedback is directed to the device. It is expected that such a user will further impair the impression of the input / output terminal 10 by being asked for explicit feedback by a surrounding person. Therefore, when the presence of a third party is detected around the user, the output control unit 360 causes the input / output terminal 10 to provide implicit feedback, thereby reducing the risk of damaging the user's emotions. be able to.

<< 2.4. Explicit feedback with reasons for improvement >>
The selection of the mode related to the feedback by the output control unit 360 has been described above. Next, explicit feedback with a reason for improvement according to the present embodiment will be described. The output control unit 360 according to the present disclosure can cause the input / output terminal 10 to execute feedback with a reason for improvement in the second mode in which explicit feedback is performed. By controlling the feedback so that the output control unit 360 also presents the reason for improving the utterance mode to the user, it is possible to soften the expression of the explicit feedback and reduce the possibility of damaging the user's emotions. It becomes.

FIG. 10 is a flowchart showing a flow of adding the reason for improvement by the output control unit 360. In the example shown in FIG. 10, the output control unit 360 first acquires the analysis result of the utterance by the voice analysis unit 320 and determines the number of detected voices (S1501). That is, the output control unit 360 determines whether or not the sound collection information includes the voices of a plurality of people.

In step S1501, when the output control unit 360 determines that only one voice is detected (S1501: one voice), the output control unit 360 sets the improvement reason 1 in the feedback information (S1502). Here, the reason for improvement 1 set in the feedback information may be additional information such as "because the surrounding sound is too loud".

Further, in step S1501, when the output control unit 360 determines that the voices of a plurality of people are detected (S1501: voices of a plurality of people), the output control unit 360 sets the improvement reason 2 in the feedback information (S1503). ). Here, the improvement reason 2 set in the feedback information may be additional information such as "because it seems that another person is talking".

Further, in step S1501, when it is difficult for the output control unit 360 to determine the number of voices (S1501: identification difficulty), the output control unit 360 sets the improvement reason 3 in the feedback information (S1504). Here, the improvement reason 3 set in the feedback information may be additional information such as "because it is a little hard to hear".

When any of the improvement reasons is set in steps S1502 to S1504, the output control unit 360 subsequently generates feedback information for executing explicit feedback with the improvement reason added, and transmits the feedback information to the input / output terminal 10. (S1505). The feedback information generated in step S1505 may be, for example, information that combines output information such as "Please say a little louder" and the reason for improvement. For example, when the reason for improvement 1 is set in step S1502, the information generated in step S1505 may be output information such as "The surrounding sounds are too loud, so please speak a little louder."

As described above, the output control unit 360 according to the present embodiment can cause the input / output terminal 10 to execute feedback with a reason for improvement in the second mode in which explicit feedback is performed. The above control by the output control unit 360 makes it possible to soften the expression of explicit feedback and realize a more natural dialogue.

<< 2.5. Additional control of feedback by visual information >>
Next, the addition of feedback by visual information according to the present embodiment will be described. The output control unit 360 according to the present embodiment can control feedback by visual information in addition to feedback by audio output. Further, the output control unit 360 can add feedback by visual information based on the fact that the user's utterance mode does not change sufficiently. Hereinafter, the above control by the output control unit 360 will be described in detail with reference to FIG.

First, the output control unit 360 determines whether or not the user's utterance volume is insufficient as the volume for recognition (S1601). Here, when the user's utterance volume is sufficient for recognition (S1601: No), the output control unit 360 ends the control related to the addition of feedback by visual information. On the other hand, when the user's utterance volume is insufficient (S1601: Yes), the output control unit 360 generates feedback information for executing the implicit feedback by the level 2 volume shown in FIG. 6 (S1602). ..

When the user's utterance is acquired again after the implicit feedback in step S1602 is executed, the output control unit 360 again determines whether or not the acquired user's utterance volume is insufficient as the volume for recognition. The determination is made (S1603). Here, when the user's utterance volume is sufficient for recognition (S1603: No), the output control unit 360 ends the control related to the addition of feedback by visual information.

On the other hand, when the user's utterance volume is insufficient again (S1603: Yes), the output control unit 360 generates feedback information for executing the implicit feedback by the level 3 volume shown in FIG. 6 (S1604). ). Further, the output control unit 360 generates feedback information for executing implicit feedback by visual information (S1605). The feedback information for executing the implicit feedback may be, for example, an instruction to display the text information similar to the feedback by the voice output on the display unit 140 of the input / output terminal 10. Further, the feedback information for executing the implicit feedback may be an instruction for executing the feedback using an image or animation described later.

When the user's utterance is acquired again after the implicit feedback is executed in steps S1604 to S1605, the output control unit 360 determines whether or not the acquired user's utterance volume is insufficient as the volume for recognition. Or, the determination is made again (S1606). Here, when the user's utterance volume is sufficient for recognition (S1606: No), the output control unit 360 ends the control related to the addition of feedback by visual information.

On the other hand, when the user's utterance volume is insufficient again (S1606: Yes), the output control unit 360 generates feedback information for executing explicit feedback by voice output (S1607). Further, the output control unit 360 generates feedback information for executing explicit feedback by visual information (S1608).

As described above, the output control unit 360 according to the present embodiment can control feedback by visual information in addition to feedback by audio output. In addition, the output control unit 360 can add visual feedback step by step in the same manner as the control of feedback by audio output. By performing the above control by the output control unit 360, it is possible to improve the recognition accuracy while based on the implicit feedback by voice.

<< 2.6. Example of feedback by visual information >>
The control of feedback by visual information according to the present embodiment has been described above. Hereinafter, an example of feedback by visual information according to the present embodiment will be described with reference to FIGS. 12 to 16. The above visual information may include changes in characters, symbols, avatars, indicators, or images.

(Implicit feedback by indicator)
FIG. 12 is an example of an indicator used for implicit feedback by visual information of the present embodiment. Referring to FIG. 12A, two indicators i1 and i2 are displayed on the display unit 140 of the input / output terminal 10. Here, the indicator i1 may be an indicator indicating the utterance volume of the user. Further, the indicator i2 may be an indicator indicating the output volume of the input / output terminal 10. The proportions of the gradations of the indicators i1 and i2 may change toward the upper part of the display unit 140 according to the change in the utterance volume of the user or the output volume of the input / output terminal 10. That is, the louder the user's utterance volume, the wider the gradation of the indicator i1 toward the upper part of the screen of the display unit 140, and the louder the output volume of the input / output terminal 10, the more the indicator i2 faces the upper part of the screen of the display unit 140. The gradation may spread.

Further, FIG. 12B is a diagram showing an example of another indicator. Referring to FIG. 12B, two indicators i3 and i4 are displayed on the display unit 140 of the input / output terminal 10. Here, the indicator i3 may be an indicator indicating the utterance volume of the user. Further, the indicator i4 may be an indicator indicating the output volume of the input / output terminal 10. In each of the indicators i3 and i4, the number of bars indicating the volume level may change toward the center of the display unit 140 according to the change in the utterance volume of the user or the output volume of the input / output terminal 10. That is, the louder the user's utterance volume, the more bars the indicator i3 has toward the center of the screen of the display unit 140, and the louder the output volume of the input / output terminal 10, the more the indicator i4 has the center of the screen of the display unit 140. The number of bars may increase towards.

By checking the indicator displayed on the display unit 140, the user can compare the output volume of the input / output terminal 10 with the utterance volume of the user. As a result, the user recognizes that the utterance volume is insufficient, and the effect of improving the utterance mode is expected.

(Implicit feedback on the direction of speech by the avatar)
Next, with reference to FIG. 13, an example of an avatar used for implicit feedback by visual information of the present embodiment will be described. The avatar shown in FIG. 13 may be an image or animation for providing implicit feedback on the user's utterance direction. Referring to FIG. 13A, the avatar a1 is displayed on the display unit 140 of the input / output terminal 10. Further, a voice input unit 110 is arranged below the input / output terminal 10. Here, the avatar a1 may be an example of an avatar displayed when the user's utterance direction is appropriate.

FIG. 13B is an example of an avatar displayed when the user's utterance direction is not appropriate. Referring to FIG. 13B, unlike FIG. 13A, the voice input unit 110 is arranged on the left side of the input / output terminal 10. Further, the avatar a2 is displayed on the left side of the display unit 140, and the line of sight is directed to the voice input unit 110.

Subsequently, reference is made to FIG. 13C. FIG. 13C also shows an example of an avatar displayed when the user's utterance direction is not appropriate, as in FIG. 13B. Referring to FIG. 13C, unlike FIG. 13A, the voice input unit 110 is arranged on the right side of the input / output terminal 10. Further, the avatar a3 is displayed on the right side of the display unit 140, and the line of sight is directed to the voice input unit 110.

As described above, in the example shown in FIG. 13, when the user's utterance direction is not appropriate, implicit feedback is provided by suggesting the position of the voice input unit 110 by the avatar. When the user visually recognizes the movement of the avatar image or animation, the effect of improving the user's utterance direction is expected.

(Implicit feedback on the direction of speech using graphics)
Next, with reference to FIG. 14, an example of the graphic used for the implicit feedback by the visual information of the present embodiment will be described. The graphic g1 shown in FIG. 14 may be an image or animation for giving implicit feedback to the user's utterance direction. Referring to FIG. 14A, the graphic g1 is displayed on the display unit 140 of the input / output terminal 10. Further, a voice input unit 110 is arranged below the input / output terminal 10. Here, the graphic g1 may be an example of a graphic displayed when the user's utterance direction is appropriate.

FIG. 14B is an example of a graphic displayed when the user's utterance direction is not appropriate. Referring to FIG. 14B, unlike FIG. 14A, the voice input unit 110 is arranged on the left side of the input / output terminal 10. Further, the graphic g2 is displayed on the left side of the display unit 140, and is deformed so as to spread toward the voice input unit 110 side.

Subsequently, reference is made to FIG. 14C. FIG. 14C, like FIG. 14B, also shows an example of a graphic displayed when the user's utterance direction is not appropriate. Referring to FIG. 14C, unlike FIG. 14A, the voice input unit 110 is arranged on the right side of the input / output terminal 10. Further, the graphic g3 is displayed on the right side of the display unit 140, and is deformed so as to spread toward the voice input unit 110 side.

As described above, in the example shown in FIG. 14, when the user's utterance direction is not appropriate, implicit feedback is provided by suggesting the position of the voice input unit 110 by the graphic. When the user visually recognizes the movement of the graphic image or animation, the effect of improving the user's utterance direction is expected. Although the example of changing the shape and display position of the graphic has been described above, the position of the voice input unit 110 may be indicated by changing the color gradation displayed on the entire screen of the display unit 140. Good.

(Implicit feedback on the utterance position by the avatar)
Next, with reference to FIG. 15, an example of implicit feedback on the utterance position by the avatar will be described. The avatar shown in FIG. 15 may be an image or animation for providing implicit feedback on the user's utterance position. Referring to FIG. 15A, the avatar a4 is displayed on the display unit 140 of the input / output terminal 10. Here, the avatar a4 may be an example of an avatar displayed when the user's utterance position is appropriate.

FIG. 15B is an example of an avatar displayed when the user's utterance position is not appropriate (when the distance is long). With reference to FIG. 15B, it can be seen that the avatar a5 is displayed smaller than the avatar a4, and the facial expression is changed. As described above, in the example shown in FIG. 15B, by changing the size and facial expression of the avatar a5, implicit feedback to the utterance position of the user is provided. Although FIG. 15B shows an example of clouding the facial expression of the avatar a5, another expression may be used for the facial expression of the avatar a5. For example, the line of sight of the avatar a5 may indicate a distance, suggesting that the user's utterance position is far.

Subsequently, reference is made to FIG. 15C. FIG. 15C also shows an example of an avatar displayed when the user's utterance position is not appropriate, as in FIG. 15B. With reference to FIG. 15C, it can be seen that the outline of the avatar a6 is displayed lighter than that of the avatar a4. As described above, in the example shown in FIG. 15C, by changing the density of the contour of the avatar a6, implicit feedback to the utterance position of the user is provided.

As described above, in the example shown in FIG. 15, implicit feedback is provided by the avatar suggesting that the user's utterance position is not appropriate. When the user visually recognizes the movement of the avatar image or animation, the effect of improving the user's utterance direction is expected.

(Implicit feedback on the utterance direction or position by the arrow)
Next, with reference to FIG. 16, an example of implicit feedback on the utterance direction or utterance position by the arrow will be described. The graphic including the arrow shown in FIG. 16 may be an image or animation for providing implicit feedback on the user's utterance direction or utterance position.

Referring to FIG. 16A, graphics g4 and g5 are displayed on the display unit 140 of the input / output terminal 10. Here, the graphic g4 is displayed as an arrow pointing to the position of the voice input unit 110, and the graphic g5 is shown as an ear icon. As described above, in the example shown in FIG. 16A, by indicating the position of the voice input unit 110 by a graphic in the shape of an arrow or an ear, implicit feedback to the user's utterance direction is provided.

Subsequently, the description will be continued with reference to FIG. 16B. In the example shown in FIG. 16B, unlike FIG. 16A, feedback indicating the relative position between the input / output terminal 10 and the user is performed. Referring to FIG. 16B, graphics g6 to g9 are shown on the display unit 140 of the input / output terminal 10. Here, the graphic g6 is displayed as an arrow pointing to the position of the voice input unit 110. The graphics g7 and g8 are icons indicating the input / output terminal 10 and the user, respectively. As described above, in the example shown in FIG. 16B, the position of the voice input unit 110 is indicated by an arrow while indicating the relative position between the input / output terminal 10 and the user, thereby providing implicit feedback to the user's utterance direction. .. Further, in the example shown in FIG. 16B, as shown in the graphic g9, feedback indicating a sound source other than the user may be provided. When the user visually recognizes the graphic showing the sound source, the effect of improving the user's utterance position is expected.

<3. Hardware configuration example of input / output terminal 10 and information processing device 30>
Next, a hardware configuration example common to the input / output terminal 10 and the information processing device 30 according to the present disclosure will be described. FIG. 17 is a block diagram showing a hardware configuration example of the input / output terminal 10 and the information processing device 30 according to the present disclosure. Referring to FIG. 17, the input / output terminal 10 and the information processing device 30 include, for example, a CPU 871, a ROM 872, a RAM 873, a host bus 874, a bridge 875, an external bus 876, an interface 877, and an input unit 878. It has an output unit 879, a storage unit 880, a drive 881, a connection port 882, and a communication unit 883. The hardware configuration shown here is an example, and some of the components may be omitted. Further, components other than the components shown here may be further included.

(CPU871)
The CPU 871 functions as, for example, an arithmetic processing device or a control device, and controls all or a part of the operation of each component based on various programs recorded in the ROM 872, the RAM 873, the storage unit 880, or the removable recording medium 901. ..

(ROM872, RAM873)
The ROM 872 is a means for storing a program read into the CPU 871 and data used for calculation. The RAM 873 temporarily or permanently stores, for example, a program read into the CPU 871 and various parameters that change as appropriate when the program is executed.

(Host bus 874, Bridge 875, External bus 876, Interface 877)
The CPU 871, ROM 872, and RAM 873 are connected to each other via, for example, a host bus 874 capable of high-speed data transmission. On the other hand, the host bus 874 is connected to the external bus 876, which has a relatively low data transmission speed, via, for example, the bridge 875. Further, the external bus 876 is connected to various components via the interface 877.

(Input unit 878)
For the input unit 878, for example, a mouse, a keyboard, a touch panel, buttons, switches, levers, and the like are used. Further, as the input unit 878, a remote controller (hereinafter, remote controller) capable of transmitting a control signal using infrared rays or other radio waves may be used.

(Output unit 879)
The output unit 879 uses the acquired information, for example, a display device such as a CRT (Cathode Ray Tube), an LCD, or an organic EL, an audio output device such as a speaker or a headphone, a printer, a mobile phone, or a facsimile. It is a device that can notify the user visually or audibly.

(Memory unit 880)
The storage unit 880 is a device for storing various types of data. As the storage unit 880, for example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, an optical magnetic storage device, or the like is used.

(Drive 881)
The drive 881 is a device that reads information recorded on a removable recording medium 901 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, or writes information on the removable recording medium 901.

(Removable recording medium 901)
The removable recording medium 901 is, for example, a DVD media, a Blu-ray (registered trademark) media, an HD DVD media, various semiconductor storage media, and the like. Of course, the removable recording medium 901 may be, for example, an IC card equipped with a non-contact type IC chip, an electronic device, or the like.

(Connection port 882)
The connection port 882 is a port for connecting an external connection device 902 such as a USB (Universal Serial Bus) port, an IEEE1394 port, a SCSI (Small Computer System Interface), an RS-232C port, or an optical audio terminal. is there.

(External connection device 902)
The externally connected device 902 is, for example, a printer, a portable music player, a digital camera, a digital video camera, an IC recorder, or the like.

(Communication unit 883)
The communication unit 883 is a communication device for connecting to the network 903, and is, for example, a wired or wireless LAN, a Bluetooth (registered trademark), or a communication card for WUSB (Wireless USB), a router for optical communication, and an ADSL (Asymmetric). It is a router for Digital Subscriber Line), a modem for various communications, and the like.

<4. Summary>
As described above, the information processing apparatus 30 can select the feedback mode for the user's utterance mode from a plurality of modes according to the information related to the user's utterance recognition. The plurality of modes described above may include a first mode in which implicit feedback is provided and a second mode in which explicit feedback is provided. In addition, the above-mentioned utterance mode may include utterance volume, utterance speed, utterance pitch, pronunciation clarity, utterance position, utterance direction, utterance content, and environmental sound. According to such a configuration, it is possible to realize a more natural dialogue between the user and the system.

Although the preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that anyone with ordinary knowledge in the technical field of the present disclosure may come up with various modifications or modifications within the scope of the technical ideas set forth in the claims. Of course, it is understood that it belongs to the technical scope of the present disclosure.

In addition, the effects described herein are merely explanatory or exemplary and are not limited. That is, the techniques according to the present disclosure may exhibit other effects apparent to those skilled in the art from the description herein, in addition to or in place of the above effects.

The following configurations also belong to the technical scope of the present disclosure.
(1)
A control unit that selects a feedback mode for the user's utterance mode from a plurality of modes according to information related to the user's utterance recognition.
With
The plurality of modes include a first mode in which implicit feedback is provided and a second mode in which explicit feedback is provided.
Information processing device.
(2)
In the first mode, feedback including an indirect improvement method for the user's utterance mode is provided.
In the second mode, feedback is provided indicating a direct improvement method for the user's speech mode.
The information processing device according to (1) above.
(3)
The information related to the user's utterance recognition includes user information, content information, environmental information, and device information.
The information processing device according to (2) above.
(4)
The user information includes the utterance mode of the user.
The control unit selects the first mode or the second mode based on the utterance mode of the user.
The information processing device according to (3) above.
(5)
The utterance mode includes at least one of utterance volume, utterance speed, utterance pitch, pronunciation clarity, utterance position, or utterance direction.
The information processing device according to (4) above.
(6)
The control unit selects the second mode based on the fact that no improvement is observed in the utterance mode of the user who has received the feedback from the first mode.
The information processing device according to (4) or (5) above.
(7)
After giving feedback in the first mode, the control unit selects the second mode based on the fact that the user's utterance is not recognized within a predetermined number of times.
The information processing device according to any one of (4) to (6) above.
(8)
The user information includes the content of the utterance of the user.
The control unit selects the first mode or the second mode based on the content of the user's utterance.
The information processing device according to any one of (3) to (7) above.
(9)
The control unit selects the second mode based on the presumed content of the user's utterance to include privacy information.
The information processing device according to (8) above.
(10)
The control unit selects the first mode based on the estimation of the existence of another person different from the user from the environmental information.
The information processing device according to any one of (3) to (9) above.
(11)
The user information includes attribute information of the user.
The control unit selects the first mode or the second mode based on the attribute information of the user.
The information processing device according to any one of (3) to (10).
(12)
The user information includes emotional information of the user.
The control unit selects the first mode or the second mode based on the user's emotional information estimated from the user's utterance.
The information processing device according to any one of (3) to (11).
(13)
In the first mode, feedback is performed at a volume having a higher recognition accuracy than the utterance volume of the user.
The information processing device according to any one of (4) to (12).
(14)
In the first mode, the feedback is performed at a speed higher in recognition accuracy than the utterance speed of the user.
The information processing device according to any one of (4) to (13).
(15)
In the first mode, the feedback is performed at a pitch of sound having higher recognition accuracy than the pitch of sound spoken by the user.
The information processing device according to any one of (4) to (14).
(16)
In the second mode, feedback is provided with a reason for improving the utterance mode.
The information processing device according to any one of (2) to (15).
(17)
The feedback includes visual feedback.
The information processing device according to any one of (2) to (16).
(18)
In the second mode, feedback is given to the effect that the user should speak to a sensor different from the sensor that detected the user's utterance.
The information processing device according to any one of (2) to (17).
(19)
The user's attribute information includes at least one of gender, age, language used, or tendency of speech mode.
The information processing device according to (11) above.
(20)
The control unit selects the first mode based on the presumption that the user is in an excited state.
The information processing device according to (12) above.
(21)
In the first mode, feedback by artificial voice is performed according to the utterance mode of the user.
The information processing device according to any one of (1) to (20).
(22)
The visual information includes changes in characters, symbols, avatars, indicators, or images.
The information processing device according to (17) above.
(23)
The processor selects a mode of feedback for the user's utterance mode from a plurality of modes according to the information related to the user's utterance recognition.
Including
The plurality of modes include a first mode in which implicit feedback is provided and a second mode in which explicit feedback is provided.
Information processing method.
(24)
Computer,
A control unit that selects a feedback mode for the user's utterance mode from a plurality of modes according to information related to the user's utterance recognition.
With
The plurality of modes include a first mode in which implicit feedback is provided and a second mode in which explicit feedback is provided.
Information processing device,
A program to function as.

10 Input / output terminal 110 Voice input unit 120 Sensor unit 130 Voice output unit 140 Display unit 150 Terminal control unit 160 Server communication unit 20 Network 30 Information processing device 310 Terminal communication unit 320 Voice analysis unit 330 Voice recognition unit 340 State storage unit 350 Position Detection unit 360 Output control unit 370 Output DB

Claims (18)

A control unit that selects a feedback mode for the user's utterance mode from a plurality of modes according to information related to the user's utterance recognition.
With
Wherein the plurality of modes, viewed contains a first mode in which implicit feedback is performed, a second mode in which explicit feedback is performed, and
In the first mode, feedback including an indirect improvement method for the user's utterance mode is provided.
In the second mode, as feedback indicating a direct improvement method for the utterance mode of the user, feedback is provided to the effect that the utterance should be made to a sensor different from the sensor that detected the utterance of the user.
Information processing device. The information related to the user's utterance recognition includes user information, content information, environmental information, and device information.
The information processing device according to claim 1 . The user information includes the utterance mode of the user.
The control unit selects the first mode or the second mode based on the utterance mode of the user.
The information processing device according to claim 2 . The utterance mode includes at least one of utterance volume, utterance speed, utterance pitch, pronunciation clarity, utterance position, or utterance direction.
The information processing device according to claim 3 . The control unit selects the second mode based on the fact that no improvement is observed in the utterance mode of the user who has received the feedback from the first mode.
The information processing device according to claim 3 or 4 . After giving feedback in the first mode, the control unit selects the second mode based on the fact that the user's utterance is not recognized within a predetermined number of times.
The information processing device according to any one of claims 3 to 5 . A control unit that selects a feedback mode for the user's utterance mode from a plurality of modes according to information related to the user's utterance recognition.
With
The plurality of modes include a first mode in which implicit feedback is provided and a second mode in which explicit feedback is provided.
In the first mode, feedback including an indirect improvement method for the user's utterance mode is provided.
In the second mode, feedback is provided indicating a direct improvement method for the user's speech mode.
The information related to the user's utterance recognition includes user information, content information, environmental information, and device information.
The user information includes the content of the utterance of the user.
The control unit selects the first mode or the second mode based on the content of the user's utterance.
Information processing apparatus. The control unit selects the second mode based on the presumed content of the user's utterance to include privacy information.
The information processing device according to claim 7 . A control unit that selects a feedback mode for the user's utterance mode from a plurality of modes according to information related to the user's utterance recognition.
With
The plurality of modes include a first mode in which implicit feedback is provided and a second mode in which explicit feedback is provided.
In the first mode, feedback including an indirect improvement method for the user's utterance mode is provided.
In the second mode, feedback is provided indicating a direct improvement method for the user's speech mode.
The information related to the user's utterance recognition includes user information, content information, environmental information, and device information.
The control unit selects the first mode based on the estimation of the existence of another person different from the user from the environmental information.
Information processing apparatus. The user information includes attribute information of the user.
The control unit selects the first mode or the second mode based on the attribute information of the user.
The information processing device according to any one of claims 2 to 9 . A control unit that selects a feedback mode for the user's utterance mode from a plurality of modes according to information related to the user's utterance recognition.
With
The plurality of modes include a first mode in which implicit feedback is provided and a second mode in which explicit feedback is provided.
In the first mode, feedback including an indirect improvement method for the user's utterance mode is provided.
In the second mode, feedback is provided indicating a direct improvement method for the user's speech mode.
The information related to the user's utterance recognition includes user information, content information, environmental information, and device information.
The user information includes emotional information of the user.
The control unit selects the first mode or the second mode based on the user's emotional information estimated from the user's utterance.
Information processing apparatus. In the first mode, feedback is performed at a volume having a higher recognition accuracy than the utterance volume of the user.
The information processing device according to any one of claims 3 to 6 . In the first mode, the feedback is performed at a speed higher in recognition accuracy than the utterance speed of the user.
The information processing device according to any one of claims 3 to 6 . In the first mode, the feedback is performed at a pitch of sound having higher recognition accuracy than the pitch of sound spoken by the user.
The information processing device according to any one of claims 3 to 6 . In the second mode, feedback is provided with a reason for improving the utterance mode.
The information processing device according to any one of claims 1 to 14 . The feedback includes visual feedback.
The information processing device according to any one of claims 1 to 15 . The processor selects a mode of feedback for the user's utterance mode from a plurality of modes according to the information related to the user's utterance recognition.
Including
Wherein the plurality of modes, viewed contains a first mode in which implicit feedback is performed, a second mode in which explicit feedback is performed, and
In the first mode, feedback including an indirect improvement method for the user's utterance mode is provided.
In the second mode, as feedback indicating a direct improvement method for the utterance mode of the user, feedback is provided to the effect that the utterance should be made to a sensor different from the sensor that detected the utterance of the user.
Information processing method. Computer,
A control unit that selects a feedback mode for the user's utterance mode from a plurality of modes according to information related to the user's utterance recognition.
With
Wherein the plurality of modes, viewed contains a first mode in which implicit feedback is performed, a second mode in which explicit feedback is performed, and
In the first mode, feedback including an indirect improvement method for the user's utterance mode is provided.
In the second mode, as feedback indicating a direct improvement method for the utterance mode of the user, feedback is provided to the effect that the utterance should be made to a sensor different from the sensor that detected the utterance of the user.
Information processing device,
A program to function as.

Images