JP2019124855A - Apparatus and program and the like - Google Patents

Abstract

To provide an apparatus provided with a function for performing communication, for example.SOLUTION: A robot 1 is provided with a function for outputting sound and a function for communicating with a user, the robot 1 interacts with the user by using an interactive engine, and the user is allowed to visually check the contents uttered by the user by means of a touch panel part 7 which is displayed on the touch panel part as a display part by converting the contents spoken by the user immediately before the conversation into character string data, thereby contributing correction and trend of subsequent communication.SELECTED DRAWING: Figure 1

Description

  The present invention relates to, for example, an apparatus and a program having a function of performing communication and the like.

  Patent Document 1 discloses a technology related to an interactive communication robot.

JP, 2011-000681, A

However, there is a problem that conventional communication robots do not have sufficient ability. Therefore, it is an object of the present invention to provide an apparatus, a program, and the like having a capability superior to the conventional one.
The object of the invention of the present application is not limited to this, and the intention is to acquire the right of division application / correction etc. even for the construction aiming to obtain the effects exerted from the parts of the construction disclosed in this specification and the drawings etc. . For example, in the present specification, a subject is disclosed in which a part described as “can be” is read as “a subject is a subject” in the present specification. The issues are described as being independent of each other, and the organization for solving the issues is also willing to acquire the right by divisional application / correction etc. alone. Even if the subject is grasped silently from the description of the specification, the applicant has the intention of claiming a part of the configuration described herein in the correction or divisional application. . Moreover, the subject which combined these independent subjects is also disclosed.

(1) A device comprising a function of performing communication by outputting output information to at least one of a user or another device, the function of controlling generation of the output information for the communication, or It is preferable to have a function of controlling the timing of outputting the output information for the communication.
In this way, the user or at least one of the other devices can obtain at least one of output information whose generation is controlled and / or output information whose timing is controlled. An apparatus superior to the conventional one can be provided.
The apparatus may generate output information according to communication, for example, under control of generation of the output information for communication. In this way, the convenience in communicating with the device especially by the user or another device is enhanced. Although communication may be performed by outputting any output information, it is preferable to store history information of communication in the past and to perform based on the history information. Also, it may be performed based on history information of communication with different users or other devices. In particular, although the output information may be output from one output means, it is preferable that the configuration is possible from a plurality of different output means, and one of these output means is selected for output. The output means may be, for example, an audio output means, a display means, a communication means or the like. The communication may be configured to communicate with voice, visual or other five senses, such as tactile sensation.

In particular, the "device" may have a function of outputting the output information as an audio. In this way, it is possible to control, for example, a device that operates by inputting voice. The device may comprise an interface, in particular for communication, and may comprise decision means for carrying out the communication.
In addition, although the part in which the structure of an "apparatus" is contained may be comprised with several housings, it is good to comprise with one housing especially.
Also, the device may be, for example, a system provided with a function of accessing a network through a wired or wireless connection. In particular, for example, a smartphone, a tablet terminal, a smart speaker, a smart camera or the like may be used. In addition, although the appearance is not limited, in particular, it is preferable to use a device that communicates with others. In particular, it is better to use a robot. For example, it is particularly preferable to use a robot that imitates a person or an animal, or, for example, has an anthropomorphic form other than them.

The device may have an interface on the input side to communicate, for example an interface with an input device such as a keyboard, eg Optical Character Recognition (OCR), which reads and digitizes characters, but the input It is preferable to have an audio interface as a side interface. For example, it is preferable to have a function of acquiring audio data based on an audio signal converted into an electrical signal by a microphone.
An audio interface may be provided as an interface on the output side, for example, a speaker device, an earphone, and the like. It is preferable to have a visual interface as an interface on the output side. For example, a display capable of changing the display content may be provided. It is good. In addition, for example, an output by a printed matter may be provided. In addition, it is preferable to provide an interface on the output side that actually generates motion. An actuator may be provided to actually generate movement. For example, a motor or the like may be provided. In particular, the device may be a robot with members that actually generate movement. The device may in particular output the output information as a movement of a member that actually generates a movement. In particular, as an interface on the output side, it is preferable to provide both an audio interface, a visual interface and an interface which actually generates motion.
The “user” is, for example, a person who can handle the device, and may be one person or plural persons.
The “other device” may be the same as or different from the specific one of the above devices, for example, in appearance, function, etc. Other devices may not have the function of outputting voice. It is preferable to have an audio output function. Further, the other device may not have the function of inputting voice, but may have the function of inputting voice.
The other device may be a device that can not access the network, but may be a device that can access the network. In particular, the device may have access to the Internet.
The output information may be configured to output an output from the output means. “A certain output” is, for example, a notification to the outside. Even if it is not a form of "inform", it can be interpreted as "inform" even if there is any change as a result. The "certain output" may not necessarily be intended for notification. For example, the output of sound or light having some information or no information is good, for example, change in some physical quantity, movement of an object, etc.

  (2) The device may have a display function for displaying so as to change the display mode on the display unit by the communication by voice.

When communication is performed by voice, the display mode can be changed on the display unit in relation to the communication by voice, so that the voice can be changed to a visual display and communication can be performed, and convenience in communication can be enhanced.
The “display unit” may be a device for displaying so as to change the display mode in the display unit by the communication by voice, for example, a liquid crystal display (LCD), a plasma display (PDP), an organic EL display, a cathode ray tube, etc. Such a display device is good. In particular, the display may be provided in the device.
“Display the display mode on the display unit according to the communication by voice” so as to display the voice of the user or another device on the display unit and change the mode, and also display the voice of the device itself Only one of them may be displayed on the part and the mode may be changed, but it is preferable to provide both particularly, and at this time, only one or both may be displayed, but only one is displayed. It is preferable to have both a state in which the user is allowed to display and a state in which both are displayed, so that the configuration is switchable. For example, an example in which only one side is displayed is an aspect when the face screen S1 is displayed in the robot 1 of the following first embodiment, and an example in which both are displayed is a chat screen in the robot 1 of the following first embodiment It is an aspect when S2 is displayed.
As the display mode, for example, the screen may be changed variously in relation to the voice, and for example, an animation may be performed such that an object displayed on the display screen is moved by voice. For example, it is preferable to change the image according to the sound output or to superimpose another image on the image by the change of the sound. Also, for example, voice data may be converted into character data and displayed. The display of the character data may be changed every moment according to the change of the voice.
The communication by voice may be controlled by the function of controlling the generation of the output information or the function of controlling the timing of outputting the output information for the communication, and in particular, the communication controls the generation of the output information. It is good to control by the function which controls the timing which outputs a function and the said output information for the said communication.
Further, the function of changing the display mode in the display unit may be controlled by the function of controlling the generation of the output information or the function of controlling the timing of outputting the output information for the communication, in particular, the communication It is good to control by the function which controls the generation of the above-mentioned output information, and the function which controls the timing which outputs the above-mentioned output information for the above-mentioned communication.
Similarly, in the following (3) and later, the configuration for outputting from the device is preferably controlled by the function of controlling the generation of the output information or the function of controlling the timing of outputting the output information for the communication. Particularly, the communication may be controlled by a function of controlling the generation of the output information and a function of controlling the timing of outputting the output information for the communication.

  (3) The change in the display mode on the display unit may be configured based only on the utterance of at least one of the user or the other device.

When the device changes the display mode on the display unit based on the utterance from the user or another device, the user or the other device visually recognizes whether or not the user's utterance is recognized by the device, and communication is performed. Convenience in planning is enhanced. In addition, it becomes a special communication by different human interface and feels fresh and interesting.
The utterance may be made directly from the user or another device, or indirectly, for example, the utterance may be converted into character data. Alternatively, the utterance may be converted into some corresponding information, such as another sound or a visualized pattern, and the display mode may be changed on the basis of the information.
In (3), the device itself may be a voice communication function, for example, because it is "based on the utterance of the user or at least one of the other devices".
As a configuration based on the utterance of at least one of the user or the other device, for example, voice is converted to a character string by a voice recognition function, and the utterance content of at least one of the user or the other device is converted. It is preferable to have a configuration to identify.

(4) The change of the display mode in the display unit is alternately performed with the audio output from the device.
Communication is not one-sided, and communication can be made while stably communicating. “Alternately” may be basically configured such that communication between the device side and the user or other device side proceeds in an interactive manner, and it may be configured not to have only one side speak unilaterally.

  (5) The display mode may include character information in which an utterance of at least one of the user or the other device is converted.

In this way, the user or other equipment can know specifically from the displayed character information how the user's speech is recognized by the device, and it is displayed whether communication has been correctly performed. It can be judged from the content of the text information. In addition, it is possible to visually confirm the content of the utterance. Also, if recognition is wrong, it is possible to lead again to the correct communication by re-speaking or rephrasing in another expression.
For example, in the case of Japanese, "character information" is a character based on an utterance of a user such as ordinary kanji, hiragana or katakana or other device, and when the utterance constitutes a sentence, for example, It is preferable that the sentence has a mixture of kanji, hiragana, katakana, and foreign language notation. When uttered in a foreign language, such as English or Chinese, it is preferable that these characters be displayed.
For example, it has a voice recognition function and a voice output function, and causes the display unit to display the content of the utterance of at least one of the user or the other device which is voice-recognized by the voice recognition function and converted into character information It is particularly preferable to have a function of generating response character information based on the content, converting the response character information into speech information by a speech synthesis function, and outputting the speech information.

(6) It is preferable that all the contents from the start to the end of the utterance be displayed on the display unit at the same time as the characters into which the utterance has been converted.
As the entire utterance having a certain length from the user or another device is received by the device side, correct communication from the device to the utterance can be expected. In addition, visual confirmation of the content spoken by oneself can be made instantaneously, which contributes to the correction and directionality of the subsequent communication.
The start and end of the utterance may be set in consideration of the time in which a person can speak in a single breath. The end of the utterance may be, for example, a point in time at which a predetermined amount of sound continues to be regarded as having no speech. The start of speech may be started, for example, on condition that the loudness exceeds a predetermined level. The start of the speech may be started, for example, on the condition that a sudden change in the level of the sound is detected above a predetermined level.

  (7) The device may have a function of causing the display unit to display an interaction history of communication by voice of at least one of the device and the user or the other device as character information.

It is easy to see how the device interacts with the user or other device, and the convenience in voice communication is enhanced.
The display of the dialogue history may be made to show, for example, which one has interacted. For that purpose, for example, it is preferable to provide a speech balloon to distinguish which speech information is based on which speech. About past dialogue, it is good to be able to scroll and confirm on a screen. Different avatar characters may be displayed on the device side and the user or other device side to indicate that it is interactive. It is good for the user to remember the past from the dialogue history if the date and time are displayed at the same time when it speaks

  (8) When the apparatus displays the dialog history as character information, the user or the other device is replaced, and when the dialog target of the apparatus is replaced, the display indicates that effect It is good to have a function to

There is also a change in the contents of dialogue if the dialogue target of the device is replaced. By displaying the fact that the dialogue target has been replaced, for example, when the past dialogue history is viewed, it is understood that the dialogue contents are replaced before and after that display, so that the dialogue contents can be understood. Become.
The device may have a function of detecting the change of the user or the other device. The detection of alternation preferably has a function of detecting from changes in voice characteristics, but a configuration having a function of acquiring and detecting the status of a surrounding person or device using a camera is desirable, and in particular detection based on both. Configuration.

  (9) The device may have a function of causing a display unit of the device to display a recognition status of at least one of the user and the other device by using a voice recognition function.

For example, when the user or another device is speaking, it is possible to make the user etc. understand that the dialogue is being carried out smoothly by making the user etc. understand that he or she is surely listening to it.
For example, as a function through vision, for example, the color of an object displayed on a display screen or a display screen may be changed according to the degree of speech recognition on a display unit, or different objects may be displayed according to a speech recognition situation. In accordance with the degree of speech recognition, it is preferable to display quantitatively, for example, a high numerical value if well recognized. As a function through hearing, for example, it is preferable to make the sound larger or smaller depending on the degree of speech recognition, for example, it is preferable to change the timbre.
In particular, it is preferable to display on the display portion in a personified mode, and to change the expression thereof.

  (10) A function is provided to perform the communication by performing the voice output using the result of recognizing the voice and converting it into a character string, and the result of recognizing the voice and converting it into a character string is the result in advance It is preferable to have a function of voice-outputting the output content corresponding to the character string when there is a portion that matches the character string stored in the storage means storing the correspondence between the character string and the output content.

If there is a part where the character string recognized and converted to voice matches the character string stored in the voice storage means, if necessary voice output can be performed only in the device, the user or other equipment can utter quickly It can respond. In addition, the connection cost can be reduced because the server is not connected to an external server.
For example, it is good to prepare a large number of built-in scenarios as a character string stored in the storage means. The built-in scenario is a scheduled dialogue, for example, a simple greeting such as a response from the user to "Good morning" to "Good morning" from the user, or to execute certain processing, for example Scenarios such as “Open setting screen” (user), “Is it really good” (device), “Yes” (user), “Just open the setting screen” (device), etc. are good. As a storage means, for example, a ROM or an SSD inside a computer, an external SD card microSD card, a CD-ROM or the like is preferable.

Here, “when there is a portion that matches a character string” is a case where it matches a completely stored character string and a case where it is a regular expression in which a certain portion may be different. A regular expression is one of the language processing methods for representing a set of character strings as one character string. Even if there are different parts such as "increase the volume,""increase the volume," etc., it is a case where it is recognized as an expression that "increases the volume" as an interpretation if the main parts coincide. Then, "output voice corresponding to the character string" means, for example, a voice such as "Yes, increase the volume according to the character string of" increase the volume ". The output is good. Also, the device may perform some processing following such voice output. For example, after the utterance "Yes, increase the volume", the device may actually increase the volume of its own utterance in subsequent dialogues.
The process of recognizing speech and converting it into a character string may be performed by the device, but the speech data is sent to a speech recognition server connected to the network, and the character string converted by the speech recognition server is It may be made to receive. Preferably, both may be provided, and a function may be provided to determine which result is to be used according to the communication situation or the like.

  (11) If there is no part where the result of speech recognition and conversion into a character string matches the character string stored in the storage means in which the correspondence between the character string of the result and the output content is stored in advance It is preferable to have a function of outputting voice data by connecting to a server provided with an engine.

The connection cost can be reduced because the connection is made to the server equipped with the dialogue engine when there is no part where the character string recognized and converted to speech matches the character string stored in the speech storage means.
The server may be, for example, an apparatus having a storage unit connected using an internet connection and a computer function as a control unit. The present invention may include a dialog engine. In the case of an external server, connection is possible by, for example, ID, password or electronic authentication. The external server is preferably a cloud server. The server may include a speech recognition engine, which may convert speech into text data by the speech recognition engine. The converted string data may be sent to the device using an internet connection.
The function of connecting to a server equipped with a dialog engine to output voice data, for example, transmits a voice-recognized character string to the dialog engine, and receives from the dialog engine a string including dialog content corresponding to the string. The character string of the dialogue content may be converted into speech data by a speech synthesis function.
The conversion of the character string into voice data may be performed by a voice synthesis engine provided in the device, but the character string is transmitted to a voice recognition server that converts the character string into voice data, and converted from the voice recognition server It is preferable to receive voice data corresponding to the character string.

  (12) Even if there is a portion where the result of speech recognition and conversion into a character string matches the character string stored in the storage means in which the correspondence between the character string of the result and the output content is stored in advance By satisfying the condition, it is preferable to connect to a server provided with a speech recognition engine and have a function of outputting speech data.

If there is a portion that matches the character string stored in the storage means with a character string that is recognized and converted into speech, providing a fixed voice output that can be predicted by the user also reduces the motivation for interaction. Therefore, daringly connecting to an external server in this way is better for more human interaction.
For example, the utterance as "Hello" from the user, if it was recognized device side, is in the original scenario, "Hello, your mood is How about" when was the built-in scenario to the dialogue and so on, the to request the voice data of "Hello" to an external server without the use of a scenario, it is possible to to request the creation of the response data for the "Hello" to use the interactive engine of an external server. A certain condition may be, for example, a number of times or random timing.

  (13) A function of detecting that a voice is interrupted and becomes silent after voice recognition, a storage unit for storing voice data from voice recognition to silence, and voice data stored in the storage unit It is preferable to have a function of outputting voice data by connecting to a server provided with a voice recognition engine at the timing of silence.

  Often in the dialogue there may be silence. However, even if silence occurs, connecting to a server equipped with an external speech recognition engine results in unnecessary costs. Therefore, by storing voice data from voice recognition to silence in advance in storage means and sending the voice data not at real time but at timing when silence is achieved, it is possible to cut the time of silent part, Cost can be reduced.

(14) The device may have a recording function, and may be connected to a server provided with a voice recognition engine by detection of voice of a predetermined sound pressure level to output voice data.
Always connected to a server with an external speech recognition engine costs money. This can reduce the cost for connection since the connection to the external server is made only when the necessary dialogue is started without being connected when the dialogue is not silent or close to silence.
The server provided with the speech recognition engine may receive the recording data of the past predetermined period already recorded by the device, and may return a character string for the recording data, but in particular, for example, in real time as streaming data. It is preferable to receive voice data and return a character string. Although there are many cases in which pay-as-you-go charges are often made in the form of how many yen per voice data reception time, it is possible to significantly reduce the cost.

  (15) Example of dialogue selected from dialogue data stored in storage means for the user when the server is in a busy state when connected to a server equipped with a speech recognition engine and outputting voice data It is preferable to have a function of outputting a voice.

  When busy, it is normal to notify that effect, but for example, such an alert during a dialogue is a sudden and an utterance that has nothing to do with the dialogue, and there is a possibility that the dialogue may be obscured. . Therefore, instead of announcing that it is busy, for example, if it is made to connect the dialogue by making a speech such as "Are you going again?" It is possible to connect to the engine and continue the appropriate dialogue, and the dialogue does not become unnatural.

  (16) When it is determined that the user's recognized speech is too long, it has a function of outputting an example of dialogue selected from speech data stored in storage means without connecting to a server equipped with a speech recognition engine You should do it.

  If the user speaks too long, the speech recognition engine may misinterpret. And, as a result, a wrong response may be returned. Therefore, when it comes to a sentence of a certain level or more, in a dialogue such as “Yes” or “Maj?” Or “Is it true?” In order to daringly eliminate such possibility and regroup the dialogue. It is good to select and output voice examples of interactive examples such as correlations that can be taken in any way, thereby enabling appropriate dialogue to be continued.

(17) In the communication by dialogue, when the user misses the voice of the device, the device may output the last voice again based on a certain utterance of the user.
For example, the device spoke immediately before such a call when the device speaks immediately before, such as "Say again" or "Speak again" or such a call is made if it is inadvertently forgotten You can make them speak words. By this, it is possible to continue as it is without interrupting the dialogue which has been performed until just before.

(18) The voice may be output from the device so as to prompt the user to speak again if the voice can not be recognized.
By this, it is possible to continue as it is without interrupting the dialogue which has been performed until just before.

(19) It is preferable to cause the display unit to display when the recognized voice content satisfies a certain condition.
For example, when an utterance including a predetermined word is voice-recognized, the display unit is made to make "a certain display" corresponding to the word. The “predetermined word” may be, for example, the user's birthday, the name of the user's child, the nickname of the device, the name of the company, a tagline for a specific advertisement, or the like. It is preferable to have a function of setting in advance a correspondence between a predetermined word and a certain display. By this, communication can be performed including visual observation as well as simple dialogue, and the aspect of communication with the device is increased, and convenience in communication is enhanced.

(20) The device has a function of moving the housing or a part connected to the housing, and the housing or the part connected to the housing may move when the recognized audio content satisfies a certain condition .
For example, when an utterance including a predetermined word is speech-recognized, the housing or a portion connected to the housing causes a certain movement, for example, a gesture corresponding to the word. By this, communication can be performed including not only mere dialogue but also the movement of the device, and the aspect of communication with the device can be increased, and convenience in communication can be enhanced. It is particularly good in combination with the above "a certain display".

  (21) The device has an eye portion which is a portion that the user can recognize as an eye, a user position recognition function that recognizes the position of the user, and a function that moves the eye portion; It is preferable to have a function to move the eye part so as to face the position direction of the user recognized in the above.

When the eyes that can be recognized as eyes turn to the position direction of the user, it is possible to obtain a pseudo sensation that you are actually talking to a person, and there is also a desire to communicate with the device. Improve.
The “eye portion that is a portion that can be recognized by the user as an eye” may be an eye as an object displayed on the display screen, or an eye that actually operates mechanically that is not such a virtual image. The device itself may be controlled to be directed toward the position of the user in synchronization with the eyes. Only the device for turning the eye towards the user may be controlled to turn towards the user's position.

(22) The device may have a face recognition function that recognizes the face of the user.
Since the face of each person can be identified, it is possible to communicate according to each individual personality. For example, by associating the name with the authenticated face of each person, it is possible to call the person who recognized the face during the dialogue by that name. In addition, it is particularly preferable to adopt a configuration in which a dialog specialized for a person whose face is recognized based on a past dialog history.

(23) The apparatus may include a display unit, and may have a function of causing the display unit to display the recognition status of the user's face by the face recognition function.
In this way, the user can grasp the recognition status of the user's own face by looking at the display unit. In particular, it is preferable to display whether face recognition has been completed or not yet recognized as a person's face as a recognition situation. In this way, the user recognizes as much as possible if the recognition is not yet completed. You can work together by keeping your face stationary for ease.

(24) The position recognition function includes the triangle and the position of the sound source based on the difference between the three microphones arranged at the vertex of the triangle and the arrival time of sound from the sound source to each of the three microphones. Specifying a sound source direction identifying a sound source direction from a position projected on a plane including the triangle along a direction perpendicular to the plane to a reference position inside the area surrounded by the triangle on the plane It should be a function.
This makes it possible to specify the sound source direction with three microphones. Then, if the direction of the sound source can be specified, the user can feel as if communicating by interaction, since the device can be directed to that direction if the user speaks.

(25) The device may include an infrared remote control signal output unit, and the communication may be communication with the other device having an infrared remote control reception function.
This allows easy communication with the device via the device's infrared remote control signal output.
For example, an infrared remote control signal is output from the device to a receiving side apparatus including an infrared remote control signal receiving unit, for example, a receiving side apparatus including an infrared remote control signal receiving unit, such as a television, an audio It becomes possible to execute control such as ON · FF. In particular, the apparatus may have a voice interactive function, and the apparatus may be configured to control the infrared remote control signal output unit based on the instruction when an instruction word such as "turn on TV" or "turn off TV" is uttered.

(26) The device may be controlled remotely via the Internet from the other device.
Since the device can be remotely controlled from another device, the convenience of the device is enhanced. For example, a smartphone as another device may be used. The device may be equipped with a camera. The device may be equipped with a mechanism to change the orientation of the camera. It may be accessed from another device, for example, to view a camera moving image on the apparatus side or to change the camera orientation. This allows control of the device without being close to the device. For example, it is good to access from a smart phone, for example, set a watching function of the device to ON, and notify the smart phone that a person has moved by e-mail. Further, for example, it is preferable to always move when watching for a sick person or a care receiver, and for example, when the person is not moving for a predetermined time or more, notification may be made. The other device may be, for example, a tablet terminal or a personal computer.

(27) The apparatus may output the voice using character information transmitted from the other device via the Internet.
For example, it is preferable to have a function of reading out a received e-mail character string. By setting that e-mail from someone can receive it, the contents of the e-mail are outputted from the device by voice, so it is not necessary to visually check the own terminal. The other device may be, for example, a smartphone, a tablet terminal, a personal computer or the like, but may be another “device”.

(28) It is preferable that the time, time or number of times of performing the voice output can be changed according to the contents of the character information.
For example, an e-mail saying "I took a medicine" would like to speak at a fixed time. For example, if the description of the subject line matches, the device speaks at a predetermined time, or if important content is uttered twice, for example, the mail content may be incorrect for the user near the device. It can be run without.

(29) The device may have a function of transmitting voice-recognized character information to the other device via the Internet.
If the voice communication function of the device is used to convert the voice into characters and send it as character data to another device, it is possible to send, for example, an e-mail without sending it manually to its own terminal.

(30) The voice-recognized character string is transmitted as the input character string to a plurality of different servers provided with a dialogue engine that outputs an output character string corresponding to the input character string, and the output character string from the plurality of different servers It is preferable to have a function to receive a string output as and to select and interact with the longest output string.
With the longest response, it feels as if it were interacting, the monotony of the interaction disappears, and the listener (user) can enjoy the interaction.

(31) The voice-recognized character string is transmitted as the input character string to a plurality of different servers provided with a dialogue engine that outputs an output character string corresponding to the input character string, and the output character string from the plurality of different servers It is preferable to have a function to receive a string output as and to select and interact with the output string with a question mark appended to the end.
When a question mark is attached to the end, it becomes a flow of speech that answers the question further, so that the conversation can be continued and the listener (user) can enjoy the dialogue.

(32) The voice-recognized character string is transmitted as the input character string to a plurality of different servers provided with a dialogue engine that outputs an output character string corresponding to the input character string, and the output character string from the plurality of different servers It is good to have the function to receive the character string outputted as and to make the question text combine and interact after combining the positive sentences.
Because arranging in this way makes the response look like thinking and writing sentences, the user feels like they are seriously listening to their own speech, and they want to continue talking. The conversation can be continued easily and the listener (user) can enjoy the dialogue. In addition, it is possible to gain an output scale and to request a response to a listener (user).

(33) The voice-recognized character string is transmitted as the input character string to a plurality of different servers provided with a dialogue engine that outputs an output character string corresponding to the input character string, and the output character string from the plurality of different servers It is preferable to have a function to receive the character string output as and to combine and interact so as to place the switched character string at the end.
By arranging in this way, it becomes a dialogue that invites the next utterance by having a topic change, and the dialogue becomes easy to continue.

(34) The voice-recognized character string is transmitted as the input character string to a plurality of different servers provided with a dialogue engine that outputs an output character string corresponding to the input character string, and the output character string from the plurality of different servers It is preferable to have a function to receive a string output as and to combine and interact so that the friendly output string is placed first.
This arrangement makes it easy for the listener (user) to be drawn into the dialogue and makes the dialogue easy to continue.

(35) The voice-recognized character string is transmitted as the input character string to a plurality of different servers provided with a dialogue engine that outputs an output character string corresponding to the input character string, and the output character string is output from the plurality of different servers It is good to have the function to receive the character string output as and to combine them in random order and to make it interact.
Since the variation of the dialogue increases, the same response does not return even when the listener (user) makes the same utterance, and the dialogue can be continued without getting tired of the dialogue.

(36) The voice-recognized character string is transmitted as the input character string to a plurality of different servers provided with a dialogue engine that outputs an output character string corresponding to the input character string, and the output character string from the plurality of different servers Receive the character string output as and when there is the output character string including the emoticon in them, it is not the dialog target, and displayed together with the output character string targeted for the dialog on the display unit It is good to have a function to
Although emoticons can not be output as voices, by displaying emoticons on the display unit, it is possible to create unusual fun of dialogue by combining them with speech as part of the dialogue.

(37) The voice-recognized character string is transmitted as the input character string to a plurality of different servers provided with a dialogue engine that outputs an output character string corresponding to the input character string, and the output character string from the plurality of different servers It is preferable to have a function of receiving the character string output as and selecting only one of the output character strings including the same character string and interacting with another output character string.
If the same string is repeated, the dialogue will be disturbed, and the listener may feel uncomfortable.

(38) The voice-recognized character string is transmitted as the input character string to a plurality of different servers provided with a dialogue engine that outputs an output character string corresponding to the input character string, and the output character string from the plurality of different servers It is preferable to have a function of receiving the character string output as and converting the end of the output character string by the word conversion engine and then combining and interacting.
It should be a more familiar expression than the text of a normal dialogue engine.

(39) The voice-recognized character string is transmitted as the input character string to a plurality of different servers provided with a dialogue engine that outputs an output character string corresponding to the input character string, and the output character string from the plurality of different servers Receive the character string output as ???, and store some of the output character strings in the storage unit without using all the output character strings, and take them out from the storage unit in subsequent dialogues and use them for dialogue It is good to have a function to
The dialog can be connected without interruption by using it when speech recognition fails or when a response from an external server does not easily come, contributing to a natural dialogue.

(40) When using a server equipped with a speech recognition engine, it is preferable to mix and use a free server and a paid server.
This enables, for example, heavy users of the device interaction to save on server connection charges.

(41) The other device may be a smart speaker, and the device may output voice to the smart speaker to communicate with the smart speaker.
Since the smart speaker does not have a display unit, the convenience is enhanced by using it in combination with the device.
The smart speaker may be, for example, a speaker having a wireless communication connection function and an assistant function of voice operation. For example, GoogleHome, AmazonEcho, LINE Clova etc. are good. The smart speaker may have, for example, a function to realize various functions / skills. It is possible to execute the function by uttering to the smart speaker from the device that communicates by voice. For example, when uttering from the device, the user utters a phrase for activating the skill of the smart speaker, the device performs speech recognition of the utterance, stores it as character string data, and synthesizes the character string data at a certain timing. It is preferable that the smart speaker be uttered to execute the skill.

(42) The other device may be a smart speaker, and the device may output sound to the smart speaker to communicate with the smart speaker.
Activate smart speakers or activate skills. Or make an inquiry. Even if the smart speaker is not activated by itself, it is possible to automatically execute the smart speaker skill at a predetermined timing or at a certain scheduled time.

(43) The other device may be a smart speaker, and the device may have an adaptation function for adapting an audio output of the smart speaker.
This is useful when it is troublesome to listen to a fixed long answer of a smart speaker in response to a question or when you want to roughly reconfirm the contents.

(44) The audio output of the device may have a function of reading web articles.
Even if you do not read Web articles, you can listen only by interacting with the device.

(45) It is preferable that the certain output be performed when execution of a predetermined processing unit of robotics process automation is completed.
Robotics process automation is difficult to understand the execution status of the processing unit, but by making the device “output” according to the execution of the processing unit, the processing status can be easily understood and the convenience is enhanced.

(46) The certain output may be a notification operation.
It will be understood that a certain output has been made by the notification operation.
(47) When the computer in execution of robotics process automation is in the state of waiting for an input from the user, the notification operation is performed.
As a result, it is possible to inform the user that the input is in the waiting state and to prompt the next processing.

(48) A client computer for performing robotic process automation, and a server computer for giving an instruction to execute robotic process automation to the client computer, and when the client computer receives an instruction from the server computer, a notification operation is performed. You should do it.
This makes it possible to inform the user that there is an instruction from the server computer and to prompt the next processing.

(49) The output information may be generated to perform an operation pointing at a computer executing robotics process automation.
This makes it possible for the user to know on which computer the execution has been performed and to prompt the user to perform the next processing.
(50) It is preferable that different output information be generated depending on the execution state of robotics process automation.
This makes it possible to distinguish what kind of execution has been done.

(51) A program for causing a computer to realize the function of the device according to any one of (1) to (50).
For example, a portion described as "presence" such as "a certain output" may be "predetermined".
The inventions described in (1) to (50) above can be arbitrarily combined. For example, at least a part of the configuration of at least one invention of (2) or later may be added to the configuration of all or part of the invention shown in (1). In particular, at least a part of the invention of (2) or later may be added to the invention of (1). Further, any configuration may be extracted from the inventions shown in (1) to (50), and the extracted configurations may be combined. The applicant of the present application has the intention to acquire the rights to the invention including these configurations. Further, even if there is a description of "in case of""at time of", it is not described as a configuration limited in that case or at that time. We also disclose the configuration in these cases and not at times, and have the intention to acquire the right. Moreover, the part which is described with the order is not limited to this order. It also discloses the configuration in which some parts are deleted or the order is changed, and has the intention to acquire the right.

When communicating with a user or another device, the device can generate output information according to the communication. Therefore, the convenience when communicating with the device in the user or another device is enhanced.
The effect of the invention of the present application is not limited to this, and the effect to be exerted from the portion of the configuration disclosed in the present specification and the drawings is also disclosed. Have the intention to For example, in the present specification, a portion described as “can be” is a description that clearly indicates the effect to be exerted, and there is a portion showing an effect even if it is not described as “can”. In addition, even if such a description is not made, there are effects to be grasped by the configuration.

The front view of the robot of Embodiment 1 concerning this invention. The side view of the robot of the same embodiment 1. The back view of the robot of the same Embodiment 1. FIG. FIG. 2 is a block diagram illustrating an electrical configuration of a robot. Explanatory drawing which captured an example of the facial expression with the face screen displayed on the face part of a robot. The explanatory view which caught an example of the chat state with the chat screen displayed on the face part of a robot. Explanatory drawing explaining the standby image which makes the background the face screen displayed on the face part of a robot. Explanatory drawing explaining the standby image which makes the background the chat screen displayed on the face part of a robot. (A)-(d) is explanatory drawing explaining the deformation | transformation pattern of the eye object displayed on the face part of a robot. (A)-(c) is explanatory drawing explaining a mode that the user's utterance content appears gradually as a character string on the face part of a robot. Explanatory drawing explaining the state which the eye object is following the user's face, and is moving in the face screen displayed on the face part of a robot. Explanatory drawing explaining an example of a smart phone. Explanatory drawing explaining the relationship of starting of a robot-wake up mode-dialog mode-sleep mode. Explanatory drawing explaining the relationship between a robot and a smart speaker in Embodiment 7. FIG. FIG. 32 is an explanatory diagram for explaining an example of a smartphone in Embodiment 9;

Embodiment 1
As shown in FIGS. 1 to 3, the robot 1, which is a communication robot that operates in response to human voice, has a fixed unit 2 that is a lower body and a movable unit 3 that is an upper body placed on the fixed unit 2. Is provided as a housing. The movable portion 3 is composed of a body 4 adjacent to the fixed portion 2 and a head 5 supported by the body 4. The fixing portion 2 is formed in a bowl-like appearance opened upward, and the body portion 4 is formed in a cylindrical shape formed of a continuous curve in the vertical direction with the upper edge of the fixing portion 2. In the robot 1, the connecting portion between the fixing portion 2 and the body portion 4 is formed to have the largest diameter, and the connecting portion is shaped so as to be narrowed in the vertical direction at the boundary. The body portion 4 is largely cut out in a semicircular shape at the front of its cylindrical shape. The head 5 is fitted to be embedded in the upper portion of the body 4. The body 4 rotates in the horizontal direction (the direction of the arrow in FIG. 1) with respect to the fixed portion 2 and the head 5 with respect to the body 4 in the longitudinal direction (the direction of the arrow in FIG. 2) In the direction of the arrow).

  The head 5 is generally formed in a spherical shape, which is the remainder of a portion of the sphere (the front surface portion) cut in one plane. The front portion formed in a cut shape appears in a circular shape and constitutes the face portion 6 of the robot 1. A rectangular portion formed on the surface of the face portion 6 is a touch panel portion 7 as a display portion which is a liquid crystal display (LCD) having a touch panel function. The contents displayed on the touch panel unit 7 will be described later. A smoked panel is disposed in the area of the face portion 6 around the touch panel portion 7 so that the entire face portion 6 has a uniform dark background. An illuminance sensor 8 and a high-brightness white LED 9 are respectively disposed at upper left and right accommodation positions of the face portion 6 inside the head 5. The lens 11 of the face recognition camera 10 is disposed at the upper center position of the touch panel unit 7 in the face unit 6.

  The microphones 12 are disposed at three positions of the same height offset by 120 degrees in the front left and right positions of the trunk 4 and the rear center position inside the trunk 4. A pair of left and right speaker devices 13 is disposed on the fixed portion 2 inside the fixed portion 2. An opening 14 for outputting the sound generated by the speaker device 13 is formed on the side of the speaker device 13. At positions adjacent to the speaker opening 14, an up switch 16 and a down switch 17 for adjusting the volume of the power switch 15 and the speaker device 13 are respectively disposed. A terminal 18 for OTG (On-The-Go) of USB, a power jack 20 for DC 12 V, and a socket (reader) 19 for micro SD card are disposed at the rear position of the fixed part 2.

  The robot 1 can be connected to a predetermined external cloud server using an internet connection. The cloud server includes a storage area as storage means for storing data required by the robot 1, various engines for performing various processes required by the robot 1, and the like. Therefore, in a broad sense, the robot 1 can be interpreted as an apparatus including portions of software and the like of these cloud servers.

Next, the electrical configuration of the robot 1 of the first embodiment will be described based on the block diagram of FIG.
The touch panel 7, the illuminance sensor 8, the high brightness white LED 9, the face recognition camera 10, the microphone 12, the speaker device 13, the terminal 18, and the micro SD card socket 19 are connected to the controller MC as control means In addition to the above, the wireless LAN device 21, the Doppler sensor 22, the first to third motors 23 to 25 and the like are connected.

  The touch panel unit 7 has an input operation function of inputting by touching the surface. The touch panel unit 7 can display different screens as shown in FIG. 5 or 6 in the natural interaction mode described later. The controller MC causes the touch panel unit 7 to displaceably display the facial expression of the robot 1 as shown in FIG. The face screen S1 is a screen displayed by default, in which the eye object 27 of the robot 1, the cheek object 28 and the elliptical area 29 are displayed. The eye object 27 is provided with deformation patterns of several eye objects 27 as animation images (FIGS. 9A to 9D). In addition, the pupil object 27a moves to the left and right as an animation image.

Further, the controller MC causes the touch panel unit 7 to display a chat screen S2 as shown in FIG. 6 as a second screen. The chat screen S2 can display the chat screen S2 on the touch panel unit 7 instead of the face screen S1 by touching and sliding the touch panel unit 7 in the state of the face screen S1. The face screen S1 and the chat screen S2 can be mutually switched by slide operation. The chat screen S2 will be described later.
Further, the touch panel unit 7 can display the standby image of FIG. 7 or 8 on the touch panel unit 7 in the standby mode (wake-up mode). The standby image will be described later.
The robot 1 is provided with a setting screen as a different image other than these, and can be moved from the chat screen S2 to the setting screen. When the robot 1 is activated for the first time, access is made from the setting screen, for example, setting registration of the ID and password to the server, setting registration of the Wi-Fi password, user registration (eg name, age, gender etc.), face authentication Input necessary initial setting items such as e-mail address settings to which data is to be transferred.

The illuminance sensor 8 recognizes the brightness of the environment in which the robot 1 is installed. The high brightness white LED 9 is automatically turned on based on the numerical value detected by the illumination sensor 8 when the light intensity is insufficient for photographing by the face recognition camera 10.
The microphone 12 is an audio input means for acquiring the user's speech in the dialog with the user, and simultaneously using three microphones 12 arranged at the apex of the triangle, the arrival time of the sound among them It is also a direction detection means that can specify the sound source direction by the difference The controller MC obtains the arrival time difference from the phase difference of the electric signals acquired by the microphones 12. The controller MC calculates the sound source angle with respect to the reference direction based on the arrival time difference. For example, a microphone specialized for interaction with the user may be provided on the face portion 6.
The speaker device 13 is an audio output unit with which the robot 1 utters (voice output) in the dialog with the user.
The micro SD card socket 19 reads and rewrites data of the inserted micro SD card.
The wireless LAN device 21 is a device for wirelessly connecting the robot 1 which is a Wi-Fi compatible device to the Internet. In this embodiment, the international standard of IEEE 802.11b is used.
The Doppler sensor 22 is a sensor using microwaves, emits microwaves, compares the frequency of the reflected microwaves with the frequency of emitted radio waves, and determines whether the object (person) is moving or not. To detect. It utilizes that the frequency of the reflected wave when the object (person) is moving is changed by the Doppler effect. For example, it is a device used to detect an abnormality around the robot 1, such as the presence or absence of a suspicious person when the user is absent.
The first motor 23 is a servomotor for rotating the body 4 in the horizontal direction (the direction of the arrow in FIG. 1) with respect to the fixed portion 2. The second motor 24 is a servomotor for rotating the head 5 in the longitudinal direction (the direction of the arrow in FIG. 2) with respect to the body 4. The third motor 25 is a servomotor for rotating the head 5 relative to the body 4 in the left-right rotational direction (the direction of the arrow in FIG. 3). When the direction in which the user speaks is determined by the microphone 12, the controller MC controls the first motor 23 so that the face 6 faces the user's direction, and the trunk 4 (movable Turn part 3).

The controller MC is composed of a known CPU, ROM, RAM, memory as storage means such as an SSD, a bus, a real time clock (RTC), and the like. In the ROM of the controller MC, various programs for executing various functions of the robot 1 are stored.
As various programs, for example, a dialogue program for controlling dialogue with the user through the microphone 12 and the speaker device 13, a face recognition program related to face recognition using the face recognition camera 10, the touch panel unit 7, the first to the first A display fluctuation / gesturing program for changing the facial expression or the motion of the robot 1 during the dialogue with the robot 1 by controlling the motors 23 to 25 of FIG. Screen display program for displaying an image on the touch panel unit 7 Data transmission / reception program for processing e-mails from camera images, smartphones, etc. acquired on the robot 1 side with other computers or smartphones, user absent Out-of-time setting program for watching over, GUI Noh net connectivity, OS or the like for operation, operation and operation of the process management, etc. are stored. In the RAM, input / output data and the like in dialogue and face recognition are temporarily stored. Each program can realize functions such as dialogue, face recognition, gesture, etc. in multitasking in cooperation with other programs or independently.

A. Regarding Operation Contents at the Time of Dialogue In the above-described configuration, the controller MC controls communication by dialogue with the user by executing the dialogue program. The face recognition that can be performed in synchronization with the start of the dialogue will be described later in “B. About the operation contents at the time of face recognition” below.
Here, the dialogue program is
1) A request is issued to the cloud server for user utterance data (voice data) acquired from the microphone 12 and a response is made to the user utterance data (character string data) converted into text using the server-side voice recognition engine Subprogram 2) Built-in scenario subprogram for executing built-in scenario dialogue based on user's utterance (character string data) 3) Cloud data of utterance data (character string data) again when the user's utterance does not correspond to the built-in scenario Utterance data transfer subprogram 4 to make the dialog engine create response data (character string data) of the robot 1 using the interaction API (application programming interface) Character string data display subprogram 5 to display on the touch panel unit 7 as a display unit) response data (character string data) that has been responded is converted into speech data by a speech synthesis engine and speech from the speaker device 13 on the robot 1 side Voice data subprogram 6 for outputting voice as a display mode / operation variation subprogram for varying the display mode on the touch panel unit 7 or the operation of the robot 1 based on the user side character string data and the robot 1 side character string data,
Etc.
Hereinafter, an example of the control contents of the controller MC mainly based on the dialogue program will be described together with the relationship between the standby mode (wakeup mode) after activation, the natural dialogue mode, and the sleep mode. These interrelations are as shown in FIG.
FIGS. 5 and 6 show screens in the natural dialogue mode, and FIGS. 7 and 8 show screens in the standby mode. In the sleep mode, these screens are darkened because the backlight of the touch panel unit 7 is turned off.

1. Startup The robot 1 is started by turning on the power switch 15 (process M0 in FIG. 13). The controller MC executes the boot program, and then when the OS is booted up, the OS waits for an “instruction” from the user, ie, wakes up. In this initial standby mode, the standby screen of FIG. 7 is displayed.
In the following, the state after the initial setting is completed, that is, the ID and password of the robot 1 are registered in the cloud server, the user registration is completed, the face authentication of a plurality of users is performed, and the e-mail address of the smartphone is the robot 1 It is assumed that it starts after being registered in
[1. Effect on launch
As described above, one screen (FIG. 7) selected from the plurality of standby screens is displayed at first. That is, a fixed screen is always displayed at startup. The user can easily recognize that the robot 1 is in the standby mode from the fact that the eyes of the robot 1 are closed (indicating that the user can not interact).

2. Standby mode (wake up mode)
The standby mode is a state in which interaction with the robot 1 is enabled when there is a trigger for the start of the natural interaction mode. In addition, it is also a state of transition when the dialogue ends in the natural dialogue mode. In addition, if the natural dialogue mode is not started for a certain period of time, the sleep mode is set. The sleep mode is a state in which communication with the robot 1 can not be achieved by dialogue.
Here, "natural interaction" means that the user interacts with the voice on the side of the device (robot 1) on which voice synthesis is performed using the built-in scenario of the robot 1 or the interaction engine (interaction software) on the server. The natural dialogue mode is a state in which natural dialogue is possible.
There are a plurality of screens in the standby mode, and two types shown in FIG. 7 and FIG. 8 are prepared in this embodiment.
FIG. 7 shows a standby screen for transitioning from the screen in the natural dialogue mode of FIG. 5 (process M2 of FIG. 13). It is also a standby screen displayed at the time of activation by the process M0 of FIG. In FIG. 7, the layer screen of the face screen S1 with the eyes of the robot 1 (eye object 27) closed is lightly displayed on the background of the screen of the clock layer where the current time and day are displayed largely. .
FIG. 8 is a standby screen for transitioning from the screen in the natural dialogue mode of FIG. 6 (process M2 of FIG. 13). In FIG. 8, the layer screen of the chat screen S <b> 2 is lightly displayed on the background of the clock layer on which the date and time, the day of the week, and the current time are largely displayed. That is, it is in the standby mode but not in the natural dialogue mode.
[2. Effect in standby mode (wake up mode)
As described above, since different standby screens are prepared, when the natural dialogue mode is started from a certain standby screen, the user can perform a dialog on the screen accessed immediately before, which is convenient. Further, by displaying a screen unique to the standby mode, the user can easily recognize that the robot 1 is in the standby mode.

3. Starting and stopping the natural interaction mode From the state of being in the standby mode or the sleep mode, the controller MC performs processing to shift to the natural interaction mode by a plurality of timings as follows, that is, triggering of mode transition (FIG. 13) Processing M1, M5). The following trigger is an example. In the natural dialogue mode, the mode is switched to the face recognition mode as described in “B. Operation contents at the time of face recognition” below (a face can be recognized).

1-1) In the standby mode, when the controller MC recognizes an utterance (voice) such as "Ye, Yupibo" from the microphone 12 as a start-up phrase within a predetermined time, the controller MC sets it as a natural dialogue mode (FIG. 13). Process M1). In addition, when it is determined that there is a touch operation on the touch panel unit 7, the natural interaction mode is set as a trigger (processing M1 in FIG. 13).
1-2) In the sleep mode, the controller MC determines whether or not there is a touch operation on the touch panel unit 7 of the standby screen at a predetermined timing. The touch operation on the touch panel unit 7 differs depending on the display mode in the face unit 6, and a touch on the entire touch panel unit 7 is on the standby screen of the face screen S1, and a dialog start button object 36 described later is on the standby screen of the chat screen S2. It starts with That is, different screens are started by different operations.
If it is determined that there is a touch operation on the touch panel unit 7, the controller MC once enters the standby mode (processing M3 in FIG. 13), and subsequently determines that there is another touch (in FIG. 13, the natural interaction mode). M1).
1-3) In the sleep mode, the controller MC determines whether there is a touch operation on the touch panel unit 7 as in 1-2). When it is determined that the touch operation has been performed, the controller MC once enters the standby mode (processing M3 in FIG. 13). In this state, when a speech (voice) such as "Hey, Ipipbo" is recognized from the microphone 12 as a start-up phrase within a predetermined time, the controller MC takes it as a trigger to set it as a natural dialogue mode (process M1 in FIG. 13) .
2) In the sleep mode, the controller MC determines whether or not a predetermined time preset by the RTC has come, and sets the natural interaction mode at the timing when the predetermined time has come (process M5 in FIG. 13).
3) In the sleep mode, the controller MC outputs speech (voice data) randomly present at random time intervals from the speaker device 13 by using, for example, random numbers generated at predetermined timing. That is, as a kind of soliloquy, for example, the robot 1 outputs a voice that invites a dialogue such as "Hey, what are you doing?" Or "I'm free!" To make it a natural dialogue mode (processing M5 in FIG. 13), Prompt the user to speak.
4) In the sleep mode, the controller MC determines by the Doppler sensor 22 whether or not the object (person) is moving, and sets the natural interaction mode at the timing when it is detected that the object (person) is moving (processing M5 in FIG. 13). ).

5) In the sleep mode, when the controller MC senses a weather abnormality or a change in weather such as an earthquake, it notifies the user of it and starts a natural dialogue triggered by this. The external cloud server acquires and stores abnormal weather information including, for example, weather abnormalities (for example, heavy snow, typhoons, etc.), earthquakes, lightning strikes, etc. at predetermined time intervals using a abnormal weather detection engine on a fixed basis. The predetermined time may be all the same timing, or the timing of acquisition may be changed according to the content of the weather. In the first embodiment, the abnormal weather information is distributed from the server to the device (controller MC) by adopting a push type distribution system, for example. When the controller MC acquires the information, the controller MC sets the natural interaction mode (process M5 in FIG. 13).
6) When the controller MC does not detect speech from the user for a certain period of time in the above-described 1) to 5) in the natural interaction mode, the controller MC enters the standby mode (processing M2 in FIG. 13). After time, the sleep mode is set (processing M4 in FIG. 13). The lengths of these mode displacement times can be set and changed as appropriate, for example, by the terminal device or by speech as a built-in scenario.

"3. Effect on start and stop of natural dialogue mode
By preparing the start of various types of natural interaction modes in this way, the user interacts with the robot 1 at various timings, which increases the opportunity to interact with the robot 1, thereby increasing the opportunity to enjoy the natural interaction. As a result, the user feels the merit of owning the robot 1. In addition, since the sleep mode is set after the interactive mode is ended once after the standby mode, the power cost is reduced.
Moreover, since the screen of the natural dialogue mode is displayed by jumping over the standby mode from the sleep mode, the dialogue can be started immediately and the dialogue start is smooth. In addition, since the dialog screen (FIG. 5 and FIG. 7) is displayed as long as the dialogue continues, the user is motivated to interact.

4. Built-in scenario dialogue in natural dialogue mode In natural dialogue mode, several dialogue processes of normal dialogue using the dialogue of built-in scenario and the dialogue engine of the server are prepared.
The controller MC first determines whether the utterance data (string data) based on the user's utterance conforms to the built-in scenario, otherwise the dialogue using the dialogue engine via the cloud server (hereinafter referred to as normal) Control to be interactive). From the user's point of view, it seems to always interact with the robot 1, but in reality there will be multiple internal processes in the natural interaction mode.
The controller MC executes processing of comparing the user's speech (character string data) created by the speech recognition engine on the cloud server side with the text data of the built-in scenario (script). In the present embodiment, the text data of the built-in scenario is stored in the memory. The built-in scenario may be added to the SD card. If it is an SD card, it is easy to rewrite built-in scenarios one after another by rewriting.
When the controller MC recognizes the user's speech, it determines whether the character string data matches the scheduled regular expression or non-regular expression, and if it matches, the script corresponding to the character string data is voice data by the speech synthesis engine And the speaker device 13 outputs a voice as an utterance of Robo and 1.
The built-in scenario, for example to promote the user's utterance "Hello" and a simple scenario, such as the greeting such as "today sounds good weather", in order to determine the processing of something based on the speech from the user Many built-in scenarios are set up (provided), such as scenario-like ones. Tables 1 to 3 disclose an example of such a built-in scenario. Of course, many built-in scenarios are set in addition to these built-in scenarios.

If the interaction is not performed according to the built-in scenario, the interaction in the built-in scenario ends halfway. The case where the dialogue is not conducted according to the built-in scenario is, for example, the following case.
1) In the case where it does not match the scheduled regular expression or non-regular expression This is the case where the built-in scenario does not match the regular expression or the non-regular expression from the beginning or from the middle. Also included is the case where the user's slippage was bad and the utterance could not be acquired correctly. In this case, the controller MC determines that it is a normal dialogue and immediately connects to the external cloud server, and thereafter issues a request for speech data to the external cloud server, and the external cloud server side dialogue engine sends character string data Create a formatted response data. Then, the response data is converted into voice data by the voice synthesis engine and the dialogue is continued by causing the speaker device 13 to output voice.

2) When the dialog in the built-in scenario is finished as scheduled For example, when the user utters "Are you OK to do?" According to the scenario, "Yes" or If there is a positive utterance such as "Please" and the dialogue in the built-in scenario is finished as scheduled, the dialogue disappears, or the dialogue disappears in the middle of the scenario. In this case, the standby mode is set after a predetermined time.
3) In the case where the user's utterance is negative about whether to proceed with a certain process When the user utters, for example, "Are you OK to do?" The built-in scenario also ends when there is a negative utterance such as "No" or "Wrong" instead of a positive utterance such as "Yes" or "Please", and the subsequent dialogue is 1 Or 2).
In the case of this negative utterance, the controller MC confirms whether or not to cancel the process, such as "Is it really good?" By this, it is possible to cope with the user's mistake or mental change. For example, in the power off scenario, when the user utters a question asking "Is it really necessary to turn off the power?" According to the scenario, if the user utters "Yes" The question "Does it really do not need to be turned off?" Is repeated several times (for example, three times in the embodiment), and if "Yes", the dialogue in the built-in scenario is ended.

"Effect of built-in scenario"
If the built-in scenario is prepared like this, it is not necessary to request an external server for all interaction, and it can be processed inside the device, so the communication cost for connecting to the server is reduced, and the communication time and server side Since the calculation time becomes unnecessary, the user does not feel uncomfortable that the response to the user's speech becomes too late and the conversation is interrupted. Further, for example, when executing a predetermined process, by providing such a built-in scenario, the user needs to operate the touch panel unit 7 to execute the process or access the robot 1 from another terminal. There is no need to interact with the process and it is user-friendly.

5. Request and response in normal dialogue On the other hand, when the utterance data (character string data) is not a built-in scenario, the controller MC connects to the cloud server and transmits utterance data (character string data) to the server again, and response data by the dialogue engine Request to create When the user has been authenticated, the controller MC sends authentication information (for example, an ID and a password) for each user at the beginning of the speech data in the request. In that case, past dialogue information is added to create response data. On the other hand, when the user can not be authenticated, the authentication information is not transmitted as a person who has not been identified in the past, so the past dialogue information is not added. When there is a request, the cloud server makes the dialogue engine create response data (character string data) based on the utterance data (character string data) using the dialogue API (Application Programming Interface), and makes the robot 1 (controller MC) Respond. If there is a past user interaction history, response data is created in consideration of the contents. The controller MC converts the response data into voice data and causes the speaker device 13 to output it as an utterance on the robot 1 side.
Similar to the above built-in scenario, if there is a user's silence for a certain period of time or more, it will be in the standby mode.
[5. Effects of request and response in normal dialogue "
Unlike built-in scenarios, connecting to an external cloud server and interacting normally makes it possible to perform advanced dialogue analysis with a significant amount of data more quickly than built-in scenarios, and it seems that you are actually interacting with people Advanced dialogue can be realized.

6. About the behavior of the robot 1 at the time of interaction The controller MC performs the following operations while the dialog is being performed in the natural dialog mode regardless of the built-in scenario or the normal dialog. ~ D. To control the behavior like.
B. The gesture controller MC of the robot 1 in the start-natural dialogue mode-standby mode controls the first to third motors 23 to 25 at various timings of the robot 1 to change the posture of the robot 1. The following is an example.
1) At start-up: When the face portion 6 of the head 5 is not facing the front, or when the head 5 is tilted, the head 5 is moved to the default position on the front.
2) When the screen is touched: the same as 1) (for facing the camera 10 for face recognition in face recognition with the user)
3) At the time of the utterance occurrence of the trigger "Nee yupibobo": 1) (in order to make the camera 10 for face recognition in face recognition face the user)
4) At the time of voice direction detection: The face portion 6 of the head 5 is directed in that direction.
5) As special emotional speech, for example, when happy: The third motor 25 is controlled to turn the head 5 in the left-right direction (clockwise and counterclockwise) with the face portion 6 facing the user.
6) As a special utterance, for example, in the case of sadness: The second motor 24 is controlled so as to stand still for a while with the head 5 nodding and then return to the default position.
7) emotions as speech, for example, "Good morning,""Hello,""Goodevening" during the speech occurs for greeting such as "Hello": to control the second motor 24 so that the head 5 to bow.
8) As a non-special emotional speech, for example, when an utterance of a simple positive communication term such as "Oka""Ok""Oh""Yeah""Yeah!" Nod head 5 To control the second motor 24. In 6 to 8), it is preferable to change the speed and time of the second motor 24 so that sadness and bowing and contact may be different.
9) As a non-special emotional speech, for example, when an utterance of a simple negative communication term such as "No", "I can not" or "I'm sorry": Move the movable part 3 to the left or right several times The first motor 23 is controlled as follows.
10) Various gestures, for example, rotating the head 5 in the lateral direction (clockwise and counterclockwise) or back and forth according to special emotional speech and non-special emotional speech and various dialogues, It may be combined with rotating the whole movable part 3 to the left and right, or rotating it so as to make it larger or smaller.
The gesture of the robot 1 may be combined with the display on the touch panel unit 7 (face unit 6) described below.
[6. About the action of robot 1 at the time of dialogue
By making the robot 1 make such gestures as described above, the user feels familiar with the robot 1 and enjoys the interaction with the robot 1 and the pleasure of actively touching the robot 1 at the same time.

B. Change in display mode in the state of the face screen S1 1) When the user speaks from the user When the controller MC obtains the user's speech from the microphone 12 and recognizes it, the face of the touch panel unit 7 as shown in FIG. In the screen S1, the elliptical area 29 is displayed in blue, and animation display is performed to change the area (that is, the size) of the area according to the user's speech volume. Specifically, the controller MC enlarges the elliptical area 29 while maintaining the elliptical shape when the volume of the user's speech increases, and reduces the elliptical area when the volume decreases. In addition, the cheek object 28 is displayed in green.

Further, the controller MC causes the touch panel unit 7 to display the user's utterance data (character string data) sent from the cloud server in a predetermined manner.
For example, as shown in FIG. 5, when the touch panel section 7 of the face screen S1, the example of the user to get the speech as "Hello", the controller MC is based on the utterance layer screen face screen S1 "Hello" Display the string. As the order, this display is started earlier than the speech of the robot 1 which is the reply of the speech of the user.
As the display mode, for example, the face screen S1 is displayed so as to gradually become opaque from the transparent state as shown in FIG. 10 (a) to FIG. 10 (b), and finally it is displayed behind as shown in FIG. Make the face screen S1 completely hidden. In other words, display the character string gradually. This figure shows the layer screen with the text displayed in reverse, after stopping displaying the state of FIG. 10 (c) with only the text displayed bright against the dark background for a very short period of time 10 (c) → FIG. 10 (b) → FIG. 10 (a) and gradually disappear to return to the default face screen S1. At this time, all the character strings in one utterance appear simultaneously and are displayed so as to disappear simultaneously. This display mode is an example, and may be displayed in a different mode.
[6. About effect of robot 1's action at the time of dialogue
As a result, the user can see his / her words on the robot 1 by eye, so that it can check whether the robot 1 has correctly heard, and can judge whether the dialog is definitely being carried out, which is a strange dialogue. Can lead to dialogue. In addition, although the off-target conversation is just frustrating, it is possible to change the way of speaking and try to talk again because the reason is also understood by confirming.
In addition, the user's speech data is subject to both the built-in scenario target and the normal interaction target to request the cloud server to perform character string data once, so the process of string data digitization does not take time, and such characters Since the creation of response data by the dialogue engine is requested for the first time after the column data, the display of the user's speech on the touch panel unit 7 can be performed at least before the speech by the reply data of the robot 1. There is no risk of mistakes.

When the speech from the user is a character string, the lengths are not the same because they differ depending on the speech. In addition, when it is not a word but a "sentence" with a clause, it may be quite long. The controller MC adjusts the font size of the character string so that the contents of one utterance are all simultaneously displayed on the touch panel unit 7 even in the case of the utterance of such a long sentence. That is, if one utterance is short, the font is displayed in a large font, and as the one utterance is long, a relatively small font is displayed.
[6. Effects of Robot 1 on the Action of Dialogue 1
This makes it difficult to interrupt the dialogue until the full text appears, and makes it difficult for the next user's utterance and fogging, because the user can confirm it with a single visual check, no matter what the utterance. In addition, since one utterance appears simultaneously at a time, the whole sentence can be understood at once, and even if the displayed time is short, the user can fully understand it. Further, since the character strings are expanded on the entire touch panel unit 7, each character can be displayed large and easy for the user to view, and even a very short display time can be sufficiently confirmed.

2) When the robot 1 speaks, the elliptical area 29 is displayed in red on the face screen S1 as shown in FIG. 5 of the touch panel unit 7 and the area (that is, the size) of the area changes according to the speech volume of the robot Display animation to make Specifically, the controller MC enlarges the elliptical area 29 while maintaining the elliptical shape when the output level from the speaker device 13 increases, and reduces the elliptical area when the volume decreases. In addition, the cheek object 28 is displayed in light red.
[6. Effects of Robot 1 on Action of Dialogue 4
As described above, the display mode on the face screen S1 is different according to the alternate dialogue between the user and the robot 1, and there is an interest that the dialogue is alternately performed not only on the actual dialogue but also on the screen, and the conversation is broken. become.

Ha. Change of Display Mode of Chat Screen S2 A display mode according to the dialogue of the chat screen S2 of the touch panel unit 7 will be described based on FIGS. 6 and 8. As described above, the display changes from the face screen S1 to the chat screen S2 by the operation of the user.
First, the configuration of the chat screen S2 will be described again.
As shown in FIG. 6, at the lower left position of the chat screen S2, the user object 31 is disposed as an avatar character, and at the lower right position, the robot object 32 is also displayed oppositely. In the user object 31, different objects are prepared for each authenticated user recognized in the face recognition mode described later or for a user without authentication, and different objects are displayed according to the currently interacting user. In the central area, a balloon object 33, in which dialogue contents of the user side and the robot 1 side are arranged in a character string, is sequentially displayed along the time axis. A dialogue stop button object 34 is displayed at the upper left position of the chat screen S2. A setting button object 35 is displayed at the upper right position of the chat screen S2.

In the chat screen S2, the balloon object 33 is displayed so as to be added every moment according to the interaction between the user side and the robot 1 side. In the balloon object 33, the user's utterance content converted into character string data and the utterance content of the robot 1 similarly converted into character string data are displayed in a line along the time axis and can be displayed on the chat screen S2. The latest utterance content is displayed in the new balloon object 33 at the lowest position of the past balloon object 33 in synchronization with the utterance.
The direction of speech of the balloon object 33 is indicated so as to indicate whether it is the speech contents on the user side or the speech contents on the robot 1 side. Since it is not possible to display all the dialogue history on the screen at one time, the chat screen S2 has a screen configuration which can be scrolled in the vertical direction, and the balloon object 31 can be displayed retroactively. When not going back to the past, the balloon object 33 of the latest dialogue is always displayed.
In the first embodiment, after the dialog is ended and the standby mode is set, the dialog is resumed, and at that time, when the user recognized again in the face recognition mode to be described later is changed, the balloon object 33 is displayed In the middle of the display of “user alternation” is displayed, and a different user object 31 is displayed according to the user whose user object 31 has been recognized again.

Further, by touching the dialogue stop button object 34 of the chat screen S2, the dialogue is actively interrupted by the user, and the standby mode is set. In this case, the standby screen of the chat screen S2 of FIG. 8 is displayed instead of FIG. 6, but the dialog start button object 36 is displayed in place of the dialog stop button object 34. When the natural interaction mode is to be set again, it is possible to return to the chat screen S2 of FIG. 6 by touching the interaction start button object 36.
[6. Effects of robot 1's behavior during dialogue Part 5
In this way, the user object 31 of the user in the interactive relationship and the robot object 32 of the robot 1 are arranged to face each other, and the balloon objects 33 which interact with each other are lined up to chat like a feeling of interaction It can be received from the screen S2.
Also, since the past chat history can be confirmed later, the chat can be used instead of the diary. In addition, because it is possible to know who interacts with what, it is also possible to confirm data such as who is often used by family members. Since the display "user change" is displayed, it is understood that the dialogue has been interrupted once, and there is no confusion when reading the past history.

D. Special Actions in Normal Dialogue In normal dialogue, the cloud server responds to the user with a command to execute a special action when it is determined that the user's speech data contains a specific word. The controller MC executes, for example, the following specific actions based on the screen display program, the gesture program, and the dialogue program according to the command. The following control is an example, and may be controlled to be another behavior, or may be controlled to perform behavior continuously or simultaneously if there are a plurality of commands in the user's speech. The following special actions may be implemented separately or in combination. B. Above about “6. Action of robot 1 at the time of dialogue” Instead of the gesture of the robot 1 in the above, display on the following display unit may be performed, or the display on the following display unit may be appropriately combined.

1) When a negative expression is uttered by the user in a normal person-to-person conversation, the display on the touch panel 7 is made simultaneously with the utterance on the robot 1 side from the normal eye of FIG. Make the animated display change to the angry eye object of). In the present embodiment, the eye object is stored in the memory.
2) When a user utters an expression that makes the person's conversation between people more enjoyable, the display on the touch panel unit 7 simultaneously with the utterance on the robot 1 side can be seen from the normal eye of FIG. The animated object which changes in c) laughing is displayed. In the present embodiment, the eye object is stored in the memory.
3) When a user utters an expression that makes him sad in a normal conversation between people, the display on the touch panel unit 7 simultaneously with the utterance on the robot 1 side can be seen from the normal eye of FIG. d) Make the sad eye object change animation display. In the present embodiment, the eye object is stored in the memory.
4) When the user's child's name is uttered from the user, the second motor 24 is controlled so that the head 5 of the robot 1 makes a nodding gesture. In the present embodiment, the program for the eye gesture is stored in the memory.
5) When the name of the company manufacturing the robot 1 is uttered by the user, the head 4 of the robot 1 moves back and forth and right and left while the body 4 repeatedly swings with respect to the fixed part 2 The first to third motors 23 to 25 are controlled to perform such gesture operations. At the same time, the text data of the company name is subjected to voice synthesis, and the company name is continuously called from the speaker device 13 as voice. In the present embodiment, a program for gestures is stored in the memory.
[6. Effects of the robot 1 on the behavior of the robot during dialogue Part 6
By performing such special movements, the user can expect unexpected movements of the robot 1 at the same time as the dialogue, and can actively enjoy the dialogue with the robot 1.

B. About operation contents at the time of face recognition 1. Activation and Deactivation of Face Recognition Mode The controller MC recognizes and authenticates the user's face by executing a face recognition program. In the face recognition program, the acquired image is recognized as a human face by face pattern recognition, and various positions of the recognized face are quantified and stored to match the numerical data of the face registered in the past Determine the degree and perform authentication. The controller MC sets the face recognition mode in synchronization with the natural dialogue mode, and executes face recognition each time the standby mode is shifted to the natural dialogue mode.

In the face recognition mode, the controller MC performs face recognition of the user using the face recognition camera 10. In particular,
1) Activate the face recognition camera 10. The user's face image captured by the face recognition camera 10 is displayed on the touch panel unit 7 (face display mode). That is, the user is urged to look at his face on the touch panel unit 7. This enables face recognition processing, and by previewing in this way, it is possible to determine whether the person has been authenticated in the past.
2) If the face recognition camera 10 can not capture a face in 1) and face recognition can not be performed within a predetermined time, the third motor 25 is driven to swing the head 5 up and down. That is, the face recognition camera 10 is caused to scan in the vertical direction. Then, the first motor 23 is driven while the face recognition camera 10 is scanned in the vertical direction, and the face recognition operation is performed while the face recognition camera 10 is rotated 360 degrees.
3) If face recognition is possible in 1) or 2), authentication is performed. If the user is already registered, the natural interaction mode is set by using data of a specific authenticated user in the interaction. If it is a user who is not registered, it will be recognized as an unspecified person, and it will be said natural dialogue mode. The touch panel unit 7 returns from the face display mode to either the face screen S1 (FIG. 5) immediately before or the chat screen S2 (FIG. 6).
4) If face recognition can not be performed in 2), it is determined that a person can not be recognized, and the natural dialogue mode itself is ended together with the face recognition mode, and a standby mode is set. The touch panel unit 7 returns from the face display mode to either the standby screen (FIG. 7) of the face screen S1 which is the last standby mode or the standby screen (FIG. 8) of the chat screen S2.
[1. Effect in start and stop of face recognition mode
Since dialogue is based on speaking while looking at the face of the other party, the user is made to actively recognize the face by not allowing the dialogue to be made if the face can not be recognized. If you don't face it, you can't interact, so you get the feeling that the user is actually interacting.

2. About the action of the robot 1 at the time of face recognition 1) After face recognition, the controller MC causes the face recognition camera 10 to acquire an image, and always performs face pattern recognition at a fixed timing. Then, the user's face is recognized within the angle of view of the face recognition camera 10, and a state in which the center position C of the user's face is located at a predetermined position within the angle of view Position. When the central position C deviates from the default position, the controller MC displays an animation such that the pupil object 27a moves in either the left or right direction according to the amount of deviation. Normally, the entire pupil object 27a appears as an elliptical shape in the white eyes in the eye object 27 as shown in FIG. 9A, for example, but it is shown in FIG. 11 in a state where the user's face is moving Thus, the pupil object 27a is displayed as the partially hidden eye object 27 as if looking in the direction.
When the user moves and the face comes out from the angle of view of the face recognition camera 10 and the face can not be recognized, the controller MC drives the first motor 23 to move the movable portion 3 in the direction in which the central position C is shifted. It is controlled to turn the whole to direct the face recognition camera 10. The driving of the first motor 23 is stopped when the face is recognized. If face recognition is not performed even if the movable part 3 is rotated 45 degrees to some extent, for example, the controller MC stops driving the first motor 23 at that stage, and face pattern recognition is always performed in that state. To continue.
[2. Effects of Robot 1's Action on Face Recognition
As a result, the user feels as if the robot 1 is always watching and interacting, and the fun of the dialog is increased.

2) The controller MC always performs face pattern recognition with the face recognition camera 10, but face recognition can not be performed when the user is moving or is not within the angle of view of the face recognition camera 10. Therefore, it is preferable to notify the user of the face recognition state as a change on the screen. In the first embodiment, the face recognition state is notified by the change in the color density of the eyelid-like reflective object 27b expressing the reflection on the pupil inside the pupil object 27a. In the first embodiment, the controller MC displays a very pale blue color when it is not recognized, a darker color when it is in recognition, and a dark blue color when a normal face is recognized.
[2. Effects of robot 1 on face recognition in face recognition Part 2
As a result, since the user can easily recognize whether or not the robot 1 is face-recognized, the user cooperates to actively face-recognize, and a smooth dialogue progresses.

C. Regarding the operation at the time of absence setting In the first embodiment, it is possible to transfer an image to the absence setting mode, that is, an e-mail registered at the time of absence setting. In the present embodiment, the absence setting mode is set and released on the setting screen displayed after the user touches the setting button object 35 of the chat screen S2 of the robot 1. Hereinafter, a process based on the absence setting program of the controller MC when the absence setting mode is selected will be described.
The controller MC controls the B.I. in the above-mentioned “2. In the standby mode in 4), if it is determined by the Doppler sensor 22 that the object (person) is moving, the following control is performed.

1) The controller MC notifies the user of this state by e-mail when there is any moving object around the robot 1. The controller MC writes the URL of the cloud server of the robot 1 to the terminal device of a user (hereinafter, referred to as an external user) who is not near the robot 1 whose e-mail address is registered through the Internet. And send. In the subject line of the e-mail or in the transmitted text, write an expression that shows the intention of this notification. For example, a sentence such as "It seems that someone is coming" or an icon that means that.
"C. The effect on the operation when there is no answer setting
As a result, the external user can be notified of the state in which an object (person) is moving around the robot 1 and recognized as a result of the e-mail being sent. You will be given the opportunity to

2) The controller MC sends an e-mail to an external user and at the same time activates the face recognition camera 10 to acquire an image.
3) The controller MC sends an e-mail to the external user, and at the same time determines whether there is a trigger utterance that is within a predetermined time. For example, it is an utterance of a greeting like "I'm happy now". Based on this utterance, a built-in scenario in the natural dialogue mode is started to urge the user (here, a person near the robot 1 who has uttered "I'm happy now") face recognition. If the controller MC determines that it is one of the registered users as a result of face authentication, it sends a second e-mail to the terminal device of the external user. This e-mail is information for an external user that the person around the robot 1 is not a suspicious person. That is, the second e-mail informs that it is a related person, such as a family. In the e-mail, the name of the registered user based on the registration information is described in the subject line and the transmission text as information. Note that, as a trigger, for example, the touch panel unit 7 may be touched to face-recognize other than the utterance, and the user may be confirmed as a registered user.
"C. About the operation at the time of absence setting Effect in part 2
As a result, when, for example, a family such as a child comes home while he is away, it is not necessary to check the appearance of the away home via the smart phone because the second e-mail indicates that fact.

4) In 1), the external user who received the e-mail inputs the ID and password of the robot 1 and connects to the cloud server, especially when there is no second e-mail transmission in 3). A camera image of the face recognition camera 10 provided by the cloud server can be viewed in real time on the browser. It is possible if there is a second e-mail transmission.
FIG. 12 shows an example of the smartphone 41 of the user. After connection to the cloud server, a predetermined camera image of the face recognition camera 10 is displayed on the display screen 43 which also serves as a touch panel. In the camera image, four operation icons 44a to 44d for remotely controlling the direction of the face recognition camera 10 are displayed. The external user operates the operation icons 44a to 44d to output a control command to the controller MC via the cloud server, and the first motor 23 or the third motor 25 is driven and controlled based on the control command, and the robot 1 is operated. The head 5 and the torso 4 can be rotated to change the direction of the face recognition camera 10. Moreover, by touching the recording button icon 45, recording can be stopped by starting recording and touching again.
Further, voice data uttered to a microphone (not shown) of the smartphone 41 is voice-outputted from the speaker device 13 of the robot 1 via the cloud server, while voice data uttered from the microphone 12 of the robot 1 is smartphone 41 via the cloud server The sound is output from a speaker device (not shown). Therefore, the external user can interact with the user in the vicinity of the robot 1 and the smartphone 41 using the robot 1 while viewing the image of the face recognition camera 10.
"C. About the operation at the time of absence setting Effect in 3
As a result, the external user can remotely change the direction of the face recognition camera 10 to check the situation around the robot 1 and, for example, can check the safety situation of the home at the time of absence. In addition, even if a family such as a child comes home while away from home, using a smartphone to actively communicate in this way contributes to a good relationship with others including the family.

<Modification 1 of Embodiment 1>
Next, a first modification of the first embodiment will be described.
In the built-in scenario dialogue in the above natural dialogue, in the case where the user's bad tongue is mixed or other sounds are mixed and the voice data acquired from the microphone 12 does not match the regular expression or non-regular expression of the built-in scenario, The MC may immediately output a voice such as "please say again" as an utterance for prompting the user to re-speak without immediately determining that the dialog is a normal dialogue.
The controller MC causes such a prompting utterance to be issued from the speaker device 13 and, if the user makes a correct utterance according to the built-in scenario within a certain time, it is treated as a built-in scenario dialogue again. Make it On the other hand, even in such a case, the built-in scenario is connected to an external cloud server when the built-in scenario does not match the regular expression or the non-regular expression.
In this way, communication can be performed only inside the robot 1 without being unnecessarily connected to an external cloud server.

<Modification 2 of Embodiment 1>
Next, a second modification of the first embodiment will be described.
When the user misses the word (speech) of the robot 1 in the above natural dialogue, the user utters a speech of a request to repeat the immediately preceding speech of the robot 1 within a predetermined time, so that the robot 1 is given a second time. You may make it speak (voice output). This process can be either built-in scenario dialogue or natural dialogue.
The controller MC performs speech recognition as to whether there is an utterance that triggers a missed speech during the interactive mode, for example, an utterance of "say again". Then, if it is determined that there is a request from the user after the utterance from the robot 1 to repeat the utterance, the content uttered immediately before by the robot 1 is voice-outputted again. Then, the previous utterance is canceled, and the second utterance is processed as the first utterance.
In this way, even if the user misses in the middle of the dialogue, the dialogue is resumed without interruption.

<Modification 3 of Embodiment 1>
Next, a third modification of the first embodiment will be described.
If the user confirms the speech of the user displayed on the touch panel 7 by the robot 1 in the above natural dialogue and if it is found that voice recognition has been made erroneously, it is pointed out within a fixed time. It may be possible to correct to the correct dialogue. This process can be either built-in scenario dialogue or natural dialogue.
The controller MC, while in the interactive mode, makes an utterance that causes the user to indicate that voice recognition is incorrect, such as an utterance such as "I'm wrong" or "No, I'll say it again." Speech recognition whether there was. Then, when controller MC determines that the utterance is within a predetermined time after the user's utterance is displayed on touch panel unit 7, controller MC outputs a voice prompting the user's utterance again. For example, the utterance content "I'm sorry. Please say again."
And
(1) In the case of the dialogue of the built-in scenario, the immediately preceding user's utterance is canceled, and the content of the user's utterance is voice-recognized again as the correct utterance.
(2) In the normal dialogue, the dialogue contents such as "I'm wrong" or "I say again" are also transmitted as speech data (voice data) to an external cloud server, and the user utters again including such speech Request to create response data by content.
In this way, even if the robot 1 mistakenly recognizes the user's speech, it can be corrected to the correct interaction.

<Modification 4 of Embodiment 1>
For example, the following configuration, which is different from the configuration of the first embodiment, may be adopted.
(1) In the above, when the built-in scenario is not executed, the cloud server is requested to shift to normal interaction. In other words, if built-in scenarios are to be executed, they are all configured as built-in scenarios, but they are requested to request to the cloud server without responding locally under conditions that may correspond to built-in scenarios. May be A certain condition may be executed, for example, once in several times or at random timing.
This makes it possible to interact with the robot 1 unpredictably, and to provide an unconventional and more human interaction.
(2) In the first embodiment, the controller MC is not provided with the speech recognition engine, and is connected to an external server provided with the speech recognition engine to convert the user's speech (speech data) into text. The burden on the robot 1 is thereby reduced.
However, the controller MC has a speech recognition engine in the memory, and calls the speech recognition engine to text the user's speech data (speech data) acquired from the microphone 12 into text by itself using the speech recognition engine. Column data may be created. That is, the controller MC of the robot 1 may have the ability to convert voice data of the user into text data. As a result, character string data can be created without using a speech recognition engine, and the internal processing time can be shortened, for example.
(3) In the first embodiment, initial setting is performed from the setting screen. However, for example, a terminal device such as a smartphone may be used to register from the outside via the cloud server. This makes setting easier and saves time, especially for people who are used to terminal devices.
(4) The first to third motors 23 to 25 may use other drive means than the servomotor. Other drive means are, for example, other types of motors, hydraulic cylinders, and the like.
(5) In the first embodiment, the character string data is speech-synthesized inside the robot 1. In this way, using the internal speech synthesis engine is better without making the data too heavy than exchanging the speech data with the server as it is, but the response data (character string data) acquired using the dialogue engine on the cloud server side May be synthesized, and the speech data may be sent to the robot 1 as a response.
(6) In the first embodiment, the configuration is such that the setting button object 35 of the chat screen S2 shifts to the setting screen, but the setting screen may be shifted by sliding the touch panel unit 7 .

(7) In "Ha. Special Actions in Normal Dialogue", even in the built-in scenario, even if it is determined that the user's speech data contains a specific word, the special actions should be executed. Good. For example, if it is determined that the user's speech data contains a specific word, the controller MC may control to cause a special action as described above.
(8) In “C. Special action in normal dialogue”, control may be performed to cause different gestures if the shape of the robot 1 is different. For example, if there is a hand or foot on the robot 1, the controller MC may control the drive means to move them.
(9) As the animation of the eye object 27 of the face screen S1, an animation which blinks occasionally may be inserted. For example, from the state of the eye object 27 of FIG. 9A, the control is performed such that the eye object 27 of the closed state of FIG. 7 is inserted. By doing so, it is possible to create a sense of realism that the robot 1 is really looking at here, and to enjoy the dialogue with the robot 1 more.
(10) In the face recognition mode, an unregistered user is recognized as an unspecified person, but when the state is shifted from the state to a setting screen and authentication is registered as another new user It is convenient.
(11) It is convenient to be able to set the absence setting mode from the smartphone in “C.
(12) In “C. Operation at the time of absence setting”, the controller MC performs processing such as notifying the user by e-mail that there is some moving object around the robot 1 by e-mail. On the contrary, it may be recognized that it is moving at a constant interval, and a process may be provided to notify the user by e-mail when there is no moving object within a predetermined time.
For example, when there is a sick person or a care recipient, by placing the robot 1 near it, it can be watched on the assumption that there is always a movement.

Second Embodiment
Next, the second embodiment will be described.
Instead of the high-intensity white LED 9 of the robot 1 according to the first embodiment, an infrared LED may be provided. At this time, if the module of the face recognition camera 10 is equipped with an infrared filter, it is removed. Since the infrared LED is invisible to human beings, when there is an intruder such as a void at night, the high brightness white LED 9 may be surprised to be escaped by the intruder. On the other hand, since it is hard to understand that the infrared LED is photographed, the intruder does not run away, so that it is possible to confirm or save the intruding image.

<Modification 1 of Embodiment 2>
Next, a first modification of the second embodiment will be described.
When the robot 1 includes an infrared LED, the infrared LED may be used to control various devices in the room provided with the infrared remote control signal reception unit. As various devices, for example, a television, an audio device, an air conditioner, etc. are preferable.
The robot 1 may be installed at a place where the infrared remote control signal receiver of the apparatus provided with the infrared remote control signal receiver can be directly seen. In the first modification of the second embodiment, the controller MC includes a shape recognition program related to shape recognition using the face recognition camera 10. For example, in the case of a television, the controller MC is based on the characteristics (square, large, black, etc.) of the shape. Can be recognized. When the robot 1 has an utterance such as "Turn on the television" as a trigger for outputting an infrared remote control signal to various devices of the user, the first motor 23 or the third motor is based on the utterance. The control unit 25 controls the robot 1 to rotate the face recognition camera 10 up and down while swinging the face 5 360 degrees to photograph the surroundings and operate the shape recognition program to recognize the shape of the television.
When the controller MC determines that there is a television, it stores the direction and simultaneously causes the infrared LED to output an infrared remote control signal in the direction of the object to execute control of ON / OFF etc. which operates the television. When there is a trigger for the next television, first, shape recognition is performed in that direction. The infrared remote control signal can be controlled not only by ON / OFF switching control, but also by changing the infrared frequency so as to cope with temperature change etc. Become. Such fine control requires different types of infrared remote control signals of different types of frequencies. For example, in the case of FIG. 12, the setting of the infrared frequency may be performed via the server using the smartphone 41 of the user.
Further, without searching for the direction by the shape recognition program, the direction of the various devices may be acquired by operating the face recognition camera 10 via the smartphone 41 to change the direction, and the direction may be registered.

Embodiment 3
Next, the third embodiment will be described.
In the third embodiment, control will be described focusing on reducing running costs associated with connection such as use of the dialog API when using a server equipped with a speech recognition engine.
Further, the dialogue program of the third embodiment includes a sub program capable of detecting a silent state of the voice acquired from the microphone 12. Further, the dialogue program includes a recording / output subprogram for acquiring speech data of the user's speech from the microphone 12 and temporarily recording the speech data for output to the server.
If there is an utterance, the controller MC does not immediately connect to the server but first records the speech data of the user's speech once, and connects to the server only when the silence state of the recorded speech data of the user is detected. The recorded voice data is output, and the dialogue engine is made to create response data. In this way, it is not always connected to the server, and since it is not necessary to connect to the server for a long time including silence time, silent connection time can be cut.
The speech recognition engine has one process dedicated to the user during the user's speech. In other words, a very long time is occupied as "a few seconds" for one process in one process, and as a result, the burden on the cost of the user who uses the speech recognition engine is increased. In this way, the process required per unit user can be reduced, which contributes to the cost reduction of the user.

<Modification 1 of Embodiment 3>
Next, a first modification of the third embodiment will be described.
Also in the first modification of the third embodiment, control will be described focusing on reducing the running cost of connection when using a server equipped with a voice recognition engine. The speech recognition server consumes the process if there is a time lag before the user's speech starts, or if the user's speech is waiting for the user to end the connection with the server after a timeout without a speech. It will cost you a lot of money.
The dialogue program of the robot 1 according to the first modification of the third embodiment includes a recording / output subprogram for acquiring speech data of speech from the microphone 12 and temporarily recording the speech data, and causing the server to output the speech data. Further, the dialogue program includes a sub program which detects and outputs the sound pressure level of the recorded voice data.
The controller MC is connected to the server when voice data being constantly recorded in a state of natural interaction is equal to or more than a predetermined sound pressure, and causes the data being recorded to be chased and reproduced. In other words, ignore small utterances that can not be silenced or can not be recognized, connect to the server only when there is an utterance that can be interacted, and output voice data, and let the server create response data with the voice recognition engine Make it
This makes it possible to eliminate useless connection time waiting for speech.

Fourth Preferred Embodiment
Next, the fourth embodiment will be described.
The voice recognition server is expensive, so it may not be possible to prepare sufficient resources beforehand, and if the server resource is not enough, the server is busy when trying to connect to the voice recognition server There is.
In the normal interaction, the server transmits speech data (speech data) to request creation of response data. And the server creates response data converted into character string data based on the speech data and responds. However, if it is busy, response data may not be sent and an error may occur. An error message will be sent back from the server.
When the controller MC of the robot 1 receives an error message from the server, the built-in scenario outputs a reply that allows the user to continue the dialogue, without giving the user an utterance indicating that the server connection error. For example, it is good to make an ambiguous response such as "Say again", "Uneun", or "What is it?", Or return an appropriate sumo wrestling to wait for the server to become available.

<Modification 1 of Embodiment 4>
Next, a first modification of the fourth embodiment will be described.
If the user speaks too long, the speech recognition engine may misinterpret. Therefore, when it is determined that the speech of the recognized user is too long, a function of outputting an example of an interaction selected from the speech data stored in the storage unit without connecting to a server provided with a speech recognition engine is provided. Good.
If the controller MC of the robot 1 determines that the user's speech data has become longer than a certain length, it is not connected to the server, and is "true" or "true" or "true" from the built-in scenario? And so on, such as an utterance that can be taken in any way in the dialogue such as. The speech data may be voice data as it is or may be converted to character string data by the controller MC or server.
This can prevent the return of off-target words, restructure the dialogue, and prompt the user to interact again.

The Fifth Preferred Embodiment
Next, the fifth embodiment will be described.
In the fifth embodiment, a case where a plurality of speech recognition engines are combined and used will be described.
The speech recognition engine includes a local speech recognition engine (that is, when processing in the device without connecting to a net server) and a cloud speech recognition engine that connects to a net server and responds to interaction data created by a request. is there. There are multiple voice recognition engines, both locally and in the cloud, and some are free and paid. Therefore, when using servers with these different speech recognition engines, a mix of free servers and paid servers will be used.
In the cloud server to which the robot 1 is connected using the Internet, in the interactive mode, when the speech data is issued from the robot 1 and the speech recognition engine generates response data, for example, the cloud server responds as follows: It is good to do.
(1) Access the paid interaction API until a certain time A set per month.
(2) From a certain time A set up per month to a certain time B, mixed use of paid API and free speech recognition engine. For example, the first few consecutive recognitions use a paid speech recognition server, and the subsequent consecutive recognitions use a mixed speech, such as a free speech recognition server.
(3) When a certain time B set per month is exceeded, only the free speech recognition engine is used. This is an example, and for example, it may be set to use only the free speech recognition engine immediately when the set time A per month is exceeded.
In this way, it is possible to interact without significantly exceeding the pay range. The cloud server connected to the robot 1 calculates the charge and free based on the time set per month based on the program that executes such processing, and processes the request issuance from the robot 1.
The robot 1 itself performs such processing and instructs the cloud server to use a paid interaction API or use a free server speech recognition engine when issuing a request. It is also good.
Embodiment 5-1
When combining and using a plurality of speech recognition engines in the fifth embodiment, the same applies to the case of combining a plurality of dialogue engines.

Embodiment 6
A sixth embodiment will now be described.
By using only one fixed dialogue engine, the response may become a well-defined pattern and the user may get bored with the conversation with the robot 1. Therefore, in the sixth embodiment, in order to solve this, processing for arranging the result so as not to get tired of conversation using a result of outputs of a plurality of dialogue engines will be described.
The dialogue engine may use not only the dialogue engine of the cloud but also the local dialogue engine in the robot 1.
This process may be performed on the side of the robot 1 to which a plurality of response data has been sent, or may be performed on the cloud server side when response data from a number of servers equipped with a dialogue engine is acquired.
(1) Output the result of the engine which has sent the longest reply of the number of characters of the character string among the chat dialog engines.
With the longest response, it feels like you are interacting, the monotony of the interaction is eliminated, and the user can enjoy the interaction.
A. For example, when the user utters "I'm hungry", the answer from the three engines of a to c is "A engine: Do you eat well?""B engine: Do not eat rice?"" c) Engine: If it is "something", adopt engine a and output its response data.
B. For example, when the user utters that "the weather is fine today", the answers from the three engines of ac are "a engine: I will go to the sky right now and check""b engine: clear weather If it is "c engine: whether it is sunny or rainy, the mood of the day may be determined", the c engine is adopted and the response data is output.

(2) Of the chat dialogue engine, the one with "?" Attached to the end is taken last and output. At this time, if there are multiple answers with "?", They are output continuously.
A question mark at the end of the word makes the conversation more responsive to the question, making it easier for the user to continue the conversation.
For example, the above (1) A. In the option, output "Do you eat snacks often? Are you not eating rice?" In addition, the above (1) B.1. If it is an option, output "Are it clear?"
(3) After combining positive sentences, combine and output question sentences.
Because arranging in this way makes the response look like thinking and writing sentences, the user feels like they are seriously listening to their own speech, and they want to continue talking. The conversation is easy to continue and the user can enjoy the dialogue. In addition, it is possible to gain an output scale and to request a response to people.
For example, the above (1) B.1. If response data such as is acquired, "I will go to the sky right now and check it. It may be decided whether it is sunny or rainy, the mood of the day. Is it sunny?"

(4) The results from engines that make topic conversion more frequent than others are brought in and output later than the results of other engines.
By arranging in this way, it becomes a dialogue that invites the next utterance by having a topic change, and the dialogue becomes easy to continue.
A. For example, when the user utters "Domo Domo", the answers from the three engines a to c are "a engine: yes"-"b engine: yes""c engine: baseball is watched If it is, c engine data will be output last and output as “Don't you think so? Do you watch baseball?”.
B. For example, when the user utters "I can not find it easily", the answers from the three engines a to c are "a engine: yes," b engine: there is "c engine: family In the case of “How many people are you?”, The data of c engine is output at the end as “Yes, there is one – how many families are you?”.
(5) The engine that makes a more friendly response than others is output first, and then the responses from other engines are attached and output.
This arrangement makes it easy for the user to be drawn into the dialogue and the dialogue to be continued. Whether it is friendly or not can be determined in which position it should be placed by ordering relative to the word (word).
A. For example, in (4) B. In the case of b, the result of the engine is first, and the output is "Are there-it is true. How many families are there?"
B. For example, when the user utters "I see," the answer from the two engines a and b is "a engine: I think it's a proper sumo wrestling" and "b engine: umu". At the end of the data, the output is "Hohoho, you're right."

(6) Arbitrarily combining the processes of (1) to (5) Since this will increase the variation of the dialogue, the same response will not be returned even when the user makes the same utterance, and the dialogue It is easy to continue the dialogue without getting tired.
For example, when the user utters "What is old age", the answers from the three engines a to c are "a engine: please wait for a moment""b engine: I do not hate support""c engine : If you are healthy now? "If you are the most friendly b engine first, then combine the positive statements, then combine the questions," I don't hate support. Do you come? "

(7) In the engine for text output, parentheses and emoticons may be returned, so when these are returned, the voice output is suppressed and output. And they are displayed on the screen.
Although emoticons can not be output as voices, by displaying emoticons on the display unit, it is possible to create unusual fun of dialogue by combining them with speech as part of the dialogue.
A. For example, if the user as "hungry" is uttered, the answer is "a engine :( deliberately ignored)" from the three engines of a~c "b engine: Hello Take care I am", "c engine: Hello In the case of “a”, only the a engine is not output as voice, and is displayed on the touch panel unit 7 (display screen).
B. For example, when the user utters "I'm originally in", the answer from the three engines a to c is "A engine: Do you do well?""B engine: ('· ω ·`) "" c Engine: Is it a night type? ”, only b does not output voice and is displayed on the touch panel 7 (display screen).
(8) Output any one of the responses including the same character string.
If the same string is repeated, the dialogue will be disturbed, and the listener may feel uncomfortable.
For example, when the user utters “Chinese food”, the answer from the three engines a to c is “a engine: good sounding” “b engine: yes, Chinese food” “c engine: eating Chinese food If you go to "?", Because there is a string of "Chinese" in b engine and c engine, only one of them is output. For example, output like "Are you going to eat Chinese?"

(9) Use an end-of-speech conversion means, such as an end-of-speech conversion API, to give a sense of unity. At this time, although the end conversion may be performed on all the responses, the end conversion may be performed on only one of the sentence output last and the sentence output first.
It should be a more familiar expression than the text of a normal dialogue engine.
For example, when there is a response data "Nemuina" from a certain engine, the word ending is converted by the vocabulary conversion API and output as "Nemuina".
(10) In preparation for recognition failure, only a part of the responses obtained from multiple engines is used for voice output, and the remaining responses are retained, and when the next voice recognition fails or from the dialogue system If there is no response, return the stored response.
The dialog can be connected without interruption by using it when speech recognition fails or when the response from the external server does not come easily, it contributes to the natural dialogue

Seventh Embodiment
As shown in FIG. 14, the seventh embodiment is an apparatus (system) in which the smart speaker 51 is disposed in the vicinity of the robot 1 and the robot 1 and the smart speaker 51 are combined. The distance between the robot 1 and the smart speaker 51 is a distance that allows the microphones to pick up the sound, and may be adjacent to each other, for example, within 1 to 2 m.
The smart speaker 51 is a network terminal having a wireless LAN device, a wireless communication function using the Internet, a telephone line connection function and the like and a network module mounted thereon, and is also a kind of computer provided with a microphone and a speaker device. . The smart speaker 51 performs various initial registrations (for example, user's name, address, telephone number, mail address registration, plural voice registrations, network-compatible AI by Bluetooth) via a server using a terminal device such as a smartphone. Set up the device, etc.), connect to the Internet by the voice (instruction) from the user who registered the voice, execute the predetermined processing using the search engine of the server, and use the result as the voice information from the speaker device Output.

The robots 1 can compensate each other's functions by cooperating with the smart speakers 51. Specifically, the following function is exerted using the robot 1 and the smart speaker 51 as an audio interface.
(1) Instruction function from the robot 1 to the smart speaker 51 a. For example, when the user sings a phrase that activates the smart speaker skill, the robot stores a character string of the speech recognition result of the phrase, and the robot sings the character string by voice synthesis at a predetermined timing by itself. Make it
In the seventh embodiment, the controller MC of the robot 1 is a voice identification program that analyzes user's speech by analyzing frequency components and identifies an individual's voice, and acquires it by the phone 12 when the user's speech is distinguished for each individual and dances A phrase recording / reproducing program is provided which is stored as string data, is speech synthesized based on the character string data, and is reproduced from the speaker device 13.
The robot 1 has a function of storing, for example, an utterance serving as a trigger for storing an utterance, for example, a word after the utterance “I speak from now, then record”. Then, the user uses this function to make the smart speaker 51 store words that activate or perform some processing. For example, words such as “OK, ××× with illumination.” Are good. At this time, the voice of the user personal registered in the robot 1 is the voice of the user personal registered in the smart speaker 51.
Then, the robot 1 is made to speak at a predetermined timing. For example, the setting for uttering at a predetermined timing can be registered by operating a terminal such as a smartphone.
B. The “utterance at a predetermined timing” of the robot 1 is, for example, that the robot 1 detects any change, for example, a touch operation on the touch panel unit 7, detection of an object (person) by the Doppler sensor 22, or the like.
For example, when the controller MC of the robot 1 detects a person by the Doppler sensor 22, the controller MC causes the speaker device 13 to output an audio as "OK, xx. Turn on the light." In response to this, the smart speaker 51 is controlled to turn on the room lighting which is the AI device compatible with the network. The illumination to be controlled is registered in advance as an object to be controlled by the smart speaker 51. Besides the lighting, for example, an air conditioner, a television, a curtain opening and closing device, etc. may be used as the AI device.

(2) A function to distinguish whether the user speaks from the smart speaker 51 or the user's voice By identifying the user's voice and setting and registering the robot 1, whether the robot 1 speaks from the smart speaker 51 or the user's voice It can be equipped with a function to distinguish whether it is a direct utterance. As a result, the robot 1 can be controlled not to respond to the sound from the smart speaker 51 which is a voice not registered, and conversely, the smart speaker 51 and the robot 1 each have individual user's By registering the voice, it is possible to instruct the smart speaker 51 not only from the user but also from the robot 1. The ability to distinguish individual utterances in this manner also has a function of not disturbing the voice operation on the smart speaker 51.
(3) Display Function of Utterance from Smart Speaker 51 The robot 1 may acquire the utterance content instructed by the user to the smart speaker 51 and convert it into text and display it on the touch panel unit 7.
The controller MC of the robot 1 may have a program for acquiring speech content and connecting to the server itself or to a server to convert it into text.
For example, the user utters "OK, ×××. Tell me the weather of today." In contrast, the smart speaker 51 says, "The weather in Okazaki, Aichi Prefecture today is fine, the maximum temperature is 15 degrees, the rainfall probability Is 20%, "for example, the touch panel unit 7 of the robot 1 displays all these dialogs, for example, the dialogs of both heard as follows.
"User: OK, xx. Tell me the weather today.
Smart Speaker: Today's weather in Okazaki, Aichi Prefecture is fine, the maximum temperature is 15 degrees, the probability of precipitation is 20%. "
In addition, it is preferable that the controller MC of the robot 1 have a program for short adaptation and summarization of the text contents. The robot 1 may be configured to cause the contents adapted or summarized to be displayed as voice output or on the touch panel unit 7.
(4) Function that robot 1 listens to smart speaker 51 in various ways and learns when there are no people It has a word, and makes the controller MC of the robot 1 voice output the question word together with the phrase for activating the smart speaker 51 at a predetermined timing when the user is away (the voice of the robot 1 is the smart speaker 51 Registered with). At this time, the controller MC causes the robot 1 to acquire and store the utterance content from the smart speaker 51 by the microphone 12 and causes the content to be voice-outputted at a predetermined timing. The predetermined timing is, for example, when the user has detected a person by the Doppler sensor 22 for a predetermined time, the user utters the robot 1 as a built-in scenario such as "Is there any news?" .

<Modification 1 of Embodiment 7>
A function that does not disturb the voice operation of another voice recognition device such as a smart speaker may be set. For example, when the robot 1 recognizes the voice of the activation phrase (voice recognition start word) of another smart speaker, the voice output of the robot itself may be stopped.
For example, when the voice of the activation phrase such as “OK, xxx” which is the startup phrase of company A, which is another smart speaker, is recognized, the controller MC of the robot 1 acquires it from the microphone 12 If it is determined that the activation phrase has been registered, the voice output of its own is temporarily suspended.
This allows the function to be exhibited without disturbing the voice operation of the voice recognition device.

<Modification 2 of Embodiment 7>
The robot 1 recognizes the speech recognition activation keyword of another speech recognition device such as the smart speaker 51, recognizes the subsequent pronunciation on the smart speaker 51 of the user, makes a request to the cloud server, and searches the search engine etc. Let me do it and get the corresponding answer.
The robot 1 recognizes voice output (for example, "Sorry") that the smart speaker 51 fails in voice recognition (for example, "error") or can not output an appropriate answer to the voice recognition result. In this case, the robot 1 outputs an answer it has obtained in advance. Alternatively, the robot 1 may request the cloud server to obtain an answer after receiving an audio output indicating that the speech recognition has failed or an appropriate answer to the speech recognition result can not be output.

Embodiment 8
The robot 1 may read the news articles on the web site by voice, for example, at the request of the user. The controller MC of the robot 1 requests a news article using a search engine on the server, and the cloud server responds to the request, for example, news data of a registered site as text data.
The robot 1 synthesizes speech data of the news data and reads it out (outputs it), and at the same time, it synthesizes the name of the information source of the article and reads it out (outputs it). At the same time, the URL of the information source of the article is displayed on the touch panel 7 as a display screen, and when the URL is touched on the touch panel 7, the contents of the page of the URL are displayed on the touch panel 7. Let's do it.
In addition, when reading out an article or an information source of an article, it is preferable to output in an expression that indicates that the article is a citation.
For example, the case where the article content of URL "https: // \\\\ .jp / archives / 92 ###" is read out will be described.
"I'll read the article on the" XX News "site. "We will sell ... from January 15, this year.""
For example, let's utter it so that the listener can see it as a regular expression, citing something other than an article or article information source such as "Read the article of the site's site" or "Yes". So, for example, 'https: // \\\\ .jp / archives / 92 ### information on the screen display. And so on, to display the URL. Then, touching the URL portion causes the touch panel unit 7 to display again the contents of the read article.
"Speech so as to be understood by the listener" may be, for example, changing the tone or voice between the article part and the other part.
In this way, even if the Web article is not read, the news contents can be understood simply by listening to the contents read by the robot 1 and in some cases, the news contents can be checked visually just in case.

<Embodiment 9>
In the ninth embodiment, the case of showing the invisible motion of the computer by the motion of the actuator of the robot will be described.
(1) Robotics Process Automation Robotics Process Automation (hereinafter referred to as RPA) is software that automates a simple personal computer task. The software can be configured on the server or on the user's computer. An example of the case where the software is set in the server based on FIG. 15 and the robot 1 of each of the above embodiments is incorporated in the network of RPA will be described.
As shown in FIG. 15, a cloud server 55 and user computers 56, 57 are connected by a network using the Internet. The cloud server 55 and the robot 1 are also connected by a network. The user side computer 56 is controlled by the cloud server 55 by the RPA program. Further, the robot 1 is notified of processing information such as processing being executed, executed, or executed in a predetermined process in a program for RPA executed by the cloud server 55 soon. It has become.
The cloud server 55 causes the user-side computer 56 to process processes 1 to 4 in order. In the present embodiment, processes 1 and 2 are executed by the computer 56, and processes 3 and 4 are executed by the computer 57. More processes may be set, and one or more user computers 56 may be involved in the process.
Process 1 includes, for example, access / login of the user to the computer 56, and process 2 includes, for example, creation / sorting of a list based on data in the computer 56. Process 3 is, for example, a customer executed following process 2 The modification of the contents of each bill, process 4 is, for example, issuance of a bill to be executed following process 3. In the ninth embodiment, for example, in process 3, the user is prompted to input for correction, and after the input is made, the process proceeds to the next process 4. That is, after the process 2, it temporarily becomes a standby mode until the power in the process 3 is completed.
The cloud server 55 outputs different notification information to the robot 1 immediately before, during, and after these processing, and the robot 1 notifies the processing status around the robot 1 based on the notification information. Good. Alternatively, one notification may be made for each process.

For example,
a. The robot 1 reports which process is in what state, by the difference in sound or sound, or music.
b. Report on the display screen. a. You may go at the same time.
c. In the process 3, since the user's input is required, only the process 3 may be notified, and only the process 3 may be an alert (differentiable) different from other informing.
d. The state of processing is transferred from the robot 1 to another terminal device and notified.
e. The robot 1 operates so as to know the processing status. For example, if the processing of the computer 56 is being performed, it is controlled to turn in that direction. Therefore, the direction of each computer 56, 57 relative to the robot 1 may be recognized in advance using some direction specifying means, for example, the shape recognition program described above. For example, the robot 1 may be provided with a direction indicating member such as an arrow or an arm member, and the computer 56 or 57 as a notification target may be operated in the pointing direction.

(2) Blockchain The blockchain is a mechanism in which a large number of computers distribute and record. In particular, in a public type block chain, data to be recorded and recorded data are released.
Therefore, the robot 1 is placed in the block chain network so that the robot 1 notifies the user before the block data is transmitted. The transmission may be stopped by outputting an instruction “wait” to the robot 1 (that is, a request for waiting without transmitting data).

<Embodiment 10>
Although the natural dialogue mode has been described in each embodiment, the foreign language learning mode may be provided instead of the natural dialogue mode or in addition to the natural dialogue mode. When a foreign language learning mode is provided in addition to the natural dialogue mode, for example, it is preferable to switch from the natural dialogue mode to the foreign language learning mode when a voice "switch to foreign language learning mode" is recognized in the natural dialogue mode. In addition, it is preferable to switch from the foreign language learning mode to the natural dialogue mode when the voice "switch to the natural dialogue mode" is recognized in the "foreign language learning mode".
It is often said that it is better to make foreign friends to master foreign languages, but not many people have such an opportunity. Therefore, it is preferable to have a foreign language learning function which is a dialogue system that can be a partner in foreign language learning.
It can be a combination of any first language and second language. In the following, the configuration will be described in which a Japanese dialog system and an English dialog system are linked.
Basically, it is a system that speaks in English, and it is preferable to output a request such as "Speak again" in Japanese during the conversation. Furthermore, in cooperation with the English analysis Web service, etc., it has a function to explain English in conversation in Japanese in response to a request such as "explain." If you don't know what to say in English when speaking in English, when you ask for "translate", it will output what it is in English. You can continue the conversation if you read the English output. Since you do not have to check it yourself, you can expect smooth English language learning without interruption in conversation.
Both the native language (for example, Japanese if Japanese) speech recognition engine and the foreign language (for example English) speech recognition engine in the foreign language learning mode use a speech recognition engine operating on the cloud. However, for native language (for example, Japanese), since the request content is a fixed phrase and the format of the response to the request is determined, a speech recognition engine is provided locally (for example, in the robot 1). It is good to use. The dialog engine is similar. The speech synthesis engine may be provided anywhere, but in particular locally.
When speech data based on the signal from the microphone 12 is thrown to the speech recognition engine of both languages, some result is returned from both speech recognition engines. For example, if you throw the Japanese saying "Tell me once more" to both engines, the Japanese engine will return the text data "Tell me once more" and the English engine will say "Tell me once more" It returns unwanted text data interpreted as English. In such a case, since the Japanese request is a fixed phrase, the text data returned by the Japanese engine and the fixed phrase of the request are compared, and if they match, it is determined that the Japanese request has been received. If you do not do so, it is better to separate English and Japanese by judging that English is spoken.
Although the request may be accepted in English, the inventor has found that it is desirable to be able to request in Japanese, especially in the case where the request is not understood in English.
The function to explain English text in conversation in Japanese in response to a request such as “explain” may be as simple as outputting Japanese translation of English text, but it is especially possible to use phrases and grammars used in English text. You should output the commentary of. In particular, it is good to output the explanation about the syntax used in the English language. For explanation of the syntax, for example, it is preferable to draw each phrase as a vertex (node), for example, to draw each phrase, and to draw a branch (edge) such as a line segment indicating the relationship of each related phrase. For example, it may be displayed on the touch panel unit 7 in the form of a graph structure (in particular, a tree structure).
Further, it is preferable to output the displayed content from the speaker device 13 by voice. For example, it is assumed that the parsing service API described in https://gigazine.net/news/20160602-foxtype-review/ is called, the result is received, and the analysis result is output in Japanese. Good.
For example, the following processing and output are performed.
"Process Phrase from English dialogue engine Robot I'm a fantastic robot.
People "Say it one more time"
Robot I'm a fantastic robot.
Person "explain"
Process (recognize “describe”) → Parsing API call → Generate Japanese commentary from syntax analysis result Robot I is the subject, am is the verb, robot is the object.
Fantastic is an adjective that means to modify the robot.
English text means "I am a wonderful robot".
Person "I don't think so."
Robot Don't say it!
If you don't know what you want to talk in English, you can request that you teach me how to speak in English in Japanese, so it has the excellent effect that the conversation is uninterrupted. If you can not make a request in Japanese, and you do not know how to speak in English, you need to look in a dictionary or translate it on the net, which makes you feel that you are studying and feel stressed. If you can request in Japanese, you can learn without stress as if you are just talking with bilingual. Since it is very important to continue language learning, it is extremely important to keep stress as low as possible. According to this configuration, it is possible to realize the robot 1 capable of performing language learning continuously.

Embodiment 11
In addition to the functions described in each of the embodiments, it is preferable to provide a function of speaking when it is detected that a person has entered the room in which the robot 1 is installed. In addition, it is preferable to provide a function to speak when it is detected that a person leaves the room in which the robot 1 is installed.
For example, if a sensor is provided in the passage of the room and a place other than the room and it is detected that a person has entered into the room in which the robot 1 is installed and a person is out of the room in which the robot 1 is installed. Good. In particular, when there is an automatic door for entering and exiting the room in which the robot 1 is installed, the sensor may be a sensor that detects that a person is approaching the automatic door, particularly for opening and closing the automatic door. In particular, the first person detection sensor outside the automatic door, the second person detection sensor indoors through the automatic door, and the person at the automatic door when the automatic door is open The controller MC of the robot 1 may be connected to at least two of the third human detection sensors (sensors for preventing a person from being pinched) to detect movement in and out of the room. In this way, it is possible to detect entering and leaving the room where the robot 1 is installed without installing a new sensor. For example, it is preferable to capture and detect an edge of a signal that rises when a person of each sensor is detected.
For example, when a person is detected by the first person detection sensor and then a person is detected by the third person detection sensor, a voice of a phrase that welcomes a person who has come in, such as “Welcome me,” can be used as the speaker device 13 It is good to output from. For example, after a person is detected by the second person detection sensor, when a person is detected by the third person detection sensor, the speaker device 13 sounds a phrase that thanks the person saying "Thank you". It is good to output from.
At these times, the first to third motors 23 to 25 may be moved to perform an operation of pointing the automatic door set in advance from the installation position of the robot 1. If the first human detection sensor and the second human detection sensor detect a person at the same time, priority may be given to the first human detection sensor. In this way, it becomes easier for the incoming person to notice the presence of the robot 1, and it is possible to reduce the sense of incongruity that the incoming person says "thank you" to the robot 1 suddenly.

<Other Embodiments>
(1) Although the wireless LAN device 21 is provided in each embodiment and the like, a wired LAN device may be provided instead of or in addition to this and connected to a wired LAN network. The wired LAN device may be built in the robot 1 or attached externally. A wired LAN device may be connected to the terminal 18 for USB OTG (On-The-Go). The wired LAN network may be connected to the Internet via a router or the like. Depending on the environment, there may be cases where communication is not stable or can not be connected depending on the environment. For example, in the case where the robot 1 is installed in a place where there are a large number of wireless LAN devices, such as a smartphone, it is desirable that the Internet be accessed via the wired LAN device.
(2) In each embodiment etc., although the example of the dialogue of the person and the robot 1 in a half duplex system is shown, the dialogue between the person and the robot 1 may be performed in a full duplex system. . For example, after the person says "open ..." in the dialog of Table 1, the voice of "cancel" is continued while the person's voice is recognized while the "Is it really good?" If it is recognized during the utterance of "Is it really OK?", The utterance may be interrupted if it is in the process of speech and the robot may immediately say "cancelled" from the robot.
(3) The configuration in which the interaction is performed in the half duplex mode is particularly preferable because the configuration and processing can be simplified and the cost can be reduced. However, it is difficult to understand when the robot 1 turns on the microphone 12, and even if the user hears, there is a problem that the head portion of the user's voice to be recognized by the robot is often missing. We found it. In order to solve this problem, it is preferable to perform characteristic screen display at the timing when the controller MC turns on the microphone. This supports smooth conversations. As a mode of performing a characteristic screen display at the timing when the controller MC turns on the microphone, at least one of 1) lighting the four corners of the screen and 2) displaying the icon of the microphone may be performed. If both of 2) are performed, they exert excellent effects.
(4) The motors constituting the first to third motors 23 to 25 may be various motors such as a DC motor, but it is particularly preferable to use a stepping motor, and the robot 1 controls the attitude by the stepping motor It is good to be configured. The magnitude of torque changes in proportion to the current flowing through the stepping motor. A large torque can be obtained if a large amount of current is supplied, but problems such as heat generation and battery life will occur. Therefore, when the robot 1 is at rest, it is particularly preferable to supply a minimum current necessary to maintain its posture and to supply a large current only when the robot 1 changes its posture. Although all the servomotors 23 to 25 may be supplied with the minimum current necessary to maintain the posture when they are at rest, the body portion which can be supported by the detent torque (torque in the non-energized state) The first motor 23 is not energized, while the head portion loses the detent torque, so that the second motor 24 and the third motor 25 may be energized even at rest.
In addition, if the torque at the time of stationary is rotated, the torque is insufficient and the step-out occurs and the rotational does not rotate well. Therefore, it is better to increase the torque by flowing a large amount of current when rotating. On the other hand, since no large torque as in the case of rotation is required when stationary, it is better to reduce the current as compared to when rotating.
In addition, when changing the direction to a certain direction, it is preferable to increase the speed to a constant speed while accelerating, and control to decelerate and stop when the target angle approaches.
In addition, when the robot is driven by a motor, mechanical and electrical noises are generated. Since this noise reduces the recognition rate of speech recognition, the motor may be controlled to stop during speech recognition.
The scope of the present invention is not limited to the configuration or limitation explicitly described in the specification, and includes within the scope a combination of the various aspects of the present invention disclosed herein. Among the present inventions, the configuration to be patented is specified in the appended claims, but the present invention is disclosed in this specification even if the configuration is not specified in the claims. In the future, we will have the intention to set the scope of claims as the scope of claims.
The present invention is not limited to the configuration described in the above-described embodiment. The components of the above-described embodiments and modifications may be arbitrarily selected and combined. In addition, any component of each embodiment or modification, and any component described in the means for solving the invention or a component embodying the any component described in the means for solving the invention And may be configured in any combination. These also have the intention to acquire the right in the correction or divisional application etc. of the present application. Further, even if there is a description of "in case of""at time of", it is not described as a configuration limited in that case or at that time. We also disclose the configuration in these cases and not at times, and have the intention to acquire the right. Moreover, the part which is described with the order is not limited to this order. It also discloses the configuration in which some parts are deleted or the order is changed, and has the intention to acquire the right.
In addition, he / she has the intention to acquire the right for a full design or a partial design by filing a change to a design application. Although the drawing depicts the entire device as a solid line, it also includes not only the overall design but also the partial design claimed for part of the device. For example, it is a drawing in which a part of the device is not limited to a partial design, and a part of the device is included as a partial design regardless of the members. A part of the device may be a part of the device or a part of the member. In addition to the overall design, we intend to entitle the partial design where any part of the solid line part of the drawing is the broken line part.

  1 ... robot as a device, 41 ... a smartphone as another device, 51 ... a smart speaker as another device.

Claims (51)

An apparatus comprising: a function of communicating by outputting output information to at least one of a user and another apparatus,
An apparatus comprising a function of controlling generation of the output information for the communication or a function of controlling timing of outputting the output information for the communication.   The device according to claim 1, wherein the device has a display function of causing a display mode on a display unit to be changed by the communication by voice.   The apparatus according to claim 2, wherein the change in the display mode on the display unit is based on an utterance of at least one of the user or the other device.   The apparatus according to claim 2 or 3, wherein the change in the display mode on the display unit is performed alternately with the audio output from the apparatus.   The apparatus according to any one of claims 2 to 4, wherein the display mode is character information obtained by converting an utterance of at least one of the user and the other device.   The apparatus according to any one of claims 2 to 5, wherein all characters from the start to the end of the utterance are displayed on the display unit simultaneously as the character into which the utterance is converted.   7. The apparatus according to claim 2, wherein the apparatus has a function of causing the display unit to display an interaction history of communication by voice of at least one of the apparatus and the user or the other apparatus as character information. The device described in any one.   The device has a function of causing the display unit to display the interaction history when the user or the other device is replaced when the interaction history is displayed as character information. The apparatus of claim 7, comprising.   The device according to any one of claims 2 to 8, wherein the device has a function of causing a display unit of the device to display the recognition status of at least one of the user and the other device by using a voice recognition function. Device described. It has a function for performing the communication by performing speech output using the result of recognizing speech and converting it into a character string,
Corresponds to the character string when there is a portion where the result of speech recognition and conversion to a character string matches the character string stored in the storage means in which the correspondence between the character string of the result and the output content is stored in advance The apparatus according to any one of claims 2 to 9, further comprising: an audio output function for outputting the output content.   A dialog engine is provided when there is no part where the result of speech recognition and conversion to a character string matches the character string stored in the storage means in which the correspondence between the character string of the result and the output content is stored in advance. The apparatus according to any one of claims 2 to 10, comprising a function of connecting to a server and outputting voice data.   Even if there is a portion where the result of speech recognition and conversion into a character string matches the character string stored in the storage means in which the correspondence between the character string of the result and the output content is stored in advance, a certain condition is satisfied The apparatus according to any one of claims 2 to 11, further comprising a function of outputting voice data by connecting to a server provided with a voice recognition engine.   A function of detecting that a voice is interrupted after voice recognition and becomes silent, storage means for storing voice data from voice recognition to silence, and voice data stored in the storage means being silent The apparatus according to any one of claims 2 to 12, further comprising a function of outputting voice data by connecting to a server provided with a voice recognition engine at a time when it has become.   The apparatus according to any one of claims 2 to 13, wherein the apparatus has a recording function, and outputs a voice data by connecting to a server having a voice recognition engine by detecting voice of a predetermined sound pressure level. Device described.   When the server connected to a server equipped with a voice recognition engine and outputting voice data, if the server is in a busy state, voice output of an example of dialogue selected from the dialogue data stored in the storage means for the user The device according to any of the claims 2-14, characterized in that it comprises   It is characterized by having a function of outputting an example of dialogue selected from the voice data stored in the storage means as voice without connecting to a server equipped with a voice recognition engine, when it is judged that the recognized user's speech is too long. An apparatus according to any one of claims 2 to 15, wherein:   The apparatus according to any one of claims 2 to 16, wherein, in the communication by dialogue, when the user misses the voice of the device, the device re-outputs the immediately preceding voice based on an utterance of the user. Device. The apparatus according to any one of claims 2 to 17, wherein when the apparatus fails to recognize speech, the apparatus outputs speech to prompt the user to speak again.
.   The apparatus according to any one of claims 2 to 18, wherein the display unit is caused to display when a recognized voice content satisfies a certain condition.   The device has a function of moving a housing or a part connected to the housing, and the housing or the part connected to the housing performs a movement when the recognized audio content satisfies a certain condition. An apparatus according to any one of claims 2-19. The device includes an eye portion which is a portion that the user can recognize as an eye, a user position recognition function that recognizes the position of the user, and a function that moves the eye portion.
The apparatus according to any one of claims 2 to 20, further comprising a function of moving the eye to face the user's position direction recognized by the position recognition function as the communication.   22. A device according to any of claims 2 to 21, wherein the device comprises a face recognition function that recognizes the face of the user.   The apparatus according to claim 22, wherein the apparatus comprises a display unit, and the face recognition function comprises a function of displaying the recognition status of the user's face on the display unit. The position recognition function is
Three microphones placed at the top of the triangle,
Based on the difference in arrival time of sound from the sound source to each of the three microphones, the position of the sound source is projected from the position projected onto the plane including the triangle along the direction perpendicular to the plane including the triangle. The apparatus according to any one of claims 2 to 23, further comprising: a sound source direction identification function comprising: an identifying unit identifying an audio source direction going to a reference position inside the area surrounded by the triangle.   The apparatus according to any one of claims 2 to 24, wherein the apparatus comprises an infrared remote control signal output unit, and the communication is communication with the other device having an infrared remote control reception function.   The device according to any one of claims 2 to 25, wherein the device is remotely controlled from the other device via the Internet.   The apparatus according to any one of claims 2 to 26, wherein the apparatus performs voice output using character information transmitted from the other device via the Internet.   The apparatus according to claim 27, wherein a time, a time or a number of times of voice output can be changed according to the contents of the character information.   The apparatus according to any one of claims 2 to 28, wherein the apparatus has a function of transmitting voice-recognized character information to the other apparatus via the Internet. The voice-recognized character string is transmitted as the input character string to a plurality of different servers provided with a dialogue engine that outputs an output character string corresponding to the input character string,
30. The apparatus according to any one of claims 2 to 29, further comprising a function of receiving the character string output as the output character string from the plurality of different servers, selecting the longest output character string, and causing interaction. Device. The voice-recognized character string is transmitted as the input character string to a plurality of different servers provided with a dialogue engine that outputs an output character string corresponding to the input character string,
The method according to any one of claims 2 to 30, characterized in that it has a function of receiving a character string output as the output character string from the plurality of different servers, selecting the output character string with a question mark appended to the end and interacting. The device described in any one. The voice-recognized character string is transmitted as the input character string to a plurality of different servers provided with a dialogue engine that outputs an output character string corresponding to the input character string,
34. The method according to any one of claims 2 to 31, further comprising receiving a character string output as the output character string from the plurality of different servers, combining positive sentences, and combining question texts for interaction. Device described. The voice-recognized character string is transmitted as the input character string to a plurality of different servers provided with a dialogue engine that outputs an output character string corresponding to the input character string,
33. The apparatus according to claim 2, further comprising a function of receiving a character string output as the output character string from the plurality of different servers, and combining and interacting so as to arrange the character string converted into a topic last. The device described in any one. The voice-recognized character string is transmitted as the input character string to a plurality of different servers provided with a dialogue engine that outputs an output character string corresponding to the input character string,
A character string having a function of receiving a character string output as the output character string from the plurality of different servers and combining and interacting so as to initially arrange the friendly output character string. The device according to any of the above. The voice-recognized character string is transmitted as the input character string to a plurality of different servers provided with a dialogue engine that outputs an output character string corresponding to the input character string,
The apparatus according to any one of claims 2 to 34, further comprising a function of receiving the character strings output as the output character string from the plurality of different servers, combining them in random order and making them interact. . The voice-recognized character string is transmitted as the input character string to a plurality of different servers provided with a dialogue engine that outputs an output character string corresponding to the input character string,
If the character string outputted as the output character string is received from the plurality of different servers, and the output character string including the emoticon is included in them, it is not considered as a dialogue target, and the display unit 36. The apparatus according to any of the claims 2-35, characterized in that it has a function to be displayed together with the output string. The voice-recognized character string is transmitted as the input character string to a plurality of different servers provided with a dialogue engine that outputs an output character string corresponding to the input character string,
The character string output as the output character string is received from the plurality of different servers, and only one of the output character strings including the same character string is selected and combined with the other output character string 37. A device according to any of the claims 2-36, characterized in that it has the ability to interact. The voice-recognized character string is transmitted as the input character string to a plurality of different servers provided with a dialogue engine that outputs an output character string corresponding to the input character string,
It has a function to receive a character string output as the output character string from the plurality of different servers, convert the end of the output character string by the word conversion engine, and then combine and interact. The apparatus in any one of -37. The voice-recognized character string is transmitted as the input character string to a plurality of different servers provided with a dialogue engine that outputs an output character string corresponding to the input character string,
The character string output as the output character string from the plurality of different servers is received, and a part of the output character string is stored in the storage unit without using all the output character strings, and the dialogue is subsequently performed The apparatus according to any one of claims 2 to 38, characterized in that it has a function of taking it out of the storage means and using it for interaction.   The apparatus according to any one of claims 2 to 39, characterized in that a server with a voice recognition engine is used in combination with a free server and a paid server.   The device according to any one of claims 2 to 40, wherein the other device is a smart speaker, and the device performs audio output to the smart speaker to communicate with the smart speaker.   42. The device according to any of the claims 2-41, wherein the other device is a smart speaker and the device provides an audio output to the smart speaker to communicate with the smart speaker.   43. A device according to any of the claims 2-42, wherein the other device is a smart speaker and the device has a adaptation function for adapting the audio output of the smart speaker.   The apparatus according to any one of claims 2 to 43, wherein the audio output of the apparatus has a function of reading web articles.   45. The apparatus according to any one of the preceding claims, characterized in that the timing of said output is done when execution of a predetermined processing unit of robotics process automation is completed.   46. Apparatus according to any of the preceding claims, wherein the output is a notification operation.   47. The apparatus according to any one of claims 1 to 46, wherein an alerting operation is performed when a computer in execution of robotics process automation is in a state of waiting for an input from a user.   A client computer for performing robotics process automation, and a server computer for giving an execution instruction of robotics process automation to the client computer, and performing an alerting operation when the client computer receives an instruction from the server computer 48. Apparatus according to any of the preceding claims, characterized in that.   49. An apparatus according to any of the preceding claims, wherein said output information is generated to perform an operation pointing in the direction of a computer running robotics process automation.   50. An apparatus according to any of the preceding claims, wherein different said output information is generated depending on the state of execution of robotics process automation. A program for causing a computer to realize the function of the device according to any one of claims 1 to 50.