JP2022169645A - Device and program, or the like - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device equipped with a communication function, and a program.

SOLUTION: A robot 1 has a voice output function and a function for communicating with a user. The robot 1 interacts with the user using an interaction engine, and converts the last speech of the user to character string data simultaneously with the interaction, to be displayed on a touch panel section 7 serving as a display unit.

EFFECT: The user can visually confirm the speech of the user, thereby contributing to correction or direction of subsequent communication.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2023,JPO&INPIT

Description

TECHNICAL FIELD The present invention relates to a device, a program, and the like having a function of, for example, communication.

Patent Literature 1 discloses a technology related to an interactive communication robot.

JP 2011-000681 A

However, there was a problem that conventional communication robots did not have sufficient capabilities. Therefore, it is an object of the present invention to provide a device, a program, and the like, which have better performance than the conventional ones.
The purpose of the invention of the present application is not limited to this, and we intend to acquire rights for the configuration aiming to obtain the effect produced by the configuration disclosed in the specification and drawings by divisional application, amendment, etc. . For example, the present specification discloses a problem in which the phrase "can be done" is read as "is the problem." The problems are described as independent ones, and we have the intention to independently acquire the rights for the structure for solving the problems by filing a divisional application, an amendment, etc. Even if the problem is implicitly understood from the description of the specification, the applicant intends to claim part of the configuration described in this specification in an amended or divisional application. . Also disclosed is a combination of these independent tasks.

(1) A device having a function of communicating by outputting output information to at least one of a user or other device, wherein the function of controlling the generation of the output information for the communication; It is preferable to provide a function of controlling the timing of outputting the output information for the communication.
In this way, the user and/or other device can obtain output information whose generation is controlled and/or whose timing is controlled. It is possible to provide a better device than before.
The device preferably controls the generation of the output information for communication, and generates output information according to communication, for example. This will enhance the convenience of communicating with the device, especially for users or other devices. Communication may be performed by outputting any kind of output information, but it is particularly preferable to store history information of past communication and perform communication based on the history information. Moreover, it is preferable to carry out based on history information of communication with a plurality of different users or other devices. In particular, the output information may be output from one output means, but it is preferable that the configuration is such that a plurality of different output means can be used, and one of these output means is selected for output. The output means may be, for example, audio output means, display means, communication means, or the like. Communication may be performed by voice, sight, or other five senses, such as tactile sensation.

The "device" preferably has a function of outputting the output information by voice. In this way, for example, a device that operates by inputting voice can be controlled. The device may in particular comprise an interface for communicating, and may comprise decision means for carrying out the communication.
Note that the part including the configuration of the "apparatus" may be composed of a plurality of housings, but it is particularly preferable to be composed of a single housing.
Also, the device may be, for example, a system that has the ability to access a network via wire or wireless. In particular, for example, a smart phone, a tablet terminal, a smart speaker, a smart camera, or the like may be used. Also, the appearance is not limited, but it is preferable to use a device that communicates with other people. In particular, it is preferable to use a robot. For example, it is particularly good if the robot is in an anthropomorphic form, such as a human, animal, or other anthropomorphic form.

The device preferably has an interface on the input side for communication. For example, it may be an interface with an input device such as a keyboard, or an optical character recognition (OCR) that reads characters and converts them into data. It is preferable to provide a voice interface as the side interface. For audio, for example, it is preferable to have a function of acquiring audio data based on an audio signal converted into an electric signal by a microphone.
It is preferable to provide an audio interface as an output side interface, for example, a speaker device, an earphone, or the like. It is preferable to provide a visual interface as an output side interface, for example, it is preferable to provide a display capable of changing display contents, for example, a display device such as a liquid crystal display (LCD), a plasma display (PDP), an organic EL display, or a cathode ray tube is provided. Good. In addition, it is preferable to provide an output in the form of printed matter, for example. In particular, it is preferable to provide an interface on the output side that actually generates movement. An actuator may be provided to actually generate the movement. For example, a motor or the like may be provided. In particular, the device may be a robot with members that actually generate movement. In particular, the device may output the output information as the movement of the member that actually produces the movement. In particular, as an interface on the output side, it is preferable to provide an audio interface, a visual interface, and an interface that actually generates movement.
A "user" is, for example, a person who can handle the device, and may be one person, but may be a plurality of persons.
The "other device" may be the same as or different from a specific one of the devices described above, for example, in appearance, function, and the like. Other devices may not have the function of outputting audio. It is preferable to have an audio output function. Further, the other device may not have the function of inputting voice, but it is preferable to have the function of inputting voice.
The other device may be a device that cannot access the network, but may be a device that can access the network. In particular, it is preferable to use a device that can access the Internet.
It is preferable that the output information is configured to cause a certain output from the output means. "Some output" is, for example, notification to the outside. Even if it is not in the form of a clear "notification", it can be interpreted as "notification" even if there is some change as a result. "A certain output" does not necessarily have to be intended for notification. For example, the output of sound or light that has some information or no information is good, for example, the change of some physical quantity, the movement of an object, etc. is good.

(2) Preferably, the device has a display function to change the display mode on the display section according to the voice communication.

Since the display mode of the display unit can be changed in relation to the communication by voice when communicating by voice, it is possible to communicate by changing the display of voice to be seen by eyes, and the convenience of communication is enhanced.
The “display unit” may be a device that displays so as to change the display mode on the display unit by the communication by voice, such as a liquid crystal display (LCD), a plasma display (PDP), an organic EL display, a cathode ray tube, etc. Such a display device is preferable. In particular, the display unit is preferably provided in the device.
"Display so as to change the display mode on the display unit according to the communication by voice" refers to the case where the voice of the user or other device is displayed on the display unit and the mode is changed, and the voice of the device itself is also displayed. In the case of displaying on the part and changing its mode, only one of them may be provided, but it is particularly preferable to provide both. At this time, either one or both may be displayed, but only one is displayed. It is preferable to have both a state in which the display is displayed and a state in which both are displayed, and to have a switchable configuration. For example, an example of displaying only one face screen is the case where the face screen S1 is displayed in the robot 1 of Embodiment 1 below, and an example of displaying both is the chat screen of the robot 1 of Embodiment 1 below. This is a mode when S2 is displayed.
As a display mode, for example, the screen may be changed in various ways in relation to the sound. For example, an animation may be executed in which an object displayed on the display screen is moved by the sound. For example, it is preferable to change the image according to the sound output, or to superimpose another image on the image according to the sound change. Also, for example, voice data may be converted into character data and displayed. The display of the character data should be changed every moment according to the change of voice.
The communication by voice may be controlled by a function of controlling the generation of the output information or a function of controlling the timing of outputting the output information for the communication, and in particular the communication controls the generation of the output information. It is preferable to control by a function for controlling the timing of outputting the output information for the function and the communication.
Further, the function of changing the display mode on the display unit may be controlled by the function of controlling the generation of the output information or the function of controlling the timing of outputting the output information for the communication. The control may be performed by a function of controlling the generation of the output information and a function of controlling the timing of outputting the output information for the communication.
In the following (3) and later, similarly, the configuration for outputting from the device may be controlled by the function of controlling the generation of the output information or the function of controlling the timing of outputting the output information for the communication. In particular, the communication may be controlled by a function of controlling the generation of the output information and a function of controlling the timing of outputting the output information for the communication.

(3) It is preferable that the change of the display mode on the display unit is based on only speech by at least one of the user and the other device.

By changing the display mode of the device based on the speech from the user or other device, the user or other device can visually check whether the device recognizes their own speech, thereby improving communication. Convenience at the time of planning increases. In addition, it becomes a special communication by a different kind of human interface, and it feels fresh and interesting.
The speech may be made directly by the user or another device, or may be indirectly converted into character data, for example. Alternatively, the utterance may be converted into some corresponding information, such as other sounds or visualized patterns, and the display mode may be changed on the display unit based on the converted information.
In (3), since it is "based on the speech of at least one of the user and/or other device", the device itself may have a voice communication function, for example.
As a configuration based on the utterance of at least one of the user and/or the other device, for example, speech is converted into a character string by a speech recognition function, and the content of the utterance of at least one of the user and/or the other device is obtained. A specific configuration may be provided.

(4) The change of the display mode on the display unit is alternately performed with the voice output from the device.
Communication is not one-sided, and communication can be achieved while communicating stably. "Alternately" means, for example, a configuration in which communication between the device side and the user or other device side basically progresses in an interactive manner, and may be a configuration in which only one side does not speak unilaterally.

(5) Preferably, the display mode is character information obtained by converting the speech of at least one of the user and the other device.

In this way, the user or other device side can specifically understand from the displayed character information how their own speech is recognized by the device, and it is displayed whether communication is being performed correctly. It can be judged from the contents of character information. In addition, it is possible to visually confirm the uttered contents. Also, if the recognition is wrong, you can say it again or restate it with other expressions to guide you to correct communication.
In the case of Japanese, for example, "character information" is characters such as normal kanji, hiragana, katakana, etc. based on the utterance of the user or other equipment, and when the utterance constitutes a sentence, It should be a sentence that has a mixture of kanji, hiragana, katakana, foreign language notation, etc. When spoken in a foreign language, such as English or Chinese, it is preferable to display in those characters.
For example, having a voice recognition function and a voice output function, displaying on a display unit the content of at least one of said user or said other device that has been speech-recognized by the voice recognition function and converted into character information, and It is particularly preferable to have a function of generating response character information based on the content, converting the response character information into voice information by a voice synthesis function, and outputting it as voice.

(6) It is preferable that all contents from the start to the end of the speech are simultaneously displayed on the display unit as to the characters converted from the speech.
Since the entire utterance of a certain length from the user or other device can be received by the device, correct communication from the device in response to the utterance can be expected. In addition, it is possible to instantly visually confirm the content of what the user has said, which contributes to the correction and direction of subsequent communication.
It is preferable to set the time from the start to the end of the speech taking into consideration the time that a person can speak in one breath. The end of the speech may be, for example, a point in time at which the volume of sound continues for a predetermined period of time, at which no speech is assumed to occur. Speech may be started, for example, on the condition that the loudness of the sound exceeds a predetermined level. Speech may be started, for example, on the condition that a sudden change in the loudness level of a predetermined level or more is detected.

(7) The device preferably has a function of displaying, as character information, a dialogue history of voice communication between the device and at least one of the user and the other device on the display unit.

It is easy to understand how the device interacts with the user or other equipment, which increases the convenience of voice communication.
It is preferable to display the dialogue history so that, for example, it can be seen which one has interacted with the other. For this purpose, for example, it is preferable to provide speech balloons to distinguish character information based on which utterance. It is preferable that past dialogues can be checked by scrolling on the screen. It is preferable to display different avatar characters on the device side and on the user or other device side to indicate the interactive format. If the date and time of the utterance are displayed at the same time, it will be an opportunity for the user to recall the past from the dialogue history.

(8) When the device displays the dialogue history as character information, if there is a change in the user or the other device and the dialogue target of the device is changed, the device displays that fact on the display unit. It is better to have a function to let

If the dialogue object of the device changes, the contents of the dialogue also change. By displaying that the dialogue target has changed, for example, when looking at the past dialogue history, the reader can understand that the dialogue content has changed before and after that display, so that the discontinuity of the dialogue content can be understood. Become.
The device may be equipped with functionality to detect replacement of said user or said other device. It is preferable to have a function to detect changes in voice characteristics for detection of alternation, but it is desirable to have a configuration that has a function to acquire and detect the state of surrounding people or equipment using a camera, especially detection based on both. It should be configured to

(9) The device preferably has a function of displaying the speech recognition status of at least one of the user and the other device on the display section of the device by means of a speech recognition function.

For example, when the user or another device is speaking, the user can be made to understand that the conversation is going on smoothly by making the user understand that they are definitely listening to it.
For example, as a visual function, depending on the degree of speech recognition on the display unit, for example, the display screen or the color of the object displayed on the display screen can be changed, or, for example, different objects can be displayed according to the speech recognition status. It is preferable to display a quantitative display according to the degree of speech recognition, for example, to indicate a high numerical value if the speech is well recognized. As an auditory function, for example, it is preferable to increase or decrease the sound according to the degree of speech recognition, for example, it is preferable to change the timbre.
In particular, it is preferable to have a configuration in which an anthropomorphic form is displayed on the display section and the facial expression is changed.

(10) A function for performing the communication by outputting the voice using the result of recognizing voice and converting it into a character string, and the result of recognizing voice and converting it into a character string is provided in advance. When there is a part that matches the character string stored in the storage means storing the correspondence relationship between the character string and the output content, it is preferable to provide a function of outputting the output content corresponding to the character string by voice.

When a character string converted by recognizing voice has a part that matches a character string stored in a voice storage means, if the necessary voice output can be performed only within the device, it is possible to quickly respond to the user's or another device's speech. can respond. In addition, since there is no connection to an external server, connection costs can be reduced.
For example, it is preferable to prepare a large number of built-in scenarios as character strings stored in storage means. A built-in scenario is a scheduled interaction, for example, a simple greeting such as "How are you?" Scenarios such as "Open the setting screen" (user), "Are you sure?" (device), "Yes" (user), "Okay, open the setting screen" (device) are good. As the storage means, for example, a ROM or SSD inside the computer, an external SD card, a micro SD card, a CD-ROM, or the like may be used.

Here, "the case where there is a part that matches the character string" means the case where the character string matches the completely stored character string, and the case where the regular expression is allowed to differ in some part. A regular expression is one of the language processing methods that expresses a set of character strings with a single character string. Even if there are different parts such as "increase the volume" and "increase the volume", if the essential parts match, the interpretation is to interpret "increase the volume". And "output the output content corresponding to the character string by voice" means, for example, a voice saying "Yes, increase the volume" in response to the character string "Increase the volume" Good output. Also, following such audio output, the device may perform certain processing. For example, after saying "yes, turn up the volume", the device should actually increase the volume of its own utterance in subsequent dialogues.
The process of recognizing the voice and converting it into a character string may be performed by the device. You may make it receive and carry out. Preferably, both are provided, and a function for determining which result to use according to the communication situation or the like is preferably provided.

(11) If the result of recognizing the voice and converting it into a character string does not match the character string stored in the storage means that stores the corresponding relationship between the resulting character string and the output content in advance, the dialogue is performed. It is preferable to have a function of connecting to a server having an engine and outputting voice data.

If the character string converted by recognizing the voice does not match the character string stored in the voice storage means, the server with the dialog engine is connected, so the connection cost can be reduced.
The server may be, for example, a device that has computer functions as a storage unit and a control unit that are connected using an Internet line. The present invention preferably includes a dialog engine. In the case of an external server, for example, it is possible to connect using an ID, password, or electronic authentication. A cloud server is good for an external server. The server preferably has a speech recognition engine and can convert speech into character string data by the speech recognition engine. The converted character string data is preferably transmitted to the device using an internet line.
The function of connecting to a server equipped with a dialogue engine and outputting voice data is, for example, sending a character string recognized by speech to the dialogue engine, and receiving a character string containing dialogue content corresponding to the character string from the dialogue engine. , it is preferable to convert the character string of the dialogue contents into speech data by a speech synthesis function.
The conversion of character strings into speech data may be performed by a speech synthesis engine provided in the device, but the character strings are sent to a speech recognition server that converts the character string into speech data, It is preferable to receive voice data corresponding to the character string.

(12) Even if the result of recognizing the voice and converting it to a character string has a part that matches the character string stored in the storage means that stores the corresponding relationship between the resulting character string and the output content, there is a part. It is preferable to have a function of connecting to a server having a speech recognition engine and outputting speech data by satisfying the conditions.

When the character string converted by recognizing the voice has a part that matches the character string stored in the storage means, outputting a fixed voice that the user can predict will discourage dialogue. Therefore, intentionally connecting to an external server in this way may enable more human-like interaction.
For example, when the user says "Hello" and the device side recognizes it, the original scenario was a built-in scenario in which a conversation such as "Hello, how are you?" It is better to request voice data of "Hello" from the external server without using a scenario, and to request creation of response data to that "Hello" using the dialogue engine of the external server. The certain condition may be, for example, once every several times or random timing.

(13) A function to detect that the voice is interrupted after voice recognition and silence, a storage means for storing voice data from voice recognition to the silence, and a storage means for storing the voice data stored in the storage means. It is preferable to have a function of connecting to a server having a speech recognition engine and outputting speech data at the timing of silence.

There is often silence in dialogue. However, even if there is no sound, connecting to a server equipped with an external speech recognition engine will incur unnecessary costs. Therefore, by storing the voice data from voice recognition to silence in advance in the storage means and sending the voice data at the timing of silence instead of real time, it is possible to cut the time of the silence part. Cost can be reduced.

(14) The device preferably has a recording function, and a function of connecting to a server having a speech recognition engine and outputting speech data upon detection of speech at a predetermined sound pressure level.
Constantly connecting to a server equipped with an external speech recognition engine would result in unnecessary costs. As a result, the external server is connected only when the necessary dialogue is started without connecting when the dialogue is not silent or nearly silent, so the cost for connection can be reduced.
A server equipped with a speech recognition engine may receive recorded data of a predetermined period in the past that has already been recorded by the device and return a character string corresponding to the recorded data. It may be of a type that receives voice data and returns a character string. Although there are many cases in which pay-as-you-go billing is performed in the form of how many yen per audio data reception time, it is possible to significantly reduce costs.

(15) Dialogue example selected from dialogue data stored in storage means for said user when connected to a server having a speech recognition engine and outputting voice data, and said server is in a busy state. It is preferable to provide a function for outputting as voice.

When it is busy, it is normal to announce that fact, but for example, such an announcement in the middle of a conversation is abrupt and unrelated to the conversation, and there is a possibility that the conversation will be shy. . For this reason, instead of notifying that the system is busy, for example, if you ask "Can you come back again?" It becomes possible to connect to the engine and continue an appropriate dialogue, and the dialogue does not become unnatural.

(16) A function of outputting as voice a dialogue example selected from voice data stored in a storage means without connecting to a server equipped with a voice recognition engine when it is determined that the recognized user's utterance is too long. It is better to

If the user's utterance is too long, the speech recognition engine may misinterpret it. And as a result, you may get a wrong answer. Therefore, if the sentence becomes more than a certain amount, in order to dare to eliminate such a possibility and reorganize the dialogue It is preferable to select a dialogue example, such as a back-and-forth that can be taken in any way, and output it by voice, so that appropriate dialogue can be continued.

(17) In the communication by dialogue, when the voice of the device is missed, the device may re-output the previous voice based on the user's utterance.
For example, if you can't hear the words spoken by the device just before, such as "say it again" or "speak again", or if you forget to hear it, you can use this kind of call to can speak words. As a result, it is possible to continue the conversation that has been going on until just before without interruption.

(18) When the voice cannot be recognized, the device preferably outputs voice so as to prompt the user to speak again.
As a result, it is possible to continue the conversation that has been going on until just before without interruption.

(19) When the recognized voice content satisfies a certain condition, it is preferable to make a certain display on the display unit.
For example, when an utterance including a predetermined word is uttered and the speech is recognized, the display unit is caused to display a "certain display" corresponding to the word. The "predetermined word" may be, for example, the user's birthday, the user's child's name, the nickname of the device, the name of the company, a particular advertising catchphrase, or the like. It is preferable to have a function of presetting a correspondence relationship between a predetermined word and a certain display. As a result, it is possible to communicate not only by simple dialogue but also by visual observation, and the mode of communication with the device increases, thereby increasing the convenience of communication.

(20) The device preferably has a function of moving the housing or the part connected to the housing, and moves the housing or the part connected to the housing when the recognized speech content satisfies a certain condition. .
For example, when an utterance including a predetermined word is uttered and the speech is recognized, the housing or a part connected to the housing is caused to make a certain movement, such as a gesture, corresponding to the word. As a result, it is possible to communicate not only with a simple dialogue but also with the movement of the device, and the mode of communication with the device increases, thereby increasing the convenience of communication. It is especially good when combined with the "certain display" above.

(21) The device includes an eye part that is a part that the user can recognize as an eye, a user position recognition function that recognizes the position of the user, and a function that moves the eye part, and the position recognition function as the communication. It is preferable to provide a function of moving the eyes so as to face the direction of the position of the user recognized in .

By turning the eyes that can be recognized as eyes to the direction of the user's position, it is possible to obtain the pseudo sensation of actually talking to a person, and the desire to communicate with the device is increased, and the utility value of the device. improves.
The 'eye part, which is a part that the user can recognize as an eye', may be an eye as an object displayed on a display screen, or may be an eye that is not a virtual image but actually operates mechanically. The device itself may also be controlled to face the user's position in synchronization with the eyes. Only the device for directing the eye toward the user may be controlled to direct toward the user's position.

(22) The device preferably has a face recognition function for recognizing a user's face.
Since each person's face can be identified, it is possible to communicate according to each person's individuality. For example, by associating each person's authenticated face with a name, the person whose face has been recognized can be called by that name during dialogue. Moreover, it is particularly preferable to have a configuration in which a dialogue specialized for a person whose face has been recognized based on the past dialogue history is performed.

(23) Preferably, the device includes a display section, and has a function of displaying the recognition status of the user's face on the display section by the face recognition function.
In this way, the user can grasp the recognition status of his/her own face by looking at the display section. In particular, it is preferable to display as the recognition status whether face recognition has been completed or whether the face has not yet been recognized as a person's face. You can cooperate without moving your face so that it is easy.

(24) The position recognition function identifies the position of the sound source based on three microphones arranged at the vertices of a triangle and the difference in the arrival time of sound from the sound source to each of the three microphones. a sound source direction specifying unit that specifies a sound source direction from a position projected onto a plane containing the triangle along a direction perpendicular to the plane to a reference position inside the area surrounded by the triangle on the plane. It should be a function.
This makes it possible to specify the sound source direction with three microphones. If the direction of the sound source can be specified, the device can be turned in that direction when the user speaks, so the user can feel as if they are communicating through dialogue.

(25) The device may include an infrared remote control signal output unit, and the communication may be communication with the other device having an infrared remote control reception function.
This makes it possible to easily communicate with the device via the infrared remote control signal output section of the device.
For example, an infrared remote control signal is output from a device to a receiving device equipped with an infrared remote control signal receiving section, for example, a receiving device equipped with an infrared remote control signal receiving section, such as a television, an audio device, an air conditioner, etc. Control such as ON/OFF can be executed. In particular, the device may be provided with a voice interaction function, and may be configured to control the infrared remote control signal output section based on the commands, for example, "Turn on the TV" or "Turn off the TV".

(26) The device may be remotely controlled via the Internet from the other device.
Since the device can be remotely operated from another device, the convenience of the device is enhanced. For example, a smart phone or the like as another device may be used. The device may be equipped with a camera. The device may be provided with a mechanism for changing the orientation of the camera. By accessing from another device, for example, it is preferable to view a video camera on the device side or change the direction of the camera. This allows control of the device without being near the device. Also, for example, it is preferable to access from a smart phone, turn on the monitoring function of the device, and notify the smart phone that a person (object) has moved by e-mail. In addition, for example, as a watch over a sick person or a care recipient, it is preferable to notify the person when the person is not moving for a certain period of time or longer, assuming that the person is always moving. The other device may be, for example, a tablet terminal, a personal computer, or the like.

(27) It is preferable that the device performs the voice output using character information transmitted from the other device via the Internet.
For example, it is preferable to have a function to read aloud the character string of the received e-mail. By setting to receive an e-mail from someone, the content of the e-mail is output by voice from the device, so there is no need to visually check one's own terminal. The other device may be, for example, a smart phone, a tablet terminal, a personal computer, or the like, but may be another “device”.

(28) It is preferable that the voice output can change the time, time or number of times the voice output is performed depending on the content of the character information.
For example, I want to make the e-mail "I took medicine" be spoken at a fixed time. For example, if the description of the subject matches, the device will speak at a predetermined time. can be run without

(29) The device may have a function of transmitting character information obtained by speech recognition to the other device via the Internet.
By using the voice communication function of the device to convert voice into text and send it to another device as text data, for example, when you want to send an e-mail, you can send it without manually entering it into your own terminal.

(30) Transmitting a speech-recognized character string as the input character string to a plurality of different servers having a dialogue engine that outputs an output character string corresponding to the input character string, and sending the output character string from the different servers; , and select the longest output string for interaction.
The longest reply gives a feeling of dialogue, eliminates the monotony of the dialogue, and allows the listener (user) to enjoy the dialogue.

(31) Transmitting a speech-recognized character string as the input character string to a plurality of different servers having a dialogue engine that outputs an output character string corresponding to the input character string, and sending the output character string from the different servers; It is preferable to have a function to receive a character string output as , select the output character string with a question mark at the end of the word, and interact with the output character string.
When a question mark is added to the end of a word, the flow of the story becomes one that further answers the question.

(32) Transmitting a speech-recognized character string as the input character string to a plurality of different servers having a dialogue engine that outputs an output character string corresponding to the input character string, and sending the output character string from the different servers; It is preferable to have a function to receive the character string output as , combine affirmative sentences, and then combine interrogative sentences to have a dialogue.
By arranging it in this way, the response becomes as if the user has carefully thought out the sentences. Conversation is easier to continue and the listener (user) can enjoy the dialogue. In addition, it is possible to earn an output scale and to request a reply from the listener (user).

(33) Transmitting a speech-recognized character string as the input character string to a plurality of different servers having a dialogue engine that outputs an output character string corresponding to the input character string, and sending the output character string from the different servers; It is preferable to have a function to receive the character string output as , and combine and interact so that the character string that changed the topic is arranged at the end.
By arranging in this way, the topic is changed, and the conversation invites the next utterance, making it easier to continue the conversation.

(34) Transmitting a speech-recognized character string as the input character string to a plurality of different servers having a dialogue engine that outputs an output character string corresponding to the input character string, and sending the output character string from the different servers; It is preferable to have a function to receive a character string output as , combine and interact so as to place the friendly output character string first.
Arranging in this way makes it easier for the listener (user) to be drawn into the dialogue, making it easier for the dialogue to continue.

(35) Transmitting a speech-recognized character string as the input character string to a plurality of different servers having a dialogue engine that outputs an output character string corresponding to the input character string, and sending the output character string from the different servers; It is preferable to have a function to receive the character strings output as , combine them in random order, and interact with them.
Since the variation of the dialogue will increase, even if the listener (user) makes the same utterance, the exact same response will not be returned, making it easier to continue the dialogue without getting bored.

(36) Transmitting a speech-recognized character string as the input character string to a plurality of different servers having a dialogue engine that outputs an output character string corresponding to the input character string, and sending the output character string from the different servers; , and if there is an output character string containing emoticons in them, it is not interactive, and is displayed on the display unit together with the output character string that is interactive It is preferable to have a function to allow
Emoticons cannot be output as voice, but by intentionally displaying emoticons on the display unit and making them part of the dialogue together with the voice, it is possible to create an unusually interesting dialogue.

(37) Transmitting a speech-recognized character string as the input character string to a plurality of different servers having a dialogue engine that outputs an output character string corresponding to the input character string, and sending the output character string from the different servers; , and selects only one of the output character strings containing the same character string, combines it with the other output character string, and interacts with it.
This is because if the same character string is repeated, the dialogue becomes verbose and the listener feels uncomfortable.

(38) Transmitting the speech-recognized character string as the input character string to a plurality of different servers having a dialogue engine that outputs an output character string corresponding to the input character string, and sending the output character string from the different servers. It is preferable to have a function to receive a character string output as , convert the ending of the output character string by an ending conversion engine, and then combine and interact.
It is good because it becomes a more friendly expression compared to the sentences of a normal dialogue engine.

(39) Transmitting a speech-recognized character string as the input character string to a plurality of different servers having a dialogue engine that outputs an output character string corresponding to the input character string, and sending the output character string from the different servers Receives the character string output as , stores a part of the output character string in a storage means without using all the output character string, and retrieves it from the storage means in the subsequent dialogue and uses it for the dialogue It is preferable to have a function to allow
By using it when speech recognition fails or when it is difficult to receive a response from an external server, dialogue can be continued without interruption, contributing to natural dialogue.

(40) When using a server equipped with a speech recognition engine, it is preferable to use a mixture of free servers and paid servers.
This can save server connection charges, for example, especially for heavy users of interaction with the device.

(41) The other device may be a smart speaker, and the device may communicate with the smart speaker by outputting audio to the smart speaker.
Since the smart speaker does not have a display, using it in combination with the device enhances convenience.
The smart speaker may be, for example, a speaker having a wireless communication connection function and a voice operation assistant function. For example, Google Home, Amazon Echo, LINE Clova, etc. The smart speaker may be provided with functions for realizing various functions and abilities (skills), for example. The function can be executed by speaking to the smart speaker from a device that communicates by voice. When making the device speak, for example, the user speaks a phrase that activates the skill of the smart speaker, the device recognizes the speech, saves it as character string data, and synthesizes the character string data at a certain timing. to execute the skill by speaking to the smart speaker.

(42) The other device may be a smart speaker, and the device may communicate with the smart speaker by outputting audio to the smart speaker.
Activate a smart speaker or activate a skill. Or make an inquiry. It is possible to automatically execute the skill of the smart speaker at a certain timing or a certain scheduled time without starting the smart speaker by oneself.

(43) The other device may be a smart speaker, and the device may have an adaptation function for adapting the audio output of the smart speaker.
It's useful if you're tired of listening to the smart speaker's fixed, long answer to a question, or if you just want a quick refresher.

(44) It is preferable that the voice output of the device has a function of reading out a Web article.
You can listen to the article only by interacting with the device without reading the article on the Web.

(45) It is preferable that the certain output is performed when execution of a predetermined processing unit of robotics process automation is completed.
In robotics process automation, it is difficult to understand the execution status of each processing unit.

(46) The certain output may be a notification operation.
It will be understood that a certain output has been made by the notification operation.
(47) A notification operation is performed when a computer that is executing robotics process automation waits for an input from a user.
This makes it possible to notify the user of the input waiting state and prompt the next process.

(48) A client computer that performs robotics process automation, and a server computer that gives instructions to the client computer to execute the robotics process automation, and when the client computer receives an instruction from the server computer, the notification operation is performed. It is recommended that you do so.
This makes it possible to notify the user that an instruction has been received from the server computer and prompt the next process.

(49) Preferably, the output information is operable to point toward a computer executing robotics process automation.
This makes it possible for the user to know on which computer the execution was performed, and to prompt the user for the next process.
(50) Different output information may be generated according to the execution state of the robotics process automation.
This makes it possible to distinguish what kind of execution was performed.

(51) A program for causing a computer to implement the function of the device according to any one of (1) to (50).
The part described as "exists" such as "some output" may be changed to "predetermined", for example.
The inventions shown in (1) to (50) above can be combined arbitrarily. For example, a configuration may be adopted in which at least part of the configuration of at least one of the following inventions (2) is added to all or part of the configuration of the invention shown in (1). In particular, the invention shown in (1) may be added with at least a part of the configuration of at least one invention after (2). Also, arbitrary configurations may be extracted from the inventions shown in (1) to (50) and the extracted configurations may be combined. The applicant of this application intends to obtain rights to inventions including these configurations. In addition, even if there is a description of "in case of" or "when", it is not described as a configuration limited to that case or that time. We disclose and intend to reserve these occasions and occasions. Moreover, the parts described with order are not limited to this order. It also discloses a configuration in which some parts are deleted or the order is changed, and there is an intention to acquire the right.

When communicating with a user or other device, the device can generate output information in response to the communication. This increases the convenience of communicating with the device by the user or other device.
The effect of the invention of the present application is not limited to this, and the effect produced by the parts of the configuration disclosed in the specification and drawings, etc. is also disclosed, and the configuration that produces the effect is also acquired by divisional application, amendment, etc. have the intention to For example, in this specification, the description of "can be done" is a description that clearly shows the effect, and there are parts showing the effect even if there is no description of "can be done". Moreover, even without such a description, there is an effect that can be grasped by the configuration.

1 is a front view of a robot according to Embodiment 1 of the present invention; FIG. The side view of the robot of the same Embodiment 1. FIG. FIG. 2 is a rear view of the robot of the same Embodiment 1; FIG. 2 is a block diagram for explaining the electrical configuration of the robot; Explanatory drawing capturing an example of an expression with a face screen displayed on the face of the robot. Explanatory diagram capturing an example of a chat state with a chat screen displayed on the face of the robot. FIG. 4 is an explanatory diagram for explaining a standby image with a face screen displayed on the face of the robot as a background; FIG. 4 is an explanatory diagram for explaining a standby image with a chat screen displayed on the face of the robot as a background; (a) to (d) are explanatory diagrams for explaining deformation patterns of eye objects displayed on the face of the robot. 4(a) to 4(c) are explanatory diagrams for explaining how the user's utterance content gradually appears as a character string on the robot's face. FIG. 4 is an explanatory diagram for explaining a state in which an eye object moves following the user's face on the face screen displayed on the face of the robot; Explanatory drawing explaining an example of a smart phone. FIG. 10 is an explanatory diagram for explaining the relationship between robot activation, wakeup mode, interactive mode, and sleep mode. FIG. 21 is an explanatory diagram for explaining the relationship between a robot and a smart speaker in Embodiment 7; FIG. 20 is an explanatory diagram illustrating an example of a smartphone in Embodiment 9;

<Embodiment 1>
As shown in FIGS. 1 to 3, a robot 1, which is a communication robot that operates in response to human voice, has a fixed part 2 as a lower body and a movable part 3 as an upper body placed on the fixed part 2. is provided as a housing. The movable part 3 is composed of a body part 4 arranged adjacent to the fixed part 2 and a head part 5 supported by the body part 4 . The fixed part 2 is formed in a bowl-like appearance that opens upward, and the body part 4 is formed in a cylindrical shape composed of a curve continuous in the vertical direction with the upper edge of the fixed part 2 . The robot 1 has the largest diameter at the connecting portion between the fixed portion 2 and the body portion 4, and has an outer shape that narrows in the vertical direction with the connecting portion as a boundary. The trunk portion 4 has a large semicircular cutout at the front of its tubular shape. The head portion 5 is fitted into the upper portion of the body portion 4 so as to be embedded therein. The body 4 rotates horizontally (in the direction of the arrow in FIG. 1) with respect to the fixed part 2, and the head 5 rotates vertically (in the direction of the arrow in FIG. 2) and in the direction of horizontal rotation (in the direction of the arrow in FIG. arrow direction)).

The head 5 as a whole is configured in a shape of a sphere, which is the remainder of a sphere cut in one plane (front part). The cut front portion appears in a circular shape and constitutes the face portion 6 of the robot 1 . A rectangular portion formed on the surface of the face portion 6 serves as a touch panel portion 7 as a display portion, which is a liquid crystal display (LCD) having a touch panel function. The contents displayed on the touch panel section 7 will be described later. A smoke panel is arranged in the area of the face part 6 around the touch panel part 7, and the entire face part 6 is a unified dark background. Inside the head 5, an illuminance sensor 8 and a high-brightness white LED 9 are arranged at accommodation positions on the upper left and right sides of the face part 6, respectively. A lens 11 of a face recognition camera 10 is arranged at the upper center position of the touch panel section 7 in the face section 6 .

Microphones 12 are arranged inside the body section 4 at three positions at the same height, which are shifted by 120 degrees between the front left and right positions and the rear center position of the body section 4 . A pair of left and right speaker devices 13 are arranged on the fixed portion 2 inside the fixed portion 2 . An opening 14 for outputting the sound generated by the speaker device 13 is formed on the side of the speaker device 13 . A power switch 15 and an up switch 16 and a down switch 17 for adjusting the volume of the speaker device 13 are provided at positions adjacent to the speaker opening 14, respectively. A USB OTG (On-The-Go) terminal 18 , a DC 12 V power jack 20 , and a micro SD card socket (reader) 19 are arranged at the rear position of the fixed portion 2 .

The robot 1 can be connected to a predetermined external cloud server using an internet line. The cloud server has a storage area as storage means for storing data required by the robot 1, various engines for performing various processes required by the robot 1, and the like. Therefore, in a broad sense, the robot 1 can be interpreted as a device including portions such as software of these cloud servers.

Next, based on the block diagram of FIG. 4, the electrical configuration of the robot 1 of Embodiment 1 will be described.
The above-described touch panel unit 7, illuminance sensor 8, high-brightness white LED 9, face recognition camera 10, microphone 12, speaker device 13, terminal 18, and micro SD card socket 19 are connected to the controller MC as control means. , a wireless LAN device 21, a Doppler sensor 22, first to third motors 23 to 25, etc. are connected.

The touch panel unit 7 has an input operation function of inputting by touching the surface thereof. The touch panel unit 7 can display different screens as shown in FIG. 5 or 6 in a natural dialogue mode, which will be described later. The controller MC causes the touch panel section 7 to displaceably display a face screen S1 as a first screen as shown in FIG. The face screen S1 is a screen displayed by default, and the eye object 27, cheek object 28, and elliptical area 29 of the robot 1 are displayed. The eye object 27 has several deformation patterns of the eye object 27 as an animation image (FIGS. 9(a) to (d)). Also, as an animation image, the pupil object 27a moves left and right.

Further, the controller MC causes the touch panel section 7 to display a chat screen S2 as shown in FIG. 6 as a second screen. The chat screen S2 can be displayed on the touch panel section 7 instead of the face screen S1 by touching and sliding the touch panel section 7 in the state of the face screen S1. The display of the face screen S1 and the chat screen S2 can be switched between each other by a slide operation. The chat screen S2 will be described later.
Further, the touch panel section 7 can display the standby image shown in FIG. 7 or 8 on the touch panel section 7 in the standby mode (wake-up mode). The standby image will be described later.
A setting screen is prepared for the robot 1 as a different image other than these, and it is possible to move to the setting screen from the chat screen S2. When the robot 1 is activated for the first time, access from the setting screen, for example, ID / password setting registration to the server, Wi-Fi password setting registration, user registration (e.g. name, age, gender, etc.), face authentication , enter the necessary initial setting items such as setting the e-mail address to which the data is to be transferred.

The illuminance sensor 8 recognizes the brightness of the environment in which the robot 1 is installed. The high-brightness white LED 9 is automatically turned on based on the numerical value detected by the illuminance sensor 8 when the brightness is insufficient for photographing by the face recognition camera 10 .
The microphone 12 is a voice input means for capturing the user's utterance in a dialogue with the user. It is also a direction detection means capable of specifying the direction of the sound source by the difference between the . The controller MC obtains the arrival time difference from the phase difference of the electrical signals acquired by each microphone 12 . The controller MC calculates the sound source angle with respect to the reference direction based on the arrival time difference. A microphone specialized for dialogue with the user may be provided, for example, on the face portion 6 .
The speaker device 13 is an audio output means through which the robot 1 speaks (outputs audio) in dialogue with the user.
The micro SD card socket 19 reads and rewrites data on the inserted micro SD card.
The wireless LAN device 21 is a device for wirelessly connecting the robot 1, which is a Wi-Fi compatible device, to the Internet. In this embodiment, it is the international standard of IEEE802.11b.
The Doppler sensor 22 is a sensor that uses microwaves, emits microwaves, compares the frequency of the reflected microwaves with the frequency of the emitted radio waves, and determines whether an object (person) is moving. To detect. It utilizes the fact that the frequency of reflected waves changes when an object (person) moves due to the Doppler effect. For example, it is a device used to detect abnormalities around the robot 1, such as whether or not there is a suspicious person when the user is absent.
The first motor 23 is a servomotor for rotating the body portion 4 in the horizontal direction (in the direction of the arrow in FIG. 1) with respect to the fixed portion 2 . The second motor 24 is a servomotor for rotating the head 5 in the vertical direction (in the direction of the arrow in FIG. 2) with respect to the body 4. As shown in FIG. A third motor 25 is a servomotor for rotating the head 5 in the left-right direction (the arrow direction in FIG. 3) with respect to the body 4 . When the direction in which the user speaks is determined by the microphone 12, the controller MC controls the first motor 23 so that the face part 6 faces the direction of the user, and moves the body part 4 (movable) with respect to the fixed part 2. Part 3) is rotated.

The controller MC is composed of a well-known CPU, ROM, RAM, memory as storage means such as SSD, bus, real-time clock (RTC), and the like. Various programs for executing various functions of the robot 1 are stored in the ROM of the controller MC.
Various programs include, for example, a dialogue program for controlling dialogue with the user via the microphone 12 and the speaker device 13, a face recognition program for face recognition using the face recognition camera 10, the touch panel unit 7, the first to the first A display variation/gesture program for controlling the motors 23 to 25 of 3 to change the facial expressions and actions of the robot 1 during a dialogue with the robot 1, a different screen and gesture program based on the dialogue with the user and the operation of the touch panel unit 7 A screen display program for displaying an image on the touch panel unit 7, a data transmission/reception program for processing, for example, a camera image acquired by the robot 1 side with another computer or smartphone, e-mails from a smartphone, etc. It stores an absence setting program for watching over the home, a GUI function, a network connection function, an OS for operation, operation, and operation such as process management. In the RAM, input/output data and the like in dialogue and face recognition are temporarily stored. Each program realizes functions such as dialogue, face recognition, and gestures through multitasking in cooperation with other programs or independently.

A. Contents of Operations During Dialogue In the configuration described above, the controller MC controls communication through dialogue with the user by executing a dialogue program. Note that the face recognition that is enabled synchronously with the start of the dialogue will be described later in the following "B. Contents of operation at the time of face recognition".
where the dialogue program is
1) For issuing a request to the cloud server for the user's utterance data (voice data) acquired from the microphone 12 and responding with the user's utterance data (character string data) converted to text using the server-side speech recognition engine. Sub-program 2) Built-in scenario sub-program that executes built-in scenario dialogue based on user's utterance (character string data) 3) If user's utterance does not correspond to built-in scenario, utterance data (character string data) is returned to cloud server Utterance data transfer subprogram that issues a request to , and uses the dialogue API (application programming interface) to create the response data (character string data) of the robot 1 in the dialogue engine 4) The response data (character string data) is sent to Character string data display subprogram for displaying on the touch panel unit 7 as a display unit 5) The response data (character string data) is converted into voice data by the voice synthesis engine, and voice output from the speaker device 13 as an utterance of the robot 1 side. 6) a display mode/motion variation subprogram for varying the display mode on the touch panel unit 7 and the motion of the robot 1 based on the user side character string data and the robot 1 side character string data;
etc.
An example of the control contents of the controller MC mainly based on the dialogue program will be described below together with the mutual relationship between the standby mode (wakeup mode) after activation, the natural dialogue mode, and the sleep mode. These mutual relationships are as shown in FIG.
5 and 6 are screens in the natural dialogue mode, and FIGS. 7 and 8 are screens in the standby mode. In the sleep mode, these screens are darkened by turning off the backlight of the touch panel section 7 .

1. Activation The robot 1 is activated by turning on the power switch 15 (processing M0 in FIG. 13). A boot program is executed in the controller MC, and then when the OS is activated, the OS is in a state of waiting for a "command" from the user, that is, in a wake-up state. In this initial standby mode, the standby screen of FIG. 7 is displayed.
In the following, the state after the initial setting is completed, that is, the robot 1 ID and password are registered in the cloud server, user registration is completed, face authentication of multiple users is performed, and the email address of the smartphone is registered with the robot 1 It is assumed that it will be started later, such as being registered in
"1. Effect on Startup』
In this way, one screen (FIG. 7) selected from a plurality of standby screens is displayed first. In other words, a fixed screen is always displayed at startup. Since the eyes of the robot 1 are closed (implying that the robot 1 cannot interact), the user can easily understand that the robot 1 is in the standby mode.

2. Standby mode (wake-up mode)
The standby mode is a state in which a dialogue with the robot 1 is possible when a trigger for starting the natural dialogue mode is given. It is also a transition state when the dialogue ends in the natural dialogue mode. It is also a state in which the sleep mode is entered if the natural dialogue mode is not started for a certain period of time. The sleep mode is a state in which communication with the robot 1 is not possible.
Here, "natural dialogue" means that the user interacts with the voice of the device (robot 1) that is synthesized by using the built-in scenario of the robot 1 or the dialogue engine (dialogue software) on the server. Natural dialogue mode is a state in which natural dialogue is possible.
There are a plurality of standby mode screens, and two types of screens shown in FIGS. 7 and 8 are prepared in this embodiment.
FIG. 7 shows a standby screen that is shifted from the screen in the natural dialogue mode of FIG. 5 (process M2 of FIG. 13). It is also a standby screen displayed at startup by the process M0 of FIG. In FIG. 7, the layer screen of the face screen S1 with the eyes (eye objects 27) of the robot 1 closed is lightly displayed in the background of the clock layer screen on which the date and day of the week and the current time are displayed in a large size. .
FIG. 8 shows a standby screen that is shifted from the screen in the natural dialogue mode of FIG. 6 (process M2 of FIG. 13). In FIG. 8, the layer screen of the chat screen S2 is lightly displayed in the background of the clock layer in which the date and time, the day of the week, and the current time are displayed in a large size. That is, it is in standby mode but not in natural dialogue mode.
"2. Effect in standby mode (wake-up mode)
Since different standby screens are prepared in this way, when the natural dialogue mode is started from a certain standby screen, the user can interact with the screen accessed immediately before, which is convenient. By displaying a screen specific to the standby mode, the user can easily understand that the robot 1 is in the standby mode.

3. Starting and Stopping Natural Dialogue Mode From the standby mode or sleep mode after activation, the controller MC performs processing to shift to the natural dialogue mode at the following multiple timings, that is, mode transition triggers (FIG. 13). processing M1, M5). The following trigger is an example. In the natural dialogue mode, the mode is switched to the face recognition mode (facial recognition becomes possible) as described in "B. Details of operations during face recognition" below.

1-1) In the standby mode, when the controller MC recognizes an utterance (voice) such as "Hey, Yupibo" from the microphone 12 as an activation phrase within a certain period of time, it triggers the natural dialogue mode (FIG. 13). process M1). Also, when it is determined that there has been a touch operation on the touch panel section 7, this is used as a trigger to set the natural dialogue mode (process M1 in FIG. 13).
1-2) In the sleep mode, the controller MC determines whether or not there is a touch operation on the touch panel section 7 of the standby screen at a predetermined timing. The touch operation on the touch panel portion 7 differs depending on the display mode on the face portion 6. On the waiting screen of the face screen S1, touching the entire area of the touch panel portion 7 corresponds to touching the dialogue start button object 36, which will be described later, on the waiting screen of the chat screen S2. is started with That is, it is started by different operations on different screens.
When it is determined that there is a touch operation on the touch panel section 7, the controller MC temporarily switches to the standby mode (process M3 in FIG. 13), and then switches to the natural dialogue mode when it determines that there has been another touch (process in FIG. 13). M1).
1-3) In the sleep mode, the controller MC determines whether or not there is a touch operation on the touch panel section 7 as in 1-2). If it is determined that there has been a touch operation, the controller MC temporarily enters the standby mode (process M3 in FIG. 13). In this state, when an utterance (voice) such as "Hey, Yupibo" is recognized from the microphone 12 as an activation phrase within a certain period of time, the controller MC uses this as a trigger to set the natural dialogue mode (process M1 in FIG. 13). .
2) In the sleep mode, the controller MC determines whether or not a predetermined time set in advance by the RTC has come, and when the predetermined time has come, it switches to the natural dialogue mode (process M5 in FIG. 13).
3) In the sleep mode, the controller MC outputs random utterances (audio data) from the speaker device 13 at random time intervals, for example, using random numbers generated at predetermined timings. In other words, as a kind of soliloquy, the robot 1 is caused to output a voice that invites dialogue, such as "Hey, what are you doing?" Prompt the user to speak.
4) In the sleep mode, the controller MC determines whether or not the object (person) is moving by the Doppler sensor 22, and switches to the natural dialogue mode at the timing when the object (person) is detected to be moving (process M5 in FIG. 13). ).

5) In the sleep mode, when the controller MC senses a change in the weather such as an abnormal weather or an earthquake, it notifies the user of the change and uses this as a trigger to start a natural dialogue. The external cloud server uses an abnormal weather detection engine to acquire and store abnormal weather information including weather abnormalities (e.g., heavy snow, typhoons, etc.), earthquakes, lightning strikes, etc. at regular intervals based on a fixed standard. The fixed time may be the same timing for all, or may be acquired at different timings depending on the contents of the weather. In the first embodiment, information on abnormal weather is distributed from a server to a device (controller MC) by adopting a push-type distribution system, for example. When the controller MC acquires the information, it switches to the natural dialogue mode (process M5 in FIG. 13).
6) If the controller MC does not detect the user's utterance for a certain period of time while in the natural dialogue mode in the above 1) to 5), it enters the standby mode (process M2 in FIG. 13), and Sleep mode is set after a period of time (process M4 in FIG. 13). The length of these mode transition times can be appropriately set and changed, for example, by the terminal device or by speech as a built-in scenario.

"3. Effect on Starting and Stopping Natural Dialogue Mode”
By preparing various kinds of natural dialogue modes to be started in this way, dialogues with the robot 1 occur at various timings, increasing the chances of dialogue with the robot 1, which naturally increases the opportunities to enjoy the dialogue. As a result, the user will feel the merit of owning the robot 1 . In addition, when the interactive mode ends, the standby mode is set and then the sleep mode is set, so that the power cost can be reduced.
In addition, since the standby mode is skipped from the sleep mode and the screen of the natural dialogue mode is displayed, the dialogue can be started immediately, so that the dialogue can be started smoothly. In addition, since the dialog screens (FIGS. 5 and 7) are displayed as long as the dialog continues, the user will be motivated to have the dialog.

4. Dialogue of Built-in Scenarios in Natural Dialogue Mode In the natural dialogue mode, a plurality of dialogue processes of built-in scenario dialogues and normal dialogues using the dialogue engine of the server are prepared.
The controller MC first determines whether or not the utterance data (character string data) based on the user's utterance matches the built-in scenario, and if not, the dialogue using the dialogue engine via the cloud server (hereinafter referred to as normal dialogue). From the user's point of view, it seems that they are constantly interacting with the robot 1, but actually there are a plurality of internal processes in the natural interaction mode.
The controller MC executes a process of comparing the user's utterance (character string data) created by the voice recognition engine on the cloud server side with the text data of the built-in scenario (script). In this embodiment, the text data of the built-in scenario is stored in memory. A built-in scenario may be added to the SD card. If it is an SD card, it is easy to increase the number of built-in scenarios one after another by rewriting.
When the controller MC recognizes the user's utterance, it determines whether or not the character string data matches a predetermined regular expression or non-regular expression. , and output from the speaker device 13 as an utterance of Robo and 1.
Built-in scenarios include simple scenarios such as greetings like "Hello" and "It's nice weather today" to prompt the user to speak, and scenarios to ask for some action based on the user's utterance. Many built-in scenarios such as scenarios are set (prepared). Tables 1-3 disclose an example of such a built-in scenario. Of course, in reality, many built-in scenarios are set in addition to these built-in scenarios.

If the dialogue is not carried out according to the built-in scenario, the dialogue in the built-in scenario ends halfway. Examples of cases where dialogue is not performed according to the built-in scenario are as follows.
1) When the expected regular expression or non-regular expression is not matched This is the case where the built-in scenario does not match the regular expression or non-regular expression from the beginning or in the middle. In addition, it also includes the case where the user's articulation is poor and the utterance cannot be acquired correctly. In this case, the controller MC judges that it is a normal dialogue and immediately connects to the external cloud server. create customized response data. Then, the response data is converted into voice data by the voice synthesizing engine, and the voice is output from the speaker device 13 to continue the dialogue.

2) When the dialogue in the built-in scenario ends as planned It is conceivable that there is no dialogue because there is an affirmative utterance such as "please" and the dialogue in the built-in scenario ends as planned, or that there is no dialogue during the scenario. In this case, the standby mode is set after a certain period of time.
3) When the user's utterance is negative about whether or not to proceed with a certain process. , the built-in scenario ends even if there is a negative utterance such as "no" or "it was a mistake" instead of a positive utterance such as "yes" or "please", and the subsequent dialogue is 1 ) or 2).
At the time of this negative utterance, the controller MC confirms whether or not it is okay to stop the processing, such as "Are you sure?" This makes it possible to deal with misrepresentation, change of mind, etc. of the user. For example, in a scenario for turning off the power, when the user utters the question "Is it really okay to turn off the power?" Is it really necessary to turn off the power?" is repeated a plurality of times (for example, three times in the embodiment), and if "yes" is given, the dialogue in the built-in scenario ends.

"Effects of Built-in Scenarios"
With built-in scenarios like this, there is no need to request all interactions with an external server, and they can be processed inside the device, reducing the communication cost of connecting to the server, reducing the communication time and saving time on the server side. Since no calculation time is required, the user will not feel a sense of incongruity that the conversation will be cut off due to too late a reply to the user's utterance. Also, for example, when executing a predetermined process, by providing such a built-in scenario, the user does not need to operate the touch panel unit 7 or access the robot 1 from another terminal in order to execute the process. It is user-friendly because it can be executed interactively.

5. Requests and responses in normal dialogue On the other hand, if the speech data (character string data) is not a built-in scenario, the controller MC connects to the cloud server, sends the speech data (character string data) to the server again, and returns the response data from the dialogue engine. request the creation of If the user has been authenticated, the controller MC sends authentication information (for example, ID and password) for each user in the request to the beginning of the utterance data. In that case, past dialogue information is taken into account to create response data. On the other hand, if the user has not been authenticated, the user is regarded as a person who has not been identified in the past, and no particular authentication information is transmitted, so past conversation information is not taken into consideration. When there is a request, the cloud server makes the dialogue engine create response data (character string data) based on the utterance data (character string data) using the dialogue API (Application Programming Interface), and sends it to the robot 1 (controller MC). respond. If there is a history of user interaction in the past, response data is created that takes into account the content of that history. The controller MC converts this response data into voice data and outputs it from the speaker device 13 as an utterance on the robot 1 side.
Similar to the built-in scenario above, if there is no word from the user for a certain amount of time or more, it will enter standby mode.
"5. Effects on Requests and Responses in Normal Dialogue”
Unlike the built-in scenario, by connecting to an external cloud server and having a normal conversation, advanced dialogue analysis can be performed quickly with a much larger amount of data than the built-in scenario, making it possible to make it feel like you are actually interacting with a person. Advanced dialogue can be realized.

6. Actions of the Robot 1 During Dialogue The controller MC performs the following a. ~D. Performs control of actions such as
stomach. Gestures of the robot 1 in startup-natural dialogue mode-standby mode The controller MC controls the first to third motors 23 to 25 at the following various timings of the robot 1 to change the posture of the robot 1. The following is an example.
1) At start-up: When the face part 6 of the head 5 is not facing the front or when the head 5 is tilted, it is moved to the front default position.
2) When the screen is touched: Same as 1) (to face the face recognition camera 10 facing the user in face recognition)
3) When the trigger utterance "Hey Yupibo" occurs: Same as 1) (to face the face recognition camera 10 in face recognition facing the user)
4) When detecting the voice direction: turn the face part 6 of the head 5 in that direction.
5) As a special emotional utterance, for example, when happy: The third motor 25 is controlled to rotate the head 5 left and right (clockwise and counterclockwise) with the face 6 facing the user.
6) As a special utterance, for example, in the case of being sad: the head 5 is nodded and kept still for a while, and then the second motor 24 is controlled to return to the default position.
7) Emotion is an utterance, for example, when a greeting utterance such as "good morning", "hello", "good evening", and "hello" occurs: the second motor 24 is controlled so that the head 5 bows.
8) As non-special emotional utterances, when utterances of simple positive communication terms such as "Soka", "Understood", "Sure", "Yeah!" The second motor 24 is controlled as follows. In 6 to 8), the speed and time of the second motor 24 should be changed so that sadness, bowing, and nodding are different.
9) When utterances of simple negative communication terms such as "No", "I can't", "No" are generated as non-special emotional utterances: Rotate the movable part 3 left and right several times. The first motor 23 is controlled as follows.
10) In response to special emotional utterances and non-special emotional utterances and various dialogues, various gestures, such as turning the head 5 in left and right directions (clockwise and counterclockwise), forward and backward, In combination with this, the entire movable portion 3 may be rotated left and right, rotated largely, or rotated in a slightly nodding motion.
This gesture of the robot 1 may be combined with the display on the touch panel section 7 (face section 6) described below.
"6. Effect 1 on the actions of robot 1 during dialogue”
By causing the robot 1 to make such gestures, the user will feel familiarity with the robot 1, and will enjoy interacting with the robot 1 and at the same time will feel the enjoyment of actively interacting with the robot 1. - 特許庁

B. Changes in the display mode in the state of the face screen S1 1) When the user is speaking On the screen S1, an elliptical area 29 is displayed in blue, and an animation is displayed in which the area (that is, the size) of that area is changed according to the user's speech volume. Specifically, the controller MC expands the elliptical area 29 while maintaining the elliptical shape when the volume of the user's speech increases, and reduces the elliptical shape while maintaining the elliptical shape when the volume decreases. Also, the cheek object 28 is displayed in green.

Further, the controller MC causes the touch panel unit 7 to display the user's utterance data (character string data) responded from the cloud server in a predetermined manner.
For example, as shown in FIG. 5, when the touch panel unit 7 is the face screen S1, when an utterance such as "Hello" is obtained from the user, the controller MC displays "Hello" based on this utterance on the layer screen of the face screen S1. display the string. In terms of order, this display is started before the robot 1's utterance, which is a reply to the user's utterance.
As a display mode, for example, the face screen S1 is displayed so as to gradually become opaque from a transparent state as shown in FIGS. The face screen S1 is completely hidden. That is, the character string is gradually displayed. After stopping and displaying the state of FIG. 10(c) in which only the character string is displayed brightly against the dark background for a very short period of time, the layer screen displaying the character string is shown in reverse. 10(c)→FIG. 10(b)→FIG. 10(a), and returns to the default face screen S1. At this time, all the character strings in one utterance appear at the same time and disappear at the same time. This display mode is an example, and may be displayed in a different mode.
"6. Effect 2 on the actions of robot 1 during dialogue”
As a result, the user can visually see his or her own words on the robot 1, so that the user can check whether the robot 1 has heard correctly, judge whether the dialogue is correct, and prevent strange and irrelevant dialogue. It is possible to guide the dialogue so that it does not happen. In addition, irrelevant conversations tend to irritate people, but by confirming them, they can understand why, so they can change their way of speaking and try again.
In addition, since the user's utterance data, whether the object of the built-in scenario or the object of normal dialogue, is requested to the cloud server to be converted into character string data once, it does not take much time to process the character string data. Since the creation of response data by the dialogue engine is requested only after the column data, the display of the user's utterance on the touch panel section 7 can be performed at least before the utterance of the response data by the robot 1, and the order of dialogue can be changed. You can't go wrong.

When the user's utterance is used as a character string, the length varies depending on the utterance, so they are not the same. In addition, it may become quite long when it is a "sentence" with clauses instead of words. The controller MC adjusts the font size of the character string so that the contents of one utterance are all displayed on the touch panel unit 7 at the same time even if the utterance is such a long sentence. In other words, the shorter the utterance, the larger the font, and the longer the utterance, the smaller the font.
"6. Effect 3 on the actions of robot 1 during dialogue”
As a result, any utterance by the user can be confirmed with one eye, making it difficult for the user to interrupt the dialogue until the full sentence appears, making it difficult for the user to overlap with the next utterance. In addition, since one utterance appears at the same time, the entire sentence can be understood at once, and the user can fully understand even if the display time is short. In addition, since the character string is developed over the entire touch panel portion 7, each character can be displayed in a large size, making it easy for the user to see, and the user can sufficiently confirm even if the display time is very short.

2) When the robot 1 is speaking On the face screen S1 as shown in FIG. 5 of the touch panel unit 7, an elliptical area 29 is displayed in red, and the area (that is, the size) of that area is changed according to the speech volume of the robot. Animate display. Specifically, the controller MC expands the elliptical area 29 while maintaining its elliptical shape when the output level from the speaker device 13 increases, and reduces it while maintaining its elliptical shape when the volume decreases. Also, the cheek object 28 is displayed in light red.
"6. Effect 4 on the actions of robot 1 during dialogue”
In this way, the display mode on the face screen S1 changes according to the alternating dialogue between the user and the robot 1, and it is interesting that not only the actual dialogue but also the dialogue takes place alternately on the screen, and the conversation is lively. become.

c. Change in Display Mode of Chat Screen S2 The display mode of the chat screen S2 of the touch panel unit 7 accompanying interaction will be described with reference to FIGS. 6 and 8. FIG. As described above, the display changes from the face screen S1 to the chat screen S2 according to the user's operation.
First, the configuration of the chat screen S2 will be explained again.
As shown in FIG. 6, a user object 31 as an avatar character is displayed on the lower left side of the chat screen S2, and a robot object 32 is displayed on the lower right side so as to face each other. A different user object 31 is prepared for each authenticated user recognized in a face recognition mode, which will be described later, or for an unauthenticated user, and different objects are displayed according to the currently interacting user. Balloon objects 33 in which the contents of the dialogue between the user side and the robot 1 side are converted into character strings and arranged are displayed in order along the time axis in the central region. A dialogue stop button object 34 is displayed in the upper left position of the chat screen S2. A setting button object 35 is displayed on the upper right side of the chat screen S2.

The chat screen S2 is displayed such that balloon objects 33 are added moment by moment according to the dialogue between the user side and the robot 1 side. In the balloon object 33, the character string data of the user's utterance content and the character string data of the robot 1 utterance content are displayed in a row along the time axis and can be displayed on the chat screen S2. The content of the most recent speech is displayed in the new speech balloon object 33 at the bottom of the row of past speech balloon objects 33 in synchronism with the speech.
The speech balloon object 33 indicates the speech direction so that the content of speech on the user side or the content of speech on the robot 1 side can be identified. Since all dialogue histories cannot be displayed on the screen at once, the chat screen S2 has a vertically scrollable screen configuration, and the balloon object 31 can be displayed retroactively. When not going back in time, the balloon object 33 of the most recent dialogue is always displayed.
In the first embodiment, after the dialogue is temporarily terminated and the standby mode is set, the dialogue is resumed, and at that time, if the user recognized again in the face recognition mode described later is changed, the balloon object 33 column "Change user" is displayed in the middle of , and a different user object 31 is displayed according to the user whose user object 31 is recognized again.

Further, by touching the dialogue stop button object 34 on the chat screen S2, the dialogue is actively interrupted by the user, and the standby mode is entered. In this case, the standby screen of the chat screen S2 of FIG. 8 is displayed instead of FIG. When switching to the natural dialogue mode again, by touching the dialogue start button object 36, it is possible to return to the chat screen S2 of FIG.
"6. Effect 5 on the actions of robot 1 during dialogue”
In this manner, the user object 31 of the user and the robot object 32 of the robot 1, which are in a relationship of dialogue, are arranged so as to face each other, and the conversation balloon objects 33 are lined up between them, giving the feeling of having a conversation. It can be received from the screen S2.
In addition, past chat history can be checked later, so chat can be used instead of a diary. In addition, since it is possible to know who had what type of conversation, it is possible to check data such as which family member frequently uses the system. Since "user change" is displayed, it can be understood that the dialogue has been interrupted, and there is no confusion when reading the past history.

D. Special Actions in Regular Dialogues In regular conversations, when the cloud server determines that a specific word is included in the user's utterance data, it responds with a command to execute a special gesture along with character string data. The controller MC causes the following specific actions, for example, to be executed based on the above screen display program, gesture program, and dialogue program according to the command. The following control is an example, and it may be controlled to perform other actions, or if there are multiple commands during the user's speech, the actions may be performed continuously or simultaneously. Each of the following special actions may be performed separately or in combination. Regarding the behavior of the robot 1 during dialogue, b. Instead of the gestures of the robot 1 in , the following display units may be displayed, or the following display units may be combined as appropriate.

1) When the user utters a negative expression in a normal conversation between people, the display on the touch panel unit 7 is changed from the normal eye in FIG. 9A to FIG. ) to animate the angry eyes object. In this embodiment, the eye object is stored in memory.
2) When the user utters an expression that makes normal conversation between people enjoyable, the display on the touch panel unit 7 is changed from the normal eyes of FIG. 9A to FIG. Animate the smiling eye object of c) to change. In this embodiment, the eye object is stored in memory.
3) When the user utters an expression that makes him sad in a normal conversation between people, the display on the touch panel unit 7 is changed from the normal eyes of FIG. d) Animate the sad eye object. In this embodiment, the eye object is stored in memory.
4) When the name of the user's child is uttered by the user, the second motor 24 is controlled so that the head 5 of the robot 1 makes a nodding gesture. In this embodiment, a program for eye gestures is stored in memory.
5) When the user speaks the name of the company that manufactures the robot 1, the head 5 of the robot 1 moves back and forth, left and right, and at the same time the body 4 repeatedly swings with respect to the fixed part 2. The first to third motors 23 to 25 are controlled so as to perform such gesture motions. At the same time, text data of the company name is voice-synthesized, and the company name is repeatedly called from the speaker device 13 as voice. In this embodiment, the program for gestures is stored in the memory.
"6. Effect 6 on the behavior of robot 1 during dialogue”
By performing such special actions, the user can anticipate unexpected actions of the robot 1 at the same time as the dialogue, and can positively enjoy the dialogue with the robot 1 .

B. About the operation contents at the time of face recognition 1. Starting and Stopping Face Recognition Mode The controller MC recognizes and authenticates the user's face by executing a face recognition program. The face recognition program recognizes the acquired image as a human face by recognizing face patterns, and by digitizing and storing the various positions of the recognized face, it matches the numerical data of the face registered in the past. Authentication is performed by judging the degree. The controller MC sets the face recognition mode in synchronization with the natural dialogue mode, and executes face recognition each time the standby mode is switched to the natural dialogue mode.

In the face recognition mode, the controller MC uses the face recognition camera 10 to recognize the user's face. In particular,
1) Activate the face recognition camera 10 . The user's face image captured by the face recognition camera 10 is displayed on the touch panel section 7 (face display mode). That is, the user is urged to look at his/her own face on the touch panel section 7 . This enables face recognition processing, and by previewing in this way, it is possible to determine whether the person has been authenticated in the past.
2) If the face recognition camera 10 cannot photograph the face in 1) and the face cannot be recognized within a certain period of time, the third motor 25 is driven to swing the head 5 up and down. That is, the face recognition camera 10 is caused to scan in the vertical direction. While scanning the face recognition camera 10 in the vertical direction, the first motor 23 is driven to rotate the face recognition camera 10 by 360 degrees to perform face recognition operation.
3) Perform authentication when the face is recognized in 1) or 2). If the user is already registered, the natural dialogue mode is set using the data of the specific authenticated user in the dialogue. If the user is not registered, the user is recognized as an unspecified person, and the natural dialogue mode is set. The touch panel unit 7 returns from the face display mode to either the previous face screen S1 (FIG. 5) or the chat screen S2 (FIG. 6).
4) If the face cannot be recognized in 2), the face recognition mode and the natural dialogue mode itself are ended as the person cannot be recognized, and the standby mode is set. The touch panel unit 7 returns from the face display mode to either the standby screen of the face screen S1 (FIG. 7) or the standby screen of the chat screen S2 (FIG. 8), which is the standby mode immediately before.
"1. Effect on Starting and Stopping Face Recognition Mode”
In dialogue, it is basic to talk while looking at the face of the other party, so if the face cannot be recognized, the dialogue is not allowed. You can't interact without face-to-face, so users can get the feeling that they are actually interacting.

2. Actions of the Robot 1 During Face Recognition 1) After face recognition, the controller MC causes the face recognition camera 10 to acquire an image and always executes face pattern recognition at a constant timing. Then, the user's face is recognized within the angle of view of the face recognition camera 10, and the default state is the center position C of the two eyes of the user's face at a predetermined position within the angle of view, for example, the origin at the center of the angle of view. position. When the center position C deviates from the default position, the controller MC displays an animation such that the pupil object 27a moves in either the left or right direction according to the amount of deviation. Normally, the pupil object 27a generally appears as an ellipse in the white of the eye in the eye object 27 as shown in FIG. 9(a). Thus, the pupil object 27a is displayed as the partially hidden eye object 27 as if looking in that direction.
When the user moves and the face is out of the angle of view of the face recognition camera 10 and the face cannot be recognized, the controller MC drives the first motor 23 to move the movable part 3 in the direction in which the center position C is displaced. Control is performed so that the whole is rotated and the face recognition camera 10 is directed. The driving of the first motor 23 is stopped when the face is recognized. When the face is not recognized even after a certain amount of rotation, for example, by rotating the entire movable part 3 by 45 degrees, the controller MC stops the driving of the first motor 23 at that stage, and face pattern recognition is always performed in that state. to continue.
"2. Effect on behavior of robot 1 during face recognition”
As a result, the user feels as if he or she is having a conversation while being watched by the robot 1 all the time, which makes the conversation more interesting.

2) The controller MC always performs face pattern recognition with the face recognition camera 10, but when the user is moving or out of the angle of view of the face recognition camera 10, face recognition cannot be performed. Therefore, it is preferable to inform the user of the face recognition state as a change on the screen. In Embodiment 1, the state of face recognition is notified by a change in the color depth of a sickle-shaped reflection object 27b expressing reflection on the pupil inside the pupil object 27a. In the first embodiment, the controller MC displays the face in very light blue when the face is not recognized, in darker blue when the face is being recognized, and in dark blue when the face is normally recognized.
"2. Effect 2 on behavior of robot 1 during face recognition”
As a result, the user can easily know whether or not the robot 1 recognizes the face, so that the user actively cooperates with the robot 1 in recognizing the face, and the conversation progresses smoothly.

C. Operation when setting to be absent In the first embodiment, images can be transferred to an e-mail registered in an answering setting mode, that is, when setting to be absent. In the first embodiment, the absence setting mode is set and canceled on the setting screen displayed after the user touches the setting button object 35 on the chat screen S2 of the robot 1. FIG. In the following, processing based on the absence setting program of the controller MC when the absence setting mode is set will be described.
The controller MC performs B. in the above "2. Start and stop of the natural dialogue mode" in the absence setting mode. In the waiting mode of 4), when the Doppler sensor 22 determines that an object (person) is moving, the following control is performed.

1) The controller MC notifies the user of the presence of some moving object around the robot 1 by e-mail. The controller MC writes the URL of the cloud server of the robot 1 in an e-mail to a terminal device, for example a smartphone, of a user who is not near the robot 1 (hereafter referred to as an external user) whose e-mail address is registered via the Internet line. and send. The subject of the e-mail and the sent text should include an expression that indicates the intention of this notification. For example, it is a sentence such as "Someone seems to be coming" or an icon representing it.
"C. About the operation when you are away, the effect of
As a result, the external user is notified of the state in which an object (person) is moving around the robot 1 by first receiving the e-mail, and can recognize the state, and the external user can take countermeasures against this state. You will be given the opportunity to take

2) The controller MC sends an e-mail to the external user and at the same time activates the face recognition camera 10 to acquire an image.
3) The controller MC sends an e-mail to the external user and at the same time determines whether there is a certain triggering utterance within a certain period of time. For example, it is a greeting utterance such as "I'm home, Yupi-bo." Based on this utterance, a built-in scenario in the natural dialogue mode is started, prompting the user (here, a person near the robot 1 who uttered "I'm home, Yupibo") to perform face recognition. When the controller MC determines that the user is one of the registered users as a result of face authentication, it sends a second e-mail to the terminal device of the external user. This e-mail serves as information for the external user that the person around the robot 1 is not a suspicious person. In other words, the second e-mail notifies that the person is a related person, such as a family member. In the e-mail, the name of the user registered based on the registration information is written as information in the subject or in the text to be sent. It should be noted that the registered user may be confirmed by, for example, touching the touch panel portion 7 to recognize the face as a trigger other than the utterance.
"C. Effect 2 on the behavior when set to be away』
As a result, when a family member such as a child returns while the user is away, the second e-mail informs the user of the fact, so there is no need to check the state of the house during the absence via the smartphone.

4) The external user who received the e-mail in 1) enters the ID and password of the robot 1 to connect to the cloud server, especially when the second e-mail is not sent in 3), and accesses the smartphone. The camera image of the face recognition camera 10 provided by the cloud server can be viewed in real time on the browser. It is possible even after sending the e-mail a second time.
FIG. 12 shows an example of a user's smart phone 41. After connecting to the cloud server, a predetermined camera image of the face recognition camera 10 is displayed on its display screen 43, which also serves as a touch panel. Four operation icons 44a to 44d for remotely controlling the direction of the face recognition camera 10 are displayed in the camera image. By operating the operation icons 44a to 44d by the external user, a control command is output to the controller MC via the cloud server. The direction of the face recognition camera 10 can be changed by rotating the head 5 and the body 4 of the face. Further, by touching the record button icon 45, recording can be started, and by touching it again, recording can be stopped.
Voice data spoken to a microphone (not shown) of the smartphone 41 is output from the speaker device 13 of the robot 1 via the cloud server, while voice data spoken from the microphone 12 of the robot 1 is sent to the smartphone 41 via the cloud server. is output from a speaker device (not shown). Therefore, the external user can interact with a user near the robot 1 using the smartphone 41 and the robot 1 while viewing the image of the face recognition camera 10 .
"C. Effect 3 on the operation when set to be away』
As a result, the external user can change the direction of the face recognition camera 10 by remote control and check the surroundings of the robot 1, for example, to check the safety situation at home when the user is away. In addition, even when a family member such as a child returns while the user is away, positive contact from the outside using a smartphone in this way contributes to a good relationship with others including the family member.

<Modification 1 of Embodiment 1>
Next, Modification 1 of Embodiment 1 will be described.
In the dialogue of the built-in scenario in the above natural dialogue, if the speech data obtained from the microphone 12 does not match the regular expression or non-regular expression of the built-in scenario because the user's speech is poor or other sounds are mixed, the controller The MC may output an utterance such as "Please say it again" as an utterance for prompting the user to repeat the utterance without immediately judging that it is a normal dialogue.
The controller MC makes such prompting utterances from the speaker device 13, and if the user utters correct utterances according to the built-in scenario within a certain period of time, the conversation is again processed as a built-in scenario dialogue. to On the other hand, even in such a case, if the user's utterance does not match the built-in scenario with a regular expression or a non-regular expression, it will be connected to an external cloud server.
By doing so, it is possible to have a dialogue only inside the robot 1 without connecting to an external cloud server in vain.

<Modification 2 of Embodiment 1>
Next, Modification 2 of Embodiment 1 will be described.
If the user fails to hear the words (utterances) of the robot 1 in the natural dialogue, the user utters a request utterance that repeats the previous utterance of the robot 1 within a certain period of time. You may make it speak (audio output). This processing can be done in either built-in scenario dialogue or natural dialogue.
The controller MC recognizes whether or not there is an utterance that triggers missed hearing, for example, utterance "Tell me again", during the dialogue mode. When it is determined that the user has requested to repeat the utterance after the utterance from the robot 1, the content uttered by the robot 1 immediately before is output as voice again. Then, the previous utterance is canceled, and the second utterance is processed as the first utterance.
In this way, even if the user misses listening during the dialogue, the dialogue can be restarted without interruption.

<Modification 3 of Embodiment 1>
Next, Modification 3 of Embodiment 1 will be described.
In the above natural dialogue, if the user confirms the user's utterance displayed on the touch panel unit 7 by the robot 1 and finds that the voice was recognized incorrectly, it is pointed out within a certain period of time. It may be possible to modify the correct dialogue. This processing can be done in either built-in scenario dialogue or natural dialogue.
During the interactive mode, the controller MC receives an utterance that triggers an indication that the speech recognition is incorrect from the user, such as an utterance such as "You're wrong" or "You're wrong. Say it again." Recognize whether or not there was a voice. Then, when the controller MC judges that the utterance is within a certain period of time after displaying the user's utterance on the touch panel section 7, the controller MC outputs voice prompting the user to utter again. For example, the content of the utterance "I'm sorry. Say it again" is output as voice.
and,
(1) In the case of dialogue in the built-in scenario, the user's previous utterance is canceled, and the content of the user's utterance again is recognized as the correct utterance.
(2) In normal conversations, the conversational content such as "I made a mistake" or "I'll say it again" is sent to the external cloud server as speech data (voice data), and the user speaks again including such speech. Request the creation of response data in the content.
In this way, even if the robot 1 erroneously recognizes the user's speech, it can correct the conversation to a correct one.

<Modification 4 of Embodiment 1>
For example, the following configuration different from the configuration of the first embodiment may be employed.
(1) In the above, the settings were such that if the built-in scenario was not executed, a request would be made to the cloud server and the normal dialogue would be started. In other words, if a built-in scenario is executed, it was set to be a built-in scenario. may The certain condition may be, for example, execution once every several times or at random timing.
As a result, an unpredictable dialogue with the robot 1 can be carried out, and an unstructured and more human dialogue can be achieved.
(2) In Embodiment 1, the controller MC does not have a speech recognition engine, but connects to an external server having a speech recognition engine to convert the user's utterance (speech data) into text. The burden on the robot 1 is thereby reduced.
However, the controller MC is provided with a speech recognition engine in its memory, and calls the speech recognition engine to convert the user's utterance data (speech data) acquired from the microphone 12 into text by itself using the speech recognition engine. Column data may be created. That is, the controller MC of the robot 1 may itself have the ability to convert the user's voice data into text data. As a result, the character string data can be created without using the speech recognition engine, thereby shortening the internal processing time, for example.
(3) In the first embodiment, initial settings are made from the setting screen, but a terminal device such as a smart phone may be used to register from the outside via a cloud server. This makes setting easier and saves time, especially for those who are accustomed to using terminal devices.
(4) The first to third motors 23 to 25 may use drive means other than servo motors. Other drive means are, for example, other types of motors, hydraulic cylinders, and the like.
(5) In the first embodiment, the character string data is voice-synthesized inside the robot 1 . In this way, using the internal speech synthesis engine does not make the data too heavy compared to exchanging voice data with the server as it is, but the response data (character string data) obtained using the dialogue engine on the cloud server side may be synthesized and the resulting voice data may be sent to the robot 1 as a response.
(6) In the first embodiment, the setting screen is shifted from the setting button object 35 of the chat screen S2 to the setting screen. .

(7) In "C. Special actions in normal dialogue", even in the built-in scenario, if it is determined that a specific word is included in the user's utterance data, a special action may be executed. good. For example, when the controller MC determines that a specific word is included in the user's utterance data, it may be controlled to perform a special action in the same manner as described above.
(8) If the shape of the robot 1 is different in "C. Special gestures in normal dialogue", the robot 1 may be controlled to make further different gestures. For example, if the robot 1 has arms and legs, the controller MC may control the drive means to move them.
(9) As the animation of the eye object 27 on the face screen S1, an animation such as blinking may be inserted from time to time. For example, it is executed by performing control such that the eye object 27 in the closed state in FIG. 7 is inserted from the state of the eye object 27 in FIG. 9A. By doing so, a realistic feeling is created as if the robot 1 is actually looking at you, so that the dialogue with the robot 1 can be enjoyed more.
(10) In face recognition mode, an unregistered user was recognized as an unspecified person. , convenient.
(11) In "C. Operation when setting to be away", it would be convenient if the setting mode can be set from the smartphone.
(12) In "C. Operation when set to be away", the controller MC performs processing such as notifying the user of this state by e-mail when some kind of moving object exists around the robot 1. Conversely, it is also possible to provide a process of recognizing that the object is moving at regular intervals and notifying the user of this state by e-mail when there is no moving object within a certain period of time.
For example, when there is a sick person or a person to be cared for, the robot 1 can be placed near the person to watch over the person assuming that there is always movement.

<Embodiment 2>
Next, Embodiment 2 will be described.
In place of the high-intensity white LED 9 of the robot 1 of the first embodiment, or in addition to it, an infrared LED may be provided. At this time, if the module of the face recognition camera 10 is provided with an infrared filter, it is removed. Since the infrared LEDs are invisible to humans, if there is an intruder such as a thief at night, there is a possibility that the intruder will be frightened by the lighting of the high-intensity white LED 9 and escape. On the other hand, if it is an infrared LED, since it is difficult to recognize that the image is being captured, the intruder will not run away, so that the image of the intrusion can be confirmed and saved.

<Modification 1 of Embodiment 2>
Next, Modification 1 of Embodiment 2 will be described.
When the robot 1 is equipped with an infrared LED, the infrared LED may be used to control various indoor devices equipped with an infrared remote control signal receiving section. Examples of various devices include a television, an audio device, an air conditioner, and the like.
It is preferable that the robot 1 is installed in a place where it can directly see the infrared remote control signal receiving section of a device equipped with an infrared remote control signal receiving section. In Modified Example 1 of Embodiment 2, the controller MC has a shape recognition program for shape recognition using the face recognition camera 10. For example, in the case of a television, based on its shape features (square, large, black, etc.) can be recognized. When the robot 1 utters an utterance such as "Turn on the TV" as a trigger for outputting an infrared remote control signal to various devices of the user, the first motor 23 or the third motor 23 is activated based on the utterance. 25, the robot 1 rotates the face recognition camera 10 vertically by 360 degrees while swinging the face 5 up and down to photograph the surroundings, and the shape of the television is recognized by the shape recognition program.
When the controller MC determines that there is a television, it stores the direction of the television and simultaneously outputs an infrared remote control signal from the infrared LED in the direction of the object to execute ON/OFF control for operating the television. When there is a trigger for the next television, shape recognition is first performed in that direction. Infrared remote control signals can be used not only for ON/OFF switching control, but also by changing the infrared frequency so as to change the channel of a television, for example, to adjust the temperature of an air conditioner. Become. For such detailed control, infrared remote control signals with different frequencies are required. For example, in FIG.
Further, even if the direction is not searched by the shape recognition program, the direction of various devices may be obtained by operating the face recognition camera 10 via the smartphone 41 to change its direction, and the direction may be registered.

<Embodiment 3>
Next, Embodiment 3 will be described.
In the third embodiment, a description will be given of control aimed mainly at reducing running costs associated with connection, such as using a dialogue API, when using a server equipped with a speech recognition engine.
Further, the interactive program of the third embodiment includes a subprogram capable of detecting the silent state of the voice acquired from the microphone 12. FIG. The dialogue program also includes a recording/output subprogram for obtaining voice data of the user's speech from the microphone 12, temporarily recording it, and outputting it to the server.
The controller MC does not immediately connect to the server when there is an utterance, but first records the voice data of the user's utterance and connects to the server for the first time at the stage when the silence state of the recorded voice data of the user's utterance is detected. The recorded voice data is output by the dialogue engine, and response data is created by the dialogue engine. In this way, it is not necessary to connect to the server all the time, and it is not necessary to connect to the server for a long time including silent time, so the silent connection time can be cut.
The speech recognition engine will be dedicated to a user during a user's speech. In other words, a very long computer time of "how many seconds" is occupied for one process, and as a result, the cost burden on the user who uses the speech recognition engine increases. By doing so, the number of processes required per unit user can be reduced, contributing to cost reduction for the user.

<Modification 1 of Embodiment 3>
Next, Modification 1 of Embodiment 3 will be described.
Also in Modification 1 of Embodiment 3, control aimed mainly at reducing the running cost of connection when using a server equipped with a speech recognition engine will be described. The speech recognition server consumes processes if there is a time lag before the user starts speaking, or if the user does not speak while waiting for the user to speak and the connection with the server is terminated due to a timeout. As a result, it costs the user.
The dialogue program of the robot 1 of Modification 1 of Embodiment 3 includes a recording/output subprogram for acquiring speech data from the microphone 12, temporarily recording it, and outputting it to the server. The interactive program also includes a subprogram for detecting and outputting the sound pressure level of recorded voice data.
The controller MC connects to the server when the sound pressure of the voice data constantly recorded in the state of natural conversation is equal to or higher than a certain sound pressure, and plays back the data being recorded. In other words, ignore silence or small utterances that cannot be recognized, connect to the server only when there is an utterance that can be interacted with, output voice data, and let the server side create response data with the voice recognition engine. to
This makes it possible to eliminate useless connection time waiting for speech.

<Embodiment 4>
Next, Embodiment 4 will be described.
Speech recognition servers are expensive, so it may not be possible to pre-provision sufficient resources, and if there are not enough server resources, the server may be busy when the terminal tries to connect to the speech recognition server. There is
In the above-mentioned normal dialogue, utterance data (voice data) is sent to the server to request the creation of response data. and the server creates response data converted into character string data based on the utterance data and responds. However, if it is busy, the response data may not be sent, resulting in an error. An error message will be returned from the server.
When receiving an error message from the server, the controller MC of the robot 1 does not utter an utterance to the effect that there is a server connection error to the user, but outputs a reply by voice so that the dialogue can be continued from the built-in scenario. For example, it is better to wait for the server to become available by giving vague responses such as "Tell me again", "Yeah, yeah" or "What is it?"

<Modification 1 of Embodiment 4>
Next, Modification 1 of Embodiment 4 will be described.
If the user's utterance is too long, the speech recognition engine may misinterpret it. Therefore, when it is determined that the recognized user's utterance is too long, it is possible to provide a function of outputting a dialogue example selected from the voice data stored in the storage means without connecting to the server equipped with the voice recognition engine. good.
When the controller MC of robot 1 determines that the length of the user's utterance data is longer than a certain length, it does not connect to the server and responds to the built-in scenario by saying "Yes", "Seriously?", "Are you sure?" It outputs utterances such as back-and-forth that can be taken in any way in a dialogue. The utterance data may be voice data as it is, or may be data after being converted into character string data by the controller MC or server.
As a result, it is possible to prevent irrelevant words from being returned, reorganize the dialogue, and prompt the user to engage in dialogue again.

<Embodiment 5>
Next, Embodiment 5 will be described.
Embodiment 5 describes a case where a plurality of speech recognition engines are used in combination.
The voice recognition engine has a local voice recognition engine (that is, when processing within the device without connecting to the net server) and a cloud voice recognition engine that responds to the dialogue data created by request by connecting to the net server. be. There are multiple types of speech recognition engines, both local and cloud, some free and some paid. Therefore, when using servers with these different speech recognition engines, free servers and paid servers are mixed and used.
In the cloud server to which the robot 1 is connected using the Internet line, in interactive mode, when the robot 1 issues a request for utterance data and the voice recognition engine creates response data, the cloud server responds as follows, for example. It is better to
(1) Access to the paid dialogue API until a certain time A set per month.
(2) Between a set time A and a set time B per month, use a mixture of a paid API and a free speech recognition engine. For example, for the first few consecutive recognitions, a paid speech recognition engine server is used, and for subsequent consecutive recognitions, a free speech recognition engine server is used.
(3) Use only the free speech recognition engine when a certain time B set per month is exceeded. This is just an example, and it may be set so that only the free speech recognition engine is immediately used when a certain time A set per month is exceeded, for example.
In this way, it is possible to have a dialogue without greatly exceeding the range of charge. A cloud server connected to the robot 1 processes the issuance of a request from the robot 1 by calculating charge and free based on a set time per month based on a program for executing such processing.
The robot 1 itself executes such processing, and when issuing a request, instructs the cloud server whether to use the paid dialogue API or to use the speech recognition engine of the free server. good too.
<Embodiment 5-1>
The same applies to the case of combining a plurality of speech recognition engines according to the fifth embodiment, and the case of combining a plurality of dialogue engines.

<Embodiment 6>
Next, Embodiment 6 will be described.
If only one fixed dialogue engine is used, there is a possibility that the user will get bored with the conversation with the robot 1 because the responses will follow a fixed pattern. Therefore, in the sixth embodiment, in order to solve this problem, a process for arranging the output results of a plurality of dialogue engines so as not to get bored with the conversation will be described.
The dialog engine may use not only a cloud dialog engine but also a local dialog engine within the robot 1 .
This processing may be performed on the side of the robot 1 to which a plurality of response data has been sent, or may be performed on the side of the cloud server when response data is obtained from a number of servers equipped with dialogue engines.
(1) Output the result of the engine that has responded with the longest character string among the chat dialogue engines.
The longest response creates a dialogue-like feel, eliminates the monotony of the dialogue, and allows the user to enjoy the dialogue.
A. For example, when the user utters "I'm hungry", the answers from three engines a to c are "a engine: do you often snack between meals?" c-engine: eat something", adopt a-engine and output its response data.
B. For example, when the user says, "Today's weather is fine," the responses from three engines a to c are "a engine: I'll go to the sky right now to check" and "b engine: it looks like it's going to be sunny." ?” and “c-engine: the mood of the day can be determined by whether it is sunny or rainy.”, the c-engine is employed to output the response data.

(2) Out of the conversational dialogue engines, those with "?" at the end of words are brought to the end and output. At this time, if there are multiple answers with "?", they are output consecutively.
Adding a question mark at the end of a word makes the flow of the story more answerable to that question, making it easier to continue the conversation and allowing the user to enjoy the dialogue.
For example, the above (1) A. In the option of , "Do you often snack between meals? Do you not eat rice?" In addition, the above (1)B. If it is the option of , it outputs "is it sunny?".
(3) After combining affirmative sentences, combine and output interrogative sentences.
By arranging it in this way, the response becomes as if the user has carefully thought out the sentences. Conversation is easier to continue and users can enjoy dialogue. In addition, it is possible to earn an output scale and to request a reply from the person.
For example, the above (1)B. When response data such as "I'm going to go to the sky right now and check it. Sunny or rainy can determine the mood of the day. Does it look like a sunny day?" is output.

(4) Bringing and outputting results from engines that change topics more frequently than others, later than results from other engines.
By arranging in this way, the topic is changed, so that the dialogue invites the next utterance, making it easier to continue the dialogue.
A. For example, when the user utters "domo domo", the answers from the three engines a to c are "a engine: yes~""b engine: yes""c engine: I watch baseball Do you want to watch baseball?”, the c-engine data is put at the end and the output is “Hey, I see. Do you watch baseball?”
B. For example, when the user utters, "I can't find it easily," the answers from the three engines a to c are "a engine: that's right,""b engine: yes," and "c engine: your family." How many people are in your family?", the c-engine data is put at the end, and "That's right. How many people do you have in your family?"
(5) Output the engine that gives a more friendly reply than the others first, then output the replies from the other engines together.
Arranging in this way makes it easier for the user to be drawn into the dialogue, making it easier for the dialogue to continue. Whether it is friendly or not can be determined in which position to place by giving relative ranking to words (words).
A. For example, the above (4)B. In the case of , the result of the b engine is put first and the output is "Yes, that's right. How many people are in your family?"
B. For example, when the user utters "I see," and the answers from two engines a and b are "engine a: Oh, that's an appropriate backtracking," and "engine b: um." Put the data at the end and output "Hohoho Oh, that's a suitable backhand."

(6) Arbitrarily combining the processing of (1) to (5) This will increase the variation of dialogue, so that even if the user makes the same utterance, the exact same response will not be returned, and the dialogue It makes it easier to continue the dialogue without getting tired of it.
For example, when a user utters "What is old age?", the answers from three engines a to c are "a engine: please wait a moment", "b engine: I don't hate support", "c engine : Are you healthy now?", the most friendly b-engine is first, combined with affirmative sentences, then combined with interrogative sentences, "I don't hate support, please wait a minute. I'm healthy now." Are you here?" is output.

(7) In the text output engine, parentheses and emoticons may be returned, so when these are returned, speech output is suppressed and output. and display them on the screen.
Emoticons cannot be output as voice, but by intentionally displaying emoticons on the display unit and making them part of the dialogue together with the voice, it is possible to create an unusually interesting dialogue.
A. For example, when the user utters "I'm hungry", the responses from the three engines a to c are "a engine: (ignoring on purpose)", "b engine: hello how are you", and "c engine: hello". , only the a engine does not output the voice, but displays it on the touch panel unit 7 (display screen).
B. For example, when the user utters, "It's already included," the answers from the three engines a to c are "A engine: Do you often do it?" B engine: (´・ω・｀) c Engine: Are you a night owl?", only b is displayed on the touch panel unit 7 (display screen) without being output as a voice.
(8) Output one of the responses containing the same character string.
This is because if the same character string is repeated, the dialogue becomes verbose and the listener feels uncomfortable.
For example, when the user utters “Chinese food”, the answers from the three engines a to c are “a engine: Oh my goodness”, “b engine: Yes, it’s Chinese food”, and “c engine: eat Chinese food”. Are you going to go to?", only one of them is output because there is a character string of "Chinese" in b engine and c engine. For example, output like "Oh, that's nice. Are you going to eat Chinese food?"

(9) Using ending conversion means, such as ending conversion API, to give a sense of unity. At this time, the endings may be changed for all the responses, but the endings may be changed for only one of the last output sentence and the first output sentence.
It is good because it becomes a more friendly expression compared to the sentences of a normal dialogue engine.
For example, when there is response data "sleepy" from a certain engine, the ending of the word is converted by the endings conversion API and output as "sleepy nya".
(10) In preparation for recognition failure, only some of the responses obtained from a plurality of engines are used for speech output, and the remaining responses are retained, and when the next speech recognition fails or the dialogue system When there is no reply, the held reply is returned.
By using it when speech recognition fails or when it is difficult to receive a response from an external server, dialogue can be continued without interruption, contributing to natural dialogue.

<Embodiment 7>
As shown in FIG. 14, the seventh embodiment is an apparatus (system) in which a smart speaker 51 is arranged near the robot 1 and the robot 1 and the smart speaker 51 are combined. The distance between the robot 1 and the smart speaker 51 is such that the sound can be picked up by each other's microphone, and it is preferable that they are adjacently arranged within 1 to 2 m, for example.
The smart speaker 51 is a network terminal having a built-in wireless LAN device, a wireless communication function using the Internet, a telephone line connection function, and a network module, and is also a kind of computer equipped with a microphone and a speaker device. . The smart speaker 51 uses a terminal device such as a smartphone to perform various initial registrations (for example, user name, address, phone number, email address registration, multiple voice registration, Bluetooth network compatible AI device settings, etc.), connects to the Internet according to the utterances (commands) from the registered user, executes predetermined processing using the server's search engine, and outputs the result as voice information from the speaker device. Output.

The robot 1 can complement each other's functions by cooperating with the smart speaker 51 . Specifically, the robot 1 and the smart speaker 51 perform the following functions using voice as an interface.
(1) Instruction function from the robot 1 to the smart speaker 51 a. For example, when the user speaks a phrase that activates a smart speaker skill, the robot memorizes the character string of the voice recognition result of that phrase, and the robot itself speaks the character string by speech synthesis at a predetermined timing. to
In Embodiment 7, the controller MC of the robot 1 has a voice identification program that analyzes the frequency components of the user's utterances to identify individual voices, and acquires the user's utterances by personal phone 12 and uses character A recording/playback program is provided for a phrase that is stored as string data, synthesized into voice based on the character string data, and played back from the speaker device 13 .
The robot 1 has a function of memorizing an utterance that serves as a trigger for memorizing an utterance, for example, words after an utterance such as "I'm going to speak now, so record it." The user can then use this feature to have the smart speaker 51 remember words that wake it up or make it do something. For example, "OK, XXX. Turn on the lights." At this time, the user's personal voice registered in the robot 1 is the user's personal voice registered in the smart speaker 51 .
Then, the robot 1 is made to speak at a predetermined timing. The setting to speak at a predetermined timing can be set and registered by operating a terminal such as a smartphone, for example.
B. The “speech at a predetermined timing” of the robot 1 is, for example, the robot 1 detecting some kind of change, such as a touch operation on the touch panel section 7, detection of an object (person) by the Doppler sensor 22, or the like.
For example, when the Doppler sensor 22 detects a person, the controller MC of the robot 1 causes the speaker device 13 to output a voice such as "OK, XXX. Turn on the lights." In response to this, the smart speaker 51 controls the lighting in the room, which is an AI device compatible with the network. Note that the lights to be controlled are registered in advance to be controlled by the smart speaker 51 . In addition to lighting, for example, air conditioners, televisions, curtain opening/closing devices, etc. may be used as AI devices.

(2) A function to distinguish between utterances from the smart speaker 51 and user utterances. It is possible to provide a function for distinguishing between direct utterances. As a result, the robot 1 can be controlled not to respond to sounds from the smart speaker 51, which are voices that are not registered. By registering the voice, not only the user but also the robot 1 can give instructions to the smart speaker 51 . By being able to distinguish individual utterances in this way, it also has a function of not interfering with voice operations on the smart speaker 51 .
(3) Display Function of Speech from Smart Speaker 51 The robot 1 may acquire the content of the speech given by the user to the smart speaker 51 , convert it to text, and display it on the touch panel section 7 .
The controller MC of the robot 1 preferably has a program that acquires the content of speech and converts it into text by itself or by connecting to a server.
For example, the user utters "OK, XXX. Tell me about today's weather." is 20%.", all these conversations are displayed on the touch panel unit 7 of the robot 1, for example, the following conversations between the two.
"User: OK, xxx. What's the weather like today?
Smart Speaker: Today's weather in Okazaki City, Aichi Prefecture is sunny with a maximum temperature of 15 degrees and a 20% chance of rain.
In addition, the controller MC of the robot 1 preferably has a program for briefly adapting or summarizing the content converted into text. It is preferable that the robot 1 output the adapted or summarized content by voice or display it on the touch panel section 7 .
(4) A function for the robot 1 to learn various things by listening to the smart speaker 51 when no one is present. When the user is away, the controller MC of the robot 1 is made to output the question word with a phrase to activate the smart speaker 51 at a predetermined timing (the voice of the robot 1 is the smart speaker 51 ). At this time, the controller MC acquires and stores the content of the speech from the smart speaker 51 in the robot 1 using the microphone 12, and outputs the content as voice at a predetermined timing. The predetermined timing is, for example, when a person is detected by the Doppler sensor 22 for a predetermined time, or when the user speaks to the robot 1 as a built-in scenario such as "Is there any news?" .

<Modification 1 of Embodiment 7>
A function that does not interfere with voice operations of other voice recognition devices, such as a smart speaker, may be set. For example, when the robot 1 recognizes the voice of another smart speaker activation phrase (voice recognition start word), the robot 1 may stop its own voice output.
For example, when recognizing the speech of an activation phrase such as “OK, XXX”, which is the activation phrase of Company A, which is another smart speaker, the controller MC of the robot 1 acquires it from the microphone 12. , when it determines that it is a registered activation phrase, temporarily stops its own voice output.
As a result, the function can be exhibited without interfering with the voice operation of the voice recognition device.

<Modification 2 of Embodiment 7>
The robot 1 recognizes the speech recognition activation keyword of another speech recognition device such as the smart speaker 51, recognizes the pronunciation of the user to the smart speaker 51 after that, requests the cloud server, and searches with a search engine etc. and get the corresponding answer.
The robot 1 recognizes a case where the smart speaker 51 has failed in speech recognition (for example, "There is an error") or a speech output indicating that an appropriate answer to the speech recognition result cannot be output (for example, "I'm sorry"). If so, the robot 1 outputs the answer it has obtained in advance. Alternatively, the robot 1 may make a request to the cloud server to obtain an answer after receiving an audio output indicating that the speech recognition has failed or that an appropriate answer to the speech recognition result cannot be output.

<Embodiment 8>
The robot 1 may, for example, read aloud news articles on a website according to a user's request. The controller MC of the robot 1 makes a request for news articles using a search engine on the server, and the cloud server responds to the request with, for example, news data from registered sites as text data.
The robot 1 synthesizes and reads out (outputs) the news data, and at the same time synthesizes and reads out (outputs) the name of the information source of the article. At the same time, the URL of the information source of the article is displayed on the touch panel section 7 as a display screen, and when the URL is touched on the touch panel section 7, the content of the page of the URL is displayed on the touch panel section 7. make it
Also, when reading out an article or the information source of the article, it is preferable to output in an expression that makes it possible to understand that they are quotes.
For example, a case will be described in which the content of the article at the URL "https://\\\\.jp/archives/92###" is read.
“I will read out the article on the site of “XX News”. "... will be on sale from January 15th this year."That's right. 』
For example, "I'll read the article on the site." or "That's right." Then, on the screen display, for example, "This is the information of https://\\\\.jp/archives/92###. ] to display the URL. By touching the URL portion, the content of the read-out article is displayed on the touch panel portion 7 again.
"Speaking so that the listener can understand" means to change the tone or voice between the article part and the non-article part, for example.
In this way, the content of the news can be understood just by listening to the content read by the robot 1 without reading the Web article, and in some cases, the content of the news can be confirmed visually just in case.

<Embodiment 9>
In the ninth embodiment, a case will be described in which invisible movements of the computer are made visible by movements of actuators of the robot.
(1) Robotics process automation Robotics process automation (hereinafter referred to as RPA) is software that automates simple personal computer work. The software can be set up on a server or it can be set up on the user's computer. An example of the case where the software is set in the server based on FIG. 15 and the robot 1 of each of the above embodiments is incorporated into the RPA network will be described.
As shown in FIG. 15, a cloud server 55 and user-side computers 56 and 57 are connected by a network using the Internet. The cloud server 55 and the robot 1 are also connected via a network. The user side computer 56 is controlled by the cloud server 55 by the RPA program. In addition, the robot 1 is notified of processing information such as a predetermined processing in a program for RPA executed by the cloud server 55 that will be executed soon, is being executed, or has been executed. It's becoming
The cloud server 55 causes the user-side computer 56 to process the processes 1 to 4 in order. In this embodiment, the computer 56 executes the processes 1 and 2, and the computer 57 executes the processes 3 and 4. FIG. More processes may be set, and one or more user-side computers 56 may be involved in the process.
Process 1 is, for example, user access/login to computer 56, etc. Process 2 is, for example, list creation/sorting based on data in computer 56, etc. Process 3 is, for example, a customer executed following process 2 Correction of each billing content, process 4 is, for example, issuance of a bill executed subsequent to process 3. FIG. In the ninth embodiment, for example, in process 3, the user is prompted to make an input for correction, and after receiving the input, the next process 4 is performed. In other words, after processing 2, the system temporarily enters a standby mode until the force in processing 3 is completed.
The cloud server 55 outputs different notification information to the robot 1 immediately before, during, and after executing these processes, and the robot 1 notifies the surroundings of the robot 1 of the processing status based on the notification information. good. Alternatively, the notification may be made once for each process.

for example,
a. The robot 1 notifies which process is in what state by voice, sound, music, or the like.
b. Notify on the display screen. a. can be done at the same time.
c. Since processing 3 requires user input, only processing 3 may be notified, or only processing 3 may be notified differently (identifiable) from other notifications.
d. The status of processing is transferred from the robot 1 to another terminal device and notified.
e. The robot 1 behaves so that the processing status can be understood. For example, if the processing of the computer 56 is being performed, it is controlled to face in that direction. Therefore, the directions of the computers 56 and 57 with respect to the robot 1 should be recognized in advance using some means for identifying directions, such as the above shape recognition program. For example, the robot 1 may be provided with a direction indicating member such as an arrow or an arm member so that the computers 56 and 57 as notification targets are located in the indicated direction.

(2) Blockchain Blockchain is a mechanism in which a large number of computers are distributed and recorded. Especially in public blockchains, the data to be recorded and the recorded data are made public.
Therefore, the robot 1 is placed in the blockchain network and the robot 1 notifies the user before the block data is sent. It is preferable to stop the transmission by causing the robot 1 to output a command to "wait" (that is, to make a request to wait without transmitting data).

<Embodiment 10>
Although the natural dialogue mode has been described in each embodiment, a foreign language learning mode may be provided in place of the natural dialogue mode or together with the natural dialogue mode. When a foreign language learning mode is provided in addition to the natural dialogue mode, it is preferable to switch from the natural dialogue mode to the foreign language learning mode, for example, when a voice saying "switch to foreign language learning mode" is recognized in the natural dialogue mode. Also, it is preferable to switch from the foreign language learning mode to the natural dialogue mode when the voice of "switch to the natural dialogue mode" is recognized in the "foreign language learning mode".
It is often said that the best way to learn a foreign language is to make foreign friends, but not many people have such an opportunity. Therefore, it is preferable to provide a foreign language learning function, which is a dialogue system that can be a partner for foreign language learning.
It can be any first and second language association. A configuration in which a Japanese dialogue system and an English dialogue system are linked will be described below.
Basically, the system should allow conversations in English, and a request such as "Tell me again" should be output in Japanese during the conversation. Furthermore, in cooperation with an English language analysis web service, etc., it has a function to explain the English sentences in conversation in Japanese in response to requests such as "explain". If you don't know how to say what you want to say in English during a conversation, you can request "translate" and it will output what you mean in English. If you read the output English sentence, you can continue the conversation as it is. Since there is no need to do research on your own, you can expect smooth English conversation learning without interruptions in conversation.
In the foreign language learning mode, both the native language (e.g. Japanese for Japanese) speech recognition engine and the foreign language (e.g. English) speech recognition engine should use the speech recognition engine running on the cloud. However, for native languages (e.g., Japanese), since the content of the request is a fixed phrase and the format of the response to the request is fixed, a voice recognition engine is provided locally (e.g., in the robot 1). Good to use. So is the dialogue engine. The text-to-speech engine may also be located anywhere, but preferably locally.
When the speech data based on the signal from the microphone 12 is fed to speech recognition engines in both languages, either speech recognition engine will return some result. For example, if you throw the Japanese word "say it again" to both engines, the Japanese engine will return the text data "say it again" and the English engine will return "say it again" Returns random text data interpreted as English. In such a case, since the Japanese request is a fixed phrase, the text data returned by the Japanese engine is compared with the fixed phrase of the request. If they do not match, it is judged that English was spoken, and a process for separating English and Japanese should be performed.
Although the request may be accepted in English, the inventor has found that it is desirable to be able to make the request in Japanese as well, especially assuming a case where the user does not understand English even when making a request.
The function to explain English in Japanese in response to a request such as "explain" may simply output a translation of the English in Japanese, but especially the phrases and grammar used in English It is good to output the explanation of . In particular, it is good to output a commentary on the syntax used in the English sentence. In order to explain the syntax, it is preferable to draw each phrase as a vertex (node), for example, to draw surrounding each phrase, and to draw branches (edges) such as line segments that indicate the relationship between related phrases. For example, it is preferable to display on the touch panel unit 7 in the form of a graph structure (especially a tree structure).
Moreover, it is preferable to output the displayed content from the speaker device 13 by voice. For example, parsing service AP as described at https://gigazine.net/news/20160602-foxtype-review/
I should be configured to call I, receive the result, and output the analysis result in Japanese.
For example, the following processing and output are performed.
"processing get phrases from english dialogue engine robot i'm a fantastic robot.
Person: Say it again
Robot I'm a fantastic robot.
person "explain"
Processing (recognizes "Explain me") → Syntax analysis API call → Generates Japanese explanation from the result of syntax analysis Robot I is the subject, am is the verb, and robot is the object.
Fantastic modifies robot with an adjective meaning fantastic.
In English, it means "I am a wonderful robot."
Hito "I don't think so."
Robot Don't say it!
If you don't understand what you want to say in English, you can ask for Japanese to teach you how to say it in English. If you can't make a request in Japanese, when you don't know how to say it in English, you have to look it up in a dictionary or translate it on the internet. If you can make a request in Japanese, you can learn without stress as if you were just talking to a bilingual. Since it is very important to continue learning a language, it is extremely important to keep learning as stressful as possible. According to this configuration, it is possible to realize the robot 1 that can continuously learn the language.

<Embodiment 11>
In addition to the functions described in each embodiment, it is preferable to provide a function of speaking when it is detected that a person has entered the room in which the robot 1 is installed. Further, it is preferable to provide a function of speaking when it is detected that a person leaves the room where the robot 1 is installed.
For example, if sensors are provided in the room and passages outside the room to detect that a person has entered the room in which the robot 1 is installed and that a person has left the room in which the robot 1 has been installed. good. In particular, when there is an automatic door for entering and exiting the room where the robot 1 is installed, it is preferable to use a sensor that detects that a person is approaching the automatic door to open and close the automatic door. In particular, the first human detection sensor outside the automatic door, the second human detection sensor inside the automatic door, and the person at the automatic door when the automatic door is open. The controller MC of the robot 1 may be connected to at least two of the third human detection sensor sensors (sensors for preventing a person from being caught in the door) to detect entry and exit into the room. In this way, it is possible to detect the entry and exit of the room in which the robot 1 is installed without installing a new sensor. For example, detection may be performed by capturing an edge of a signal that rises when each sensor detects a person.
For example, after a person is detected by the first person detection sensor, when a person is detected by the third person detection sensor, the loudspeaker device 13 outputs a phrase such as "Welcome" to welcome the person who has entered. should be output from For example, when a person is detected by the third person detection sensor after the person is detected by the second person detection sensor, the loudspeaker device 13 outputs a voice saying "thank you very much" to thank the person. should be output from
At these times, the first to third motors 23 to 25 may be operated to move the robot 1 from the installation position toward the preset automatic door. Note that if the first human detection sensor and the second human detection sensor detect a person at the same time, the first human detection sensor should be prioritized. In this way, it becomes easier for people who enter to notice the existence of the robot 1, and it is possible to reduce the sense of incongruity when the people who enter suddenly say "thank you" to the robot 1.

<Other embodiments>
(1) In each embodiment, etc., the wireless LAN device 21 is provided, but a wired LAN device may be provided instead of or together with this and connected to a wired LAN network. The wired LAN device may be built in the robot 1 or may be externally attached. A wired LAN device may be connected to the USB OTG (On-The-Go) terminal 18 . The wired LAN network may be configured to be connected to the Internet via a router or the like. Depending on the environment, wireless LAN communication may not be stable or connection may not be possible. For example, when the robot 1 is installed in a place where there are many wireless LAN devices such as smartphones, it is desirable to access the Internet via the wired LAN devices.
(2) In each embodiment, etc., an example of interaction between the human and the robot 1 in the half-duplex method is shown, but the interaction between the person and the robot 1 may be performed in the full-duplex method. . For example, in the dialog shown in Table 1, after the person says "Open...", the human voice recognition continues while the person says "Are you sure?" If the recognition is made during the utterance of "Are you sure?", the utterance may be interrupted if the utterance is still in progress, and the robot may immediately utter "Cancelled".
(3) The configuration for half-duplex communication is particularly good because it simplifies the configuration and processing and reduces the cost. However, it is difficult to know when the robot 1 turns on the microphone 12, and even if the user speaks, the first part of the user's speech that the robot recognizes, that is, the first part of the words, is often missing. The inventors have found. In order to solve this problem, it is preferable to perform a characteristic screen display at the timing when the controller MC turns on the microphone. This supports smooth conversation. As a mode of displaying a characteristic screen at the timing when the controller MC turns on the microphone, it is preferable to perform at least one of 1) lighting the four corners of the screen and 2) displaying a microphone icon, especially 1). , and 2) exhibit excellent effects.
(4) Various motors such as DC motors can be used as the motors constituting the first to third motors 23 to 25, but stepping motors are particularly preferable, and the posture of the robot 1 is controlled by the stepping motors. configuration. A stepping motor changes its torque in proportion to the current flowing through the motor. If a large amount of current is applied, a large amount of torque can be obtained, but problems such as heat generation and battery life arise. Therefore, when the robot 1 is at rest, it is particularly preferable to apply a minimum current required to maintain its posture, and apply a large current only when the robot 1 changes its posture. It should be noted that although the minimum current required to maintain the attitude of all the servo motors 23 to 25 at rest may be energized, the body portion that can be supported by the detent torque (torque when not energized) While the first motor 23 is not energized, the second motor 24 and the third motor 25 should be energized even when the head is stationary because detent torque is lost.
In addition, if the motor is rotated with the torque at rest, it will lose synchronism due to lack of torque and will not rotate well. On the other hand, when the motor is stationary, it does not require a large torque as it does when rotating, so it is better to lower the current compared to when rotating.
Also, when changing the direction in a certain direction, it is preferable to perform control such that the speed is increased to a constant speed while accelerating, and then decelerates and stops when the target angle is approached.
In addition, when a robot is driven by a motor, it generates mechanical and electrical noise. Since this noise lowers the recognition rate of speech recognition, the motor should be controlled to stop during speech recognition.
The scope of the invention is not limited to the configurations or limitations expressly set forth herein, but rather includes within its scope any combination of the various aspects of the invention disclosed herein. Although the patentable configuration of the present invention is specified in the appended claims, any configuration not currently specified in the claims is disclosed herein. It is our intention to claim such configurations in the future.
The present invention is not limited to the configurations described in the above-described embodiments. The constituent elements of the above-described embodiments and modifications may be arbitrarily selected and combined. In addition, arbitrary constituent elements of each embodiment and modifications, arbitrary constituent elements described in Means for Solving the Invention, or constituent elements embodying arbitrary constituent elements described in Means for Solving the Invention and may be configured in any combination. We intend to acquire the rights for these as well in the amendment of the present application or in a divisional application. In addition, even if there is a description of "in case of" or "when", it is not described as a configuration limited to that case or time. We disclose and intend to reserve these occasions and occasions. Moreover, the parts described with order are not limited to this order. It also discloses a configuration in which some parts are deleted or the order is changed, and there is an intention to acquire the right.
I also intend to acquire rights to the whole design or partial design by filing a conversion application to a design application. Although the drawing shows the entire device in solid lines, the drawing includes not only the overall design but also the partial design claimed for a part of the device. For example, it is a drawing that includes a part of the device as a partial design regardless of the member, as well as a partial design of a member of the device. The part of the device may be a part of the device or a part of the member. We intend to acquire rights not only for the overall design, but also for the partial design in which any part of the solid line part of the drawing is the broken line part.

1... A robot as a device, 41... A smart phone as another device, 51... A smart speaker as another device.

Claims (13)

A character string based on speech recognition is transmitted as the input character string to a plurality of different servers equipped with a dialogue engine that outputs an output character string corresponding to the input character string, and is output as the output character string from the plurality of different servers. and outputting output information to at least one of the user or other device based on the character string output as the output character string received from the different servers Or have a function to interact with other devices
A device characterized by: 2. The apparatus according to claim 1, having a function of receiving character strings output as said output character strings from said plurality of different servers, and selecting the longest said output character string for interaction. 3. The apparatus according to claim 1, further comprising a function of receiving character strings output as said output character strings from said plurality of different servers, selecting said output character strings with question marks at the end of words, and causing dialogue. Apparatus as described. 4. A function according to any one of claims 1 to 3, characterized in that it has a function of receiving character strings output as said output character strings from said plurality of different servers, combining affirmative sentences and then combining interrogative sentences to have a dialogue. Apparatus as described. It has a function of receiving character strings output as said output character strings from said plurality of different servers, combining them so as to arrange the character strings changed in topic at the end, and interacting with them. A device according to any one of the preceding claims. Claims 1 to 5, characterized by having a function of receiving character strings output as said output character strings from said plurality of different servers, combining said friendly output character strings so as to place them first, and interacting with said character strings. A device according to any of the preceding claims. 7. The device according to any one of claims 1 to 6, characterized by having a function of receiving character strings output as said output character strings from said plurality of different servers, combining them in random order, and interacting with them. . When character strings output as the output character strings are received from the plurality of different servers, and if the output character strings including emoticons are included in the character strings, they are not regarded as a dialogue target, and displayed on a display unit as a dialogue target. 8. The apparatus according to any one of claims 1 to 7, characterized by having a function of displaying together with said output character string. receiving character strings output as the output character strings from the plurality of different servers, and selecting only one of the output character strings containing the same character string and combining it with the other output character strings 9. The device according to any one of claims 1 to 8, characterized in that it has a function of interacting with the 2. A function of receiving character strings output as said output character strings from said plurality of different servers, converting the endings of said output character strings by an ending conversion engine, and then combining and interacting with them. 10. The device according to any one of -9. Receiving character strings output as the output character strings from the plurality of different servers, storing part of the output character strings in a storage means without using all the output character strings, and subsequent interaction 11. The device according to any one of claims 1 to 10, further comprising a function of retrieving from said storage means and using it for interaction. 12. The apparatus according to any one of claims 1 to 11, wherein when a server having a speech recognition engine is used, a free server and a charged server are mixed and used. A program for causing a computer to implement the functions of the device according to any one of claims 1 to 12.

Images