JP2002261966A - Communication support system and photographing equipment - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable to provide a mental care for loneliness by simulating human conversations and to watch whether there is anything wrong by transmitting an image to a mobile terminal within the extent not invading the privacy. SOLUTION: A robot 1 comprises a conversation processing means 21 to which a microphone 11 and a speaker 12 are connected and which enables voice conversations with the other party using a natural language, an image processing means 22 which is connected by a TV camera 13 for photographing the other party and outputs an image data which is a digitized image photographed by the TV camera 13, and a communication means 23 for sending out the image data onto a network. The image data is transferred to an image server 4 via a communication line 3. By accessing the image server 4 from a mobile terminal 2 which is allowed to access the image server, the image data can be displayed on the mobile terminal 2.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to a communication support system and a photographing device.

[0002]

2. Description of the Related Art In general, elderly people have few opportunities to go out and have few conversations with others, so that their family is often the partner of conversation. In addition, many elderly people are at home due to physical weakness, and their families need to be constantly watching to prevent accidents before they occur and to respond quickly if accidents occur. Under such circumstances, it is difficult for a family living with the elderly to leave the elderly at home and go out.

On the other hand, recently, a kind of robot called a pet robot that acts as if it has intention has been commercialized. As this kind of robot, for example, a dog-shaped pet robot is called “AIBO” ( It is commercialized under the name of Sony Corporation. This robot monitors and learns the tone, facial expression, and how to touch a person's voice. It becomes possible to behave as if growing and to make the illusion of communicating with people, and it has a kind of healing function. However, this pet robot is intended for entertainment by imitating the movement of a dog, and is not intended to support communication with the elderly.

On the other hand, Matsushita Electric Industrial Co., Ltd. has proposed a cat-type or bear-type robot as a pet robot whose main purpose is to communicate with the elderly. In order to contribute, it can perform simple daily conversations and behave like a pet, and also has a function to indirectly grasp the usage status of a pet robot from a remote location. Therefore, this pet robot can talk to the elderly and provide mental care for the elderly, and can confirm the safety of the elderly living alone from a remote location. This pet robot is connected to a center facility installed in a welfare service support center via a communication line (a communication line in which digital communication technology and mobile communication technology are combined with a two-way CATV facility). When confirming the safety of the elderly, a message to be uttered by the pet robot is transmitted from the center facility to the pet robot through a communication line, and the message is transmitted from the pet robot to the elderly by voice. At this time, the response of the elderly is returned to the center equipment, and the safety of the elderly is confirmed by inferring the state of life with the inference device provided in the center equipment. In the inference device, for example, when the elderly person does not respond to the utterance of the pet robot,
It infers that something may have gone wrong.

[0005]

The above-mentioned pet robot has a function of reacting like a living thing, and the former pet robot has no interactive function, but the latter pet robot has an interactive function. It is thought that it is possible to reduce the loneliness of the elderly. However, since the latter pet robot is connected to a welfare service support center, which is a public facility, via a communication line, it is necessary to ensure that the privacy of the elderly is not violated. It does not have a function to transfer the video of the situation. Therefore, even if it is inferred that an abnormality has occurred, it is not possible to confirm what abnormality has occurred unless the person goes to the site, and prompt action is required such as illness or injury. May be delayed.

To check the state of the elderly left at home when a family goes out or travels, it is conceivable to use a device capable of transferring images, such as a videophone. However, this type of device only enables the transfer of video during a call, and there is no function to talk to elderly people except during calls, and provides mental care for elderly people except during calls No function. Such a situation applies not only to families with elderly people but also to families with sick people.

SUMMARY OF THE INVENTION The present invention has been made in view of the above circumstances, and aims to simulate a dialogue with a person to enable mental care of loneliness of a person and not to violate privacy. It is an object of the present invention to provide a communication support system and a photographing device that allow an image of an interlocutor to be transferred to a portable terminal within a range so that the presence or absence of an abnormality of the interlocutor can be monitored.

[0008]

According to a first aspect of the present invention, there is provided a dialogue processing means for connecting a voice input device and a voice output device and performing a voice dialogue with a dialogue person using a natural language,
A video input device connected to the video input device for capturing an image of the interlocutor; a video processing unit for outputting video data obtained by digitizing the video captured by the video input device; a communication unit for transmitting the video data to a network; The portable terminal is provided with a portable terminal that is permitted to browse data and a conversion unit that converts video data into a format that can be displayed on the portable terminal. According to this configuration, since the dialogue with the person is simulated using the dialogue processing means, the interlocutor is provided with mental care for the feeling of loneliness. In addition, since browsing of the video data of the interlocutor is possible only with the authorized mobile terminal, the privacy of another person is not infringed by using the mobile terminal so that only the interlocutor's family can carry it. The image of the interlocutor can be transferred to the mobile terminal within the range, and the presence or absence of the interlocutor's abnormality can be monitored. As a result, even in a home that has the elderly and the sick, it is possible to view the state of the elderly and the sick from a distance using a mobile terminal,
You will be able to go out with peace of mind.

According to a second aspect of the present invention, in the first aspect of the present invention, the conversion means is provided in a video server for storing video data transferred via a network, and the video server is allowed by a portable terminal holder. In this case, only the video data is selected and transferred to the portable terminal. According to this configuration, by providing the video server that stores the video data of the interlocutor, it is possible to browse the video data retroactively.

According to a third aspect of the present invention, in the first aspect of the present invention, the conversion means is provided in one of the video processing means and the communication means. According to this configuration, the video data of the interlocutor can be browsed from the portable terminal without providing the video server, and the system configuration is simple.

According to a fourth aspect of the present invention, in the first to third aspects of the present invention, the video processing means sends the video data of the interlocutor to the network through the communication means in response to an instruction from the portable terminal. According to this configuration, since the transmission of the video data is instructed from the mobile terminal, it is possible to view the video when the owner of the mobile terminal requests.

According to a fifth aspect of the present invention, in the first to fourth aspects of the present invention, the image processing means operates in response to a command instructed by an interlocutor through the interactive processing means. According to this configuration, since the interlocutor causes the user to shoot a video image by himself / herself, for example, when the interlocutor feels abnormal, it is possible to send video data to notify the portable terminal holder.

According to a sixth aspect of the present invention, in accordance with the first to fifth aspects of the present invention, it is specified whether the video processing means is operated according to an instruction from an interlocutor or is operated according to an instruction from the portable terminal. It is provided with a notification means for notifying the side. According to this configuration, while it is possible to control the video processing means from both the interlocutor and the mobile terminal, the instructions of both are not confused.

According to a seventh aspect of the present invention, in the first to sixth aspects of the present invention, a contact sensor for detecting a contact of the interlocutor is added, and when the contact sensor detects the contact of the interlocutor, the interaction processing means is provided. Is to start. According to this configuration, the dialogue processing means is activated by touching the contact sensor when the dialogue person tries to talk, so that the dialogue processing means malfunctions due to ambient noise or voice of another person other than the talker. Can be prevented.

According to an eighth aspect of the present invention, in the first to seventh aspects, a means is provided for indicating to the interlocutor that the image of the interlocutor is being captured by the image input means. According to this configuration, since it is possible to know when the interlocutor is being photographed, it is possible to take a measure such as moving to a place where no photographing is performed when the photographer does not want to be photographed due to privacy.

According to a ninth aspect of the present invention, in the first to eighth aspects of the present invention, the interaction processing means has a function of asking the interlocutor to confirm a voice input by the interlocutor. Upon receiving a positive response, the voice input by the interlocutor is determined. According to this configuration, since the voice input of the interlocutor is confirmed by repetition, it is possible to surely understand the voice generated by the interlocutor.

According to a tenth aspect of the present invention, in the ninth aspect, the interaction processing means learns a voice generated by the interlocutor using a positive response from the interlocutor as a teacher signal. According to this configuration, the recognition rate is increased by learning the voice generated by the interlocutor in the dialog with the interlocutor.

According to an eleventh aspect of the present invention, in the first to tenth aspects, there is provided means for transferring video data browsed by the portable terminal to another terminal on the network. According to this configuration, by transferring the video data to another terminal, for example, when the owner of the portable terminal notices an abnormality of the interlocutor, the video data is sent to the doctor to receive advice on coping. Becomes possible.

In a twelfth aspect of the present invention, in the first to eleventh aspects of the present invention, there is provided a notifying means for notifying the interlocutor of a request to view video data from the portable terminal. According to this configuration, even when the family uses the portable terminal to monitor the elderly or the sick from outside the home, even if the family has the elderly or the sick who are interlocutors, Since the notification means notifies that the family has confirmed, the elderly or sick can know that the family has confirmed and can feel reassured.

According to a thirteenth aspect, in the twelfth aspect, the notifying means has a function of notifying a viewer of the video data using the identification information of the portable terminal. According to this configuration, it is possible to know who has confirmed the video data, so that the user can be more relieved.

According to a fourteenth aspect, in the thirteenth aspect, the notifying means includes a notifying unit assigned to each viewer of the video data. According to this configuration,
Since the viewer can be identified by visually recognizing the notification unit without depending on the character data or the like, it is possible to intuitively know not only that the viewer has checked the video data but also who has viewed the video data.

According to a fifteenth aspect of the present invention, a voice input device and a voice output device are connected to each other, and a dialogue processing means for performing a voice dialogue with a dialogue person using a natural language, and a video input device for imaging the talker. A video processing unit that is connected and outputs video data obtained by digitizing a video captured by a video input device, a communication unit that sends video data to a network,
Conversion means for converting the video data into a format that can be displayed on a portable terminal permitted to view the video data of the interlocutor through the network. According to this configuration, since the dialogue with the person is simulated using the dialogue processing means, the interlocutor is provided with mental care for the feeling of loneliness. In addition, since browsing of the video data of the interlocutor is possible only with the authorized mobile terminal, the privacy of another person is not infringed by using the mobile terminal so that only the interlocutor's family can carry it. The image of the interlocutor can be transferred to the mobile terminal within the range, and the presence or absence of the interlocutor's abnormality can be monitored. As a result, even in a home that has the elderly and the sick, the state of the elderly and the sick can be monitored from a distance using a portable terminal, and the user can go out with peace of mind.

According to a sixteenth aspect of the present invention, in the invention of the fifteenth aspect, the clock means for measuring the current date and time and the date and time measured by the clock means when the interlocutor is imaged are synthesized as characters into the video data. And superimpose processing means for sending out from the communication means. According to this configuration, since the date and time when the video was captured is attached to each video data, when there is a large time difference between the date and time when the video was captured and the date and time when the video data was browsed by the mobile terminal, By checking the date and time attached to the video data, the viewer can take measures such as requesting the latest video.

According to a seventeenth aspect, in the sixteenth aspect, the video data transmitted to the network through the communication means and the result of the success or failure of the communication are associated with the date and time measured by the clock means. A storage means for sequentially recording and accumulating and holding the recorded contents without power supply is added. According to this configuration, even when the video data is not transmitted from the communication means and the video data cannot be confirmed by the portable terminal, the health condition of the interlocutor can be confirmed later by the video data stored in the storage means. Since the result of communication success / failure is stored in the storage means as a transmission history, it is possible to easily know whether or not the communication means is abnormal during maintenance.

According to an eighteenth aspect of the present invention, in the invention of the fifteenth to seventeenth aspects, an attitude sensor for detecting a tilt of the image input device is added, and the image processing means converts an image captured by the image input device into an image. The image is corrected based on the inclination detected by the posture sensor so as to be erect. According to this configuration, since the correction is performed so as to remove the inclination of the image based on the information obtained by the posture sensor, when the image input device is inclined,
For example, even when the photographing device is arranged in a place that is not horizontal or when an interlocutor lifts the photographing device, it is possible to obtain an image similar to the state where the photographing device is arranged on a horizontal plane. .

According to a nineteenth aspect of the present invention, in accordance with the fifteenth to eighteenth aspects of the present invention, there is provided a swing driving means capable of changing the direction of the visual field of the video input device, and A template that specifies that the image is captured can be registered, and a pattern processing unit that controls the swing driving unit so as to increase the similarity by comparing the template with the image captured by the video input device is added. Is what is being done. According to this configuration, the direction of the visual field of the video input device is controlled so that the similarity between the video captured by the video input device and the template is increased. It is possible to adjust the orientation of the video input device so that it is captured near the center of the visual field of the video input device.

According to a twentieth aspect of the present invention, in accordance with the fifteenth to eighteenth aspects of the present invention, the second audio input device in which the direction of the visual field of the video input device matches the direction having directivity, and the video input device Swing drive means for simultaneously changing the direction of the visual field of the device and the direction in which the second voice input device has directivity, and the head in a direction in which the sound level of the output of the second voice input device increases. A sound level processing means for controlling the driving means is added. According to this configuration, the direction in which the interlocutor exists can be estimated based on the sound level. Therefore, by adjusting the field of view of the video input device to a direction in which the sound level increases, the interlocutor can be estimated within the field of view of the video input device. Is more likely to be captured.

According to a twenty-first aspect of the present invention, in accordance with the fifteenth to eighteenth aspects of the present invention, there are provided a plurality of the video input devices arranged so that a phase difference of an output with respect to a sound source present in a direction of a visual field becomes zero. And a phase difference detecting means for controlling the swing driving means in a direction to reduce the phase difference between the third sound input apparatus and the third sound input apparatus. According to this configuration, by adjusting the visual field of the video input device so as to reduce the phase difference between the plurality of audio input devices, it becomes possible to capture the interlocutor within the visual field of the video input device.

According to a twenty-second aspect of the present invention, in accordance with the fifteenth to twenty-first aspects, a half mirror is provided in front of the video input device so as to be orthogonal to the optical axis direction of the video input device. Is what it is. According to this configuration, since the interlocutor is present in front of the video input device in a state where the interlocutor reflects his / her figure on the half mirror, the interlocutor can be surely interposed by the video input device by the interlocutor's intention. It becomes possible to take an image.

According to a twenty-third aspect of the present invention, in accordance with the fifteenth to twenty-first aspects, an image can be displayed on a display device to which an external display device can be connected and image data picked up by the image input device can be displayed on the display device. Output means is added. According to this configuration, the interlocutor can check the image captured by the video input device on the spot by the display device connected to the video output means, so that the interlocutor can only check his own reflection condition. It is also possible to check whether or not an appropriate video is captured by checking the surroundings.

According to a twenty-fourth aspect of the present invention, in the invention of the fifteenth to twenty-third aspects, there is provided an outer shell capable of being held by the interlocutor. And a heart rate recording device that records a heartbeat sound detected by the heart rate sensor. According to this configuration, it is possible to not only know the health of the interlocutor by the video but also check the health of the interlocutor by the heartbeat sound. It is possible to know the health condition of the person more accurately. Moreover,
Since the heartbeat sensor is configured to collect heartbeat sounds when the interlocutor is holding the heartbeat sensor, heartbeat sounds can be collected without any trouble.

In a twenty-fifth aspect of the present invention, in any one of the fifteenth to twenty-fourth aspects, a notifying means is provided for notifying the interlocutor of a request to view video data from the portable terminal. According to this configuration, even when the family uses the portable terminal to monitor the elderly or the sick from outside the home, even if the family has the elderly or the sick who are interlocutors, Since the notification means notifies that the family has confirmed, the elderly or sick can know that the family has confirmed and can feel reassured.

According to a twenty-sixth aspect, in the twenty-seventh aspect, the notifying means has a function of notifying a viewer of the video data using the identification information of the portable terminal. According to this configuration, it is possible to know who has confirmed the video data, so that the user can be more relieved.

According to a twenty-seventh aspect, in the twenty-sixth aspect, the notifying unit includes a notifying unit assigned to each viewer of the video data. According to this configuration,
Since the viewer can be identified by visually recognizing the notification unit without depending on the character data or the like, it is possible to intuitively know not only that the viewer has checked the video data but also who has viewed the video data.

[0035]

DESCRIPTION OF THE PREFERRED EMBODIMENTS (First Embodiment) First, the overall configuration of a system used in the present invention will be described. In the present embodiment, as shown in FIG. 1, a robot 1 placed in a house where an elderly person or a sick person (hereinafter, simply referred to as a “talker” in the sense of a person interacting with the robot 1), and the robot 1 1 and a portable terminal 2 connected via a communication line 3 including a network such as the Internet. The robot 1 has a function of dialogue by voice, a function of capturing an image of the surroundings, and a function of communicating data including video. In the present embodiment, the robot 1 will be described as having a dog-shaped appearance. However, it is not essential to simulate the movement of an animal or the like. You don't have to worry about movement. In the present embodiment, a mobile phone having an image display function is assumed as the mobile terminal 2. The communication line 3 is connected to a video server 4 for temporarily storing video captured by the robot 1.

The robot 1 includes a microphone 11 (speech input device) as an ear and a speaker 12 (speech output device) as a mouth in order to perform a dialogue by voice. , A small T as an eye
A V camera 13 (video input device) is provided. The microphone 11 and the loudspeaker 12 are connected to the dialogue processing means 21, and the dialogue processing means 21 enables the dialogue processing in a natural language through the microphone 11 and the speaker 12. On the other hand, the TV camera 13 is
The video processing unit 22 adjusts the brightness and contrast of the video imaged by the TV camera 13,
For example, digitizing video signals. The dialog processing means 21 and the video processing means 22 constitute a signal processing unit 20 having a microcomputer as a main component.

The communication means 23 is provided after the video processing means 22.
The communication means 23 compresses the video signal into a data format that can be sent to the communication line 3 and sends it to the communication line 3. Compressed video signal (below,
Video data) is temporarily stored in a video server 4 connected to the communication line 3. Here, the video server 4
Has a function as a conversion unit that converts the size and color tone of the video data into a format that can be displayed on the screen of the mobile terminal 2. Therefore, the video server 4 is provided from the mobile terminal 2 through a dedicated Internet connection service (such as an Internet connection service provided by a mobile phone).
By accessing, the image stored in the video server 4 can be displayed on the screen of the mobile terminal 2. Since the video data is converted by the video server 4 into a format that can be displayed on the screen of the mobile terminal 2, the video captured by the robot 1 can be confirmed by the mobile terminal 2. The video data may be a still image, but is preferably a moving image. However, since the mobile terminal 2 generally has a limited memory capacity for storing video, the video data is transferred to the mobile terminal 2.
It is difficult to reproduce the video after it has been captured, and in consideration of the current communication speed on the communication line 3 and the video compression technique, the video becomes a moving picture equivalent to a videophone.

The communication means 23 has a notification means 28
Is connected, and when the image captured by the robot 1 and stored in the video server 4 is browsed by the mobile terminal 2, the notification unit 28 uses an auditory unit (speaker) or a visual unit (lamp). To inform the interlocutor that the image has been viewed. That is, if the family of the interlocutor browses the images stored in the video server 4 using the mobile terminal 2, the browsing of the image of the video server 4 is notified to the robot 1, and the interlocutor is notified by the notifying unit 28. Will be. Thus, the interlocutor can have a sense of security that the family or the like has confirmed his / her situation, and can eliminate the feeling of anxiety.

The timing for storing video data in the video server 4 is determined by an instruction from the portable terminal 2. That is, it requests the robot 1 to accumulate video data through the video server 4 according to an instruction from the mobile terminal 2, and transfers the image stored in the video server 4 to the mobile terminal 2. However, when the mobile terminal 2 requests the transfer of the video data, the interlocutor interacting with the robot 1 may not be near the robot 1, and the user may view the video data past the time when the request of the video data transfer was made. Sometimes I want to
The video data may be stored in the video server 4 by performing the imaging by the V-camera 13 periodically while the video server 13 is designated in advance. As described above, if the timing at which the video data captured by the TV camera 13 is stored in the video server 4 and the timing at which the mobile terminal 2 reads the video data from the video server 4 are shifted, the mobile terminal 2 can instruct the video server 4 to issue an instruction. After giving the video data, the video data can be viewed by waiting for the time required to transfer the video data from the video server 4 to the portable terminal 2, so that the response time until the video is viewed can be reduced. In this case, it is desirable to attach an imaging time to each video data.

Further, the video server 4 is designed to prevent the images stored in the video server 4 from being viewed by others.
Access to each of the video data stored in the portable terminal 2 is restricted according to the identification information from each portable terminal 2. For example, when a mobile phone is used as the mobile terminal 2, the phone number of the mobile phone is used for the identification information, and the video server 4 transfers only the video data associated with the phone number to the mobile phone. The identification information is not limited to the telephone number, and it is also possible to use the personal identification number as the identification information after accessing the video server 4 or use a combination of a plurality of information as the identification information.

As described above, when accessing the video data stored in the video server 4, in order to confirm the identification information from the portable terminal 2, the video server 4 uses the identification information and the personal information such as the name. A database that associates data with the data is provided. The database transfers personal data to the robot 1 when identification information is input from the mobile terminal 2, and notifies the interlocutor of the personal data through the notification means 28. Notification means 2 for this purpose
Numeral 8 notifies personal data by voice or displays a personal data by separately providing a small display device. With this configuration, the interlocutor can confirm the person who has accessed the video server 4, and can give the interlocutor a sense of security.

By the way, in this embodiment, it is possible to select a necessary one from the videos obtained through the portable terminal 2, attach it to an e-mail, and transfer it to an appropriate destination.
That is, the video server 4 adds an index to the video data specified by the mobile terminal 2 so that the video data can be handled as a document attached to the electronic mail input by the mobile terminal 2. Therefore, in the case where any abnormalities are recognized in the interlocutor imaged by the video obtained through the mobile terminal 2, it becomes possible to send an e-mail with the video data attached to the doctor at home, and the doctor's advice Appropriate measures such as receiving

As described above, since the video that can be reproduced on the portable terminal 2 through the video server 4 is captured by the TV camera 13, the timing at which the video is captured by the TV camera 13 is inconvenient for a person who interacts with the robot 1. Sometimes. For example, it is not preferable from the viewpoint of privacy that an image is taken while changing clothes even in a relative. Therefore, in order to notify the interlocutor interacting with the robot 1 that the TV camera 13 is taking an image, the robot 1 is provided with a display unit 2.
7 is provided. The display means 27 may be a light emitting diode, and can be informed to the interlocutor by turning on the light during the imaging by the TV camera 13.

Incidentally, the dialogue processing means 21 has the configuration shown in FIG. 2 and enables a simple dialogue by voice. The voice input through the microphone 11 is input to the voice recognition unit 31. In view of the environment in which the robot 1 is used, ambient noise is often input through the microphone 11.
1 to extract the feature parameters of the input speech, separate speech and noise based on the continuity and frequency distribution of the speech, and perform speaker recognition. The speaker recognition is performed in order to avoid erroneous recognition between the voice from the television or the radio and the voice of the interlocutor. The information required for speaker recognition and noise separation is herein referred to as an acoustic model M1. The speech recognition unit 31 performs word recognition by collating with a standard pattern, converts speech into text data TXT1, and outputs the text data TXT1. Here, the information of the standard pattern for performing the word recognition is herein referred to as a language model M2.

The text data TXT1 is stored in the dialog management unit 32.
Is input to The dialogue management unit 32 is a kind of inference engine, and uses a natural language processing technology for collating the text data TXT1 with the knowledge stored in the knowledge base IB.
The syntax analysis and the keyword extraction of the text data TXT1 are performed, and the semantic analysis and the context analysis are performed based on the result. That is, the meaning of the utterance of the interlocutor and the intention of the interlocutor are analyzed and expressed in a predetermined format, and the response content is generated in accordance with the analysis result of the meaning of the utterance and the intention of the interlocutor, and the utterance content is generated in the predetermined format. Deliver to part 33. Here, the predetermined format is a format of a semantic expression. The utterance content generation unit 33 generates text data TXT2 according to the response content passed from the dialog management unit 32. This text data T
The XT 2 is delivered to the voice generating unit 34, and performs voice synthesis from the text, or edits a pre-registered voice according to the text, and outputs the voice as voice through the speaker 12. However, it is desirable to make the sound output from the speaker 12 as close to natural sound as possible.

In the dialog management section 32, as described below, it is possible to generate a response corresponding to "scrutiny" and "schedule management". Which response content is to be generated is transmitted by the interlocutor to the robot 1 through the microphone 11 by voice.

In order to perform "grinding", the interlocutor gives a command "I want to grind" to the robot 1 by voice. Here, after the first and second attacks are negotiated, the scouring is started. In the rejection, since the utterance of the interlocutor is only words in principle, it is sufficient to perform only word recognition without performing syntax analysis or context analysis. The processing procedure at the time of descaling is as shown in FIG. Now, assuming that the interlocutor is the first player, the dialogue management unit 32
Text data TXT1 (word) from the voice recognition unit 31
Is input (S1), it is determined whether or not the input word is a word that has already been input after the start of filtering (S1).
2). If the word has not been input since the beginning of the rejection, it is stored in an appropriate storage device, and the syllable at the end is extracted (S3).
Next, a word having the beginning of the same syllable as the extracted syllable is searched and extracted from the dictionary in the knowledge base IB (S
4). Here, the extracted word is collated with the above-mentioned storage device, and it is determined whether or not the word has already been used after the start of descaling (S5). If the word has already been used, steps S4 and S5 are performed. By repeating, another word is searched and extracted. Here, if there is no change in the order of the words in the dictionary, the words are extracted in the same order each time and the interest is lost. Therefore, a search method in which words are searched randomly at the time of search is desirable. When an appropriate word is extracted in this way, the extracted word is stored in the above-described storage device, and the utterance of the extracted word is instructed to the utterance content generation unit 33. It waits for an input (S6). By the way, in step S2, the text data TXT1 input from the voice recognition unit 31 is collated with the above-mentioned storage device, and when it is determined that the word has already been used after the start of filtering, the dialog management unit 32 sends the utterance content generation unit 33. Is instructed to generate text data TXT2 saying "I told you before" (S7). This processing prompts the interlocutor to utter another word. To end the rejection, the interlocutor may give a command “stop it” to the robot 1 by voice. At the time when the removal is stopped, the contents of the storage device are erased.

Note that the above example shows a simple procedure, and the processing does not include the rule of losing if a word ending in "n" is used or the rule of losing if the response is slow. It is desirable to include processing for implementing these rules. For example, it is desirable to include a process of confirming the ending of the word used by the interlocutor and the robot 1. In this case, the robot 1 also checks the ending after adopting the word extracted from the dictionary. In the above-described processing, the robot 1 can use all the words registered in the dictionary, and the robot 1 is unlikely to lose to the interlocutor. It is desirable to perform processing such as causing

Hereinafter, an example of the progress of “grinding” will be described. Here, the interlocutor calls "apple" and the robot 1
Replies "gorilla" as an example. The voice of the interlocutor is recognized by the voice recognition unit 31 and "apple"
Is generated and passed to the dialog management unit 32. Here, the dialog management unit 32 instructs the utterance content generation unit 33 to check the content of the text data TXT1. That is, the utterance content generation unit 33 includes:
A text for confirmation of "(text data TXT1) is registered as a fixed sentence, and the dialog management unit 32 gives an instruction to the utterance content generating unit 33 to use the fixed sentence. In this example, since (text data TXT1) = (apple), the utterance content generation unit 33 generates the text data TXT2 of "I'm an apple", and responds to the interlocutor through the voice generation unit 34 and the speaker 12. When the interlocutor gives an affirmative response to the robot 1 saying “Is an apple” from the speaker 12 (for example, “Yes”, “Yes”, “Yeah”), the content of the interlocutor's utterance is changed. You can confirm that it was "apple". That is, the robot 1 repeats the utterance of the interlocutor and confirms the word. Here, if the speech recognition unit 31 has a learning function, it is possible to make the speech recognition unit 31 learn the characteristic parameters of the pronunciation of the interlocutor by repeating the repetition for each word. The recognition rate of word recognition can be increased.

When there is a positive response from the interlocutor,
By performing the process of the procedure shown in FIG. 3, the dialog management unit 32 extracts “go” from the ending of the word spoken by the talker, and extracts a word having “go” as the head from the dictionary of the knowledge base IB. I do. In this example, “gorilla” is extracted, and the dialog management unit 32 instructs the utterance content generation unit 33 to generate the text data TXT2 of “gorilla”. Here, the words "apple" and "gorilla" are registered in the above-described storage device.
Thus, the robot 1 outputs the voice of the word “gorilla” from the speaker 12. In the same manner, the above processing is repeated until one of the interlocutor and the robot 1 loses or the interlocutor instructs the end.

Next, "schedule management" will be described. In the “schedule management”, the voice recognition unit 3
1 to the text data TX given to the dialogue management unit 32
It is activated by determining whether or not T1 includes a keyword related to a schedule. For example,
When text data TXT1 including concepts such as “get up” and “drink medicine” is input to the dialogue management unit 32, the dialogue management unit 32 recognizes the information as schedule information. Here, the alarm function is exemplified. In order to realize this function, although not shown in FIG.
Is provided with clock means 51 (see FIG. 6) for measuring the current date and time. Now, if the interlocutor says to the robot 1 "wake up at 5 o'clock", the dialog management unit 32
Then, by performing syntactic analysis and context analysis using the knowledge base IB, "5:00 o'clock" is understood as time, and "wake up" is understood as an instruction of an alarm clock set. That is, in the knowledge base IB, the knowledge that "(numeral)" of "(numerical) time" represents time is set, and an alarm clock is set with reference to such knowledge.

By the way, in the word recognition in the voice recognition unit 31, when a plurality of word candidates are extracted, those having a high degree of similarity are extracted. There is also. For example,
"5 o'clock" may be erroneously recognized as "for construction". In this case, the text data TX input to the dialog management unit 32
T1 is "get up for construction". Since the dialog managing unit 32 analyzes the meaning of the input text data TXT1 from the syntax and the context, the knowledge base IB
, It is determined that “wake up” after “to construction” cannot be combined and meaning analysis cannot be performed. In such a case, the speech recognition unit 31 determines that the recognition is erroneous, and can combine before "wake up" and a candidate having a similar phoneme as "to construction" and combining after "to construction" A candidate that is similar to “wake up” and has similar phonemes is extracted, and temporary text data is generated for a candidate that has a meaning. Now
It is assumed that temporary text data of “wake up at 5:00” is generated by replacing “to work” with “at 5:00”. Since the text data generated in this way is temporary, the text data is verified by confirming with the interlocutor. That is, in the same manner as in the case of “removing”, the text data TXT of the question sentence for confirming the temporary text data
2 is instructed to generate the utterance content 2. For example, since the tentative text data is "wake up at 5 o'clock", the utterance content generation unit 33 generates a question sentence "set an alarm at 5 o'clock", and uses this text data TXT2 to inform the interlocutor. Confirm by voice. If there is a positive response from the interlocutor to this question, the wake-up is set at 5:00 assuming that the provisional text data was correct. If the response is negative to the provisional text data, the robot 1 outputs a voice such as "Please say again slowly". Through such a dialogue, the interlocutor can give a simple schedule management instruction to the robot 1 by voice.

Here, when words that cannot be combined by syntactic analysis or semantic analysis in the dialog management unit 32 are combined, provisional text data is generated on the assumption that there is a possibility that the words have been erroneously recognized during word recognition. In addition, the process of generating a question sentence and confirming it by voice is performed not only in schedule management but also in ordinary dialogue. That is, a kind of repetition processing is performed. Further, by repeating and confirming, the robot 1 can learn using the response of the interlocutor as the teacher signal, and the recognition rate of the interlocutor's voice can be increased.

In this embodiment, an example has been described in which the function as the conversion means for converting the video data into a format which can be displayed on the portable terminal 2 is provided in the video server 4. However, the video processing means 22 of the robot 1 and the communication means are provided. 23 may be provided with a function as a conversion unit for converting the video data into a format that can be displayed on the mobile terminal 2. Further, the communication means 23 does not necessarily need to be built in the robot 1, and the communication means 23 connected to the communication line 23 and the robot 1 may be connected by forming a wireless communication path in the house. .

(Second Embodiment) In the first embodiment, the robot 1 does not move like a dog even though it is a dog type.
4 has an appropriate joint, and as shown in FIG. 4, an example in which a dog-like operation is enabled by controlling a robot driving unit 24 including a plurality of actuators to bend and stretch or rotate the joint. . Robot drive unit 24
Is connected to the signal processing unit 20 through the robot control unit 25, and the robot driving unit 24 is controlled according to an instruction from the signal processing unit 20. The robot control unit 25 converts an instruction from the signal processing unit 20 into an operation of the robot driving unit 24. That is, the vocabulary (word) of the command to the robot can be extracted by the voice recognition unit 31 of the dialog processing unit 21. When the dialog management unit 32 recognizes that the command is the command to the robot 1, the analysis result of the command content is recognized by the robot control. Hand over to means 25. The robot controller 25 determines the operation amount of each actuator of the robot drive unit 24 according to the analysis result of the instruction content. For example, regarding the movement of the robot 1, commands such as "turn right", "turn left", "slight right", and "slight left" can be recognized, and ambiguous words such as "slight" can be handled. There is.

Further, in this embodiment, the direction of the optical axis and the focal length of the TV camera 13 can be changed without moving the robot 1. That is, the camera control means 2 having a function of shaking the TV camera 13 right and left and a function of driving a zoom lens provided in the TV camera 13.
6 are provided. The instruction to the camera control means 26 is given through the signal processing unit 20. The signal processing unit 20 controls the field of view of the TV camera 13 by giving an instruction by a voice of an interlocutor to the camera control unit 26. For example, when controlling the field of view of the TV camera 13 by the voice of the interlocutor, the interactive processing unit 21 performs “zoom in”, “zoom out”, and “slightly right” as in the case of instructing the movement of the robot 1. , "Slightly left", etc., are made recognizable, and T
The focal length and direction of the V camera 13 are controlled. Since the TV camera 13 according to the present invention is mainly intended to capture an image of an interlocutor,
In, the position of a person's face is recognized based on the video signal from the TV camera 13, and when zooming in, the extent that the upper part of the person's chest enters the field of view is maximized; It is desirable to minimize the degree of entry. In addition, for the command of "slightly right" and "slightly left", it is desirable that the left and right of the person be within the field of view.

Command for moving robot 1 and TV
The command related to the control of the camera 1 can be given from the portable terminal 2 through the communication line 3 in addition to the instruction by the voice of the interlocutor. However, since a command is given to the robot 1 from the outside through the communication line 3, authentication work based on identification information is necessary. For example, by using a personal identification number, it is possible to prevent a person other than a specific person from giving a command to the robot 1. is there. It is possible to give a command from the mobile terminal 2 to the robot 1 by a combination of numbers. The technique of giving an instruction by using a command can be used as it is.

By the way, if the configuration for giving a command from the microphone 11 to the robot 1 and the configuration for giving a command from the portable terminal 2 are used together, the robot 1 may receive mutually contradictory commands almost simultaneously. In this case, even if the robot 1 sequentially processes the commands, the robot 1 performs an operation corresponding to the later processed command. That is, the operation that was performed. Therefore, when giving an instruction to the robot 1, if one of the instructions is executed, the other instruction is not accepted for the time being (the time for completing one operation for the instruction), and the other instruction is not accepted. Is informed to the other that the one instruction is being executed. For example,
If the interlocutor interacting with the robot 1 is executing a command from the mobile terminal 2 when trying to give a voice command, a voice message indicating that “operation is being performed from the mobile terminal” is output through the speaker 12. I do. Conversely, mobile terminal 2
When the interlocutor interacting with the robot 1 is executing the command given by voice when the interlocutor operating the mobile terminal 2 gives the command, a message indicating that “the operation is being performed by the dialogue” is transmitted to the mobile terminal 2. Display on the screen. If the mobile terminal 2 gives a voice command to the robot 1,
A voice message indicating “during operation by dialogue” may be output through the mobile terminal 2. As described above, when the robot 1 is operating in accordance with an instruction from one of the interlocutor and the portable terminal 2, a notification that the instruction is not accepted is issued to the other instructed side of the instruction. like,
The signal processing unit 20 is provided with a notifying unit (not shown).

In the present embodiment, the microphone 11 and the T are used as interfaces for obtaining information from the interlocutor.
A contact sensor 14 is provided in addition to the V camera 13.
The contact sensor 14 is arranged at an appropriate position near the surface of the robot 1, and the output of the contact sensor 14 is
The signal is input to the signal processing unit 20 in the same manner as the output of the V camera 13. Thus, when the contact sensor 14 detects that the interlocutor has touched the robot 1, the dialog processing means 21 is activated. That is, if the dialog processing means 21 is constantly operated, the robot 1 responds to the surrounding noise.
It is desirable that the dialog processing means 21 be activated only during the period during which the interlocutor is speaking, but it is not possible to predict when the interlocutor will speak. A technology that activates the dialogue processing means 21 when a voice louder than the surrounding noise is input to the microphone 11 can be considered. Become. Further, the interactive processing means 21
However, even if the speaker recognition technology is used, it is difficult to completely specify a speaker only by voice using the current technology, and therefore, there is a possibility that surrounding noise is erroneously recognized as a speaker.

Therefore, in the present embodiment, when the contact sensor 14 detects that the interlocutor has touched the robot 1, the interactive processing means 21 is activated. Can be almost avoided.

In the present embodiment, as in the first embodiment, personal data is transferred when a video stored in the video server 4 is viewed from the portable terminal 2. However, the communication means 28 is provided with a table for associating the movement of the robot 1 with the personal data. If the robot 1 is of a dog type as shown in FIG. 5, the head 1a and the tail 1b are associated with each personal data. . That is, according to the personal data, the head 1a
And the action of shaking the tail 1b. This allows the interlocutor to intuitively know who accessed the video server 4. In addition to using the operation of each unit of the robot 1 as described above as a notification unit for identifying personal data, it is also possible to use a display lamp in which colors are associated with personal data as the notification unit.

By the way, the robot 1 in this embodiment
Is premised on connection to the Internet, and the communication means 23 includes an e-mail processing unit 41 and an Internet search processing unit 42. The process of exchanging e-mails and searching for information via the Internet can be instructed by the interlocutor to the robot 1 by voice through the microphone 11, as in the case of "scrutiny" and "schedule management".

Now, an example in which electronic mail is exchanged with the portable terminal 2 will be described. Here, it is assumed that the mail address of the transmission destination is set in the e-mail processing unit 41. When an e-mail is transmitted from the interlocutor interacting with the robot 1 to the portable terminal 2, the interlocutor informs the robot 1 of “I want to transmit the e-mail to XX” by voice, and the e-mail processing unit 41. Will launch and you will be able to enter the body of the email. Here, when the interlocutor prepares the body of the e-mail by voice and gives an instruction to the robot 1 by voice such as "send the e-mail", the robot 1 reads out the body and confirms the content. When the interlocutor accepts, the created e-mail is stored in the mail server of the Internet provider. The e-mail stored in the mail server is the same as a normal e-mail, and can be read by the mobile terminal 2.

The robot 1 periodically accesses the mail server in order to read an e-mail addressed to the interlocutor interacting with the robot 1 from the mail server. Even when the instruction is given to 1, the electronic mail can be read. When the electronic mail is read from the mail server in the electronic mail processing unit 41, the text of the electronic mail is input to the voice generating unit 34 as text data TXT2.
Therefore, the body of the e-mail is read out and transmitted to the interlocutor by voice.

The internet search processing section 42 is used for keyword search. The Internet search processing unit 42 has a function of connecting to a preset homepage for search. The homepage has a search engine and can transfer search results as text data. That is, a dedicated search homepage is provided for the robot 1. This search homepage basically has a telephone directory function, and has a function of searching for a trader of a desired business type in a district desired by the interlocutor. For example, when the interlocutor tells the robot 1 by voice "search for a courier company", the dialog management unit 32 recognizes that a search on the Internet is required, and searches for the courier company through the Internet search processing unit 42. I do. The search result is given as text data TXT2 to the voice generating unit 34, and the searched delivery company names are read out by voice in order. Here, when the number of search results is large, the Internet search processing unit 42 instructs the utterance content generation unit 33 to promote the narrowing down.
Search results are communicated to the interlocutor by voice. further,
When the interlocutor selects a desired courier and notifies the robot 1 by voice, the telephone number of the courier is read from the homepage and transmitted to the interlocutor by voice.

In each of the above-described embodiments, the robot 1 operates when the interlocutor gives an instruction to the robot 1. However, in the pet robot described as the conventional configuration, the robot 1 is operated by the autonomous agent.
Has a function of automatically starting and talking to the interlocutor, and also has a multimodal dialog function of performing voice recognition using information such as the expression and gesture of the interlocutor. By adding these functions, a smoother conversation between the interlocutor and the robot 1 becomes possible. Other configurations and operations are the same as those of the first embodiment.

(Third Embodiment) In the present embodiment, the first
In the embodiment of the present invention, a function is added to enhance convenience. The functions added in the present embodiment can be applied to the second embodiment.

In the present embodiment, as shown in FIG.
The date and time when the image was captured is added to the image captured by the camera 13.
Clock means 51 for counting the current date and time to add
The date and time measured by the clock means 51 are recorded on the TV camera 13.
Superimposed display as characters on the captured video
And a superimpose processing means 52 for performing the operation. sand
That is, the image as shown in FIG.
The imaged date and time when the image is captured and timed by the clock means 51
It was 15:23 on November 15, 2000
Then, as shown in FIG.
On the top or bottom of the video
15:23 ", etc.
Are synthesized. The video with such date and time is stored on the video server
4, the screen of the mobile terminal 2 also has T
The date and time when the video was captured by the V camera 13 is displayed.
become. That is, the interlocutor is imaged by the TV camera 13.
From the point in time, an image is displayed on the screen of the mobile terminal 2 and the
Since there is a time lag before the situation is confirmed,
By displaying the date and time when the video was captured on screen 2
To ensure at least the state of the interlocutor at that date and time.
And become possible. The superimpose processing means 52
Is appropriately programmed by the video processing means 22 comprising a microcomputer.
May be implemented by adding a system, but using a dedicated integrated circuit
With this configuration, the load on the video processing unit 22 is reduced.
You.

Although the storage capacity of the video server 4 is very large, if the number of robots 1 increases, the amount of video data becomes enormous. Instead of storing all video data, video data viewed on the mobile terminal 2 is deleted in principle. On the other hand, in the case where some trouble occurs in the interlocutor and there is a large time difference between the time of occurrence of the trouble and the time when the video is viewed on the portable terminal 2, if the trouble occurs before the occurrence of the trouble There is a case where a history of video data of a straddling period is required. As described above, since the video data stored in the video server 4 is sequentially deleted, it may not be possible to sufficiently cope with such a request. Further, since the video is transferred from the robot 1 to the video server 4 through the communication means 23, depending on the state of the communication line 3 and the status of the video server 4, there may be cases where the video data is not stored in the video server 4.

Therefore, in the present embodiment, the storage means 53 for storing the video data to be transferred to the video server 4 through the communication means 23 is added to the robot 1. The storage means 53 is capable of retaining the stored contents even when the power is turned off. This type of storage means 53 includes a flexible disk, a hard disk, and a CD.
-R, various types such as a nonvolatile memory can be used. The information stored in the storage unit 53 includes the video data transmitted from each communication unit 23, the date and time of transmission of each video data, and data on the success or failure of transmission.
For example, assign a unique name to each video data to be transmitted,
Data having a format such as “transmission date and time, name of video data, success / failure of transmission” may be stored together with the video data. Further, the individual name given to the video data may be set using the transmission date and time. For example, if the transmission date and time is 2
If it is 8:13 on November 13, 000, the name of the video data may be "200011130811a" or the like. Thus, by accumulating the video data transmitted from the communication means 23 and the transmission history in the robot 1, it is possible to browse the video data retrospectively as necessary. Here, regarding the video data and transmission history stored in the storage unit 53, the storage unit 53
By removing the recording medium from, the data can be read by another personal computer or the like.

The above description is based on the premise that the interlocutor is reliably imaged by the TV camera 13. However, in actuality, when the robot 1 is tilted or when the interlocutor
In some cases, it does not exist near the center of the third visual field. Therefore,
In the following, a technique for appropriately capturing an image of the interlocutor with the TV camera 13 will be described.

If the robot 1 is installed at an angle or if the interlocutor is holding the robot 1, the image captured by the TV camera 13 may be tilted. Therefore,
In the present embodiment, a posture sensor 54 for detecting the inclination of the robot 1 is added. A gyro sensor can be used as the attitude sensor 54. The tilt of the robot 1 detected by the attitude sensor 54 is given to the video processing unit 22, and the tilt of the video is corrected by the video processing unit 22 so that the video is erected. Here, the image being erect means that a plane orthogonal to the direction of gravity is parallel to the lower edge of the image. Also, it is only necessary to correct the left and right inclination of the video, and the up and down inclination is not necessarily corrected.
It is possible to obtain an image similar to the state where the camera 13 is installed on a horizontal plane. For example, when it is determined that the image is rotated 30 degrees counterclockwise by the inclination of the robot 1 detected by the posture sensor 54, the image processing unit 2
In 2, the image is rotated clockwise 3 around the center of the screen.
Correction for rotating by 0 degrees is performed, and the subsequent processing is performed using the corrected image.

Further, in order to capture the interlocutor within the field of view of the TV camera 13, a swinging driving means 55 for changing the direction of the TV camera 13 and a required image to be picked up by the TV camera 13 are used as a template in advance. There is provided a pattern processing means 56 for controlling the swing driving means 55 so as to increase the degree of similarity when the image registered by the TV camera 13 is compared with the template. The swing driving means 55 is provided so as to change the direction of the optical axis of the TV camera 13.
If the robot 3 is disposed on the head of the robot 1, the position or the direction of the head of the robot 1 may be movable by the swing driving means 55. Further, the swing drive means 56 may be configured so that only the TV camera 13 can move with respect to the main body of the robot 1 without moving the robot 1 in particular.

On the other hand, the pattern processing means 56 checks the image taken by the TV camera 13 with the image registered as a template, and swings the driving means in the direction in which the similarity becomes maximum in the range of the obtained image. Drive 55 to T
The direction of the optical axis of the V camera 13 is changed. For example, when a video as shown in FIG. 8A is registered as a template, the video of FIG. 8C has a higher similarity than the video of FIGS. 8B and 8D. Therefore, the swing driving means 55 changes the direction of the TV camera 13 so that an image as shown in FIG. If an appropriate template of the interlocutor is registered in the pattern processing unit 56 in this manner, the interrogator can always be captured by the TV camera 13 by appropriately controlling the swing driving unit 55. Note that interlocutors do not maintain the same attitude,
The swing drive unit 55 may be controlled by extracting features relating to the shape, the moving speed, and the like, and using the knowledge for determining whether or not the person is a person in combination with the template.

The pattern processing means 56 cannot determine how to drive the swing drive means 55 unless there is an interlocutor in the video imaged by the TV camera 13. Therefore, it is possible to know the direction in which the interlocutor is present even by the sound level by using a microphone 11 having strong directivity. That is, in the present embodiment, the microphone 11 functions as a second voice input device. The direction in which the directivity of the microphone 11 is maximized is set to be parallel to the direction of the optical axis of the TV camera 13, and the swing drive unit 55 keeps the positional relationship between the microphone 11 and the TV camera 13 fixed. And the direction of the TV camera 13 is changed. The sound level of the sound input from the microphone 11 is detected by the sound level processing means 57, and when the similarity is less than or equal to a predetermined value in the pattern processing means 56, the sound level detected by the sound level processing means 57 is increased. It is configured to drive the swing drive means 55. In short, while the direction of the microphone 11 is being changed by the swinging driving means 55, FIG.
(A) Assuming that the state where the sound level is relatively low as shown in FIG. 9 (c) changes to the state where the sound level is relatively high as shown in FIG. 9 (b), the microphone 1 in the state shown in FIG.
It can be considered that the interlocutor exists in a direction in which the directivity of 1 is maximized.

For example, the voice of the interlocutor is
1, the direction of the microphone 11 is changed by the swing driving means 55 while the voice of the interlocutor is being input, and the microphone is input in such a direction that the sound level of the voice input to the microphone 11 becomes almost maximum. 11 is aimed. At this time, the optical axis of the TV camera 13 is also oriented in the same direction, and the direction in which the sound level of the sound input to the microphone 11 increases is likely to be the direction in which the interlocutor is present. This increases the possibility that the interlocutor can be captured within the field of view of the camera 13. In addition,
In the case of using a technique for capturing the interlocutor using the directivity of the microphone 11, the pattern processing unit 56 can be omitted. In addition, a microphone having strong directivity used for capturing the interlocutor can be provided separately from the microphone 11 used for the dialog with the interlocutor.

Further, as shown in FIG. 10, a half mirror 58 may be arranged in front of the TV camera 13 so as to be orthogonal to the optical axis Ax of the TV camera 13. By arranging the half mirror 58 in front of the TV camera 13 in this manner, the half mirror 58 reflects the figure of the talker himself so that the talker can enter the field of view of the TV camera 13 without fail. In other words, it can be ensured that the interlocutor is captured in the field of view of the TV camera 13.
In short, the figure of the interlocutor is reflected by the half mirror 58 and reflected on the half mirror 58, and the interlocutor can be imaged by the TV camera 13 by the light transmitted through the half mirror 58.

In order to confirm by itself that the interlocutor is within the field of view of the TV camera 13, the video processing means 2
A video output means 59 for outputting the video signal handled in 2 to a normal television receiver may be added. In this case,
If a television receiver is connected to the video output means 59, the interlocutor himself can confirm on the screen of the television receiver that the interlocutor is being imaged by the TV camera 13. Further, the interlocutor can recognize not only his / her own figure but also the surrounding state such as the indoor state on the screen of the television receiver on the spot. That is, the television receiver can be used as a display device for checking images. Further, the video output means 59 does not necessarily need to be capable of displaying a video on the screen of a television receiver, as long as it has a display device capable of displaying a video captured by the TV camera 13 It is also possible to output an image to a display device of a personal computer or the like. It is desirable to provide the robot 1 with a function that allows the interlocutor to delete the video in case the video output to the display device is unsatisfactory for the interlocutor. It is desirable that this function can be instructed to the robot 1 by voice.

As described above, the robot 1 has a dog-like appearance, and can be used as a so-called pet robot. Therefore, by forming the outer shell such as a stuffed animal and making it lightweight so that the interlocutor can hold the robot 1, the interlocutor can hold the robot 1 as necessary. If the robot 1 is held by the interlocutor, the robot 1 comes into contact with the interlocutor, so that it is possible to obtain information about the interlocutor not only by voice and video but also by a sense of touch. Therefore, in the present embodiment, a heartbeat microphone 60 as a heartbeat sensor capable of detecting a heartbeat sound is arranged on the chest or abdomen of the robot 1, and the heartbeat microphone 60 is used by the heartbeat microphone 60 when the user holds the robot 1. The sound is collected.

Since the output of the heartbeat microphone 60 has a waveform as shown in FIG. 11, for example, the output of the heartbeat microphone 60 is sampled, and the characteristic of the waveform is extracted to determine whether the heart rate or the heartbeat waveform is abnormal. Etc. can be detected. However, the heartbeat microphone 60 does not necessarily detect only the heartbeat sound, and the output of the heartbeat microphone 60 contains many noise components, so that appropriate filtering is necessary.

Information obtained based on the output of the heartbeat microphone 60 is transferred to the video server 4 by superimposing and displaying, for example, characters representing the heart rate in the video. Alternatively, the heartbeat sound collected by the heartbeat microphone 60 may be compressed, attached to the video data, transferred to the video server 4, or stored in the storage unit 53. This makes it possible to obtain an indication of the health status of the interlocutor. When recording the heartbeat sound in the storage unit 53, the storage unit 53 is also used as a heartbeat recording device. Further, a heart rate recording device for recording a heartbeat sound may be provided separately from the storage means 53. Since the heartbeat sound cannot be collected unless the interlocutor holds the robot 1, it is desirable to instruct the robot 1 to hold the robot 1 by a voice message from the robot 1. It is desirable to generate such a voice message at every predetermined time,
For example, detecting daily heartbeat sounds immediately after the wake-up time is useful for managing the health condition.

(Fourth Embodiment) In the third embodiment, a microphone 11 having directivity is used as a microphone 11 in order to capture an interlocutor in the field of view of the TV camera 13, and one microphone 11 is used. Has been controlled in such a direction that the sound level of the sound detected in step (1) increases, in the present embodiment, as shown in FIG. 12, two microphones 11a and 11b are used and both
11b based on the phase difference detected at 11b
5 is controlled. That is, the microphones 11a and 11b function as a third voice input device. The two microphones 11a and 11b are arranged corresponding to the left and right ears of the robot 1, and when the distance between the interlocutor as the sound source and each of the microphones 11a and 11b is equal, as shown in FIG. Both microphones 11a, 11
The phase difference between the outputs b is zero. Also,
When the interlocutor is present at a position where the phase difference between the outputs of the two microphones 11a and 11b becomes zero, the two microphones 1 are positioned so that the interlocutor is positioned at the center of the visual field of the TV camera 13.
The positional relationship between 1a, 11b and the TV camera 13 is set.

Therefore, when the interlocutor is located on the left side of the field of view of the TV camera 13, the output phase of the left microphone 11a leads the output phase of the right microphone 11b as shown in FIG. That means
Conversely, when the interlocutor is present on the right side with respect to the field of view of the TV camera 13, the output phase of the right microphone 11a leads the output phase of the left microphone 11b as shown in FIG. . Therefore, a phase difference detecting means 61 for detecting the phase difference between the outputs of the microphones 11a and 11b is provided, and the swinging drive means 55 is controlled in such a direction that the phase difference becomes smaller, so that the interlocutor can view the field of view of the TV camera 13. TV camera 13 so that it is located near the center
It is possible to adjust the direction.

Further, in the present embodiment, the direction in which the sound source exists is obtained from the phase difference between the outputs of the two microphones 11a and 11b. It becomes possible to grasp accurately. Other configurations and operations of this embodiment are the same as those of the other embodiments described above.

[0085]

According to the first aspect of the present invention, a voice input device and a voice output device are connected to each other, and a dialogue processing means for performing a voice dialogue with a talker using a natural language, and a video image of the talker. An image processing means for outputting video data obtained by digitizing a video image captured by the video input device to which the input device is connected, a communication means for transmitting the video data to a network,
A mobile terminal that is permitted to view the video data of the interlocutor through the network; and a conversion unit that converts the video data into a format that can be displayed on the mobile terminal. Simulates the spiritual care of loneliness. Also, since the viewing of the video data of the interlocutor is possible only with the mobile terminal that is permitted, by using the mobile terminal so that only the family of the interlocutor can carry,
The image of the interlocutor can be transferred to the portable terminal within a range that does not violate the privacy of the other person, and the presence or absence of the interlocutor's abnormality can be monitored. As a result, even in a home that has the elderly and the sick, the state of the elderly and the sick can be monitored from a distance using a portable terminal, and the user can go out with peace of mind.

According to a second aspect of the present invention, in the first aspect of the present invention, the conversion means is provided in a video server for storing video data transferred via a network, and the video server is allowed by a portable terminal holder. In this case, only the video data is sorted and transferred to the portable terminal. By providing the video server for storing the video data of the interlocutor, the video data can be browsed retroactively.

According to a third aspect of the present invention, in the first aspect of the present invention, the conversion means is provided in one of the video processing means and the communication means. Can be browsed from a portable terminal, and the system configuration is simple.

According to a fourth aspect of the present invention, in the first to third aspects of the present invention, the video processing means transmits the video data of the interlocutor to the network through the communication means in accordance with an instruction from the portable terminal. Since the transmission of the video data is instructed from the mobile terminal, the video at the time of the request by the holder of the mobile terminal can be viewed.

According to a fifth aspect of the present invention, in the first to fourth aspects, the video processing means operates in response to a command instructed by the interlocutor through the interactive processing means. Since the user is allowed to shoot the video image by himself / herself, for example, when the interlocutor feels abnormal, it becomes possible to send the video data to notify the portable terminal owner.

According to a sixth aspect of the present invention, in accordance with the first to fifth aspects of the present invention, it is specified whether the video processing means is operated by an instruction from the interlocutor or is operated by an instruction from the portable terminal. It is provided with a notification means for notifying the video processing means from both the interlocutor and the portable terminal, but the instructions of the two are not confused.

According to a seventh aspect of the present invention, in the first to sixth aspects of the present invention, a contact sensor for detecting a contact of the interlocutor is added, and when the contact sensor detects the contact of the interlocutor, the interaction processing means is provided. Is activated, and the dialogue processing means is activated by contacting the contact sensor when the dialogue person tries to talk, and the dialogue processing means malfunctions due to ambient noise or voice of another person other than the dialogue person. Can be prevented.

The invention according to claim 8 is the invention according to any one of claims 1 to 7, further comprising means for indicating to the interlocutor that the image of the interlocutor has been photographed by the image input means. Since it is possible to know when is photographed, it is possible to take measures such as moving to a place where photographing is not performed when the user does not want to be photographed for privacy reasons.

According to a ninth aspect of the present invention, in the first to eighth aspects, the interaction processing means has a function of asking the interlocutor to confirm a voice input by the interlocutor. If a positive response is received, the voice input by the interlocutor is determined. Since the voice input of the interlocutor is confirmed by repetition, the voice generated by the interlocutor can be surely understood.

According to a tenth aspect of the present invention, in the ninth aspect, the interaction processing means learns a voice generated by the interlocutor using a positive response from the interlocutor as a teacher signal. The recognition rate increases by learning the voice generated by the interlocutor through the dialogue.

The invention of claim 11 is the invention of claim 1 to claim 10, further comprising means for transferring video data browsed by the portable terminal to another terminal on the network. Is transferred to another terminal, for example, when the owner of the portable terminal notices an abnormality of the interlocutor, it is possible to receive the advice of coping by sending the video data to the doctor.

According to a twelfth aspect of the present invention, in the first to eleventh aspects of the present invention, there is provided notifying means for notifying the interlocutor that browsing of video data has been requested from the portable terminal. Even in a home with a certain elderly or sick person, when the family uses a mobile terminal to monitor the state of the elderly or sick from a distance, the family confirmed that the elderly and sick were notified by means of notification. Since the notification is given, the elderly and the sick can be informed that the family has confirmed them, and a sense of security can be obtained.

According to a thirteenth aspect of the present invention, in the twelfth aspect, the notifying means has a function of notifying a viewer of the video data using the identification information of the portable terminal. You can know more, so you can feel more at ease.

According to a fourteenth aspect of the present invention, in the thirteenth aspect, the notifying means includes a notifying unit assigned to each viewer of the video data. Since the user can be identified, it is possible not only to know that the viewer has checked the video data, but also to intuitively know who viewed the video data.

A fifteenth aspect of the present invention is directed to a dialogue processing means for connecting a voice input device and a voice output device to perform a voice dialogue using a natural language with a user, and a video input device for imaging the user. A video processing unit that is connected and outputs video data obtained by digitizing a video captured by a video input device, a communication unit that sends video data to a network,
Conversion means for converting the video data into a format that can be displayed on a portable terminal that is permitted to view the video data of the interlocutor through the network. Simulating gives the interlocutor mental care for loneliness. Also,
Since the video data of the interlocutor can be viewed only on the authorized mobile device, using the mobile device so that it can be carried only by the interlocutor's family can be performed within a range that does not violate the privacy of others. The image of the interlocutor can be transferred to the portable terminal, so that the presence or absence of the anomaly of the interlocutor can be monitored. As a result, even in a home that has the elderly and the sick, the state of the elderly and the sick can be monitored from a distance using a portable terminal, and the user can go out with peace of mind.

According to a sixteenth aspect of the present invention, in the invention of the fifteenth aspect, the clock means for measuring the current date and time and the date and time measured by the clock means when the interlocutor is imaged are combined with the video data as characters. And superimpose processing means for sending from the communication means, and the date and time when the video was captured is attached to each video data.
When there is a large time difference between the date and time when the video was captured and the date and time when the video data was viewed by the mobile terminal, the viewer can check the date and time attached to the video data to obtain the latest video, etc. Can be dealt with.

In a seventeenth aspect of the present invention, in the sixteenth aspect of the present invention, the video data transmitted to the network through the communication means and the result of the success or failure of the communication are associated with the date and time measured by the clock means. A storage means for sequentially recording and accumulating and storing the recorded contents without power supply is added, and when the video data is not transmitted from the communication means and the video data cannot be confirmed by the portable terminal, Also, the health status of the interlocutor can be confirmed at a later date based on the video data stored in the storage means, and the result of communication success / failure is stored in the storage means as a transmission history, so that the communication means can be used during maintenance. Can be easily known.

According to an eighteenth aspect of the present invention, in the invention according to the fifteenth to seventeenth aspects, an attitude sensor for detecting a tilt of the image input device is added, and the image processing means converts an image picked up by the image input device. This is to correct the image based on the inclination detected by the posture sensor so as to erect it, and to correct so as to remove the inclination of the image based on the information obtained by the posture sensor. Even when the image capturing device is placed on a horizontal plane, for example, when the image capturing device is arranged in a place that is not horizontal or when an interlocutor lifts the image capturing device, Video can be obtained.

A nineteenth aspect of the present invention is the invention according to the fifteenth to eighteenth aspects, wherein a swinging drive means for changing the direction of the field of view of the video input device, and a talker near the center of the video input device Can be registered, and a pattern processing unit that controls the swinging driving unit so as to increase the similarity by comparing the template image with the image captured by the video input device is added. Since the orientation of the visual field of the video input device is controlled so that the similarity between the image captured by the video input device and the template is increased, if the appropriate template is set, the Of the video input device can be adjusted so that the image is captured near the center of the visual field of the video input device.

According to a twentieth aspect of the present invention, in accordance with the fifteenth to eighteenth aspects, the second audio input device in which the direction of the visual field of the video input device matches the direction having directivity, and the video input device Swing drive means for simultaneously changing the direction of the field of view of the device and the direction in which the second voice input device has directivity; The sound level processing means for controlling the driving means is added, and the direction in which the interlocutor is present can be estimated based on the sound level. Therefore, the visual field of the video input device is adjusted to increase the sound level. By doing so, it is more likely that the interlocutor can be captured within the field of view of the video input device.

In a twenty-first aspect of the present invention, in accordance with the fifteenth to eighteenth aspects, a plurality of the video input devices are arranged so that a phase difference of an output with respect to a sound source present in a direction of a visual field becomes zero. And a phase difference detecting means for controlling the swinging drive means in a direction to reduce the phase difference between the third voice input apparatus and the third voice input apparatus. By adjusting the field of view of the video input device to reduce the phase difference, it becomes possible to capture the interlocutor within the field of view of the video input device.

According to a twenty-second aspect of the present invention, in addition to the fifteenth to twenty-first aspects, a half mirror is provided in front of the video input device so as to be orthogonal to the optical axis direction of the video input device. In the state where the interlocutor reflects himself in the half mirror, the interlocutor is present in front of the video input device, so the interlocutor can be reliably imaged by the video input device according to the will of the interlocutor. It becomes possible to do.

[0107] According to a twenty-third aspect of the present invention, in accordance with the fifteenth to twenty-first aspects of the present invention, an image can be displayed on the display device to which an external display device can be connected and image data captured by the image input device can be displayed. The output means is added, and the interlocutor can check the image captured by the video input device on the spot by the display device connected to the video output means, so that the interlocutor can confirm the state of his own reflection. In addition to confirming whether or not an appropriate image has been captured, the user can also check the surroundings.

[0108] According to a twenty-fourth aspect of the present invention, in the invention of the fifteenth to twenty-third aspects, there is provided an outer shell capable of being held by the interlocutor. A heart rate sensor capable of detecting the heart rate and a heart rate recording device that records the heart rate sound detected by the heart rate sensor are added. It is possible to check the health status of the interlocutor, and to know the health status of the interlocutor more accurately by combining the information obtained about the interlocutor. In addition, since the heartbeat sensor is configured to collect the heartbeat sound when the interlocutor is holding the heartbeat sensor, the heartbeat sound can be collected without any trouble.

[0109] According to a twenty-fifth aspect of the present invention, in the invention of the fifteenth to twenty-fourth aspects, there is provided a notifying means for notifying the interlocutor of a request to view video data from the portable terminal. Even in a home with a certain elderly or sick person, when the family uses a mobile terminal to monitor the state of the elderly or sick from a distance, the family confirmed that the elderly and sick were notified by means of notification. Since the notification is given, the elderly and the sick can be informed that the family has confirmed them, and a sense of security can be obtained.

According to a twenty-sixth aspect of the present invention, in the twenty-seventh aspect, the notifying means has a function of notifying a viewer of the video data using the identification information of the portable terminal. You can know more, so you can feel more at ease.

According to a twenty-seventh aspect of the present invention, in the invention of the twenty-sixth aspect, the notifying means includes a notifying unit assigned to each viewer of the video data. Since the user can be identified, it is possible not only to know that the viewer has checked the video data, but also to intuitively know who viewed the video data.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a first embodiment of the present invention.

FIG. 2 is a block diagram showing a dialog processing means used in the embodiment.

FIG. 3 is an operation explanatory view showing an operation example of the dialog processing means according to the first embodiment;

FIG. 4 is a block diagram showing a second embodiment of the present invention.

FIG. 5 is a perspective view showing the appearance of the robot used in the above.

FIG. 6 is a block diagram showing a third embodiment of the present invention.

FIG. 7 is an operation explanatory diagram of the above.

FIG. 8 is an operation explanatory view of the above.

FIG. 9 is an operation explanatory view of the above.

FIG. 10 is an operation explanatory view of the above.

FIG. 11 is a configuration diagram of a main part of the above.

FIG. 12 is a block diagram showing a fourth embodiment of the present invention.

FIG. 13 is an explanatory diagram of the operation of the above.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Robot 2 Mobile terminal 3 Video server 4 Communication line 11 Microphone 11a, 11b Microphone 12 Speaker 13 TV camera 14 Contact sensor 20 Signal processing unit 21 Interactive processing means 22 Video processing means 23 Communication means 27 Display means 28 Notification means 51 Clock means 52 Superimpose processing means 53 Storage means 54 Attitude sensor 55 Swing drive means 56 Pattern processing means 57 Sound level processing means 58 Half mirror 59 Video output means 60 Heartbeat microphone 61 Phase difference detection means

Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat II (Reference) G10L 15/22 G10L 3/00 551H 5D045 H04N 7/15 571U 5K101 7/18 R (72) Inventor Takashi Nishiyama Kadoma, Osaka 1048 Ichidai Kadoma Matsushita Electric Works Co., Ltd.F-term (reference) 5C054 AA02 CC02 CD03 CE04 CG01 CH01 CH05 DA01 DA07 EA01 EA03 EA05 EA07 EH01 FA04 FC12 FF02 GA04 GB04 GD03 HA00 HA04 5C064 AA06 AB04 AC02 AC06 A08 AC12 BA01 CA09 CA28 CB26 CB36 DA08 DA14 DA33 EA45 FA06 FA18 5C087 AA02 AA08 AA10 AA24 AA25 AA37 AA44 BB12 BB20 BB46 BB65 BB74 DD03 DD24 EE05 FF01 FF02 FF04 FF16 FF19 FF23 GG02 GG20 GG02 GG20 GG02 GG01 GG02 GG01 GG02 GG20 GG20 GG20 NN08 NN13 NN18 NN21 RR12

Claims (27)

[Claims] An audio input device and an audio output device are connected to each other, and a dialogue processing means for performing a voice dialogue with a dialogue person using a natural language with a dialogue person is connected to a video input device for imaging the talker. Video processing means for outputting video data obtained by digitizing video captured by the device, communication means for sending video data to a network, a portable terminal permitted to view video data of an interlocutor through the network, A communication support system, comprising: conversion means for converting data into a format that can be displayed on a mobile terminal. 2. The image processing apparatus according to claim 1, wherein the conversion unit is provided in a video server that stores video data transferred via a network,
2. The communication support system according to claim 1, wherein the video server selects only video data permitted to the portable terminal holder and transfers the selected data to the portable terminal. 3. The communication support system according to claim 1, wherein said conversion means is provided in one of a video processing means and a communication means. 4. The image processing apparatus according to claim 1, wherein said video processing means transmits video data of the interlocutor to a network through a communication means in accordance with an instruction from said portable terminal. Communication support system. 5. The communication support system according to claim 1, wherein said video processing means operates in response to a command instructed by an interlocutor through said interactive processing means. . 6. A notifying means for notifying the instructed side whether the video processing means is operating according to an instruction from an interlocutor or operating according to an instruction from the portable terminal. A communication support system according to claim 5. 7. A system according to claim 1, wherein a contact sensor for detecting a contact of the interlocutor is added, and when the contact of the interlocutor is detected by the contact sensor, the dialog processing means is activated.
The communication support system according to claim 6. 8. The communication support according to claim 1, further comprising means for indicating to the interlocutor that a video of the interlocutor has been photographed by the video input means. system. 9. The dialogue processing means has a function of asking the interlocutor to confirm the voice input by the interlocutor, and confirming the voice input by the interlocutor when receiving a positive response from the interlocutor. The communication support system according to any one of claims 1 to 8, wherein: 10. The communication support system according to claim 9, wherein said dialogue processing means learns a voice generated by the interlocutor using a positive response from the interlocutor as a teacher signal. 11. The communication support according to claim 1, further comprising means for transferring video data browsed on the portable terminal to another terminal on the network. system. 12. The communication according to claim 1, further comprising a notifying unit that notifies the interlocutor of a request to view video data from the mobile terminal. Support system. 13. The communication support system according to claim 12, wherein said notifying means has a function of notifying a viewer of video data using identification information of said portable terminal. 14. The communication support system according to claim 13, wherein said notifying unit includes a notifying unit assigned to each viewer of the video data. 15. A voice input device and a voice output device are connected to each other, and a dialogue processing means for performing a voice dialogue with a talker using a natural language with a talker; Image processing means for outputting image data obtained by digitizing the image picked up by the device, communication means for transmitting the image data to a network, and display on a portable terminal permitted to view the image data of an interlocutor through the network A conversion unit for converting video data so as to have a simple format. 16. Clock means for measuring the current date and time, and superimposing processing means for synthesizing the date and time measured by the clock means as text into video data when the interlocutor is imaged and transmitting the video data from the communication means. The photographing apparatus according to claim 15, wherein is added. 17. The video data transmitted to the network through the communication means and the result of communication success / failure are sequentially recorded and stored in association with the date and time measured by the clock means, and the recorded contents are supplied without power supply. 17. The photographing apparatus according to claim 16, further comprising a storage unit for storing the image data. 18. An image processing apparatus, further comprising: a posture sensor for detecting a tilt of the image input device, wherein the image processing means detects an image based on the inclination detected by the posture sensor so as to erect the image captured by the image input device. 18. The photographing apparatus according to claim 15, wherein the correction is performed. 19. A video camera, comprising: a swing drive unit capable of changing a direction of a field of view of the video input device; and a template defining an image of an interlocutor near a center of the video input device. 19. The image processing apparatus according to claim 15, further comprising pattern processing means for controlling the swing driving means so as to increase the degree of similarity by comparing the image captured by the input device with the template. The imaging device according to claim 1. 20. A second audio input device in which the direction of the visual field of the video input device matches the direction having directivity, and the direction of the visual field of the video input device and the second audio input device have directivity. And a sound level processing means for controlling the swing driving means so as to increase the sound level of the output of the second voice input device. The imaging device according to any one of claims 15 to 18, wherein: 21. A plurality of third audio input devices arranged so that a phase difference between outputs with respect to a sound source present in the direction of the visual field of the video input device becomes zero, 19. The photographing apparatus according to claim 15, further comprising a phase difference detecting unit for controlling the swing driving unit in a direction to reduce the phase difference. 22. The apparatus according to claim 15, further comprising a half mirror disposed in front of the image input device so as to be orthogonal to an optical axis direction of the image input device. 2. The photographing apparatus according to claim 1. 23. An image display device according to claim 15, further comprising a video output means connectable to an external display device and enabling display of video data picked up by said video input device on said display device. Claim 2
The imaging device according to any one of claims 1 to 7. 24. A heart rate sensor having an outer shell capable of being held by the interlocutor, capable of detecting a heartbeat sound of the interlocutor when the interlocutor is holding the heartbeat sound, and a heartbeat sound detected by the heartbeat sensor The imaging apparatus according to any one of claims 15 to 23, further comprising: a heartbeat recording device for recording a heart rate. 25. The photographing apparatus according to claim 15, further comprising a notifying unit that notifies the interlocutor that a request for browsing video data from the portable terminal is provided. apparatus. 26. The photographing apparatus according to claim 25, wherein the notifying unit has a function of notifying a viewer of the video data using identification information of the mobile terminal. 27. The photographing apparatus according to claim 26, wherein the notifying unit includes a notifying unit assigned to each viewer of the video data.