JP2011097531A - System for continuing listening interaction - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a video phone for continuing an interaction between persons handicapped in communication. <P>SOLUTION: A listening interaction continuation system 100 includes a monitor 16a, a PC 14a to which a microphone 20a and a monitor camera 22a are connected, and a robot 10a including a belly camera 12a. In the PC 14a, the action of a user A is determined based on an image of the user A captured by the monitor camera 22a and the belly camera 12a, and the sound of the user A collected by the microphone 20a, and stored in a memory. In the PC 14a, a condition of the user A is recognized based on action data for a first predetermined time. When the condition of the user is recognized as "active talk monitor", the PC 14a gives an operating instruction to the robot 10a to perform simulated listening to the user A. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a listening dialog sustaining system, and more particularly to a listening dialog system that performs a dialog using, for example, a videophone.

In the interactive annotation system disclosed in Patent Document 1, a videophone used by a care recipient who is a dementia patient is connected to a computer used by a caregiver via a network. A server for transferring commands between the computer and the videophone is connected to the network, and a database for storing picture image data is coupled to the server. Also, the caregiver can display the videophone used by the partner (caregiver) by opening the URL of the server with a computer and designating the photo. Thereby, as a part of care, a caregiver and a cared person can have a conversation using a videophone.
JP 2007-150955 A [H04N 7/14]

  In recent years, attention activities have been focused on listening to stories of patients with mild dementia, elderly people, and people with speech impairments and providing emotional support. As a part of the listening activity, the listening volunteer is talking with a care recipient such as a patient with mild dementia as a caregiver of Patent Document 1. However, the number of listening volunteers is smaller than the number of caregivers seeking a conversation partner. Therefore, since patients with mild dementia and the like can also have a dialogue, activities for conducting a dialogue between patients with mild dementia who seek a talking partner are being promoted.

  However, in a dialogue between patients with mild dementia, the patient with mild dementia often lacks concentration on the dialogue, and the dialogue often does not continue.

  Therefore, the main object of the present invention is to provide a novel listening dialogue sustaining system.

  Another object of the present invention is to provide a listening dialog sustaining system that maintains a dialog between persons with communication disabilities.

  The present invention employs the following configuration in order to solve the above problems. The reference numerals in parentheses, supplementary explanations, and the like indicate the corresponding relationship with the embodiments described in order to help understanding of the present invention, and do not limit the present invention.

  A first invention is a listening dialogue sustaining system including a video phone and a robot including a first camera and a microphone, and based on an image captured by the first camera and a sound collected by the microphone, the user's Based on the determination means for determining the action, the recognition means for recognizing the user's state from the action for the first predetermined time determined by the determination means, and the conversation based on the user's state recognized by the recognition means It is a listening dialogue continuous system provided with the operation | movement provision means to operate a robot.

  In the first invention, the listening dialogue sustaining system (100) includes a first camera (22a, 22b) provided near the monitor (16a, 16b) and a microphone (20a, 20b) that collects a voice spoken by the user. And a stuffed toy robot (10a, 10b, 14a, 14b).

  The determination means (64, S65, S67) uses a result obtained by performing predetermined template matching processing and face recognition processing on the image captured by the first camera, and the sound level of the sound collected by the microphone. Based on this, the user's behavior is determined. The recognition means (64, S75-S81) recognizes the state of the user based on the action for the first predetermined time that is, for example, 30 seconds. The action giving means (64, S171, S173, S177, S179), for example, pseudo-listens the user's utterance, adjusts the user's utterance, or attracts the user's attention based on the user's state. The conversation is sustained by operating the robot.

  According to the first aspect, since the robot operates so as to maintain the dialogue according to the user's state, the dialogue between persons with communication disabilities can be sustained.

  A second invention is dependent on the first invention, and the recognition means includes a positive recognition means for recognizing positive and non-positive actions, and the action giving means is based on the recognition result of the positive recognition means. A listening operation giving means for causing the robot to perform a pseudo-listening with respect to.

  In the second invention, the aggressiveness recognition means (64, S75) is configured to actively respond to the dialogue based on, for example, the time when the user looks at the other party (WT), the forward tilt line of sight time (FT), and the frequency of conflict (RF). Recognize (active state) and inactive (passive state). The listening motion giving means (64, S171, S173, S201, S225, S265, S285, S305, S367, S435, S441, S443) simulates the user's utterance, for example, when the user is active in dialogue. Move the robot to listen.

  According to the second aspect of the invention, the user can feel as if he / she is listening to his / her story by the pseudo-listening of the robot, so that the dialogue continues.

  A third invention is dependent on the first invention or the second invention, and the recognition means further includes a speaker state recognition means for recognizing a listener side state and a speaker side state in a dialog, and the utterance of the user or the other party Measuring means for measuring time is further provided, and the action giving means further includes utterance control action giving means for operating the robot so as to control the user's utterance based on the utterance time measured by the measuring means.

  In the third invention, the speaker state recognition means (64, S77) recognizes the listening (listen state) and the utterance (talk state) in the dialogue based on the presence or absence of the user's utterance. The measuring means (64, S141, S143, S145) measures the utterance time of the user or the other party when, for example, the sound level is a predetermined value or more. The utterance control operation giving means (64, S207, S227, S297, S319) controls the user's utterance when the utterance time exceeds a threshold value, for example, in a state where one of the users is speaking unilaterally. Next, operate the robot.

  According to the third invention, since the utterances are controlled, the utterances are uttered in a well-balanced manner, so that the dialogue continues.

  A fourth invention is dependent on the third invention, and the utterance control means is an active utterance state that is recognized as active by the aggressiveness recognition means and is recognized as a speaker state by the speaker state recognition means, And the speech suppression operation | movement provision part which operates a robot is included so that a user's speech may be suppressed when speech time becomes more than a threshold value.

  In the fourth invention, the utterance suppression operation giving means (64, S297, S405, S411, S417, S423, S425) is configured such that the utterance time exceeds the threshold when the user is actively speaking. Then, the robot is operated so as to suppress the user's speech.

  According to the fourth invention, when the user is unilaterally speaking, the utterance is suppressed.

  A fifth invention is dependent on the fourth invention, and the utterance suppression operation giving means includes attention guiding means for operating the robot so as to guide the user's attention.

  In the fifth aspect of the invention, the attention guiding means (64, S425, S455, S461), for example, operates the robot so that the user and the monitor (16) on which the other party is displayed are alternately viewed so as to alert the user. Induce.

  According to the fifth aspect, since the user's unilaterally speaking user's attention is induced, the utterance is suppressed. In addition, when the user's attention is guided to the other party, the other party can have an opportunity to speak, and the dialogue is more sustained.

  The sixth invention is dependent on any one of the third to fifth inventions, and the utterance control means is recognized as positive by the positiveness recognition means and is recognized as the listening state by the speaker state recognition means. In the active listening side state, the apparatus further includes an utterance promoting action giving means for operating the robot so that the user's utterance is promoted.

  In the sixth invention, when the user actively listens to (listens to) the other party's story, the utterance promoting action giving means (64, S227, S319, S471), for example, If the threshold value is exceeded, the robot is operated so that the user's speech is promoted.

  According to the sixth aspect of the invention, speech is promoted if the user is actively listening to the other party for a long time.

  A seventh invention is according to any one of the second to fifth inventions, and the motion giving means causes the user to participate in the dialogue when recognized as inactive by the positiveness recognition means. It further includes a participation action giving means for operating the.

  In the seventh invention, the participation operation giving means (64, S177, S231, S319, S321) operates the robot so as to cause the user to participate in the dialogue if the user is inactive in the dialogue.

  According to the seventh aspect of the present invention, the dialog is maintained by allowing users who are not active in the dialog to participate in the dialog.

  An eighth invention is according to the seventh invention, and the participation motion giving means includes attention attracting means for operating the robot so as to attract the user's attention.

  In the eighth invention, the attention attracting means (64, S321, S335, S341, S343) attracts the user's attention by, for example, operating the robot so as to see or talk to the user.

  According to the eighth aspect of the invention, the user is allowed to participate in the dialogue by attracting the attention of the non-aggressive user.

  A ninth invention is dependent on the seventh invention or the eighth invention, and the participation motion giving means further includes speech prompting means for operating the robot so as to prompt the user to speak.

  In the ninth invention, the speech prompting means (64, S319, S471) prompts the user to speak by operating the robot so as to ask the user a question, for example.

  According to the ninth aspect of the present invention, the user who is not active in the dialogue is allowed to speak to participate in the dialogue.

  A tenth invention is according to any one of the first to ninth inventions, and transmits a network to which the robot is connected, a server connected to the network, and a user action determined by the determination means to the server. The transmission means and an acquisition means for acquiring the other user's action from the server are further provided, and the action giving means gives the robot a pseudo-listening to the other user based on the other user's action and the user's action acquired by the acquisition means. It further includes an opponent listening operation giving means to be performed.

  In the tenth invention, a robot and a server (24) are connected to the network (200) by, for example, a wireless LAN. A transmission means (64, S69) transmits a user's action to a server for every fixed time. The acquisition means (64, S269, S289, S309) acquires the other user's action transmitted to the server. If the other user is speaking, the other party listening operation giving means (64, S275, S295, S315, S385, S391, S393) performs, for example, a pseudo listening operation on the monitor on which the other user is displayed. Move the robot.

  According to the tenth aspect, even if the user is not listening to the other user's story, the robot may feel as if the other user is listening to his / her story because the robot listens to the other's story. it can.

  An eleventh invention is according to any one of the first to tenth inventions, wherein the recognition means further includes an interest object recognition means for recognizing a user's interest object, and the action giving means is the interest object recognition means. The robot further includes an interest motion giving means for operating the robot so that the user is interested in the dialogue based on the recognition result obtained by the above.

  In the eleventh invention, the object recognition means (64, S79) recognizes what the user is viewing as the user's object of interest. The interest motion giving means (64, S179, S241, S321, S335, S341, S343, S405) operates the robot so that the user is interested in the dialogue if the user is not interested in the robot, for example.

  According to the eleventh aspect of the present invention, the dialog is sustained by making the user interested in the dialog.

  A twelfth invention is according to the eleventh invention, wherein the robot further includes a face recognition unit that includes a second camera and performs face recognition processing on each image by the first camera and the second camera, The interest object recognition means includes first recognition rate calculation means for calculating a first recognition rate of the first camera from a face recognition result for a second predetermined time by the face recognition means, and face recognition for a second predetermined time by the face recognition means. A second recognition rate calculating means for calculating a second recognition rate of the second camera from the result; and a setting means for setting the recognition result based on the first recognition rate and the second recognition rate.

  In the twelfth invention, the robot is provided with the second camera (12) in the abdomen, for example. The face recognition means (64, S41) recognizes the user's face photographed by the first camera and the second camera. The first recognition rate calculating means (64, S123) calculates the first recognition rate (M) of the user's face taken by the first camera. The second recognition rate calculation means (64, S125) calculates the second recognition rate (S) of the user's face photographed by the second camera. And a setting means (64, S131, S133, S135) sets a user's interest object based on a 1st recognition rate and a 2nd recognition rate.

  According to the twelfth aspect, since the object of interest is set based on the recognition rates M and S of the user's face, the object the user is looking at can be accurately recognized.

  A thirteenth invention is according to the twelfth invention, and further comprises image transmission means for transmitting either the first camera image or the second camera image based on the object of interest set by the setting means.

  In the thirteenth invention, the image transmission means (64, S489, S491) transmits the image of the first camera if, for example, the user's interest is a videophone, and transmits the image of the second camera if the user is a robot. .

  According to the thirteenth aspect, even if the partner is in a state where the user is talking to the robot, an image of the user's face in front is sent to the partner. As a result, the other party feels as if they are speaking to them, and the dialogue continues.

  A fourteenth invention is dependent on any one of the first to thirteenth inventions, and the determining means is a posture determining means for determining the user's posture, an utterance determining means for determining the presence or absence of the user's utterance, A head direction determining means for determining a head direction, a gaze direction determining means for determining a user's gaze direction, a whirling determining means for determining the presence or absence of a user's whisper, and a conflict determining means for determining a user's conflict, The behavior is determined based on the posture determination result, the utterance determination result, the head direction determination result, the gaze direction determination result, the whirl determination result, and the conflict determination result.

  In the fourteenth invention, the posture determination means (64, S9), the head direction determination means (64, S25), the gaze direction determination means (64, S33) and the whirl determination means (64, S49) Determine based on. The utterance determination means (64, S17) determines based on the user's voice. Further, the conflict determination means (64, S59) determines based on the determination result of the user's speech and whisper. And a user's action is decided based on each judgment result.

  According to the fourteenth aspect, a plurality of determination results are included in the user's behavior data, so that the user's state can be recognized accurately.

  According to the present invention, since the robot operates so as to maintain the conversation according to the state of the user, it is possible to maintain the conversation between persons with communication disabilities.

  The above object, other objects, features, and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

FIG. 1 is an illustrative view showing an outline of a listening dialogue sustaining system according to an embodiment of the present invention. FIG. 2 is an illustrative view showing an example of a lateral positional relationship among the monitor camera, the monitor, the robot, and the user shown in FIG. 1 and an imaging range of the monitor camera and the abdominal camera. FIG. 3 is an illustrative view showing the appearance of the robot shown in FIG. 1 from the front. FIG. 4 is a block diagram showing an example of the electrical configuration of the robot shown in FIG. FIG. 5 is a block diagram showing an example of the electrical configuration of the PC shown in FIG. FIG. 6 is an illustrative view showing one example of an action table stored in the memory shown in FIG. FIG. 7 is an illustrative view showing one example of a face recognition result by the monitor camera and the abdominal camera shown in FIG. FIG. 8 is a block diagram showing an example of the electrical configuration of the server shown in FIG. FIG. 9 is an illustrative view showing one example of a memory map of the memory of the PC shown in FIG. FIG. 10 is an illustrative view showing one example of a data storage area shown in FIG. FIG. 11 is an illustrative view showing one example of a configuration of the situation recognition program shown in FIG. FIG. 12 is an illustrative view showing one example of a configuration of the robot control program shown in FIG. FIG. 13 is a flowchart showing image / sound acquisition processing of the processor of the PC shown in FIG. FIG. 14 is a flowchart showing the attitude determination processing of the processor of the PC shown in FIG. FIG. 15 is a flowchart showing speech determination processing of the processor of the PC shown in FIG. FIG. 16 is a flowchart showing head direction determination processing of the processor of the PC shown in FIG. FIG. 17 is a flowchart showing the gaze direction determination processing of the processor of the PC shown in FIG. FIG. 18 is a flowchart showing face recognition processing of the processor of the PC shown in FIG. FIG. 19 is a flowchart showing the whirling determination process of the PC processor shown in FIG. FIG. 20 is a flowchart showing the compatibility determination process of the processor of the PC shown in FIG. FIG. 21 is a flowchart showing synchronization processing of the processor of the PC shown in FIG. FIG. 22 is a flowchart showing state recognition processing of the processor of the PC shown in FIG. FIG. 23 is a flowchart showing the Ac / Pa recognition process of the processor of the PC shown in FIG. FIG. 24 is a flowchart showing Ta / Li recognition processing of the processor of the PC shown in FIG. FIG. 25 is a flowchart showing the interest object recognition processing of the processor of the PC shown in FIG. FIG. 26 is a flowchart showing the utterance time measurement processing of the processor of the PC shown in FIG. FIG. 27 is a flowchart showing the overall processing of the processor of the PC shown in FIG. FIG. 28 is a flowchart showing the active talk processing of the processor of the PC shown in FIG. FIG. 29 is a flowchart showing an active listening process of the processor of the PC shown in FIG. FIG. 30 is a flowchart showing inactive processing of the processor of the PC shown in FIG. FIG. 31 is a flowchart showing the other processing of the processor of the PC shown in FIG. FIG. 32 is a flowchart showing the utterance continuation processing of the PC shown in FIG. FIG. 33 is a flowchart showing speech suppression processing of the processor of the PC shown in FIG. FIG. 34 is a flowchart showing the speech promotion process of the processor of the PC shown in FIG. FIG. 35 is a flowchart showing the attention attracting process of the processor of the PC shown in FIG. FIG. 36 is a flowchart showing an attendant pseudo-listening process of the processor of the PC shown in FIG. FIG. 37 is a flowchart showing the first positive pseudo-listening process of the processor of the PC shown in FIG. FIG. 38 is a flowchart showing user utterance suppression processing of the processor of the PC shown in FIG. FIG. 39 is a flowchart showing the second active pseudo-listening process of the processor of the PC shown in FIG. FIG. 40 is a flowchart showing attention guidance processing of the processor of the PC shown in FIG. 41 is a flowchart showing user utterance promotion processing of the processor of the PC shown in FIG. FIG. 42 is a flowchart showing camera control processing of the processor of the PC shown in FIG.

  Referring to FIG. 1, the listening dialogue sustaining system 100 of this embodiment is used for a dialogue between a user A having a mild brain disorder such as a dementia patient and a user B in a remote place. Therefore, the listening dialogue sustaining system 100 includes a stuffed robot (hereinafter simply referred to as “robot”) 10a, a PC 14a, and a monitor 16a including an abdominal camera 12a (second camera) installed in the room 1 where the user A is present. A robot 10b, a PC 14b, a monitor 16b, a speaker 18b, a microphone 20b, and a speaker 18a, a microphone 20a, a monitor camera 22a (first camera), and an abdominal camera 12b installed in a room 2 (remote location) where the user B is located. A monitor camera 22b and a server 24 connected to the network 200 are provided. In addition, in this specification, when it is necessary to describe the corresponding devices and people in the room 1 and the room 2 without distinction, the reference numerals may be displayed only without reference numerals. Please keep in mind.

  The robot 10 listens and speaks based on the control signal from the PC 14. An abdominal camera 12 provided on the abdomen of the robot 10 captures the user and outputs an image to the PC 14 via the robot 10. The PC 14 outputs a control signal to the robot 10 and receives an image captured by the abdominal camera 12 and the monitor camera 22 and a sound collected by the microphone 20. Then, the PC 14 determines and recognizes the user's action and state based on the input image and sound, and transmits the result to the server 24 via the network 200.

  The PC 14, the monitor 16, the speaker 18, the microphone 20, and the monitor camera 22 function as a video phone. For example, the PC 14a receives the image and sound of the user B transmitted from the PC 14b on the user B side. Therefore, the monitor 16a displays the image of the user B, and the speaker 18 outputs the user B's voice. Further, the microphone 20 collects the voice of the user A and outputs it to the PC 14, and the monitor camera 22 captures an image of the user A and outputs it to the PC 14. Then, the PC 14 transmits the image and sound of the user A to the PC 14b via the network 200.

  When the server 24 receives the data of the actions and states of the user A and the user B transmitted from the PC 14a and the PC 14b, the server 24 accumulates them in a database (DB). And when there exists a request | requirement which acquires the data of action and a state from PC14, data are transmitted to PC14 according to the request | requirement.

  In another embodiment, the robot 10 and the PC 14 may be wirelessly connected instead of wiredly connected. Further, the connection between the PC 14 and the server 24 with the network 200 may be a wired connection or a wireless connection.

  FIG. 2 shows an embodiment of the embodiment shown in FIG. 1 viewed from the side. As can be seen from FIG. 2, the monitor camera 22 is placed on the monitor 16, and the robot 12 and the monitor 16 are placed on the desk. The user is photographed by the abdominal camera 12 and the monitor camera 22 while facing the monitor 16 and the monitor camera 22 placed on the desk. Furthermore, since the robot 10 is disposed between the user and the monitor 16, the monitor camera 22 photographs the robot 10 and the user at the same time. Thereby, the robot 10 performs a pseudo-listening operation (pseudo-listening operation) on the user A, or performs a pseudo-listening operation on the monitor 16a on which the user B is displayed.

  Note that the robot 10 does not have to be placed on the desk as long as it is captured by the monitor camera 22 and can capture the user.

  FIG. 3 illustrates the appearance of the robot 10. The robot 10 includes a head 26 and a body 28 that supports the head 26. The left arm 30L and the right arm 30R are provided on the left and right of the upper part of the body 28 (corresponding to a human shoulder), and the abdomen camera 12 is provided on the abdomen of the body 28. As the abdomen camera 12, for example, a camera using a solid-state imaging device such as CCD or CMOS can be employed. The head 26 is provided with a mouth 32 on the front surface, and an eyeball 34 is provided above the mouth 32. An ear 36 is attached to the upper side surface of the head 26.

  The head 26 is supported by the body 28 so as to be able to turn and rise, and the eyeball 34 is also operatively held. In addition, the body 28 can be tilted in the left-right direction around the waist. Further, a speaker 56 (FIG. 4) is built in the mouth 32, and a microphone 58 (FIG. 4) is built in the ear 36.

  If the microphones 58 are incorporated in both ears 36, they function as stereo microphones, whereby the position of the sound input to the stereo microphones can be specified as necessary. Further, the appearance of the robot 10 is not limited to a bear, but may be another animal or a humanoid.

  FIG. 4 is a block diagram showing an electrical configuration of the robot 10. The robot 10 incorporates a processor 38, also called a microcomputer or a CPU, via a bus 40, which is an example of a communication path, via an abdominal camera 12, a memory 42, a motor control board 44, and an audio input / output board 54. , Connected to the sensor input / output board 60 and the I / O 62.

  The memory 42 includes a ROM and a RAM (not shown). The ROM mainly includes a listening operation by the robot 10, a program for uttering, voice data output from the speaker 56 when uttering, and the like. Stored in advance. The RAM is used as a temporary storage memory and a working memory.

  The motor control board 44 is constituted by, for example, a DSP (Digital Signal Processor), and controls each axis motor of each arm and head of the robot 10 shown in FIG. That is, the motor control board 44 receives the control data from the processor 38 and controls the three angles for controlling the X, Y, and Z axes so that the right arm 30R (FIG. 3) can be moved back and forth and left and right. The rotation angle of the motor 46R (collectively shown as “right arm motor” in FIG. 4) 46R is adjusted. Further, the motor control board 44 adjusts the rotation angle of three motors 46L of the left arm 30L (collectively shown as “left arm motor” in FIG. 4) 46L. The motor control board 44 also adjusts the rotation angle of three motors 48 (collectively shown as “head motors” in FIG. 4) that control the turning angle and the elevation angle of the head 26. The motor control board 44 also controls an eyeball motor 50 that moves the eyeball 34 and a waist motor 52 that tilts the body 28.

  Note that the motors described above are stepping motors or pulse motors in order to simplify the control, but may be direct current motors.

  The speaker 56 is provided with the synthesized voice data from the processor 38 via the voice input / output board 54, and accordingly, the speaker 56 outputs voice or voice according to the data. Then, the sound collected by the microphone 58 is taken into the processor 38 via the sound input / output board 54.

  Similar to the motor control board 44, the sensor input / output board 60 is composed of a DSP, takes in a signal from the abdominal camera 12, and provides it to the processor 38. The video signal from the abdominal camera 12 is input to the processor 38 after being subjected to predetermined processing by the sensor input / output board 60 as necessary.

  The I / O 62 is a digital port that can control input / output, and a video signal is output from the output port and applied to the PC 14. On the other hand, a control signal is output from the PC 14 and applied to the input port.

  FIG. 5 is a block diagram showing the electrical configuration of the PC 14. Like the robot 10, the PC 14 includes a processor 64, which is also called a microcomputer or CPU, and is connected to the memory 68, the line-of-sight server 70, the audio input / output board 72, and the I / O 74 via the bus 66. The The processor 64 has a built-in RTC (Real Time Clock) 64a for outputting date information.

  The memory 68 incorporates a ROM, a RAM, and an HDD (not shown). The ROM mainly stores a program for realizing a telephone function and a program expressed by flowcharts (FIGS. 13 to 42) described later. The The RAM is mainly set with a buffer for temporarily storing images taken by the abdominal camera 12 and the monitor camera 22 and sounds collected by the microphone 20. The HDD mainly stores the result of determining the user's action, the result of recognizing the state, and the like as needed.

  The line-of-sight server 70 detects the user's line-of-sight direction and position in real time from the image of the user's face taken by the abdominal camera 12 and the monitor camera 22. Then, the processor 20 receives the data indicating the user's line-of-sight direction and position identified or detected by the line-of-sight server 70 through the bus 66 to determine an object to be watched by the user.

  Note that the method of detecting the user's line-of-sight direction and position by the line-of-sight server 70 has been disclosed in Japanese Patent Application Laid-Open No. 2008-113875 previously filed by the applicant of the present application and is therefore omitted here. To do.

  The audio data of the other user is given to the speaker 18 from the processor 64 via the audio input / output board 72, and the audio according to the data is output from the speaker 18 accordingly. Then, the voice of the partner user collected by the microphone 20 is taken into the processor 64 via the voice input / output board 72.

  Similar to the I / O 62 of the robot 10, the I / O 74 is a digital port capable of controlling input / output. From the output port, a control signal is output to the robot 10 and an image signal is output to the monitor 16. Is done. In addition, a video signal is output from the robot 10 and the monitor camera 22 and applied to the input port. Note that the audio data output from the speaker 18 and the audio data input from the microphone 20 may be input / output to / from the audio input / output board 72 via the I / O 74.

  The processor 64 is connected to the communication LAN board 76 via the bus 66. The communication LAN board 76 is constituted by, for example, a DSP, and supplies transmission data given from the processor 64 to the wireless communication device 78. The wireless communication device 78 transmits the transmission data to an external computer (the server 24 and the partner PC 14) via the network 200. The communication LAN board 76 receives data via the wireless communication device 78 and gives the received data to the processor 64.

  For example, the transmission data may be a command, image data and audio data necessary for a videophone, a result of determining a user's action, and a result of recognizing the user's state. The received data may be image data and audio data of the other party obtained as a video phone, or a result of determining the action and state of the other user and recognizing the state.

  FIG. 6 shows an action table stored in the memory 68 of the PC 14. The behavior table is a table in which the result of determining the user's behavior is used as behavior data, and the behavior data is recorded every predetermined time (for example, 1 second).

  Referring to FIG. 6, the action table includes “time”, “forward tilt posture”, “utterance”, “head direction (abdomen)”, “gaze direction (abdomen)”, “face recognition (abdomen) from the left side. ”,“ Head direction (monitor) ”,“ line-of-sight direction (monitor) ”,“ face recognition (monitor) ”,“ blink ”and“ contrast ”. Each action data is composed of a determination result recorded in each column in synchronization with the “time” column.

  The numerical value recorded in the “time” column is the date and time information output by the RTC 64a. For example, “10:00:30” represents “10:00:30”.

  In the column of “forward leaning posture”, a determination result indicating whether or not the user is leaning forward with respect to the monitor 16 is recorded. For example, if “Yes” is recorded in the “Forward leaning” column, the user is leaning forward with respect to the monitor 16, and if “None” is recorded, the user is directed to the monitor 16. It shows that it is not taking forward leaning posture. Then, the presence / absence of the forward tilt posture is determined by applying a template matching process to images taken by the abdominal camera 12 and the monitor camera 22. For example, the forward tilt posture is determined as “present” when the user posture matches the template of the forward tilt line of sight, and the forward tilt posture is determined as “none” when it does not match the template. Note that a sensor that can measure a distance, such as an ultrasonic sensor, may be attached to the robot 10 to determine the user's posture.

  In the “utterance” column, a determination result indicating whether or not the user is speaking is recorded. For example, if “Yes” is recorded in the “Speech” column, it indicates that the user is speaking, and if “None” is recorded, it indicates that the user is not speaking. Then, “Yes” / “No” of the utterance is determined from the voice level of the voice data collected by the microphone 20. For example, if the sound level of the sound data is greater than or equal to a predetermined value, it is determined as “present”, and if it is less than the determined value, it is determined as “not present”.

  In the columns of “head direction (abdomen)” and “head direction (monitor)”, an object in the direction in which the user's head is facing is recorded. For example, if “monitor” is recorded, it indicates that the monitor 16 is in the direction facing the user's head, and if “robot” is recorded, the robot 10 is directed in the direction toward the user's head. If “other” is recorded, it indicates that the robot 10 and the monitor 16 are not in the direction in which the user's head faces. The column of “head direction (abdomen)” is determined by applying template matching processing to the image captured by the abdomen monitor 12. On the other hand, the column of “head direction (monitor)” is determined by applying template matching processing to the image captured by the monitor camera 22. For example, if the shape of the user's head matches the template facing the direction of the monitor 16, it is determined as “monitor”. If the shape of the user's head matches the template facing the direction of the robot 10, “ If the shape of the user's head does not match any of the templates, it is determined as “other”.

  In the columns “line-of-sight direction (abdomen)” and “line-of-sight direction (monitor)”, objects in the direction in which the user's line of sight faces are recorded. Similarly to the “head direction (abdomen)” column, “monitor”, “robot”, and “other” are recorded in the “line of sight direction (abdomen)” and “line of sight direction (monitor)” columns. The Then, “monitor”, “robot”, and “other” in the columns of “line-of-sight direction (abdomen)” and “line-of-sight direction (monitor)” are determined based on the output of the line-of-sight server 70. For example, if the user's line-of-sight direction is directed to the monitor 16, it is determined as “monitor”, and if the user's line-of-sight direction is directed toward the robot 10, it is determined as “robot”, and the user's line-of-sight direction is determined as the robot 10 and the monitor 16. If it is not in any direction, it is determined as “other”.

  In the columns of “face recognition (abdomen)” and “face recognition (monitor)”, the recognition results of the user's face are recorded. For example, “success” is recorded if the user's face is recognized, and “failure” is recorded if the user's face is not recognized. The column of “face recognition (abdomen)” is determined by performing face recognition processing on an image photographed by the abdomen monitor 12. On the other hand, the column “face recognition (monitor)” is determined by performing a predetermined face recognition process on the image captured by the monitor camera 22.

  7A shows a success sequence of face recognition results by the monitor camera 22, and FIG. 7B shows a successful example of face recognition results by the abdominal camera 12. FIG. First, referring to FIG. 7A, the left side is an image by the abdominal camera 12, the right side is an image by the monitor camera 22, and both images are images taken at the same time. At this time, the user is watching the monitor camera 22. Therefore, in the image by the monitor camera 22, the face of the user is reflected in the front, and thus the face recognition is successful. On the other hand, in the image obtained by the abdominal camera 12, the user's face is tilted and the face recognition fails.

  Next, referring to FIG. 7B, as in FIG. 7A, the left side is an image by the abdominal camera 12, the right side is an image by the monitor camera 22, and is an image taken at the same time. is there. At this time, since the user is gazing at the abdomen camera 12, the face recognition is successful in the image by the abdomen camera 12. On the other hand, face recognition has failed in the image by the monitor camera 22.

  Returning to FIG. 6, the presence / absence of whispering by the user is recorded in the “whispering” column. For example, if “yes” is recorded in the “whispering” column, it indicates that the user has asked, and if “none” is recorded, it indicates that the user has not spoken. The presence / absence of the user's whisper is determined by applying a predetermined template matching process to the images taken by the abdominal camera 12 and the monitor camera 22. For example, “Yes” is determined if the user's head shape matches the crawling template, and “No” is determined if the template does not match the template.

  The presence / absence of a user's conflict is recorded in the “contrast” column. For example, if “Yes” is recorded in the column of “Consideration”, it indicates that the user has asked, and if “No” is recorded, it indicates that the user has not asked. The presence / absence of the conflict is determined based on the determination result of the whispering and the determination result of the voice prosody that can determine a sound that is compatible with the voice data such as “Ah” and “Ye”. For example, if there is a sound that is compatible when the user's whisper is “Yes”, the determination result of the conflict is determined as “Yes”.

  Note that the template matching process and the face recognition process described above use widely general methods, and thus detailed description thereof is omitted.

  And PC14 recognizes a user's state for every 1st predetermined time (for example, 30 seconds) from the action data recorded for every fixed time by these processes. Specifically, Ac / Pa recognition, Ta / Li recognition, and interest recognition processing are executed based on the behavior data for each first predetermined time, and the state of the user is determined from the recognition result of each processing.

  First, Ac / Pa recognition refers to an “active (Ac)” state where the user actively participates in the dialogue and concentrates on the dialogue, or the user is inactive in the dialogue, This is a process for recognizing a “passive (Pa)” state in which concentration or attention is distracting. Therefore, in the Ac / Pa recognition of the present embodiment, the AC value indicating the likelihood of the active state is calculated based on the line-of-sight direction, the forward leaning posture, the whirling, and the conflict among the user behavior data for the first predetermined time. To recognize the active or passive state.

  For example, when WT is the time to watch the opponent, AT is the time of the forward leaning posture approaching the body, and RF is the frequency of reciprocity, the time WT to look at the opponent is the “sight line direction (abdomen)” in the action table and It can be obtained by counting the number of times determined as “present” in the row of “first line of sight (monitor)” for the first predetermined time. Further, the forward tilt posture time FT can be obtained by counting the number of times determined to be “present” in the first predetermined time row in the “forward tilt posture” column. Further, the frequency RF of the conflict can be obtained by dividing the number of times determined as “Yes” in the column of “Consult” by the number of times determined as “Yes” in the column of “Swing”.

  Further, the AC value is calculated based on the equation shown in Formula 1 from the calculated time WT for watching the opponent, the time AT for approaching the body by defeating, and the frequency RF of the companion.

[Equation 1]
C1 × WT + C2 × FT + C3 × RF = AC values C1, C2, C3: constants By determining the AC value calculated in this way based on the threshold A, the active state or the passive state can be recognized. That is, if the AC value calculated by the equation shown in Equation 1 is larger than the threshold value A, the active state is recognized, and if the AC value is equal to or less than the threshold value, the passive state is recognized. Each parameter can be weighted by changing the values of the constants C1, C2, and C3.

  Next, Ta / Li recognition is a “talk: Talk (Ta)” state (speaker side state) in which a user speaks as a speaker, or “listen: Listen” listening to the other user's story. This is a process of recognizing whether the state is (Li) ”(listening side state). Therefore, in Ta / Li recognition, it recognizes based on a user's utterance among a user's action for the 1st predetermined time. In this embodiment, the utterance amount of the user is used as the utterance amount, and the utterance amount recognizes the talk state or the listen state based on the threshold value T.

  For example, when the utterance amount is Ta, the utterance amount Ta is obtained by counting the number of times determined to be “present” in the row of the first predetermined time in the “utterance” column in the action table. be able to. Then, the talk state or the passive state can be recognized by comparing the utterance amount Ta thus calculated with the threshold value T. That is, if the utterance amount Ta is larger than the threshold value T, it is recognized as a talk state, and if the utterance amount Ta is equal to or less than the threshold value T, it is recognized as a listening state.

  Interest object recognition is processing for recognizing whether the user's interest is “robot”, “monitor”, or “other”. Therefore, in interest recognition, the recognition rate M of the face on the monitor camera 22 side and the abdomen based on the user's face recognition and line-of-sight direction of the user's behavior data for a second predetermined time (for example, 10 seconds) By calculating the face recognition rate S on the camera 12 side, an object that the user is interested in is recognized.

  For example, the recognition rate M is determined as “success” and “monitor” in the first predetermined time row in the columns of “face recognition (monitor)” and “gaze direction (monitor)” in the action table. It can be obtained by counting the number of times. On the other hand, the recognition rate S can be obtained by counting the number of times determined as “success” and “robot” in the columns of “face recognition (abdomen)” and “gaze direction” in the action table. Then, the user's interest can be recognized by comparing the recognition rate M and the recognition rate S calculated in this way with the threshold value In. That is, if both the recognition rate M and the recognition rate S are equal to or less than the threshold value In, the object of interest is recognized as “other”. If the recognition rate M and the recognition rate S are both greater than the threshold value In and the recognition rate M is greater than the recognition rate S, the object of interest is recognized as “monitor”, and if the recognition rate S is greater than the recognition rate M, The target is recognized as a “robot”.

  Note that the recognition rate S and the recognition rate M may be obtained only from the result of face recognition. Further, when the result of face recognition is shown as a percentage, the total product of recognition results every second may be used as the recognition rate. Furthermore, in interest recognition, instead of the recognition rate, recognition may be performed based on numerical values obtained from the time ratio of the forward tilt posture, the time ratio in the line-of-sight direction, and the time ratio in the head direction. The first predetermined time and the second predetermined time may be the same length.

  Thus, after the process of Ac / Pa recognition, Ta / Li recognition, and interest object recognition is performed, a user's state is determined by combining each recognition result. Specific user states include “active talk monitor”, “active listen monitor”, “passive talk monitor”, and “passive listen monitor” in which the user is interested in the monitor 16. There are four types, “Active Talk Robot”, “Active Listen Robot”, “Passive Talk Robot”, and “Passive Listen Robot” whose users are interested in the robot 10, and the user A total of 4 types of "active talk other", "active listen other", "passive talk other", and "passive listen other" that are not interested in robot 10 or monitor 16 There are 12 types.

  Here, when the predetermined time is 1 second and the first predetermined time is 30 seconds, "active talk monitor", "active listen monitor", "passive talk monitor" and "passive listen monitor" An example of the state of the user recognized as “monitor” will be described. Note that the user states of “robot” and “other” are the same as in the case of “monitor” except that the user's interests are different, and thus detailed description thereof is omitted.

  First, in a state where the user's interest is recognized as “monitor”, the duration of the utterance is 15 seconds (50%) or more, and “monitor” is 18 times out of 30 times (60 %) Or more is judged, and if there are two or more whisperings, it is recognized as “active talk monitor”.

  Next, in a state where the user's interest is recognized as “monitor”, the duration of the utterance is 9 seconds (30%) or less, and “monitor” is determined 15 times out of 30 times in the determination of the gaze direction ( 50%) or more, and if “Yes” is determined 21 times out of 30 (70%) or more in the determination of the forward tilt posture, it is recognized as “active listen monitor”.

  Next, in a state where the user's interest is recognized as “monitor”, the duration of the utterance is 12 seconds (40%) or less, and “monitor” is 6 times out of 30 times in the determination of the line-of-sight direction ( 20%) or more, and if there is no further whispering, it is recognized as “passive talk monitor”.

  Then, in a state where the user's interest is recognized as “monitor”, the duration of the utterance is 9 seconds (30%) or less, there is no whispering, and there is “Yes” in the determination of the forward leaning posture. If it is 6 times (20%) or less out of 30 times, it is recognized as a “passive listen monitor”.

  The user status recognized in this way is stored in the memory 64 as status data 354 (FIG. 10) and transmitted to the server 24. Further, a support vector machine (SVM) may be used for recognizing the user state.

  FIG. 8 is a block diagram showing the electrical configuration of the server 24. Similarly to the processors 38 and 64, the server 24 includes a processor 80 called a microcomputer or CPU. The processor 80 is connected to the memory 84, the first robot information DB 86, the second robot information DB 88, and the communication LAN board 90 via the bus 82.

  The memory 84 incorporates a ROM and a RAM (not shown), and the ROM mainly stores a program for performing data communication between the server 24 and the PCs 14a and 14b and the like. The RAM is used as a temporary storage memory and a working memory.

  The first robot information DB 86 is a database for accumulating user A's action data and state data transmitted from the PC 14a. The second robot DB 88 is a database for accumulating user B's action data and state data transmitted from the PC 14b. The first robot information DB 86 and the second robot information DB 88 are configured using a storage medium such as an HDD or an SSD.

  Similar to the communication LAN board 76 of the PC 14, the communication LAN board 90 is configured by a DSP, for example, and provides transmission data given from the processor 80 to the wireless communication device 92. The wireless communication device 92 transmits the transmission data to an external computer (PC 14a, 14b) via the network 200. The communication LAN board 90 receives data via the wireless communication device 92 and provides the received data to the processor 80.

  For example, the received data is the action data of the user A transmitted from the PC 14a, and the processor 80 stores the action data of the user A in the first robot information DB 86. Further, when the action data acquisition request of the user A is given to the processor 80 from the PC 14b as the received data, the processor 80 gives the action data of the user A to the communication LAN board 90 as transmission data.

  Here, in the listening dialogue sustaining system of the present invention, the operation of the robot 10 is controlled based on the user's behavior data and state data and the other user's behavior data and state data so that the conversation between the two people is sustained. . Then, the robot 10 performs three types of operations, “pseudo-listening operation”, “speech control operation”, and “attention attracting operation”, with respect to the dialogue between the user A and the user B, and continues the dialogue.

  The pseudo-listening operation is an operation that behaves as if listening to one of the utterances when the user A and the user B are actively interacting with each other. Furthermore, when one user is speaking unilaterally, the utterance control action is to suppress the utterance by looking at the user or to talk to the user in order to balance the two utterances. It is an action that promotes speech. The attention inducing and attracting actions are actions to attract the user's attention by talking to the user when the user is not concentrated on the dialogue. A specific operation of the robot 10 will be described with reference to flowcharts shown in FIGS.

  FIG. 9 is an illustrative view showing one example of a memory map 300 of the memory 68 in the PC 14 shown in FIG. As shown in FIG. 9, the memory 68 includes a program storage area 302 and a data storage area 304. The program storage area 302 stores a data communication program 312, a situation recognition program 314, a robot control program 316, a camera control program 318, an utterance time measurement program 320, and a random number generation program 322 as programs for operating the PC 14. The

  The data communication program 312 is a program for performing data communication with the server 24. The situation recognition program 314 is a program for determining a user's action and recognizing a state. The robot control program 316 is a program for determining the operation of the robot 10. The camera control program 318 is a program for determining a camera image to be transmitted to the other party. The utterance time measurement program 320 is a program for measuring the utterance time of the user A and the user B. The random number generation program 322 is a process executed when determining the operation of the robot 10 at random.

  Although illustration is omitted, the program for operating the PC 14 includes a program for realizing a speech function on a television.

  Referring to FIG. 10, the data storage area 304 includes a time buffer 330, a monitor camera buffer 332, an abdominal camera buffer 334, an audio buffer 336, a determination result buffer 338, a face recognition result buffer 340, and an interest recognition result buffer. 342, a data communication buffer 344, a partner action data buffer 346, a state data buffer 348, and a random number buffer 350 are provided. Further, the action table data 352 and the state data 354 are stored in the data storage area 304, and an Ac / Pa flag 356, a Ta / Li flag 358, a state counter 360, and an utterance counter 362 are further provided.

  The time buffer 330 is a buffer in which date / time information output from the RTC 64a is temporarily stored. The monitor camera buffer 332 is a buffer in which an image taken by the monitor camera 22 is temporarily stored. The abdominal camera buffer 334 is a buffer in which an image taken by the abdominal camera 12 is temporarily stored. The audio buffer 336 is a buffer in which audio collected by the microphone 20 is temporarily stored.

  The determination result buffer 338 determines the presence / absence of the user's forward tilt posture, determines the user's utterance, determines the user's head direction, determines the user's head direction, and determines the user's gaze direction. It is a buffer for temporarily storing each determination result of gaze direction determination, whirl determination for determining a user's whirl, and conflict determination for determining a user's conflict. The face recognition result buffer 340 is a buffer that temporarily stores the results of face recognition performed on images taken by the monitor camera 22 and the abdominal camera 12. The interest recognition result buffer 342 is a buffer for temporarily storing a result determined that the user is interested. For example, data indicating “robot”, “monitor”, and “other” is temporarily stored. Remembered.

  The data communication buffer 344 is a buffer that temporarily stores the other party's action data, state data, and the like obtained by data communication with the server 24. The opponent action data buffer 346 is a buffer for temporarily storing the action data of the opponent. The status data buffer 348 is a buffer for temporarily storing the status data of the other party and the status data of the user on this side. The random number data buffer 350 is a buffer in which random numbers generated by the random number generation program 322 are temporarily stored.

  The action table data 352 is the action table shown in FIG. 6, and the latest action data is added every certain time. The state data 354 is data indicating the state of the user recognized from the action data for the first predetermined time, and is composed of, for example, a character string indicating “active talk robot”.

  The Ac / Pa flag 356 is a flag indicating an Ac / Pa recognition result. For example, the Ac / Pa flag 356 is composed of a 1-bit register. When the Ac / Pa flag 356 is turned on (established), a data value “1” is set in the register. On the other hand, when the Ac / Pa flag 356 is turned off (not established), a data value “0” is set in the register. Further, the Ac / Pa flag 356 is turned on when recognized as an active state, and is turned off when recognized as a passive state. The Ta / Li flag 358 is a flag indicating a Ta / Li recognition result. That is, the Ta / Li flag 358 is turned on when recognized as a talk state, and turned off when recognized as a listen state. The configuration of the Ta / Li flag 358 is the same as that of the Ac / Pa flag 356, and thus detailed description thereof is omitted.

  The state counter 360 is a counter used for acquiring action data for a first predetermined time when recognizing the state of the user. For example, the state counter 360 starts counting when the power of the PC 14 is turned on, and is reset every time action data for a first predetermined time is acquired. The utterance counter 362 is a counter for counting the utterance time measured by the utterance time measurement program 320.

  Although not shown, the data storage area 304 is provided with a template for use in each determination, a buffer for temporarily storing various calculation results, and the like, and is necessary for the operation of the PC 14. Other counters and flags are also provided.

  FIG. 11 shows programs constituting the situation recognition program 314. Referring to FIG. 11, the situation recognition program 314 includes an image / sound acquisition program 314a, a gaze determination program 314b, an utterance determination program 314c, a head direction determination program 314d, a gaze direction determination program 314e, a face recognition program 314f, and a whisper determination. The program 314g, the conflict determination program 314h, the synchronization program 314j, the state recognition program 314k, the Ac / Pa recognition program 314m, the Ta / Li recognition program 314n, and the interest recognition program 314p.

  The image / sound acquisition program 314a is a program for taking in an image captured by the monitor camera 22 and the abdominal camera 12 and the sound collected by the microphone 20 into a buffer. The posture determination program 314b is a program for determining whether or not the user's posture is a forward leaning posture. The speech determination program 314c is a program for determining whether or not the user is speaking. The head direction determination program 314d is a program for determining the user's head direction. The gaze direction determination program 314e is a program for determining the gaze direction of the user. The face recognition program 314f is a program for recognizing a face area in images taken by the monitor camera 22 and the abdominal camera 12. The whisper determination program 314g is a program for determining whether or not the user has whispered. The conflict determination program 314h is a program for determining whether or not the user has a conflict. The synchronization program 314j is a program for synchronizing action determination data, speech determination results, head direction determination results, gaze direction determination results, face recognition results, whisper determination results, and conflict determination results into action data.

  The state recognition program 314k is a program for recognizing the user state every first predetermined time. The Ac / Pa recognition program 314m is a program for recognizing whether the user is in an active state or a passive state. The Ta / Li recognition program 314n is a program for recognizing whether the user is in a talk state or a listen state. The interest object recognition program 314p is a program for recognizing a user's interest versus object.

  FIG. 12 shows programs constituting the robot control program 316. Referring to FIG. 12, overall program 316a, active talk program 316b, active listen program 316c, inactive program 316d, other program 316e, speech continuation program 316f, speech suppression program 316g, speech promotion program 316h, attention attracting program 316j, It consists of a by-participant pseudo-listening program 316k, a first active pseudo-listening program 316m, a user utterance suppression program 316m, a second active pseudo-listening program 316p, an attention guidance program 316q, and a user utterance promotion program 316r.

  The overall program 316a is also called a main routine, and is a program that performs overall processing for controlling the robot 10 so as to maintain a dialogue according to state data. The active talk program 316b is a program that is executed when the state data is “active talk robot”. The active listen program 316c is a program executed when the state data is “active listen robot”. The inactive program 316d is a program that is executed when the user is in a passive state or when the object of interest is recognized as a “robot”. The other program 316e is a program that is executed when the user's interest is “other”.

  The utterance continuation program 316f is a program for operating the robot 10 so that the dialogue between the two users is continued. The speech suppression program 316g is a program for operating the robot 10 so as to suppress speech when the user is speaking unilaterally. The utterance promotion program 316h is a program for operating the robot 10 to encourage utterance when the user unilaterally listens to the other user's utterance without speaking. The attention attracting program 316j is a program for operating the robot 10 so as to attract the attention of a user who has lost interest in the dialogue.

  The bystander-like pseudo-listening program 316k is a program for operating the robot 10 as if listening to the story of one of the users when the conversation between the two users continues. . The “collateral participant” indicates a person who participates in a conversation but is neither a “speaker” nor a “listener”.

  The first active pseudo-listening program 316m is a program for operating the robot 10 as if listening to the other user's story. The user utterance suppression program 316n is a program for operating the robot 10a so as to suppress, for example, the utterance of the user A who is near the robot 10a. The second active pseudo-listening program 316p is a program for operating the robot 10a, for example, as if listening to the utterance of the user A near the robot 10a. The attention guidance program 316q is a program for operating the robot 10 so as to guide the user's attention. The user utterance promotion program 316r is a program for operating the robot 10 so as to urge the user A who is in the vicinity of the robot 10a, for example.

  Hereinafter, a flowchart of the present invention executed by the PC 14 will be described. 13 to 25 show the processing of each program constituting the situation recognition program 314, FIG. 26 shows the processing by the utterance time measurement program 320, and FIGS. 27 to 41 show the robot control program 316. FIG. 42 shows processing by the camera control program 318.

  FIG. 13 is a flowchart showing the processing of the image / sound acquisition program 314a. For example, when the power of the PC 14 is turned on by the user, the processor 64 of the PC 14 acquires image data from the abdominal camera 12 in step S1. That is, image data is acquired from the video signal input from the robot 10 and is temporarily stored in the abdominal camera buffer 334. Subsequently, in step S3, image data obtained by the monitor camera 22 is acquired. That is, image data is acquired from the video signal input from the monitor camera 22 and temporarily stored in the monitor camera buffer 332. Subsequently, in step S5, audio data is acquired, and the process returns to step S1. That is, voice data is extracted from the voice collected by the microphone 20 and temporarily stored in the voice buffer 336. In addition, the process of step S1-S5 is repeated about every 1 second.

  FIG. 14 is a flowchart showing the processing of the posture determination program 314b. The processor 64 determines whether image data has been acquired in step S7. That is, it is determined whether new image data is stored in the monitor camera buffer 332 and the abdomen camera buffer 334. If “NO” in the step S7, that is, if the image data is not acquired, the process of the step S7 is repeatedly executed. On the other hand, if “YES” in the step S7, that is, if image data is acquired, the posture of the user is determined in a step S9. For example, the presence or absence of the user's forward leaning posture is determined with respect to the image data stored in the monitor camera buffer 332 and the abdomen camera buffer 334. If the forward tilt posture is determined to be “present” in both images, the posture determination result is determined to be “present”. Note that the processor 64 that executes the process of step S9 functions as an attitude determination unit.

  Subsequently, in step S11, the current time is acquired. That is, the date information stored in the time buffer 330 is acquired. In step S13, the current time is associated with the posture determination result. That is, the current time is associated in order to synchronize a plurality of determination results. The posture determination result associated with the current time is temporarily stored in the determination result buffer 338.

  In another embodiment, posture determination may be performed using only one image of the monitor camera 22 or the abdominal camera 12. Also, in the other determination processes shown in FIGS. 15 to 20, there is a process for acquiring date / time information in the same manner as in step S <b> 11 and associating the date / time information as in step S <b> 13. Detailed description is omitted in other flowcharts.

  FIG. 15 is a flowchart showing the processing of the utterance determination program 314c. The processor 64 determines whether or not audio data has been acquired in step S15. That is, it is determined whether or not new audio data is stored in the audio buffer 336. If “NO” in the step S15, that is, if audio data is not acquired, the process of the step S15 is repeatedly executed. On the other hand, if “YES” in the step S15, that is, if voice data is acquired, the user's utterance is determined in a step S17. For example, it is determined whether or not the sound level of the sound data stored in the sound buffer 336 is greater than or equal to a certain value. In step S19, the current time is acquired, and in step S21, the current time is associated with the speech determination result. Then, when the process of step S19 ends, the process returns to step S15. In step S 19, the determination result of the utterance associated with the current time is temporarily stored in the determination result buffer 338. The processor 64 that executes the process of step S17 functions as an utterance determination unit.

  FIG. 16 is a flowchart showing the processing of the head direction determination program 314d. The processor 64 determines whether or not image data has been acquired in step S23. That is, it is determined whether new image data is stored in the monitor camera buffer 332 and the abdomen camera buffer 334. If “NO” in the step S23, that is, if the image data is not newly acquired, the process in the step S23 is repeated. On the other hand, if “YES” in the step S23, that is, if new image data is acquired, the head direction of the user is determined in a step S25. For example, “monitor”, “robot”, and “other” are determined by executing template matching processing on the image stored in the monitor camera buffer 332. Further, the same determination is made for the image stored in the abdominal camera buffer 334. The processor 64 that executes the process of step S25 functions as a head direction determination unit.

  Subsequently, in step S27, the current time is acquired, and in step S29, the current time is associated with the determination result of the head direction. Then, when the process of step S29 ends, the process returns to step S23. In step S29, the determination result of the head direction of the image captured by the monitor camera 22 and the determination result of the head direction of the image captured by the abdominal camera 12 are stored in the determination result buffer 338, respectively.

  FIG. 17 is a flowchart showing the processing of the gaze direction determination result program 314e. The processor 64 determines whether image data has been acquired in step S31. That is, it is determined whether or not a new image is stored in the monitor camera buffer 332 and the abdomen camera buffer 334. If “NO” in the step 31, that is, if new image data is not acquired, the process of the step S31 is repeatedly executed. On the other hand, if “YES” in the step S31, that is, if the image data is acquired, the user's line-of-sight direction is determined in a step S33. For example, in step S <b> 33, first, the image data stored in the monitor camera buffer 332 is input to the line-of-sight server 70, thereby identifying the user's line-of-sight direction. Next, “monitor”, “robot”, and “other” are determined based on the line-of-sight direction specified by the line-of-sight server 70. The same determination is performed on the image stored in the abdominal camera buffer 334. The processor 64 that executes the process of step S33 functions as a line-of-sight direction determination unit.

  Subsequently, in step S35, the current time is acquired, and in step S37, the current time is associated with the visual line direction determination result. And if the process of step S37 is complete | finished, it will return to step S31. In the process of step S37, the determination result of the line-of-sight direction of the image captured by the monitor camera 22 and the determination result of the line-of-sight direction of the image captured by the abdominal camera 12 are obtained as in step S29 (FIG. 16). Each is stored in the determination result buffer 338.

  FIG. 18 is a flowchart showing the processing of the face recognition processing program 314f. The processor 64 determines whether or not image data has been acquired in step S39. That is, it is determined whether new images are stored and updated in the monitor camera buffer 332 and the abdomen camera buffer 334. If “NO” in the step S39, that is, if the image data is not acquired, the process of the step S39 is repeated. On the other hand, if “YES” in the step S39, that is, if image data is acquired, the user's face is recognized in a step S41. For example, it is determined whether or not the face area has been recognized by applying a predetermined face recognition process to the image data stored in the monitor camera buffer 332. If the face is recognized, the face recognition result is determined as “success”. If the face recognition result is not recognized, the face recognition result is determined as “failure”. The same determination is performed on the image data stored in the abdominal camera buffer 334. The processor 64 that executes the process of step S41 functions as a face recognition unit.

  Subsequently, in step S43, the current time is acquired, and in step S45, the current time is associated with the face recognition result. And if the process of step S45 is complete | finished, it will return to step S39. In the process of step S39, each recognition result associated with the current time is temporarily stored in the face recognition result buffer 340.

  FIG. 19 is a flowchart showing the processing of the whirl determination program 314g. The processor 64 determines whether or not image data has been acquired in step S47. That is, it is determined whether or not the monitor camera buffer 332 and the abdomen camera buffer 334 have been updated. If “NO” in the step S47, that is, if the image data is not acquired, the step S47 is repeated. On the other hand, if “YES” in the step S47, that is, if the image data is acquired, it is determined whether or not the user is whispered in a step S49. For example, a template matching process is applied to the image data stored in the monitor camera buffer 332 and the abdomen camera buffer 334 to determine whether or not the user has whispered. Then, when it is determined that the whisper is “Yes” based on the two determination results, the user's whisper is determined to be “Yes”. Note that the processor 64 that executes the process of step S49 functions as a whirl determination unit.

  Subsequently, the current time is acquired in step S51, and the current time is associated with the result of the determination in step S53. And if the process of step S53 is complete | finished, it will return to step S47. In addition, the determination result of the whisper associated with the date / time information is stored in the determination result buffer 338.

  FIG. 20 shows a flowchart showing the processing of the conflict determination program 314h. The processor 64 determines whether or not the whisper determination has been completed in step S55. For example, it is determined whether or not the determination result of whispering is stored in the determination result buffer 338. If “NO” in the step S55, that is, if the whispering determination is not ended, the process of the step S55 is repeated. On the other hand, if “YES” in the step S55, that is, if the whispering determination is completed, it is determined whether or not the speech determination is completed in a step S57. For example, it is determined whether or not an utterance determination result is stored in the determination result buffer 338. If “NO” in the step S57, that is, if the speech determination is not ended, the process returns to the step S55. On the other hand, if “YES” in the step S57, that is, if the speech determination is completed, the user's conflict is determined based on the date information in a step S59. For example, each is synchronized based on date and time information associated with an utterance determination result and a whisper determination result. Then, when the determination result of “whit” is “Yes”, the above-described sound method is determined for the sound data stored in the sound buffer, thereby determining the conflict. The processor 64 that executes the process of step S59 functions as a conflict determination unit.

  Subsequently, in step S61, time is associated with the determination result of the conflict, and the process returns to step S55. That is, in step S61, the date / time information associated with the speech or whisper determination result is associated with the conflict determination result. Then, the determination result of the conflict associated with the time is stored in the determination result buffer 340.

  FIG. 21 is a flowchart showing the processing of the synchronization program 314j. The processor 64 determines whether or not each determination is completed in step S63. For example, it is determined whether or not the determination result buffer 338 stores posture, speech, head direction, line-of-sight direction, whispering, and conflict determination results, and the face recognition result buffer 340 stores face recognition results. If “NO” in the step S63, that is, if each determination is not completed, the process of the step S63 is repeatedly executed. On the other hand, if “YES” in the step S63, that is, if each determination is completed, each determination result and the face recognition result are synchronized on the basis of the time in a step S65. That is, synchronization is performed based on the time associated with each determination result and the face recognition result.

  Subsequently, in step S67, each synchronized determination result is set as action data and recorded in the action table. That is, in the action table shown in FIG. 6, each determination result and face recognition result are recorded in a new line. Subsequently, in step S69, the current behavior data is transmitted to the server 24. And if the process of step S69 is complete | finished, it will return to step S63. That is, in step S69, the action data corresponding to the newly added row is transmitted to the server 24 in the action table.

  As described above, the processing of FIG. 13 to FIG. 21 is executed in parallel at regular time intervals, whereby the user behavior data is determined and transmitted to the server 24. And since a several determination result is contained in a user's action data, a user's state is recognized correctly.

  The processor 64 that executes the processes of steps S65 and S67 functions as a determination unit. The processor 64 that executes the process of step S69 functions as a transmission unit.

  FIG. 22 is a flowchart showing the processing of the state recognition program 314k. For example, when the power of the PC 14 is turned on by the user, the processor 64 of the PC 14 determines whether or not a first predetermined time has elapsed in step S71. For example, it is determined whether or not the value of the status counter 360 exceeds a value indicating the first predetermined time. If “NO” in the step S71, that is, if the first predetermined time has not elapsed, the process of the step S71 is repeatedly executed. On the other hand, if “YES” in the step S71, that is, if the first predetermined time has elapsed, the action data for the first predetermined time is acquired in a step S73. That is, action data from the current time to the first predetermined time before is acquired from the action table. Note that when the process of step S73 ends, the state counter 360 is reset.

  Subsequently, Ac / Pa recognition processing is executed in step S75, Ta / Li recognition processing is executed in step S77, and interest recognition processing is executed in step S79. Note that the processing in steps S75, S77, and S79 will be described later with reference to the flowcharts shown in FIGS. 23, 24, and 25, and thus detailed description thereof is omitted here.

  Subsequently, in step S81, each recognition result is stored as state data 354. For example, if the Ac / Pa recognition result is active, the Ta / Li recognition result is talk, and the interest recognition result is “robot”, the state data 354 is stored in the memory 68 as “active talk robot”. Is remembered. Subsequently, in step S83, the status data 354 is transmitted to the server 24, and the process returns to step S71.

  The processor 64 that executes the process of step S75 functions as an aggressiveness recognition unit, the processor 64 that executes the process of step S77 functions as a speaker state recognition unit, and the processor 64 that executes the process of step S79 is an interesting object. Functions as a recognition means. And the processor 64 which performs the process of step S75-81 functions as a recognition means.

  FIG. 23 is a flowchart showing the processing of the Ac / Pa recognition program 314m. The processor 64 calculates the opponent watching time WT in step S91, calculates the forward tilting time FT in step S93, and calculates the conflict frequency RF in step S95. In other words, in steps S91, S93, and S95, based on the action data acquired in step S73 of the upper routine, the time WT for watching the opponent, the time FT of the forward tilt posture, and the frequency RF of the conflict are calculated. In step S97, an AC value is calculated from each calculation result. That is, the AC value is calculated based on the formula shown in the above equation 1. Subsequently, in step S99, it is determined whether or not the calculated AC value is larger than a threshold value A.

  If “YES” in the step S99, that is, if the AC value is larger than the threshold A, the active state is set in a step S101. That is, the Ac / Pa flag 356 is set to ON. On the other hand, if “NO” in the step S99, that is, if the AC value is equal to or less than the threshold A, the passive state is set in a step S103. That is, the Ac / Pa flag 356 is set to off. In addition, if the process of step S101 or step S103 is complete | finished, an Ac / Pa recognition process is complete | finished and it returns to a state recognition process.

  FIG. 24 is a flowchart showing the processing of the Ta / Li recognition program 314n. In step S111, the processor 64 calculates the utterance amount Ta from the utterance determination result for the first predetermined time. That is, the utterance amount Ta is calculated based on the action data acquired in step S73 of the upper routine. Subsequently, in step S113, it is determined whether or not the calculated utterance amount Ta is larger than a threshold value T. If “YES” in the step S113, that is, if the utterance amount Ta is larger than the threshold T, the talk state is set in a step S115. That is, in step S115, the Ta / Li flag 358 is turned on. On the other hand, if “NO” in the step S113, that is, if the utterance amount Ta is equal to or less than the threshold value Ta, the listening state is set in a step S117. That is, in step S117, the Ta / Li flag 358 is turned off.

  And if the process of step S115 or step S117 is complete | finished, a Ta / Li recognition process will be complete | finished and it will return to a state recognition process.

  FIG. 25 is a flowchart showing the processing of the interest recognition program 314p. The processor 64 acquires action data for the second predetermined time in step S121. That is, the action data for the second predetermined time is acquired from the action data acquired in step S73, which is the upper routine. Subsequently, in step S123, the recognition rate M is calculated from the face recognition result and the line-of-sight determination result of the monitor camera 123, and in step S125, the recognition rate S is calculated from the face recognition result and the line-of-sight determination result of the abdominal camera 12. That is, in steps S123 and S125, the recognition rate M and the recognition rate S are calculated based on the behavior data for the second predetermined time acquired in step S121. The processor 64 that executes the process of step S123 functions as a first recognition rate calculation unit, and the processor 64 that executes the process of step S125 functions as a second recognition rate calculation unit.

  Subsequently, in step S127, it is determined whether each recognition rate is greater than a threshold value In. That is, it is determined whether the recognition rate M and the recognition rate S are larger than the threshold value In. If “NO” in the step S127, that is, if both the recognition rate M and the recognition rate S are equal to or less than the threshold value In, the process proceeds to a step S135. If “YES” in the step S127, that is, if the recognition rate M and the recognition rate S are larger than the threshold value In, it is determined whether or not the recognition rate M is larger than the recognition rate S in a step S129.

  If “YES” in the step S129, that is, if the recognition rate M is larger than the recognition rate S, the monitor state is set in a step S131. That is, since the user's interest is the monitor 16, data indicating “monitor” is temporarily stored in the interest result buffer 342. On the other hand, if “NO” in the step S129, that is, if the recognition rate M is equal to or less than the recognition rate S, the robot state is set in a step S133. That is, since the user's recognition target is the robot 12, the data indicating “robot” is temporarily stored in the interest target recognition result buffer 342. If the recognition rate M and the recognition rate S are less than or equal to the threshold value In, the other state is set in step S135. That is, in step S135, since the user's interest is not the robot 12 or the monitor 16, the data indicating “other” is temporarily stored in the interest recognition result buffer 342.

  In addition, if the process of step S131, S133, and step S135 is complete | finished, an interested object recognition process will be complete | finished and it will return to a state recognition process. In the present embodiment, since the object of interest is set based on the recognition rates M and S of the user's face, the object that the user sees can be accurately recognized.

  The processor 64 that executes the processes of steps S131, S133, and S135 functions as a setting unit.

  As described above, by executing the processing shown in FIGS. 22 to 25, the user status data is stored in the memory 68 and transmitted to the server 24. In another embodiment, the Ac / Pa recognition process, the Ta / Li recognition process, and the interest object recognition process may be performed in parallel. Recognize the user's interest by acquiring it every hour.

  FIG. 26 is a flowchart showing the processing of the utterance time measurement program 320. In step S141, the processor 20 determines whether or not the sound level is equal to or higher than a predetermined value. For example, it is determined whether or not the sound level of the sound collected by the microphone 20 is equal to or higher than a predetermined value that can be determined as a human speech. If “NO” in the step S141, that is, if the sound level is less than a predetermined value, the process of the step S141 is repeated. On the other hand, if “YES” in the step S141, that is, if the sound level is equal to or higher than a predetermined value, the utterance counter 362 is incremented in a step S143. That is, in step S143, the utterance counter 362 is incremented to count the utterance time. The processor 64 that executes the processes of steps S141 to S145 functions as a measurement unit.

  Subsequently, in step S145, it is determined whether or not the audio level has become less than a predetermined value. For example, it is determined whether or not the sound level of the sound collected by the microphone 20 has become less than a predetermined value. That is, if “NO” in the step S145, that is, if the sound level is equal to or higher than a predetermined value, the process returns to the step S143. On the other hand, if “YES” in the step S145, that is, if the sound level is less than a predetermined value, the utterance counter 362 is initialized in a step S147, and the process returns to the step S141. That is, since the user's utterance is completed, the utterance counter 362 for measuring the utterance time is initialized.

  In step S141 and step S145, the determination is made not only based on the sound level of the sound input to the microphone 20, but also based on the sound level of the sound output from the speaker 18. Thereby, it is also possible to measure the utterance time of the other user.

  In another embodiment, the utterance time may be measured based on the utterance determination result in the action table without using the utterance counter 362. That is, the utterance time can be measured by counting the number of times that “Yes” is continuously determined in the utterance determination. Further, in this case, for example, it becomes possible to measure the utterance time only for the user A, and when the user A and the user B speak at the same time, the utterance time of each user can be accurately measured. become. Furthermore, if all the utterance determination results during the conversation are recorded in the action table, the total utterance time during the conversation and each utterance time can be calculated.

  FIG. 27 is a flowchart showing the processing of the entire program 316a included in the robot control program 316. For example, when the telephone call by the videophone function is started, the processor 64 of the PC 14 determines whether or not the end operation is performed in step S161. For example, it is determined whether or not an operation for ending a call using the videophone function has been performed. If “YES” in the step S161, that is, if an end operation is performed, the entire process is ended. On the other hand, if “NO” in the step S161, that is, if the end operation is not performed, the state data 354 is referred to in a step S163.

  Subsequently, in step S165, it is determined whether or not the monitor state is set. That is, it is determined whether or not the status data 354 includes “monitor” indicating that the user is interested in the monitor 16. If “NO” in the step S165, that is, if the user's interest is not the monitor 16, the process proceeds to a step S175. On the other hand, if “YES” in the step S165, that is, if the user's interest is the monitor 16, it is determined whether or not it is active in a step S167. That is, it is determined whether or not the status data 354 includes “active” indicating that the user is in the active status. If “NO” in the step S167, that is, if the user is in a passive state, the process proceeds to a step S177.

  On the other hand, if “YES” in the step S167, that is, if the user is in an active state, it is determined whether or not the talk is in a step S169. That is, it is determined whether or not the status data 354 includes “talk” indicating that the user is in the talk state. If “YES” in the step S169, that is, if the user is in the talk state, the state data 354 is “active talk monitor”, so the active talk process is executed in a step S171, and the process returns to the step S161. Since this active talk process will be described later with reference to the flowchart shown in FIG. 28, a detailed description thereof is omitted here.

  If “NO” in the step S169, that is, if the user is in a listening state, the state data 354 is “active listening monitor”, so an active listening process is executed in a step S173, and the process returns to the step S161. Since this active listening process will be described later with reference to the flowchart shown in FIG. 29, a detailed description thereof will be omitted here.

  If the user's interest is other than the monitor 16, it is determined in step S175 whether the robot is in the robot state. That is, it is determined whether or not the status data 354 includes “robot” indicating that the user is interested in the robot 10. If “YES” in the step S175, that is, if the user's object of interest is the robot 10, an inactive process is executed in a step S177, and the process returns to the step S161. Since this inactive process will be described later with reference to the flowchart shown in FIG. 30, a detailed description thereof will be omitted here. On the other hand, if “NO” in the step S175, that is, if the user is not interested in the robot 10 or the monitor 16, the other process is executed in a step S179, and the process returns to the step S161. Since the other process will be described later with reference to the flowchart shown in FIG. 31, detailed description thereof is omitted here.

  In other embodiments, the processes in steps S165, S175, and S179 may be omitted, and it may be determined in step S167 whether or not the monitor is an active monitor. In addition, the processor 64 that executes the processes of steps S171, S173, S177, and S179 functions as an operation providing unit.

  FIG. 28 is a flowchart showing the processing of the active talk program 316b. Note that the processing shown in the flowcharts of FIGS. 28 to 42 will be described as being executed by the PC 14a. Therefore, “user” indicates user A, and “partner” indicates user B, that is, the partner user.

  When the process of step S171 is executed, the processor 64 of the PC 14a establishes data communication with the server 24 in step S191. That is, data communication with the server 24 is established by executing data communication processing. Subsequently, in step S193, the other party's state data is acquired. That is, the state data of user B stored in the server 24 is acquired by data communication with the server 24. The acquired partner status data is temporarily stored in the data communication buffer 344.

  Subsequently, in step S195, it is determined whether or not the partner is in a monitoring state. That is, based on the state data of the user B temporarily stored in the data communication buffer 344, it is determined whether or not the object of interest of the user B is the monitor 16b. If “NO” in the step S195, that is, if the user B is not interested in the monitor 16b, the process proceeds to a step S207. On the other hand, if “YES” in the step S195, that is, if the object of interest of the user B is the monitor 16b, it is determined whether or not the partner is active in a step S197. That is, based on the acquired state data of user B, it is determined whether or not user B is in an active state. If “NO” in the step S197, that is, if the user B is in a passive state, the process proceeds to a step S207.

  On the other hand, if “YES” in the step S197, that is, if the user B is in an active state, it is determined whether or not the other party is a talk in a step S199. That is, based on the state data of the user B, it is determined whether or not the user B is in the talk state. If “YES” in the step S199, that is, if the user B is in a talk state, the speech continuation process is executed in a step S201. When the utterance continuation process ends, the active talk process ends, and the process returns to the overall process. Since this utterance continuation process will be described later with reference to the flowchart shown in FIG. 32, detailed description thereof will be omitted here.

  If “NO” in the step S199, that is, if the user B is in the listening state, the state data of the user and the partner is temporarily stored in a step S203. That is, the status data 354 indicating the status of the user A and the status data of the user B stored in the data communication buffer 344 are temporarily stored in the status data buffer 348. Subsequently, in step S205, it is determined whether or not the utterance time is shorter than the threshold value L. That is, it is determined whether or not the utterance time of the user A counted by the utterance counter 362 is shorter than the threshold value L.

  If “YES” in the step S205, that is, if the utterance time of the user A is shorter than the threshold L, the process proceeds to a step S201. On the other hand, if “NO” in the step S205, that is, if the utterance time of the user A is equal to or more than the threshold L, the utterance suppression process is executed in a step S207 because the user A is speaking unilaterally. When the process in step S207 is completed, the active talk process is terminated and the process returns to the overall process. Note that the utterance suppression process will be described later with reference to the flowchart shown in FIG. 33, and thus detailed description thereof is omitted here.

  In the flowcharts shown in FIGS. 29 to 31, the detailed description of the same flow as that in FIG. 28 is omitted.

  FIG. 29 is a flowchart showing the processing of the active listen program 316c. When the process of step S173 is executed, the processor 64 of the PC 14a establishes data communication with the server 24 in step S211 and acquires the partner state data in step S213. Subsequently, in step S215, it is determined whether or not the partner is in a monitoring state. If “NO” in the step S215, that is, if the object of interest of the user B is other than the monitor 16b, the process proceeds to a step S227.

  If “YES” in the step S215, that is, if the object of interest of the user B is the monitor 16b, it is determined whether or not the partner is active in a step S217. If “NO” in the step S217, that is, if the user B is in a passive state, the process proceeds to a step S227. On the other hand, if “YES” in the step S217, that is, if the user B is in an active state, it is determined whether or not the other party is a talk in a step S219. If “NO” in the step S219, that is, if the user B is in the listening state, the process proceeds to a step S227. On the other hand, if “YES” in the step S219, that is, if the user B is in the talk state, the state data of the user and the other party are temporarily stored in a step S221.

  Subsequently, in step S223, it is determined whether or not the utterance time is shorter than the threshold value L. That is, it is determined whether or not the utterance time of the user B counted by the utterance counter 362 is shorter than the threshold value L. If “YES” in the step S223, that is, if the utterance time of the user B is shorter than the threshold L, the utterance continuing process is executed in a step S225. When the process of step S225 ends, the active listen process ends, and the process returns to the overall process. On the other hand, if “NO” in the step S223, that is, if the utterance time of the user B is equal to or greater than the threshold value L, the user B who is the partner user is unilaterally speaking, so the utterance promoting process in the step S227. Execute. When the process of step S227 is completed, the active listen process is terminated and the process returns to the overall process. Note that the utterance promotion process will be described later with reference to the flowchart shown in FIG. 34, and thus detailed description thereof is omitted here.

  When the speech suppression process in step S207 and the speech promotion process in step S227 are executed, the robot 10a operates to control the user A's speech. Therefore, since the user A and the user B speak in a well-balanced manner, the dialogue continues.

  FIG. 30 is a flowchart showing the processing of the inactive program 316d. When the process of step S177 is executed, the processor 64 of the PC 14a executes the speech promotion process in step S231. And if the process of step S231 is complete | finished, an inactive process will be complete | finished and it will return to the whole process. For example, by promoting the utterance of a user who is not active in the dialogue, the user is allowed to participate in the dialogue and the dialogue is continued.

  FIG. 31 is a flowchart showing the processing of the other program 316e. When the processing of step S179 is executed, the processor 64 of the PC 14a executes attention attracting processing in step S241. Then, when the process of step S241 ends, the other process ends, and the process returns to the overall process.

  Although a detailed description of the attention attracting process is omitted, when this process is executed, the robot 10a operates to attract the user A's attention. That is, even if the object of interest of the user A is the monitor 16 (user B) but not the robot 10a, the robot 10a attracts attention so that the user A is interested in the dialogue. In this way, the conversation is continued by making the user who is not in an active state interested in the conversation.

  FIG. 32 is a flowchart showing the processing of the utterance continuation program 316f. When the process of step S201 or step S225 is executed, the processor 64 of the PC 14a refers to the behavior data in step S261. That is, in step S261, the action data of the user A is referred from the action table data 352. Subsequently, in step S263, it is determined whether or not the user is speaking. That is, it is determined whether or not the utterance determination result is “Yes” in the action data of the user A. If “YES” in the step S263, that is, if the user A is speaking, an attendant pseudo-listening process is executed in a step S265. When the process of step S265 is completed, the speech continuation process is terminated and the process returns to the upper routine. Note that the attendant-like pseudo-listening process will be described later with reference to the flowchart of FIG. 36, and a detailed description thereof will be omitted here.

  If “NO” in the step S263, that is, if the user A is not speaking, the data communication with the server 24 is established in a step S267 similarly to the step S191. In step S269, the other party's action data is acquired. That is, the action data of the user B is acquired from the server 24. Subsequently, in step S271, the other party's action data is temporarily stored. That is, the acquired action data of the user B is temporarily stored in the opponent action data buffer 346.

  Subsequently, in step S273, it is determined whether or not the other party is speaking. That is, the behavior data of the user B stored in the partner behavior data buffer 346 is read, and it is determined whether or not the speech determination result is “Yes”.

  If “YES” in the step S273, that is, if the user B is speaking, the first positive pseudo-listening process is executed in a step S275. When the process of step S275 is completed, the utterance continuation process is terminated and the process returns to the upper routine. Since the first active pseudo-listening process will be described later with reference to the flowchart shown in FIG. 37, a detailed description thereof is omitted here. On the other hand, if “NO” in the step S273, that is, if the user B is not speaking, the immediately preceding operation is continued in a step S277, and if the process is ended, the utterance continuing process is ended and the process returns to the upper routine. For example, in step S277, if an operation command “view user A” has been issued in the previous process, the same operation command is issued in the current process.

  In the bystander-like pseudo-listening process and the first positive pseudo-listening process, an operation command is given to the user A or the user B so that the robot 10 performs the pseudo-listening. That is, since the user A and the user B can feel as if they are listening to their own stories by the pseudo-listening of the robot 10, the dialogue continues.

  In the flow charts shown in FIGS. 33 and 34, detailed description of the same flow as that in FIG. 32 is omitted.

  FIG. 33 is a flowchart showing the processing of the speech suppression program 316g. When the process of step S207 is executed, the processor 64 of the PC 14a acquires behavior data in step S281. Subsequently, in step S283, it is determined whether or not a speech is being made. If “YES” in the step S283, that is, if the user A is speaking, the by-participant pseudo-listening process is executed in a step S285, and if the process in the step S285 is ended, the speech suppressing process is ended. Return to the active talk process.

  On the other hand, if “NO” in the step S283, that is, if the user A is not speaking, the data communication with the server 24 is established in a step S287, and the action data of the other party is acquired in a step S289. In step S291, the other party's action data is temporarily stored. Subsequently, in step S293, it is determined whether or not the other party is speaking. If “YES” in the step S293, that is, if the user B speaks, the first positive pseudo-listening process is executed in a step S295. When the process of step S295 ends, the speech suppression process ends and the process returns to the active talk process.

  If “NO” in the step S293, that is, if the user B does not speak, the user utterance suppressing process is executed in a step S297. And if the process of step S297 is complete | finished, an utterance suppression process will be complete | finished and it will return to an active talk process.

  Although detailed description of the user utterance suppression process will be described later, when this process is executed, the robot 10 operates to suppress the user's utterance. That is, when user A speaks unilaterally, user A's speech is suppressed.

  FIG. 34 is a flowchart showing the processing of the speech promotion program 316h. When the process of step S227 or step S231 is executed, the processor 64 of the PC 14a refers to the behavior data in step S301. Subsequently, in step S303, it is determined whether or not the user is speaking. If “YES” in the step S303, that is, if the user A is speaking, an attendant pseudo-listening process is executed in a step S305. On the other hand, if “NO” in the step S303, that is, if the user A is not speaking, the data communication with the server 24 is established in a step S307, and the action data of the other party is acquired in a step S309. In step S311, the other party's action data is temporarily stored. Subsequently, in step S313, it is determined whether or not the other party is speaking. If “YES” in the step S313, that is, if the user B is speaking, the first positive pseudo-listening process is executed in a step S315.

  If “NO” in the step S313, that is, if the user B is not speaking, it is determined whether or not the user is watching the robot 10 in a step S317. That is, it is determined from the action data of the user A whether the gaze direction determination result is “robot”.

  If “YES” in the step S317, that is, if the user A is looking at the robot 10a, the user speech promoting process is executed in a step S319. And if the process of step S319 is complete | finished, a speech promotion process will be complete | finished and it will return to a high-order routine.

  Although detailed description of the user utterance promotion process will be described later, when this process is executed, the robot 10a operates so that the user A speaks. For example, if the upper routine is an inactive process, the robot 10 operates so as to promote, that is, encourage, speech so that the user A can participate in the conversation. As a result, a user who is not active in the dialogue speaks, so that the user A participates in the dialogue.

  The processor 64 that executes the processes of steps S207, S227, S297, and S319 functions as an utterance control operation giving unit. Furthermore, the processor 64 that executes the processes of steps S269, S289, and S309 functions as an acquisition unit.

  On the other hand, if “NO” in the step S317, that is, if the user A is not looking at the robot 10a, the attention attracting process is executed in a step S321. And if the process of step S321 is complete | finished, a speech promotion process will be complete | finished and it will return to a high-order routine.

  Although a detailed description of the attention attracting process will be described later, when this process is executed, the robot 10a operates to attract the user A's attention. For example, if the upper routine is an inactive process, the robot 10 operates to attract the user A's attention so that the user A can participate in the conversation. This attracts the attention of users who are not active in the dialogue and participates in the dialogue.

  Note that the processor 64 that executes the processes of steps S177, S231, S319, and S321 functions as a participation operation giving unit.

  Here, when the upper routine of the speech promotion process is the active listen process, the robot 10 operates to promote the user's speech or attract attention in the process of step S319 or step S321, so that the user A begins to speak. That is, if the user is actively listening to the other person's story for a long time, the user A is required to speak.

  FIG. 35 is a flowchart showing the processing of the attention attracting program 316j. For example, when the process of step S251 is executed, the processor 64 of the PC 14a executes a random number generation process in step S331. For example, in step S331, one random number with a small number of digits is generated in the random number generation process, and the generated random number is stored in the random number buffer 350.

  Subsequently, in step S333, it is determined whether or not the random number is an odd number. That is, it is determined whether or not the remainder when the random number stored in the random number buffer 350 is divided by “2” is “1”. If “YES” in the step S333, that is, if the random number is an odd number, an operation command “view user” is issued in a step S335. That is, an operation command for viewing the user A is given to the robot 10a. When the process of step S335 ends, the attention attracting process ends and the process returns to the upper routine.

  If “NO” in the step S333, that is, if the random number is an even number, the random number generation process is executed again in a step S337. In addition, the random number created in the process of step S337 is stored after the random number created in step S331 is deleted. That is, when the process of step S337 is executed, the random number stored in the random number buffer 350 is updated.

  Subsequently, in step S339, it is determined whether or not the random number is an odd number. That is, it is determined whether or not the random number generated in step S337 is an odd number. If “YES” in the step S339, that is, if the random number is an odd number, an operation command “speak to the user” is issued in a step S341. For example, the issued operation command causes a synthesized voice such as “Hey” to be output from the speaker 56 provided in the mouth 32. Then, the user A feels as if talking to the robot 10a by listening to the synthesized voice.

  If “NO” in the step S339, that is, if the random number is an even number, an operation command “view the same as the user” is issued in a step S343. That is, an operation command for driving the head motor 48 and the eyeball motor 50 of the robot 10a is given to the robot 10a based on the line-of-sight determination result in the action data of the user A.

  For example, if the user A's line-of-sight determination result is “robot”, the robot 10a issues an operation command for driving the head motor 48 and the eyeball motor 50 so as to look at its body 28. If the line-of-sight determination result is “monitor”, an operation command for operating the head 28 is issued so that the robot 10a looks at the monitor 16a. If the line-of-sight determination result is “other”, an operation command for operating the head 28 is issued so that the robot 10a stumbles around. When there are four or more line-of-sight determination results, the operation commands are further diversified. Further, when the line-of-sight determination result in the behavior data is indicated by spatial coordinates, the target to be watched by the user A is more accurately determined based on the space coordinate position of the robot 10a and the space coordinate position of the line-of-sight determination result. You may make it look.

  As described above, in the attention attracting process, an operation command for operating the robot 10 so as to look at the user, speak to the user, or look at the same thing as the user is randomly issued. And the user's attention is attracted by the robot 10 operating in this way.

  The user A's attention may be drawn by moving the right arm 30L or the left arm 30L. Further, the processor 64 that executes the processes of steps S321, S335, S341, and S343 functions as an attention attracting unit.

  In the flowcharts shown in FIGS. 36 to 40, the detailed description of the same flow as that in FIG. 35 is omitted.

  FIG. 36 shows a flowchart showing the processing of the by-participant pseudo-listening program 316k. The processor 64 of the PC 14a refers to the behavior data in step S361. Subsequently, in step S363, it is determined whether to start talking. For example, when the utterance time is counted by the utterance counter 362, it is determined whether or not the value of the utterance counter 362 is smaller than a predetermined value (for example, a value indicating 2 seconds). Further, when the utterance time is measured based on the utterance determination result in the action table, whether or not the utterance determination result is within a predetermined number of times (for example, two times) after “None” is changed to “Yes”. Determine whether.

  If “YES” in the step S363, that is, if the user A starts speaking, the second positive pseudo-listening process is executed in a step S367. And if the process of step S367 is complete | finished, an attendant pseudo-listening process will be complete | finished and it will return to a high-order routine. Since the second active pseudo-listening process will be described later with reference to the flowchart shown in FIG. 39, detailed description thereof is omitted here.

  If “NO” in the step S363, it is determined whether or not the robot 10a is viewed in a step S365. That is, it is determined whether or not the line-of-sight direction determination is “robot” based on the action data of the user A referred to. If “YES” in the step S365, that is, if the user A is looking at the robot 10a, the process proceeds to a step S367. On the other hand, if “NO” in the step S365, that is, if the user A is not looking at the robot 10a, a random number generation process is executed in a step S369, and it is determined whether or not the random number generated in the step S371 is an odd number. To do. If “NO” in the step S371, that is, if the random number is an even number, the process proceeds to a step S367. On the other hand, if “YES” in the step S371, that is, if the random number is an odd number, the first positive pseudo-listening process is executed in a step S373. Then, when the process of step S373 is finished, the attendant pseudo-listening process is finished, and the process returns to the upper routine.

  Thus, in the by-participant pseudo-listening process, the first pseudo-listening process for performing pseudo-listening with respect to the user B and the second pseudo-listening process for performing pseudo-listening with respect to the user A include the utterance of the user A. Or based on the line of sight.

  FIG. 37 shows a flowchart showing the processing of the first active pseudo-listening program 316m. For example, when the process of step S373 is executed, the processor 64 of the PC 14a executes a random number generation process in step S381, and determines in step S383 whether or not the generated random number is an odd number. If “YES” in the step S383, that is, if the random number is an odd number, an operation command “face to the monitor 16a” is issued in a step S385. That is, an operation command for driving the head motor 48 and the eyeball motor 50 so that the head direction of the head 28 and the line-of-sight direction of the eyeball 34 are directed in the direction of the monitor 16a on which the image of the user B is displayed. To 10a. Then, when the process of step S385 ends, the first active pseudo-listening process ends, and the process returns to the upper routine.

  If “NO” in the step S383, that is, if the random number is an even number, the random number generation process is executed again in a step S387, and it is determined whether or not the random number is an odd number in a step S389. If “YES” in the step S389, that is, if the random number is an odd number, an operation command “tilt the body toward the monitor 16a” is issued in a step S391. That is, an operation command for driving the waist motor 52 is given to the robot 10a so that the body (the head 26, the torso 28, etc.) is tilted toward the monitor 16a. And if the process of step S391 is complete | finished, a 1st active pseudo-listening process will be complete | finished, and it returns to a high-order routine.

  Further, if “NO” in the step S389, that is, if the random number is an even number, an operation instruction “depress the companion” is issued in a step S393. For example, a synthetic voice such as “Ah” is output from the speaker 56 like a human interaction, and an operation command for driving the head motor 48 so that the head 26 crawls toward the monitor 16a is given to the robot 10a. To do. When the process of step S393 is completed, the first positive pseudo-listening process is terminated, and the process returns to the upper routine.

  In this way, in the first active pseudo-listening process, the face of the robot 10 (the direction of the head 26 and the eyeball 34) is directed toward the monitor 16 on which the image of the other user is reflected, or the body of the robot 10 is tilted. The robot 10 operates as if it is listening to the other user by reciprocal. Therefore, even if the user A is not listening to the user B's story, the robot 10a can pseudo-listen to the user B's story, so that the user B can feel as if he / she is listening to his / her own story.

  In addition, the processor 64 which performs the process of step SS275, S295, S315, S373, S385, S391, and S393 functions as an other party listening action provision means.

  FIG. 38 is a flowchart showing the processing of the user utterance suppression program 316n. When the process of step S297 is executed, the processor 64 of the PC 14a refers to the action data in step S401. Subsequently, in step S403, as in step S365, it is determined whether or not the robot 10a is being viewed. If “NO” in the step 403, that is, if the user A is not looking at the robot 10a, the attention attracting process is executed in a step S405. And if the process of step S405 is complete | finished, a user utterance suppression process will be complete | finished and it will return to an utterance suppression process. In addition, the processor 64 which performs the process of step S179, S241, S321, S335, S341, S343, and S405 functions as an interest motion giving means.

  If “YES” in the step S403, that is, if the user A is not looking at the robot 10a, a random number generation process is executed in a step S407, and it is determined whether or not the random number is an odd number in a step S409. If “YES” in the step S409, that is, if the random number is an odd number, an operation command “view user” is issued in a step S411. That is, an operation command for driving the head motor 48 and the eyeball motor 50 is given to the robot 10a so that the head 28 and the eyeball 34 are directed toward the user A. And if the process of step S411 is complete | finished, a user utterance suppression process will be complete | finished and it will return to an utterance suppression process.

  On the other hand, if “NO” in the step S409, the random number generation process is executed again in a step S413, and it is determined whether or not the random number is an odd number in a step S415. If “YES” in the step S415, that is, if the random number is an odd number, an operation command “stop utterance” is issued in a step S417. That is, in step S417, an operation command for outputting the synthesized voice that stops the speech from the speaker 56 is given to the robot 10a. Synthetic speech that stops speech is “wait a moment”. And if the process of step S417 is complete | finished, a user's speech suppression process will be complete | finished and it will return to a speech suppression process.

  Furthermore, if “NO” in the step S415, that is, if the random number is an even number, the random number generation process is executed again in a step S419, and it is determined whether or not the random number is an odd number in a step S421. If “YES” in the step S421, that is, if the generated random number is an odd number, an operation command “point to the monitor 16a” is issued in a step S423. That is, an operation command for driving the right arm motor 46R or the left arm motor 46L is given to the robot 10a so that the tip of the left arm 30L or the right arm 30R points toward the monitor 16a. And if the process of step S423 is complete | finished, a user utterance suppression process will be complete | finished and it will return to an utterance suppression process.

  If “NO” in the step S421, that is, if the generated random number is an even number, the attention guidance process is executed in a step S425. Moreover, if the process of step S425 is complete | finished, a user utterance suppression process will be complete | finished and it will return to an utterance suppression process. Note that the attention guidance process will be described later with reference to the flowchart shown in FIG. 40, and thus detailed description thereof will be omitted.

  In this way, in the user utterance suppression process, the robot 10a watches the user A, utters a voice that stops the utterance, instructs the monitor 16a, attracts attention to the user A who speaks actively, Suppress utterances by inducing attention. Accordingly, since the user's attention is guided, the utterance is suppressed. In addition, when the user's attention is guided to the other party, the other party can have an opportunity to speak, and thus the dialogue is further sustained.

  In addition, the processor 64 which performs the process of step S297, S405, S411, S417, S423, and S425 functions as an utterance suppression operation | movement provision means.

  FIG. 39 is a flowchart showing the processing of the second active pseudo-listening program 316p. When the process of step S367 is executed, the processor 64 of the PC 14a executes a random number generation process in step S431, and determines whether or not the random number is an odd number in step S433. If “YES” in the step S433, that is, if the random number is an odd number, an operation command “face the user” is issued in a step S435. That is, an operation command for driving the head motor 48 and the eyeball motor 50 is given to the robot 10a so that the face (head 26 and eyeball 34) faces the direction of the user A. Moreover, if the process of step S435 is complete | finished, a 2nd active doubt listening process will be complete | finished, and it will return to a by-participant pseudo-listening process.

  On the other hand, if “NO” in the step S433, that is, if the random number is an even number, a random number generation process is executed in a step S437, and it is determined again whether or not the random number is an odd number in a step S439. If “YES” in the step S439, that is, if the random number is an odd number, an operation command “tilt the user A side” is issued in a step S441. That is, an operation command for driving the waist motor 52 is given to the robot 10a so that the body (head 26 and torso 28) of the robot 10a is inclined toward the user A side.

  If “NO” in the step S439, that is, if the random number is an even number, an operation instruction “depress the companion” is issued in a step S443. That is, the head motor 48 is driven so that the head 26 crawls toward the user A side, and an operation command for outputting synthesized speech like a human interaction from the speaker 56 is given to the robot 10a. And if the process of step S443 is complete | finished, a 2nd positive pseudo-listening process will be complete | finished, and it will return to an attendant pseudo-listening process.

  In this way, the robot 10a performs an operation as if listening to the user A's utterance by turning his / her face, tilting his / her body, or listening to the user A.

  The processor 64 that executes the processes of steps S171, S173, S201, S225, S265, S285, S305, S367, S435, S441, and S443 functions as a listening operation imparting unit.

  FIG. 40 is a flowchart showing the processing of the attention guidance program 316q. When the process of step S425 is executed, the processor 64 of the PC 14a executes a random number generation process in step S451, and determines whether or not the random number is an odd number in step S453. If “YES” in the step S453, that is, if the random number is an odd number, an operation command “view the monitor 16a” is issued in a step S455. For example, in step S455, an operation command for driving the head motor 48 and the eyeball motor 50 is given to the robot 10a so that the head direction of the head 28 and the line-of-sight direction of the eyeball 34 are directed toward the monitor 16a. . And if the process of step S455 is complete | finished, an attention guidance process will be complete | finished and it will return to a user utterance suppression process.

  On the other hand, if “NO” in the step S453, that is, if the random number is an even number, the random number generation process is executed again in a step S457, and it is determined whether or not the random number is an odd number in a step S459. If “YES” in the step S459, that is, if the random number is an odd number, the process proceeds to a step S455. On the other hand, if “NO” in the step S459, that is, if the generated random number is an even number, an operation command “view user A” is issued in a step S461 similarly to the step S411. And if the process of step S461 is complete | finished, an attention guidance process will be complete | finished and it will return to a user utterance suppression process.

  As described above, when the attention inducing process is executed several times, the user A and the monitor 16a are alternately viewed so that the user A's attention is induced. In particular, in this embodiment, the robot 10a is set so that the number of times of watching the monitor 16a is larger than that of the user A, so that the user A's attention is guided to the monitor 16a, that is, the user B. In another embodiment, the user A's attention may be more effectively guided by repeating the attention guidance process a plurality of times.

  The processor 64 that executes the processes of steps S425, S455, and S461 functions as a caution guiding unit.

  FIG. 41 is a flowchart showing the processing of the user utterance promotion program 316r. When the process of step S319 is executed, the processor 64 of the PC 14a issues an operation command “Ask the user a question” in step S471. For example, in step S471, an operation command is output to the robot 10a so that a synthesized voice such as “What should I ask about today's meal?” Is output from the speaker 56 provided in the mouth 32. And if the process of step S471 is complete | finished, a user's speech promotion process will be complete | finished and it will return to a speech promotion process.

  Thus, when the robot 10a speaks that the user A asks the user B a question, the user B speaks, and the dialogue continues.

  Note that the processor 64 that executes the processes of steps S227, S319, and S471 functions as an utterance promoting operation imparting unit. In addition, the processor 64 that executes the processes of steps S319 and S471 functions as a speech prompting unit.

  FIG. 42 is a flowchart showing the processing of the camera control program 318. When a call is started by the videophone function, the processor 64 of the PC 14a transmits an image of the monitor camera 22a in step S481. That is, the image data used in the videophone function is transmitted to the PC 14b via the network 200. Subsequently, in step S483, an interest recognition result is referred to. That is, the recognition result of the object of interest included in the state data 354 is referred to.

  Subsequently, in step S485, it is determined whether or not the state is the other state. That is, it is determined whether or not the recognition result of the object of interest of the user A is “other”. If “YES” in the step S485, that is, if the user A is not looking at the robot 10a or the monitor 16a, the process returns to the step S483. On the other hand, if “NO” in the step S485, that is, if the user A is looking at the robot 10a or the monitor 16a, it is determined whether or not the monitor state is in a step S487. That is, it is determined whether or not the object of interest of the user A is the monitor 16.

  If “YES” in the step S487, that is, if the object of interest of the user A is the monitor 16a, the image of the monitor camera 22a is transmitted in a step S489. That is, if the user A looks at the monitor 16 and the face of the user A is photographed from the front by the monitor camera 22a, the image of the monitor camera 22a is transmitted to the PC 14b in step S489. On the other hand, if “NO” in the step S487, that is, if the object of interest of the user A is the robot 10a, the image of the abdominal camera 12a is transmitted in a step S491. That is, if the user A looks at the robot 10a and the face of the user A is photographed from the front by the abdominal camera 12a, the image of the abdominal camera 12a is transmitted to the PC 14b in step S491.

  In addition, if the process of step S489 or step S491 is completed, the process returns to step S483. The processor 64 that executes the processes of steps S489 and S491 functions as an image transmission unit.

  Thereby, for example, an image in which the face of the user A is recognized is transmitted to the PC 14b, so that the face of the user A is always displayed in the image displayed on the monitor 16b. Therefore, even if the user A is in a state where the user A is talking to the robot 10a, the user B feels that the user B is talking to the user B himself, so that the dialogue continues.

  According to this embodiment, the listening dialogue sustaining system 100 includes a monitor 16 on which the other user is displayed, a microphone 20 on which the user's voice is collected, and a PC 14 connected to the monitor camera 22 that photographs the user, and an abdominal camera. And a robot 10 provided with 12.

  In the PC 14, the user's action is determined based on the user's image taken by the monitor camera 22 and the abdominal camera 12 and the sound collected by the microphone 20, and is further stored in the memory 68. Further, the PC 14 recognizes the state of the user from the action data for the first predetermined time. For example, if the user's state is recognized as “active talk monitor”, the robot 10 is given an operation command from the PC 14 so as to perform pseudo-listening to the user. Further, if the user's state is recognized as “passive listen monitor”, the robot 10 draws attention by talking to the user and causes the user to participate in the dialogue.

  Thus, since the robot 10 operates so as to maintain the dialogue according to the state of the user, the dialogue between persons with communication disabilities can be sustained.

  In addition to the abdominal camera 12 and the monitor camera 22, a third camera that captures the face of the user at a position other than the monitor 16 and the robot 10 is installed, and whether the user's interest is “other” by the third camera. You may make it complementarily determine whether it is no. In other words, the third camera can distinguish between a state in which the user's face recognition has failed and a state in which the user's interest is “other”. Then, for example, by not reflecting the action data when the face recognition fails in the interest recognition process, the listening dialogue sustaining system 100 may recognize the user's interest with high accuracy.

  Furthermore, a fourth camera for specifying the user's line-of-sight direction may be installed, and the determination of whether or not the robot 10 is in the user's field of view may be performed complementarily by the fourth camera. Then, in the listening dialogue sustaining system 100, the necessity of utterance by the robot 10 or the user's attention may be determined based on the determination result. For example, if the robot 10 is not in the user's field of view, the PC 14 operates the robot 10 to speak. On the other hand, if the robot 10 is in the user's field of view, the PC 14 always operates the robot 10 so as to attract the user's attention.

  In this way, by installing three or more cameras, it is possible to improve the recognition accuracy of the object of interest or to operate the robot 10 more effectively. Note that the above two determinations may be performed by one camera instead of the third camera and the fourth camera. Further, by installing more cameras, it is possible to improve the accuracy of the determination of the user's action and the recognition of the state, or to make the determination of the operation command given to the robot 10 more advanced.

  In another embodiment, the PC 14 may be built in the robot 10, and in this case, the robot 10 autonomously performs a listening operation and a speech function. Further, in this case, the robot 10 has an interaction-oriented function having a function of executing a communication action including at least one of a body motion such as gesture gesture and a voice with a communication target (communication target) such as a user. Or a robot (communication robot).

  Further, the user's image and sound may be transmitted to and received from the PC 14 via a telephone network or the like without using the PC 14 and the network 200. If the monitor 16 or the like has a telephone function, the video phone function may be realized by using only the monitor 16, the speaker 18, the microphone 20, and the monitor camera 22 without using the PC 14. And the monitor 16, the speaker 18, the microphone 20, and the monitor camera 22 may be integrated in the same housing | casing. Further, in this case, a videophone call is started with the robot 10 and the videophone connected.

  Further, instead of the PC 14, the user's action and state may be determined and recognized by the server 24, and an operation command may be given to the robot 10. In this case, the image captured by the abdominal camera 12 and the monitor camera 22 and the sound collected by the microphone 20 are directly transmitted to the server 24, and the robot 10 operates according to an operation command from the server 24. Further, only the process for recognizing the user state may be executed by the server 24. That is, the processing shown in FIGS. 22 to 25 is executed by the processor 80 of the server 24, and the PC 14 acquires the state recognition result every first predetermined time.

  As another example of the listening dialogue sustaining system 100, the robot 10 a performs “pseudo-listening operation”, “utterance control operation” based on only the state data of the user A regardless of the state recognition results of the user A and the user B. And “attracting attention” may be performed. For example, if the user A speaks, the robot 10a performs a pseudo-listening operation, and if the speaking time of the user A becomes longer, the robot 10a looks at the user A and suppresses the utterance. If the user A is not speaking or is not looking at the monitor 16a and the robot 10a, the robot 10a performs an operation to attract the user A's attention.

  And state data may be memorize | stored in the format like the table shown in FIG. 6 similarly to action data. Further, when the state data is stored in a table format, it may be used as log data indicating a user state change. A graph created based on the state data table may be displayed on the monitor 16.

10a, 10b ... Robot 12a, 12b ... Abdominal camera 14a, 14b ... PC
16a, 16b ... monitor 20a, 20b ... microphone 22a, 22b ... monitor camera 24 ... server 26 ... head 28 ... trunk 30R ... right arm 30L ... left arm 32 ... mouth 34 ... eyeball 46R ... right arm motor 46L ... left arm motor 48 ... head Motor 50 ... Eye motor 52 ... Waist motor 56 ... Speaker 64 ... Processor 68 ... Memory 70 ... Line of sight server 72 ... Communication LAN board 70 ... Wireless communication device 100 ... Listening dialogue sustaining system 200 ... Network

Claims (14)

A listening dialogue sustaining system comprising a video phone and a robot including a first camera and a microphone,
Determining means for determining a user's action based on an image captured by the first camera and sound collected by the microphone;
Recognizing means for recognizing the state of the user from the action for the first predetermined time determined by the determining means, and the robot so as to maintain the conversation based on the state of the user recognized by the recognizing means. A listening dialogue sustaining system, comprising an action imparting means for operating the device. The recognizing means includes positive recognition means for recognizing active and inactive dialogues,
2. The listening dialogue sustaining system according to claim 1, wherein the motion giving unit includes a listening motion giving unit that causes the robot to perform a pseudo-listening to the user based on a recognition result of the positiveness recognition unit. The recognizing means further includes a speaker state recognizing means for recognizing a listener side state and a speaker side state in a dialogue,
It further comprises measuring means for measuring the utterance time of the user or the other party,
3. The listening according to claim 1, wherein the action giving unit further includes an utterance control action giving unit that moves the robot so as to control the user's utterance based on the utterance time measured by the measuring unit. Dialog persistence system.   The utterance control means is an active utterance state that is recognized as active by the aggressiveness recognition means and is recognized as a speaker side state by the speaker state recognition means, and when the utterance time exceeds a threshold value, The listening dialogue sustaining system according to claim 3, further comprising an utterance suppression operation giving unit that operates the robot so that the user's utterance is suppressed.   5. The listening dialogue sustaining system according to claim 4, wherein the utterance suppression operation giving unit includes a caution induction unit that operates the robot so as to guide the user's attention.   When the utterance control means is in an active listening state that is recognized as active by the positiveness recognition means and is recognized as a listening state by the speaker state recognition means, the utterance of the user is promoted. The listening dialogue sustaining system according to any one of claims 3 to 5, further comprising an utterance promotion operation giving means for operating the robot.   6. The operation giving means further includes participation action giving means for operating the robot so that the user participates in a dialogue when the action recognition means is recognized as inactive by the positiveness recognition means. Listening dialog system described in Crab.   The listening dialogue sustaining system according to claim 7, wherein the participation motion giving means includes attention attracting means for operating the robot to attract the user's attention.   The listening dialogue sustaining system according to claim 7 or 8, wherein the participation motion giving unit further includes an utterance prompting unit that operates the robot so as to prompt the user to speak. A network to which the robot is connected;
A server connected to the network;
A transmission means for transmitting the action of the user determined by the determination means to the server; and an acquisition means for acquiring the action of the other user from the server,
The said action provision means further contains the other party listening action provision means which makes the said robot perform pseudo-listening with respect to the said other party user based on the other party's action acquired by the said acquisition means, and a user's action. 9. The listening dialogue sustaining system according to any of 9 The recognition means further includes an interest object recognition means for recognizing the interest object of the user,
11. The action giving means further includes an interest action giving means for operating the robot so that the user is interested in a dialogue based on a recognition result by the interest object recognition means. The dialogue listening and sustaining system described. The robot includes a second camera;
Face recognition means for executing face recognition processing on each of the images from the first camera and the second camera;
The interesting object recognition means includes a first recognition rate calculation means for calculating a first recognition rate of the first camera from a face recognition result for a second predetermined time by the face recognition means, and a second predetermined time by the face recognition means. And a setting means for setting a recognition result based on the first recognition rate and the second recognition rate. 11. The dialogue listening and sustaining system according to 11.   13. The dialogue listening sustaining system according to claim 12, further comprising image transmission means for transmitting either the image of the first camera or the second camera based on the object of interest set by the setting means. The determination unit includes a posture determination unit that determines the posture of the user, an utterance determination unit that determines the presence or absence of the user's utterance, a head direction determination unit that determines the head direction of the user, and a gaze direction of the user. Gaze direction determining means for determining;
The said user action is determined based on a posture determination result, an utterance determination result, a head direction determination result, a gaze direction determination result, a whisper determination result, and a conflict determination result. Interactive listening system.

Images