JP5186723B2 - Communication robot system and communication robot gaze control method - Google Patents

Description

  The present invention relates to a communication robot system, and in particular, to a communication robot system that detects a human line of sight, for example.

For example, Patent Document 1 by the present applicant discloses a communication robot that gazes at the position of an object detected based on a camera image. In this technique, it is possible to turn the eyes to a human face by detecting the human face as an object. Then, by gazing at the position of the detected face, the person's face is watched or the line of sight with the person is expressed.
JP 2004-42151 A

  However, in the prior art, since it is impossible to detect a person's line of sight, it is impossible to grasp where the person is looking. Accordingly, it has been difficult to exchange the line of sight with the person, such as changing the line of sight according to the line of sight of the person. In addition, since the position of a person's eyes cannot be accurately detected, it is difficult to gaze at the person's eyes. Therefore, it is difficult for a person to realize that the eyes are stared or not seen from the robot.

  Therefore, a main object of the present invention is to provide a communication robot system capable of exchanging gaze with a person.

  Another object of the present invention is to provide a communication robot system that can make a person feel strongly what is being seen or not being seen.

A first invention is a communication robot system including a robot that performs a communication action with a person, an infrared light emitting device that is attached to a person and emits infrared rays toward the human eye, and is attached to the human eye A first infrared camera that captures an infrared image of the camera, a visual field camera that is mounted on a person and that captures a visual field image of the person, a gazing point detection unit that detects a human gazing point based on the infrared image and the visual field image, and a robot second infrared camera for capturing an infrared image, is mounted on the robot, the eye position detection means for detecting a position of the human eye by detecting the infrared radiation reflected to the human eye by the second infrared camera, and the gazing point detected Gaze control means for moving the gaze according to the gaze point of the person detected by the means and the eye position of the person detected by the eye position detection means. A Interview Nikeshon robot system.
The second invention is dependent on the first invention, and the gaze control means detects the person detected by the eye position detection means when it is determined that the person is seen based on the detection result by the gazing point detection means. It is a communication robot system that directs the line of sight to the eye position.

In the second invention, when it is determined that the person is watching based on the detected gaze point of the person, the line of sight control means directs the line of sight to the position of the person's eye. Therefore, when seen by a person, the person's eyes can be looked back.

The third invention is dependent on the first invention, and the gaze control means detects the person detected by the eye position detection means when it is determined that the person is seen based on the detection result by the gazing point detection means. It is a communication robot system that directs the line of sight other than the eye position.
A fourth invention is dependent on the third invention, and the line-of-sight control means directs the line of sight to a position other than the position of the human eye by selectively setting a position separated from the position of the human eye by a predetermined distance or more. , A communication robot system.

In 3rd or 4th invention, when it determines with the said person seeing based on the detected gaze point of the detected person, a gaze control means directs a gaze other than the position of the said person's eyes. Therefore, when seen by a person, the line of sight can be diverted from the person's eyes.

The fifth invention is dependent on the first invention, and the gaze control means detects the person detected by the eye position detection means when it is determined that the person is not seen based on the detection result by the gazing point detection means. It is a communication robot system that directs the line of sight to the eye position.

In the fifth invention, when it is determined that the person is not seen based on the detected line of sight of the person, the line of sight control means directs the line of sight to the position of the person's eye. Therefore, when not seen by a person, the person's eyes can be seen.

A sixth invention is according to the first invention, and when it is determined that the line-of-sight control means is not seen by a person based on a detection result by the gaze point detection means, the line of sight is in the same direction as the person's gaze point. Is a communication robot system.

In 6th invention, when it determines with the said person not being seen based on the detected gaze point of a person, a gaze direction is turned to the same direction as the said gaze point of the said person by a gaze control means. Therefore, when not being seen by a person, the person's line of sight can be followed.

A seventh invention includes a robot that performs a communication action with a person, and is attached to a person and irradiates infrared rays toward the eyes of the person, and an infrared image of the person's eyes that is attached to the person A first infrared camera that is attached to a person and that captures a visual field image of the person, a gazing point detection unit that detects a gazing point of the person based on the infrared image and the visual field image, and an infrared ray that is attached to the robot. A second infrared camera for taking an image; an eye position detecting means for detecting the position of the human eye by detecting an infrared ray attached to the robot and reflected by the second infrared camera; Ru and a sight line control means for moving the line of sight in accordance with the position of the human eye which is detected by the gazing point and the eye position detecting means of the person detected by the communication robot system Kicking a line of sight control method of the robot, the first determining step of determining whether the robot is watching the person, if it is determined that the robot is watching the person in the first determination step, people cutlet to watch the robot A second determination step for determining whether or not the mutual gaze time when the robot is looking at a person is less than a first threshold; and when the robot does not determine that the robot is looking at a person at the first determination step, the mutual gaze time is a second threshold value When it is determined that the mutual gaze time is less than the first threshold in the third judgment step and the second judgment step for judging whether or not the time is less than the first threshold, or when the mutual gaze time is less than the second threshold in the third judgment step When the person is watching the robot, it is determined that the mutual gaze time is less than the first threshold in the first gaze control step and the second judgment step for controlling the gaze so as to look at the position of the human eye Or a second line-of-sight control step for controlling the line of sight so as to see the same direction as the person when the person is not looking at the robot when it is determined in the third determination step that the mutual gaze time is less than the second threshold. When the second determination step does not determine that the mutual gaze time is less than the first threshold value or when the third determination step does not determine that the mutual gaze time is less than the second threshold value, the person watches the robot. The third gaze control step for controlling the gaze so as to look outside the position of the human eye, and the second judgment step when the mutual gaze time is not judged to be less than the first threshold, or the third A fourth line-of-sight control step of controlling the line of sight so as to look at the position of the eyes of the person when the person is not looking at the robot when the determination step does not determine that the mutual gaze time is less than the second threshold; community This is a line-of-sight control method for a communication robot.

According to the seventh aspect , when the time for which the person and the robot stare out exceeds a predetermined time, it is possible to stop the staring by looking away from the person's eyes. For this reason, it is possible to stare at people for an appropriate time, and more natural dialogue and communication can be performed.

According to the seventh invention, when the mutual gaze time is smaller than the second threshold, the line of sight may be directed to the position of the human eye. In such a configuration, it is possible to control the line of sight so that the person and the robot stare at each other until the predetermined time has passed. For this reason, it is possible to stare at people for an appropriate time, and more natural dialogue and communication can be performed. Also, for example, if the second threshold value is set to a value smaller than the first threshold value and the mutual gaze time is subtracted when the two are not staring, when the mutual gaze time exceeds the first threshold value, the stare is fixed. It is possible to control the line of sight so that when the mutual gaze time becomes smaller than the second threshold after that, the gaze is stopped again. Therefore, since it is possible to control the line of sight so as to stare at the appropriate time and not to keep an eye on the appropriate time, more natural dialogue and communication can be performed.

  According to this invention, since the line of sight of a person is detected and the line of sight is moved according to the line of sight of the person, the line of sight can be exchanged with the person. In addition, since the line of sight of the communication robot is moved according to the line of sight of the person, it is possible to make the person strongly feel that the communication robot is seen or not seen. Therefore, it becomes possible to execute more natural dialogue and communication.

  The above object, other objects, features and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

  Referring to FIG. 1, a communication robot system (hereinafter simply referred to as “system”) 200 according to this embodiment includes a communication robot (hereinafter simply referred to as “robot”) 10 and a human 12 that is a communication target. A line-of-sight measurement device 14 for detecting the line of sight is included. The robot 10 has a function of interacting with the human 12 using at least one of gestures and utterances. For example, the robot 10 is active while interacting with the human 12 in daily life, and is an equivalent partner in the same environment as the human 12. It is possible to behave. The robot 10 can operate according to the line of sight of the human 12 measured using the line-of-sight measurement device 14 in communication with the human 12.

  The line-of-sight measurement device 14 includes a visual field camera 16, an infrared light emitting device 18, and an infrared camera 20, which are attached to the human 12 using, for example, a cap 22 and are portable computers carried on the human 12, for example. 24.

  The visual field camera 16 is, for example, a color video camera, captures an image of the visual field of the human 12, and outputs the video signal to the computer 24. The infrared light emitting device 18 includes an infrared LED and irradiates infrared rays toward the eyes of the human 12. The infrared camera 20 is, for example, a video camera with an infrared filter, and takes an infrared video image of the human 12 eye and outputs the video signal to the computer 24.

  The computer 24 detects the line of sight of the person 12 based on, for example, an infrared image and a visual field image at regular intervals. The infrared image captured by the infrared camera 20 includes a pupil and a bright spot (corneal reflection image) obtained by reflecting infrared rays from the infrared light emitting device 18 on the cornea. The computer 24 detects the eye movement of the human 12 based on the positions of the pupil and the bright spot in the infrared image, and detects the line of sight and the gazing point of the human 12 by associating the rotational movement of the eye with the visual field image. Further, by providing a set of the infrared light emitting device 18 and the infrared camera 20 for both eyes and performing simultaneous measurement of both eyes, the convergence angle and the gaze point distance can be measured. Further, the computer 24 can communicate with the robot 10 wirelessly, and can communicate with the robot 10 via a network such as a wireless LAN or directly. The computer 24 transmits information regarding the detected line of sight or gaze point of the human 12, a visual field image captured by the visual field camera 16, and the like to the robot 10.

  As an example of the line-of-sight measurement device 14 as described above, the technique of the line-of-sight measurement system “eye mark recorder” (http://www.eyemark.jp/) of NAC Image Technology Co., Ltd. can be applied.

  With reference to FIG. 2, the robot 10 includes a carriage 26, and wheels 28 for autonomously moving the robot 10 are provided on the lower surface of the carriage 26. The wheel 28 is driven by a wheel motor (indicated by reference numeral “30” in FIG. 3), and the carriage 26, that is, the robot 10 can be moved in any direction. Although not shown in FIG. 2, a collision sensor (indicated by reference numeral “32” in FIG. 3) is attached to the front surface of the carriage 26, and this collision sensor 32 is used for a person or other person to the carriage 26. Detects contact with obstacles. When contact is detected while the robot 10 is moving, the driving of the wheels 28 can be stopped immediately.

  A polygonal column sensor mounting panel 34 is provided on the carriage 26, and an ultrasonic distance sensor 36 is mounted on each surface of the sensor mounting panel 34. The ultrasonic distance sensor 36 is for measuring the distance between the mounting panel 34, that is, the human 12 around the robot 10.

  Further, the body of the robot 10 is mounted on the carriage 26 so that the lower portion of the body is surrounded by the mounting panel 34 described above and stands upright. This body is composed of a lower body 38 and an upper body 40, and the lower body 38 and the upper body 40 are connected by a connecting portion 42. Although not shown, the connecting portion 42 has a built-in lifting mechanism, and the height of the upper body 40, that is, the height of the robot 10 can be changed by using the lifting mechanism. The lifting mechanism is driven by a waist motor (indicated by reference numeral “44” in FIG. 3).

  The upper body 40 is provided with one omnidirectional camera 46. The omnidirectional camera 46 is installed, for example, on a column 48 extending from substantially the center of the upper end on the back side. The omnidirectional camera 46 captures the surroundings of the robot 10 and is distinguished from an eye camera 50 described later. As this omnidirectional camera 46, for example, a camera using a solid-state imaging device such as a CCD or a CMOS can be adopted. In addition, one microphone 52 is provided in the approximate center of the front side of the upper body 40. The microphone 52 captures ambient sounds, particularly the voice of the person 12 who is a communication target. The installation positions of the omnidirectional camera 46 and the microphone 52 are not limited to the upper body 40 and can be changed as appropriate.

  Upper arms 56R and 56L are attached to both shoulders of upper body 40 by shoulder joints 54R and 54L, respectively. The shoulder joints 54R and 54L each have three axes of freedom. That is, the right shoulder joint 54R can control the angle of the upper arm 56R around each of the X, Y, and Z axes. The Y axis is an axis parallel to the longitudinal direction (or axis) of the upper arm 56R, and the X axis and the Z axis are axes orthogonal to the Y axis from different directions. The left shoulder joint 54L can control the angle of the upper arm 56L around each of the A, B, and C axes. The B axis is an axis parallel to the longitudinal direction (or axis) of the upper arm 56L, and the A axis and the C axis are axes orthogonal to the B axis from different directions.

  Forearms 60R and 60L are attached to the tips of upper arms 56R and 56L via elbow joints 58R and 58L. The elbow joints 58R and 58L can control the angles of the forearms 60R and 60L around the axes of the W axis and the D axis, respectively.

  In the X, Y, Z, W axis and A, B, C, and D axes that control the displacement of the upper arms 56R and 56L and the forearms 60R and 60L, “0 degree” is the home position. The upper arms 56R and 56L and the forearms 60R and 60L are directed downward.

  Although not shown in FIG. 2, touch sensors (FIG. 3) are provided on the shoulder portion of the upper body 40 including the shoulder joints 54R and 54L and the above-described upper arms 56R and 56L and the arm portions including the forearms 60R and 60L, respectively. And the touch sensor 62 detects whether or not the human 12 has touched these parts of the robot 10.

  Spheres 64R and 64L corresponding to hands are fixedly attached to the tips of the forearms 60R and 60L, respectively. Instead of the spheres 64R and 64L, a “hand” in the shape of a human hand can be used when a finger function is required unlike the robot 10 of this embodiment.

  A head 68 is attached to an upper center of the upper body 40 via a neck joint 66. The neck joint 66 has three degrees of freedom and can be controlled in angle around each of the S, T, and U axes. The S-axis is an axis that goes directly from the neck, and the T-axis and the U-axis are axes that are orthogonal to the S-axis in different directions. The head 68 is provided with a speaker 70 at a position corresponding to a human mouth. The speaker 70 is used for the robot 10 to communicate with the surrounding people by voice or voice. Further, the speaker 70 may be provided in another part of the robot 10, for example, the trunk.

  The head 68 is provided with eyeballs 72R and 72L at positions corresponding to the eyes. Eyeball portions 72R and 72L include eye cameras 50R and 50L, respectively. The left and right eyeball portions 72R and 72L may be collectively indicated by reference numeral “72”, and the left and right eye cameras 50R and 50L may be collectively indicated by reference numeral “50”. The eye camera 50 captures the image of the face of the human 12 approaching the robot 10, other parts or objects, and the like. As the eye camera 50, an image sensor similar to the omnidirectional camera 46 described above can be used.

  For example, the eye camera 50 is fixed in the eyeball part 72, and the eyeball part 72 is attached to a predetermined position in the head 68 via an eyeball support part (not shown). The eyeball support unit has two degrees of freedom and can be controlled in angle around each of the α axis and the β axis. The α axis and the β axis are axes set with respect to the head 68, the α axis is an axis in the direction toward the top of the head 68, the β axis is orthogonal to the α axis and the front side of the head 68 It is an axis in a direction orthogonal to the direction in which (face) faces. In this embodiment, when the head 68 is at the home position, the α axis is set to be parallel to the S axis and the β axis is set to be parallel to the U axis. In such a head 68, when the eyeball support portion is rotated around each of the α axis and β axis, the tip (front) side of the eyeball portion 72 or the eye camera 50 is displaced, and the camera axis, that is, the line of sight moves. Is done.

  In the α axis and the β axis that control the displacement of the eye camera 50, “0 degree” is the home position. At this home position, the camera axis of the eye camera 50 is the head 68 as shown in FIG. The direction of the front side (face) is directed, and the line of sight is in the normal viewing state.

  Further, an infrared camera 74 is fixedly provided at the upper center of the head 68 with its camera axis facing the front of the head 68. Therefore, the camera axis of the infrared camera 74 is displaced with the displacement of the head 68. The infrared camera 74 is an image sensor with an infrared filter, for example, and is provided to detect infrared rays reflected by the eyes of the person 12 wearing the line-of-sight measurement device 14. Note that the infrared camera 74 may be provided in another location of the head 68 or in another part such as the upper body 40.

  Referring to FIG. 3, the robot 10 includes a microcomputer or CPU 76 for overall control, and this CPU 76 is connected to a memory 80, a motor control board 82, a sensor input / output board 84, and a voice input through a bus 78. / Output board 86 is connected.

  Although not shown, the memory 80 includes a ROM, an HDD, a RAM, and the like, and a program and data for controlling the operation of the robot 10 are stored in the ROM or the HDD in advance. The CPU 76 executes processing according to this program. The RAM is used as a buffer memory or a working memory.

  The motor control board 82 is composed of, for example, a DSP (Digital Signal Processor) and controls a motor for driving body parts such as the right arm, the left arm, the head, and the eyes. That is, the motor control board 82 receives the control data from the CPU 76 and controls the angles of the three motors for controlling the X, Y and Z axes of the right shoulder joint 54R and the axis W of the right elbow 58R. The rotation angles of a total of four motors including one motor (collectively shown as “right arm motor” in FIG. 3) 88 are adjusted. The motor control board 82 includes a total of four motors including three motors for controlling the angles of the A, B, and C axes of the left shoulder joint 54L and one motor for controlling the angle of the D axis of the left elbow joint 58L. The rotation angle of the motor (collectively shown as “left arm motor” in FIG. 3) 90 is adjusted. The motor control board 82 also adjusts the rotation angle of three motors (collectively shown as “head motors” in FIG. 3) 92 that control the angles of the S, T, and U axes of the neck joint 66. . The motor control board 82 also controls two motors 30 (shown collectively as “wheel motors” in FIG. 3) 30 that drive the waist motor 44 and the wheels 28. Further, the motor control board 82 adjusts the rotation angle of two motors 94 (collectively shown as “right eyeball motor” in FIG. 3) that control the angles of the α axis and β axis of the right eyeball portion 72R. In addition, the rotation angle of two motors 96 that collectively control the angles of the α axis and β axis of the left eyeball portion 72L (collectively shown as “left eyeball motor” in FIG. 3) 96 is adjusted.

  The above-described motors of this embodiment are stepping motors or pulse motors for simplifying the control except for the wheel motors 30, but may be direct current motors as with the wheel motors 30. In this embodiment, a motor using electric power as a drive source is used as an actuator for driving a body part such as an arm, a head, and an eye of the robot 10. However, the robot 10 may be a robot that drives a body part by other actuators such as air pressure (or negative pressure), oil pressure, a piezoelectric element, or a shape memory alloy.

  Similarly, the sensor input / output board 84 is configured by a DSP, and takes in signals from each sensor and gives them to the CPU 76. That is, data relating to the reflection time from each of the ultrasonic distance sensors 36 is input to the CPU 76 through the sensor input / output board 84. The video signal from the omnidirectional camera 46 is input to the CPU 76 after predetermined processing is performed on the sensor input / output board 84 as necessary. Similarly, the video signal from the eye camera 50 is also supplied to the CPU 76. The video signal from the infrared camera 74 is also supplied to the CPU 76 in the same manner. Further, signals from the touch sensor 62 and the collision sensor 32 are given to the CPU 76 via the sensor input / output board 84.

  Audio data is given to the speaker 70 from the CPU 76 via the audio input / output board 86, and accordingly, audio or voice according to the data is output from the speaker 70. Also, the voice input from the microphone 52 is taken into the CPU 76 as voice data via the voice input / output board 86.

  Further, a communication LAN board 98 is connected to the CPU 76 via the bus 78. The communication LAN board 98 is similarly configured by a DSP and is connected to the wireless communication apparatus 100. In this embodiment, the CPU 76 of the robot 10 is communicatively connected to a network such as a wireless LAN using the communication LAN board 98 and the wireless communication device 100. The communication LAN board 98 gives the transmission data given from the CPU 76 to the wireless communication apparatus 100 and transmits the data from the wireless communication apparatus 100. The communication LAN board 98 receives data via the wireless communication device 100 and provides the received data to the CPU 76.

  Furthermore, the above-described line-of-sight measurement device 14 is connected to the CPU 76 wirelessly via the communication LAN board 98, the wireless communication device 100, and the network. The line-of-sight measuring device 14 includes the computer 24 as described above, and the computer 24 acquires the video signal of the visual field camera 16 and the video signal of the infrared camera 20 and controls the light emission of the infrared light emitting device 18. Further, the computer 24 detects the position of the gazing point of the human 12 based on the visual field image captured by the visual field camera 16 and the infrared image captured by the infrared camera 20. Further, the computer 24 transmits necessary information such as the detected information regarding the gazing point of the human 12 and the visual field image captured by the visual field camera 16 to the CPU 76.

  In such a system 200, the gaze measurement device 14 detects the gaze point of the person 12 who is the communication partner, and controls the gaze of the robot 10 according to the position of the gaze point of the person 12. For example, a follow-up mode, an escape mode, and a match mode are prepared.

  In the follow-up mode, the line of sight of the robot 10 is controlled so as to follow the position and line of sight of the human 12. Specifically, first, a gaze point and a visual field image of the person 12 are acquired from the line-of-sight measurement device 14, and it is determined whether or not the person 12 is looking at the robot 10. When the robot 10 is being watched by the human 12, the position of the eyes of the human 12 is detected based on the image from the infrared camera 74, and the head 68 and / or so as to see the position of the eyes of the human 12. Alternatively, the line of sight of the robot 10 is controlled by displacing the eyeball unit 72. In this way, when the human 12 is looking at himself / herself, the robot 10 turns his gaze toward the eyes of the human 12 and looks back at the human 12.

  On the other hand, when the robot 10 is not watched by the human 12, the position of the human 12 is detected, and the head 68 and / or the eyeball unit 72 is displaced so as to look in the same direction as the human 12. Thus, the line of sight of the robot 10 is controlled. In this way, when the human 12 is not looking at himself, the robot 10 follows the line of sight of the human 12 and sees what the human 12 is viewing together with the human 12.

  Thereafter, for example, when a certain time has elapsed or when a change in the gazing point of the human 12 is detected, the line of sight of the robot 10 is controlled again depending on whether or not the human 12 is looking at the robot 10. .

  Thus, in the follow-up mode, the line of sight of the robot 10 is moved so as to stare back at the eyes of the human 12 or follow the line of sight of the human 12 according to the movement of the line of sight of the human 12. According to this system 200, it is possible to make the human 12 feel strongly that the robot 10 is looking at the human 12 and watching the movement of the eyes of the human 12. Therefore, an intellectual impression can be given to the person 12 and the place where the person is looking can be clearly communicated, so that more natural dialogue and communication with the person 12 can be performed.

  FIG. 4 shows an example of the operation of the robot 76 in the follow mode of the CPU 76. In step S1, the gazing point of the person 12 is detected. Specifically, the gaze point of the person 12 detected by the line-of-sight measurement device 14 is acquired. In addition, necessary information such as a visual field image is also acquired.

  Next, in step S <b> 3, it is determined whether or not the human 12 is looking at the robot 10. In this embodiment, it is determined whether or not the face of the robot 10 is being viewed. For example, when the position of the face of the robot 10 in the visual field image is detected based on the face image of the robot 10 stored in advance, and the coordinates of the gazing point are included in the face position (region) of the robot 10. Is determined that the human 12 is gazing at the robot 10, and if the coordinates of the gazing point are not included in the position (region) of the face of the robot 10, the human 12 is not gazing at the robot 10. It is judged.

  If “YES” in the step S3, that is, if the robot 10 is seen by the human 12, the eye position of the human 12 is detected in a step S5. For example, the position of the eye of the human 12 is calculated by detecting a bright spot (corneal reflection image) in an infrared image acquired from the infrared camera 74.

  In a succeeding step S7, the line of sight is controlled so as to see the position of the eyes of the human 12. As a result, the robot 10 stares back at the person 12. For example, the line of sight is moved only by the movement of the eyeball 72. Specifically, control data related to the α-axis and β-axis of the eyeball 72 such that the eyeball 72, which is the eye of the robot 10, sees the position of the eye of the human 12 is calculated. Then, the control data is given to the motor control board 82, and the rotations of the right eye motor 94 and the left eye motor 96 are controlled. Alternatively, the line of sight may be moved only by the movement of the head 68. In this case, control data related to the S-axis, T-axis, and U-axis of the head 68 (neck joint 66) such that the eyeball unit 72 sees the position of the eye of the human 12 is calculated. The rotation is controlled. Alternatively, the line of sight may be moved by movement of both the eyeball unit 72 and the head 68. In this case, control data related to the α and β axes of the eyeball 72 and the S, T, and U axes of the head 68 such that the eyeball 72 sees the position of the eyes of the human 12 is calculated. The rotations of the right eyeball motor 94, the left eyeball motor 96, and the head motor 92 are controlled. When step S7 ends, the process proceeds to step S13.

  On the other hand, if “NO” in the step S3, that is, if the robot 10 is not seen by the human 12, the position where the human 12 is looking is detected in a step S9. Specifically, three-dimensional position measurement of the gazing point of the human 12 is performed. For example, the three-dimensional position of the gazing point with respect to the position of the head of the human 12 is measured by the plurality of field cameras 16. Further, the position and posture of the head of the human 12 are measured by the eye cameras 50R and 50L of the robot 10 or a camera (not shown) installed in the environment. Then, the three-dimensional position of the gazing point with respect to the eye camera 50 of the robot 10 or the three-dimensional position of the gazing point with respect to the environment is calculated. Alternatively, object recognition is performed. For example, an object is recognized using an image around the gazing point of the human 12 in the visual field image of the visual field camera 16. Further, based on the image of the eye camera 50 of the robot 10 or a camera installed in the environment, an object that matches the object viewed by the human 12 is searched, and the position of the object is calculated.

  In the subsequent step S11, the line of sight is controlled so as to look in the same direction as the person 12. As a result, the line of sight of the robot 10 follows the line of sight of the human 12 and sees what the human 12 is looking at. The line of sight of the robot 10 is moved by moving the eyeball 72 and / or the head 68 in the same manner as in step S7 described above. Control data that allows the eyeball unit 72 to see the position of the human 12 detected in step S9 is calculated, and the rotation of the corresponding motors 92, 94, and 96 is controlled based on the control data. When step S11 ends, the process proceeds to step S13.

  In step S13, a change in the point of gaze or the elapse of a certain time is waited. For example, information on a gazing point is acquired from the line-of-sight measurement device 14, and it is determined whether or not the gazing point is different from the previously acquired gazing point. When there is no change in the gazing point, for example, the determination of the change in the gazing point is repeated until a predetermined time elapses. On the other hand, when a change in the gazing point is detected or when a certain time has elapsed, the process proceeds to step S15.

  In step S15, it is determined whether or not to end the follow-up mode. For example, it is determined whether a condition for ending the follow-up mode has been satisfied, for example, when an end instruction from the operator is received wirelessly or when the person 12 is gone. If “NO” in the step S15, the process returns to the step S1, and the operation in the follow-up mode is continued. If “YES”, the follow-up mode is ended.

  In the escape mode, the line of sight of the robot 10 is controlled so that the line of sight does not match the human 12. Specifically, first, a gaze point and a visual field image of the person 12 are acquired from the line-of-sight measurement device 14, and it is determined whether or not the person 12 is looking at the robot 10. When the robot 10 is viewed by a human 12, the position of the eyes of the human 12 is detected, and the head 68 and / or the eyeball unit 72 is displaced so as to see other than the positions of the eyes of the human 12. Thus, the line of sight of the robot 10 is controlled. In this way, when the human 12 is looking at himself, the robot 10 turns his eyes to a position other than the position of the eyes of the human 12 and turns away from the positions of the eyes of the human 12.

  On the other hand, when the robot 10 is not seen by the human 12, it is determined whether or not the robot 10 is watching the human 12. When the robot 10 is not looking at the human 12, the line of sight of the robot 10 is controlled by displacing the head 68 and / or the eyeball unit 72 so as to see the detected eye position of the human 12. In this way, the robot 10 looks to the position of the eyes of the person 12 when the person 12 is not looking at the person and is not looking at the person 12. That is, the robot 10 steals the eyes of the human 12.

  Further, when the robot 10 is not seen by the human 12 and when the robot 10 is looking at the human 12, as in the case where the robot 10 is seen by the human 12, it is assumed that other than the eye position of the human 12 is seen. The line of sight is controlled. As described above, the robot 10 looks away from the position of the eyes of the person 12 even when the person 12 is not looking at the person 12. Therefore, in this embodiment, the robot 10 turns away from the position of the eyes of the human 12 after snooping the eyes of the human 12.

  Thereafter, for example, when a certain time elapses or a change in the gazing point of the human 12 is detected, whether or not the human 12 is watching the robot 10 again, or whether or not the robot 10 is watching the human 12. Accordingly, the line of sight of the robot 10 is controlled.

  In this way, in the escape mode, the line of sight of the robot 10 is such that the eyes of the human 12 are looked away or the eyes of the human 12 are looked at in accordance with the movement of the line of sight of the human 12 or the current line of sight of the robot 10. Is moved. According to this system 200, it is possible to make the human 12 feel strongly that the robot 10 is seeing the movement of the eyes of the human 12 and that he / she is not looking at the eyes of the human 12 (not to see). . Therefore, it is possible to give an intellectual impression to the human 12 and clearly communicate where he / she is looking or not, so that more natural dialogue and communication with the human 12 can be performed. It becomes possible.

  FIG. 5 shows an example of the operation of the CPU 76 of the robot 10 in the escape mode. In step S31, the gaze point of the person 12 is detected in the same manner as in step S1 described above.

  Next, in step S33, it is determined whether or not the human 12 is looking at the robot 10 in the same manner as in step S3 described above. If “YES” in the step S33, that is, if the robot 10 is seen by the human 12, the process proceeds to a step S37.

  On the other hand, if “NO” in the step S33, that is, if the robot 10 is not seen by the human 12, it is determined whether or not the robot 10 is watching the human 12 in a step S35. For example, it is determined whether or not the robot 10 is looking at the face of the human 12. Specifically, in the image acquired by the eye camera 50, it is determined whether or not an image indicating the face of the human 12 is included in an area corresponding to the front of the eyeball unit 72. Alternatively, in the image of the infrared camera 74, it may be determined whether or not a bright spot due to infrared irradiation exists in a region corresponding to the front of the eyeball unit 72. If “YES” in the step S35, that is, if the robot 10 is looking at the person 12 who is not looking at the user, the process proceeds to a step S37.

  In step S37, the eye position of the human 12 is detected in the same manner as in step S5 described above. In a succeeding step S39, the line of sight is controlled so as to look outside the position of the human eye 12. As a result, the robot 10 turns away from the position of the eyes of the human 12. In step S39, the line of sight is moved by moving the eyeball 72 and / or the head 68, as in the case of the following mode. However, control data is calculated such that the eyeball unit 72 sees a position other than the position of the eyes of the human 12. For example, a position away from the human eye 12 by a predetermined distance or more may be selected at random. When step S39 ends, the process proceeds to step S45.

  On the other hand, if “NO” in the step S35, that is, if neither the robot 10 nor the human 12 is looking at the opponent, the eye position of the human 12 is detected in the same manner as the above-described step S1 in a step S41. Is done. In the subsequent step S43, the line of sight is controlled so as to see the position of the eyes of the human 12 in the same manner as in step S7 described above. As a result, the robot 10 steals the eyes of the person 12 who has not seen him. When step S43 ends, the process proceeds to step S45.

  In step S45, in the same manner as in step S13 described above, the change of the gazing point or the elapse of a certain time is waited. When the gazing point changes or when the certain time elapses, the process proceeds to step S47. In step S47, similarly to step S15, it is determined whether or not to end this escape mode. If “NO” in the step S47, the process returns to the step S31 to continue the escape mode operation, and if “YES”, the escape mode is ended.

  In the staring mode, the line of sight of the robot 10 is controlled so as to stare at the human 12. In this embodiment, the mutual gaze time is measured, and the line of sight is controlled by a different method depending on the value of the mutual gaze time. The mutual gaze time indicates a time when the robot 10 and the human 12 are staring at each other. In this embodiment, when it is determined that the human 12 is looking at the robot 10 and the robot 10 is looking at the human 12, it is determined that they are staring at each other, and the mutual gaze time is a unit time. When it is determined that the two are not staring, the mutual gaze time is decreased by unit time. That is, as the mutual gaze time, a time when both of them are continuously staring at each other is measured.

  The value of the mutual gaze time is evaluated based on a predetermined value that varies depending on whether or not the robot 10 is looking at the person 12. In this embodiment, the first threshold is used when the robot 10 is looking at the person 12, and the second threshold is used when the robot 10 is not looking at the person 12. The first threshold is set to a value larger than the second threshold. The first threshold indicates a limit at which the robot 10 does not want to stare at the human 12, and the second threshold indicates a limit at which the robot 10 does not want to stare at the human 12. For example, both values are appropriately set such that the first threshold is 3 seconds and the second threshold is 2 seconds.

  Specifically, when the robot 10 is not looking at the person 12 and the mutual gaze time is smaller than the second threshold, the line of sight of the robot 10 is controlled so that the person 12 and the robot 10 can stare at each other. In this embodiment, the line of sight is controlled in the follow-up mode described above, and the robot 10 looks back at the person 12 when viewed by the person 12 and follows the line of sight of the person 12 when not seen by the person 12.

  Thereafter, the mutual gaze time is increased while the robot 10 stares back at the eyes of the human 12. When the robot 10 is looking at the human 12 and the mutual gaze time exceeds the first threshold, the line of sight of the robot 10 is controlled so as not to match the line of sight of the human 12. In this embodiment, the line of sight is controlled in the above-described escape mode, and the robot 10 looks away from the position of the eyes of the human 12. Since the determination of the first threshold is performed when the robot 10 is looking at the human 12, the snooping is not executed in the escape mode in this case.

  Alternatively, when the human 12 turns away from the robot 10 and follows the line of sight of the human 12 before the mutual gaze time exceeds the first threshold, the robot gazes when the gaze time exceeds the second threshold. The line of sight of 10 is controlled so as not to match the line of sight of human 12. In this embodiment, the line of sight is controlled in the above-described escape mode, and the robot 10 turns away from the position of the eyes of the human 12 when viewed by the human 12 and the human 12 when not viewed by the human 12. To steal your eyes.

  In the meantime, when the mutual gaze time falls below the second threshold, the line of sight of the robot 10 is controlled so as to stare at the human 12 again as described above.

  As described above, in the staring mode, the robot 10 controls the line of sight so as to stare at the human 12 when the continuous staring time has not reached the first threshold value. When the continuous staring time exceeds the first threshold, the robot 10 looks away from the human 12 and stops staring. Thereafter, when a predetermined time (first threshold value−second threshold value) has elapsed after stopping staring, and the mutual gaze time becomes smaller than the second threshold value, the robot 10 again stares at the human 12. Control your gaze. According to the system 200, it is possible to make the person 12 feel strongly that the robot 10 is looking at the eyes of the person 12 or not looking at the eyes of the person 12. In addition, it is possible to match eyes with the person 12 for an appropriate time (first threshold), and then look away from the person 12 for an appropriate time (first threshold−second threshold). Therefore, it is possible to give an intellectual impression to the human 12 and clearly communicate where he / she is looking or not, so that more natural dialogue and communication with the human 12 can be performed. It becomes possible.

  FIG. 6 shows an example of the operation of the CPU 76 of the robot 10 in the check mode. In step S61, the gaze point of the person 12 is detected in the same manner as in step S1 described above.

  Next, in step S63, it is determined whether or not the human 12 is looking at the robot 10 and whether or not the robot 10 is looking at the human 12, that is, whether or not both are staring. Whether or not the human 12 is looking at the robot 10 is determined in the same manner as in step S3 described above, and whether or not the robot 10 is looking at the human 12 is determined in the same manner as in step S35 described above.

  If “YES” in the step S63, that is, if both of them are staring, the mutual gaze time is increased by a unit time in a step S65. On the other hand, if “NO” in the step S63, that is, if both are not staring, the mutual gaze time is decreased by a unit time in a step S67. Note that the initial value of the mutual gaze time is 0, and the minimum value is also 0. If the mutual gaze time becomes smaller than 0 by subtraction, it is reset.

  When step S65 or S67 is completed, it is determined in step S69 whether or not the robot 10 is looking at the person 12. If the robot 10 is looking at the person 12, the mutual gaze time is determined based on the first threshold value in step S71. On the other hand, if the robot 10 is not looking at the person 12, the step is step S73. The determination of the mutual gaze time is executed based on the two threshold values. Then, the line of sight is controlled according to the determination result of the mutual gaze time.

  That is, if “YES” in the step S69, it is determined whether or not the mutual gaze time is less than a first threshold value in a step S71. If "YES" in the step S71, that is, if the mutual gaze time has not reached the limit at which it is difficult to stare, the line of sight is controlled in the same manner as in the follow-up mode in the step S75. Specifically, the processes in steps S3 to S11 in the follow-up mode in FIG. 4 are executed, and the robot 10 looks back at the eyes of the human 12 when the human 12 is looking at the robot 10, and the human 12 is looking at the robot 10. If not, the person 12 is followed. On the other hand, if “NO” in the step S71, that is, if the mutual gaze time has reached a limit at which it is not necessary to check, the line of sight is controlled in the same manner as in the escape mode described above in a step S77. Specifically, the processing of steps S33 to S43 in FIG. 5 is executed, and the robot 10 looks away from the position of the human eye 12.

  If “NO” in the step S69, it is determined whether or not the mutual gaze time is less than a second threshold value in a step S73. If “YES” in the step S73, that is, if the mutual gaze time exceeds a lower limit at which it becomes difficult to check, the line of sight is controlled in the same manner as the follow-up mode in the step S75. Thereby, the robot 10 stares back at the eyes of the human 12 when viewed by the human 12, and follows the line of sight of the human 12 when not viewed by the human 12. On the other hand, if “NO” in the step S73, that is, if the mutual gaze time is equal to or more than the second threshold value, the line of sight is controlled in the same manner as in the above escape mode in the step S77. As a result, the robot 10 looks away from the eyes of the human 12 when viewed by the human 12, and steals the eyes of the human 12 when not viewed by the human 12.

  When step S75 or S77 is completed, in step S79, the change of the gazing point or the elapse of a fixed time is waited in the same manner as in step S13 described above, and when the gazing point changes or when the fixed time elapses, The process proceeds to step S81. In step S81, as in step S15, it is determined whether or not to end the check mode. If “NO” in the step S81, the process returns to the step S61 to continue the operation of the check mode, and if “YES”, the check mode is ended.

  In the check mode of the above-described embodiment, the mutual gaze time is determined using a different threshold depending on whether or not the robot 10 is looking at the person 12. However, in another embodiment, when the mutual gaze time is less than a predetermined threshold, the line of sight is controlled in, for example, the follow-up mode so as to stare at the human 12, and when the mutual gaze time is equal to or greater than the predetermined threshold, For example, the line of sight may be controlled in the escape mode so that the eyes of the human 12 are not matched.

  In each of the above embodiments, the computer 24 of the line-of-sight measuring device 14 detects the gaze or gazing point of the human 12 and transmits the detection information to the robot 10. In other embodiments, The computer 24 may transmit the field-of-view image captured by the field-of-view camera 16 and the infrared image captured by the infrared camera 20 to the robot 10 so that the CPU 76 of the robot 10 detects the line of sight or gazing point of the human 12.

  Further, in each of the above-described embodiments, the eyeball portion 72 is provided so as to be displaceable, but in another embodiment, the eyeball portion 72 may be fixedly provided on the head 68. In that case, the direction of the eyeball unit 72, that is, the line of sight is moved by displacing the head 68.

It is an illustration figure which shows the outline of the communication robot system of one Example of this invention. It is an illustration figure which shows an example of the external appearance of the communication robot of FIG. 1 Example. It is a block diagram which shows an example of an electrical configuration of the communication robot system of FIG. 1 Example. It is a flowchart which shows an example of operation | movement in the follow-up mode of a communication robot. It is a flowchart which shows an example of operation | movement in escape mode of a communication robot. It is a flowchart which shows an example of operation | movement in the check-up mode of a communication robot.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Communication robot 14 ... Eye-gaze measuring device 16 ... Field-of-view camera 18 ... Infrared light-emitting device 20, 74 ... Infrared camera 24 ... Computer 66 ... Neck joint 68 ... Head 72 ... Eyeball part 76 ... CPU
DESCRIPTION OF SYMBOLS 80 ... Memory 82 ... Motor control board 84 ... Sensor input / output board 92 ... Head motor 94 ... Right eyeball motor 96 ... Left eyeball motor 200 ... Communication robot system

Claims (7)

A communication robot system including a robot that performs communication behavior with a person,
An infrared light emitting device that is attached to the person and emits infrared rays toward the human eye;
A first infrared camera mounted on the person and taking an infrared image of the person's eye;
A field-of-view camera mounted on the person and capturing a field-of-view image of the person;
Gazing point detection means for detecting the gazing point of the person based on the infrared image and the visual field image;
Wherein mounted on the robot, a second infrared camera for photographing an infrared image,
Eye position detection means for detecting the position of the person's eye by detecting infrared light reflected on the person's eye by the second infrared camera, and the person detected by the gaze point detection means A communication robot system comprising gaze control means for moving the gaze according to the eye point of the person and the position of the human eye detected by the eye position detection means.   The line-of-sight control means directs the line of sight to the eye position of the person detected by the eye position detection means when it is determined that the person is looking based on a detection result by the gazing point detection means. The communication robot system according to claim 1.   The line-of-sight control means determines the line of sight in addition to the position of the human eye detected by the eye position detection means when it is determined that the person is looking based on the detection result by the gazing point detection means. The communication robot system according to claim 1, which is directed.   The communication robot system according to claim 3, wherein the line-of-sight control unit directs the line of sight to a position other than the position of the human eye by selectively setting a position that is a predetermined distance or more away from the position of the human eye.   The line-of-sight control means directs the line of sight to the eye position of the person detected by the eye position detection means when it is determined that the person is not seen based on the detection result by the gazing point detection means. The communication robot system according to claim 1.   The communication according to claim 1, wherein the gaze control unit directs the gaze in the same direction as the human gaze point when it is determined that the gaze control unit is not seen by the person based on a detection result by the gaze point detection unit. Robot system. An infrared light emitting device that includes a robot that performs communication behavior with a person and that is attached to the person and emits infrared rays toward the eyes of the person; and a first that is attached to the person and captures an infrared image of the eyes of the person An infrared camera, a field-of-view camera attached to the person and capturing a field-of-view image of the person, and a point-of-gaze detection means attached to the robot and detecting the point of interest of the person based on the infrared image and the field-of-view image When the mounted on the robot, and a second infrared camera for photographing an infrared image, the mounted on the robot, the eyes by detecting the infrared ray reflected into the eye of the person by the second infrared camera An eye position detecting means for detecting a position; a position of the eye of the person detected by the eye position detecting means; Flip and a sight line control means for moving the line of sight, a said robot eye control method in the communication robot system,
A first determination step of determining whether the robot is watching a person;
A second determination step of determining whether or not the mutual gaze time when the robot is watching the person and the robot is watching the person is less than a first threshold when the robot determines that the robot is watching the person in the first determination step;
A third determination step of determining whether or not the mutual gaze time is less than a second threshold when it is not determined in the first determination step that the robot is watching a person;
When it is determined in the second determination step that the mutual gaze time is less than the first threshold, or when it is determined in the third determination step that the mutual gaze time is less than the second threshold, the person is watching the robot. A first line-of-sight control step for controlling the line of sight to look at the position of the human eye,
When it is determined in the second determination step that the mutual gaze time is less than the first threshold, or when it is determined in the third determination step that the mutual gaze time is less than the second threshold, the person is not looking at the robot. A second line-of-sight control step for controlling the line of sight to look in the same direction as a person,
When the second determination step does not determine that the mutual gaze time is less than the first threshold, or when the third determination step does not determine that the mutual gaze time is less than the second threshold, A third line-of-sight control step for controlling the line of sight so as to look outside the position of the human eye, and when the mutual gaze time is not determined to be less than the first threshold in the second determination step, Alternatively, in the third determination step, when it is determined that the mutual gaze time is not less than the second threshold, and when the person is not looking at the robot, the fourth line-of-sight control is performed to control the line of sight so as to look at the position of the human eye. A communication robot gaze control method including steps.

Images