JP2006088276A - Motion generation system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a motion generation system, especially the motion generation system, causing a communication robot to perform motion (cooperative motion or imitating motion) corresponding to the motion of the other party of communication such as human being. <P>SOLUTION: This motion generation system 10 includes: a communication robot 12 making gestures, and a robot control device 16 detects the state of motion of the human being 14 according to the body motion data of the human being 14 and the communication robot 12 measured by a motion capture system 18 and the speech measurement data obtained from a microphone 24. A behavior module corresponding to the detected state is selected from a behavior module DB20 storing a plurality of behavior modules for realizing the motion (the imitation motion and the sympathizing motion) cooperative with the motion of the human being 14. The control data for realizing the cooperative motion is generated by performing the behavior module, and in the communication robot 12, the cooperative motion is realized according to the control data. According to the invention, the communication robot is caused to perform suitable cooperative motion such as the imitation motion and the sympathizing motion corresponding to the state of the other party. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an action generation system, and more particularly to an action generation system that causes a communication robot to execute an action (such as a cooperative action or an imitation action) corresponding to the action of a communication partner such as a human.

  The present applicant has proposed a communication robot that interacts with humans in Patent Document 1 and the like.

In addition, in the previous researches of the present inventors, humans perform cooperative body movements corresponding to the movements of the communication robots, such as eye contact and synchronous movements, for example. It has been found that entrainment in interactions occurs (eg, Non-Patent Document 1).
Japanese Patent Application Laid-Open No. 2002-355783 Takayuki Kanda (T. Kanda, H. Ishiguro, M. Imai, and T. Ono): “Body Movement Analysis of Human-Robot Interaction”, Proc. of International Joint Conference on Artificial Intelligence, pp. 177-182, 2003.

  It has been found that humans behave cooperatively in response to robots for natural communication, but communication robots that can autonomously perform cooperative actions on humans have not yet been developed. Absent.

  Further, for example, when considering creating a robot that imitates a human motion as a cooperative motion using a communication robot of the background art, a problem arises if a robot that imitates a person is simply created. For example, since a robot always imitates a human being, communication with a human is no longer possible. Further, when communication is performed at a close distance, there is a risk of hitting a human if the same operation as that of a human is always performed.

  Therefore, a main object of the present invention is to provide a motion generation system capable of causing a communication robot to execute an appropriate motion corresponding to the motion of a communication partner.

  Another object of the present invention is to provide a motion generation system capable of causing a communication robot to execute appropriate cooperative motion and imitation motion according to the motion state of the opponent.

  The invention of claim 1 is a motion generation system comprising a communication robot having at least a body part including an arm and an actuator for driving the body part, and causing the communication robot to perform an action corresponding to the action of a communication partner. Program storage means for storing a plurality of behavior programs for causing the communication robot to perform actions corresponding to the actions of the opponent, action detection means for detecting actions of the opponent and the communication robot and generating data relating to the actions, Based on the state detection means for detecting the state of the opponent based on the state of the opponent detected by the state detection means, the selection means for selecting an action program from a plurality of action programs stored in the program storage means, the selection by the selection means Accordance behavioral program includes an actuator control means for controlling the actuator based generator to generate, and the control data control data for realizing the operation is an operation generating system.

  According to the first aspect of the present invention, the motion detection means detects the motion of the communication robot and the communication partner and generates data related to the motion. In the embodiment, for example, data related to body movement is acquired by a motion capture system, and data related to speech is acquired by a microphone. Based on the data relating to the detected operation, the state detecting means detects the state of the other party. The selection means selects an action program corresponding to the detected partner's state from a plurality of programs for the action corresponding to the opponent's action stored in the program storage means. The generation means generates control data of the communication robot for realizing the operation according to the selected behavior program. The actuator control means controls the actuator that drives the body part based on the generated control data. In this way, the body part including at least the arm of the communication robot is driven, and an operation corresponding to the operation of the other party is realized. Therefore, according to the first aspect of the present invention, the action program for the corresponding action is selected according to the detected state of the opponent based on the data related to the detected action. Therefore, it is possible to cause the communication robot to execute an appropriate operation according to the state of the other party.

  The invention of claim 2 is dependent on the invention of claim 1, wherein the body part of the communication robot further includes a head, and the behavior program stored in the program storage means causes the communication robot to cooperate with the opponent. Includes an action program to execute. According to the second aspect of the present invention, it is possible to cause the communication robot to execute a cooperative operation according to the condition of the other party using not only the arm but also the head. Therefore, communication between the communication robot and the other party can be made more natural.

  The invention of claim 3 is dependent on the invention of claim 1 or 2, and the action program stored in the program storage means includes an action program for causing the communication robot to perform a mimic action imitating the opponent, and detects the action. The means includes body motion detection means for detecting body motion of the opponent and the communication robot to generate body motion data, the state detection means detects the state of the partner based on at least the body motion data, and the generation means includes: When the action program for executing the imitation action is selected by the selection means, the control data is generated based on the body action data.

  In the invention of claim 3, the program storage means stores a program for executing an operation imitating the operation of the other party. The motion detection means detects body motions of the communication robot and the other party and generates body motion data. In the embodiment, for example, an optical motion capture system is applied, and the three-dimensional position coordinate data of the feature points of the communication robot and the partner's body are acquired. The state detection means detects the state of the other party based on at least the body movement data. Depending on the detected state of the other party, the selection means can select an action program for executing the imitation operation. When the action program for executing the imitation operation is selected by the selection unit, the generation unit generates control data for realizing the imitation operation based on the body operation data. Therefore, according to the invention of claim 3, it is possible to cause the communication robot to execute an imitation operation of an appropriate opponent according to the state of the opponent. Therefore, the imitation operation as a cooperative operation with the other party can be appropriately executed, so that natural and smooth communication can be realized.

  The invention of claim 4 is dependent on the invention of claim 3, wherein the program storage means stores a plurality of behavior programs for executing the imitation operation, and the state detection means is configured such that the partner performs a specific action. A first state detecting unit that detects a first state relating to a hand that is present, and a second state detecting unit that detects a second state relating to an arm of a communication robot that may collide with an opponent. When one of the opponent's hands is shown, an action program for executing an action imitating the specific action with an arm different from the arm indicated in the second state is selected.

  According to a fourth aspect of the present invention, the first state detecting means detects the first state relating to the hand in which the opponent is performing the specific motion based on the body motion data. For example, in the embodiment, the first state indicates a hand that the opponent is pointing at. The second state detection means detects a second state related to the arm of the communication robot that may collide with the opponent based on the body motion data. When the first state indicates one of the opponent's hands, that is, when the opponent is performing a specific action, the selecting means imitates a specific action by an arm different from the arm indicated by the second state. The action program for is selected from the program storage means. Therefore, according to the invention of claim 4, when there is a possibility of colliding with the opponent if the same action as the opponent's specific action is performed, it is possible to execute the imitation operation using the arm that is not likely to collide. Therefore, it is possible to safely execute an operation that imitates the operation of the other party.

  The invention according to claim 5 is dependent on the invention according to claim 4, wherein the state detection means further includes third state detection means for detecting a third state relating to the direction of the specific action, and the selection means includes any one of the second states. If no arm is shown, an action program for causing the arm on the side corresponding to the direction indicated by the third state to perform an action imitating the specific action is selected.

  In the invention of claim 5, the third state detecting means detects the third state relating to the direction of the specific operation. For example, in the embodiment, the third state indicates the direction in which the specific operation is performed as viewed from the communication robot. When the second state does not indicate any arm, that is, when there is no possibility that any arm collides with the opponent, the selecting means performs a specific action by the arm on the side corresponding to the direction indicated by the third state. Select an action program for imitation action. Therefore, according to the invention of claim 5, when the safety is ensured, the imitation operation can be executed with the arm corresponding to the direction indicated by the opponent's operation, so that a more natural imitation operation is realized. it can.

  The invention of claim 6 is dependent on the invention of claim 4, and the state detecting means determines the fourth state relating to the hand used by the opponent for the specific action, the distance from the shoulder of the opponent's hand, the opponent's hand. Detection is performed based on the movement of a predetermined time in the past and the detected fourth state, and the selection means executes an operation that is not an imitation operation when the hand indicated by the first state is not the same as the hand indicated by the fourth state To choose an action program.

  In the invention of claim 6, the state detecting means detects the hand used by the opponent for the specific action as the fourth state. The detection of the fourth state is performed based on the distance from the shoulder of the opponent's hand, the movement of the opponent's hand in the past fixed time, and the fourth state detected last time. For example, in the embodiment, only when both the distance and the movement concerning the hand not shown in the previous fourth state are larger than both the distance and the movement concerning the hand shown in the previous fourth state, Assuming that the hand being used has been changed, the hand indicated in the fourth state is switched. And the selection means becomes different when the hand in the first state and the hand in the fourth state are not the same, that is, the hand performing the specific action and the hand used for the specific action are different. Sometimes, it is considered that the specific action is finished, and an action program for executing an action that is not an imitation action is selected. Therefore, according to the invention of claim 6, since imitation operation is not continued even if the other party's operation is changed in an appropriate manner when the specific operation changes, imitation without a sense of incongruity. Operation can be realized.

  The invention of claim 7 is dependent on any of the inventions of claims 3 to 6, and the selection means detects the state when the selection of the action program for executing the action imitating the specific action continues for a certain time or more. The action program is reselected according to the state of the opponent detected by the means.

  According to the seventh aspect of the present invention, when the selection of the action program for the specific action continues for a certain time or more, the action program is reselected according to the state of the opponent. Therefore, according to the invention of claim 7, the communication robot does not continue the imitation operation for a long time, and the time for realizing the imitation operation can be made appropriate. Can be done naturally.

  The invention of claim 8 is dependent on any of the inventions of claims 3 to 7, and the specific action includes a pointing action or a body action indicating a direction. In the invention of claim 8, the state detection means detects the state of the other party with respect to the pointing operation as the specific operation or the body operation indicating the direction. Therefore, for example, when a pointing operation such as the route guidance situation in the embodiment is performed by a human, the communication robot can appropriately perform the imitation operation of the opponent, and smooth communication can be realized.

  The invention according to claim 9 is dependent on the invention according to claim 2, wherein the motion detection means includes speech detection means for detecting speech of the other party and generating data related to speech, and the state detection means is based on the data related to speech. An action program for causing the partner to perform cooperative actions using the head when the partner's utterance state detected by the state detector satisfies a predetermined condition. Select.

  According to another aspect of the present invention, the utterance detection means detects the other party's utterance and generates data related to the utterance, and the state detection means detects the other party's utterance state. The selection means uses the head to select an action program for causing the partner to perform a cooperative action when the partner's utterance state satisfies a predetermined condition. Therefore, the communication robot can realize a cooperative operation according to the utterance state of the other party by a physical operation using the head.

  The invention of claim 10 is dependent on any one of the inventions of claims 1 to 9, wherein the communication robot further comprises voice output means for outputting voice, and the behavior program stored in the program storage means Action detecting means includes utterance detecting means for detecting utterance of the other party and generating data relating to the utterance, and the state detecting means detects the state of the other party's utterance based on the data relating to the utterance. The selecting means selects an action program for each body part and utterance of the communication robot, and selects an action program for uttering at least according to the state of the other party's utterance detected by the state detecting means. When the action program for speaking is selected by the selecting means, the generating means selects the utterance. Generates control data for the speech output means outputs the sound based on the control data.

  In the invention of claim 10, the communication robot is provided with a voice output means. The program storage means stores an action program for speaking. The utterance detection means detects the utterance of the other party and generates data related to the utterance, and the state detection means detects the state of the utterance of the other party based on the data concerning the utterance. Since the selection means can select an action program for each body part and utterance of the communication robot, the action program for uttering is selected according to at least the detected utterance state. When an action program for uttering is selected, the generation means generates control data for uttering the utterance. The voice output means outputs voice based on the control data for speaking. Therefore, the communication robot can make an appropriate utterance according to the detected utterance state of the other party, so that the movement corresponding to the other party's movement using the utterance together with the body movement, cooperative movement, imitation movement, etc. Can be executed, and smoother communication can be realized.

  According to the present invention, based on the data related to the communication partner and the operation of the communication robot, the partner's state is detected, and in accordance with the detected state, the cooperative operation as the operation corresponding to the partner's operation or The action program for imitation operation is selected. Therefore, since the communication robot can execute appropriate cooperative operation or imitation operation according to the state of the other party's operation, natural and smooth communication can be realized.

  The above object, other objects, features and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

  With reference to FIG. 1, the motion generation system 10 of this embodiment executes a cooperative motion on a communication robot (hereinafter also simply referred to as “robot”) 12 according to the motion (behavior) of a human 14. In particular, as a cooperative operation, an operation that imitates (i.e., is similar to or the same as) the body motion of the human 14 or an operation that synchronizes with the motion of the human 14 is executed. The motion generation system 10 includes a robot 12 having a body part including at least an arm and capable of expressing the body, a robot control device 16 for controlling a behavior to be executed by the robot 12, and a motion capture system 18 for detecting the body motion. Including.

  The robot control device 16 includes a computer such as a personal computer or a workstation, and includes a CPU, RAM, ROM, HDD, and the like. A program and data for controlling the behavior of the robot 12 are stored in an HDD or the like as a main storage device, and the CPU executes processing according to this program. A behavior module database (DB) 20 and an utterance measurement data DB 22 are provided inside or outside the robot control device 16. As will be described later, the behavior module DB 20 stores each modularized program (called a behavior module) for realizing each behavior of the robot 12. In the utterance measurement data DB 22, data related to the utterance of the human 14 acquired by the microphone 24 is stored. The microphone 24 is provided in the robot 12 in this embodiment. The robot 12 measures the volume of the utterance of the human 14 from the microphone 24 at a predetermined cycle, and transmits the acquired utterance measurement data to the robot control device 16. The robot control device 16 receives the utterance measurement data from the robot 12 and stores it in the utterance measurement data DB 22. The utterance measurement data is stored in association with the measurement time (frame).

  The robot control device 16 and the motion capture system 18 are connected to each other by a wired or wireless LAN via, for example, a HUB (not shown). Also, the robot 12 is connected to the robot control device 16 via a wireless LAN via an access point (not shown) connected to the HUB, for example.

  The CPU of the robot control device 16 selects and executes an appropriate behavior module based on the detected motion (body motion and speech) of the human 14. Of the data related to the motion of the human 14, data related to the physical motion is acquired from the motion capture system 18. As a result of the execution of the behavior module, the CPU of the robot control device 16 transmits command data for realizing the behavior to the robot 12.

  A known motion capture system is applied as the motion capture system (three-dimensional motion measurement device) 18. For example, an optical motion capture system of VICON (http://www.vicon.com/) may be applied. The motion capture system 18 includes a computer such as a personal computer or a workstation, and a three-dimensional motion measurement data DB 26 that stores numerical data related to the body motion of the human 14 and the robot 12 is provided inside or outside the computer. In this motion capture system 18, as shown in FIG. 2, a plurality of (at least three) cameras 28 having an infrared irradiation function are arranged in an appropriate space such as an entrance, a hallway, a room, etc. Are arranged in directions different from each other, and detects a body motion in the interaction between the robot 12 and the human 14. A plurality of infrared reflection markers 30 are attached to the robot 12 and the human 14. Each marker 30 has a feature point that characterizes the movement of the human 14, such as the top of the head, the top of the eye, the neck, the shoulder, the elbow, and the hand (wrist). It is attached. For example, the human 14 may wear a hat or clothes to which the marker 30 is attached. Similarly, each marker 30 is attached to the robot 12 at feature points (the top of the head, the top of the eye, the neck, the shoulder, the elbow, the hand, etc.) that can specify the operation. The computer of the motion capture system 18 acquires image data from the camera 28 at, for example, 60 Hz (60 frames per second), and by image processing of the image data, the two-dimensional position of each marker 30 in all the image data at the time of measurement. To extract. Then, the computer calculates the three-dimensional position coordinate data of each marker 30 in the real space based on the two-dimensional position, and uses the calculated three-dimensional position coordinate data as the three-dimensional movement measurement data in the three-dimensional movement measurement data DB 26. Store. The three-dimensional motion measurement data is stored in association with the measurement time (frame).

  The robot 12 has a human-like body, and uses the body to generate complex body movements necessary for communication. Specifically, referring to FIG. 3, the robot 12 includes a carriage 32, and wheels 34 for autonomously moving the robot 12 are provided on the lower surface of the carriage 32. The wheel 34 is driven by a wheel motor (indicated by reference numeral “36” in FIG. 4 showing the internal configuration of the robot 12), and the carriage 32, that is, the robot 12 can be moved in any direction. Although not shown in FIG. 3, a collision sensor (indicated by reference numeral “38” in FIG. 4) is attached to the front surface of the carriage 32. Detect obstacle contact. When contact with an obstacle is detected during the movement of the robot 12, the driving of the wheels 34 is immediately stopped and the movement of the robot 12 is suddenly stopped.

  In this embodiment, the height of the robot 12 is about 100 cm so as not to intimidate people, particularly children. However, this height can be arbitrarily changed.

  A polygonal column sensor mounting panel 40 is provided on the carriage 32, and an ultrasonic distance sensor 42 is mounted on each surface of the sensor mounting panel 40. The ultrasonic distance sensor 42 measures the distance between the mounting panel 40, that is, the person around the robot 12 mainly.

  Further, the body of the robot 12 is mounted on the carriage 32 so that the lower portion thereof is surrounded by the mounting panel 40 described above and stands upright. The body is composed of a lower body 44 and an upper body 46, and the lower body 44 and the upper body 46 are connected by a connecting portion 48. Although not shown, the connecting portion 48 has a built-in lifting mechanism, and the height of the upper body 46, that is, the height of the robot 12 can be changed by using the lifting mechanism. As will be described later, the elevating mechanism is driven by a waist motor (indicated by reference numeral “50” in FIG. 4). The height 100 cm of the robot 12 described above is a value when the upper body 46 is at its lowest position. Therefore, the height of the robot 12 can be 100 cm or more.

  One omnidirectional camera 52 and one microphone 24 are provided in the approximate center of the upper body 46. The omnidirectional camera 52 photographs the surroundings of the robot 12 and is distinguished from an eye camera 54 described later. As described above, the microphone 24 captures ambient sounds, particularly human voices.

  Upper arms 58R and 58L are attached to both shoulders of the upper body 46 by shoulder joints 56R and 56L, respectively. The shoulder joints 56R and 56L each have three degrees of freedom. That is, the right shoulder joint 56R can control the angle of the upper arm 58R around each of the X, Y, and Z axes. The Y axis is an axis parallel to the longitudinal direction (or axis) of the upper arm 58R, and the X axis and the Z axis are axes orthogonal to the Y axis from different directions. The left shoulder joint 56L can control the angle of the upper arm 58L around each of the A, B, and C axes. The B axis is an axis parallel to the longitudinal direction (or axis) of the upper arm 58L, and the A axis and the C axis are axes orthogonal to the B axis from different directions.

  Forearms 62R and 62L are attached to the respective distal ends of upper arms 58R and 58L via elbow joints 60R and 60L. The elbow joints 60R and 60L can control the angles of the forearms 62R and 62L around the W axis and the D axis, respectively.

  In the X, Y, Z, W axes and the A, B, C, D axes that control the displacement of the upper arms 58R and 58L and the forearms 62R and 62L (FIG. 3), “0 degree” is the home position. In this home position, the upper arms 58R and 58L and the forearms 62R and 62L are directed downward.

  Although not shown in FIG. 3, touch sensors (FIG. 4) are provided on the shoulder portion of the upper body 46 including the shoulder joints 56R and 56L and the above-mentioned upper arms 58R and 58L and the arm portions including the forearms 62R and 62L, respectively. The touch sensor 64 detects whether or not a person has touched these parts of the robot 12.

  Spheres 66R and 66L corresponding to hands are fixedly attached to the tips of the forearms 62R and 62L, respectively. In place of the spheres 66R and 66L, a “hand” in the shape of a human hand can be used when a finger function is required unlike the robot 12 of this embodiment.

  A head 70 is attached to an upper center of the upper body 46 via a neck joint 68. The neck joint 68 has three degrees of freedom and can be controlled in angle around each of the S, T, and U axes. The S-axis is an axis that goes directly from the neck, and the T-axis and the U-axis are axes that are orthogonal to the S-axis in different directions. The head 70 is provided with a speaker 72 at a position corresponding to a human mouth. The speaker 72 is used for the robot 12 to communicate with a person around it by voice or voice. However, the speaker 72 may be provided in another part of the robot 12, for example, the trunk.

  The head 70 is provided with eyeball portions 74R and 74L at positions corresponding to the eyes. Eyeball portions 74R and 74L include eye cameras 54R and 54L, respectively. The right eyeball portion 74R and the left eyeball portion 74L may be collectively referred to as an eyeball portion 74, and the right eye camera 54R and the left eye camera 54L may be collectively referred to as an eye camera 54. The eye camera 54 captures the video signal by photographing the face of the person approaching the robot 12 and other parts or objects.

  Note that each of the omnidirectional camera 52 and the eye camera 54 described above may be a camera using a solid-state imaging device such as a CCD or a CMOS.

  For example, the eye camera 54 is fixed in the eyeball part 74, and the eyeball part 74 is attached to a predetermined position in the head 70 via an eyeball support part (not shown). The eyeball support unit has two degrees of freedom and can be controlled in angle around each of the α axis and the β axis. The α axis and the β axis are axes set with respect to the head 70, the α axis is an axis in a direction toward the top of the head 70, the β axis is orthogonal to the α axis and the front side of the head 70 It is an axis in a direction orthogonal to the direction in which (face) faces. In this embodiment, when the head 70 is at the home position, the α axis is set to be parallel to the S axis and the β axis is set to be parallel to the U axis. In such a head 70, when the eyeball support portion is rotated around each of the α axis and the β axis, the tip (front) side of the eyeball portion 74 or the eye camera 54 is displaced, and the camera axis, that is, the line-of-sight direction is changed. Moved.

  In the α axis and β axis that control the displacement of the eye camera 54, “0 degree” is the home position. At this home position, the camera axis of the eye camera 54 is the head 70 as shown in FIG. The direction of the front side (face) is directed, and the line of sight is in the normal viewing state.

  FIG. 4 is a block diagram showing the internal configuration of the robot 12. As shown in FIG. 4, the robot 12 includes a microcomputer or CPU 76 for overall control. The CPU 76 is connected to a memory 80, a motor control board 82, a sensor input / output board 84, and a voice through a bus 78. An input / output board 86 is connected.

  Although not shown, the memory 80 includes a ROM, an HDD, a RAM, and the like, and the control program and data for the robot 12 are stored in the ROM or the HDD in advance. The CPU 76 executes processing according to this program. The RAM can be used as a working memory as well as a temporary storage memory.

  The motor control board 82 is composed of, for example, a DSP (Digital Signal Processor) and controls a motor for driving body parts such as the right arm, the left arm, the head, and the eyes. That is, the motor control board 82 receives control data from the CPU 76, and controls the angles of the three motors for controlling the X, Y, and Z axes of the right shoulder joint 56R and the axis W of the right elbow joint 60R. The rotation angles of a total of four motors including one motor (collectively shown as “right arm motor” in FIG. 4) 88 are adjusted. The motor control board 82 includes a total of four motors including three motors that control the angles of the A, B, and C axes of the left shoulder joint 56L and one motor that controls the angle of the D axis of the left elbow joint 60L. The rotation angle of the motor (collectively shown as “left arm motor” in FIG. 4) 90 is adjusted. The motor control board 82 also adjusts the rotation angles of three motors 92 (collectively shown as “head motors” in FIG. 4) that control the angles of the S, T, and U axes of the neck joint 68. To do. The motor control board 82 also controls the waist motor 50 and the two motors 36 that drive the wheels 34 (collectively shown as “wheel motors” in FIG. 4). Further, the motor control board 82 adjusts the rotation angle of two motors 94 (collectively shown as “right eyeball motor” in FIG. 4) that control the angles of the α axis and β axis of the right eyeball portion 74R. In addition, the rotation angle of two motors 96 (collectively shown as “left eyeball motor” in FIG. 4) for controlling the angles of the α axis and the β axis of the left eyeball portion 74L is adjusted.

  The above-described motors of this embodiment are stepping motors or pulse motors for simplifying the control except for the wheel motors 36, but may be direct-current motors as with the wheel motors 36.

  Similarly, the sensor input / output board 84 is also constituted by a DSP, and takes in signals from each sensor and camera and gives them to the CPU 76. That is, data relating to the reflection time from each of the ultrasonic distance sensors 42 is input to the CPU 76 through the sensor input / output board 84. The video signal from the omnidirectional camera 52 is input to the CPU 76 after being subjected to predetermined processing by the sensor input / output board 84 as necessary. Similarly, the video signal from the eye camera 54 is also supplied to the CPU 76. Further, a signal from the touch sensor 64 is given to the CPU 76 via the sensor input / output board 84.

  Synthetic voice data is given to the speaker 72 from the CPU 76 via the voice input / output board 86, and accordingly, voice or voice according to the data is outputted from the speaker 72. Further, the voice input from the microphone 24 is taken into the CPU 76 via the voice input / output board 86.

  Similarly, the communication LAN board 98 is also configured by a DSP, and sends the transmission data given from the CPU 76 to the wireless communication apparatus 100 and causes the wireless communication apparatus 100 to transmit the transmission data. The communication LAN board 98 receives data via the wireless communication device 100 and provides the received data to the CPU 76.

  In this embodiment, for example, in a route guidance situation in which the human 14 is pointing or gesturing (gesturing) indicating a direction, a cooperative operation (imitation operation, tuning operation, etc.) is performed on the human 14. ) Is assumed to be performed by the robot 12. The situation where the cooperative operation including the imitation operation is performed is not limited to the route guidance situation and may be changed as appropriate.

  The route guidance situation is caused, for example, when the robot 12 talks to the person 14 about the route asking "How do I get to the conference room?" In such a situation, the robot 12 executes a cooperative operation in response to the operation of the human 14. In other words, for example, in human-to-human communication, when a person performs a specific action indicating the direction by pointing in a route guidance situation, an imitation action is often performed in which the other person points in the same way. . In addition, when a person points and indicates a direction, the face is turned in that direction. When the person explains route guidance with utterances, he / she is nodded or nodded. Thus, the operation | movement which synchronizes with a human motion is often performed. Humans unconsciously perform cooperative actions on the movements of the other party to realize smooth and natural communication.

  Therefore, in this embodiment, as an example of the imitation operation, when the human 14 performs a specific operation in the route guidance situation, that is, when pointing to indicate the direction and indicating the direction, the robot 12 Pointing in the same direction as the direction the human 14 points (gesture indicating the direction). Further, when the human 14 is moving his hand instead of pointing, the robot 12 moves in the same manner as the human 14. In these cases, the arm of the robot 12 used for the imitation operation is selected in consideration of the danger of a collision with the human 14, and therefore, not only the same operation but also a similar operation is performed as the imitation operation. . In this way, the robot 12 can realize natural and smooth communication with the human 14 by imitating the movement of the human 14, that is, an action similar to or the same as that of the human 14. it can.

  As an example of the synchronized operation, when the human 14 points to indicate the direction in order to guide the route, the robot 12 looks at the direction pointed by the human 14 (turns his face). Further, the robot 12 makes eye contact with the human 14 when the finger is not pointed. In addition, depending on the state of the utterance of the human 14, he / she confers or nods.

  Specifically, the body movements of the human 14 and the robot 12 are measured by the motion capture system 18 and stored in the three-dimensional movement measurement data DB 26 as the three-dimensional position coordinate data of each marker attachment point. Further, the utterance of the human 14 is acquired by the microphone 24 provided in the robot 12 and is stored as the utterance measurement data in the utterance measurement data DB 22 of the robot control device 16. Based on the body motion data (three-dimensional motion measurement data) stored in the three-dimensional motion measurement data DB 26 and the utterance data (speech measurement data) stored in the utterance measurement data DB 22, the robot control device 16 The behavior module to be executed by the robot 12 is selected from a plurality of behavior modules stored in the behavior module DB 20.

  FIG. 5 shows an example of the contents of a plurality of behavior modules stored in the behavior module DB 20. Action module is a program for executing the action with body movements and speech to the robot 12, for example bow, say "Hello", it has been prepared in advance for each action, such as to hug. FIG. 5 shows an example of a plurality of behavior modules including behavior modules for cooperative operations (imitation operation and tuning operation) used in the route guidance situation of this embodiment. In the behavior module DB 20, the behavior modules for the right arm, the left arm and the head, and the speech are stored in association with each identifier.

  Regarding the movement of the right arm, an action module (Rsr: identifier) that moves the same as the human right hand, an action module (Rsl) that moves the same as the human left hand, and an action module (Rpr) that points in the direction pointed by the right hand ), A behavior module (Rpl) that points in the direction pointed with the left hand, and a behavior module (Rno) that does nothing. Regarding the movement of the left arm, an action module (Lsl) that moves the same as the human left hand, an action module (Lsr) that moves the same as the human right hand, an action module (Lpr) that points in the direction pointed by the right hand, An action module (Lpl) that points in the direction pointed with the left hand and an action module (Lno) that does nothing are stored. Regarding the movement of the head, a behavior module (Hec) for making eye contact, a behavior module (Hrp) for viewing the direction pointed with the right hand, a behavior module (Hlp) for viewing the direction pointed with the left hand, and a nod behavior module ( Hnd) is stored. In addition, regarding speech, an action module (Seh) that says “E?”, An action module (Sun) that says “Yes”, an action module (Suu) that says “Yes”, and an action module that says “So” Ssd) is stored.

  In each of the behavior modules related to both arms and the head, based on the three-dimensional motion measurement data acquired by the motion capture system 18, each axis motor (right arm motor 88, left arm motor 90 and A target angle (angle control data) of the head motor 92) is calculated. For example, in the behavior modules (Rsr, Rsl, Lsl, Lsr) that perform the same movement as the hand of the human 14, first, the angles of the shoulder and elbow of the right hand and the left hand of the human 14 are respectively calculated based on the three-dimensional motion measurement data. . Then, as the robot 12 shows the same movement as the human 14 does, the motors (the right arm motor 88 and the left arm) of the shoulder joint 56R and the elbow joint 60R of the right arm and the shoulder joint 56L and the elbow joint 60L of the left arm of the robot 12 are shown. The target angle of the motor 90) is calculated based on the right and left hand angles of the human 14 respectively. Similarly, the behavior modules (Rpr, Rpl, Lpr, Lpl) pointing in the direction pointed by the human 14 calculate the angles of the shoulder and elbow of the right hand and the left hand of the human 14 respectively. Then, the target angles of the right arm motor 88 or the left arm motor 90 of the robot 12 are calculated based on the angles of the right hand and the left hand of the human 14 so that the robot 12 points in the direction pointed by the human 14. Further, in the action modules (Rno, Lno) that do nothing, the target angles of the right arm motor 88 and the left arm motor 90 of the robot 12 are changed to the angles indicating the home positions, respectively, or held at the same angle as the present. Further, the behavior module (Hec) for making eye contact calculates the direction vector of the head of the human 14 and the direction vector of the head 70 of the robot 12, and the two vectors indicate opposite directions. A desired angle of the head 70, that is, a target angle of the head motor 92 is calculated. In the behavior modules (Hrp, Hlp) for viewing the direction pointed by the human 14, first, the angles of the shoulder and elbow of the right hand or the left hand of the human 14 are respectively calculated. Then, the target angle of the head motor 92 of the robot 12 is calculated so that the robot 12 sees the direction pointed by the human 14. Note that in the behavior module related to utterance, synthesized voice data (utterance content data) indicating the corresponding utterance content is read from the HDD or the like.

  From such behavior modules, the robot control device 16 selects an appropriate behavior module according to the state of operation of the human 14. At the time of selection, the current operation state and the past operation state are detected as the operation state of the human 14.

  Based on the three-dimensional motion measurement data acquired from the three-dimensional motion measurement data DB 26 and the utterance measurement data acquired from the utterance measurement data DB 22 as the current motion state, the following six states are present in this embodiment. Determined.

  That is, (1) it is determined whether the human 14 is pointing with the right (or left) hand. As a result, in the case of the right hand, data indicating the right hand is set in the variable Point indicating the state related to pointing, and in the case of the left hand, the data indicating the left hand is set in the variable Point. When no insertion is made, data indicating no pointing is set in the variable Point. The condition that the action of the person 14 is a pointing is, for example, that the values of the shoulder and elbow angles of the person 14 are within a predetermined angle range. Note that the initial value of the variable Point is set to data indicating none. (2) It is determined to which side (right or left) of the robot 12 the human 14 is pointing. As a result, in the case of the right, data indicating the right is set in the variable Direction indicating the state relating to the direction, and in the case of the left, the data indicating the left is set in the variable Direction. The initial value of the variable Direction is set to data indicating no direction. (3) It is determined whether the right (or left) hand is moving. Specifically, it is determined whether the speed of movement of the right (or left) hand is greater than or equal to a threshold value. As a result, when both hands are used, data indicating both hands is set in the variable Move indicating the state relating to movement, when the right hand is set, data indicating the right hand is set in the variable Move, and when the left hand is set, the variable Data indicating the left hand is set in Move, and data indicating no movement is set in the variable Move when both hands are not moving. Note that the initial value of the variable Move is set to data indicating none. (4) It is determined whether the human 14 is using the right (or left) hand for route guidance (such as pointing or gesturing indicating direction). As a result, in the case of the right hand, data indicating the right hand is set in the variable Active indicating the state of use, and in the case of the left hand, the data indicating the left hand is set in the variable Active. Data indicating no use is set in the variable Active. Specifically, the state of use or activity is determined based on the current state and the past state. That is, the determination is made based on the three viewpoints of the hand set to Active last time, the movements of the right and left hands in the past certain time, and the distance from the shoulder of the hand in the current frame. If the previous Active is set to None, the hand with the most movement for a certain period of time in the past is set to Active. If the previous Active is either hand, both the distance from the shoulder and the movement for a certain period of time for the hand that is not set to Active are the same for the hand that is set to Active. Only when the distance from the shoulder and the movement over the past certain period of time have become larger, the hand set to Active can be switched. Note that the initial value of the variable Active is set to data indicating none. (5) It is determined whether the human 14 is in a place where the right (or left) arm of the robot 12 may hit. As a result, in the case of the right hand, the data indicating the right hand is set in the variable Hit indicating the state related to the collision, and in the case of the left hand, the data indicating the left hand is set in the variable Hit, and the human 14 is not hit. If there is, data indicating no collision is set in the variable Hit. Note that the initial value of the variable Hit is set to data indicating none. (6) It is determined whether the human 14 is speaking. Specifically, the time taken for utterance is measured, and the measured utterance time is set in the variable Speech related to utterance. Further, the elapsed time without utterance is measured and set to the variable Nospeech. Note that the initial values of the variable Speech and the variable Nospeech are each set to zero.

  Further, as the past operation states, the following four states are determined in this embodiment based on the past selection of the behavior module. That is, (1) With regard to the selection of arm movement, it is determined whether the action module Rpr, Rpl, Lpr or Lpl corresponding to the pointing has continued for a certain past time. (2) With regard to the selection of head movement, it is determined whether the behavior module related to the head has continued for a predetermined time in the past. (3) With regard to the selection of the head movement, it is determined whether the action module Hrp or Hlp responding to the pointing has continued for a certain past time. (4) Regarding the selection of an utterance, it is determined what the action module related to the last selected utterance was.

  The robot control device 16 selects a behavior module to be executed according to the behavior selection rule based on the detected motion state of the human 14. In this embodiment, the action selection rule is stored in advance in the HDD or the like as each selection processing program relating to arm movement, head movement and speech, and each selection process of arm movement, head movement and speech is executed according to each selection processing program. Is done. This action selection rule is created based on knowledge obtained through experiments by the inventors. Then, the robot control device 16 executes the selected action module, calculates the target angle of each motor as described above as necessary, and sends command data for instructing the execution of the action to the robot 12. Send to. In this embodiment, in the case of an action module related to body movement, the command data to be transmitted includes angle control data (target angle data) of each motor for realizing the action, and action related to speech In the case of a module, speech content data (synthesized speech data) indicating the content of speech is included.

  When the behavior module is executed, the robot 12 receives the command data from the robot control device 16 and executes the behavior based on the command data. In the case of command data related to body movement, each motor is controlled according to the angle control data, and in the case of command data related to speech, a voice is output according to the speech synthesis data. In this way, an action that imitates or synchronizes with the action of the human 14 is realized by the robot 12.

  FIG. 6 shows an example of the operation of the robot control device 16 in the normal route guidance situation. In the first step S1 of FIG. 6, the CPU of the robot control device 16 executes a state detection process. The operation of this process is shown in detail in FIGS.

  In the first step S21 in FIG. 7, the three-dimensional motion measurement data necessary for detecting the state of the body motion of the human 14 is acquired from the three-dimensional motion measurement data DB 26 via the computer of the motion capture system 18. Next, in step S23, it is determined whether or not the human 14 is pointing with the right hand. For example, the right shoulder and right elbow angles of the human 14 are calculated based on the three-dimensional motion measurement data, and it is determined whether or not these angles are within a predetermined angle range indicating the pointing state.

  If “YES” in the step S23, data indicating the right hand is set in a variable Point indicating the pointing state in a step S25. Subsequently, in step S27, it is determined whether or not the right hand of the human 14 points to the right side of the robot 12. For example, it is determined whether the direction indicated by the right forearm of the human 14 is the side where the right arm of the robot 12 is present or the side where the left arm is present when viewed from the center of the robot 12. If “YES” in the step S27, data indicating the right is set in the variable Direction indicating the direction state. On the other hand, if “NO” in the step S27, that is, if the human 14 points to the left side of the robot 12, data indicating the left is set in the variable Direction in a step S31. When step S29 or step S31 is completed, the process proceeds to step S45.

  On the other hand, if “NO” in the step S23, it is determined whether or not the human 14 is pointing with the left hand in a step S33. For example, the angles of the left shoulder and the left elbow of the human 14 are calculated, and it is determined whether or not these angles are within a predetermined angle range indicating the pointing state.

  If “YES” in the step S33, data indicating the left hand is set in the variable Point in a step S35. Subsequently, in step S37, it is determined whether or not the left hand of the human 14 points to the right side of the robot 12. For example, it is determined whether the direction indicated by the human left forearm is the side where the right arm of the robot 12 is present or the side where the left arm is present when viewed from the center of the robot 12. If “YES” in the step S37, data indicating the right is set in the variable Direction, and if “NO” in the step S37, data indicating the left is set in the variable Direction. When step S39 or step S41 is completed, the process proceeds to step S45.

  If “NO” in the step S33, that is, if the human 14 is not pointing with either the right hand or the left hand, data indicating no pointing is set in the variable Point in a step S43.

  In step S45, it is determined whether or not the right hand of the human 14 is moving. For example, the moving speed of the right hand of the human 14 is calculated based on the three-dimensional motion measurement data, and it is determined whether or not the calculated speed is equal to or higher than a predetermined threshold. If “NO” in the step S45, it is determined whether or not the left hand of the human 14 is moving in a step S47. For example, the moving speed of the human left hand is calculated, and it is determined whether the speed is equal to or higher than a predetermined threshold. If “NO” in the step S47, it is considered that both hands of the human 14 are not moving, and in a step S49, data indicating no movement is set in the variable Move indicating the movement state. On the other hand, if “YES” in the step S47, data indicating the left hand is set in the variable Move in a step S51. If “YES” in the step S45, it is determined in a succeeding step S53 whether or not the left hand of the human 14 is moving in the same manner as the above-described step S47. If “YES” in the step S53, it is considered that both hands of the human 14 are moving, and data indicating both hands is set in the variable Move in a step S55. On the other hand, if “NO” in the step S53, data indicating the right hand is set in the variable Move in a step S57. When step S49, S51, S55, or S57 is completed, the process proceeds to step S59 in FIG.

  In step S59 of FIG. 8, it is determined whether or not data indicating none is set in the variable Active indicating the use state of the hand. If “YES” in the step S59, that is, if it is determined that neither hand is used for the route guidance at the time of the previous process, the left hand of the human 14 in the step S61. It is determined whether or not the movement in the past certain time (for example, 1 second) is greater than the movement in the past certain time in the right hand. If “YES” in the step S61, it is assumed that the left hand is used for route guidance, and data indicating the left hand is set in the variable Active in a step S63. On the other hand, if “NO” in the step S61, it is determined whether or not the movement of the left hand in the past certain time is equal to or less than the movement of the right hand in the past certain time. If “YES” in the step S65, it is assumed that the right hand is used for route guidance, and data indicating the right hand is set in the variable Active in a step S67. If “NO” in the step S65, that is, if neither hand has moved during the past fixed time, data indicating absence is set in the variable Active in a step S69. When step S63, S67 or S69 ends, the process proceeds to step S89 in FIG.

  On the other hand, if “NO” in the step S59, it is determined whether or not the left hand movement is larger than the right hand movement in the step S71 as in the above-described step S61. If “YES” in the step S71, it is determined whether or not the distance from the shoulder of the left hand is longer than the distance from the shoulder of the right hand based on the three-dimensional motion measurement data of the current frame in a step S73. Here, the distance from the shoulder of the hand is the distance between the hand (hand or wrist) and the shoulder when the human body 14 is viewed from directly above. If “YES” in the step S73, that is, if the movement of the left hand in the past certain time is more than the movement of the right hand and the left hand is farther from the shoulder than the right hand, the left hand is used for route guidance. Assuming that it is being used, data indicating the left hand is set in the variable Active in step S75.

  On the other hand, if “NO” in the step S71, it is determined whether or not the left-hand movement is equal to or less than the right-hand movement in a step S77 in the same manner as the above-described step S65. If “YES” in the step S77, it is determined whether or not the distance from the shoulder of the left hand is equal to or less than the distance from the shoulder of the right hand based on the three-dimensional motion measurement data of the current frame in a step S79. If “YES” in the step S79, that is, if the movement of the right hand in the past certain time is more than the movement of the left hand and the right hand is more than the left hand and away from the shoulder, the right hand is used for route guidance. Assuming that it is used, data indicating the right hand is set in the variable Active in step S81. When step S75 or S81 ends, the process proceeds to step S89 in FIG.

  On the other hand, if “NO” in step S73, “NO” in step S77, or “NO” in step S79, the process proceeds to step S83, and the same hand as in the previous time is set to Active. Set. That is, in step S83, it is determined whether or not data indicating the left hand is set in the variable Active. If “YES” in the step S83, the data indicating the left hand is set in the variable Active in a step S85, and if “NO” in the step S83, the data indicating the right hand is set in the variable Active in a step S87. Advances to step S89 in FIG.

  In step S89 in FIG. 9, it is determined whether or not the human 14 is in a place where the right hand of the robot 12 may hit. That is, for example, based on the three-dimensional motion measurement data, it is determined whether or not the human 14 is present in the movable range of the right arm (sphere 66R, forearm 62R, elbow joint 60R, upper arm 58R, and shoulder joint 56R) of the robot 12. Is done. If “YES” in the step S89, data indicating the right hand is set in a variable Hit indicating a collision state in a step S91.

  On the other hand, if “NO” in the step S89, it is determined whether or not the human 14 is in a place where the left hand of the robot 12 may hit in a step S93. For example, it is determined whether or not the human 14 exists in the movable range of the left arm (the sphere 66L, the forearm 62L, the elbow joint 60L, the upper arm 58L, and the shoulder joint 56L) of the robot 12. If “YES” in the step S93, data indicating the left hand is set in the variable Hit in a step S95.

  If “NO” in the step S93, since there is no possibility that the both hands of the robot 12 collide with the human 14 in a step S97, data indicating absence is set in the variable Hit.

  Subsequently, in step S99, utterance measurement data necessary for detecting the utterance state of the person 14 is acquired from the utterance measurement data DB 22. Based on the utterance measurement data, it is determined in step S101 whether or not the human 14 is speaking.

  If “YES” in the step S101, the speech time speech measuring process is executed in a step S103. For example, the utterance time is counted in response to the detection of the start of utterance, the utterance time is continuously counted until the end of the utterance is detected, and when the end of the utterance is detected, the measured utterance time is set in the variable Speech.

  On the other hand, if “NO” in the step S101, a measurement process of the no-speech time Nospeech is executed in a step S105. For example, in response to the detection of the end of utterance, the utterance-free time is counted, and the utterance-free time is counted until the utterance start is detected, and the no-speech time is set in the variable Nospeech. When step S103 or S105 ends, this state detection process ends.

  Returning to FIG. 6, when step S <b> 1 is completed, arm motion selection processing is executed in step S <b> 3. The operation of the process for selecting the arm behavior module is shown in detail in FIGS.

  In the first step S131 of FIG. 10, it is determined whether or not the previous action module related to the arm movement is any pointing (Rpr, Rpl, Lpr or Lpl). If “YES” in the step S131, it is determined whether or not the pointing operation Rpr, Rpl, Lpr, or Lpl continues for a predetermined time or more in a step S133. If “NO” in the step S133, that is, if the pointing action module does not continue for a certain period of time, the same action module as the previous one is selected in the step S135 in order to continue the execution of the action module. . When step S135 ends, the arm motion selection process ends.

  On the other hand, if “YES” in the step S133, that is, if the pointing action module continues for a certain time or longer, the processing is performed so as to reselect the action module in accordance with the state detected in the step S1. Advances to step S137. Thereby, the robot 12 does not continue the operation of imitating the pointing operation for a long time, the time for realizing the imitation operation can be made appropriate, and a natural imitation operation can be realized. If “NO” in the step S131, the process similarly proceeds to the step S137.

  In step S137, it is determined whether data indicating the left hand is set in the variable Point indicating the pointing state. If “YES” in the step S137, it is determined whether or not data indicating the left hand is set in the variable Active indicating the use state in a step S139. If “YES” in the step S139, it is determined whether or not data indicating the left hand is set in the variable Hit indicating the collision state. If “YES” in the step S141, that is, if the human 14 is performing a pointing operation using the left hand and is present at a position where the left arm of the robot 12 can be hit, in a step S143. The action module Rpl for pointing the direction pointed with the left hand with the right arm is selected.

  In this way, when there is a possibility of hitting the person 14 if the body action of the person 14 is imitated as it is, a similar action in which the arm to be used is changed is selected as the imitation action. Therefore, the operation imitating the operation of the human 14 can be performed safely, and smooth and natural communication can be realized.

  On the other hand, if “NO” in the step S141, it is determined whether or not data indicating the right hand is set in the variable Hit in a step S145. If “YES” in the step S145, that is, if the human 14 is performing a pointing operation using the left hand and is present at a position where the right arm of the robot 12 can be hit, in a step S147. The action module Lpl for pointing the direction pointed with the left hand with the left arm is selected. In this case, since safety is ensured, the same operation using the same arm is selected as the imitation operation.

  If “NO” in the step S145, that is, if data indicating none is set in the variable Hit, data indicating the right is set in the variable Direction indicating the pointing direction in a step S149. Judge whether or not. If “YES” in the step S149, that is, if the human 14 is not present at a position where the arm of the robot 12 can be hit and is pointing to the right side when viewed from the robot 12 with the left hand, in a step S151, The action module Rpl for pointing the direction pointed with the left hand with the right arm is selected.

  On the other hand, if “NO” in the step S 149, that is, if data indicating the left is set in the variable Direction, that is, the human 14 is not present at a position where the arm of the robot 12 can hit, and the left hand is used. When pointing to the left side as viewed from the robot 12, the action module Lpl for pointing the direction pointed with the left hand with the left arm is selected in step S153.

  In this way, when safety is ensured, the imitation operation using the arm on the side corresponding to the direction in which the human 14 is pointing is selected. Therefore, since the robot 12 uses the arm on the same side as the pointing direction, it is possible to perform an operation imitating the operation of the human 14 in a more natural form, and to realize natural and smooth communication. .

  When step S143, S147, S151, or S153 is completed, the pointing operation time is counted for use in the determination in step S133 and the arm operation selection process is terminated.

  If “NO” in the step S 139, that is, if the left hand is pointing, but it is not recognized that the left hand is used, the pointing with the left hand is finished. Accordingly, in step S155, the behavior modules Rno and Lno for performing the operation of doing nothing with the right arm and the left arm are selected. As described above, since the human 14 detects the hand used for the motion, it is possible to accurately cope with the change of the motion. That is, since the imitation operation does not continue even if the operation is changed, a natural and uncomfortable imitation operation can be realized. When step S155 ends, the arm motion selection process ends.

  If “NO” in the step S137, it is determined whether or not data indicating the right hand is set in the variable Point in a step S157. If “YES” in the step S157, the process proceeds to a step S159 in FIG.

  In step S159 of FIG. 11, it is determined whether data indicating the right hand is set in the variable Active. If “YES” in the step S159, it is determined whether or not data indicating the right hand is set in the variable Hit in a step S161. If “YES” in the step S161, that is, if the human 14 is performing a pointing operation using the right hand and is present at a position where the right arm of the robot 12 can be hit, in a step S163. The action module Lpr for pointing the direction pointed with the right hand with the left arm is selected. In this way, the imitation operation can be performed safely.

  On the other hand, if “NO” in the step S161, it is determined whether or not data indicating the left hand is set in the variable Hit in a step S165. If “YES” in the step S165, that is, if the human 14 is performing a pointing operation using the right hand and is present at a position where the left arm of the robot 12 can be hit, in a step S167. The action module Rpr for pointing the direction pointed with the right hand with the right arm is selected.

  If “NO” in the step S165, that is, if data indicating none is set in the variable Hit, it is determined whether or not data indicating the right is set in the variable Direction in a step S169. To do. If “YES” in the step S169, that is, if the human 14 does not exist at a position where the arm of the robot 12 can hit and is pointing to the right side when viewed from the robot 12 with the right hand, in a step S171, The action module Rpr for pointing the direction pointed with the right hand with the right arm is selected.

  On the other hand, if “NO” in the step S169, that is, if data indicating the left is set in the variable Direction, that is, the human 14 is not present at a position where the arm of the robot 12 can be hit, and the right hand is used. When pointing to the left side when viewed from the robot 12, in step S173, the action module Lpr for pointing the direction pointed with the right hand with the left arm is selected. In this way, an imitation operation can be performed naturally.

  When step S163, S167, S171, or S173 is completed, the pointing operation time is counted for use in the determination in step S133 and the arm operation selection process is ended.

  If “NO” in the step S 159, that is, the finger is pointing with the right hand, but it is not recognized that the right hand is used, the finger pointing with the right hand is finished. Accordingly, the behavior modules Rno and Lno are selected in step S155 of FIG.

  If “NO” in the step S157 in FIG. 10, that is, if data indicating none is set in the variable Point, the process proceeds to a step S175 in FIG.

  In step S175 of FIG. 12, it is determined whether or not data indicating absence is set in the variable Hit. If “YES” in the step S175, it is determined whether or not data indicating both hands is set in the variable Move indicating the movement state in a step S177. If “YES” in the step S177, that is, if the human 14 moves both hands instead of pointing, and does not exist at a position where the arm of the robot 12 can be hit, in the step S179, the right arm is used. The action module Rsr that moves the same as the right hand of the human 14 and the action module Lsl that moves the same as the left hand of the human 14 with the left arm are selected. Thus, in this embodiment, even when the human 14 is not pointing, if the safety is ensured, the imitation operation is executed.

  On the other hand, if “NO” in the step S177, it is determined whether or not data indicating the right hand is set in the variable Move in a step S181. If “YES” in the step S181, that is, if the human 14 moves the right hand instead of pointing, and does not exist at a position where the arm of the robot 12 can be hit, the action module in the step S183. Select Rsr.

  If “NO” in the step S181, it is determined whether or not data indicating the left hand is set in the variable Move in a step S185. If “YES” in the step S185, that is, if the human 14 moves the left hand instead of pointing, and does not exist at a position where the arm of the robot 12 can hit, the action module Lsl in the step S187. Select. Note that when step S179, S183, or S187 ends, the arm motion selection process ends.

  If “NO” in the step S 185, that is, if data indicating none is set in the variable Move, it is assumed that the human 14 has not moved the hand, and the process proceeds to the step S 155 in FIG. Then, the behavior modules Rno and Lno are selected.

  On the other hand, if “NO” in the step S175 of FIG. 12, that is, if the human 14 is present at a position where the arm of the robot 12 can hit, data indicating the right hand is set in the variable Active in a step S189. It is judged whether it is done. If “YES” in the step S189, it is determined whether or not data indicating the right hand is set in the variable Hit in a step S191. If “NO” in the step S191, that is, if the human 14 is not pointing but uses the right hand and is present at a position where the left arm of the robot 12 can hit, the action is performed in the step S193. Module Rsr is selected.

  On the other hand, if “YES” in the step S191, that is, if data indicating the right hand is set in the variable Hit, that is, the human 14 is not pointing but uses the right hand, and the robot 12 If it exists at a position where the right arm can be hit, in step S195, the behavior module Lsr for making the same movement as the right hand of the human 14 with the left arm is selected. Also in this case, it is possible to safely perform the imitation operation by changing the arm to be used. When step S193 or S195 ends, the arm motion selection process ends.

  On the other hand, if “NO” in the step S189, it is determined whether or not data indicating the left hand is set in the variable Active in a step S197. If “YES” in the step S197, it is determined whether or not data indicating the left hand is set in the variable Hit in a step S199. If “YES” in the step S199, that is, if the human 14 is using the left hand instead of the pointing hand and is present at a position where the left arm of the robot 12 can be hit, the right arm in the step S201. The action module Rsl for making the same movement as the left hand of the human 14 is selected. Also in this case, the human 14 can be safely imitated by changing the arm.

  On the other hand, if “NO” in the step S199, that is, if data indicating the right hand is set in the variable Hit, that is, the human 14 is using the left hand instead of pointing, and the right arm of the robot 12 is used. If it exists at a position that can be hit, the action module Lsl is selected in step S203. When step S201 or S203 ends, the arm motion selection process ends.

  If “NO” in the step S197, that is, if data indicating that the variable Active is not set is set, that is, the human 14 does not point and does not use both hands, and the robot If it exists in a position that can hit either of the 12 arms, the behavior modules Rno and Lno are selected in step S155 of FIG.

  Returning to FIG. 6, when the arm motion selection process in step S3 is completed, the head motion selection process is executed in step S5. The operation of the process for selecting the head behavior module is shown in detail in FIG.

  In the first step S221 in FIG. 13, it is determined whether or not the previous action module related to the head movement continues for a predetermined time or more. If “NO” in the step S221, the same behavior module as the previous one is selected in a step S223 to continue the execution of the previous behavior module. When step S223 ends, this head movement selection process ends.

  On the other hand, if “YES” in the step S221, in the step S225, the previous behavior module is synchronized with the pointing or cooperative head movement, that is, the behavior module Hrp for viewing the direction pointed with the right hand or It is determined whether or not the action module Hlp is for viewing the direction pointed with the left hand. If “NO” in the step S225, the process proceeds to a step S229 so as to select an action module according to the operation state of the human 14.

  On the other hand, if “YES” in the step S225, it is determined whether or not the action module Hrp or Hlp corresponding to the pointing is continued for a predetermined time or more in a step S227. Note that the fixed time determined in step S227 is longer than the predetermined time determined in step S221, and is set to a time sufficient to be regarded as synchronized with the pointing action of the human 14, for example. Has been. If “NO” in the step S227, the process proceeds to a step S229 so as to perform an operation according to the state of the human 14.

  In step S229, it is determined whether data indicating the left hand is set in the variable Point. If “YES” in the step S229, that is, if the behavior module Hrp or Hlp has not continued for a certain time and the human 14 is pointing with the left hand, or if the previous behavior module is If the operation is other than the operations Hrp and Hlp that synchronize with the pointing and the human 14 is pointing with the left hand, the behavior module Hlp is selected in step S231.

  On the other hand, if “NO” in the step S229, it is determined whether or not data indicating the right hand is set in the variable Point in a step S233. If “YES” in the step S233, that is, if the action module Hrp or Hlp has not continued for a certain time and the human 14 is pointing with the right hand, or the previous action module is If the action is other than the actions Hrp and Hlp that are cooperative with pointing, and the human 14 is pointing with the right hand, the behavior module Hrp is selected in step S235.

  When step S231 or S235 is completed, counting of the execution time of the behavior module related to the head motion is started for use in the determination in step S221, step S227, and the head motion selection process is terminated. However, if “NO” is determined in the step S227, the head movement selection process is terminated without starting the count.

  If “NO” in the step S233, that is, if data indicating none is set in the variable Point, that is, if the human 14 is not pointing, the eye contact is performed in the step S237. The action module Hec to do is selected. Note that if “YES” in the step S227, that is, if the user 14 is synchronized with the finger 14 for a sufficient time, the behavior module Hec is selected in the step S237. When step S237 is ended, counting of the execution time of the behavior module related to the head motion is started for use in the determination in step S221 and the head motion selection process is ended.

  Returning to FIG. 6, when step S5 is completed, an utterance selection process is executed in step S7. The operation of the process for selecting the speech action module is shown in detail in FIG.

  In the first step S251 in FIG. 14, it is determined whether or not the value of the variable Speech indicating the time when the human 14 spoke is, for example, 1 second or more and less than 2 seconds. If “YES” in the step S251, it is determined whether or not the previous action module regarding the utterance is the action module Seh for speaking “E?” In a step S253. If “NO” in the step S253, the behavior module Seh is selected in a step S255. In this case, since the utterance time of the human 14 is short, the robot 12 expresses that it cannot be heard. If “YES” in the step S253, the utterance selection process is ended.

  On the other hand, if “NO” in the step S251, it is determined whether or not the value of the variable Speech is, for example, not less than 2 seconds and less than 5 seconds in a step S257. If “YES” in the step S257, an action module Sun for speaking “no” and an action module Hnd for nodding are selected in a step S259. In this case, since the utterance time of the human 14 is an appropriate length, the robot 12 normally hits each other. In addition, the action module regarding head movement is also selected here.

  If “NO” in the step S257, it is determined whether or not the variable Speech is, for example, 5 seconds or more in a step S261. If “YES” in the step S261, a behavior module Suu for speaking “no” and a behavior module Hnd for nodding are selected in a step S263. In this case, since the utterance time of the human 14 is relatively long, the robot 12 makes a match according to the long utterance.

  On the other hand, if “NO” in the step S261, it is determined whether or not the variable Nospeech indicating the no-speech time is, for example, 3 seconds or more in a step S265. If “YES” in the step S265, an action module Ssd for speaking “So” is selected in a step S267. In this case, since the silent time of the human 14 has become longer, the robot 12 prompts the human 14 to talk first.

  When step S255, S259, S263, or S267 ends, or if “NO” in the step S265, the utterance selection processing is ended.

  Returning to FIG. 6, when step S <b> 7 is completed, it is determined in step S <b> 9 whether or not the behavior module is to be executed. The behavior module is executed, for example, when each selection of the behavior module regarding the arm, the head, or the utterance is changed, or when a predetermined time required for executing each behavior module has passed.

  If “NO” in the step S9, the process proceeds to a step S15 without executing the behavior module. On the other hand, if “YES” in the step S9, it is determined whether or not a reaction delay time has elapsed in a step S11. When the human 14 does something, the robot 12 immediately performs this kind of imitation or tuning (for example, when the human 14 points, the head of the robot 12 immediately faces the same direction). If you do, it is clearly unnatural. Therefore, the inventors mount the reaction delay time acquired in the communication experiment between humans in the robot controller 16 so that the robot 12 performs the imitation operation and the tuning operation after a predetermined delay time elapses. According to the experiments by the inventors, the reaction delay time is felt to be natural if the reaction delay time is 0.89 seconds on average, so this value or a value in the vicinity thereof (for example, 1) Second).

  If “YES” in the step S11, that is, if a predetermined delay time has elapsed since an action module to be executed (that is, an action module whose selection has been changed or an action module that has passed a predetermined time) has been selected. In step S13, behavior module execution processing is executed. Details of the operation of this processing are shown in FIG.

  In the first step S281 in FIG. 15, the behavior module to be executed is read from the behavior module DB 20 and activated. Thus, in the case of an action module related to arm movement or head movement, the target angle of each motor is calculated using, for example, three-dimensional movement measurement data. In the case of an action module related to speech, speech content data is read from the HDD or the like.

  In step S283, command data including target angle data (angle control data) or utterance content data of each motor is generated, and the command data is transmitted to the robot 12.

  Subsequently, it is determined whether or not there is a behavior module to be executed in step S285. If “YES”, the processing of steps S281 and S283 is executed for the behavior module. If “NO” in the step S285, the action module executing process is ended.

  Returning to FIG. 6, in step S <b> 15, it is determined whether to end the imitation operation or the tuning operation. For example, it is determined whether or not the route guidance situation has ended, for example, the person 14 has left. If “NO” in the step S15, the process returns to the step S1. If “YES” in the step S15, the action processing in the route guidance situation is ended.

  FIG. 16 shows an example of the operation of the robot 12 when the behavior module is executed. In step S301, the CPU 76 of the robot 12 determines whether or not command data from the robot control device 16 has been received. If “NO” in the step S301, the process at the time of executing the behavior module is ended.

  On the other hand, if “YES” in the step S301, it is determined whether or not the command data is related to physical movement in a step S303. If “YES” in the step S303, the target angle data is given to the motor control board 82 in a step S305 to control the respective motors 88, 90, or 92. When the behavior module that realizes the cooperative operation is selected, the right arm, the left arm, or the head of the robot 12 shows an imitation operation or a tuning operation according to the state of the operation of the human 14.

  If “NO” in the step S303, that is, if the command data is data relating to the utterance, the utterance content data is given to the voice input / output board 86 in the step S307, and the voice is outputted from the speaker 72. Output. As a result, the robot 12 speaks in accordance with the state of movement of the human 14. When step S305 or S307 ends, this process ends.

  According to this embodiment, when imitating the human 14, the body motion data of the human 14 and the robot 12 is measured, the motion state of the human 14 is detected based on the body motion data, and the detected state is detected. The appropriate action module is selected. Therefore, it is possible to cause the robot 12 to perform an appropriate imitation operation according to the state of the human 14 operation, and to realize natural and smooth communication.

  Further, in this embodiment, in addition to the operation imitating the operation of the human 14, for example, the operation of turning the face in the pointing direction or the operation synchronized with the operation of the human 14 such as the reconciliation according to the utterance is also detected. Since the selection is made according to the performed state, the robot 12 can appropriately execute the operation synchronized with the action of the human 14. Therefore, natural and smooth communication between the human 14 and the robot 12 can be realized by causing the human 14 to appropriately perform cooperative operations on the robot 12.

  According to experiments conducted by the inventors using a subject, by performing a cooperative action corresponding to a human (subject) with which the robot 12 interacts (imitation operation and tuning operation in the above-described embodiment), the human It has been proved that the robot 12 can communicate smoothly. In this experiment, it has been verified that the body movement factor has an important effect in terms of listening, sharing, and empathy, and the speech factor has an important effect in terms of sharing. Furthermore, it has been verified that both body movement factors and utterance factors affect the empathic and reliable communication aspects. Therefore, the harmony between the body movement factor and the utterance factor promotes sympathetic and reliable communication, thereby realizing smoother communication.

  In the above-described embodiment, the volume of the utterance of the human 14 is measured, and the action module is selected according to the length of the utterance time. However, the robot controller 16 may be provided with a program and data for voice recognition. That is, in this case, the robot control device 16 acquires the voice data of the utterance of the human 14 and recognizes the voice to detect the utterance content, and the behavior related to the utterance or the body motion according to the detected utterance content. It is possible to select a module.

  In each of the above-described embodiments, a motor using electric power as a drive source is used as an actuator for driving a body part such as an arm, a head, or an eye of the robot 12. However, the robot 12 may be a robot 12 that expresses the body by other actuators such as air pressure (or negative pressure), hydraulic pressure, a piezoelectric element, or a shape memory alloy.

  In each of the above-described embodiments, the robot 12 is caused to execute a cooperative operation such as an imitation operation. However, in the motion generation system 10 according to the present invention, the motions of the human 14 and the robot 12 are detected in real time by, for example, the motion capture system 18 or the microphone 24, and data related to the motions (physical motion data and data related to speech) are generated. Therefore, the robot 12 can appropriately execute the operation corresponding to the operation of the human 14 in a wider range than the operation cooperative with the operation of the human 14. For example, in a situation where the robot 12 and the human 14 are separated or reunited, the robot 12 can be hugged by the human 14 when the human 14 raises his hand. Of course, a behavior module for realizing such a hugging operation is stored in the behavior module DB 20 in advance. Further, when the human 14 raises his / her hand, the robot motion control device 16 also performs an arm motion selection processing program in which the action module for the hugging motion is selected according to the motion state (Hit, Active, etc.) of the human 14. Is remembered. In this case, the range of movement of the hand of the robot 12 is calculated, the relative positional relationship with the human 14 is detected, and the human 14 is appropriately held by the signal of the touch sensor 64 disposed on the hand or arm. be able to. Further, when the touch sensor 64 is not a sensor that detects the on / off binary value but can detect the contact strength (pressure), for example, the applicant of the present application is effective as of March 24, 2003. In the case of a skin sensor or the like shown in detail in Japanese Patent Application No. 2003-80106 filed, it is possible to prevent the human 14 from being tightened by detecting the sensor output at the time of hugging.

  In each of the above-described embodiments, the robot control device 16 is provided separately from the robot 12. However, the functions of the robot control device 16, that is, programs for performing imitation operations, etc. are stored in the behavior module DB 20 and the speech. The measurement data DB 22 may be included in the robot 12.

  In each of the above-described embodiments, the communication partner is the human 14, but the robot 12 is not limited to a human, and may be another communication robot, for example.

It is an illustration figure which shows the outline | summary of the action production | generation system of one Example of this invention. It is an illustration figure which shows an example of the attachment position of a marker. It is an illustration figure (front view) which shows an example of the external appearance of the communication robot of FIG. It is a block diagram which shows an example of an electrical configuration of the communication robot of FIG. It is an illustration figure which shows an example of the content of the action module memorize | stored in action module DB of FIG. It is a flowchart which shows an example of the operation | movement at the time of imitation of the robot control apparatus of FIG. It is a flowchart which shows a part of example of operation | movement of the state detection process of FIG. It is a flowchart which shows a part of continuation of FIG. FIG. 9 is a flowchart showing a continuation of FIG. 8. It is a flowchart which shows a part of example of operation | movement of the arm operation | movement selection process of FIG. It is a flowchart which shows a part of continuation of FIG. FIG. 11 is a flowchart showing another part of the continuation of FIG. 10. It is a flowchart which shows an example of operation | movement of the head movement selection process of FIG. It is a flowchart which shows an example of the operation | movement of the speech selection process of FIG. It is a flowchart which shows an example of operation | movement of the action module execution process of FIG. It is a flowchart which shows an example of the operation | movement at the time of the action of the communication robot of FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Motion generation system 12 ... Communication robot 14 ... Human 16 ... Robot control apparatus 18 ... Motion capture system 20 ... Action module DB
24 ... Microphone 56R, 56L ... Shoulder joint 58R, 58L ... Upper arm 60R, 60L ... Elbow joint 62R, 62L ... Forearm 68 ... Neck joint 70 ... Head 82 ... Motor control board 88 ... Right arm motor 90 ... Left arm motor 92 ... Head motor

Claims (10)

An operation generation system comprising a communication robot having at least a body part including an arm and an actuator for driving the body part, and causing the communication robot to execute an action corresponding to an action of a communication partner,
Program storage means for storing a plurality of behavior programs for causing the communication robot to execute an action corresponding to the action of the opponent;
Motion detection means for generating motion data by detecting motions of the opponent and the communication robot;
State detecting means for detecting the state of the other party based on the data relating to the operation;
Selection means for selecting an action program from a plurality of action programs stored in the program storage means according to the state of the opponent detected by the state detection means;
A motion generation system comprising: generation means for generating control data for realizing the operation according to the behavior program selected by the selection means; and actuator control means for controlling the actuator based on the control data. The body part of the communication robot further includes a head,
The action generation system according to claim 1, wherein the action program stored in the program storage unit includes an action program for causing the communication robot to execute a cooperative action on the opponent. The action program stored in the program storage means includes an action program for causing the communication robot to execute a mimic action that imitates the opponent,
The motion detection means includes body motion detection means for generating body motion data by detecting the body motion of the opponent and the communication robot,
The state detection means detects the state of the other party based on at least the body movement data,
The motion generation system according to claim 1, wherein the generation unit generates the control data based on the body motion data when an action program for executing the imitation operation is selected by the selection unit. The program storage means stores a plurality of behavior programs for executing the imitation operation,
The state detection means detects a first state detection means for detecting a first state relating to a hand in which the opponent is performing a specific action, and a second state relating to an arm of the communication robot that may collide with the opponent. Including a second state detecting means;
The selection means selects an action program for executing an action imitating the specific action with an arm different from the arm indicated in the second state when the first state indicates either hand of the opponent. The motion generation system according to claim 3. The state detecting means further includes third state detecting means for detecting a third state relating to the direction of the specific action,
When the second state does not indicate any arm, the selection unit selects an action program for executing an operation imitating the specific operation with an arm on a side corresponding to the direction indicated by the third state. The motion generation system according to claim 4. The state detection means detects the fourth state relating to the hand used by the opponent for the specific action, the distance from the shoulder of the opponent's hand, the movement of the opponent's hand for a certain past time, and the previous detection. Detecting based on the fourth state,
The operation generation according to claim 4, wherein the selection means selects an action program for executing an operation that is not the imitation operation when the hand indicated by the first state and the hand indicated by the fourth state are not the same. system.   The selection means reselects the action program according to the state of the opponent detected by the state detection means when selection of the action program for executing an action imitating a specific action continues for a certain time or more. The motion generation system according to any one of claims 3 to 6.   The motion generation system according to claim 3, wherein the specific motion includes a finger motion or a physical motion indicating a direction. The motion detection means includes utterance detection means for detecting the utterance of the opponent and generating data related to the utterance,
The state detection means detects the state of the other party's utterance based on the data related to the utterance,
The selection unit selects an action program for causing the partner to perform a cooperative action when the state of the partner's utterance detected by the state detection unit satisfies a predetermined condition. Item 3. The motion generation system according to Item 2. The communication robot further includes voice output means for outputting voice,
The action program stored in the program storage means includes an action program for speaking,
The motion detection means includes utterance detection means for detecting the utterance of the opponent and generating data related to the utterance,
The state detection means detects the state of the other party's utterance based on the data related to the utterance,
The selection means is for selecting an action program for each body part and utterance of the communication robot, and for making the utterance according to at least the utterance state of the opponent detected by the state detection means. Select an action program,
The generating means generates control data for making an utterance when an action program for making an utterance is selected by the selecting means,
The motion generation system according to claim 1, wherein the voice output unit outputs a voice based on the control data.

Images