JP2006268607A - Communication robot and motion identification system using it - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a communication robot accurately specifying a human motion while reducing a calculation time, and a motion identification system using it. <P>SOLUTION: The motion identification system 10 comprises the communication robot 12. The communication robot 12 detects a tag ID of an article owned by a person or present in the vicinity of the person. The communication robot 12 determines motion candidates of the person from the detected tag ID. The communication robot 12 takes an image of the person by an eye camera. The communication robot calculates the similarity of each of the motion candidates to the taken image, and specifies a motion shown by the taken image as a motion with the highest similarity. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a communication robot and a motion identification system using the communication robot, and more particularly to a communication robot and a motion identification system using the communication robot for identifying a human motion, for example.

The present applicant has proposed a communication robot that interacts with a human, as represented by Patent Document 1.
Japanese Patent Application Laid-Open No. 2002-355783

  When using a communication robot of the background art to identify human movements around it, for example, video images of all human movements (behaviors) are recorded in advance, and recorded video images are recorded in advance. It compares with the stored video and identifies any one action. In this case, the calculation amount is enormous. In addition, when an operation is determined only by a video, there is a possibility that an operation similar to the video even if it is a different operation is specified as an erroneous operation.

  Therefore, a main object of the present invention is to provide a novel communication robot and a motion identification system using the communication robot.

  Another object of the present invention is to provide a communication robot and an action identification system using the same that can accurately specify human behavior by reducing calculation processing.

  According to the first aspect of the present invention, a photographing means for photographing a human, an identification information detecting means for detecting identification information about an object existing in the vicinity of the human, and a human action based on the identification information detected by the identification information detecting means. The communication robot includes a motion predicting unit that predicts, and a motion specifying unit that specifies the actual motion from a human actual motion indicated by a captured image captured by the capturing unit and a predicted motion predicted by the motion predicting unit.

  According to the first aspect of the present invention, the communication robot includes a photographing unit, an identification information detecting unit, an operation predicting unit, and an operation specifying unit. The photographing means photographs a human, for example. The identification information detection means detects identification information about an object (article) that exists in the vicinity of the robot in the environment where the robot itself exists, strictly speaking, an article that exists in the vicinity of a human. In the embodiment, the tag ID transmitted from the wireless tag attached to the article is detected. The motion prediction means predicts a human motion based on the identification information detected by the identification information detection means. For example, if a person has a pencil or a pencil is in the vicinity of a person, predict the movement of the person who is writing or reading the character. Can do. Then, the action specifying unit specifies the actual action from the human actual action indicated by the photographed image taken by the photographing means and the predicted action predicted by the action predicting unit.

  According to the first aspect of the present invention, since the motion is predicted based on the object existing in the vicinity of the human and the actual motion is specified from the predicted motion, it is much more significant than the case where one motion is specified from every motion. Specific processing can be reduced. Further, since one motion is specified by the combination of the object and the actual motion, the motion can be accurately specified.

  The invention of claim 2 is dependent on claim 1, and further comprises similarity calculation means for calculating the similarity between each of the human actual motion and the predicted action indicated by the captured image, and the action specifying means is the similarity calculation means. The predicted motion with the highest similarity calculated by is identified as an actual motion.

  In the invention of claim 2, the communication robot further includes a similarity calculation means. This similarity calculation means calculates the similarity between each human actual motion and predicted motion indicated by the captured image. The action specifying unit specifies a predicted action having the highest similarity as an actual action.

  According to the second aspect of the present invention, since the degree of similarity with the predicted motion narrowed down based on the identification information is only calculated, the processing load can be reduced as compared with the case of calculating the degree of similarity with all motions. In addition, since the actual motion is specified from the narrowed predicted motion, the identification accuracy can be made higher than when the similarity is calculated for all motions.

  The invention of claim 3 is an operation identification system comprising a communication robot and a server provided so as to be communicable with the communication robot, wherein the communication robot is used for photographing means for photographing a person and an object existing in the vicinity of the person. Identification information detecting means for detecting the identification information of the user, identification information transmitting means for transmitting the detected identification information to the server, predicted motion information receiving means for receiving human predicted motion information from the server, and a photographed image taken by the photographing means The operation identification means for identifying the actual movement from the human actual movement indicated by the prediction movement information received by the prediction movement information receiving means, and the server receives the identification information transmitted to the identification information transmission means. Based on the information receiving means and the identification information received by the identification information receiving means The prediction operation information obtained by human comprising a prediction operation information transmitting means for transmitting to the communication robot, an operation identification system.

  The invention of claim 3 is an operation identification system comprising a communication robot and a server provided so as to be communicable with the communication robot. In this motion identification system, when the communication robot detects identification information of an object existing in the vicinity of a person, the communication robot transmits the identification information to a server. The server predicts the operation based on the identification information, and transmits the predicted operation information to the communication robot. That is, the invention of claim 2 is different from the invention of claim 1 in that a server as a means or role for predicting the operation is provided.

  According to the invention of claim 3, as in the invention of claim 1, it is possible to greatly reduce the calculation processing for the action specification. Moreover, the burden of calculation processing in the robot can be reduced.

  The invention of claim 4 is an operation identification system comprising a communication robot and a server provided so as to be communicable with the communication robot, wherein the communication robot is for photographing means for photographing a person, and an object existing in the vicinity of the person. Identification information detecting means for detecting the identification information, and transmission means for transmitting the photographed image photographed by the photographing means and the identification information detected by the identification information detecting means to the server, and the server is transmitted to the transmitting means. Receiving means for receiving the captured image and identification information, operation predicting means for predicting human motion based on the identification information received by the receiving means, and actual human motion indicated by the captured image received by the identification information receiving means And a motion characteristic that identifies the actual motion from the motion predicted by the motion prediction means. Comprising means, an operation identification system.

  In the invention of claim 4, the communication robot transmits identification information and a human captured image to the server, the server predicts an action based on the identification information, and the captured image is obtained from each of the predicted actions and the captured image. The present invention is the same as the invention of claim 3 except that the human action to be shown is specified.

  In the invention of claim 4, as in the invention of claim 1, the calculation processing for specifying the operation can be greatly reduced. Moreover, the burden of calculation processing in the robot can be reduced.

  According to the present invention, since the human motion is estimated from an object existing in the vicinity of the human and is identified as any one of the estimated motions, the amount of calculation can be greatly reduced. Further, since the motion is specified not only by the image but also by the combination with the object, the motion can be accurately specified.

  The above object, other objects, features and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

<First embodiment>
Referring to FIG. 1, an operation identification system (hereinafter simply referred to as “system”) 10 according to the first embodiment includes a communication robot (hereinafter simply referred to as “robot”) 12 and a plurality of tags 14. The robot 12 can take a body motion or action (hereinafter sometimes referred to as “communication action”) such as gestures with a human. However, the communication action may include a conversation between the robot 12 and a human. The tag 14 is, for example, a passive type wireless tag (RF tag, IC tag, etc.), and is attached (pasted) to an object (not shown in FIG. 1) that exists at least in the environment where the robot 12 exists.

  The robot 12 has a human-like body and generates complex body movements necessary for communication using the body. Specifically, referring to FIG. 2, the robot 12 includes a carriage 32, and wheels 34 for autonomously moving the robot 12 are provided on the lower surface of the carriage 32. The wheel 34 is driven by a wheel motor (indicated by reference numeral “36” in FIG. 3 showing the internal configuration of the robot 12), and the carriage 32, that is, the robot 12 can be moved in any direction.

  Although not shown in FIG. 2, a collision sensor (indicated by reference numeral “38” in FIG. 3) is attached to the front surface of the carriage 32, and the collision sensor 38 is connected to a person or other person to the carriage 32. Detect obstacle contact. When contact with an obstacle is detected during the movement of the robot 12, the driving of the wheels 34 is immediately stopped and the movement of the robot 12 is suddenly stopped.

  In the first embodiment, the height of the robot 12 is about 100 cm so as not to intimidate people, particularly children. However, this height can be arbitrarily changed.

  A polygonal column sensor mounting panel 40 is provided on the carriage 32, and an ultrasonic distance sensor 42 is mounted on each surface of the sensor mounting panel 40. The ultrasonic distance sensor 42 measures the distance between the mounting panel 40, that is, the person around the robot 12 mainly.

  Further, the body of the robot 12 is mounted on the carriage 32 so that the lower portion thereof is surrounded by the mounting panel 40 described above and stands upright. The body is composed of a lower body 44 and an upper body 46, and the lower body 44 and the upper body 46 are connected by a connecting portion 48. Although not shown, the connecting portion 48 has a built-in lifting mechanism, and the height of the upper body 46, that is, the height of the robot 12 can be changed by using the lifting mechanism. As will be described later, the elevating mechanism is driven by a waist motor (indicated by reference numeral “50” in FIG. 3). The height 100 cm of the robot 12 described above is a value when the upper body 46 is at its lowest position. Therefore, the height of the robot 12 can be 100 cm or more.

  One omnidirectional camera 52 and one microphone 16 are provided in the approximate center of the upper body 46. The omnidirectional camera 52 photographs the surroundings of the robot 12 and is distinguished from an eye camera 54 described later. The microphone 16 captures ambient sounds, particularly human voice.

  Upper arms 58R and 58L are attached to both shoulders of the upper body 46 by shoulder joints 56R and 56L, respectively. The shoulder joints 56R and 56L each have three degrees of freedom. That is, the right shoulder joint 56R can control the angle of the upper arm 58R around each of the X, Y, and Z axes. The Y axis is an axis parallel to the longitudinal direction (or axis) of the upper arm 58R, and the X axis and the Z axis are axes orthogonal to the Y axis from different directions. The left shoulder joint 56L can control the angle of the upper arm 58L around each of the A, B, and C axes. The B axis is an axis parallel to the longitudinal direction (or axis) of the upper arm 58L, and the A axis and the C axis are axes orthogonal to the B axis from different directions.

  Forearms 62R and 62L are attached to the respective distal ends of upper arms 58R and 58L via elbow joints 60R and 60L. The elbow joints 60R and 60L can control the angles of the forearms 62R and 62L around the W axis and the D axis, respectively.

  In the X, Y, Z, W axes and the A, B, C, D axes that control the displacement of the upper arms 58R and 58L and the forearms 62R and 62L (FIG. 2), “0 degree” is the home position. In this home position, the upper arms 58R and 58L and the forearms 62R and 62L are directed downward.

  Although not shown in FIG. 2, a touch sensor (FIG. 3) is provided on the shoulder portion including the shoulder joints 56R and 56L of the upper body 46 and the arm portion including the upper arms 58R and 58L and the forearms 62R and 62L. The touch sensor 64 detects whether or not a person has touched these parts of the robot 12.

  Spheres 66R and 66L corresponding to hands are fixedly attached to the tips of the forearms 62R and 62L, respectively. Instead of the spheres 66R and 66L, a “hand” in the shape of a human hand can be used when a finger function is required unlike the robot 12 of the first embodiment. The sphere 66R is provided with a right-hand tag reader 102, and the sphere 66L is provided with a left-hand tag reader 104. However, the tag reader may be provided only on one of the spheres or hands.

  A head 70 is attached to an upper center of the upper body 46 via a neck joint 68. The neck joint 68 has three degrees of freedom and can be controlled in angle around each of the S, T, and U axes. The S-axis is an axis that goes directly from the neck, and the T-axis and the U-axis are axes that are orthogonal to the S-axis in different directions. The head 70 is provided with a speaker 72 at a position corresponding to a human mouth. The speaker 72 is used for the robot 12 to communicate with a person around it by voice or voice. However, the speaker 72 may be provided in another part of the robot 12, for example, the trunk.

  The head 70 is provided with eyeball portions 74R and 74L at positions corresponding to the eyes. Eyeball portions 74R and 74L include eye cameras 54R and 54L, respectively. The right eyeball portion 74R and the left eyeball portion 74L may be collectively referred to as an eyeball portion 74, and the right eye camera 54R and the left eye camera 54L may be collectively referred to as an eye camera 54. The eye camera 54 captures the video signal by photographing the face of the person approaching the robot 12 and other parts or objects.

  Note that each of the omnidirectional camera 52 and the eye camera 54 described above may be a camera using a solid-state imaging device such as a CCD or a CMOS.

  For example, the eye camera 54 is fixed in the eyeball part 74, and the eyeball part 74 is attached to a predetermined position in the head 70 via an eyeball support part (not shown). The eyeball support unit has two degrees of freedom and can be controlled in angle around each of the α axis and the β axis. The α axis and the β axis are axes set with respect to the head 70, the α axis is an axis in a direction toward the top of the head 70, the β axis is orthogonal to the α axis and the front side of the head 70 It is an axis in a direction orthogonal to the direction in which (face) faces. In the first embodiment, when the head 70 is at the home position, the α axis is set to be parallel to the S axis and the β axis is set to be parallel to the U axis. In such a head 70, when the eyeball support portion is rotated around each of the α axis and the β axis, the tip (front) side of the eyeball portion 74 or the eye camera 54 is displaced, and the camera axis, that is, the line-of-sight direction is changed. Moved.

  In the α axis and β axis that control the displacement of the eye camera 54, “0 degree” is the home position. At this home position, the camera axis of the eye camera 54 is the head 70 as shown in FIG. The direction of the front side (face) is directed, and the line of sight is in the normal viewing state.

  FIG. 3 is a block diagram showing the internal configuration of the robot 12. As shown in FIG. 3, the robot 12 includes a microcomputer or a CPU 76 for overall control. The CPU 76 is connected to a memory 80, a motor control board 82, a sensor input / output board 84, and a voice through a bus 78. An input / output board 86 is connected.

  Although not shown, the memory 80 includes a ROM, an HDD, a RAM, and the like, and the control program and data for the robot 12 are stored in the ROM or the HDD in advance. The CPU 76 executes processing according to this program. Specifically, a plurality of programs (referred to as action modules) for controlling the body movement of the robot 12 are stored. For example, the body motions indicated by the behavior module include “handshake”, “cuckling”, “many years”, and so on. When the body motion indicated by the behavior module is “handshake”, when the behavior module is executed, the robot 12 presents the right hand forward, for example. Further, when the body motion indicated by the behavior module is “cuddle”, when the behavior module is executed, the robot 12 presents both hands forward, for example. Further, when the body motion indicated by the behavior module is “many years”, when the behavior module is executed, the robot 12 raises and lowers both hands several times (for example, twice), for example. The RAM can be used as a working memory as well as a temporary storage memory.

  The motor control board 82 is composed of, for example, a DSP (Digital Signal Processor) and controls a motor for driving body parts such as the right arm, the left arm, the head, and the eyes. That is, the motor control board 82 receives control data from the CPU 76, and controls the angles of the three motors for controlling the X, Y, and Z axes of the right shoulder joint 56R and the axis W of the right elbow joint 60R. The rotation angle of a total of four motors including one motor (collectively shown as “right arm motor” in FIG. 3) 88 is adjusted. The motor control board 82 includes a total of four motors including three motors that control the angles of the A, B, and C axes of the left shoulder joint 56L and one motor that controls the angle of the D axis of the left elbow joint 60L. The rotation angle of the motor (collectively shown as “left arm motor” in FIG. 3) 90 is adjusted. The motor control board 82 also adjusts the rotation angle of three motors 92 (collectively shown as “head motors” in FIG. 3) that control the angles of the S, T, and U axes of the neck joint 68. To do. The motor control board 82 also controls the waist motor 50 and the two motors 36 that drive the wheels 34 (collectively shown as “wheel motors” in FIG. 3). Further, the motor control board 82 adjusts the rotation angle of two motors 94 (collectively shown as “right eyeball motor” in FIG. 3) that control the angles of the α axis and β axis of the right eyeball portion 74R. In addition, the rotation angles of two motors 96 that collectively control the angles of the α axis and β axis of the left eyeball portion 74L (collectively shown as “left eyeball motor” in FIG. 3) 96 are adjusted.

  The above-described motors of the first embodiment are stepping motors or pulse motors for simplifying the control, except for the wheel motors 36. However, like the wheel motors 36, they may be DC motors. .

  Similarly, the sensor input / output board 84 is also constituted by a DSP, and takes in signals from each sensor and camera and gives them to the CPU 76. That is, data relating to the reflection time from each of the ultrasonic distance sensors 42 is input to the CPU 76 through the sensor input / output board 84. The video signal from the omnidirectional camera 52 is input to the CPU 76 after being subjected to predetermined processing by the sensor input / output board 84 as necessary. Similarly, the video signal from the eye camera 54 is also supplied to the CPU 76. Further, a signal from the touch sensor 64 is given to the CPU 76 via the sensor input / output board 84.

  Synthetic voice data is given to the speaker 72 from the CPU 76 via the voice input / output board 86, and accordingly, voice or voice according to the data is outputted from the speaker 72. Further, the voice input from the microphone 24 is taken into the CPU 76 via the voice input / output board 86.

  Further, a communication LAN board 98 is connected to the CPU 76 through the bus 78. Similarly, the communication LAN board 98 is also configured by a DSP, and sends the transmission data given from the CPU 76 to the wireless communication apparatus 100 and causes the wireless communication apparatus 100 to transmit the transmission data. The communication LAN board 98 receives data via the wireless communication device 100 and provides the received data to the CPU 76.

  Further, the tag reader 102, the tag reader 104, and the database 106 are connected to the CPU 76 via the bus 78. However, the database 106 does not need to be provided inside the robot 12 and can be provided outside the robot 12 so as to be communicable with the robot 12. As described above, the tag reader 102 and the tag reader 104 are provided in the hand of the robot 12 (spheres 64R and 64L), and tag information (tag ID) transmitted from the tag 14 attached to an object or the like existing around the robot 12 Is given to the CPU 76.

  Further, as shown in FIGS. 4 and 5, the database 106 stores data such as article data 106a, motion candidate data 106b, motion template data 106c, and the like. As shown in FIG. 4A, the article data 106a describes the name of the article (object) to which the corresponding tag 14 is attached, corresponding to the identification information (tag ID) of the tag 14. That is, the article data 106a is table data for identifying an article from the tag ID. Further, as shown in FIG. 4B, the motion candidate data 106b describes motion candidates (motion candidates) corresponding to the names of articles. For example, when the article is “pencil”, operations such as “hold”, “write”, “see”, and “place” are described as motion candidates. When the article is a “knife”, operations such as “hold”, “place”, “cut”, “sharp”, “wash” are described. For example, if there is a pencil near a person, the person holds the pencil (holds it), writes characters with the pencil, looks at the contents written with the pencil, or places the pencil It is thought that the operation. If there is a knife near the person, the person holds the knife (holds it), puts the knife, cuts food (meat, fish, vegetables, fruits, etc.) with the knife, It is considered that the knife is sharpened or the knife is washed. In other words, the motion candidate data 106b is table data for determining candidates for human motion (behavior) that can be predicted (estimated) from an article (object).

  Also, as shown in FIG. 5, the action template data 106c describes the file name of a video (image) file when a person who performs the action is photographed, corresponding to the action. Although not shown, the database 106 also stores actual image files. However, the image file itself may be described corresponding to each operation.

  For example, as shown in FIG. 6, the system 10 is applied to a certain room 200. However, in FIG. 6, the robot 12 and some tags 14 are shown for simplicity. Further, as shown in FIG. 6, a person 202 exists in the room 200 and the robot 12 exists in the vicinity thereof. A desk 204 is disposed in the room 200, and a notebook 206 is placed on the desk 204. For example, a person 202 holds a pencil 208 and writes a character or the like on a notebook 206. Further, a trash can 210 is placed in the corner of the room 200.

  Further, as described above, the tag 14 is attached to each article (object) present in the room 200. Although FIG. 6 shows that the tag 14 is attached only to the desk 204, the tag 14 is actually attached to the notebook 206, the pencil 208, and the trash can 210. Further, when identifying the room 200 itself (or indoor or outdoor), that is, when identifying the place, the tag 14 may be attached to the entrance or wall of the room 200.

  For example, the system 10 identifies (specifies) the action (action) of the human 202. Specifically, the CPU 76 shown in FIG. 3 executes the operation identification process shown in FIGS. As shown in FIG. 7, when the CPU 76 starts the operation identification process, the tag reader 102 and the tag reader 104 are moved in step S1 to detect the tag ID. Specifically, the person 202 is detected from the captured image (video) of the eye camera 54, approaches the person 202, and detects an article (strictly, a tag ID) that exists in the vicinity of the robot 12. However, in the first embodiment, since the movement of the human 202 is identified, strictly speaking, an article existing in the vicinity (for example, within a range of about 1 to 2 m and within the reach of a human hand) is detected. I have to do it.

  In addition, since the motion of the human 202 is identified (specified) from an article existing in the vicinity of the human 202 and a captured image (video) of the human 202, the robot 12 is brought closer to the human 202. Although detailed description is omitted, the robot 12 detects (estimates) the human 202 from the captured image (video) of the eye camera 54 by a pattern matching method, and proceeds in that direction. Alternatively, when a skin color area included in the photographed image (video) is detected, the skin color area advances (or moves backward) or rotates in a direction in which the skin color area increases. In this way, the robot 12 can approach the human 202.

  However, the detection result (distance) of the ultrasonic distance sensor 42 may be used to determine whether or not the person 202 is approached, and the captured image of the eye camera 54 and the detection result of the ultrasonic distance sensor 42 are used. It may be.

  When the robot 12 approaches the human 202, it detects a tag ID received by the tag reader 102 and the tag reader 104 provided in both hands, that is, the sphere 66R and the sphere 66L. In the first embodiment, the tag reader 102 and the tag reader 104 are moved freely by moving both hands (arms), that is, the shoulder joints 56R and 56L and the elbow joints 60R and 60L, and the tag ID is detected by the tag reader 102 or the tag reader 104.

  In a succeeding step S3, it is determined whether or not the tag ID is detected. If “NO” in the step S3, that is, if the tag ID is not detected, the process returns to the step S1, the tag reader 102 or the tag reader 104 or both are moved, and the tag ID is detected. On the other hand, if “YES” in the step S3, that is, if a tag ID is detected, the tag reader 102 or 104 (in either case, either may be used) and the eye that have detected the tag ID in a step S5. The distance to the camera 54 is calculated and stored (temporarily stored) in the memory 80. Here, the position of the eye camera 54 and the lengths of both hands (such as the upper arms 58R and 58L and the forearms 62R and 62L) are fixed, and the tag reader 102 and the tag reader 104 are attached to the spheres 66R and 66L. , 56L and the angles of the elbow joints 60R, 60L are calculated.

  The distance between the eye camera 54 and the tag readers 102 and 104 is measured as described above, because the tag 14 is of a passive type, and the approximate distance between the eye camera 54 and an article (object) is measured. This is because the distance between the person 202 and the article is finally estimated. As a result of this, and as described above, the tag ID of the article that the person 202 possesses (or wears) or exists in the vicinity can be detected by moving forward so that the skin color area becomes larger. Therefore, although illustration is omitted, even when the tag ID is detected, when moving in step S1, the skin color area is not detected or the skin color area is detected, but the area is relatively small, When it is determined that the distance between the robot 12 and the human 202 is long, the detected tag ID is excluded (rejected) from the detection result. This is to accurately identify the operation of the human 202.

  Subsequently, in step S7, both hands (shoulder joints 56R and 56L, elbow joints 60R and 60L) are moved to move the tag reader 102 and the tag reader 104 to positions that are not reflected in the eye camera 54. However, it is preferable to move not only the tag readers 102 and 104 but also a position where both hands are not captured. This is because, as will be described later, the human 202 is photographed by the eye camera 54 and the operation of the human 202 is identified. That is, this is because the human 202 is photographed by the eye camera 54 as much as possible.

  In the subsequent step S9, the detected tag ID, detection time, and detection location are stored. Although omitted in FIG. 3, the detection time is acquired from a clock circuit (timer) provided in the robot 12. Moreover, as for the detection location, if a tag assigned in advance for each location is installed and this tag ID is detected, the location can be specified in the same manner as the article. In a succeeding step S11, it is determined whether or not the skin color is detected from the photographed image. If “NO” in the step S11, that is, if the skin color is not detected from the photographed image, the process proceeds to a step S15 as it is. On the other hand, if “YES” in the step S11, that is, if a skin color (region) is detected from the photographed image, the eye camera 54 is turned in a step S13 so that the skin color region is at the center of the photographed image. . This is for photographing the human 202 (the operation thereof) and accurately identifying the operation. However, not only the eye camera 54 but also the direction of the entire robot 12 may be changed (turned). In this way, the person 202 and its operation can be photographed.

  In the first embodiment, when the skin color area is detected from the photographed image, no processing is performed. However, the robot 12 is turned to detect the human 202 (skin color area). It may be.

  In subsequent step S15, it is determined whether or not all tag IDs have been read at the current location (current location). That is, it is determined whether or not all articles (tag IDs) existing around the human 202 have been detected by moving the robot 12, turning it, or moving both hands. If “NO” in the step S15, that is, if all the tag IDs are not read at the current location, the process returns to the step S1 as it is. However, if “YES” in the step S15, that is, if all the tag IDs are read at the current location, the motion candidate pattern is determined based on the one or more tag IDs detected in the step S17 shown in FIG. decide.

  Specifically, the CPU 76 refers to the article data 106a and identifies one or more articles corresponding to the detected tag ID. Subsequently, the CPU 76 refers to the motion candidate data 106b and determines a motion candidate pattern corresponding to the identified article. However, when two or more articles are specified, only the candidate action patterns for each article and overlapping in all articles are determined. For example, when “knife” and “cutting board” are detected as articles, “hold”, “place”, “cut”, and “wash” are determined as motion candidates. That is, motion candidate patterns for “sharpening” that do not overlap in two articles are eliminated.

  Subsequently, in step S19, the similarity between each motion candidate pattern and the captured image is calculated. Here, the CPU 76 calculates the similarity distance between each motion candidate pattern and the captured image. For example, the distance between the marker trajectory attached to the human 202 and the corresponding marker trajectory in each motion candidate pattern is calculated by “dynamic programming”.

  This “Dynamic Programming” is already well-known and is not an essential part of the present invention, so detailed description will be omitted. For example, “Katsuhiko Takahashi” , Seki Susumu, Oka Ryuichi. Spotting recognition of gesture moving images. IEICE Technical Report PRU92-157, pp. 9-16, 3 1993.

  In the subsequent step S21, the operation is specified. Specifically, the motion of the motion candidate pattern having the smallest similarity distance (highest similarity) calculated in step S19 is specified as the motion of the human 202. In the next step S23, the specified operation, time and place are stored. For example, action history table data may be stored in the memory 80 or the database 106, and the operation, time, and place specified may be stored. However, the time and place stored in step S23 are the detection time and detection place stored in step S9. In step S25, the operation identification result is output, and the operation identification process is terminated. For example, in step S25, the action identified from the speaker 72 of the robot 12 can be output by voice. However, if the computer is connected so as to be communicable with the robot 12 and the identification result is transmitted to the computer, the operation identified by the image display device such as a monitor (not shown) connected to the computer is performed. It is also possible to display text or to output an action identified from a speaker (not shown) connected to the computer.

  According to the first embodiment, an article existing in the vicinity of a person is detected, an action candidate is determined from the article, and a person's action is identified from the action candidates. The processing can be greatly reduced as compared with the case of identifying the operation.

  Further, according to the first embodiment, since the operation is identified not only based on the image but also based on the article, it can be accurately identified. That is, since the operation candidates are narrowed down, the identification system can be increased.

In the first embodiment, the recorded action is recorded along with the time and place where the action is detected (photographed) together with the specified action, so that the recorded contents can be used as an action memo about a certain person.
<Second embodiment>
The system 10 of the second embodiment shown in FIG. 9 is the same as the above-described embodiment except that a human motion candidate is detected by a server provided to be able to communicate with the robot 12, and therefore, a duplicate description is not provided. Omitted.

  Referring to FIG. 9, in system 10 of the second embodiment, robot 12 is connected to server 22 via network 20 so as to be communicable. A general-purpose server can be used as the server 22, and a general-purpose personal computer or workstation can be used instead of the server. Further, the network 20 may be constructed by either wired or wireless. In the system 10, a database 24 is connected to the server 22. The article data 106a and the motion candidate data 106b shown in the first embodiment are stored in the database 24, and the server 22 detects motion candidates in response to an inquiry from the robot 12, and the detected motion candidates are stored in the robot 12. Notify (send).

  Since the specific operation identification process of the robot 12 (CPU 76) is almost the same as the operation identification process shown in the flowcharts of FIGS. 7 and 8, only different processes will be described. Since the processing shown in FIG. 7 is the same, the illustration is omitted.

  As shown in FIG. 10, the CPU 76 transmits the detected one or more tag IDs to the server 22 in step S <b> 17 ′. That is, an operation candidate is inquired. Here, although illustration is omitted, when the server 22 receives one or more tag IDs from the robot 12, the server 22 determines an operation candidate pattern based on the tag ID and transmits the determined operation candidate pattern to the robot 12. To do. Specifically, the server 12 refers to the article data 106a and specifies one or more articles corresponding to the tag ID received from the robot 12. Subsequently, the server 22 refers to the motion candidate data 106b and determines a motion candidate pattern corresponding to the identified article. However, when two or more articles are specified, only the candidate action patterns for each article and overlapping in all articles are determined. For example, when “knife” and “cutting board” are detected as articles, “hold”, “place”, “cut”, and “wash” are determined as motion candidates. That is, motion candidate patterns for “sharpening” that do not overlap in two articles are eliminated.

  Returning to FIG. 10, in step S18, it is determined whether an operation candidate pattern has been received. If “NO” in the step S18, that is, if the motion candidate pattern has not been received, the process returns to the same step S18 and waits for the reception of the motion candidate pattern. On the other hand, if “YES” in the step S18, that is, if an operation candidate pattern is received, the similarity between each operation candidate pattern and the photographed image is calculated in a step S19. The subsequent processing is the same as that described with reference to FIG.

According to the second embodiment, human movements can be reliably identified with a small amount of processing, and further, since the motion candidates are detected on the server side, the processing load on the robot can be reduced.
<Third embodiment>
Since the system 10 of the third embodiment is the same as the system 10 of the second embodiment except that the operation of the human 202 is identified on the server 22 side, the duplicated explanation is omitted. Although illustration is omitted, in the system 10 of the third embodiment, the article data 106a, the motion candidate data 106b, and the motion template data 106c are stored in the database 24 connected to the server 22. Therefore, the database 106 provided in the robot 12 can be deleted.

  Specifically, in the system 10 of the third embodiment, the robot 12 detects the tag ID of an article possessed by the human 202 or present in the vicinity of the human 202, and also displays a captured image of the human 202 at that time. get. Then, the detected tag ID and captured image are transmitted to the server 22. As described in the second embodiment, the server 22 detects motion candidate patterns based on the tag ID, calculates the similarity between each detected motion candidate pattern and the captured image, and the captured image is The action of the human 202 shown is specified.

  Therefore, in the action identification process shown in FIGS. 7 and 10, the process from step S1 to step S17 ′ is executed on the robot 12 side, and the process from step S18 to step S25 is executed on the server 22 side. That's fine. However, in step S17 ′, the detected tag ID and captured image are transmitted to the server 22.

  Although detailed description is omitted, since the time and place stored in step S23 are the detection time and place stored in step S9 as described in the first embodiment, in this third embodiment, In step S17 ′, the detection time and detection location stored in step S9 are also transmitted to the server 22.

  Also in the third embodiment, it is possible to reliably identify human movements with a small amount of processing, and furthermore, since the movement identification processing is executed on the server side, the processing load on the robot can be reduced.

  In each of the above-described embodiments, a passive type tag is used. However, as another embodiment, an active type tag (such as an infrared tag) can be used. In such a case, for example, an infrared tag is provided on each article, while two cameras (infrared cameras (sensors)) are provided on the head of the robot 12 (above the eye camera 54). Then, the distance between the robot 12 and the article or the position (three-dimensional position) of the article viewed from the robot 12 is measured (calculated) by performing triangulation from the detection results (captured images) of the two cameras. Similarly, by performing triangulation from the detection result (captured image) of the eye camera 54, the distance between the robot 12 and the human 202 or the three-dimensional position of the human 202 viewed from the robot 12 is calculated. Thereby, the distance (position) between the person 202 and the article can be measured. That is, an article existing in the vicinity of the person 202 can be detected. The processing after the article is detected is the same as that in the above-described embodiment.

FIG. 1 is an illustrative view showing one example of an action identification system using the communication robot of the present invention. FIG. 2 is an illustrative view for explaining the appearance of the robot shown in FIG. 1 embodiment. FIG. 3 is an illustrative view showing an electrical configuration of the robot shown in FIGS. 1 and 2. FIG. 4 is an illustrative view showing an example of article data and motion candidate data stored in a database built in the robot. FIG. 5 is an illustrative view showing an example of operation template data stored in a database built in the robot. FIG. 6 is an illustrative view showing an application example of the system shown in FIG. FIG. 7 is a flowchart showing a part of the operation identification process of the CPU shown in FIG. FIG. 8 is another part of the CPU operation identification process shown in FIG. 3, and is a flowchart subsequent to the flowchart of FIG. FIG. 9 is an illustrative view showing another example of the motion identification system using the communication robot of the present invention. FIG. 10 is a flowchart showing a part of the operation identification processing of another embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Motion identification system using communication robot 12 ... Communication robot 14 ... Tag 16 ... Microphone 20 ... Network 22 ... Server 24, 106 ... Database 38 ... Collision sensor 42 ... Ultrasonic distance sensor 52 ... Omnidirectional camera 54 ... Eye camera 64 ... touch sensor 76 ... CPU
DESCRIPTION OF SYMBOLS 80 ... Memory 82 ... Motor control board 84 ... Sensor input / output board 86 ... Audio | voice input / output board 88-96 ... Motor 98 ... Communication LAN board 100 ... Wireless communication apparatus 102,104 ... Tag reader

Claims (4)

Photography means for photographing humans,
Identification information detecting means for detecting identification information about an object existing in the vicinity of the human,
A motion prediction unit that predicts the human motion based on the identification information detected by the identification information detection unit; and the human actual motion and the motion prediction unit indicated by the captured image captured by the imaging unit. A communication robot provided with motion specifying means for specifying the actual motion from the predicted motion. A degree of similarity calculating means for calculating the degree of similarity between the human actual motion indicated by the captured image and each of the predicted motions;
The communication robot according to claim 1, wherein the action specifying unit specifies the predicted action having the highest similarity calculated by the similarity calculation unit as the actual action. An operation identification system comprising a communication robot and a server provided to be communicable with the communication robot,
The communication robot is
Photography means for photographing humans,
Identification information detecting means for detecting identification information about an object existing in the vicinity of the human,
Identification information transmitting means for transmitting the detected identification information to the server;
Predicted motion information receiving means for receiving the predicted motion information of the human from the server; and the actual motion of the human indicated by the captured image captured by the capturing means and the predicted motion information received by the predicted motion information receiving means; Comprises an action specifying means for specifying the actual action from
The server
An identification information receiving means for receiving the identification information transmitted to the identification information transmitting means; and a prediction for transmitting the human predicted motion information acquired based on the identification information received by the identification information receiving means to the communication robot An operation identification system comprising operation information transmitting means. An operation identification system comprising a communication robot and a server provided to be communicable with the communication robot,
The communication robot is
Photography means for photographing humans,
An identification information detecting means for detecting identification information about an object existing in the vicinity of the human, and a transmission for transmitting to the server a photographed image photographed by the photographing means and the identification information detected by the identification information detecting means With means,
The server
Receiving means for receiving the captured image and identification information transmitted to the transmitting means;
Motion predicting means for predicting the human motion based on the identification information received by the receiving means;
An action identification system comprising action specifying means for specifying the actual action from the actual human action indicated by the captured image received by the identification information receiving means and the predicted action predicted by the action predicting means.

Images