JP5145569B2 - Object identification method and apparatus - Google Patents

Description

  The present invention relates to an object identification method and apparatus, and in particular, for example, an article or an object (hereinafter referred to as an “object”) designated by a person while a communication robot communicates with the person is identified and transported. The present invention relates to a method and apparatus for identifying an object and a communication robot including the same.

Patent Document 1 discloses an object specifying device that specifies an object that exists in a direction in which a human points. According to the apparatus shown in Patent Document 1, it is possible to specify an object pointed by a human.
JP 2007-80060 [G06F 3/038 G01C 21/00 G08G 1/0969 G09B 29/00 G09B 29/10]

  In the technique of Patent Document 1, since the object is specified based only on the pointing direction, for example, the direction in which a human is pointing is not always directed toward the object, and therefore the object is specified incorrectly. there's a possibility that.

  Therefore, a main object of the present invention is to provide a novel object identification method and apparatus.

  Another object of the present invention is to provide an object specifying method and apparatus capable of accurately specifying an object.

  The present invention employs the following configuration in order to solve the above problems. The reference numerals in parentheses, supplementary explanations, and the like indicate the corresponding relationship with the embodiments described in order to help understanding of the present invention, and do not limit the present invention.

The first invention among the articles that are in the same spatial coordinate system between humans, an article that human is directed to pointing and gaze, a method of identifying a target object, each to be retrieved article Register necessary information such as name, attribute, and position in the search dictionary, and (a) the distance between the pointing straight line indicating the pointing direction and each item registered in the search dictionary at each repetition time And create a pointing direction confidence table by finding the article with the shortest distance, and (b) for each repetition time, the line of sight indicating the line of sight and each article registered in the search dictionary. Create a gaze direction confidence table by calculating the distance and finding the item with the shortest distance, and (c) identify the object by simultaneously referring to the pointing direction confidence table and the gaze direction confidence table This is an object identification method.

The second invention is dependent on the first invention, and step (a) comprises: (a1) an article having the smallest distance between the fingertip and the first pointing straight line passing through the center of the face for each repetition time. And (a2) estimating an article having a minimum distance between the fingertip and the second pointing straight line passing through the elbow at each repetition time, and step (b) includes (b1) A step of estimating an article having a minimum distance from the first line of sight indicating the eye vector of one eye for each time; and (b2) a second line of sight indicating the line of sight of the other eye at each repeated time. And a step (a) to create a pointing direction confidence table based on the results of step (a1) and step (a2), and in step (b), to create a line-of-sight direction confidence table based on the result of the step (b1) and step (b2), the object JP It is a method.

  For example, the first and second pointing straight lines are estimated by motion capture, and the distance between these straight lines and each article in the vicinity of the human being is calculated to calculate the “confidence for each article for each line. Evaluate "degree". Similarly, the distance between each gaze direction straight line of each human eye and each article is calculated, and the certainty factor for each article for each gaze is evaluated.

  Such a certainty factor is evaluated at each repetition time, and an article that has acquired the most certainty factor is specified as an object.

A third invention is, among the articles that are in the same spatial coordinate system between humans, an article that human is directed to pointing and gaze, an apparatus for identifying a target object, each to be retrieved article Search dictionary for registering necessary information such as name, attribute, position, etc., and for each repetition time, calculate the distance between the pointing straight line indicating the pointing direction and each article registered in the search dictionary, and the shortest distance A pointing direction confidence table creating means for creating a pointing direction confidence table by obtaining an article having a distance between a line of sight indicating a viewing direction and each article registered in the search dictionary for each repetition time And a gaze direction certainty table creation means for creating a gaze direction certainty table by obtaining an article having the shortest distance, and a target object by simultaneously referring to the pointing direction certainty table and the gaze direction certainty table Comprising means for identifying A elephant-specifying device.

  4th invention is a communication robot provided with the instruction | indication means which instruct | indicates the target object specified with the target object specifying apparatus according to 3rd invention, and the target object specifying apparatus.

  A fifth invention is a communication robot according to the fourth invention, further comprising means for carrying the object specified by the object specifying device.

  According to the present invention, since the object is specified by simultaneously referring to the human gaze direction and the pointing direction, the object can be specified accurately.

  The above object, other objects, features, and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

  Referring to FIG. 1, a communication robot system 10 of this embodiment includes a communication robot (hereinafter simply referred to as “robot”) 12. The robot 12 can access a network 14 such as a wireless LAN. The robot 12 specifies an object designated by the human 16 and performs an operation such as taking the object to the human 16.

  The human 16 is wearing a wireless tag 18 indicating who the person is, and a marker for motion capture is attached, although not shown. The markers are typically set on the top of the human head, both shoulders, both elbows, the tip of the index finger of both hands, etc., and these markers together with the entire human 16 are controlled by the camera 22 controlled by the server 20. Taken. In the embodiment, three cameras 22 are provided, take a picture of the person 16 from three directions, and supply the camera video to the server 20.

  The server 20 is coupled to a network 14 such as a wireless LAN, and executes a motion capture process that detects the movement of the marker based on the camera video data input as described above, and also detects a skin color area, for example. Thus, the position of the face of the human 16 can be specified.

  In the system 10, as described above, the robot 12 specifies an article designated by the human 16 as an object. In this embodiment, a book (book) is used as an example of an article that can be an object. A radio tag 18 indicating what kind of book the book is attached to the book 24. The book 24 is stored in a bookshelf 26.

  However, the articles that can be the object are not only the books of the embodiment, but any household article can be considered if it is a home system. Naturally, it can be used not only for home use but also in any place (company, office, factory, etc.) that works with people.

  All articles targeted by the communication robot system 10 are registered in the article dictionary 28 attached to the server 20. The article dictionary 28 will be described later.

  In FIG. 1, one robot 12 is shown for simplicity, but two or more robots may be used. Further, the number of humans does not have to be limited to one, and can be identified by the wireless tag 18 and may be plural.

  In the embodiment shown in FIG. 1, the positions of the robot 12, the human 16, the article 24, and the like are expressed using the world coordinates of the space where the system 10 is installed. Since it is performed in coordinates, details will not be described, but the robot 12 performs a coordinate conversion process between the robot coordinates and the world coordinates as necessary in the process described later.

  The hardware configuration of the robot 12 will be described with reference to FIG. FIG. 2 is a front view showing the appearance of the robot 12 of this embodiment. The robot 12 includes a carriage 30, and two wheels 32 and one slave wheel 34 for autonomously moving the robot 12 are provided on the lower surface of the carriage 30. The two wheels 32 are independently driven by a wheel motor 36 (see FIG. 3), and the carriage 30, that is, the robot 12 can be moved in any direction, front, back, left, and right. The slave wheel 34 is an auxiliary wheel that assists the wheel 32. Therefore, the robot 12 can move in the arranged space by autonomous control.

  A cylindrical sensor attachment panel 38 is provided on the carriage 30, and a large number of infrared distance sensors 40 are attached to the sensor attachment panel 38. These infrared distance sensors 40 measure the distance to the sensor mounting panel 38, that is, the object (human being, obstacle, etc.) around the robot 12.

  In this embodiment, an infrared distance sensor is used as the distance sensor, but an ultrasonic distance sensor, a millimeter wave radar, or the like can be used instead of the infrared distance sensor.

  A body 42 is provided on the sensor mounting panel 38 so as to stand upright. Further, the above-described infrared distance sensor 40 is further provided in the upper front upper portion of the body 42 (a position corresponding to a human chest), and measures the distance mainly to a human in front of the robot 12. Further, the body 42 is provided with a support column 44 extending from substantially the center of the upper end of the side surface, and an omnidirectional camera 46 is provided on the support column 44. The omnidirectional camera 46 photographs the surroundings of the robot 12 and is distinguished from an eye camera 70 described later. As this omnidirectional camera 46, for example, a camera using a solid-state imaging device such as a CCD or a CMOS can be adopted. In addition, the installation positions of the infrared distance sensor 40 and the omnidirectional camera 46 are not limited to the portions, and can be changed as appropriate.

  An upper arm 50R and an upper arm 50L are provided at upper end portions on both sides of the torso 42 (position corresponding to a human shoulder) by a shoulder joint 48R and a shoulder joint 48L, respectively. Although illustration is omitted, each of the shoulder joint 48R and the shoulder joint 48L has three orthogonal degrees of freedom. That is, the shoulder joint 48R can control the angle of the upper arm 50R around each of three orthogonal axes. A certain axis (yaw axis) of the shoulder joint 48R is an axis parallel to the longitudinal direction (or axis) of the upper arm 50R, and the other two axes (pitch axis and roll axis) are orthogonal to the axes from different directions. It is an axis to do. Similarly, the shoulder joint 48L can control the angle of the upper arm 50L around each of three orthogonal axes. A certain axis (yaw axis) of the shoulder joint 48L is an axis parallel to the longitudinal direction (or axis) of the upper arm 50L, and the other two axes (pitch axis and roll axis) are orthogonal to the axes from different directions. It is an axis to do.

  In addition, an elbow joint 52R and an elbow joint 52L are provided at the respective distal ends of the upper arm 50R and the upper arm 50L. Although illustration is omitted, each of the elbow joint 52R and the elbow joint 52L has one degree of freedom, and the angle of the forearm 54R and the forearm 54L can be controlled around the axis (pitch axis).

  At the tip of each of the forearm 54R and the forearm 54L, a hand 56R and a hand 56L corresponding to a human hand are provided. Although the detailed illustration is omitted, these hands 56R and 56L are configured to be openable and closable so that the robot 12 can grip or hold an object using the hands 56R and 56L. However, the shape of the hands 56R and 56L is not limited to the shape of the embodiment, and may have a shape and a function very similar to a human hand.

  Although not shown, the front surface of the carriage 30, the portion corresponding to the shoulder including the shoulder joint 48R and the shoulder joint 48L, the upper arm 50R, the upper arm 50L, the forearm 54R, the forearm 54L, the sphere 56R, and the sphere 56L, A contact sensor 58 (shown generically in FIG. 3) is provided. A contact sensor 58 on the front surface of the carriage 30 detects contact of a person or another obstacle with the carriage 30. Therefore, the robot 12 can detect the contact with the obstacle during its movement and immediately stop the driving of the wheel 32 to suddenly stop the movement of the robot 12. Further, the other contact sensors 58 detect whether or not the respective parts are touched. In addition, the installation position of the contact sensor 58 is not limited to the said site | part, and may be provided in an appropriate position (position corresponding to a person's chest, abdomen, side, back, and waist).

  A neck joint 60 is provided at the upper center of the body 42 (a position corresponding to a person's neck), and a head 62 is further provided thereon. Although illustration is omitted, the neck joint 60 has a degree of freedom of three axes, and the angle can be controlled around each of the three axes. A certain axis (yaw axis) is an axis directed directly above (vertically upward) of the robot 12, and the other two axes (pitch axis and roll axis) are axes orthogonal to each other in different directions.

  The head 62 is provided with a speaker 64 at a position corresponding to a human mouth. The speaker 64 is used for the robot 12 to communicate with humans around it by voice or sound. A microphone 66R and a microphone 66L are provided at a position corresponding to a human ear. Hereinafter, the right microphone 66R and the left microphone 66L may be collectively referred to as a microphone 66. The microphone 66 captures ambient sounds, in particular, the voices of humans who are subjects of communication. Furthermore, an eyeball part 68R and an eyeball part 68L are provided at positions corresponding to human eyes. The eyeball portion 68R and the eyeball portion 68L include an eye camera 70R and an eye camera 70L, respectively. Hereinafter, the right eyeball part 68R and the left eyeball part 68L may be collectively referred to as the eyeball part 68. The right eye camera 70R and the left eye camera 70L may be collectively referred to as an eye camera 70.

  The eye camera 70 captures a human face approaching the robot 12, other parts or objects, and captures a corresponding video signal. In this embodiment, the robot 12 detects the line-of-sight directions (vectors) of the left and right eyes of the human 16 from the video signal from the eye camera 70. Specifically, the line-of-sight detection method is disclosed in Japanese Patent Application Laid-Open No. 2004-255074 as using two cameras, and Japanese Patent Application Laid-Open No. 2006-172209 and Japanese Patent Application Laid-Open No. 2006-285531 as using one camera. However, the details are not important here, so only those publications are cited.

  However, a well-known eye mark recorder or the like may be used for detecting the line-of-sight vector of the human 16.

  The eye camera 70 can be the same camera as the omnidirectional camera 46 described above. For example, the eye camera 70 is fixed in the eyeball unit 68, and the eyeball unit 68 is attached to a predetermined position in the head 62 via an eyeball support unit (not shown). Although illustration is omitted, the eyeball support portion has two degrees of freedom, and the angle can be controlled around each of these axes. For example, one of the two axes is an axis (yaw axis) in a direction toward the top of the head 62, and the other is orthogonal to the one axis and goes straight in a direction in which the front side (face) of the head 62 faces. It is an axis (pitch axis) in the direction to be performed. By rotating the eyeball support portion around each of these two axes, the tip (front) side of the eyeball portion 68 or the eye camera 70 is displaced, and the camera axis, that is, the line-of-sight direction is moved. Note that the installation positions of the speaker 64, the microphone 66, and the eye camera 70 described above are not limited to those portions, and may be provided at appropriate positions.

  As described above, the robot 12 of this embodiment includes independent two-axis driving of the wheels 32, three degrees of freedom of the shoulder joint 48 (6 degrees of freedom on the left and right), one degree of freedom of the elbow joint 52 (2 degrees of freedom on the left and right), It has a total of 17 degrees of freedom, 3 degrees of freedom for the neck joint 60 and 2 degrees of freedom for the eyeball support (4 degrees of freedom on the left and right).

  FIG. 3 is a block diagram showing the electrical configuration of the robot 12. With reference to FIG. 3, the robot 12 includes a CPU 80. The CPU 80 is also called a microcomputer or a processor, and is connected to the memory 84, the motor control board 86, the sensor input / output board 88 and the audio input / output board 90 via the bus 82.

  The memory 84 includes a ROM, an HDD, and a RAM (not shown). In the ROM and the HDD, a control program for controlling the operation of the robot 12 is stored in advance. For example, a detection program for detecting the output (sensor information) of each sensor, a communication program for transmitting / receiving necessary data and commands to / from external computers (such as the central control device 14 and the operation terminal 16), etc. To be recorded. The RAM is used as a work memory or a buffer memory.

  Further, in this embodiment, the robot 12 is configured to be able to speak and make a gesture for communication with the human 16, but the memory 84 has a speech / gesture for such a speech or gesture. A dictionary 85A is set.

  In addition, a search dictionary 85B is set in the memory 84, and this search dictionary 85B is used for the robot 12 to identify an article (in the embodiment, a book) designated by the human 16 as an object. A dictionary in which only articles (books) existing in the vicinity are extracted from the article dictionary 28 and registered, and can be dynamically rewritten according to a change in the position of the person 16.

  The motor control board 86 is constituted by, for example, a DSP, and controls driving of each axis motor such as each arm, neck joint, and eyeball unit. That is, the motor control board 86 receives control data from the CPU 80, and controls two motors (collectively indicated as “right eyeball motor 92” in FIG. 3) that control the angles of the two axes of the right eyeball portion 68R. Control the rotation angle. Similarly, the motor control board 86 receives control data from the CPU 80, and controls two angles of the two axes of the left eyeball portion 68L (in FIG. 3, collectively referred to as “left eyeball motor 94”). ) To control the rotation angle.

  The motor control board 86 receives control data from the CPU 80, and includes a total of four motors including three motors for controlling the angles of the three orthogonal axes of the shoulder joint 48R and one motor for controlling the angle of the elbow joint 52R. The rotation angle of two motors (collectively indicated as “right arm motor 96” in FIG. 3) is controlled. Similarly, the motor control board 86 receives control data from the CPU 80, and includes three motors for controlling the angles of the three orthogonal axes of the shoulder joint 48L and one motor for controlling the angle of the elbow joint 52L. The rotation angles of a total of four motors (collectively indicated as “left arm motor 98” in FIG. 3) are controlled.

  Further, the motor control board 86 receives control data from the CPU 80, and controls three motors that control the angles of the three orthogonal axes of the neck joint 60 (in FIG. 3, collectively indicated as “head motor 100”). Control the rotation angle. The motor control board 86 receives control data from the CPU 80 and controls the rotation angles of the two motors (collectively indicated as “wheel motor 36” in FIG. 3) that drive the wheels 32.

  A hand actuator 108 is further coupled to the motor control board 86, and the motor control board 86 receives control data from the CPU 80 and controls the opening and closing of the hands 56R and 56L.

  In this embodiment, a motor other than the wheel motor 36 uses a stepping motor (that is, a pulse motor) in order to simplify the control. However, a DC motor may be used similarly to the wheel motor 36. The actuator that drives the body part of the robot 12 is not limited to a motor that uses a current as a power source, and may be changed as appropriate. For example, in another embodiment, an air actuator may be applied.

  Similar to the motor control board 86, the sensor input / output board 88 is configured by a DSP and takes in signals from each sensor and gives them to the CPU 80. That is, data relating to the reflection time from each of the infrared distance sensors 40 is input to the CPU 80 through the sensor input / output board 88. The video signal from the omnidirectional camera 46 is input to the CPU 80 after being subjected to predetermined processing by the sensor input / output board 88 as necessary. Similarly, the video signal from the eye camera 70 is also input to the CPU 80. Further, signals from the plurality of contact sensors 58 described above (collectively indicated as “contact sensors 58” in FIG. 3) are provided to the CPU 80 via the sensor input / output board 88. Similarly, the voice input / output board 90 is also configured by a DSP, and voice or voice in accordance with voice synthesis data provided from the CPU 80 is output from the speaker 64. In addition, voice input from the microphone 66 is given to the CPU 80 via the voice input / output board 90.

  The CPU 80 is connected to the communication LAN board 102 via the bus 82. The communication LAN board 102 is configured by a DSP, for example, and provides transmission data provided from the CPU 80 to the wireless communication device 104, and the wireless communication device 104 transmits the transmission data to the server 20 via the network 14. In addition, the communication LAN board 102 receives data via the wireless communication device 104 and provides the received data to the CPU 80. For example, the transmission data may be a signal (command) from the robot 12 to the server 20, or operation history information (history data) regarding communication performed by the robot 12. The reason why the history data is transmitted as well as the command is to reduce the capacity of the memory 84 and to reduce power consumption. In this embodiment, the history data is transmitted to the server 20 every time communication is performed. However, the history data may be transmitted to the server 20 in units of a fixed time or a fixed amount.

  Further, the wireless tag reader 106 is connected to the CPU 80 via the bus 82. The wireless tag reader 106 receives a radio wave superimposed with identification information transmitted from the wireless tag 18 (RFID tag) via an antenna (not shown). Then, the RFID tag reader 106 amplifies the received radio wave signal, separates the identification signal from the radio wave signal, demodulates (decodes) the identification information, and supplies the identification information to the CPU 80. According to FIG. 1, the wireless tag 18 is attached to the person 16 in the reception of the company where the robot 12 is disposed or in the living room of a general household, and the wireless tag reader 106 detects the wireless tag 18 within the communicable range. To do. Note that the wireless tag 18 may be an active type or a passive type that is driven according to a radio wave transmitted from the wireless tag reader 106.

  Next, the article dictionary 28 will be described with reference to FIG. The article dictionary 28 shown in FIG. 4 assigns an ID such as a Ucode to one of the articles, and stores necessary information such as the name, attribute, and position (coordinates) for each article as a character string. Registered as. The U-code is specifically a 128-bit number, and can individually identify articles that are 1 trillion times more than 1 trillion times 340 trillion. However, the ID used for the article dictionary 28 does not necessarily need to be such a U-code, and may consist of a combination of appropriate numbers and symbols.

  Such an article dictionary 28 registers all articles, for example, household articles, which are objects to be identified by the robot, with an ID and a character string, and is equivalent to a global dictionary.

  When the robot 12 specifies an object in accordance with communication with the human 16, the robot 12 registers only the articles existing in the vicinity of the human 16 (within a predetermined distance range from the human 16), and the search illustrated in FIG. 5. A dictionary 85B is created. The search dictionary 85B functions as a local dictionary because it is registered only for articles within a predetermined distance range from the human 16 as described above. The search dictionary 85B is considered to be dynamically rewritten because an article existing in the vicinity of the person 16 changes in accordance with the change in the position of the person 16. The registered contents of the article dictionary 85B may be extracted from the article dictionary 28.

  Next, the operation of the robot 12 in the embodiment shown in FIG. 1 will be described with reference to the flowchart shown in FIG.

  In the first step S1 of FIG. 6, the CPU 80 (FIG. 3) of the robot 12 determines whether or not the person 16 (FIG. 1) has been recognized in accordance with the sensor input from the sensor input / output board 88 also illustrated in FIG. . Specifically, for example, when a human body is detected by the infrared sensor 40 and the wireless tag reader 106 recognizes the wireless tag 18 worn by the user 16 at that time, it is determined that the human (user) 16 has been recognized.

  When the user (human 16) is recognized in step S1, the CPU 80 of the robot 12 in the next step S3, as described above, is an article within a predetermined distance range from the user 16, such as a book in the example of FIG. A search dictionary 85B including an ID and a character string indicating 24 individually is created in the memory 84 by extracting the contents from the article dictionary 28.

  In the next step S5, the CPU 80 uses the utterance / gesture dictionary 85A set in the memory 84 to make an utterance such as “Would you like to bring some books?” From the speaker 64. Make it. Thereafter, if the user utters, for example, “bring it in”, the CPU 80 confirms the user's instruction and determines “YES” in step S7. At this time, the user 16 utters “bring” and indicates the book he / she wants to bring by pointing to the corresponding book.

  When the user's instruction is confirmed in step S7, the CPU 80 increments the counter 85C set in the memory 84 in the next step S9. In the initial state, “1” is set in the counter. The counter 85C counts the number of articles existing in the vicinity of the user 16, and functions as a pointer for the search dictionary 85B (FIG. 5). Therefore, different articles are designated in the search dictionary 85B according to the count value of the counter 85C. It should be understood that the following operations are performed for each article until the count value of the counter 85C is equal to the number of articles “n” listed in the search dictionary 85B.

  Subsequent to step S <b> 9, the CPU 80 executes in parallel the operation of estimating the user's (human 16) line of sight and obtaining the certainty thereof, and the operation of estimating the pointing direction and obtaining the certainty thereof. However, here, for the sake of convenience, description will be made in the order of first estimating the line of sight and then estimating the pointing direction.

  6 are executed at regular repetition times (t1, t2, t3,..., Tn) within the utterance time T in step S5. In the embodiment, 50 Hz ( The time T is set to 0.5-1 seconds.

  In step S <b> 11, the CPU 80 processes the camera video from the eye camera 70, for example, thereby estimating the eye gaze vectors of the left and right eyes of the user 16 according to any of the methods described in the above publications. . The line-of-sight directions of the left and right eyes are indicated by straight lines L1 and L2 in FIG. After estimating each line of sight L1 and L2 in this way, in the next step S13, the CPU 80 determines the distance between the line of sight that the counter 85C designates in the search dictionary 85B and the line of sight L1 and L2. calculate.

  In order to estimate the pointing direction, in step S <b> 17, first, the CPU 80 specifies the arm on which the human 16 has performed the pointing operation. Specifically, referring to the motion capture data, for example, the arm on the side where the difference between the fingertip and shoulder height of the human 16 is small is estimated as the pointing arm. This is because when the pointing operation is performed, the arm lifting operation will be performed first. Thus, after estimating which arm is used to perform the pointing operation in step S17, the CPU 80 estimates the pointing direction in the next step S19.

  In this embodiment, as shown in FIG. 7, a straight line L3 passing through the fingertip of the pointing arm and the center (center of gravity) of the face and a straight line L4 passing through the fingertip of the pointing arm and the elbow of the arm are assumed. Then, the straight lines L3 and L4 are estimated with reference to the motion capture data. In the next step S21, the distance between each straight line L3 and L4 and each article is calculated.

  Steps S11-S13 and S17-S21 described above are performed at each repetition time within the utterance time T. Then, for each repetition time (t1, t2, t3,..., Tn), an article having a minimum distance from the lines L1, L2, L3, and L4 is obtained. In each line, a high certainty factor (“◯” in FIG. 8) is assigned to the minimized article. In this way, for example, a certainty factor table as shown in FIG. 8 is created.

  If the certainty factor table is created by calculating the article having the shortest distance for each straight line as described above, the certainty factor (○) may be given to two or more straight lines for one article. As a result, an article list as will be described later in step S25 can be created.

  In the reliability table of FIG. 8, the reliability evaluated for each of the lines of sight L1 and L2 can be referred to as “line of sight reliability”, and the reliability evaluated for each of the pointing direction lines L3 and L4 is “finger”. It can be said that it is the “direction direction certainty”.

  Referring to the example shown in FIG. 8, for an article having an ID of “123... 000001”, in the example, a book named “global warming” shown in FIG. Although the distance to the book is minimized, it can be determined that no line is closest to the book in other time intervals. For the next article having an ID of “123... 000035”, in the example, the magazine named “Camera” shown in FIG. 1, any line is present at each time except for time t1. It can be seen that it has approached again. In this way, the certainty factor table shown in FIG. 8 is created in steps S13 and S21.

  In step S <b> 25, the CPU 80 refers to the certainty factor table shown in FIG. 8, and specifies an object that the human 16 (user) thinks instructed at that time. Specifically, the number of times that ○ is included in both the line of sight (L1 or L2) and the pointing (L3 or L4) in the order in which the confidence evaluations (circles in FIG. 8) are simply large or in repetition time. An article list is created according to the descending order, and the highest item is first identified as an object. This provides a variety of confidence assessments.

  This reliability evaluation will be described with reference to the example shown in FIG. 8, for example, if the book named “global warming” having the ID “123. And the certainty factor of the magazine “camera” having the ID “123... 000035” is “3”. Therefore, in this case, ID “123... 000035” and ID “123. Therefore, first, a magazine “camera” (ID “123... 000035”) is specified as an object.

  However, when the number of certainty degrees (circles) is the same, or when the number of certainty degrees (circles) is smaller than a predetermined threshold, for example, each repetition shown in FIG. What is necessary is just to specify the articles | goods to which confidence is provided in the half or more of all the sections of time as a target object.

  After specifying the object in step S25, the CPU 80 performs utterance and pointing operation in the next step S27 with reference to the utterance / gesture dictionary 85A so as to indicate the object specified in S25. When it is specified that the object is a magazine titled “Camera”, in this step S27, the book placed third from the right in FIG. Talk like "

  Thereafter, the CPU 80 performs voice recognition processing on the voice of the user (human 16) input through the microphone 66. Then, in step S31, the user determines whether the object identified by the robot 12 has been affirmed or denied, and if not, the process proceeds to the next step S33, and the unspecified article in the article list in step S25. To determine if is still left. If “YES” is determined in the step S33, the process returns to the step S25 to repeat the process. However, if “NO” is determined, the process is ended because it is not possible to specify the object.

  When the positive voice of the user 16 is recognized in step S31, the robot 12 moves in the direction of the corresponding object, holds the corresponding object, and carries it to the position of the user 16. That is, since the coordinates of the position where the object exists are already known, the CPU 80 of the robot 12 controls the wheel motor 36 to move the robot 12 to the position of the object, and then the actuator 108 (FIG. 3). By controlling the hand 56R (or 56L), the object is gripped by the hand 56R (or 56L: FIG. 2), and in this state, the wheel motor 36 is controlled again to bring the robot 12 to the position of the user 16. Move. In this way, the object specified by the robot 12 in step S25 can be carried to the user 16 in step S35.

  Thus, in the above-described embodiment, the line of sight L1 and L2 and the pointing direction lines L3 and L4 are estimated and the distance to each article is calculated, and then the determination is made based on the certainty factor. Even if the instruction is ambiguous or fluctuates from time to time, the object can be specified fairly accurately.

  However, for example, when it can be estimated that the user's lines of sight L1 and L2 are directed to the robot 12 and are not clearly directed to the object, the pointing direction straight line is not estimated for the lines of sight L1 and L2. Only L3 and L4 may be estimated.

  In addition, although individual explanation is omitted in the above-described embodiment, in order to define each line L1-L4, specify the position of an article or a person, or calculate the distance between each line and each article. All the coordinates of the world coordinate system are used. Therefore, when necessary, the robot 12 performs coordinate conversion with the robot coordinate system.

  Furthermore, although the Example which applied the target object identification method and apparatus of this invention to the communication robot system was demonstrated, it cannot be overemphasized that it can apply also to uses other than a communication robot.

FIG. 1 is an illustrative view showing an outline of a communication robot system showing an embodiment of the present invention. FIG. 2 is an illustrative view showing the appearance of the robot shown in FIG. 1 from the front. FIG. 3 is a block diagram showing an electrical configuration of the robot shown in FIG. FIG. 4 is an illustrative view showing one example of an article dictionary used in the embodiment of FIG. FIG. 5 is an illustrative view showing one example of a search dictionary used in the embodiment of FIG. FIG. 6 is a flowchart showing the operation of the robot in the embodiment of FIG. FIG. 7 is an illustrative view showing a user's (human) line of sight and pointing direction. FIG. 8 is an illustrative view showing one example of a certainty factor table used in the embodiment of FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Communication robot system 12 ... Communication robot 14 ... Network 18 ... Wireless tag 20 ... Server 22 ... Camera 24 ... Goods (book)
80 ... CPU

Claims (5)

Among the articles that are in the same spatial coordinate system between humans, an article that human is directed to pointing and gaze, a method of identifying a target object,
Register necessary information such as name, attribute, and position for each article to be searched in the search dictionary,
(a) At each repetition time, the distance between the pointing straight line indicating the pointing direction and each article registered in the search dictionary is calculated, and the pointing direction confidence table is obtained by obtaining the article having the shortest distance. Create
(b) For each repetition time, calculate the distance between the line of sight indicating the line-of-sight direction and each article registered in the search dictionary, and create a line-of-sight confidence table by finding the article with the shortest distance. And
(c) An object specifying method for specifying an object by simultaneously referring to the pointing direction certainty table and the gaze direction certainty table. The step (a) includes: (a1) estimating an article having a minimum distance between the fingertip and the first pointing straight line passing through the center of the face at every repetition time; and (a2) at every repetition time. Estimating the article having the smallest distance between the fingertip and the second pointing straight line passing through the elbow,
The step (b) includes (b1) a step of estimating an article having a minimum distance from the first line of sight indicating the line-of-sight vector of one eye for each repetition time, and (b2) for each repetition time. Estimating an article having a minimum distance from a second line of sight indicating a line-of-sight vector of the other eye,
In the step (a), based on the results of the step (a1) and the step (a2), create the pointing direction confidence table,
2. The object specifying method according to claim 1, wherein in the step (b), the line-of-sight direction certainty table is created based on the results of the step (b1) and the step (b2). Among the articles that are in the same spatial coordinate system between humans, an article that human is directed to pointing and gaze, an apparatus for identifying a target object,
A search dictionary that registers necessary information such as name, attribute, and position for each article to be searched,
For each repetition time, the distance between the pointing line indicating the pointing direction and each article registered in the search dictionary is calculated, and the pointing direction confidence table is created by obtaining the article having the shortest distance. Pointing direction confidence table creation means,
For each repetition time, calculate the distance between the line of sight indicating the line of sight and each item registered in the search dictionary, and create the line of sight direction confidence table by finding the item with the shortest distance An object specifying device comprising: a degree table creating means; and means for simultaneously specifying the object with reference to the pointing direction reliability table and the line-of-sight direction reliability table. A communication robot,
A communication robot comprising: the object specifying device according to claim 3; and an instruction unit that indicates an object specified by the object specifying device.   The communication robot according to claim 4, further comprising means for carrying an object specified by the object specifying device.

Images