JP2003266348A - Robot device and control method therefor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a robot device enabling tracking by merely specifying for what it sees with attention in a motion command generation part (high-order module). <P>SOLUTION: A short period storage part 75 is an object for holding the information related to external environment of the robot device 1 during a comparatively short period and receives result of sound (voice) detection from a sound detection part 72, result of color detection from a color detection part 71, result of face detection from a face detection part 73, and a sensor output of a joint angle from a joint angle detection part 74. It integrates a plurality of these detection information by keeping coordination timely and spatially, handles these information as integrated information having meaning, and holds these information during a comparatively short time, for example, during 15 seconds. These integrated information is delivered to a motion command part 76 and a neck part motion generation part 77. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a robot apparatus and a control method thereof for autonomously performing realistic communication, and more particularly to recognizing external information such as images and voices and responding to the information. The present invention relates to an autonomous robot apparatus having a function of reflecting its own behavior and a control method thereof.

[0002]

2. Description of the Related Art A mechanical device that makes a motion similar to a human motion by using an electrical or magnetic action is called a "robot". The origin of the robot is the Slavic word "ROBO".
It is said that it is derived from "TA (slave machine)." In Japan, robots began to be popular since the end of the 1960s, but most of them are automation of production work in factories.
It was an industrial robot such as a manipulator and a transfer robot for the purpose of unmanned operation.

Recently, a pet type robot imitating the body mechanism of a four-legged animal such as dogs, cats, and bears and its movements, or the body mechanism of an animal that performs two-legged upright walking, such as a human or a monkey, has been used. "Humanoid" that imitates movement
Alternatively, research and development on the structure of legged mobile robots such as "humanoid robots" and their stable walking control have progressed, and expectations for their practical application are increasing. These legged mobile robots are more unstable than crawler robots, making posture control and walking control difficult, but they are superior in that they can realize flexible walking and running operations such as climbing stairs and climbing over obstacles. There is.

Stationary type robots, such as arm type robots, which are implanted in a specific place and used,
Work only in fixed and local work spaces such as assembly and selection of parts. On the other hand, a mobile robot has a non-limitative work space, and can freely move on a predetermined route or on a non-route to perform a predetermined or arbitrary human work on behalf of a human or a dog. Alternatively, various services that replace other life forms can be provided.

One of the uses of legged mobile robots is to perform various difficult tasks on behalf of industrial activities and production activities. For example, maintenance work in nuclear power plants, thermal power plants, and petrochemical plants, parts transportation / assembly work in manufacturing plants, cleaning in high-rise buildings, agency of dangerous work / difficult work such as rescue at fire sites and the like. .

Further, as another application of the legged mobile robot, it is more closely related to life, that is, "symbiosis" or "entertainment" with humans, rather than the above-mentioned work support.
There are uses. This kind of robot faithfully reproduces the motion mechanism of humans, dogs (pets), bears and other relatively intelligent legged walking animals and the rich emotional expression using the limbs. In addition to simply faithfully executing the pre-entered motion pattern, it also dynamically responds to the words and attitudes received from the user (or other robot) (such as "praise", "scrib", "hit", etc.). It is also required to realize corresponding and lively response expressions.

In the conventional toy machine, the relationship between the user operation and the response motion is fixed, and the motion of the toy cannot be changed according to the user's preference. As a result, the user eventually gets tired of the toy that repeats only the same operation.

On the other hand, an intelligent robot that autonomously operates generally has a function of recognizing external information and reflecting its own behavior on it. That is, the robot realizes autonomous thinking and motion control by changing the emotion model or the instinct model based on input information such as voice, image, and tactile sense from the external environment to determine the motion. That is, by providing the emotion model and the instinct model, the robot can realize realistic communication with humans at a higher intellectual level.

In order for a robot to perform an autonomous operation according to a change in environment, conventionally, an action is described by a combination of simple action descriptions such as receiving an action for one observation result and taking an action. It was By mapping the actions to these inputs, it is possible to express complex actions that are not unique by introducing functions such as randomness, internal state (emotion / instinct), learning, and growth. The place where this action is expressed is called the action generation unit.

For example, an autonomous robot apparatus having a stereo camera composed of two CCD cameras with eyes as movable eyes with respect to the body, and further having left and right legs capable of bipedal walking. Therefore, a behavior generation unit that generates a neck motion is called a neck motion generation unit.

[0011]

By the way, conventionally, the gaze (tracking) to the feature point on the image by the neck is converted into the coordinate system of the root of the neck, and is converted into the position of the feature point in the converted coordinate, The process of turning the neck in that direction was done. Therefore, the sensor information to be gazed is specified from the upper part such as the motion command generation unit to the neck motion generation unit, and the gaze action is performed in that direction. Therefore, it is necessary for the upper module (motion command generation unit) to select the optimum information used for the gaze behavior.

The present invention has been made in view of the above circumstances, and a motion command generation unit (upper module) is capable of tracking only by designating what to look for and a robot device and its control. The purpose is to provide a method.

[0013]

In order to solve the above-mentioned problems, an autonomous robot apparatus according to the present invention is an autonomous robot apparatus which autonomously gazes at an object based on a recognition result of an external environment. A plurality of recognition means for recognizing attribute information about the object existing in the external environment,
Integrated information generating and storing means for receiving each attribute information recognized by the plurality of recognizing means, integrating so as to maintain temporal and spatial consistency for each object, and attaching and storing an identification number; Operation command generation for searching the integration information of the target object stored in the integrated information generation storage means for integration information of a predetermined target object to be watched and generating an operation command based on the identification number of the target object And an operation command based on the identification number of the object from the operation command generation means, the integrated information to which the identification number of the object is attached is received from the integrated information generation and storage means, and gaze behavior Motion generating means for selecting attribute information to be used for generating a control signal for gazing behavior.

In this robot apparatus, the integrated information generating / storing means receives each attribute information recognized by the plurality of recognizing means, integrates each object so as to maintain temporal and spatial consistency, and assigns the identification number. Attached and stored, the action command generation means searches the integrated information of the object from the integrated information of the object stored in the integrated information generation storage means, and searches for the integrated information of the object to be the target. When the operation command based on the identification number of the object is generated and the operation generation unit receives the operation command based on the identification number of the object from the operation command generation unit, the integrated information with the identification number of the object is added. Is received from the integrated information generating and storing means, attribute information used for the gaze action is selected, and a control signal for the gaze action is generated.

The control method of the robot apparatus according to the present invention is
In order to solve the above problems, in a control method of an autonomous robot device that gazes at an object autonomously based on a recognition result for the external environment, a plurality of attribute information regarding an object existing in the external environment is recognized. An integrated information generating and storing step of receiving each attribute information recognized by the recognition means, integrating so as to maintain temporal and spatial consistency for each object, and storing with an identification number, and the integrated information generation From the integrated information of the object stored in the storage step,
Based on the identification number of the target from the operation command generation step of searching for integrated information of a predetermined target object to be watched and generating an operation command based on the identification number of the target When the operation command is received, the integrated information with the identification number of the object is received from the integrated information generation / storage step, and the attribute information used for the gaze behavior is selected to generate the control signal for the gaze behavior. And a motion generating step for performing.

[0016]

BEST MODE FOR CARRYING OUT THE INVENTION A bipedal type robot apparatus shown as an example of the configuration of the present invention will be described in detail below with reference to the drawings. This humanoid robot device is a practical robot that supports human activities in various situations in the living environment and other daily life, and can act according to internal conditions (anger, sadness, joy, enjoyment, etc.), It is an entertainment robot that can express the basic actions that humans perform. That is, the behavior control can be autonomously performed based on the recognition result of the external stimulus such as the voice or the image.

As shown in FIG. 1, in the robot apparatus 1, a head unit 3 is connected to a predetermined position of a torso unit 2, two left and right arm units 4R / 4L, and two left and right legs. The units 5R / 5L are connected to each other (however, each of R and L is a suffix indicating each of right and left. The same applies hereinafter).

FIG. 2 schematically shows the joint degree of freedom structure of the robot apparatus 1. The neck joint supporting the head unit 3 includes a neck joint yaw axis 101 and a neck joint pitch axis 10
It has two degrees of freedom, namely 2 and the neck joint roll shaft 103.

Each arm unit 4R / 4L constituting the upper limb has a shoulder joint pitch axis 107, a shoulder joint roll axis 108, an upper arm yaw axis 109, and an elbow joint pitch axis 1.
10, forearm yaw axis 111, wrist joint pitch axis 112
And a wrist joint roll wheel 113 and a hand portion 114. The hand portion 114 is actually a multi-joint / multi-degree-of-freedom structure including a plurality of fingers. However, since the motion of the hand portion 114 has little contribution or influence to the posture control and the walking control of the robot apparatus 1, it is assumed that the degree of freedom is zero in this specification. Therefore, each arm has seven degrees of freedom.

Further, the trunk unit 2 has three degrees of freedom: a trunk pitch axis 104, a trunk roll axis 105 and a trunk yaw axis 106.

Each leg unit 5R / 5L constituting the lower limb has a hip joint yaw shaft 115, a hip joint pitch shaft 116, a hip joint roll shaft 117, and a knee joint pitch shaft 1.
18, an ankle joint pitch shaft 119, an ankle joint roll shaft 120, and a foot portion 121. In this specification, the intersection of the hip joint pitch axis 116 and the hip joint roll axis 117 defines the hip joint position of the robot apparatus 1. The foot 121 of the human body is actually a structure including a multi-joint, multi-degree-of-freedom foot, but the foot of the robot apparatus 1 has zero degrees of freedom. Therefore, each leg has 6 degrees of freedom.

In summary, the robot apparatus 1 as a whole has 3 + 7 × 2 + 3 + 6 × 2 = 32 degrees of freedom. However, the robot device 1 for entertainment is not necessarily limited to 32 degrees of freedom. It goes without saying that the degree of freedom, that is, the number of joints, can be appropriately increased or decreased in accordance with design / production constraint conditions and required specifications.

Each degree of freedom of the robot apparatus 1 as described above is actually implemented by using an actuator.
It is preferable that the actuator be small and lightweight in view of demands such as eliminating extra bulges in appearance and approximating the shape of a natural human body, and performing posture control for an unstable structure such as bipedal walking. .

FIG. 3 schematically shows the control system configuration of the robot apparatus 1. As shown in the figure, the robot device 1 includes a trunk unit 2, which represents human limbs.
The head unit 3, the arm units 4R / 4L, the leg units 5R / 5L, and the control unit 10 that performs adaptive control for realizing cooperative operation between the units.

The overall operation of the robot apparatus 1 is controlled by the control unit 10. Control unit 10
Is a CPU (Central Processing Unit) or DRA
An interface (not shown) for exchanging data and commands with the main control unit 11 including main circuit components (not shown) such as M and flash ROM, and the power supply circuit and each component of the robot apparatus 1. And the peripheral circuit 12 including.

In implementing the present invention, the installation place of the control unit 10 is not particularly limited. Although it is mounted on the trunk unit 2 in FIG. 3, it may be mounted on the head unit 3. Alternatively, the control unit 10 may be provided outside the robot apparatus 1 to communicate with the body of the robot apparatus 1 in a wired or wireless manner.

The degree of freedom of each joint in the robot apparatus 1 shown in FIG. 2 is realized by an actuator corresponding to each joint. That is, the head unit 3 includes a neck joint yaw axis 101, a neck joint pitch axis 102, and a neck joint roll axis 10.
3, the neck joint yaw axis actuator A ₂ representing each of
A neck joint pitch axis actuator A ₃ and a neck joint roll axis actuator A ₄ are arranged.

Further, the head unit 3 is provided with a CCD (Charge Coupled Device) camera for picking up an image of an external situation, and a distance sensor for measuring a distance to an object located in front, A microphone for collecting external sound, a speaker for outputting voice, a touch sensor for detecting the pressure received by the physical action of the user such as "stroking" or "striking" are provided. ing.

The trunk unit 2 includes a trunk pitch axis actuator A ₅ , a trunk roll axis actuator A ₆ , which expresses each of the trunk pitch axis 104, trunk roll axis 105, and trunk yaw axis 106, A trunk yaw axis actuator A ₇ is provided. Also, in the trunk unit 2,
The robot apparatus 1 is provided with a battery as a power source. This battery is composed of a chargeable / dischargeable battery.

The arm unit 4R / 4L is used for the upper arm unit.
Knit 4₁R / 4₁L and elbow joint unit 4_TwoR / 4_Two
L and forearm unit 4_ThreeR / 4_ThreeIt is subdivided into L,
Shoulder joint pitch axis 107, shoulder joint roll axis 108, upper arm yo
-Axis 109, elbow joint pitch axis 110, forearm yaw axis 11
1, wrist joint pitch axis 112, wrist joint roll axis 113
Of each shoulder joint pitch axis actuator A₈,shoulder
Joint roll axis actuator A₉, Upper arm yaw actu
Eta A₁₀, Elbow joint pitch axis actuator A ₁₁,
Elbow joint roll axis actuator A₁₂, Wrist joint pitch
Axis actuator A_Thirteen, Wrist joint roll axis actuator
Data A₁₄Has been deployed.

The leg unit 5R / 5L includes a thigh unit 5 ₁ R / 5 ₁ L and a knee unit 5 ₂ R / 5 ₂ L.
The hip joint yaw axis 115, the hip joint pitch axis 116, the hip joint roll axis 117, the knee joint pitch axis 118, the ankle joint pitch axis 119, and the ankle joint roll axis are subdivided into the tibial unit 5 ₃ R / 5 ₃ L. A hip joint yaw axis actuator A ₁₆ , a hip joint pitch axis actuator A ₁₇ , a hip joint roll axis actuator A ₁₈ , a knee joint pitch axis actuator A ₁₉ , an ankle joint pitch axis actuator A ₂₀ , and an ankle joint roll axis actuator A ₂₁ that represent each of 120. Has been deployed. The actuators A ₂ , A ₃ ... Used for the respective joints are more preferably composed of small AC servo actuators of the type that are directly connected to the gears and the servo control system is integrated into one chip and mounted in the motor unit. can do.

For each mechanical unit such as the trunk unit 2, the head unit 3, the arm units 4R / 4L, the leg units 5R / 5L, etc., the sub-control units 20, 21, 22R / of the actuator drive control unit are provided. 22L and 23R / 23L are provided. Furthermore, a ground contact confirmation sensor 30R for detecting whether or not the sole of each leg unit 5R / 5L has landed
In addition to wearing / 30L, the trunk unit 2 is equipped with a posture sensor 31 for measuring the posture.

The ground contact confirmation sensor 30R / 30L is composed of, for example, a proximity sensor or a micro switch installed on the sole of the foot. Further, the posture sensor 31 is composed of, for example, a combination of an acceleration sensor and a gyro sensor.

By the output of the ground contact confirmation sensor 30R / 30L, it is possible to determine whether each of the left and right legs is currently standing or swinging during an operation period such as walking or running. Further, the output of the posture sensor 31 can detect the inclination and posture of the trunk.

The main control unit 11 includes the sensors 30R / 30
The control target can be dynamically corrected in response to the outputs of L and 31. More specifically, the sub control unit 20,
By performing adaptive control on each of 21, 22R / 22L and 23R / 23L, it is possible to realize a whole-body movement pattern in which the upper limbs, the trunk, and the lower limbs of the robot apparatus 1 are cooperatively driven.

The whole body motion of the robot apparatus 1 on the body is
Foot movement, ZMP (Zero Moment Point) trajectory, trunk movement
Movement, upper limb movement, waist height, etc.
Command to instruct the operation according to the setting contents
Sub control unit 20, 21, 22R / 22L, 23R / 23L
Transfer to. Then, each sub control unit 20, 21, ...
・ Interpret the received command from the main controller 11
Each actuator A _Two, A_Three... Drives for etc.
Output a control signal. "ZMP" here means walking
Of the point on the floor where the moment due to the floor reaction force inside becomes zero
The “ZMP orbit” is, for example, a robot
Means a locus of movement of the ZMP during the walking motion of the device 1.
It In addition, the concept of ZMP and ZMP
Regarding the points applied to the stability criterion, see Miomir Vukob.
ratovic "LEGGED LOCOMOTION ROBOTS" (Ichiro Kato
In "Walking Robot and Artificial Feet" (Nikkan Kogyo Shimbun))
Have been described.

As described above, in the robot apparatus 1, each of the sub-control units 20, 21, ... Interprets the received command from the main control unit 11 and each actuator A ₂ , A ₃
A drive control signal is output to ... to control the drive of each unit. As a result, the robot apparatus 1 makes a stable transition to the target posture and can walk in a stable posture.

Further, in the control unit 10 of the robot apparatus 1, in addition to the posture control as described above, various sensors such as an acceleration sensor, a touch sensor, a ground contact confirmation sensor, image information from a CCD camera, and voice from a microphone. Information is handled in a centralized manner. In the control unit 10, although not shown, various sensors such as an acceleration sensor, a gyro sensor, a touch sensor, a distance sensor, a microphone, a speaker, actuators, CCD cameras, and batteries are connected to the main control unit 11 via corresponding hubs. Has been done.

The main control section 11 sequentially takes in sensor data, image data and audio data supplied from the above-mentioned respective sensors, and sequentially stores them in a predetermined position in the DRAM via the internal interface. Further, the main control unit 11 sequentially takes in the battery remaining amount data representing the battery remaining amount supplied from the battery, and stores this in the DRAM.
It is stored in a predetermined position inside. Each sensor data, image data, voice data, and battery remaining amount data stored in the DRAM are used when the main control unit 11 controls the operation of the robot apparatus 1.

The main controller 11 reads the control program and stores it in the DRAM at the initial stage when the power of the robot apparatus 1 is turned on. In addition, the main control unit 11 uses the sensor data, the image data, the audio data, and the battery remaining amount data that are sequentially stored in the DRAM from the main control unit 11 as described above, based on the self and surrounding conditions and the user. Judging whether or not there are instructions and how to work.

Further, the main control section 11 determines an action according to its own situation based on this determination result and the control program stored in the DRAM, and drives a necessary actuator based on the determination result. The robot apparatus 1 is caused to take actions such as so-called “gesture” and “hand gesture”.

In particular, the robot apparatus 1 uses specific sensor information from a plurality of sensor information to perform action control of gazing in that direction. Further, in the present invention, by using the information of the short-term storage unit (ShortTermMemory) that performs sensor fusion, appropriate sensor information is selected from the target object information of the gaze behavior, and from the higher-order command, the ID used for sensor fusion ( The gaze operation module automatically selects the optimum sensor information only by specifying the object ID).

FIG. 4 schematically shows the basic architecture of the action control system 50 including the gaze action control adopted in the robot apparatus 1.

Object-oriented programming can be incorporated into the illustrated behavior control system 50. In this case, each software is handled in module units called "objects" in which data (property) and a processing procedure (method) for the data are integrated. In addition, each object can pass properties and inherit methods by message communication and inter-object communication method using shared memory.

The behavior control system 50 uses the external environment (Envi
ronments), the visual recognition function unit 51,
The hearing recognition function unit 52 and the contact recognition function unit 53 are provided.

The visual recognition function section (Video) 51 is, for example, a charge coupled device (CCD).
Image recognition processing such as face recognition and color recognition and feature extraction are performed based on a captured image input through an image input device such as a camera. The visual recognition function unit 51 uses "MultiColorTracke
It consists of multiple objects such as "r", "FaceDetector", and "FaceIdentify".

The auditory recognition function unit (Audio) 52 performs voice recognition of voice data input via a voice input device such as a microphone to extract a feature and set a word set (text).
Do recognition. The auditory recognition function unit 52 is composed of a plurality of objects such as "AudioRecog" and "AuthurDecoder" described later.

The contact recognition function unit (Tactile) 53 recognizes a sensor signal from a contact sensor built in, for example, the head of the machine body, and recognizes an external stimulus such as "stroked" or "struck". To do.

Internal state management unit (ISM: Internal Statu)
s Manager) 54 has an instinct model and an emotional model,
The visual recognition function unit 51, the auditory recognition function unit 52,
External stimulus (E recognized by the touch recognition function unit 53
S: External Stimula) to manage internal states such as instinct and emotion of the robot apparatus 1.

The emotion model and the instinct model respectively have the recognition result and the action history as inputs, and manage the emotion value and the instinct value. The behavior model can refer to these emotional values and instinct values.

The short-term memory section (ShortTermMemory) 55 is
The visual recognition function unit 51, the auditory recognition function unit 52,
It is a functional module that holds a target or event recognized from the external environment by the contact recognition function unit 53 for a short period of time. For example, the input image from the camera is stored for a short period of about 15 seconds.

The long-term memory unit (LongTermMemory) 56 is used to hold information obtained by learning such as the name of an object for a super period. The long-term storage unit 56, for example, associatively stores a change in internal state from an external stimulus in a certain action module.

The action control of the robot apparatus 1 is realized by the "reflexive action" realized by the reflexive action unit 59, the "situation dependent action" realized by the situation dependent action layer 58, and the "thinking action layer 57". It is roughly divided into "thinking behavior".

Deliberative Layer 57
Performs a relatively long-term action plan of the robot apparatus 1 based on the stored contents of the short-term storage unit 55 and the long-term storage unit 56.

The deliberative action is an action which is made by inferring a plan for realizing the reasoning or in accordance with a given situation or an instruction from a human. Such inferences and plans are
Since the processing time and the calculation load are longer than the reaction time for maintaining the interaction, the robot apparatus 1 performs the reflexive behavior and the situation-dependent behavior while returning the reaction in real time,
Conduct deliberative actions by reasoning and planning.

Situation-dependent behavior hierarchy (SBL: Situated Beha)
The viorsLayer) 58 controls the action immediately corresponding to the current situation of the robot apparatus 1 based on the stored contents of the short-term storage unit 55 and the long-term storage unit 56 and the internal state managed by the internal state management unit 54. To do.

The state-dependent action hierarchy 58 prepares a state machine for each action, and classifies the recognition result of the external information input by the sensor depending on the action and the situation before that to classify the action as a body. Expressed above. The situation-dependent action hierarchy 58 also realizes an action for keeping the internal state within a certain range (also called "homeostasis action"), and when the internal state exceeds the specified range, the internal state Activate the behavior so that it becomes easier for the behavior to return to within the range (actually, the behavior is selected in consideration of both the internal state and the external environment). Situation-dependent behavior has a slower reaction time than reflexive behavior (described later).

Reflection action layer 57 and situation-dependent action layer 58
Can be implemented as an application.

Reflexive action part (ConfigurationDependentAc
The tionsAndReactions or ReflexiveSBL) 59 is a functional module that realizes reflexive body movements according to the external stimuli recognized by the visual recognition function unit 51, the auditory recognition function unit 52, and the contact recognition function unit 53.

Basically, the reflexive behavior is a behavior that directly receives the recognition result of the external information inputted by the sensor, classifies the recognition result, and directly determines the output behavior. For example, behaviors such as chasing a human face or nodding are preferably implemented as reflexive behaviors.

In the robot apparatus 1 according to the present embodiment, the short-term storage unit 55 temporally stores the results of a plurality of recognizers such as the visual recognition function unit 51, the auditory recognition function unit 52, and the touch recognition function unit 53 described above. They are integrated so as to maintain spatial consistency, and the perception of each object in the external environment is provided as a short-term memory to the context-dependent behavior hierarchy (SBL) 58 and the like. Further, the recognition result of the external information inputted by the sensor is directly provided to the reflexive SBL 59.

Therefore, on the action control module side configured as a higher-order module, a plurality of recognition results from the outside world are integrated and treated as meaningful symbol information,
A high degree of behavior control can be performed. In addition, by using more complex recognition results such as the correspondence problem with the previously observed recognition results, it is possible to determine which skin color area corresponds to which person in the face, and which person this voice is. Can be solved.

Further, since the short-term storage unit 55 holds the information regarding the recognized observation result as a memory, even if the observation result does not come temporarily during the autonomous action period, the behavior of the aircraft is Objects can be made to appear to be perceived by higher-level modules such as control applications at all times. For example, since the information outside the field of view of the sensor is immediately retained without being forgotten, even if the robot loses sight of the object, it can be searched for later. As a result, it is possible to realize a stable system that is resistant to mistakes in the recognizer and noise from the sensor and does not depend on the timing of notification of the recognizer. Also, even if the recognizer alone lacks information, other recognition results may be supplemented, so that the recognition performance of the entire system is improved.

Further, since the related recognition results are linked, it is possible to judge the action by using the related information in the upper module such as the application. For example, the robotic device can derive the name of the person based on the called voice. As a result, it is possible to notes, such as answer such as "Hello, XXX-san." The response of the greeting.

By applying the behavior control system 50 as described above, the robot apparatus 1 uses the information of the short-term storage unit 55 that performs the sensor fusion, as described above, and determines the attribute related to the object from the object information of the gaze behavior. Appropriate sensor information that senses the information is selected, and a gaze action is performed on the object. From the higher-level command, simply by designating the ID (object ID) used for sensor fusion, the gaze operation module automatically selects the optimum sensor information.

FIG. 5 shows each module constituting the gaze behavior control system 70 included in the behavior control system.

The color detector 71 is a module for performing color detection, receives image data from an image input device such as a camera, extracts a color based on a plurality of color models that it has in advance, and outputs a continuous color region. Split into. The color information (ColorInfo) such as the position, size, and feature amount of each divided color area is stored in a short-term storage unit (ShortTermMemory) 75 and a head motion generation unit (HeadMotionGenerator: H) described later.
MG) 77 as one of the attribute information of the object.

The sound detector 72 is a module for detecting a sound, and receives sound data from a sound input device such as a microphone and performs not only sound recognition but also the direction of the sound. The sound direction can be detected by estimating the sound source direction in the horizontal direction when the microphone is stereo. Sound information detected here (SoundInfo)
Is sent to the short-term storage unit 75 and a neck motion generation unit 77, which will be described later, as one of the attribute information regarding the object.

The face detector 73 is a module for detecting a face area in an image frame, receives image data from an image input device such as a camera, and reduces and converts the image data into a nine-step scale image. A rectangular area corresponding to the face is searched from all the images. The overlapping candidate areas are reduced to detect face information (FaceInfo) such as the position, size, and feature amount regarding the area finally determined to be the face. The face information detected here is sent to the short-term storage unit 75.

The joint angle detecting section 74 detects the current value (sensor output) of the joint angle of each joint of the leg, neck, arm and the like of the robot apparatus 1. The sensor output of the joint angle detected here is sent to the short-term storage unit 75 and the neck motion generation unit 77.

The short-term storage unit 75 is an object that holds information about the external environment of the robot apparatus 1 for a relatively short time, and the sound (voice) detection result from the sound detection unit 72, the color detection result from the color detection unit 71, The face detection unit 73 receives the face detection result, and the joint angle detection unit 74 receives the joint angle sensor output. Then, the plurality of pieces of detection information are integrated so as to maintain consistency in terms of time and space, treated as meaningful integrated information, and held for a relatively short time, for example, 15 seconds. These pieces of integrated information are passed to the motion command unit 76 and the neck motion generation unit 77.

The operation command section 76 is a higher-level module such as the above-mentioned situation-dependent action layer. A command for instructing a neck motion to the neck motion generation unit 77 is developed. In the present invention, the tracking operation is performed on the robot apparatus 1 by designating the ID (target object ID) relating to the integrated information stored in the short-term storage unit 75, that is, by designating what to track. It is something to do.

The neck motion generator 77 is a module for calculating the joint angle of the neck in response to the command to move the neck from the motion commander 76. When the command of “tracking” (in the present embodiment, the object ID) is received,
Based on the information received from the short-term storage unit 75, the joint angle of the neck facing the direction in which the object exists is calculated and output. The object ID is received, the optimum object information is selected, and the selected object information is obtained.

Next, an embodiment in which the robot apparatus 1 tracks the face of Mr. A will be described. In FIG. 6, the robot apparatus 1 is trying to track the face 85 of Mr. A based on the information obtained by the PSD sensor 80 including a CCD sensor or the like. Near A's face 85 is a skin-colored ball 87.

The set situation will be described with reference to FIG. The time Time (frameNumber) starts from 0 and is incremented by 1. Among them, 3
In this situation, the face is seen in the image acquired every 0th frame, and the skin-colored area of the face is detected every 10th frame, and when "50th frame", the voice "Hello" is heard. Since Mr. A's face was visible in the first 30 frames, he is about to execute the tracking command.

In such a situation setting, the face detector 73, the color detector 71, the sound detector 72, and the short-term storage section 7 are set.
5, detailed operations of the motion command unit 76 and the neck motion generation unit 77 will be described below.

First, the face detector 73 applies a general detection algorithm of template matching to generate face information FaceInfo which is one of the attribute information about the object. Here, as shown in FIG. 8, the face information is detected by looking at the face image of the frame at time t. Face detection result is FaceRecogResult = (frameNumber, faceID, gravity point
(x, y), sizeX, sizeY, cm). Here, frameNumber indicates the time (absolute time) in the system of the robot device, so this is also shown in the image.
frameNumber is attached. gravity point (x, y) is the coordinate of the center of gravity of the face image. sizeX and sizeY are the size on the X axis and the size on the Y axis of the face image. cm is confidence mea
sure represents the certainty (reliability) of face recognition.

The certainty (reliability) cm of this face recognition can be calculated by the following equation, for example, with the number of matched pixels among the number n of pixels of the template being (n _match ). cm = n _match / n Therefore, the face detection result signal of the image of the 30th frame in FIG. 7 is, for example, FaceRecogResult (30, human
-A, x1, y1, sizex1, sizey1,0.3).

Next, the color detector 71 applies a color detection algorithm of extracting a region matching the set color space (here, a skin color space), and the color which is one of the attribute information of the object. Generate a detection result signal.

Each pixel P (x, y) on the image has a value of color information (y, u, v) as shown in FIG. From here, the skin color (S
Detect the center and size of the area that matches the color space set in (KIN).

Assuming that the color space of pixels is represented by 8 bits, as shown in FIG. 9, 0 ≦ y ≦ 255, 0 ≦ u ≦ 255, 0
Since it is expressed by ≦ v ≦ 255, the space designation method is shown in FIG.
, (U1, v1), (u2, v
It can be defined as a rectangle defined in 2). Here, Color [y] [0] = u1; Color [y] [1] = v1; Color [y] [2] = u2; Color [y] [3] = v2; Therefore, the color space color can be defined as Color [256] [4].

Next, a plurality of color area detection algorithms will be described with reference to FIGS. 11, 12 and 13. The pixel position on the image is P (x0, x1), and the yuv data is Cu (x0, x1).
1), Cv (x0, x1), Cy (x0, x1), the image data after passing through the color space filter (1 if it matches the set color space, otherwise 0) Q (x0, x1)
Is Becomes

In the color collation image (0), when each (x0, x1) matches (Q (x0, x1) = 1), it is searched for whether the surrounding pixels match, and they are matched. If
Consider the same area.

The condition for the pixel (X, Y) is (Q (X + 1, Y) == 1‖Q (X-1, Y) == 1‖Q (X, Y + 1) == 1 ‖Q (X, Y-1) == 1‖Q (X + 1, Y + 1) == 1‖Q (X + 1, Y-1) == 1‖Q (X-1, Y + 1) ) == 1‖Q (X-1, Y-1) == 1) When this is filtered for the color matching image (0) of FIG. 11, the color matching image (1) of FIG.
A color matching image (2) of 3 is obtained.

The color detection signal is obtained by adding the color space id to the color detection result and is represented by the following equation. ColorDetectResult (id, color, centerX, centerY, sizeX, s
izeY) where id is the serial number and color is the color space used for the segment. Therefore, if color is a face,
It becomes the skin color area (SKIN in this example). Also, centerX, c
enterY indicates the barycentric coordinates in the color detection area. Also,
sizeX and sizeY indicate the size of the color detection area.

For each of the color matching images (1) and (2), center, sizeX and sizeY are calculated, and a color detection signal is calculated.
ColorDetectResult = (id, color, centerX, centerY, sizeX,
sizeY) is calculated.

Then, the color detection signal in the color detection area of the color collation image (1) is ColorDetectRes when id is 10.
ult [0] = (10, “SKIN”, 3,4,3,3).

Further, the color detection signal in the color detection area of the color matching image (2) is ColorDetectResul when id is 11.
t [0] = (11, “SKIN”, 6,8,3,3).

Then, when the color detection signal in the color detection area detected as described above is stored in the color detection module and the area where the frame overlaps is large, the same one is detected. It is considered that they have moved, and the same id is numbered. For example, in FIG.
As shown in 4, newly detected color detection areas new and Old (id =
ID threshold _S new color detection area new color detection signal if greater than _thresould with the area _{S overlap} overlap Overlap regions between the color detection area 10)
_new is the same as id _old . That is, S
_{If overlap} > S _{th result,} then ID
_{Let new} = id _old .

In the situation shown in FIG. 7, the skin color area of the face is detected every 10 frames. In this way, the color detection result signal ColorSegmentResult in the color detection area detected for each predetermined number of frames is represented by (frameNumber, i
d, x, y, sizeX, sizeY, cm). cm (Confiden
ce measure) is the reliability of color recognition. The calculation result will be described later. The detection algorithm for a plurality of skin color areas in the same image is as described above.

Therefore, as shown in FIG. 15, the color detection result signals of the two color detection areas in the image of every 10 frames are ColorSegmentResult (10,1, x2, y2, sizex2 when id is 1 and 2. , sizey2,0.8) ColorSegmentResult (10,2, x3, y3, sizex3, sizey3,0.8).

There is a method of calculating the reliability of color recognition, when a plurality of identical color detection areas, for example, a skin color area is detected, it is divided by the number. Therefore, cm = 1 / (the same color area included in the image). Next, the sound detector 72 receives voice data from a voice input device such as a microphone and performs voice recognition (voice detection). Therefore, the sound detector 72 is
It is equipped with thurDecoder and performs voice recognition using voice features, voice dictionary and syntax dictionary. The sound detector 72 detects not only the voice recognition but also the direction of the sound, that is, the sound source direction. The sound source direction is detected by estimating the sound source direction in the horizontal direction from the time difference between the output level R from the Rigt mic and the output level L from the Left mic as shown in FIG. 16 obtained when the microphone is stereo. It is possible. The sound detection signal detected here is sent to the short-term storage unit 75 and a neck motion generation unit 77 described later.

This sound source direction detection algorithm is a general method. A specific example of the sound detection algorithm will be described below. In FIG. 17, the sound velocity from the sound source Ss is v [m / s],
When the time difference between the left and right microphones is t [s], the path difference u between the left and right sounds is tv [m]. Therefore, if it is approximated by φ = θ and the distance from the neck center o to the microphone is a [m], cos θ = a / (tv) and θ = cos ₋₁ (a / (tv)).

Therefore, the voice detection result obtained by combining the voice recognition result (word) and the sound source direction (θ) is SoundRecogResu.
It becomes lt (frameNumber, word, θ, cm). Cm here
Indicates the reliability of voice recognition. Details will be described later.

Therefore, in the situation of FIG. 7, the sound detection result of the 50th frame is SoundRecogResult (50, "OHAYO", 30 degrees, 0.9).

The voice recognition reliability cm is obtained as follows. That is, it is calculated according to the average power of the sampling (here, 50 frames (1 frame 5msec) frequency domain (10kHz-7kHz) and the maximum value max value up to that point. If it is 80% or more, it is calculated according to 0.8. To do.

That is, P _current = Σp
_frame / 50, and if the average power P _{cur rent} is 80% or more of the maximum value P _ma of the power so far, the reliability cm is 0.8. Also, the average power P _current
&& P _curren is not less than 50% of the maximum value P _ma, and the reliability cm if smaller than 80% of the maximum value P _ma is 0.5
Becomes Furthermore, _{P current} && P _curren
Is a but more than 30% of the maximum value P _ma, and 50% of the maximum value P _ma
If it is smaller, the reliability cm is 0.3.

The algorithm for determining the reliability cm is shown below by an equation. _{If (P current> = P max} * 0.8) cm = 0.8 elseIf (P max * 0.5 <= P current && P current <P max * 0.8) cm = 0.5 Else if (P max * 0.3 <= P current && P curren <Pmax * 0.5) cm = 0.3.

Next, the short-term storage unit 75 stores the information about the external environment of the robot apparatus 1, that is, the sound (voice) detection result from the sound detection unit 72, the color detection result from the color detection unit 71, and the face as described above. The face detection result is received from the detection unit 73, and the joint angle sensor output is received from the joint angle detection unit 74. Then, these plural detection results are integrated so as to maintain consistency in terms of time and space, treated as meaningful integrated information, and held for a relatively short time, for example, 15 seconds.

The short-term memory information stored in the short-term memory unit 75 is an array of information called ShortTermMemory (targetID, sensorType, id, data, updated frameNumber, cm). Here, sensorType represents a face recognition result, a color detection result, and a voice detection result.

FIG. 18 schematically shows how the short-term storage section 75 operates. In the example shown in the figure, the operation when the face recognition result (FACE) and the voice recognition, and the sound source direction recognition result (VOICE) are processed at different timings and notified to the short-term storage unit 75, It is shown (however, it is drawn in a polar coordinate system whose origin is the machine body of the robot apparatus 1). In this case, each recognition result face and vo
Since ice is temporally spatially close (overlapping), it is determined that it is one object having face attributes and voice attributes, and the information from the detectors is integrated. The recognized object is stored in the memory as a memory.

A target detector for detecting a target from the external environment is provided in the short-term storage unit 75.
This target detector adds a new target or recognizes an existing target based on the detection (recognition) results of the detectors 71, 72 and 73 such as the face recognition result, the voice recognition result, and the color recognition result. Update to reflect the results. The detected target is held in a target memory (not shown).

Further, in the target memory, a garbage collector that searches for a target that is no longer observed and erases it
Or a target associate (Target) that determines the relationship between multiple targets and associates them with the same target.
There are functions such as tAssociate). The garbage collector is realized by decrementing the certainty factor of the target over time and deleting the target whose certainty factor is below a predetermined value. In addition, the target associate can identify the same target by having spatial and temporal closeness between the targets whose feature values of the same attribute (recognition type) are close to each other.

Next, as shown in FIG. 6, when the skin-colored ball 87 is near the face 85 of Mr. A, the robot apparatus 1 is based on the information obtained by the PSD sensor 80 which is a CCD sensor. A specific example of a series of processes of the short-term storage unit 75 in the embodiment for tracking the face 85 of Mr. A will be described with reference to FIGS.

In this case, based on the situation setting shown in FIG. 7, only the color detection by the color detector 71 is performed up to the 29th frame as shown in FIG. Then, the color detection result signal ColorSegmentResult (2
9,1, x2, y2, sizex2, sizey2,0.8) is detected by the target detector. Here, the short-term storage unit 75 targ the skin color area (id = 1) in which the skin color ball 87 is detected as a skin color object.
Let shortTermMemory (0, “COLOR”, 1, “SKIN_COLOR”, 20,0.
8) is stored in the target memory as. As mentioned above, ShortTermMemory is (targetID, sensorType, id, data,
Since it is an array of updated frameNumber, cm), 2
The short-term memory information stored by the 9th frame is target
This is information with an ID of 0, a color detection result of “COLOR”, an id of 1, a skin color region of “SKIN_COLOR”, an updated frameNumber of 20, and a recognition reliability of 0.8.

By the 29th frame, the short-term storage unit 75 also sets the skin-color area (id = 2) detected as a skin-colored object for Mr. A's face 85 as short-term memory information of targetID = 1.
ShortTermMemory as ShortTermMemory (1, “COLOR”, 2, “SKIN_COLOR”, 20,0.
8) is stored in the target memory as.

Next, in the 30th frame, the short-term storage section 75 stores the short-term storage information from the color detection result of the color detector 71 as ShortTermMemory (0, "COLOR", 1, "SKIN_COLOR", 30, 0.
8) ShortTermMemory (1, “COLOR”, 2, “SKIN_COLOR”, 30,0.
8) is stored in the target memory as.

In the 30th frame, the face detector 73
Performs face detection. For example, as shown in FIG. 20, face detection result FaceRecogResult (30, human-A, x1, y1, sizex1, sizey
1,0.3) is detected in the direction in which it overlaps with the skin color area of targetID = 1, the short-term memory unit 75 stores ShortTermMemory (1, “FACE”, A-san, “”, 30,0.3) Integrate with targetID = 1.

Then, as shown in FIG. 21, up to the 50th frame, the short-term storage section 75 stores the ShortTermMemory (0, "COLOR", 1, "SKIN_COLOR", 30, 0.
8) ShortTermMemory (1, “COLOR”, 2, “SKIN_COLOR”, 30,0.
8) Store ShortTermMemory (1, “FACE”, A-san, “”, 30,0.3) in the target memory section.

Next, at the 50th frame, as shown in FIG. 22, the sound detector 72 detects the sound as a large area overlapping with the skin color area of the skin color object from the direction of θ = 30 degrees, and the sound detection result is obtained. , SoundRecogResult (50, “OHAYO”, 30 degrees, 0.9) is transmitted to the short-term storage unit 75.

The short-term memory unit 75 detects the voice detection result by the target detector, and in the 51st frame shown in FIG. 23, the short-term memory information, ShortTermMemory (1, "SOUND", 1, "OHAYO", 50. , 0.9) into targetID = 1.

At this time, the short-term storage unit 75 stores the ShortTermMemory (0, "COLOR", 1, "SKIN_COLOR", 50, 0.
8) ShortTermMemory (1, “COLOR”, 2, “SKIN_COLOR”, 50,0.
8) ShortTermMemory (1, “FACE”, A-san, “”, 30,0.3) Stores ShortTermMemory (1, “SOUND”, 1, “OHAYO”, 50,0.9) in the target memory.

As described above, the short-term storage unit 75 calculates the position in the center coordinates of the neck from the angle information of the neck at that time from the sensed time and the position on the image for the image information, and the direction information thereof. To get Also, for the voice information, the position in the neck center coordinate system is calculated from the sensed time and the angle information of the sound.

After that, it is detected whether or not these pieces of information are mapped to a group (targetID) of existing information, and if they cannot be mapped, a new targetID is created.

By the way, in the gaze behavior control unit 70 shown in FIG. 5, the operation command unit 76 is a client of the short-term storage unit 75, and receives notification (Notify) of information about each target from the target memory.
The operation command unit 76 then targ as shown below.
The etID is searched, and the command is sent to the neck motion generation unit 77.

The operation command unit 76 stores all the short-term memory information Short.
For targetID of tTermMemory (), sensorType is “FAC
Search for “E” and id “A-san” under the condition of IF (sensorType == “FACE” && id == “A-san”). If a search result that matches the above conditions is obtained, command After sending the command, wait until the targetID is no longer in the short-term memory information.

That is, as shown in the processing procedure in FIG. 24, if the operation command section 76 has already transmitted the tracking command (Yes in step S1), the process proceeds to step S2, and the targetID on which the tracking command is based is determined. It is checked whether or not it exists in the latest short-term memory information, and if it exists, the processing from step S1 is repeated. Here, if the targetID does not exist in the latest short-term storage information, the process proceeds to step S3, and an action command to stop tracking is transmitted to the neck motion generation unit 77. If it is determined in step S1 that the tracking command has not been transmitted, the process proceeds to step S4, and there is a targetID in which the sensorType is "FACE" and the id is "A-san" in the short-term memory information as described above. I will check. Here, if there is no such targetID (N
O) Proceed to step S5, store a new targetID in the short-term storage unit 75, and send an action command. on the other hand,
In step S4, the above-mentioned se is included in the short-term memory information.
If nsorType is present (YES), the process returns to step S1.

Here, the motion command signal (command) transmitted from the motion command unit 76 to the neck motion generator 77 is Acti
It is onCommand (COMMAND, targetID). In this specific example, ActionCommand (“Tracking”, 1) is transmitted when the data of frameNumber = 30 is received from the short-term storage unit 75.

Next, the detailed configuration and operation of the neck motion generating unit 77 that receives the motion command signal from the motion command unit 76 and performs a gaze motion (tracking) will be described with reference to FIG. The neck motion generator 77 includes a motion command interpreter 771, a target object information selector 772, a target information receiver 773, a target position detector 774, and a target joint angle generator 775.

The operation command interpreter 771 receives the operation command signal sent from the operation command generator 76. The target object information selector 772 stores the operation object information received along with the operation command signal. The target information receiver 773 receives the tracking operation sensor information signal from the target object information selector 772 and acquires the target information. The target position detector 774 receives the target information signal from the target information receiver 773 and further receives the joint angle signal from the joint angle detection unit 74. Target joint angle generator 775
Receives a target position signal from the target position detector 774.

The details of the operation of the neck motion generator 77 will be described below with reference to FIGS. 26 to 29. The motion command interpreter 771 determines whether the command specifies the direction of the neck joint or a command for following a certain target ID. The target ID here is the short-term memory ta
Matches rgetID. Here, in the case of designating the direction of the neck joint, it is directly transmitted to the target joint angle generator 775 as shown in FIG.

As shown in FIG. 27, the target object selector 772 allows the target ID generated in the short-term storage unit 75 to be associated with the sensor information, that is, as shown in the figure, ShortTermMemory (0, "COLOR", 1, “SKIN_COLOR”, 30,0.
8) ShortTermMemory (1, “COLOR”, 2, “SKIN_COLOR”, 30,0.
8) ShortTermMemory (1, “FACE”, A-san, “”, 30,0.3) is received and stored in the database, based on the signal (targetID = 1) sent from the motion command interpreter 771. t
Holds the latest information about argetID. Also, it operates even when the information about the held targetID is updated from the short-term storage unit 75, and outputs the tracking operation sensor information signal (useTrackingInfo = color) to the target information receiver 77.
Send to 3.

As shown in FIG. 28, the target information receiver 773 uses the sensor information determined by the tracking operation sensor information signal (useTrackingInfo = color) from the target object information selector 772 to generate a sound or , Sensor information used for tracking operation is selected from the color signal information.

As shown in FIG. 29, the target position detector 774 calculates the joint angle at that time from the frameNumber attached to the sensor information, and sends it to the target joint angle generator 775. The target joint angle generator 775 is used by the target information receiver 77.
The motion command execution unit 80 generates a target joint angle from the sensor information from the target position detector 3 and the joint angle from the target position detector 774.
Send to.

The motion command execution unit 80 controls the actuator of the robot according to the target joint angle signal generated by the target joint angle generator 775 of the neck motion generation unit 77, and causes the robot apparatus 1 to perform the tracking operation.

As described above, the operation command generator 76 uses the targetID attached in the short-term memory 75 as the operation command signal Acti.
By simply adding (COMMAND, targetID) to onCommand, the target object information selector 772 selects sensor information to be used for optimal tracking according to the sensor status,
Then, the target information receiver 773 receives the sensor information used for the tracking operation by the tracking operation sensor information signal and sends it to the target joint angle generator 775. The target joint angle generator 775 uses fr associated with the sensor information.
The joint angle calculated from ameNumber is also supplied,
A target joint angle is generated from this joint angle and the sensor information, and is transmitted to the operation command execution unit 80.

In the gaze behavior control system 70, the motion command generator 76 only needs to specify what to track, and the neck motion generator 77 can perform tracking.

[0128]

In the robot apparatus according to the present invention, the integrated information generation / storage means receives each piece of attribute information recognized by the plurality of recognition means, and integrates the objects so as to maintain consistency in terms of time and space. Then, from the integrated information of the object stored in the integrated information generation / storage means by the operation command generation means, the predetermined operation generation means to be watched is integrated information of the object. To generate an operation command based on the identification number of the object, and the operation generation unit receives the operation command based on the identification number of the object from the operation command generation unit, the identification number of the object is Since the attached integrated information is received from the integrated information generating and storing means and the attribute information used for the gaze action is selected to generate the control signal for the gaze action, the operation command generation unit (upper module), Only need to specify whether to gaze allows tracking respect.

The control method of the robot apparatus according to the present invention is as follows.
The integrated information generating / storing process receives each attribute information recognized by the plurality of recognizing means, integrates it so as to maintain temporal and spatial consistency for each object, stores it with an identification number, and stores it. From the integrated information of the object stored in the integrated information creating / storing step, a predetermined motion that is a gaze target The creating process searches for integrated information of the object and operates based on the identification number of the object. When a command is generated and the motion generation step receives a motion command based on the identification number of the object from the motion command generation step, integrated information with the identification number of the object is added from the integrated information generation / storage step. Since it receives and selects the attribute information to be used for the gaze action and generates the control signal for the gaze action, the motion command generation unit (upper module) of the robot device specifies what to gaze at. Only it becomes possible tracking that.

[Brief description of drawings]

FIG. 1 is an external perspective view of a robot device.

FIG. 2 is a schematic diagram showing joint degrees of freedom of a robot apparatus.

FIG. 3 is a diagram schematically showing a control system configuration of a robot apparatus.

FIG. 4 is a diagram showing a basic architecture of a behavior control system of a robot device.

FIG. 5 is a diagram showing each module that constitutes the gaze behavior control system.

FIG. 6 is a diagram showing an example in which the robot apparatus tracks on Mr. A's face.

FIG. 7 is a diagram for explaining the situation setting of an embodiment in which the robot device tracks on Mr. A's face.

FIG. 8 is a diagram for explaining a detection algorithm of a face detector.

FIG. 9 is a diagram showing a color space.

FIG. 10 is a diagram showing a color region on a plane.

FIG. 11 is a diagram showing a color matching image (0).

FIG. 12 is a diagram showing a color matching image (1).

FIG. 13 is a diagram showing a color matching image (2).

[Figure 14] Newly detected color detection area new and Old (id = 10)
It is a figure which shows the overlap Overlap area | region with the color detection area of.

FIG. 15 is a diagram showing two color detection areas in an image every 10 frames.

FIG. 16 is a diagram for explaining detection of a sound source direction.

FIG. 17 is a diagram for explaining detection of a sound source direction.

FIG. 18 is a diagram schematically showing how the short-term storage unit operates.

FIG. 19 is a diagram showing up to the 29th frame in which the short-term storage section operates in the situation setting shown in FIG. 7;

20 is a diagram showing a 30th frame in which the short-term storage section operates in the situation setting shown in FIG.

FIG. 21 is a diagram showing up to the 50th frame in which the short-term storage section operates in the situation setting shown in FIG. 7.

22 is a diagram showing the 50th frame in which the short-term storage section operates in the situation setting shown in FIG. 7. FIG.

23 is a diagram showing the 51st frame in which the short-term storage section operates in the situation setting shown in FIG. 7.

FIG. 24 is a diagram showing a processing procedure of an operation command unit.

FIG. 25 is a diagram showing a detailed configuration of a neck motion generation unit.

FIG. 26 is a diagram for explaining the details of the operation of the neck operation generation unit.

FIG. 27 is a diagram for explaining the details of the operation of the neck operation generation unit.

FIG. 28 is a diagram for explaining the details of the operation of the neck operation generation unit.

FIG. 29 is a diagram for explaining the details of the operation of the neck operation generation unit.

[Explanation of symbols]

1 robot device, 70 gaze behavior control system, 71
Color detector, 72 Sound detector, 73 Face detector, 74
Joint angle detector, 75 short-term memory unit, 76 motion command unit, 77 neck motion generation unit

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Kenta Kawamoto             6-735 Kita-Shinagawa, Shinagawa-ku, Tokyo Soni             -Inside the corporation (72) Inventor Tomohisa Morihira             6-735 Kita-Shinagawa, Shinagawa-ku, Tokyo Soni             -Inside the corporation F term (reference) 3C007 AS27 AS32 AS36 CS08 KS01                       KS11 KS21 KS31 KS36 KS39                       KT01 KT04 LT06 LT08 MT08                       WA03 WA13 WB17 WB19

Claims (8)

[Claims] 1. In an autonomous robot apparatus that autonomously gazes at an object based on the recognition result for the external environment, a plurality of recognition means for recognizing attribute information regarding the object existing in the external environment, Integrated information generating and storing means for receiving each attribute information recognized by the recognizing means, integrating so as to maintain consistency temporally and spatially for each object, and storing with an identification number, Motion command generation means for searching the integrated information of the object stored in the generation storage means for integrated information of a predetermined object to be watched and generating an operation command based on the identification number of the object; When an operation command based on the identification number of the object is received from the operation command generation means, integrated information with the identification number of the object is received from the integrated information generation storage means. Robotic device characterized by comprising an operation generating means for generating a control signal for watching action by selecting the attribute information to be used for watching action. 2. The integrated information generation / storage means is assumed to be attribute information regarding the same object when the recognition results from at least two recognition means overlap in a predetermined range, and the same identification code. The robot device according to claim 1, wherein the robot device is stored as integrated information with a mark. 3. The integrated information generating / storing means is the same when a temporally new recognition result from one of the plurality of recognizing means overlaps with a previous recognition result within a predetermined range. 2. The robot apparatus according to claim 1, wherein it is determined that the target objects have the same attribute information as having moved. 4. The attribute information of the new object when the integrated information generation / storage means does not map a temporally new recognition result from one of the plurality of recognition means to a previous recognition result. The robot device according to claim 1, wherein the robot device is stored with an identification code as. 5. A control method for an autonomous robot apparatus that autonomously gazes at an object based on the recognition result for the external environment, comprising: a plurality of recognition means for recognizing attribute information about the object existing in the external environment. An integrated information generating and storing step of receiving each recognized attribute information, integrating so as to maintain temporal and spatial consistency for each object, and storing with an identification number, and the integrated information generating and storing step. An operation command generation step of searching for integration information of a predetermined target object to be watched from the stored integrated information of the target object and generating an operation command based on an identification number of the target object; When an operation command based on the identification number of the object is received from the generation step, the integrated information with the identification number of the object is received from the integrated information generation / storage step and used for the gazing action. And a motion generating step of generating control signals for gazing behavior by selecting attribute information to be used. 6. The integrated information generating / storing step, when the recognition results from at least two recognizing means are overlapped in a predetermined range, the attribute information is the same and the same identification code is used. 6. The control method for the robot apparatus according to claim 5, wherein the control information is stored as integrated information with a mark. 7. The integrated information generating and storing step is the same when a new recognition result temporally from one of the plurality of recognition means overlaps with a previous recognition result within a predetermined range. 6. The method for controlling a robot apparatus according to claim 5, wherein it is determined that the target objects have the same attribute information as having moved. 8. The integrated information generating and storing step includes attribute information of a new object when a temporally new recognition result from one of the plurality of recognizing means does not map to a previous recognition result. 6. The method for controlling a robot apparatus according to claim 5, wherein the identification number is added and stored.