JP2004291228A - Robot device, action control method of robot device and computer program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To make a robot conduct an action that does not make a user feel bored as an entertainment robot, and to make the robot learn the action conducted by itself. <P>SOLUTION: An action selection control system 100 has: a plurality of element actions that output an action when selected; an action value calculating part 120 that refers to a database in which an external stimulus to be inputted is made to correspond to an expected internal condition change expected to change after the action is conducted, and that calculates an action value AL of each of the element actions based on information from an internal condition control part 91 and an external stimulus recognition part 80; an action selection part 130 that selects the element action having the largest action value AL; and a learning part 140 that updates the database based on a result after the action is conducted. A predetermined internal condition and external stimulus are made to correspond to each action. The action value calculating part 120 calculates the action value AL for each action, based on a desire value for the action corresponding to the inputted internal condition and on a expected satisfaction level change on the basis of the expected internal condition change. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a robot device that expresses an action autonomously, a method of controlling the action of the robot device, and a computer program, and more particularly to a robot device and a robot device that select an action that expresses an internal state and an external stimulus. The present invention relates to a behavior control method and a computer program.

More specifically, the present invention relates to a robot apparatus which models emotions such as instinct and emotion, manages an internal state of the system, and selects an action according to a change in the internal state, a behavior control method of the robot apparatus, and a computer program In particular, the present invention relates to a robot apparatus for selecting an action in accordance with an interaction with a user or an environment with respect to an internal state and an external stimulus, an action control method of the robot apparatus, and a computer program.

  2. Description of the Related Art In recent years, research on human-coexisting robots and entertainment robots has been advanced as a new robot field in place of industrial robots. Such a robot device expresses a behavior using information of an external stimulus from a sensor or the like.

  For example, there has been proposed a method of generating a behavior of a robot based on an evaluation from the viewpoint of self-preservation of a robot device. The evaluation from the viewpoint of self-preservation is to evaluate the hardware from the viewpoint of the durability and failure rate of its own hardware. Specifically, it evaluates not only external information (visual and auditory sensor inputs) but also its own Calculates a self-preservation evaluation function that converts input information consisting of information such as battery status, power consumption, and circuit temperature into an evaluation value (failure rate) of body durability. In order to change the action state by actually giving a command to another action from the state in which the user is, the action state is changed by monitoring the internal / external sensory information and the value of the self-preservation evaluation function. Thereby, the behavior of the robot apparatus can reflect the intentionality (like / dislike) generated from the self-preservation evaluation that approaches the one where the self-preservation evaluation improves and avoids the one that worsens.

  In addition, this robot device introduces a self-preservation evaluation function for modeling "emotion". For example, when it is charged, it expresses yellow as a reaction of joy, and when it is discharged, it expresses blue as a fear response. By expressing the emotion, emotion expression can be performed.

  However, the values based on the “self-preservation” of this robot device are simple, such as good for charging and bad for discharging, and can only realize one-way communication with humans.

  In addition, many robots called human coexistence type robots and entertainment robots are toy-mechanically positioned, and only external stimuli are used as triggers for action selection. It is fixed. As a result, the user often easily feels bored with a toy that repeats only the same operation for the same external stimulus.

  Recently, a system that manages the internal state of the system by modeling emotions such as instinct and emotions and selects an action according to changes in the internal state has been proposed, but it is selected for internal states and external stimuli Behavior is often fixed and it is difficult to change it through interaction with users and the environment.

  If the robot device can be equipped with a function for predicting and performing the optimal next action and action according to the current situation, and a function for changing the next action and action based on past experience, further improvement will be made. This provides the user with a sense of closeness and satisfaction, further improves the amusement as the robot device, and facilitates the interaction with the user, which is convenient.

Tetsuya Ogata and Shigeki Sugano, "Robot Behavior Generation Based on Self-Conservation-Realization of Methodology and Mechanical Model-", Journal of the Robotics Society of Japan, 1997, Vol. 15, No. 5, p. 710-721

  An object of the present invention is to manage an internal state of a system by modeling emotions such as instinct and emotion, to select an action according to a change in the internal state, an excellent robot apparatus and a behavior control method of the robot apparatus, And to provide a computer program.

  A further object of the present invention is to provide an excellent robot apparatus, an action control method of the robot apparatus, and a computer program which can select an action according to an interaction with a user or an environment with respect to an internal state and an external stimulus. Is to do.

  A further object of the present invention is to provide an excellent robot apparatus, an action control method for a robot apparatus, and a computer, which are capable of expressing an action that does not make a user tired as an entertainment robot and capable of learning the action that the user develops.・ To provide programs.

The present invention has been made in consideration of the above problems, and a first aspect of the present invention is directed to a robot apparatus which autonomously selects and expresses an action based on an internal state and an external stimulus,
A plurality of action description modules in which actions associated with a predetermined internal state and an external stimulus are described,
An action value calculation database having a data format in which an input external stimulus is associated with an expected internal state change expected to change after the action is expressed,
The action value calculation database is referred to from the internal state and the external stimulus, the desire value for the action associated with the internal state and the degree of satisfaction based on the internal state are obtained, the desire value obtained from the current internal state, and the expected internal state change Action value calculating means for calculating the action value of the action described in each of the action description modules based on the expected satisfaction degree change obtained from
Action selecting means for selecting an action description module based on the calculated action value, and expressing the action described in the selected action description module;
Learning means for updating the action value calculation database based on the selected result after the action expression;
A robot apparatus comprising:

  In the present invention, a desire value for an action corresponding to the current internal state is obtained, and a degree of satisfaction with the current internal state and an expected internal state expected to change after an external stimulus is input and the action is expressed are calculated. The change in expected satisfaction, which is the difference from the expected satisfaction, is calculated, the action value is calculated from these, and the action is selected and expressed based on the action value. It can be updated from time to time by learning from the results. Therefore, various actions that are not unique to the internal state and various external stimuli that change in accordance with the environment and communication with the user can be expressed.

  Further, the action value calculation means calculates the action value for each action based on the desire value obtained from the current internal state, the satisfaction degree obtained from the current internal state, and the expected satisfaction degree change. May be.

  Further, the action value calculation database can have the expected internal state change associated with the value of the external stimulus, and when data that is not in the action value calculation database is input, a linear model is used. Then, the expected internal state change can be calculated by performing linear interpolation, and it is not necessary to have the expected internal state change corresponding to all the values of each external stimulus, so that the data amount can be reduced.

  Further, the learning means can learn the expected internal state change from the actual internal state change after the external stimulus is input and expresses the action, and a different database depending on a user who communicates or an environment. Can be created.

  Further, the action selecting means can select an action to be actually made to speak from various candidate action modules by various methods. For example, it is possible to adopt a method (Greedy) of always selecting a candidate action having the largest action value calculated by the action value calculation means.

  Alternatively, the action selecting means may randomly select from the candidate actions irrespective of the action value calculated by the action value calculating means. In such a case, the action selection becomes exploratory, and the possibility of updating the action value calculation database can be increased.

  Alternatively, the action selecting means may select from among the candidate actions according to the probability according to the action value calculated by the action value calculating means by SoftMax. In this case, an action having a large action value is selected with a higher probability.

  Further, the action value calculation database may manage the data format as a set of an action described in each of the action description modules, a characteristic of an object as an external stimulus, and an internal state. . In this case, the action value calculation means can search the action value calculation database using the action described in each of the action description modules as an index, and determine the internal state from the characteristics of the object as the external stimulus. .

  Further, as another method of using the action value calculation database, the action value calculation means searches the action value calculation database using a certain characteristic of an object as an external stimulus as an index to determine an internal state. May be. In this case, the action value calculating means can arbitrarily set or average values of the action or other characteristics of the object as an external stimulus to give the object an abstract value.

According to a second aspect of the present invention, there is provided a computer-readable format for executing, on a computer system, an action control of a robot apparatus for autonomously selecting and expressing an action based on an internal state and an external stimulus. In the described computer program,
A plurality of action description modules in which actions associated with a predetermined internal state and an external stimulus are described,
A step of managing an action value calculation database having a data format in which the input external stimulus and the expected internal state change expected to change after the action is expressed,
The action value calculation database is referred to from the internal state and the external stimulus, the desire value for the action associated with the internal state and the degree of satisfaction based on the internal state are obtained, the desire value obtained from the current internal state, and the expected internal state change An action value calculating step of calculating an action value of the action described in each of the action description modules based on the expected satisfaction degree change obtained from
Selecting an action description module based on the calculated action value, and an action selection step of expressing the action described in the selected action description module;
A learning step of updating an action value calculation database based on the selected result after the action is expressed;
A computer program characterized by the following.

  The computer program according to the second aspect of the present invention defines a computer program described in a computer-readable format so as to realize a predetermined process on a computer system. In other words, by installing the computer program according to the second aspect of the present invention in a computer system, a cooperative action is exerted on the computer system, and the robot apparatus according to the first aspect of the present invention. The same operation and effect can be obtained.

  According to the present invention, an excellent robot apparatus and a behavior control method for a robot apparatus, which can model emotions such as instinct and emotion, manage an internal state of the system, and select an action according to a change in the internal state, And a computer program.

  Further, according to the present invention, there is provided an excellent robot apparatus, an action control method of the robot apparatus, and a computer program capable of selecting an action according to an interaction with a user or an environment with respect to an internal state and an external stimulus. Can be provided.

  Further, according to the present invention, an excellent robot apparatus and an action control method for the robot apparatus, which can express a behavior that does not make the user tired as an entertainment robot and can learn an action that the user develops, and A computer program can be provided.

  The robot apparatus according to the present invention is a robot apparatus that autonomously selects and expresses an action based on an internal state and an external stimulus, and includes a plurality of action description modules in which the action is described, and an action value based on the internal state and the external stimulus. An action value calculating means for calculating the action value of the action described in each action description module with reference to the calculation database, and selecting an action description module based on the calculated action value; There is provided an action selecting means for expressing the action, and a learning means for updating the action value calculation database based on the selected result after the action is expressed.

The action described in each action description module is associated with a predetermined internal state and an external stimulus. The action value calculation database is a database in which an input external stimulus is associated with an expected internal state change that is expected to change after the action is expressed. Then, the action value calculating means obtains a desire value for the action associated with the internal state and a degree of satisfaction based on the internal state, and determines a desire value obtained from the current internal state and an expected satisfaction degree change obtained from the expected internal state change. Based on the above, the action value for each action is calculated.

That is, the desire value of the action that emerges from the own internal state is obtained, and the change in the degree of satisfaction expected after the action is expressed from the degree of satisfaction based on the current internal state is determined as the expected degree of satisfaction. By calculating and selecting an action based on this action value, different actions are selected depending on the value of the internal state even with the same external stimulus, and furthermore, the database used for the action value calculation can be updated at any time by learning. . Therefore, it is possible to develop an action that does not make various users who are not unique to the internal state and various external stimuli that change in accordance with the environment and communication with the user get tired.

  Further objects, features, and advantages of the present invention will become apparent from more detailed descriptions based on embodiments of the present invention described below and the accompanying drawings.

  Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings. The embodiment described in this specification is an application of the present invention to a robot device that simulates life of a pet-type agent, a human-type agent, and the like, and enables interaction with a user. Hereinafter, first, the configuration of such a robot apparatus will be described, then, among the control systems of the robot apparatus, an action selection control system that performs an action selection will be described, and finally, a robot apparatus including such an action selection control system A control system will be described.

A. The diagram 1 of the robot apparatus, and shows the appearance of the robot apparatus to be used for the practice of the present invention. As shown in the figure, the robot apparatus 1 has a head unit 3 connected to a predetermined position of a trunk unit 2, and two left and right arm units 4 R / L and two left and right leg units 5 R / L are connected to each other (however, each of R and L is a suffix indicating each of right and left. The same applies hereinafter).

  FIG. 2 schematically shows a functional configuration of the robot apparatus 1 provided for implementing the present invention. As shown in FIG. 1, the robot apparatus 1 includes a control unit 20 that performs overall control of the entire operation and other data processing, an input / output unit 40, a driving unit 50, and a power supply unit 60. . Hereinafter, each unit will be described.

  The input / output unit 40 is equivalent to a human eye as an input unit, and is disposed in a CCD (Charge Coupled Device) camera 15 for photographing an external situation, a microphone 16 corresponding to an ear, and a part such as a head and a back. When a predetermined pressure is received, a touch sensor 18 that senses a user's contact by electrically detecting this, a distance sensor for measuring a distance to an object located in front, and other elements corresponding to the five senses, Includes various sensors such as gyro sensors. Further, as an output unit, for example, an LED (Light Emitting Device) indicator provided in the head unit 3 and provided at the position of a human eye and a speaker 17 corresponding to a human mouth and expressing an emotional expression or a visual recognition state. (Eye lamp) 19 etc. are equipped. These output units can express the user feedback from the robot apparatus 1 in a format other than the voice, the blinking of the LED indicator 19, and the mechanical motion pattern by the legs.

  For example, a plurality of touch sensors 18 are provided at predetermined positions on the top of the head unit, and the touch detection by each of the touch sensors 18 is used in a complex manner, so that the user can work and “stroke” the head of the robot apparatus 1. "", "Tap", "Tap lightly", etc. can be detected. Further, when it is detected that some of the pressing sensors have sequentially contacted with a predetermined time interval, it is determined that the contact has been performed. , Etc., and the internal state changes accordingly, and such a change in the internal state can be expressed by the above-described output unit or the like.

  The drive unit 50 is a functional block that implements a body operation of the robot device 1 according to a predetermined movement pattern commanded by the control unit 20, and is a control target by behavior control. The drive unit 50 is a functional module for realizing a degree of freedom at each joint of the robot apparatus 1, and includes a plurality of drive units 541 to 54n provided on the roll, pitch, and yaw axes of each joint. You. Each of the drive units 541 to 54n includes a motor 511 to 51n for performing a rotation operation about a predetermined axis, encoders 521 to 52n for detecting a rotational position of the motor 511 to 51n, and motors 511 to 52n based on outputs of the encoders 521 to 52n. It is configured in combination with drivers 531 to 53n that adaptively control the rotation position and rotation speed of 51n.

  Although the robot device 1 according to the present embodiment is a bipedal walking, the robotic device 1 may be configured as a legged mobile robot device such as a quadrupedal walking, depending on how the drive units are combined.

  The power supply unit 60 is a functional module that supplies power to each electric circuit and the like in the robot device 1 as the name implies. The robot apparatus 1 according to the present embodiment is of an autonomous driving type using a battery, and a power supply unit 60 includes a charging battery 61 and a charging / discharging control unit 62 that manages a charging / discharging state of the charging battery 61. .

  The charging battery 61 is configured, for example, in the form of a “battery pack” in which a plurality of lithium ion secondary battery cells are packaged in a cartridge type.

  Further, the charge / discharge control unit 62 grasps the remaining capacity of the battery 61 by measuring a terminal voltage and a charge / discharge current amount of the battery 61, an ambient temperature of the battery 61, and determines a start time and an end time of charging. decide. The start and end timings of charging determined by the charge / discharge control unit 62 are notified to the control unit 20 and serve as triggers for the robot apparatus 1 to start and end charging operations.

  The control unit 20 corresponds to the “brain”, and is mounted on, for example, the body head or the body of the robot apparatus 1.

  FIG. 3 shows the configuration of the control unit 20 in more detail. As shown in FIG. 3, the control unit 20 has a configuration in which a CPU (Central Processing Unit) 21 as a main controller is bus-connected to a memory and other circuit components and peripheral devices. The bus 28 is a common signal transmission path including a data bus, an address bus, a control bus, and the like. Each device on the bus 28 is assigned a unique address (memory address or I / O address). The CPU 21 can communicate with a specific device on the bus 28 by specifying an address.

  A RAM (Random Access Memory) 22 is a writable memory including a volatile memory such as a DRAM (Dynamic RAM), and loads a program code executed by the CPU 21 or temporarily stores work data by the execution program. Used to save and so on.

  The ROM (Read Only Memory) 23 is a read-only memory that permanently stores programs and data. The program code stored in the ROM 23 includes a self-diagnosis test program executed when the power of the robot apparatus 1 is turned on, an operation control program for defining the operation of the robot apparatus 1, and the like.

  The control program of the robot apparatus 1 includes a “sensor input / recognition processing program” that processes sensor inputs from the camera 15 and the microphone 16 and recognizes the symbols as symbols, and performs storage operations (described later) such as short-term storage and long-term storage. An "action control program" that controls the action of the robot apparatus 1 based on the sensor input and a predetermined action control model, a "drive control program" that controls the drive of each joint motor and the audio output of the speaker 17 according to the action control model, and the like. Is included.

  The non-volatile memory 24 is configured by a memory element that can be electrically erased and rewritten, such as an EEPROM (Electrically Erasable and Programmable ROM), and is used to hold data to be sequentially updated in a non-volatile manner. The data to be sequentially updated includes an encryption key and other security information, a device control program to be installed after shipment, and the like.

  The interface 25 is a device for interconnecting with devices outside the control unit 20 and enabling data exchange. The interface 25 performs data input / output with the camera 15, the microphone 16, the speaker 17, and the like, for example. The interface 25 inputs and outputs data and commands to and from each of the drivers 531 to 53n in the drive unit 50.

  The interface 25 includes a serial interface such as a RS (Recommended Standard) -232C, a parallel interface such as an IEEE (Institute of Electrical and Electronics Engineers) 1284, a USB interface (Universal Serial I / E), a serial interface, and a serial interface (USB). , A general-purpose interface for connecting peripheral devices of a computer, such as a SCSI (Small Computer System Interface) interface, a memory card interface (card slot) for receiving a PC card or a memory stick, and the like. Between external devices It may be performed to move programs and data.

  As another example of the interface 25, an infrared communication (IrDA) interface may be provided to perform wireless communication with an external device.

  Further, the control unit 20 includes a wireless communication interface 26, a network interface card (NIC) 27, and the like, and performs near field wireless data communication such as Bluetooth, a wireless network represented by IEEE 802.11, or the Internet. Data communication can be performed with various external host computers via the wide area network.

  By such data communication between the robot device 1 and the host computer, complicated operation control of the robot device 1 can be calculated or remotely controlled using a remote computer resource.

B. Next, a behavior control method of the robot device will be described in detail. The above-described robot device 1 is configured to be able to autonomously act according to the situation of itself and the surroundings, and instructions and actions from the user. That is, the robot apparatus 1 can autonomously select and express an action according to the external stimulus and the internal state.

In the behavior control method of the robot device according to the present embodiment, the robot device expresses a behavior described in a behavior description module (described later) selected based on a behavior value AL (activation level: Activation Level) for the behavior. Action is generated. Here, among the behavior control of the robot apparatus, particularly, the action selection control until the action expressed in response to the internal state of the robot apparatus and the stimulus from the outside is selected and output will be described. A calculation method of the action value AL will be described. The overall configuration of the control system of the robot device will be described later in detail.

B-1. Action Selection Control of Robot Apparatus FIG. 4 shows an action selection control system part which performs a process of calculating an action value corresponding to each action and outputting an action based on the action value in the control system of the robot apparatus according to the present embodiment. ing. As shown in the figure, the action selection control system 100 includes an internal state management unit 91 that formulates and manages several types of emotions such as instinct and emotion, and an external stimulus recognition that recognizes external stimuli such as sensor inputs in an external environment. A unit 80, a plurality of elementary actions 132 that output an action when selected based on an internal state and an external stimulus, an action value calculation unit 120 that calculates an action value AL of each elemental action 132, and an action value AL And a learning unit 140 that learns based on the result after the action has occurred and updates the action value database, and the selected element action 132 outputs the action. By doing so, the robot device expresses an action.

  The elementary action 132 is configured as a module that determines an action output from an internal state and an external stimulus (hereinafter, also referred to as an “action description module”), includes a state machine for each module, and performs a previous action (action) or Depending on the situation, the recognition result of the external information input by the sensor is classified, and the operation is expressed on the machine. Although FIG. 4 shows only the element actions A to D, the present invention is not limited to this. The behavior description module performs a situation determination according to an external stimulus or an internal state, and calculates a behavior value AL. A Monitor function and a schema (Schema) having an Action function for realizing a state transition (state machine) accompanying the behavior execution. ), But details of the schema will be described later.

For each element action 132, a predetermined internal state and an external stimulus are defined according to the action described in the element action 132.

  Here, the external stimulus is, for example, perceptual information of the robot apparatus in the external stimulus recognizing unit 80, and includes, for example, target information such as color information, shape information, and face information processed on an image input from a camera. No. Specifically, for example, a color, a shape, a face, a 3D general object, and a hand gesture, as well as a motion, a voice, a contact, a distance, a place, a time, and the number of times of interaction with a user are included.

  As described above, the internal state is an emotion such as instinct and emotion managed by the internal state management unit 91. For example, fatigue (FATIGUE), pain (PAIN), nutritional state (NOURISHMENT), and dryness (THURRST) ), Affection (AFFECTION), curiosity (CURIOSITY), and the like.

  As shown in FIG. 4, each elementary action deals with an action output and an internal stimulus and an external stimulus defined according to the action output. The external stimulus is treated as a property of the target object. For example, the elementary action A whose action output is “eat” handles the type of the object (OBJECT_ID), the size of the object (OBJECT_SIZE), the distance of the object (OBJECT_DISTANCE), and the like as the external stimulus, and the internal state is “ NOURISHMENT "(" nutrition status ")," FATIGUE "(" fatigue "), and the like. In this way, the types of the external stimulus and the internal state to be handled are defined for each elemental action, and the action value for the action (elemental action) corresponding to the relevant external stimulus and the internal state is calculated. It is needless to say that one internal state or external stimulus may be associated with not only one elementary action but also a plurality of elementary actions.

  Further, the internal state management unit 91 receives an external stimulus and information such as the remaining amount of the battery of the own device and the rotation angle of the motor as inputs, and obtains the internal state values (internal state vector) corresponding to the plurality of internal states as described above. IntV) is calculated and managed. Specifically, for example, the internal state “nutrition state” can be determined based on the remaining amount of the battery, and the internal state “tired” can be determined based on the power consumption.

  The action value calculation unit 120 refers to the action value calculation database 121 (described later) in which the input external stimulus and the expected internal state change expected to change after the action is expressed are associated, and determines the external stimulus at a certain time. The action value AL of each of the elementary actions A to D at that time is calculated from the internal state. In the embodiment shown in FIG. 4, the action value calculation unit 120 is provided individually for each of the element actions A to D, but the action value calculation unit 120 calculates the action values for all the element actions. You may comprise.

  The action value AL here indicates how much the robot apparatus wants to perform the element action (execution priority). The action selection unit 130 selects one of the candidate elementary actions based on the action value AL calculated for each of the elementary actions. Then, the selected element action outputs the action described in itself. That is, each elementary action calculates its action value AL by its own action value calculation unit 120, and the action selection unit 130 selects the element action whose action value AL is, for example, the highest. A method of selecting an elementary action based on the action value AL in the action selection unit 130 will be described later in detail.

  The action value AL for each elementary action is a desire value for each action corresponding to each current internal state, a degree of satisfaction based on each current internal state, and a change amount of an internal state expected to change due to an external stimulus, ie, Is calculated based on the expected satisfaction degree change based on the expected internal state change indicating the amount of change in the internal state expected to change as a result of the input of the external stimulus and the appearance of the action.

  Here, regarding the method of calculating the action value AL in the element action, when an object of a certain “kind” and “size” exists at a certain “distance”, the action of the element action A whose action output is “eat” A specific description will be given of an example in which the value AL is calculated from the internal states “nutrition state” and “tired” defined in the element action A.

B-2. Action Value Calculation Unit FIG. 5 shows a flow of processing in which the action value calculation unit 120 calculates the action value AL from the internal state and the external stimulus. In the present embodiment, an internal state vector IntV (Internal Variable) having one or more internal state values as components is defined for each element action, and an internal state vector corresponding to each element action from the internal state management unit 91 is defined. Obtain IntV. That is, each component of the internal state vector IntV indicates a value of one internal state (internal state parameter) indicating, for example, the emotion described above, and the action value of the elemental action corresponding to each component of the internal state vector IntV. Used for calculation. Specifically, for the elementary action A having the action output “eat”, for example, the internal state vector IntVｔIntV_NOURISHMENT “nutrition state”, IntV_FATIGUE “tired”} is defined.

  In addition, an external stimulus vector ExStml (External Stimulus) having one or more external stimulus values as components is defined for each internal state, and the external stimulus recognition unit 80 outputs an external stimulus corresponding to each internal state, ie, each elemental action. Obtain the stimulus vector ExStml. That is, each component of the external stimulus vector ExStml indicates recognition information such as the size of the target object, the type of the target object, and the distance to the target object, and corresponds to each component of the external stimulus vector ExStml. Used to calculate internal state values. Specifically, for example, an external stimulus vector ExStml {OBJECT_ID “type of object”, OBJECT_SIZE “size of object”} is defined in the internal state IntV_NOURISHMENT “nutrition state”, and the internal state IntV_FATIGUE “fatigue” is defined. Defines, for example, an external stimulus vector ExStml {OBJECT_DISTANCE “distance to object”}.

  The action value calculator 120 receives the internal state vector IntV and the external stimulus vector ExStml as inputs, and calculates the action value AL. Specifically, the action value calculation unit 120 calculates, from the internal state vector IntV, a first calculation unit MV for obtaining a motivation vector (MotionVector) indicating how much the corresponding elemental action is to be performed, and an internal state vector IntV. And a second calculating unit RV for obtaining a releasing vector (ReleasingVector) indicating whether or not the corresponding elemental action can be performed from the external stimulus vector ExStml, and the action value AL is calculated from these two vectors.

B-2-1. The motivation vector, which is one element for calculating the action value AL, is calculated from the internal state vector IntV defined for the element action as a desire value vector InsV (Instinct Variable) indicating a desire for the element action. Desired. For example, the elementary action A having the action output “eating” has an internal state vector IntV {IntV_NOURISHMENT, IntV_FATIGUE}, and thereby obtains a desire value vector InsV {InsV_NOURISHMENT, InsV_FATIGUE} as a motivation vector. That is, the desire value vector InsV is a motivation vector for calculating the action value AL.

  As a calculation method of the desire value vector InsV, for example, as the value of the internal state vector IntV is larger, it is determined that the desire is satisfied, and the desire value is smaller. When the internal state vector IntV is larger than a certain value, the desire value is A function that becomes negative can be used.

  Specifically, there are functions as shown in the following equation (1) and FIG. In FIG. 6, the horizontal axis represents each component of the internal state vector IntV, and the vertical axis represents each component of the desire value vector InsV. This is shown in the graph.

  The desire value vector InsV is determined only by the value of the internal state vector IntV, as shown in the above equation (1) and FIG. Here, a function is shown in which the magnitude of the internal state is 0 to 100 and the magnitude of the desire value at that time is -1 to 1. For example, by setting an internal state-desired value curve L1 such that the desire value becomes 0 when the internal state is satisfied by 80%, the robot apparatus always has a state in which the satisfaction degree of the internal state is 80%. You will choose actions to maintain. This indicates that, for example, when the desire corresponding to the internal state “nutritional state” (IntV_NORISHMENT) is “appetite” (InsV_NORISFMENT), the appetite increases when the stomach is reduced, and the appetite disappears after the eighth minute. By using this, it is possible to express an action that expresses such an emotion.

  By variously changing the constants A to F in the above equation (1), different desire values are obtained for each internal state. For example, while the internal state is between 0 and 100, the desire value may be changed from 1 to 0. Further, for each internal state, an internal state-desired value function different from the above equation (1) may be defined. May be prepared.

B-2-2. Calculation of Releasing vector Meanwhile, Releasing vector as the other elements for calculating the activation level AL is the satisfaction vector S obtained from the internal state vector IntV (Satisfaction), expected satisfaction obtained from an external stimulus vector ExStml It is calculated from the degree change vector.

  First, from the internal state defined for each elemental action and the external stimulus defined for this internal state, a predicted internal state indicating the difference between the internal state expected to be obtained after the action is expressed and the current internal state. The state change vector is obtained by the following equation (2).

  The expected internal state change amount vector indicates a change amount that is expected to change after the appearance of the action from the current internal state vector, and is used to calculate the action value in the action value calculation database 121 that can be referred to by the action value calculation unit 120. It can be determined by referring to the data. The action value calculation data describes the correspondence between the external stimulus vector and the expected internal state change amount vector expected to change after the action appears, and by referring to this database, the action value calculation unit 120 An expected internal state change vector corresponding to the input external stimulus vector can be obtained.

  The configuration of the action value calculation database 121 will be described later in detail. Further, in the action selection control system 100 of the robot apparatus according to the present embodiment, the learning unit 140 learns a predicted internal state change predicted before the action onset from the internal state change actually changed after the action onset, and calculates the action value. It has a function of updating the database, and details of this learning function will be described later. Here, first, a method of obtaining the expected internal state change and the expected desire value change from the action value calculation database will be described.

  As the action value calculation data registered in the action value calculation database, for example, data shown in FIGS. 7A and 7B can be considered.

In the example illustrated in FIG. 7A, as for the internal state “nutrition state” (“NOURISHMENT”), the size of the target object (OBJECT_SIZE) is large as a result of expressing the action output of the elementary action “eating”. The more the internal state “nutritional state” is satisfied, the larger the amount that is expected to satisfy the nutrition. Similarly, from the object M1 whose object type (OBJECT_ID) corresponds to OBJECT_ID = 0, to the object M2 corresponding to OBJECT_ID = 1, and from the object M2 corresponding to OBJECT_ID = 1 to OBJECT_ID = 2. This shows a case where the corresponding target object M3 has a larger amount of satisfying the internal state “nutrition state” and is expected to satisfy nutrition.

  In addition, in the example illustrated in FIG. 7B, as for the internal state “fatigue” (“FATIGUE”), as a result of expressing the action output “eating” of the elementary action, the distance “OBJECT_DISTANCE” of the target object is large. The more the internal state “FATIGUE” is satisfied, the greater the amount that the internal state “FATIGUE” is satisfied.

  That is, as described above, since the internal state vector IntV and the external stimulus vector ExStml are defined for each action element, the vector having the size of the target object and the type of the target object as each component of the external stimulus vector ExStml Is supplied, the expected internal state change with respect to the action result of the element action A in which the internal state vector having the internal state IntV_NOURISHMENT (“nutrition state”) in which the external stimulus vector ExStml is defined is obtained. Can be Similarly, when a vector having the distance to the object is supplied, the action result of the element action A in which the internal state vector having the internal state IntV_FATIGUE (“tired”) in which the external stimulus vector ExStml is defined is defined. Expected internal state change with respect to.

  Next, the satisfaction degree vector S shown in the following equation (3) is calculated from the internal state vector IntV, and the expected satisfaction shown in the following equation (4) is calculated from the expected internal state change vector shown in the above equation (2). Find the degree change vector.

  The calculation method of the satisfaction level vector S for the internal state vector IntV is as follows for each component IntV_NOURISHMENT “nutrition state” and IntV_FATIGUE “fatigue” of the internal state vector {IntV_NOURISHMENT, IntV_FATIGUE} defined in the element action A. A function such as shown in Expressions (5-1) and (5-2) can be considered.

  FIG. 8 and FIG. 9 are graphs showing the functions shown in the above equations (5-1) and (5-2), respectively. In FIG. 8, the horizontal axis indicates IntV_NOURISHMENT “nutrition state”, the vertical axis indicates the degree of satisfaction S_NOURISHMENT for the internal state “nutrition state”, and FIG. 9 indicates the horizontal axis indicates IntV_FATIGUE “fatigue” and the vertical axis indicates internal state “fatigue”. The degree of satisfaction S_FATIGUE is taken to show the relationship between the internal state and the degree of satisfaction.

  The function shown in FIG. 8 is a function in which the value of the internal state “nutritional state” IntV_NOURISHMENT has a value of 0 to 100, and the corresponding degree of satisfaction S_NOURISHMENT has a positive value of 0 to 1, The curve L2 shows that the degree of satisfaction increases from 0 when the value of the state is from 0 to around 80, and thereafter decreases and the value of the internal state becomes 100 and the degree of satisfaction becomes 0 again. That is, as for the internal state “nutrition state”, the satisfaction degree S_NOURISHMENT calculated from the value of the current (a certain time) internal state “nutrition state” (IntV_NOURISHMENT = 40), and the internal state “nutrition” obtained from FIG. The expected satisfaction change corresponding to the expected internal state change of “state” (20 from 40 to 80) is both positive.

  Although only the function L2 is shown in FIG. 5 described above, a function as shown in FIG. 9 can be used. That is, the value of the internal state “fatigue” IntV_FATIGUE has a value of 0 to 100, and the corresponding degree of satisfaction S_FATIGUE is a negative value from 0 to −1, and the larger the internal state, This is a function indicating a curve L3 that reduces the degree of satisfaction. The degree of satisfaction S_FATIGUE calculated from the value of the internal state of the current internal state “fatigue” is negative, and if the expected internal state change of the internal state “fatigue” obtained by FIG. The degree change vector becomes negative.

  In the functions represented by the above equations (5-1) and (5-2), by setting each of the constants A to F variably, a function for obtaining different degrees of satisfaction corresponding to various internal states is obtained. Can be set.

  Then, by determining the value of how much the internal state is satisfied after the onset of the action in response to the external stimulus according to the following equation (6), the releasing element, which is the other element for calculating the action value AL, is determined. Can be requested.

  Here, if α in the above equation (6) is large, the releasing vector indicates an expected satisfaction change indicating how much satisfaction is obtained as a result of expressing the action, that is, how much satisfaction increases. If α is small, α tends to be strongly dependent on the value of expected satisfaction, that is, a value indicating the degree of satisfaction as a result of expressing an action.

B-2-3. Calculation of Action Value AL From the motivation vector and the releasing vector obtained as described above, the action value AL is finally calculated as in the following equation (7).

  Here, if β is large, the action value AL tends to strongly depend on the internal state (desired value), and if β is small, it tends to strongly depend on the external stimulus (expected satisfaction change and expected satisfaction). In this way, the desire value, satisfaction, and expected satisfaction are calculated from the value of the internal state (internal state vector IntV) and the value of the external stimulus (external stimulus vector ExStml), and the desired value, satisfaction, and expected satisfaction are calculated. The action value AL can be calculated based on the degree.

  Then, based on the action value AL, the action selecting unit 130 selects an action, so that, even when the same external stimulus is input, a different action is selected depending on the value of the internal state at that time. .

B-2-4. Action Value Calculation Database Next, the structure of the action value calculation data of the action value calculation database 121 and a method of referring to the database (a method of obtaining an expected internal state change) will be described.

As described above, the action value calculation data is data for obtaining an expected internal state change amount vector for an input external stimulus, and an external stimulus vector space for an internal state defined for each elemental action. A representative point (a value of an external stimulus) is defined above, and an expected internal state change indicating an expected amount of change in the internal state is defined on the representative point. When the input external stimulus is a value on the representative point in the defined external stimulus vector space, the expected internal state change becomes a value defined on the representative point.

  FIGS. 10A and 10B show an example of an action value calculation data structure.

As shown in FIG. 10 (a), when the expected internal state change of the internal state “nutrition state” (“NOURISHMENT”) is determined, the representative point {OBJECT_ID, OBJECT_SIZE} on the external stimulus vector space and this representative point are corresponded. The expected internal state change is defined, for example, as shown in Table 1 below.

  In addition, as shown in FIG. 10B, when an expected internal state change vector of the internal state “fatigue” (“FATIGUE”) is obtained, a representative point {OBJECT_DISTANCE} on the external stimulus vector space and this representative point The expected internal state change is defined, for example, as shown in Table 2 below.

  As described above, since the expected internal state change is defined only at the representative point on the external stimulus vector space, depending on the type of the external stimulus (for example, OBJECT_DISTANCE or OBJECT_SIZE), the representative internal stimulus vector space is defined. A value other than a point may be input. In that case, the expected internal state change can be obtained by linear interpolation from representative points near the input external stimulus.

  11 and 12 illustrate a linear interpolation method of one-dimensional and two-dimensional external stimuli, respectively.

When the expected internal state change is obtained from one external stimulus (OBJECT_DISTANCE) as shown in FIG. 10 (b), that is, when one external stimulus is defined for the internal state, it is shown in FIG. Thus, the horizontal axis represents the external stimulus, and the vertical axis represents the expected internal state change with respect to the external stimulus. The predicted internal state change defined by the representative point D1 and the representative point D2 which are parameters of the external stimulus (OBJECT_DISTANCE). The expected internal state change amount In of the input external stimulus Dn can be obtained from the straight line L4 as follows.

  In addition, as shown in FIG. 12, when an external stimulus which is an input to the internal state is defined as an external stimulus vector from two components, for example, an external stimulus called OBJECT_WEIGHT is defined for the internal state in addition to OBJECT_DISTANCE. If so, representative parameters (D1, W1), (D1, W2), (D2, W1), (D2, W2), which are predetermined parameters of each external stimulus, are defined, and the corresponding expected internal state When there is a change, when an external stimulus Enm (Dn, Wn) different from the above representative point is input, for example, first, when OBJECT_DISTANCE = D1, the prediction defined in the representative points W1 and W2 of OBJECT_WEIGHT A straight line L5 passing through the internal state change is obtained, and OBJECT_DISTANCE is similarly obtained. In D2, it obtains a straight line L6 passing through the predicted internal state change defined in the representative point W1 and W2 of OBJECT_WEIGHT. Then, of the two inputs of the external stimulus Enm to be input, for example, the expected internal state change in two straight lines L5 and L6 corresponding to Wn is obtained, and further, a straight line L7 connecting the two expected internal state changes is obtained. By calculating the expected internal state change Inm corresponding to the other external stimulus Dn of the external stimulus Enm input on the straight line L7, the expected internal state change corresponding to the external stimulus Enm can be obtained by linear interpolation.

  Thus, when data that is not in the action value calculation database is input, the expected internal state change can be calculated by performing linear interpolation using a linear model, and it corresponds to all the values of each external stimulus. It is not necessary to have the expected internal state change, and the data amount can be reduced.

  As shown in FIG. 10, the action value calculation database according to the present embodiment has a data format including a set of an action described in each action description module, a property of an object as an external stimulus, and an internal state. , And action value calculation data.

  In this case, the action value calculation database 121 is searched using the action described in each action description module (schema) as an index, and the internal state can be determined from the property of the object as the external stimulus.

  Further, as another method of using the action value calculation database 121, the action value calculation database 121 may be searched by using a certain property of an object as an external stimulus as an index to determine an internal state. In this case, the behavior or other characteristics of the object as an external stimulus can be set as a value or averaged in the property to give the object an abstract value. A method of using the action value calculation database 121 in this case will be described with reference to FIG.

  From the action value database of each elemental action, an abstract internal state change that does not depend on the action on the property of the target object is calculated for the property of the target object to which a certain element action pays attention. For example, the property of the target object of the element action “tremble” is “color”, and the internal state is “PAIN”. At that time, the expected internal state change is calculated from the color using the action value database of the other elementary actions (eating, kicking, talking, etc.) focusing on the property “color” and the internal state “PAIN” of the object. .

  As a calculation method at this time, an average value of dIntV_PAIN to COLORΣdIntV_PAIN (COLOR) / n. If dIntV_PAIN is not uniquely determined for COLOR in the action value database of a certain element action, for example, if dIntV_PAIN (COLOR, OBJECT_ID, DISTANCE) or the like, OBJECT_ID uses a representative value, and DISTANCE uses an average value. Then, the action value AL is calculated from the internal state value IntV and the expected change amount dIntV.

  As a result of calculating such a behavioral value AL, if you have a behavioral value database such as "I get hurt by kicking a red ball" or "I get hurt by talking to a red person", just looking at the red color "PAIN is recalled and trembles" is realized, and the abstract value (internal state change) of the property of the object irrespective of the type of action can be determined.

B-2-5. Action Selection The conventional action selection method for a behavior-based robot is to determine the activity by a heuristic method, select the action module that gives the maximum value of the activity, and express the action. However, depending on whether the calculation method of the activity is manually and the calculation is set well below, the behavior of the self-supporting robot may be appropriate or inappropriate.

  On the other hand, in the present embodiment, a behavior norm type system is configured in which an appropriate action is selected for a new external stimulus by associating a change in the internal state with the external stimulus. Further, the associated change in the internal state is stored in each action module, and this is converted into a numerical value as an action value AL using the evaluation of pleasure or discomfort, that is, the degree of satisfaction based on the internal state, and it is regarded as a value function of reinforcement learning. An appropriate action selection is realized in the framework of reinforcement learning based on maximum value selection or SoftMax selection.

  In the present embodiment, each elementary behavior calculates its behavioral value AL by its own behavioral value calculation unit 120.

As described above, in the action value calculation database 121, the expected internal state change amount ΔI (ES, B _i ) when each of the elementary actions B _i is executed in response to the input of the external stimulus ES is determined by the external stimulus. It is stored in association with the ES. Then, the elementary action B _i, is recalled internal state I based on predicted internal state variation [Delta] I ([Delta] I) is able to evaluate the satisfaction degree p at that time from the internal state (i.e., p = f (I + [Delta] I) ). Here, the degree of satisfaction can be regarded as a value function or a reward or an expected value for the elementary action B _i , and is treated as an action value AL in this specification.

In this way, the action value AL is calculated for all of the candidate elementary actions B _i , and the action selection unit 130 selects an action to be expressed from the candidate elementary actions based on the calculated action value AL. I do. For example, even when the same external stimulus is input, different actions are selected depending on the value of the internal state at that time.

For example, the k-th elementary action B _k is selected by the action selecting unit 130, and is actually expressed. At this time, the system can obtain the change amount ΔI of the actual internal state _obtained by actually expressing the elementary action B _k , but based on the actually measured value, the corresponding expected value in the action value calculation database 121 is obtained. The internal state change amount ΔI can be updated, that is, learned. By the learning action of the action value calculation database, it is possible to set a function for calculating an appropriate action value for all actions.

Learning of the action value calculation database will be described later. In this section, a selection method by the action selection unit 130 will be described.

  The action selecting unit 130 selects each element action based on the calculated action value AL. The present inventors consider, for example, the following as the action selection policy.

(1) Greedy
The action selection unit always selects the element action having the highest action value among the candidate element actions.

(2) Random
The action selecting unit 130 selects an element action at random regardless of the action value. As a result, the action selection becomes exploratory, and the possibility of updating the action value calculation database 121 increases.

(3) SoftMax
The action selecting unit 130 selects from the candidate actions according to the probability according to the calculated action value AL. Specifically, among the candidate elementary actions, an elementary action having a high action value is selected with a higher probability. For example, when the action value of an element action i is ALi, the selection probability P (i) is calculated by the following equation.

  However, in the above equation, T is a parameter called Boltzmann temperature, and the characteristics of the selection method using SoftMax can be adjusted. That is, the above equation is a form in which Randomness is given to the above-mentioned Greedy, and the Boltzmann temperature T is a measure of randomness. When T is sufficiently small, the action selection policy becomes Greedy, and when T is sufficiently large, it becomes Random (see FIG. 30).

  However, in the above (1) to (3), elementary actions (for example, a shot against a soccer ball) that are not established unless the target object is present are excluded from the options of the action selection unit.

B-3. Learning of Action Value Calculation Database Next, a learning method for learning the expected internal state change vector of such action value calculation data from the internal state change vector after the occurrence of the action will be described.

As described above, since the robot apparatus according to the present embodiment has a learning function, the action value calculation database is updated at any time according to the interaction with the user and the external environment. Therefore, depending on the learning result of the robot device, action generation that does not tire the user, such as expressing different actions even when receiving the same external stimulus, is realized.

To learn such action value calculation data, a teacher signal is required. In the present embodiment, an expected internal state change corresponding to an external stimulus is learned using an actual internal state change obtained from the result of the action as a teacher signal. Therefore, as shown in FIG. 4 described above, the action selection control system 100 includes a learning unit 140 connected to the action value calculation unit 120.

  FIG. 13 shows a flow from the input of the external stimulus to the learning of the action value calculation database. As described with reference to FIG. 5, the action selecting unit 130 refers to the action value calculation database 121, uses the predicted internal state change vector as the student signal, calculates the action value AL, and based on this value, For example, the element action having the highest action value AL is selected. The selected element action outputs an action, and the robot device expresses the action.

  As a result of the robot apparatus actually exhibiting an action, the internal state management unit 91 shown in FIG. 4 changes its own internal state. That is, for example, the internal state is changed in accordance with the lapse of time after the action has appeared, or the internal state is changed in accordance with the result of the action. Specifically, for example, as described above, if the internal state “nutrition state” is determined based on the remaining amount of the battery, and the internal state “tired” is determined based on the power consumption, the behavior is expressed. As a result, the internal state “nutrition state” also decreases due to a decrease in the remaining battery power, and the internal state “fatigue” increases in proportion to the amount of power consumed by expressing behavior. I do.

  As a result of actually expressing the behavior in this manner, the internal state of the robot device changes, and the amount of change in the internal state before and after the actual behavior is expressed can be obtained. Then, as shown in FIG. 13, the internal state change vector dIntV obtained after the appearance of the action becomes a teacher signal, and the learning unit 140 learns an expected internal state change vector predicted before the appearance of the action. The value calculation database 121 is updated according to the learning.

  The learning method differs depending on whether or not the external stimulus input here is a value on the representative point. First, when the external stimulus required for calculating the action value in a certain selected elemental action is a value on the representative point, the representative point is calculated based on the actual internal state change amount by the following equation (8). Update the expected internal state change above.

  If the external stimulus required for calculating the action value is a value other than the representative point in a selected elemental action, the expected internal state change at the representative point near the external stimulus, that is, at the representative point used for linear interpolation, Become learning targets. For each external stimulus, the ratio of the distance between the external stimulus and the representative point is multiplied by equation (8) to update the expected internal state change.

  14 and 15 show examples of updating expected internal state changes of one-dimensional and two-dimensional external stimuli, respectively. As shown in FIG. 14, when the external stimulus Dn is input and the external stimulus Dn is not on the representative point, linear interpolation is performed using the representative points D1 and D2 near the external stimulus Dn as described above. , An expected internal state change In before the action is expressed is obtained. Then, after the action is manifested, the actual internal state change amount (dIntV_Fatigue) is obtained, and the expected internal state change at the representative points D1 and D2 is obtained from the distance between the representative points D1 and D2 and the external stimulus Dn and the learning rate γ. Is performed, and the expected internal state changes of the representative points D1 and D2 are updated according to the following equations (9-1) and (9-2).

  When two external stimuli are input, as shown in FIG. 15, when the expected internal state change amount Inm corresponding to the external stimulus Enm (Dn, Wn) before the action is expressed is determined by linear interpolation. The estimated internal state changes corresponding to the used representative points (D1, W1) (D1, W2), (D2, W1), (D2, W2) near the external stimulus to be input are represented by the following equation (10-1). ) To (10-4) are learned and updated. That is, learning is performed based on the actual internal state change vector obtained after the action is expressed, the distance between the representative point and the external stimulus, and the learning rate γ, and the expected internal state change corresponding to each representative point is updated.

B-4. Action Value Calculation Method and Learning Method of Action Value Calculation Database Next, the action value calculation method in the action value calculation section 120 shown in FIG. 5, and the action value according to the action expressed by the learning section 140 shown in FIG. A method of updating the calculation database will be described with reference to flowcharts shown in FIGS.

  As shown in FIG. 16, first, when the external stimulus is recognized by the external stimulus recognition unit 80 shown in FIG. 4, this is supplied to the action value calculation unit 120. At this time, each internal state is supplied from the internal state management unit 91 by, for example, a notification from the external stimulus recognition unit 80 (step S1).

  Next, as described above, a corresponding desire value is calculated from each of the supplied internal states using a function such as the above-described equation (1), so that the internal state vector IntV becomes a motivation vector. A desire value vector is calculated (step S2).

  Further, the action value calculating unit 120 calculates a corresponding degree of satisfaction from each of the supplied internal states using a function such as Expressions (5-1) and (5-2), thereby obtaining an internal state vector. A satisfaction level vector S is calculated from IntV (step S3).

  On the other hand, from the supplied external stimulus (external stimulus vector), an expected internal state change expected to be obtained as a result of the action as described above is obtained (step S4). Then, using the same function as in step S3, the expected satisfaction change corresponding to the expected internal state change is obtained (step S5), and the expected satisfaction change obtained and the satisfaction vector obtained in step S3 are determined. A releasing vector is calculated by the above equation (6) (step S6).

  Finally, the action value AL is calculated from the above equation (7) from the motivation vector obtained in step S2 and the releasing vector obtained in step S6 (step S7).

  In the above-described steps S1 to S7, it has been described that the action value AL is calculated by the action value calculation unit 120 every time the external stimulus is recognized. However, for example, the action value is calculated at a predetermined timing. Is also good.

  Thereafter, based on the action value AL calculated in step S7, as shown in FIG. 17, the action selection unit 130 monitors the action value calculation results for all the element actions, for example, the element action with the highest action value AL. Is selected, the action is output from the element action (step S8). As described above, in addition to Greedy, a method such as Random or SoftMax can be adopted for selecting an element action.

  In the robot device, for example, the behavior of the robot causes the remaining amount of the battery to change, and the internal state calculated based on the remaining power changes compared to before the behavior. In addition, the internal state calculated based on the power consumption or the like used when the behavior is expressed changes as compared to before the behavior was expressed. The internal state management unit 91 illustrated in FIG. 4 calculates such a change in the internal state before and after the action, and supplies the change to the learning unit 140 (Step S9). As described above, the learning unit 140 determines, from the actual change in the internal state before and after the action and the expected internal state change stored in the action value calculation database, the equations (9-1) and (9-2). ) Or (10-1) to (10-4), a new expected internal state change is calculated, and the action value calculation database is updated (step S10).

C. Robot System Control System In this section, a specific example in which a behavior selection control system for calculating a behavior value AL and outputting a behavior is applied to a robot system control system will be described in detail.

FIG. 18 schematically shows a functional configuration of the control system 10 including the above-described action selection control system 100. As described above, the illustrated robot apparatus 1 can perform behavior control in accordance with the recognition result of an external stimulus and a change in an internal state. Furthermore, by providing a long-term memory function and associatively storing a change in an internal state from an external stimulus, behavior control can be performed according to a recognition result of the external stimulus or a change in the internal state.

  That is, as described above, for example, color information, shape information, face information, and the like processed on an image input from the camera 15, and more specifically, color, shape, face, 3D general object, The action value AL is calculated according to an external stimulus composed of components such as hand gesture, movement, voice, contact, smell, and taste, and an internal state indicating emotions such as instinct and emotion based on the body of the robot device, Select (generate) an action and express it.

  The instinctive elements of the internal state include, for example, fatigue, heat or temperature, pain, appetite or hunger, third, affection, curiosity. , Excretion or sexual desire. The emotional elements include happiness, sadness, anger, surprise, disgust, fear, frustration, boredom, and sleepiness. ), Sociability (greariousness), patience, tension, relaxed, alertness, Guilt, malice (spite), loyalty, submission or Jealousy and the like.

  The illustrated control system 10 can be implemented using object-oriented programming. In this case, each software is handled in a module unit called an “object” that integrates data and a processing procedure for the data. Further, each object can perform data transfer and Invoke by message communication and an inter-object communication method using a shared memory.

  The behavior control system 10 according to the present embodiment is a functional module including a visual recognition function unit 81, a hearing recognition function unit 82, a contact recognition function unit 83, and the like, in order to recognize an external environment (Environments) 70. An external stimulus recognition unit 80 shown in FIG. 4 is provided.

  The visual recognition function unit (Video) 81 performs image recognition processing such as face recognition and color recognition and feature extraction based on a captured image input via an image input device such as a CCD camera, for example.

  The auditory recognition function unit (Audio) 82 performs voice recognition of voice data input via a voice input device such as a microphone to extract features or perform word set (text) recognition.

  The contact recognition function unit (Tactile) 83 recognizes a sensor signal from a contact sensor built in, for example, the head of the body, and recognizes an external stimulus such as “patched” or “hit”.

  An internal state manager (ISM: Internal Status Manager) 91 has an emotion / instinct model for managing several types of emotions such as instinct and emotion by mathematical modeling, and includes the above-described visual recognition function section 81 and auditory recognition function section. The internal state such as instinct and emotion of the robot apparatus 1 is managed in accordance with the external stimulus (ES: External Stimula) recognized by the touch recognition function unit 83 and the touch recognition function unit 83.

  The emotion / instinct model has a recognition result and an action history as inputs, and manages emotion values and instinct values, respectively. The behavior model can refer to these emotion values and instinct values.

  In addition, in order to perform action control according to a recognition result of an external stimulus or a change in an internal state, information is compared with a short term memory (STM: Short Term Memory) 92 that stores a short term memory that is lost over time. It has a long term memory (LTM: Long Term Memory) 93 for holding for a long period of time. The classification of short-term memory and long-term memory depends on neuropsychology.

  The short-term storage unit 92 is a functional module that holds a target or an event recognized from the external environment by the visual recognition function unit 81, the auditory recognition function unit 82, and the contact recognition function unit 83 for a short period of time. For example, the input image from the camera 15 shown in FIG. 2 is stored for a short period of about 15 seconds.

  The long-term storage unit 93 is used to hold information obtained by learning, such as the name of an object, for a long time. For example, the long-term storage unit 93 can associate and store a change in an internal state from an external stimulus in a certain behavior description module.

  In addition, the behavior control of the robot apparatus 1 is performed by a “reflexive behavior layer 103” and a “reflex behavior” realized by a reflexive behavioral behavior layer 103, and by a “situation-dependent behavior layer (SBL)”. Behavior "and" contemplation behavior "realized by the Deliberative Layer 101.

  The reflex action unit 103 is a functional module that implements a reflexive body operation in response to an external stimulus recognized by the visual recognition function unit 81, the auditory recognition function unit 82, and the contact recognition function unit 83 described above. The reflex action is basically an action of directly receiving a recognition result of external information input from a sensor, classifying the result, and directly determining an output action. For example, it is preferable to implement a behavior such as chasing or nodding a human face as a reflex action.

  The situation-dependent behavior hierarchy 102 is based on the storage contents of the short-term storage unit 92 and the long-term storage unit 93 and the internal state managed by the internal state management unit 91, and the behavior corresponding to the situation where the robot apparatus 1 is currently placed. Control.

  The situation-dependent action hierarchy 102 prepares a state machine for each action (element action), classifies the recognition result of the external information input by the sensor depending on the action or situation before that, and performs the action. Is expressed on the fuselage. The situation-dependent behavior hierarchy 102 also implements an action for keeping the internal state within a certain range (also referred to as “homeostasis behavior”). When the internal state exceeds a specified range, the internal state is regarded as the relevant state. The action is activated so that the action for returning to the range easily appears (actually, the action is selected in consideration of both the internal state and the external environment). Situation-dependent behavior has a slower reaction time than reflex behavior. This situation-dependent behavior hierarchy 102 corresponds to the element behavior 131, the behavior value calculation unit 120, and the behavior selection unit 130 in the behavior selection control system 100 shown in FIG. 4 described above, and as described above, the behavior value is determined from the internal state and the external stimulus. AL is calculated, and action output is performed based on the AL.

  The reflection behavior hierarchy 101 performs a relatively long-term action plan of the robot apparatus 1 based on the storage contents of the short-term storage unit 92 and the long-term storage unit 93. Reflection behavior is behavior that is performed based on a given situation or a command from a human, with a reasoning or a plan for realizing it. For example, searching for a route from the position of the robot device and the position of the target is equivalent to deliberate behavior. Such inferences and plans may require more processing time and calculation load (that is, more processing time) than the reaction time for the robot apparatus 1 to maintain the interaction. Reflective actions make inferences and plans while responding in real time.

  The reflection behavior hierarchy 101, the situation-dependent behavior hierarchy 102, and the reflex behavior unit 103 can be described as higher-level application programs independent of the hardware configuration of the robot device 1. On the other hand, the hardware dependent layer control unit (Configuration Dependent Actions And Reactions) 104 responds to commands from these higher-level applications, that is, the behavior description module (schema), and controls the hardware (such as the drive of the joint actuator) of the body. External environment) directly. With such a configuration, the robot device 1 can determine its own and surrounding conditions based on the control program, and can act autonomously according to instructions and actions from the user.

  Next, the behavior control system 10 will be described in more detail. FIG. 19 is a schematic diagram illustrating an object configuration of the behavior control system 10 in this specific example. As shown in the figure, the visual recognition function unit 81 is composed of three objects: a Face Detector 114, a Multi Color Tracker 113, and a Face Identify 115.

  The Face Detector 114 is an object that detects a face area from within an image frame, and outputs a detection result to a Face Identify 115. The Multi Color Tracker 113 is an object that performs color recognition, and outputs a recognition result to the Face Identify 115 and the Short Term Memory (STM) 92. Further, the Face Identify 115 identifies the person by searching the detected face image using a person dictionary held by the user, and outputs the ID information of the person together with the position and size information of the face image area to the STM 92.

  The auditory recognition function unit 82 is composed of two objects, Audio Recog 111 and Speech Recog 112. The Audio Recog 111 is an object that receives voice data from a voice input device such as a microphone and performs feature extraction and voice section detection, and outputs the feature amount and sound source direction of voice data in the voice section to the Speech Recog 112 and the STM 92. The speech recog 112 is an object that performs speech recognition using the speech feature amount received from the audio recog 111, the speech dictionary, and the syntax dictionary, and outputs a set of recognized words to the STM 92.

  The tactile recognition storage unit 83 is configured by an object called a Tactile Sensor 119 that recognizes a sensor input from a contact sensor, and outputs a recognition result to the STM 92 and an Internal State Model (ISM) 91 that is an object for managing an internal state.

  The STM 92 is an object constituting a short-term storage unit, and holds a target or an event recognized from an external environment by each of the above-described recognition system objects for a short period of time (for example, an input image from the camera 15 is stored for a short period of about 15 seconds). Only), and periodically notifies the SBL client 102 of the external stimulus (Notify).

  The LTM 93 is an object that forms a long-term storage unit, and is used to hold information obtained by learning, such as the name of an object, for a long time. The LTM 93 can, for example, associate and store a change in an internal state from an external stimulus in a certain behavior description module (schema).

  The ISM 91 is an object that constitutes an internal state management unit, manages several types of emotions such as instinct and emotion by mathematical modeling, and controls external stimuli (ES: External Stimula) recognized by each of the above-described recognition system objects. The internal state such as the instinct and emotion of the robot apparatus 1 is managed according to.

  The SBL 102 is an object that forms a situation-dependent behavior hierarchy. The SBL 102 is an object serving as a client (STM client) of the STM 92. Upon receiving a notification (Notify) of information on an external stimulus (target or event) from the STM 92 periodically, a schema (Schema), that is, an action description to be executed Determine the module (described below).

  The Reflexive SBL 103 is an object that constitutes a reflexive action unit, and executes reflexive and direct body motion in response to an external stimulus recognized by each object of the above-described recognition system. For example, a behavior such as following a human face, nodding, or immediately avoiding by detecting an obstacle is performed.

  The SBL 102 selects an operation according to a situation such as an external stimulus or a change in the internal state. On the other hand, the Reflexive SBL 103 selects a reflexive operation in response to an external stimulus. Since the action selection by these two objects is performed independently, when the action description modules (schema) selected from each other are executed on the body, hardware resources of the robot apparatus 1 conflict with each other and cannot be realized. Sometimes. An object called an RM (Resource Manager) 116 arbitrates hardware conflicts between the SBL 102 and the Reflexive SBL 103 when selecting an action. Then, the body is driven by notifying each object that realizes the body operation based on the arbitration result.

  The Sound Performer 172, the Motion Controller 173, and the LED Controller 174 are objects that implement the body operation. The Sound Performer 172 is an object for performing voice output, performs voice synthesis in accordance with a text command given from the SBL 102 via the RM 116, and performs voice output from a speaker on the body of the robot apparatus 1. The Motion Controller 173 is an object for performing an operation of each joint actuator on the body, and calculates a corresponding joint angle in response to receiving a command to move a hand, a leg, or the like from the SBL 102 via the RM 116. . The LED Controller 174 is an object for performing a blinking operation of the LED 19, and performs blinking driving of the LED 19 in response to receiving a command from the SBL 102 via the RM 116.

C-1. Situation-Dependent Behavior Control Next, the situation-dependent behavior hierarchy for calculating the behavior value AL and selecting an action to be expressed will be described in further detail. FIG. 20 schematically illustrates a form of situation-dependent behavior control using a situation-dependent behavior hierarchy (SBL) (including a reflex behavior unit).

The recognition result (sensor information) 182 of the external environment 70 in the external stimulus recognition unit 80 including the visual recognition function unit 81, the auditory recognition function unit 82, and the contact recognition function unit 83 is used as the external stimulus 183 as a situation-dependent behavior hierarchy (reflex behavior). 102a). The change 184 of the internal state according to the recognition result of the external environment 70 by the external stimulus recognition unit 80 is also given to the situation-dependent behavior hierarchy 102a. In the situation-dependent behavior hierarchy 102a, the situation can be determined according to the external stimulus 183 or the change 184 in the internal state, and the behavior can be selected.

In the situation-dependent action hierarchy 102a, as described above, the action value AL of each action description module (schema) is calculated based on the external stimulus 183 and the change 184 of the internal state, and the schema is selected according to the magnitude of the action value AL. And perform the action (action). For calculating the action value AL, for example, by using a library, a unified calculation process can be performed for all schemas.

The library includes, for example, a function for calculating a desire value vector from an internal state vector, a function for calculating a satisfaction level vector from an internal state vector, and an action for predicting a predicted internal state change vector from an external stimulus, as described above. The evaluation database is stored.

C-2. Schema FIG. 21 schematically shows a situation in which the situation-dependent behavior hierarchy 102 is composed of a plurality of schemas (element behaviors) 132. The situation-dependent behavior hierarchy 102 has a behavior description module as the above-described element behavior, and prepares a state machine for each behavior description module. The state machine 102 receives a sensor input depending on a previous behavior (action) or a situation. It classifies the recognition results of the external information and expresses the operation on the aircraft.

The action description module as an elementary action is described as a schema 132 having a Monitor function for performing a situation determination according to an external stimulus or an internal state, and an Action function for implementing a state transition (state machine) accompanying the action execution. .

  The context-dependent behavior hierarchy 102b (more strictly, of the context-dependent behavior hierarchy 102, a hierarchy that controls normal context-dependent behavior) is configured as a tree structure in which a plurality of schemas 132 are hierarchically connected. Behavior control is performed by integrally determining a more optimal schema 131 according to changes in the internal state. The tree 131 includes a plurality of subtrees (or branches) such as a behavior model in which an ethological situation-dependent behavior is formalized, and a subtree for executing emotion expression.

  FIG. 22 schematically illustrates a tree structure of a schema in the context-dependent behavior hierarchy 102. As shown in the figure, the context-dependent behavior hierarchy 102 starts from the root schemas 2011, 2021, and 2031 that receive a notification (Notify) of an external stimulus from the short-term storage unit 92, and starts from the concrete behavior from the abstract behavior category. A schema is arranged for each hierarchy so as to head to the category.

  For example, in the hierarchy immediately below the root schema, schemas 2012, 2022, and 2032 of "Search (Investigate)", "Eat (Ingestive)", and "Play (Play)" are provided. Under the schema 2012 “Search (Investigate)”, a plurality of schemas 2013 that describe more specific search behaviors such as “InvestigativeLocomotion”, “HeadinAirSniffing”, and “InvestigativeSniffing” are arranged.

  Similarly, a plurality of schemas 2023 describing more specific eating and drinking behaviors such as “Eat” and “Drink” are provided below the schema 2022 “Eat (Ingestive)”, and the schema 2032 “Play”. A plurality of schemas 2033 describing more specific playing behaviors such as “Play Bowing”, “Play Greeting”, and “Play Paying” are arranged below the list.

  As shown, each schema inputs an external stimulus and an internal state. Each schema has at least a Monitor function and an Action function.

  FIG. 31 schematically shows the internal structure of the schema. As shown in the figure, the schema includes an Action function that describes a body operation in the form of a state transition model (state machine) in which a state (or a state) changes according to occurrence of a predetermined event, and an external stimulus. It comprises a Monitor function that evaluates each state of the Action function according to the internal state and returns it as an activity level value, and a state management unit that stores and manages the state of the schema. As shown in the figure, the state management unit stores and manages the state of the schema as the state machine of the Action function as one of READY (completed), ACTIVE (active), and SLEEP (waiting).

  The Monitor function is a function that calculates an action value AL that is the activity level of the schema according to the external stimulus and the internal state. When the tree structure shown in FIG. 22 is configured, the upper (parent) schema can call the Monitor function of the lower (child) schema with the external stimulus and the internal state as arguments, and the child schema has an AL value. Is the return value. The schema can also call the Monitor function of the child's schema to calculate its own AL value. Then, since the AL value from each subtree is returned to the root schema, it is possible to integrally determine the optimal schema, that is, the action according to the change of the external stimulus and the internal state.

  For example, a schema having the highest AL value may be selected, or two or more schemas whose AL values exceed a predetermined threshold may be selected and executed in parallel (however, when executing in parallel, Assume that there is no hardware resource contention between each schema).

  FIG. 32 schematically illustrates the internal configuration of the Monitor function. As shown in the figure, the Monitor function includes a behavior induction evaluation value calculator for calculating an evaluation value for inducing a behavior described in the schema as an activity level, and a used resource calculator for specifying an aircraft resource to be used. It has. In the example shown in FIG. 31, when the Monitor function is called from the schema, that is, the behavior state control unit (tentative name) that manages the behavior module, the State function of the Action function is virtually executed, and the behavior induction evaluation value (that is, the activity induction evaluation value) is obtained. Calculates the used resource and the used resource, and returns it.

  The Action function includes a state machine (or state transition model) (described later) that describes the behavior of the schema itself. When configuring the tree structure as shown in FIG. 22, the parent schema can call the Action function to start or interrupt the execution of the child schema. In this embodiment, the Action state machine is not initialized unless it becomes Ready. In other words, the state is not reset even if interrupted, and the schema saves the work data being executed, so that interrupted re-execution is possible (described later).

  In the example shown in FIG. 31, the behavior state control unit (tentative name) that manages the schema, that is, the behavior module, selects the behavior to be executed based on the return value from the Monitor function, and calls the Action function of the corresponding schema. Or instruct the transition of the state of the schema stored in the state management unit. For example, a schema having the highest activity level as a behavior induction evaluation value is selected, or a plurality of schemas are selected according to a priority order so that resources do not conflict. Also, the behavior state control unit saves the state of the lower-priority schema from ACTIVE to SLEEP when a higher-priority schema is activated and a resource conflict occurs. Control the state of the schema, such as recovering.

  As shown in FIG. 33, the behavior state control unit may be provided only one in the context-dependent behavior hierarchy 102, and may centrally manage all schemas constituting the hierarchy 102 in a centralized manner. In the example illustrated in FIG. 33, the behavior state control unit includes a behavior evaluation unit, a behavior selection unit, and a behavior execution unit.

  The behavior evaluation unit calls the Monitor function of each schema at a predetermined control cycle, for example, and acquires each activity level and used resources.

  The action selection unit performs action control and management of machine resources using each schema. For example, schemas are selected in descending order of the aggregated activity level, and two or more schemas are simultaneously selected so that resources used do not conflict.

  The action execution unit issues an action execution command to the Action function of the selected schema, and manages the state of the schema (READY, ACTIVE, SLEEP) to control the execution of the schema. For example, when a schema with a higher priority is activated and a resource conflict occurs, the state of a schema with a lower priority is saved from ACTIVE to SLEEP, and when the conflict is resolved, the schema is restored to ACTIVE.

  Alternatively, the function of the behavior state control unit may be arranged for each schema in the context-dependent behavior hierarchy 108. For example, as shown in FIG. 22, when the schema forms a tree structure (see FIG. 34), the behavior state control of the upper (parent) schema is performed by using the external stimulus and the internal state as arguments, and The Monitor function of the (child) schema is called, and the activity level and the used resource are received as return values from the child schema. The child's schema also calls the child's schema Monitor function to calculate its activity level and resources used. Then, the activity level and the used resources from each subtree are returned to the behavior state control unit of the root schema. , Action function to start or suspend execution of the child schema. In this embodiment, the Action state machine is not initialized unless it becomes Ready. In other words, the state is not reset even if interrupted, and the schema saves the work data being executed, so that interrupted re-execution is possible.

  FIG. 23 schematically illustrates a mechanism for controlling normal context-dependent behavior in the context-dependent behavior hierarchy 102. As shown in the figure, an external stimulus 183 is input (Notify) from the short-term storage unit (STM) 92 to the context-dependent behavior hierarchy (SBL) 102, and a change 184 in the internal state is received from the internal state management unit 91. Is entered.

  The context-dependent behavior hierarchy 102 is composed of a plurality of sub-trees, such as a behavior model that formulates ethological behavior-dependent behavior and a sub-tree for executing emotional expression. In response to the notification (Notify) of 183, the Monitor function of each subtree is called, and by referring to the action value AL as a return value, an integrated action selection is performed, and the subtree that realizes the selected action is called. Then, an Action function is called.

  The context-dependent behavior determined in the context-dependent behavior hierarchy 102 is applied to the machine operation (Motion Controller) through arbitration of hardware resource competition with the reflex behavior by the reflex behavior unit 103 by the resource manager RM 116. Is done.

In addition, the reflexive action unit 103 performs a reflexive and direct body motion such as, for example, promptly avoiding by detecting an obstacle, in response to the external stimulus 183 recognized by each object of the above-described recognition system. . Therefore, unlike the case of controlling the normal context-dependent behavior shown in FIG. 22, a plurality of schemas 133 for directly inputting signals from the respective objects of the recognition system are not hierarchized as shown in FIG. They are arranged in parallel.

  FIG. 24 schematically illustrates the configuration of the schema in the reflex action unit 103. As shown in the drawing, the reflex action unit 103 operates in response to the Aid Big Sound 204, Face to Big Sound 205, and Nodding Sound 209 as schemas that operate in response to the recognition result of the auditory system, and the recognition result of the visual system. A Face to Moving Object 206 and an Avoid Moving Object 207 as schemas, and a hand-withdrawing 208 as a schema that operates in response to the recognition result of the haptic system are provided in an equivalent position (in parallel).

  As shown, each schema performing reflexive behavior has an external stimulus 183 as input. Each schema has at least a Monitor function and an Action function. The Monitor function calculates the action value AL of the schema in accordance with the external stimulus 183, and determines whether or not to exhibit the corresponding reflex action accordingly. The Action function includes a state machine (described later) that describes the reflexive behavior of the schema itself, and when called, expresses the relevant reflexive behavior and transitions the state of the Action.

  FIG. 25 schematically illustrates a mechanism for controlling reflexive behavior in the reflexive behavior unit 103. As shown in FIG. 24, a schema describing a reaction behavior and a schema describing an immediate response behavior exist in parallel in the reflex behavior unit 103. When a recognition result is input from each object constituting the functional module 80 of the recognition system, the corresponding reflexive behavior schema calculates an action value AL using an Aonitor function, and it is determined whether or not the Action should be tracked according to the value. You. Then, the reflex action determined to be activated by the reflex action unit 103 is mediated by the resource manager RM 116 through the arbitration of hardware resource competition with the context-dependent behavior by the context-dependent behavior hierarchy 102, and then to the body operation (Motion Controller 173). Applied to

  Such a schema that constitutes the situation-dependent behavior hierarchy 102 and the reflex behavior unit 103 can be described as a “class object” described on a C ++ language basis, for example. FIG. 26 schematically illustrates the class definition of the schema used in the situation-dependent behavior hierarchy 102. Each block shown in the figure corresponds to one class object.

  As illustrated, the context-dependent behavior hierarchy (SBL) 102 includes one or more schemas, an Event Data Handler (EDH) 211 that assigns IDs to input / output events of the SBL 102, and a Schema Handler (which manages schemas in the SBL 102). SH) 212, one or more Receive Data Handlers (RDH) 213 for receiving data from external objects (such as STM and LTM, resource managers, and cognitive objects), and one or more receive data handlers (RDH) 213 for transmitting data to external objects. It has a Send Data Handler (SDH) 214.

  The Schema Handler 212 stores information such as schemas and tree structures (SBL configuration information) constituting the context-dependent behavior hierarchy (SBL) 102 and the reflex behavior unit 103 as a file. For example, when the system is started, the Schema Handler 212 reads this configuration information file, constructs (reproduces) the schema configuration of the situation-dependent behavior hierarchy 102 as shown in FIG. 22, and stores each schema in the memory space. Map entity.

  Each schema has an OpenR_Guest 215 positioned as the base of the schema. The OpenR_Guest 215 includes one or more class objects Dsubject 216 for transmitting data to the outside of the schema, and one or more class objects DObject 217 for receiving data of the schema to the outside. For example, when the schema sends data to an external object (STM, LTM, each object of a recognition system, etc.) of the SBL 102, the Dsubject 216 writes the transmission data to the Send Data Handler 214. Further, the DOJECT 217 can read data received from the external object of the SBL 102 from the Receive Data Handler 213.

  Both Schema Manager 218 and Schema Base 219 are class objects that inherit OpenR_Guest 215. The class inheritance is to inherit the definition of the original class. In this case, it means that a class object such as Dsubject 216 or DOobject 217 defined in OpenR_Guest 215 also includes a Schema Manager Base 218 or a Schema Base 219 (below). Similar). For example, when a plurality of schemas have a tree structure as shown in FIG. 22, the Schema Manager Base 218 has a class object Schema List 220 that manages a list of child schemas (has a pointer to the child schema), You can call functions in the child schema. The Schema Base 219 has a pointer to the parent schema, and can return the return value of the function called from the parent schema.

  The Schema Base 219 has two class objects, State Machine 221 and Pronome 222. The state machine 221 manages a state machine for an action (action function) of the schema. The parent schema can switch (state transition) the state machine of the Action function of the child schema. Also, the target to which the schema executes or applies the action (Action function) is assigned to the Pronome 222. As will be described later, the schema is occupied by the target assigned to the Pronom 222, and the schema is not released until the action (operation) ends (complete, abnormal termination, etc.). To execute the same action for a new target, the same class definition schema is created in the memory space. As a result, the same schema can be executed independently for each target (the work data of each schema does not interfere with each other), and the reentrance property of the behavior described later is secured.

  The Parent Schema Base 223 is a class object that inherits the Schema Manager 218 and Schema Base 219 multiple times, and manages a parent schema and a child schema of the schema itself, that is, a parent-child relationship in the tree structure of the schema.

  The Intermediate Parent Schema Base 224 is a class object that inherits the Parent Schema Base 223, and implements interface conversion for each class. Further, the Intermediate Parent Schema Base 224 has a Schema Status Info 225. The Schema Status Info 225 is a class object that manages a state machine of the schema itself. The parent schema can switch the state of its state machine by calling the Action function of the child schema. In addition, the Monitor function of the child schema can be called to inquire about the action value AL corresponding to the normal state of the state machine. However, it should be noted that the state machine of the schema is different from the state machine of the Action function described above.

  The And Parent Schema 226, Num Or Parent Schema 227, and Or Parent Schema 228 are class objects that inherit from the Intermediate Parent Schema Base 224. The And Parent Schema 226 has pointers to a plurality of child schemas to be executed simultaneously. The Or Parent Schema 228 has pointers to a plurality of child schemas to be executed alternatively. The Num Or Parent Schema 227 has pointers to a plurality of child schemas that execute only a predetermined number at the same time.

  The Parent Schema 229 is a class object that inherits the And Parent Schema 226, Num Or Parent Schema 227, and Or Parent Schema 228 in multiples.

  FIG. 27 schematically shows the functional configuration of the classes in the situation-dependent behavior hierarchy (SBL) 102.

  The context-dependent behavior hierarchy (SBL) 102 transmits data to one or more Receive Data Handlers (RDH) 213 that receive data from external objects such as STM, LTM, resource manager, and cognitive objects, and transmits data to external objects. And one or more Send Data Handlers (SDH) 214.

  An Event Data Handler (EDH) 211 is a class object for allocating an ID to an input / output event of the SBL 102, and receives a notification of the input / output event from the RDH 213 or the SDH 214.

  The Schema Handler 212 is a class object for managing a schema, and stores configuration information of a schema configuring the SBL 102 as a file. For example, when the system is activated, the Schema Handler 212 reads the configuration information file and constructs a schema configuration in the SBL 102.

  Each schema is generated according to the class definition shown in FIG. 26, and entities are mapped on the memory space. Each schema has an OpenR_Guest 215 as a base class object, and includes class objects such as DObject 216 and DOObject 217 for external data access.

  The functions and state machines that the schema mainly has are shown below. The following functions are described in Schema Base 219.

ActivationMonitor (): Evaluation function for making the schema Active at the time of Ready Actions (): State machine Goal () for execution at the time of Active Fail (): Schema for evaluating whether the schema has reached Goal at the time of Active SleepActions (): State machine executed before Sleep. SleepMonitor (): Evaluation function for Resume at Sleep. ResumeActions (): State machine DestroyMonitor () for Resume before Resume. Evaluation function MakePronome () that determines whether the schema is in the fail state at the time: Determines the target of the entire tree Number

C-3. The functional situation-dependent behavior hierarchy (SBL) 102 of the situation-dependent behavior hierarchy is based on the contents stored in the short-term storage unit 92 and the long-term storage unit 93 and the internal state managed by the internal state management unit 91. Control actions that respond to your situation.

  As described in the previous section, the situation-dependent behavior hierarchy 102 according to the present embodiment is configured by a tree structure of a schema (see FIG. 22). Each schema keeps its independence, knowing its child and parent information. With such a schema configuration, the situation-dependent behavior hierarchy 102 has the main features of Concurrent evaluation, Concurrent execution, Preemption, and Reentrant. Hereinafter, these features will be described in detail.

C-3-1. Concurrent rating:
It has already been described that the schema as the action description module has a Monitor function for making a situation judgment according to an external stimulus or a change in the internal state. The Monitor function is implemented by the schema having a Monitor function in the class object Schema Base. The Monitor function is a function that calculates the action value AL of the schema according to the external stimulus and the internal state.

  When the tree structure as shown in FIG. 22 is configured, the upper (parent) schema can call the Monitor function of the lower (child) schema with the external stimulus 183 and the change 184 of the internal state as arguments. Returns the action value AL as the return value. In addition, the schema can further call the Monitor function of the child's schema to calculate its own action value AL. Then, since the action value AL from each subtree is returned to the root schemas 2011 to 2031, it is possible to integrally determine the optimal schema, that is, the operation according to the external stimulus 183 and the change 184 of the internal state.

  Because of the tree structure, the evaluation of each schema by the external stimulus 183 and the change 184 of the internal state is first performed on the current from the bottom of the tree structure to the top. That is, when the child schema exists in the schema, the Monitor function of the selected child is called, and then the own Monitor function is executed. Next, an execution permission as an evaluation result is passed from top to bottom of the tree structure. Evaluation and execution are performed while resolving contention for resources used by the operation.

  The situation-dependent behavior hierarchy 102 according to the present embodiment can evaluate behavior in parallel by using a tree structure of a schema, so that the situation such as an external stimulus 183 or a change 184 of an internal state can be evaluated. Be adaptive. In addition, at the time of evaluation, evaluation is performed on the entire tree, and the tree is changed by the action value AL calculated at this time, so that the schema, that is, the operation to be executed can be dynamically prioritized.

C-3-2. Concurrent execution:
Since the action value AL from each subtree is returned to the root schema, an optimal schema, that is, an operation according to the external stimulus 183 and the change 184 of the internal state, that is, the operation can be determined in an integrated manner. For example, a schema with the highest action value AL may be selected, or two or more schemas with an action value AL exceeding a predetermined threshold may be selected and executed in parallel. Sometimes it is assumed that there is no hardware resource conflict between each schema).

  The selected schema that has been granted execution permission is executed. That is, the schema actually observes the external stimulus 183 and internal state change 184 in more detail, and executes the command. Execution is performed sequentially from the top of the tree structure to the bottom, that is, Concurrent. That is, if the schema includes a child schema, the child's Actions function is executed.

  The Action function includes a state machine that describes an action (operation) of the schema itself. When constructing the tree structure as shown in FIG. 22, the parent schema can call the Action function to start or suspend the execution of the child schema.

  The context-dependent behavior hierarchy (SBL) 102 according to the present embodiment can execute another schema that uses the surplus resources at the same time by using the tree structure of the schema when resources do not conflict. However, if there is no restriction on the resources used up to Goal, an unusual behavior may occur. The context-dependent behavior determined in the context-dependent behavior hierarchy 102 is applied to the body operation (Motion Controller) through arbitration of competition of hardware resources with the reflex behavior by the reflex behavior unit (Reflexive SBL) 103 by the resource manager. Is done.

C-3-3. Preemption:
Even if a schema has been executed once, if there is a more important (higher priority) action, the schema must be interrupted and the execution right must be given to it. In addition, when more important actions are completed (completed or stopped), it is necessary to resume the original schema and continue the execution.

  Executing a task according to such a priority is similar to a function called Preemption of an OS (Operating System) in the computer world. The OS has a policy of sequentially executing tasks with higher priorities at a timing when the schedule is considered.

  On the other hand, since the control system 10 of the robot apparatus 1 according to the present embodiment extends over a plurality of objects, arbitration between the objects is required. For example, the reflex action unit 103, which is an object for controlling reflex behavior, needs to avoid an object or balance without worrying about the behavior evaluation of the context-dependent behavior hierarchy 102, which is an object for controlling higher-level context-dependent behavior. is there. This means that the execution right is actually robbed and executed, but the upper-level behavior description module (SBL) is notified that the execution right has been robbed, and the higher-level behavior description module (SBL) performs the processing to obtain a preemptive capability. Hold.

  It is also assumed that execution of a certain schema is permitted as a result of the evaluation of the action value AL based on the external stimulus 183 and the change 184 of the internal state in the context-dependent action layer 102. Further, it is assumed that the evaluation of the action value AL based on the external stimulus 183 and the change 184 of the internal state thereafter makes the importance of another schema higher. In such a case, by switching to the sleep state using the Actions function of the schema being executed, the preemptive behavior can be switched.

  Save the state of Actions () of the running schema and execute Actions () of a different schema. Also, after Actions () of a different schema is completed, Actions () of the suspended schema can be executed again.

  Also, Actions () of the schema being executed is interrupted, and SleepActions () is executed before the execution right is transferred to a different schema. For example, if the robot apparatus 1 finds a soccer ball during a conversation, it can say, "Wait a moment" and play soccer.

C-3-4. Reentrant:
Each schema constituting the context-dependent behavior hierarchy 102 is a kind of subroutine. When a schema is called from a plurality of parents, it is necessary to have a storage space corresponding to each parent in order to store its internal state.

  This is similar to the reentrant property of the OS in the computer world, and is referred to as a schema reentrant property in this specification. As shown in FIG. 27, the schema is composed of class objects, and the reentrant property is realized by generating an entity, that is, an instance of the class object for each target (Pronome).

  The reentrancy of the schema will be described more specifically with reference to FIG. The Schema Handler 212 is a class object for managing a schema, and stores configuration information of a schema configuring the SBL 102 as a file. When the system is started, the Schema Handler 212 reads this configuration information file and constructs a schema configuration in the SBL 102. In the example illustrated in FIG. 28, it is assumed that schema entities such as Eat 221 and Dialog 222 that define an action (operation) are mapped in the memory space.

  Here, by evaluating the action value AL based on the external stimulus 183 and the change 184 of the internal state, a target (Pronom) of A is set for the schema Dialog 222, and the Dialog 222 executes a dialogue with the person A. Suppose.

  Then, the person B interrupts the dialogue between the robot apparatus 1 and the person A, and then evaluates the action value AL based on the external stimulus 183 and the change 184 in the internal state. Assume that the priority is higher.

  In such a case, the Schema Handler 212 maps another Dialog entity (instance) inheriting a class for interacting with B on the memory space. Since the conversation with B is performed using another Dialog entity independently of the previous Dialog entity, the contents of the conversation with A are not destroyed. Therefore, DialogA can maintain data consistency, and when the conversation with B ends, the dialogue with A can be resumed from the point of interruption.

  The schema in the Ready list is evaluated according to the object (external stimulus 183), that is, the action value AL is calculated, and the execution right is delivered. After that, an instance of the schema moved in the Ready list is generated, and the other objects are evaluated. Thereby, the same schema can be set to the active or sleep state.

  As described above, the control program for realizing the control system as described above is stored in the flash ROM 23 in advance, and is read at the initial stage of turning on the power of the robot apparatus 1. In this way, the robot apparatus 1 can autonomously act according to the situation of itself and the surroundings, and instructions and actions from the user.

  The present invention has been described in detail with reference to the specific embodiments. However, it is obvious that those skilled in the art can modify or substitute the embodiment without departing from the spirit of the present invention.

  The gist of the present invention is not necessarily limited to products called “robots”. In other words, if it is a mechanical device or other general mobile device that performs motion similar to human motion using electric or magnetic action, it is a product belonging to another industrial field such as a toy. Even if there is, the present invention can be similarly applied.

  In short, the present invention has been disclosed by way of example, and the contents described in this specification should not be interpreted in a limited manner. In order to determine the gist of the present invention, the claims described at the beginning should be considered.

FIG. 1 is a perspective view illustrating an appearance of a robot device according to an embodiment of the present invention. FIG. 2 is a block diagram schematically illustrating a functional configuration of the robot device according to the embodiment of the present invention. FIG. 2 is a block diagram illustrating a configuration of a control unit according to the embodiment of the present invention in further detail. FIG. 4 is a schematic diagram showing an action selection control system portion that performs a process of calculating an action value corresponding to each action and outputting an action based on the action value in the control system of the robot apparatus according to the embodiment of the present invention. It is a schematic diagram which shows the flow of the process which the action value calculation part of an upper figure calculates the action value AL from an internal state and an external stimulus. FIG. 7 is a graph showing the relationship between an internal state and a desire value, with the horizontal axis representing each component of the internal state vector IntV and the vertical axis representing each component of the desire value vector InsV. It is a figure which shows the action value calculation data in an action value calculation database. FIG. 7 is a graph showing the relationship between the internal state and the degree of satisfaction, with IntV_NOURISHMENT (nutritional state) on the horizontal axis and the degree of satisfaction S_NOURISHMENT for the internal state “nutritional state” on the vertical axis. It is a graph which shows the relationship between an internal state and a satisfaction degree, taking IntV_FATIGUE (fatigue) on a horizontal axis and satisfaction S_FATIGUE for an internal state "fatigue" on a vertical axis. (A) And (b) is a figure which shows an example of the action value calculation data structure at the time of calculating | requiring the expected internal state change of an internal state "nutrition state" ("NOURISHMENT") and "fatigue" ("FATIGUE"), respectively. . FIG. 4 is a diagram illustrating a linear interpolation method of a one-dimensional external stimulus. FIG. 7 is a diagram illustrating a linear interpolation method of a two-dimensional external stimulus. FIG. 5 is a schematic diagram showing a flow from input of an external stimulus to learning of an action value calculation database in the control system of the robot device according to the embodiment of the present invention. FIG. 11 is a diagram illustrating an example of updating a predicted internal state change of a one-dimensional external stimulus. It is a figure explaining an example of updating of an expected internal state change of a two-dimensional external stimulus. It is a flowchart which shows the method of updating the action value calculation database according to the action result expressed by learning. 5 is a flowchart illustrating a behavior value calculation method in a behavior value calculation unit. It is a schematic diagram which shows the functional structure of the action control system of the robot apparatus in the specific example of this invention. It is a schematic diagram which shows the object structure of the action control system in the example of this invention. It is a schematic diagram which shows the form of the situation dependent action control by the situation dependent action hierarchy in the specific example of the present invention. It is a schematic diagram which shows a mode that a situation dependent action hierarchy is comprised by a some schema. It is a schematic diagram which shows the tree structure of the schema in a situation dependent action hierarchy. It is a schematic diagram which shows the mechanism for controlling normal context-dependent behavior in a context-dependent behavior hierarchy. It is a schematic diagram which shows the structure of the schema in a reflex action part. It is a schematic diagram which shows the mechanism for controlling a reflex action by a reflex action part. It is a schematic diagram which shows the class definition of the schema used in a situation dependent action hierarchy. It is a schematic diagram which shows the functional structure of the class in a situation dependent action hierarchy. FIG. 4 is a diagram for explaining the reentrant property of a schema. It is a figure for explaining other use methods of action value calculation database 121. It is a figure for explaining the influence of Boltzmann temperature at the time of action selection by SoftMax. FIG. 31 schematically shows the internal structure of the schema. FIG. 32 is a diagram schematically illustrating an internal configuration of the Monitor function. FIG. 33 is a diagram schematically illustrating a configuration example of an action state control unit. FIG. 34 is a diagram schematically illustrating another configuration example of the behavior state control unit.

Explanation of reference numerals

DESCRIPTION OF SYMBOLS 1 ... Robot apparatus 10 ... Control system 15 ... CCD camera 16 ... Microphone 17 ... Speaker 18 ... Touch sensor 19 ... LED indicator 20 ... Control part 21 ... CPU
22 ... RAM
23… ROM
24 nonvolatile memory 25 interface 26 wireless communication interface 27 network interface card 28 bus 29 keyboard 40 input / output unit 50 driving unit 51 motor 52 encoder 53 driver 81 visual recognition function unit 82 ... Hearing recognition function unit 83 ... Touch recognition function unit 91 ... Internal state management unit 92 ... Short term storage unit (STM)
93 ... Long term memory (LTM)
100: Behavior selection control system 101: Reflection behavior hierarchy 102: Situation-dependent behavior hierarchy (SBL)
103 ... reflex action unit 120 ... action value calculation unit 121 ... action value calculation database 130 ... action selection unit 132 ... element action 140 ... learning unit

Claims (25)

In a robot device that autonomously selects and expresses an action based on an internal state and an external stimulus,
A plurality of action description modules in which actions associated with a predetermined internal state and an external stimulus are described,
An action value calculation database having a data format in which an input external stimulus is associated with an expected internal state change expected to change after the action is expressed,
The action value calculation database is referred to from the internal state and the external stimulus, the desire value for the action associated with the internal state and the degree of satisfaction based on the internal state are obtained, the desire value obtained from the current internal state, and the expected internal state change Action value calculating means for calculating the action value of the action described in each of the action description modules based on the expected satisfaction degree change obtained from
Action selecting means for selecting an action description module based on the calculated action value, and expressing the action described in the selected action description module;
A robot device comprising: A learning unit that updates a corresponding expected internal state change in the action value calculation database based on an internal state change amount actually obtained based on a result after the action expression selected by the action selection unit. ,
The robot device according to claim 1, wherein: The action value calculating means, based on the desire value obtained from the current internal state, the degree of satisfaction obtained from the current internal state, and the change in expected satisfaction, sets the action value for the action described in each of the action description modules. Calculate
The robot device according to claim 1, wherein: The action value calculation database has a predicted internal state change associated with the value of the external stimulus,
The robot device according to claim 1, wherein: When a value that is not in the action value calculation database is input, the action value calculation unit calculates a predicted internal state change by performing linear interpolation using a linear model.
The robot device according to claim 4, wherein: The action selecting means always selects the action having the largest action value calculated by the action value calculating means from the candidate actions,
The robot device according to claim 1, wherein: The action selecting means randomly selects from the candidate actions, regardless of the action value calculated by the action value calculating means,
The robot device according to claim 1, wherein: The action selecting means selects from among actions that are candidates according to the probability according to the action value calculated by the action value calculating means,
The robot device according to claim 1, wherein: The action value calculation database manages the data format as a set of an action described in each of the action description modules, a property of an object as an external stimulus, and an internal state,
The robot device according to claim 1, wherein: The action value calculation unit searches the action value calculation database using the action described in each of the action description modules as an index, and determines an internal state from characteristics of the target object as an external stimulus.
The robot device according to claim 9, wherein: The action value calculation unit searches the action value calculation database using a certain characteristic of the object as an external stimulus as an index, and determines an internal state.
The robot device according to claim 9, wherein: The behavioral value calculating means arbitrarily sets or averages the behavior or other characteristics of the object as an external stimulus to a value, and gives the object an abstract value.
The robot device according to claim 11, wherein: In a behavior control method of a robot device that autonomously selects and expresses an action based on an internal state and an external stimulus, each action is described as an action description module associated with a predetermined internal state and an external stimulus,
A step of managing an action value calculation database having a data format in which the input external stimulus and the expected internal state change expected to change after the action is expressed,
The action value calculation database is referred to from the internal state and the external stimulus, the desire value for the action associated with the internal state and the degree of satisfaction based on the internal state are obtained, the desire value obtained from the current internal state, and the expected internal state change An action value calculating step of calculating an action value of the action described in each of the action description modules based on the expected satisfaction degree change obtained from
Selecting an action description module based on the calculated action value, and an action selection step of expressing the action described in the selected action description module;
A learning step of updating an action value calculation database based on the selected result after the action is expressed;
A behavior control method for a robot device, comprising: The method further includes a learning step of updating a corresponding expected internal state change in the action value calculation database based on an internal state change amount actually obtained based on the result after the action expression selected in the action selection step. ,
The behavior control method for a robot device according to claim 13, wherein: In the action value calculating step, the action value for the action described in each of the action description modules is based on the desire value obtained from the current internal state, the degree of satisfaction obtained from the current internal state, and the expected degree of satisfaction change. Calculate
The behavior control method for a robot device according to claim 13, wherein: The action value calculation database has a predicted internal state change associated with the value of the external stimulus,
The behavior control method for a robot device according to claim 13, wherein: In the action value calculation step, when a value that is not in the action value calculation database is input, a predicted internal state change is calculated by performing linear interpolation using a linear model.
The behavior control method for a robot device according to claim 16, wherein: In the action selection step, always select the action having the largest action value calculated in the action value calculation step among the actions that are candidates,
The behavior control method for a robot device according to claim 13, wherein: In the action selection step, regardless of the action value calculated in the action value calculation step, randomly select from the candidate actions,
The behavior control method for a robot device according to claim 13, wherein: In the action selection step, according to the probability according to the action value calculated in the action value calculation step, to select from the candidate actions,
The behavior control method for a robot device according to claim 13, wherein: In the step of managing the action value calculation database, the data format is managed as a set of an action described in each of the action description modules, a property of an object as an external stimulus, and an internal state,
The behavior control method for a robot device according to claim 13, wherein: In the action value calculation step, the action value calculation database is searched using the action described in each of the action description modules as an index, and an internal state is determined from characteristics of the object as an external stimulus.
22. The method according to claim 21, wherein the action is controlled by the robot apparatus. In the action value calculation step, the action value calculation database is searched using a certain characteristic of an object as an external stimulus as an index, and an internal state is determined.
22. The method according to claim 21, wherein the action is controlled by the robot apparatus. In the action value calculation step, arbitrarily set or average the value of the behavior or other characteristics of the object as an external stimulus, and give the object an abstract value,
The method of controlling a behavior of a robot device according to claim 23, wherein: A computer program written in a computer-readable form to execute on a computer system an action control of a robot apparatus for selecting and expressing an action autonomously based on an internal state and an external stimulus,
A plurality of action description modules in which actions associated with a predetermined internal state and an external stimulus are described,
A step of managing an action value calculation database having a data format in which the input external stimulus and the expected internal state change expected to change after the action is expressed,
The action value calculation database is referred to from the internal state and the external stimulus, the desire value for the action associated with the internal state and the degree of satisfaction based on the internal state are obtained, the desire value obtained from the current internal state, and the expected internal state change An action value calculating step of calculating an action value of the action described in each of the action description modules based on the expected satisfaction degree change obtained from
Selecting an action description module based on the calculated action value, and an action selection step of expressing the action described in the selected action description module;
A learning step of updating an action value calculation database based on the selected result after the action is expressed;
A computer program characterized by the above-mentioned.

Images