JP4683308B2 - Learning device, learning method, and program - Google Patents

Description

  The present invention relates to a learning apparatus, a learning method, and a program, and more particularly, to a learning apparatus, a learning method, and a program that can obtain, for example, a state transition model that appropriately models a complex object.

  A sensor signal observed from the modeling target (hereinafter also referred to as modeling target), that is, for example, the modeling target is sensed, and the state of the modeling target is configured based on the sensor signal obtained as a result of the sensing As learning methods, for example, K-means clustering method and SOM (self-organization map) have been proposed.

  In the K-means method or SOM, the state is arranged as a representative vector (representative vector) on the signal space of the observed sensor signal.

  In the K-means method, representative vectors are appropriately arranged on the signal space as initialization. Then, the vector at each time of the sensor signal is assigned to the representative vector having the closest distance, and the representative vector is updated by the average vector of the vectors assigned to each representative vector.

  In the SOM, competitive neighborhood learning is used for representative vector learning (see, for example, Non-Patent Document 1).

  There are many studies related to SOM, and a learning method called Growing Grid that learns while sequentially increasing states (here, representative vectors) has been proposed (see Non-Patent Document 2, for example).

  In the K-means method or SOM, a state (representative vector) is arranged on the signal space, but information on how the state transitions is not learned.

  For this reason, it is difficult to cope with a problem called perceptual aliasing by the K-means method or SOM.

  Here, perceptual aliasing is a problem that even if the states of the modeling target are different, they cannot be distinguished when the sensor signals observed from the modeling target are the same. For example, when a mobile robot equipped with a camera observes a landscape image as a sensor signal through the camera, if there are multiple locations in the environment where the same landscape image is observed, there is a problem that they cannot be distinguished. appear.

  On the other hand, the use of an HMM (Hidden Markov Model) has been proposed as a method of learning an observed sensor signal as time-series data and learning it as a probabilistic model having both states and state transitions.

  HMM is one of the models widely used for speech recognition. It represents the state transition probability that represents the probability of state transition, and the probability density at which a certain observed value is observed when the state transitions in each state. It is a state transition model defined by an output probability density function or the like to be expressed (for example, see Non-Patent Document 3).

  The parameters of the HMM, that is, the state transition probability, the power density function, etc. are estimated so as to maximize the likelihood. A Baum-Welch re-estimation method (Baum-Welch algorithm) is widely used as an HMM parameter estimation method.

  The HMM is a state transition model that can transition from each state to another state through the state transition probability. According to the HMM, the model transition process (the sensor signal observed from) As a model.

  However, in the HMM, the state corresponding to the observed sensor signal is usually determined only probabilistically. Therefore, as a method for determining a state transition process with the highest likelihood based on the observed sensor signal, that is, a sequence of states that maximizes the likelihood (hereinafter also referred to as the maximum likelihood path), the Viterbi method ( Viterbi Algorithm) is widely used.

  According to the Viterbi method, the state corresponding to the sensor signal at each time can be uniquely determined along the maximum likelihood path.

  According to the HMM, even if the sensor signal observed from the modeled object is the same in different situations (states), it is the same depending on the difference in the time change process of the sensor signal before and after the current time. Sensor signals can be treated as different state transition processes.

  Note that HMM cannot completely solve the problem of perceptual aliasing, but it is possible to assign different states to the same sensor signal. It is possible to model in more detail.

  By the way, in the HMM learning, when the number of states and the number of state transitions increase, it becomes difficult to appropriately (correctly) estimate the parameters.

  In particular, the Baum-Welch re-estimation method is not always a method for guaranteeing that an optimum parameter is determined. Therefore, when the number of parameters increases, it is extremely difficult to estimate an appropriate parameter.

  Also, when the modeling target is an unknown target, it is difficult to set the initial values of the HMM structure and parameters appropriately, and this also makes it difficult to estimate appropriate parameters.

  HMMs are effectively used in speech recognition because sensor signals handled are limited to speech signals, a lot of knowledge about speech is available, and the structure of HMMs that properly model speech The major factor is that the left-to-right structure is effective, and that the results of many years of research have been obtained.

  Therefore, if the modeling target is an unknown target and information for predetermining the structure and initial values of the HMM is not given, it is very difficult to make a large-scale HMM function as a practical model. It is a difficult problem.

  A method for determining the structure of the HMM itself instead of giving the structure of the HMM in advance has been proposed (see, for example, Non-Patent Document 4).

  In the method described in Non-Patent Document 4, the number of HMM states and the number of state transitions are increased one by one, each time a parameter is estimated, and an evaluation criterion called Akaike information criterion (called AIC) The structure of the HMM is determined by repeating the evaluation of the HMM using.

  The method described in Non-Patent Document 4 is applied to a small-scale HMM such as a phoneme model.

  However, since the method described in Non-Patent Document 4 is not a method that takes into account the estimation of large-scale HMM parameters, it is difficult to appropriately model a complicated modeling target.

  That is, in general, it is not always guaranteed that the evaluation criterion is monotonously improved only by making corrections that add states and state transitions one by one with respect to the structure of the HMM.

  Therefore, even if the method described in Non-Patent Document 4 is used for a complicated modeling target expressed by a large-scale HMM, an appropriate HMM structure is not always determined.

T. Kohonen, “Self-Organizing Map” (Springer Fairlark Tokyo) B. Fritzke, “Growing Grid-a self-organizing network with constant neighborhood range and adaptation strength”, Neural Processing Letters (1995), Vol. 2, No. 5, page 9-13 Co-authored by Laurence Rabiner and Biing-Hwang Juang, "Basics of Speech Recognition (Up / Down)", NTT Advanced Technology Corporation Shiro Ikeda, “Generation of phoneme models by structure search of HMM”, IEICE Transactions D-II, Vol.J78-D-II, No.1, pp.10-18, January 1995

  Conventionally, as described above, for example, it has been difficult to obtain a state transition model that appropriately models a complicated modeling target expressed by a large-scale HMM or the like.

  The present invention has been made in view of such a situation, and enables a state transition model to appropriately model a modeling target even if it is a complicated modeling target. is there.

The learning device or program according to one aspect of the present invention performs learning using a Baum-Welch re-estimation method using time-series data that is a sensor signal observed in time series by sensing a modeling target. HMM (Hidden Markov Model) structure number of states and state transitions, a state restricted to a predetermined number is placed in a space of a predetermined dimension, and the state transition is restricted to a state transition to a nearby state Initial structure setting means for initializing to a predetermined initial structure and setting a state transition probability, an output probability density function, and an initial probability, which are parameters of the HMM of the initial structure, and new parameters of the HMM in the learning of the HMM Before the estimation, the sampling frequency of the adjusted time-series data adjusted by adjusting the time-series data is increased with respect to the increase in the number of learning times estimated and updated by the HMM parameters. Data adjustment means for adjusting the time series data by down-sampling and outputting the adjusted time series data, and using the adjusted time series data, the new parameters of the HMM are changed to the Baum-Welch parameters. Using the parameter estimation means for estimating by the estimation method, the HMM of the new parameter, and the time series data or the adjusted time series data, the structure of the HMM, the state of the HMM, and the state transition It is a program for causing a computer to function as a learning device or a learning device provided with a structure adjusting means for adjusting so as to increase or decrease.

A learning method according to one aspect of the present invention uses time series data that is a sensor signal observed in time series by sensing a modeling target, and is based on a Humden Markov Model (Hidden Markov Model) Baum-Welch re-estimation method. The learning device that performs the learning places the number of states and state transitions that are the structure of the HMM in a state in which a predetermined number of states are constrained in a predetermined dimension space, and the state transitions are constrained to state transitions to neighboring states. initialized to a predetermined initial structure that is, the parameter a is the state transition probability of the HMM of the initial structure, the output probability density function, and sets the initial probability, before estimating the new parameters of the HMM in the learning of the HMM In addition, the time-series data is down-scaled so that the sampling frequency of the adjusted time-series data obtained by adjusting the time-series data increases as the number of learning times updated by estimating and updating the HMM parameters is increased. The adjusted time series data is output, and the adjusted time series data is used to estimate the new parameters of the HMM by the Baum-Welch re-estimation method. Using the parameter HMM and the time-series data or the adjusted time-series data, repeatedly adjusting the structure of the HMM to increase or decrease the state and state transition of the HMM It is a learning method.

In one aspect of the present invention, an HMM (Hidden Markov Model) in which learning is performed by a Baum-Welch re-estimation method using time-series data that are sensor signals observed in time series by sensing a modeling target. The number of states and state transitions in the structure of) are arranged in a space of a predetermined dimension and the state transition is initially set to a predetermined initial structure constrained to a state transition to a neighboring state. The state transition probability, the output probability density function, and the initial probability, which are parameters of the HMM having the initial structure, are set. Further, before estimating the new parameters of the HMM in the learning of the HMM, adjusted time-series data obtained by adjusting the time-series data with respect to an increase in the number of learning times estimated and updated by the HMM parameters The time-series data is adjusted by down-sampling so that the sampling frequency increases, and the adjusted time-series data is output. Then, using the adjusted time series data, a new parameter of the HMM is estimated by the Baum-Welch re-estimation method, and the new parameter HMM and the time series data, or the adjusted Using the time series data, the structure of the HMM is adjusted so as to increase or decrease the state and state transition of the HMM.

  Note that the learning device may be an independent device or an internal block constituting one device.

  The program can be provided by being transmitted via a transmission medium or by being recorded on a recording medium.

According to one aspect of the present invention, it is possible to obtain an HMM that appropriately models a complicated modeling target.

  First, an outline of processing to which the present invention is applied will be described with reference to FIGS.

  One of the objects of the present invention is as follows. That is, one of the purposes is to automatically build a model of the external environment based on the sensor signal (hereinafter referred to as an observation signal) to be observed by the target system or agent and the action signal taken by itself. . In addition to realizing the object, one of the objects is to freely generate an intelligent behavior for realizing an arbitrary state on the internally perceived model.

  Note that an “agent” generally refers to an autonomous entity that can perceive (for example, sense) an environmental state and select an action based on the perceived content. However, the following operation subject will be described using a system, not an agent.

  Also, the modeling of the external environment itself is not particularly limited. However, in the present embodiment, a hidden Markov model (hereinafter referred to as HMM) is adopted as modeling of the external environment.

  For example, as shown in FIG. 1A, a model for obtaining an action signal from an observation signal is created.

  In this case, as shown in FIG. 1B, first, the system constructs an HMM from only the observation signal.

  Next, as shown in FIG. 2A, the system analyzes the relationship between each state transition of the constructed HMM (hereinafter abbreviated as a transition as appropriate) and the action taken by itself (action signal). Thereby, the relationship between the sensor signal and the action signal necessary for each transition is learned as a controller.

  Then, as shown in FIG. 2B, when a target state (state F in the example of FIG. 2B) is given, the system transitions from the current state (state A in the example of FIG. 2B) to the target state (see FIG. 2B). In the example of 2B, a transition sequence of thick arrows) is obtained. Hereinafter, such a transition sequence is appropriately referred to as a route. Further, obtaining such a route is hereinafter referred to as planning. After planning, the system can realize an arbitrary state by calling a controller necessary for each transition included in the route.

  The outline of the process to which the present invention is applied has been described above. Next, an embodiment of the present invention will be described.

  FIG. 3 is a functional block diagram showing a functional configuration example of an embodiment of an information processing system to which the present invention is applied (hereinafter simply referred to as the system of FIG. 3).

  The system of FIG. 3 includes a sensor unit 21, a modeling unit 22, an innate controller 23, a behavior control unit 24, and an action unit 25.

  The sensor unit 21 observes a predetermined physical quantity of the environment where the agent is placed, and provides the observation result to the modeling unit 22 as an observation signal.

  The modeling unit 22 includes a learning unit 31, an HMM storage unit 32, a recognition unit 33, and a planning unit 34.

  The learning unit 31 constructs an HMM using the observation signal of the sensor unit 21 (see FIG. 1B) and stores it in the HMM storage unit 32.

  The recognition unit 33 estimates each transition to the current state (current state) using the HMM stored in the HMM storage unit 32 and the observation signal sequence of the sensor unit 21 when the action unit 25 described later acts. To do. The estimation result of the recognition unit 33 is provided to the behavior control unit 24.

  The planning unit 34 uses the HMM stored in the HMM storage unit 32 to plan (calculate) an optimal route from the current state to the target state and provide the behavior control unit 24 (see FIG. 2B). ). The target state refers to a state given as a target to the behavior control unit 24 and is provided from the behavior control unit 24 to the modeling unit 22.

  The innate controller 23 issues various commands based on a predetermined innate rule for learning by the learning unit 41 of the behavior control unit 24 described later, and provides the commands to the learning unit 41 and the action unit 25.

  The behavior control unit 24 includes a learning unit 41, a controller table storage unit 42, a controller storage unit 43, and an execution management unit 44.

  The learning unit 41 uses each transition recognized by the recognition unit 33 based on the action result of the action unit 25 according to the command from the innate controller 23 and the command from the innate controller 23, and sets the controller for each transition. Learn (see FIG. 2A). Then, the learning unit 41 stores each controller in the controller storage unit 43. Further, the learning unit 41 stores the relationship between each controller and the transition in the controller table storage unit 42. Details of the controller will be described later.

  The execution management unit 44 generates a command for the action unit 25 so that the action unit 25 acts along the route provided from the planning unit 34, that is, realizes each transition in the route. 25. This command is reversely generated based on information stored in the controller table storage unit 42 and the controller storage unit 43. Details of the process of the execution management unit 44 will be described later.

  Hereinafter, further details of the system of FIG. 3 will be described using a case where a simple pendulum task is given as a task as an example.

  That is, for the purpose of the system of FIG. 3, for example, as shown in FIG. 4, the observation angle θ of the simple pendulum 51 is taken as an observation signal, and the simple pendulum 51 is freely controlled using the generated torque τ as a control signal (action signal). Adopt the purpose of. However, it is assumed that physical parameters such as the mass of the simple pendulum 51 and the friction coefficient are unknown. A simple pendulum task with such a task setting is often adopted as a task of reinforcement learning.

  In the task setting in the conventional simple pendulum task, two variables of the angular velocity ω are given as observation signals in addition to the angle θ so that the system state can be uniquely described. Further, as an object, it is given that the simple pendulum 51 swings up, that is, the angle θ is 180 °. Furthermore, an objective function for achieving the objective, for example, an objective function such as giving a reward when the angle θ reaches 180 ° or giving a higher value as the angle θ approaches 180 ° is given. .

  On the other hand, one of the purposes of the system shown in FIG. 3 is to realize an agent that can autonomously solve various tasks regardless of the simple pendulum task. Therefore, the system of FIG. 3 is imposed with a restriction that only the angle θ that is a part of the state can be observed. Also, one of the purposes of the system of FIG. 3 is not to provide an objective function but to realize an arbitrary internal state. Thus, the system of FIG. 3 does not require an objective function that depends on the task of swinging up.

  FIG. 5 is a flowchart for explaining an example of processing for achieving the simple pendulum task by the system shown in FIG. 3 (hereinafter referred to as simple pendulum task control processing).

  In step S1, the system of FIG. 3 executes an HMM learning process.

  In step S2, the system of FIG. 3 performs a recognition process.

  In step S3, the system of FIG. 3 executes a controller learning process.

  In step S4, the system of FIG. 3 executes a planning process.

  In step S5, the system of FIG. 3 executes a behavior control process.

  Hereinafter, details of each processing of these steps S1 to S5 will be individually described in the order.

  First, the HMM learning process in step S1 will be described.

  In the initial state, the action unit 25 outputs a control signal τ that is randomly generated or a control signal τ in which moderate perturbation is added to a pattern embedded in advance. The control signal τ is generated by the action unit 25 based on a command given by the innate controller 23, for example.

  During this time, a time series of observation signals θ output from the sensor unit 21 (hereinafter referred to as time series observation signals) is stored in a memory (not shown) of the learning unit 31. For example, the signal 52 in FIG. 6 is an example of a time series observation signal. At a timing when a certain amount of time series observation signals are stored in the memory, the learning unit 31 learns these observation time series signals to construct an HMM and stores it in the HMM storage unit 32.

  The series of processes described above is the HMM learning process.

  In the HMM learning process, the Baum-Welch algorithm is generally used. An example of an HMM applicable to such an algorithm is shown in FIGS.

  However, there is a problem that when learning a fully coupled HMM as shown in FIG. 7 without any restrictions, depending on the initial value of the parameter, it converges to a local minimum, making it difficult to learn the HMM. is there.

  Therefore, in the present embodiment, a hypothesis is adopted that most of the phenomena in the natural world can be expressed by a sparse connection like a small world network. That is, in this embodiment, the Baum-Welch algorithm restricted to sparse coupling is adopted. Specifically, in the present embodiment, it is assumed that any one of the HMMs in FIGS. 9 and 10 which is an example of a sparsely coupled HMM is employed. Here, the HMM in FIGS. 9A and 9B is a two-dimensional neighborhood constraint HMM. The HMM in FIG. 10A is an HMM due to a three-dimensional grid constraint. The HMM in FIG. 10B is an HMM due to a two-dimensional random arrangement constraint. The HMM in FIG. 10C is an HMM by a small world network.

  FIG. 11 shows a display example of the result of learning the time series observation signal by giving the 484-node two-dimensional neighborhood structure HMM as the initial structure in the simple pendulum task described above.

  The horizontal axis in FIG. 11 indicates the angle θ of the simple pendulum 51 that is an observation signal. The vertical axis in FIG. 11 indicates the angular velocity ω of the simple pendulum 51. In FIG. 11, a circle indicates a node (state), and a solid line between two circles indicates a connection (transition) between two nodes. That is, in the display example of FIG. 11, each of the (θ, ω) spaces on the basis of the average value of the true state (θ, ω) of the environment at the node perceived by the system (agent) of FIG. Nodes are plotted as circles, and among the connections between these nodes, only the connections having a transition probability of 0.01 or more are displayed as solid lines.

  From the display example of FIG. 11, it can be seen that each node is connected to only a few nearby nodes. This means that the system of FIG. 3 corresponds to a continuous change in the (θ, ω) space. In such a case, it means that the behavior of the system of FIG. 3 can be described by sparse coupling.

  The only exception is that the change from θ = −π to π is discontinuous when the simple pendulum 51 makes one rotation. The fact that learning about this discontinuous change is made can also be seen from the fact that the left and right ends and end nodes in FIG. 11 are combined in the display example of FIG.

  Further, from the display example of FIG. 11, it can be seen that different nodes are assigned for the same angle θ even though only the angle θ is observed as the observation signal. This indicates that the behavior of the system in FIG. 3 can be expressed by the angular velocity ω even at the same angle θ.

  The HMM learning process in step S1 has been described above. Next, the recognition process in step S2 will be described.

  The recognition process is a process of estimating the current state of the system in FIG. 3 using the HMM constructed by the HMM learning process in step S1. This recognition process is executed by the recognition unit 33.

  The result of the recognition process is used for a controller learning process in step S3 described later. In addition to the process of step S2, a recognition process is executed as one process of an action control process of step S5 described later (see step S61 in FIG. 14).

  The key information in this recognition process is the observation signal sequence from the past to the present. Viterbi algorithm is widely used for HMM state estimation. Therefore, in this embodiment, the state before 50 steps is undefined. That is, it is assumed that the probability of each node is equal. Next, assume that the state before 50 steps is the initial state, and observation results for 50 steps are given. Then, it is assumed that recognition processing such as estimating the state of the last 50 steps, that is, the current state is executed by determining the state at each step by the Viterbi Algorithm.

  Specifically, for example, in this embodiment, it is assumed that the recognition process is executed according to the flowchart of FIG.

  Hereinafter, the transition probability from the node i to the node j is described as aij or Aij. The initial state probability is described as πi. The observed value (observed signal level) at time t is described as o (t). The likelihood of the observation value o (t) at the node i is referred to as observation likelihood and is described as bi (o (t)). The current time is described as T.

  In step S21, the recognition unit 33 sets time t = 0.

  In step S22, the recognizing unit 33 sets the initial state probability πi by the observation likelihood b (O (0)) to each node.

  In step S23, the recognizing unit 33 multiplies the state probability at time t by the transition probability Aij and the observation likelihood b (O (t + 1)) to obtain the maximum probability at the transition destination node j. Update to probability.

  In step S24, the recognizing unit 33 stores the node i of the transition source at that time in the storage table. The construction location of the storage table is not particularly limited. In the present embodiment, for example, it is assumed that a storage table is built in the recognition unit 33.

  In step S25, the recognition unit 33 sets time t = t + 1.

  In step S <b> 26, the recognizing unit 33 determines whether the time t has reached the current time T.

  If the time t is a time before the current time T, it is determined as NO in Step S26, and the process returns to Step S23 to repeat the subsequent processes.

  That is, the loop process of steps S23 to S26 is repeated for each of times t = 0 to T. Then, when the time t becomes the current time T, it is determined as YES in Step S26, and the process proceeds to Step S27.

  In step S <b> 27, the recognizing unit 33 selects the largest node among the state probabilities at time t and sets it as a confirmed node at time t. That is, in the process of step S27 immediately after it is determined as YES in the process of step S26, since the time t = T, a definite node at the current time T is obtained.

  In step S28, the recognizing unit 33 extracts the node i that is the transition source of the node j selected in the process of step S27 from the storage table and sets it as the node at time t-1.

  In step S29, the recognition unit 33 sets time t = t−1.

  In step S30, the recognizing unit 33 determines whether or not time t = 0.

  When the time t is a time later than 0, it is determined as NO in step S30, and the process returns to step S27, and the subsequent processes are repeated.

  That is, the loop process of steps S27 to S30 is repeated for each of the times t = T to 0. Then, when the time t becomes 0, it is determined as YES in Step S30, and the recognition process is ended.

  The recognition process in step S2 has been described above. Next, the controller learning process in step S3 will be described.

  When the recognition process in step S2 is executed, the node i indicating the state at each time is determined, and the transition probability Aij from the node i to the node j indicating the state at the next time is determined. Hereinafter, the transition probability Aij is appropriately referred to as a transition edge Aij. Note that in the description of the recognition process, a lowercase letter a is used such as a transition probability aij, but here a capital letter A is used such as a transition probability Aij (transition edge Aij). . This is to prevent confusion with a lowercase letter a in action a (t) described later.

  The system of FIG. 3 takes some random or innate behavior as described above during the learning process of the HMM in step S1. Therefore, the action taken by the system of FIG. 3 in the state i during the innate action is referred to as action a (t). However, action a (t) is abbreviated as appropriate as action a. In this case, a causal model in which the transition edge Aij is generated by the action a is established.

  Therefore, the learning unit 41 of the behavior control unit 24 samples an observed value o (t) (hereinafter abbreviated as an observed value o) and an action a for each generated transition edge Aij. In this case, if the time-series observation signal is a long-time signal, the transition edge Aij occurs many times during that time. Therefore, the learning unit 41 learns a mapping a = Fij (o) for one transition edge Aij using the sampled observation value o and action a. As a learning method for the function mapping Fij (), for example, a neural network or the like can be used. As the simplest example, a learning method may be employed in which the function mapping Fij () is such that the average value of the action a is output regardless of the observation value o.

  Such a function mapping Fij () is stored in the controller storage unit 43 as a controller to be executed by the action unit 25.

  Then, the controller learning result, that is, information indicating what the corresponding controller (function mapping Fij ()) is stored for each transition edge Aij is stored in the controller table storage unit 42 in a table format. Hereinafter, such a table is referred to as a controller table.

  Here, in the present embodiment, it is assumed that an identifier (ID) that uniquely identifies each controller (function mapping Fij ()) is assigned. In this case, the ID of the controller can be used as information indicating what the controller (function mapping Fij ()) associated with the predetermined transition edge Aij is. Therefore, in the present embodiment, it is assumed that the ID of the corresponding controller (function mapping Fij ()) is stored in the controller table 42 for each transition edge Aij. In the controller storage unit 43, it is assumed that each controller (function mapping Fij ()) is stored in association with its ID. An example of how to use the ID will be mentioned in the description of step S70 in FIG.

  As described above, the controller learning process in step S3 has been described by taking as an example the process of performing learning for assigning each controller that outputs an action for each state transition. However, as the controller learning processing to which the present invention is applied, for example, the following processing can be adopted in addition to the above-described example. That is, for example, a process of performing learning for assigning each controller that outputs an action for each transition destination state can be employed.

  Next, the planning process in step S4 will be described.

  When the controller learning process in step S3 is completed, the system shown in FIG. 3 finishes learning, sets an arbitrary target in the internal state formed by the HMM, and performs an action to achieve the target. Will be able to take.

  Therefore, the planning unit 34 makes a plan (planning) for achieving the goal. The process for making such a plan is the planning process in step S4.

  That is, the planning unit 34 sets, as a goal, a target designated from the outside or obtained internally. In the system of FIG. 3, the target is provided from the execution management unit 44. Hereinafter, a node indicating a goal state is referred to as a goal node g. In this case, when the goal node g and the node i indicating the current state (hereinafter referred to as the current state node i) are known, the planning unit 34 searches for a path connecting the two nodes on the HMM. Such a process of searching for a route from the current state node i to the goal node g is the planning process in step S4.

  Here, various route search algorithms exist, and any algorithm may be adopted. However, in this embodiment, it is assumed that an algorithm applying the Viterbi Algorithm is adopted as shown in the flowchart of FIG. That is, FIG. 13 is a flowchart for explaining an example of the planning process.

  In step S41, the planning unit 34 sets the state probability of the current state node i to 1.0 and sets the state probabilities of other nodes to 0. In addition, the planning unit 34 sets time t = 0.

  In step S42, the planning unit 34 sets 0.9 as the transition probability Aij that is equal to or higher than the threshold (0.01 in this case) and 0 as the other.

  In step S <b> 43, the planning unit 34 multiplies the state probability at time t by the transition probability Aij, and updates the maximum probability at the transition destination node j to the state probability of the node j.

  In step S44, the planning unit 34 stores the transition source node i at that time in the storage table. The construction location of the storage table is not particularly limited. In the present embodiment, for example, it is assumed that a storage table is built in the planning unit 34.

  In step S45, the planning unit 34 determines whether or not the state probability of the goal node “g” exceeds zero.

  When the state probability of the target goal node g is 0, it is determined that the target has not been reached yet, NO is determined in step S45, and the process proceeds to step S46.

  In step S46, the planning unit 34 determines whether the loop process of steps S43 to S47 has been repeated N times.

  The case of repeating N times means a case where the target has not yet been reached after repeating N times. Therefore, in such a case, that is, in the case where it is determined as YES in step S46, the planning process is ended because the planning unit 34 gives up the planning.

  On the other hand, if it has not been repeated N times yet, it is determined as NO in Step S46, and the process proceeds to Step S47. In step S47, the planning unit 34 sets time t = t + 1. Thereafter, the process returns to step S43, and the subsequent processes are repeated.

  In this way, if the state probability of the target goal node g exceeds 0 as a result of repeating the loop process of steps S43 to S47 several times, it is determined that the target has been reached and YES in step S45. As a result, the process proceeds to step S48.

  The planning unit 34 selects the goal node g in step S48, and sets g = j in step S49.

  In step S50, the planning unit 34 extracts the node i, which is the transition source of the selected node j, from the storage table and sets it as the node at time t-1.

  In step S51, the planning unit 34 sets time t = t-1.

  In step S52, the planning unit 34 determines whether or not time t = 0.

  When the time t is later than 0, it is determined as NO in Step S52, and the process proceeds to Step S53. In step S53, the planning unit 34 sets j = i. Thereafter, the process returns to step S50, and the subsequent processes are repeated.

  That is, the loop process of steps S50 to S53 is repeated until time t = 0. Then, when the time t becomes 0, it is determined as YES in Step S52, and the planning process ends. A node string formed at this time, that is, a node string from the current state node i to the goal node g is determined as a route.

  The planning process in step S4 has been described above. Next, the behavior control process in step S5 will be described.

  FIG. 14 illustrates an example of behavior control processing by the behavior control unit 24, that is, an example of processing when the behavior control unit 24 performs behavior control based on the route (node sequence) calculated in the process of step S4. It is a flowchart.

  In step S61, the execution management unit 44 of the behavior control unit 24 performs HMM recognition processing, and selects the node with the highest state probability among all the nodes as the node i_max.

  In the present embodiment, as the HMM recognition process, it is assumed that the process according to the flowchart of the example of FIG. 12 described above is executed. In addition, although the operation subject of the recognition processing of the HMM is the execution management unit 44 here for convenience of explanation, it is actually the recognition unit 33. Specifically, the recognition unit 33 performs HMM recognition processing, and the execution management unit 44 selects the node i_max based on the processing result.

  In step S62, the execution management unit 44 selects, as the current node i_pathmax, the node having the highest state probability between the previous node i_pathmax and the goal node among the nodes on the route.

  In step S63, the execution management unit 44 determines whether or not the ratio between the state probabilities P (i_max) and P (i_pathmax) is less than or equal to a threshold (for example, 0.7 or less). Here, the state probability P (i_max) indicates the state probability of the node i_max. The state probability P (i_pathmax) indicates the state probability of the node i_pathmax.

  If the ratio between the state probabilities P (i_max) and P (i_pathmax) is equal to or less than the threshold value, it is determined that the current behavior of the system in FIG. Ends.

  On the other hand, if the ratio between the state probabilities P (i_max) and P (i_pathmax) exceeds the threshold, the current behavior of the system in FIG. If it is determined that there is, the process proceeds to step S64.

  In step S64, the execution management unit 44 determines whether or not the node i_pathmax is stopped at the same node i_pathmax, that is, the node i_pathmax selected in the current step S62 and the node i_pathmax selected in the previous step S62. It is determined whether or not they are the same.

  If it is not stopped at the same node i_pathmax, it is considered that the robot has moved along the route. Therefore, it is determined as NO in Step S64, and the process proceeds to Step S68. The processing after step S68 will be described later.

  On the other hand, if it remains at the same node i_pathmax, there is a possibility that it has not moved along the route. Therefore, it is determined as YES in Step S64, and the process proceeds to Step S65.

  In step S65, the execution management unit 44 determines whether or not the state probability of the next node i_next on the route is higher than the previous state probability.

  If it has not risen, the execution management unit 44 determines that the transition along the path has not been made, and determines that the answer is NO in Step S65, and sets the node i_pathmax to be the node i_next in Step S66. Thereafter, the process proceeds to step S68. The processing after step S68 will be described later.

  On the other hand, when it has risen, it determines with it being YES in step S65, and a process progresses to step S67.

  In step S67, the execution management unit 44 determines whether the same node has stopped N times (for example, 50 times) or more.

  If the same node has not stopped more than N times, it is determined as NO in step S67, and the process proceeds to step S68. The processing after step S68 will be described later.

  On the other hand, if the same node is stopped N times or more, the execution management unit 44 determines YES in step S67, and sets the node i_pathmax to the node i_next in step S66. That is, if the same node is stopped N times or more, it is considered that the route has been forcibly advanced. Thereafter, the process proceeds to step S68.

  In step S68, the execution management unit 44 determines whether or not it is already on the goal node.

  If it is already recognized that it is on the goal node, it is determined as YES in step S68, and the behavior control process is terminated assuming that the target is reached.

  On the other hand, when it is recognized that it is not yet on the goal node, it is determined as NO in Step S68, and the process proceeds to Step S69.

  In step S69, the execution management unit 44 determines a transition edge Aij for transitioning to the next node on the route.

  In step S70, the execution management unit 44 calls the controller (function mapping Fij ()) assigned to the transition edge Aij, and the action unit 25 gives the current observation value o to the controller, thereby taking action a to be taken. Ask.

  More precisely, in this embodiment, the ID of the controller (function mapping Fij ()) assigned to the transition edge Aij is read from the controller table storage unit 42. Further, the controller (function mapping Fij ()) specified by the ID is read from the controller storage unit 43. An output obtained as a result of inputting the current observation value o to the function mapping Fij () as the controller is an action a.

  This action a is provided to the action unit 25 as a command. Therefore, in step S71, the action unit 25 executes the command a.

  Thereafter, the process returns to step S61, and the subsequent processes are repeated.

  When it is determined as YES in the process of step S68 and the behavior control process ends, the execution management unit 44 may determine again whether or not the node i_max at that time is really a goal node. . If the result of this redetermination is a goal node, the entire control process of the simple pendulum task in FIG. 5 is terminated. On the other hand, if the result of the redetermination is not a goal node, the system in FIG. 3 returns the process to step S4, and executes the planning process again with the same goal node to create a new route. Later, the behavior control process in step S5 is performed again.

  The system of FIG. 3 that can achieve the simple pendulum task has been described above. However, the system of FIG. 3 cannot achieve the multimodal task described later. In contrast, FIG. 15 shows a functional configuration example of a system capable of achieving a multimodal task. 15 is an embodiment of an information processing system to which the present invention is applied (hereinafter simply referred to as the system of FIG. 15), and shows a functional configuration example different from the system of FIG. It is a functional block diagram.

  The system in FIG. 15 includes a sensor unit 61, three types of modeling units 62A to 62C, a causal unit 63, a behavior control unit 64, and an action unit 65.

  The sensor unit 61 is configured as a so-called multimodal sensor.

  Here, the multimodal sensor will be briefly described.

  One of the concepts developed from the conventional human interface is a multimodal interface. As a synonym for the multimodal interface, for example, the term multimedia interface exists. A multimedia interface simply represents an interface using a plurality of media (sound, video, touch, etc.), whereas when each media is used in various forms to transmit information, a multimodal interface is used. It is called.

  For example, as an example of a multimodal interface, an event such as utterance, action, or line of sight is made modal, and these modals are coordinated, used simultaneously, or by combining multiple types of messages, human beings are trying to convey it, or There is an interface for trying to understand messages that are transmitted naturally.

  That is, the multimodal sensor is a sensor for realizing such a multimodal interface, and can detect a physical quantity corresponding to each of a plurality of modals (events).

  For example, in the example of FIG. 15, the sensor unit 61 observes a predetermined physical quantity of the environment where the agent is placed, that is, a physical quantity corresponding to the modal, for each of the three modals, and models the observation result as an observation signal. Provided to parts 62A, 62B, 62C.

  Each of the modeling units 62A, 62B, and 62C has basically the same function and configuration as the modeling unit 22 of FIG. That is, regarding the modeling unit 62A, each of the learning unit 71A to the planning unit 74A has basically the same function and configuration as each of the learning unit 31 to the planning unit 34 of FIG. Although not illustrated, the modeling unit 62B is provided with learning units 71B to 74B having basically the same functions and configurations as the learning units 31 to the planning unit 34 in FIG. ing. Further, the modeling unit 62C includes learning units 71C to 74C having basically the same functions and configurations as the learning units 31 to the planning unit 34 in FIG.

  Therefore, each HMM constructed as a result of learning using the observation signals for each of the three modals of the sensor 61, that is, three modal HMMs, are stored in the HMM storage units 72A to 72C, respectively. Here, the modals targeted by the modeling units 62A to 62C will be referred to as modals A to C, respectively. In this case, the modal A to C HMMs are stored in the HMM storage units 72A to 72C, respectively.

  Of course, the number of modals is not limited to three and may be two or more. However, in that case, there are as many modeling units as the number of modals corresponding to the modeling unit 62A.

  The causal unit 63 includes a causal learning unit 75, a causal table storage unit 76, and a causal estimation unit 77.

  The causal learning unit 75 generates a node transition recognized by the recognition unit 73K based on the HMM structure of the modal K (K is any one of A to C) and another modal L (L is an A to other than K). C) learns the relationship between the HMM states. The learning result is stored in the causal table storage unit 76. Details of the processing of the causal learning unit 75 will be described later.

  The behavior control unit 64 includes an execution management unit 78 and a controller unit 79. The controller unit 79 includes a controller table storage unit 80 and a controller storage unit 81. The controller table storage unit 80 and the controller storage unit 81 have basically the same functions and configurations as the controller table storage unit 42 and the controller storage unit 43 of FIG.

  When a goal is given, the execution management unit 78 determines the modal K corresponding to the goal and provides it to the modeling unit 62K. The planning unit 74K of the modeling unit 62K plans a route according to this goal, and provides it to the execution management unit 78. Therefore, the execution management unit 78 controls the action unit 65 so that the system (agent) of FIG. 15 acts along this route. In other words, the execution management unit 78 first inquires the cause-and-effect estimation unit 77 about the cause node causing the transition in order to realize the path. The causal estimation unit 77 estimates the cause node and the cause modal, and provides them to the execution management unit 78. The cause node and the cause modal will be described later. If the cause modal is a controller, the execution management unit 78 inquires the controller unit 79 and outputs a command corresponding to the controller. Further, if the cause node is a node on another modal L HMM, the execution management unit 78 recursively inquires the planning unit 74L about the route. Details of the series of processes of the execution management unit 78 will be described later.

  The action unit 65 takes a predetermined action in accordance with a command from the action control unit 64.

  In the following, the system of FIG. 15 will be described in further detail, taking as an example the case where a multimodal task is given as a task.

  Specifically, for example, the following multimodal task is given. That is, as shown in FIG. 16, an object is to allow a mobile robot 85 having a round shape to freely move within an area surrounded by a wall 86. Note that point 87 indicates that there is a light source there.

  The present applicant conducted the movement of the mobile robot 85 shown in FIG. 16 as an experiment using a simulator. That is, FIG. 16 is a diagram showing the appearance of the simulator. The prototype of the simulator of FIG. 16 adopted this time is “Olivier Michel. Khepera Simulator Package version 2.0: Freeware mobile robot simulator written at the University of Nice Sophia--Antipolis by Olivier Michel. Downloadable from the World Wide Web at http </wwwi3s.unice.fr/~om/khep-sim.html ".

  Here, the reason why the prototype is described is that the simulator adopted this time is not the simulator itself disclosed in the above-mentioned document, but a simulator incorporating an observation signal and an action as shown in FIG. Because there is.

  That is, the robot 85 includes a sensor unit 61 in addition to a distance sensor 61A that detects the distance to the wall 86 and an optical sensor 61B that detects the brightness of the light, as shown in FIGS. An energy sensor 61C is also mounted. The robot 85 can move by driving left and right wheels.

  It should be noted that the arrangement positions of the distance sensor 61A, the optical sensor 61B, and the energy sensor 61C in FIG. 16 do not necessarily match the actual arrangement positions.

  As shown in FIG. 17, the distance sensor 61A is assumed to be attached in 24 directions around the robot 85, and outputs each value corresponding to the distance to the wall 86 as an observation signal for each 24 directions. That is, in FIG. 17, the bar graphs with numbers 1 to 24 represent the signal strengths (instantaneous values) of observation signals in 24 directions.

  The optical sensor 61B is attached in 24 directions around the robot 85 (the same direction as the distance sensor 61A), and outputs each value corresponding to the brightness of the light as an observation signal for each 24 directions. However, in consideration of the characteristic that light diffuses, each value of the observation signal is not only a value in one direction but also a value that affects the surrounding sensors. . That is, in FIG. 17, the bar graphs with numbers 25 to 48 represent the signal intensities (instantaneous values) of observation signals in 24 directions.

  The energy sensor 61C observes energy defined as follows and outputs the observed value as an observation signal. That is, the energy is consumed in proportion to the amount of movement and replenished in proportion to the amount of light. In FIG. 17, a bar graph with the number 49 represents the signal intensity (instantaneous value) of the observation signal.

  As an action (behavior), that is, as a command given to the action unit 65, a movement amount command is employed. Specifically, a command (Δx, Δy) (hereinafter referred to as a movement command) for moving along the horizontal and vertical axes on the simulator of FIG. 16 is employed. Here, Δx is a movement command on the x-axis (horizontal direction in FIG. 16). Δy is a movement command on the y-axis (vertical direction in FIG. 16).

  In summary, the robot 85 has a detection function using a 24-dimensional distance sensor 61A, a 24-dimensional optical sensor 61B, and a one-dimensional energy sensor 61C. Has input / output function. The robot 85 is an agent controlled by the system shown in FIG. Therefore, the robot 85 aims to self-organize the internal state and control the internal state arbitrarily by performing these functions.

  The outline of the process flow of the system of FIG. 15 for achieving the multimodal task as described above is similar to the control process of the simple pendulum task of FIG. Therefore, only the points different from the control processing of the simple pendulum task in FIG. 5 will be described below.

  First, the system shown in FIG. 15 executes an HMM learning process in the same manner as the process in step S1 of the simple pendulum task control process shown in FIG. However, the HMM learning process executed by the system of FIG. 15 differs from the learning process of FIG. 5 as follows.

  That is, the system of FIG. 15 (the robot 85 agent) takes an action based on a random or simple innate rule (for example, a rule of moving in a certain direction and changing the direction when it hits the wall 86). In addition, when performing the action based on an innate rule, suppose that the innate controller 23 of FIG. 3 is provided also in the system of FIG.

  In the control processing of the simple pendulum task in FIG. 5, learning of the HMM is performed with the time series observation signal (time series signal of angle θ) as the only observation information.

  On the other hand, in the system of FIG. 15, the modality of the sensor unit 61 is known. That is, the robot 85 has a detection function including a 24D distance sensor 61A, a 24D optical sensor 61B, and a 1D energy sensor 61C. Therefore, the HMM learning process is performed for each of the three types of observation signals, that is, the observation signal (distance) of the distance sensor 61A, the observation signal (light) of the optical sensor 61B, and the observation signal (energy) of the energy sensor 61C. Note that the HMM learning process alone for one observation signal is basically the same as the HMM learning process in the simple pendulum task control process of FIG.

  That is, in the example of FIG. 15, the modeling unit 62A constructs a distance HMM and stores it in the HMM storage unit 72A. The modeling unit 62B constructs an optical HMM and stores it in the HMM storage unit 72B. The modeling unit 62C constructs an energy HMM and stores it in the HMM storage unit 72C.

  Example of display of HMM learning processing result by modeling unit 62A, that is, a table of results of learning time series of observation signal (distance) of distance sensor 61A by giving 400 node two-dimensional neighborhood structure HMM as an initial structure An example is shown in FIG. 18A.

  Example of display of HMM learning processing result by modeling unit 62B, that is, a table of results of learning time series of observation signal (light) of distance sensor 61B by giving 100-node two-dimensional neighborhood structure HMM as an initial structure An example is shown in FIG. 18B.

  A display example of the learning result of the modeling unit 62C, that is, a display example of the result of learning the time series of the observation signal (energy) of the distance sensor 61C by giving the two-dimensional neighborhood structure HMM of 100 nodes as the initial structure is shown in FIG. It is shown in 18C.

  In FIG. 18A, nodes (open circles) are plotted at the average position where the robot 85 exists when each node is recognized. However, the horizontal axis indicates the distance in the horizontal direction x, and the vertical axis indicates the distance in the vertical direction y.

  In FIG. 18B, nodes (open circles) are plotted at the average position where the robot 85 exists when each node is recognized. However, the horizontal axis indicates the distance in the horizontal direction x, and the vertical axis indicates the distance in the vertical direction y. Further, the center position, that is, the coordinates (0, 0) indicates the position of the point 87 which is a light source. However, the coordinate (0, 0) does not mean the position of one specific point 87, but means any one of the three points 87 in FIG.

  As for FIG. 18C, a node (open circle) is on a space between the energy value (vertical axis) and the distance (horizontal axis) to the light (point 87 which is a light source) closest to the average position where the robot 87 exists. Are marked).

  The distance HMM in FIG. 18A is expressed as a topological network having a maze configuration because the distance sensor 61A senses the wall 86.

  As for the light HMM in FIG. 18B, it can be seen that a radial network is formed around the light source (each point 87).

  As for the HMM of energy in FIG. 18C, since the energy only rises and falls, it can be seen that the network is like a single chain. The plotting method in FIG. 18 is a plotting method using the distance to the light (distance to each point 87) as the horizontal axis, so that a state transition is formed in the direction in which the energy increases when it is close to the light. On the other hand, it can be seen that a network in which the direction of state transition is determined as the energy decreases when it is far from the light, that is, a so-called ladder-type network is formed.

  If the target multimodal task is considered only by the distance HMM and action (command) and is controlled to an arbitrary state, the result is as shown in FIG. That is, the behavior control process can be realized in the same way as the simple pendulum task. In other words, in this case, the system of FIG. 15 may perform steps S2 to S5 of FIG.

  However, in the task setting of the multimodal task, there is not always a direct correlation between the state transition and action (action) of each HMM in FIGS. For this reason, it is not possible to solve such a multimodal task problem simply by learning the action (behavior) when the state transition occurs.

  For example, the transition of the HMM of energy in FIG. 18C is determined by the distance relationship between the light source (each point 87) and the robot 85 on the simulator in FIG. Therefore, the transition of the HMM of energy in FIG. 18C has nothing to do with the moving action of which direction the robot 85 has moved at a certain moment. However, the internal state such as the position in the maze represented by the HMM of the distance in FIG. 18C has a high relationship between the moving action of the robot 85 and the transitioning node.

  Therefore, the causal unit 63 is provided in the system of FIG. 15 in order to realize a function that allows the agent (robot 85) to autonomously find and control the relationship between the internal state and the action even in such a case. It is.

  That is, the causal unit 63 can execute the following process instead of steps S2 and S3 in FIG. 5 in order to reach the goal of the multimodal task.

  That is, at each time step, one currently recognized node is determined by the recognition result in each HMM in FIGS. 18A to 18C. As a recognition result in a single HMM, for example, the result of the recognition process in FIG. 12 for a single HMM can be adopted.

  In addition to the determined node, the action taken at that time is also discretized so that it can be handled as one modal. Hereinafter, such a modal is referred to as an action modal. The action modal state is referred to as an action state.

Here, the state of the HMM at time t including the action state is described as S _{k, i} (t). k indicates a modal number, and k = 0 indicates an action modal. Further, i is an index representing a state in a modal.

  In addition, a stochastic causal model as shown in Equation (1) is assumed.

・・・（１）

... (1)

Equation (1) shows that the next state of a certain modal depends on the current state and the state S _{m, l} of some other modal.

  Here, this “a certain modal” is referred to as a cause modal, and the current state node in the cause modal is referred to as a cause node. Then, when the cause modal is the action modal itself, the expression (1) is a simple behavior result model in which the node that transitions from the current state node (cause node) changes according to the action (action) taken at time t. Will be shown.

  Hereinafter, finding a cause modal and a cause node for each modal node transition is referred to as causal estimation. Since the detailed explanation of the causal estimation will be described later, only the outline of the causal estimation is described here.

  That is, causal estimation means that when a transition occurs in a certain mode, the state of other modals recognized at that time is counted, and a state occurring simultaneously with the transition is determined with high frequency. This makes it possible to find the corresponding cause modal and cause node for each transition. That is, the causal learning unit 75 discovers a cause modal and a cause node corresponding to each transition by performing such causal estimation for each transition. The cause modal and cause node for each transition are stored in the causal table storage unit 76 as a table. Hereinafter, such a table is referred to as a causal table.

  The outline of causal estimation will be further described with reference to FIGS.

  FIG. 20 shows the processing of the system of FIG. 15 in the case where the movement of the mobile robot 85 in the simulator of FIG. 16 is a task and only one distance is modal. For convenience of explanation, in FIG. 20 (FIGS. 21 and 24 described later), the actions (behavior) of the robot 85 (agent) are E (east), W (west), S (south), and N Assume that only (North) moving actions in four directions are adopted.

  In this case, the system of FIG. 15 performs the following steps S81A to S84A. That is, step S81A is a process of self-organization of the internal state by distance HMM structure learning. Step S82A is a process of estimating, that is, counting, an action that causes each state transition. Step S83A is a process of generating a route. Step S84A is an action execution process.

  On the other hand, FIG. 21 shows the same task as FIG. 20, but the modal shows the processing of the system of FIG. 15 when energy is present in addition to distance. That is, as described above, the system of FIG. 15 acquires the HMM independently in each modal in step S81B. In the example of FIG. 21, a distance HMM and an energy HMM are acquired. Next, in step S82B, the system of FIG. 15 generates a “(extended) cause state-result transition model” as shown in FIG. That is, the system of FIG. 15 searches for (causes counting) a cause state that causes each modal transition, with an action as one of the states (as an action state). For example, as shown in FIG. 21, in a specific transition on the distance HMM, if the action state is always north (N), the action state is counted. Also, for example, in the energy HMM, when the energy always increases at a place where food is present, the state of food in the distance HMM is counted.

  In this way, when the causal table is stored in the causal table storage unit 76, at that stage, the system of FIG. 15 finishes learning, sets an arbitrary target in the internal state formed by itself, You will be able to take action to achieve your goals.

  Therefore, the system of FIG. 15 makes a plan (planning) for realizing the achievement of the target. A process for making such a plan is a planning process. However, the planning process here is different from the planning process executed in step S4 of the control process of the simple pendulum task in FIG. Therefore, hereinafter, the planning process performed in the multimodal task is particularly referred to as a multistage planning process.

  Then, the system of FIG. 15 executes the action control process according to the result of the multistage planning process. However, the action control process here is different from the action control process executed in step S5 of the control process of the simple pendulum task in FIG. Therefore, hereinafter, the action control process performed in the multimodal task is particularly referred to as a multistage action control process.

  Hereinafter, it is shown that multi-modal HMMs can be controlled with an arbitrary state as a target, that is, multistage action control processing can be performed.

  In the multi-stage planning process, the planning unit 74K of the modeling unit 62K (K is one of A to C) is designated from the outside or is obtained internally as in the case of the simple pendulum task. Is set as the goal. However, a predetermined state (node) in a predetermined modal is set as the goal. That is, the goal modal and the goal state are set.

  Thereafter, the modeling unit 62K executes a planning process according to the flowchart of FIG. 13, for example. As a result, a route from the current state node (start node) in the modal K to the goal node is generated.

  For example, when the planning unit 74C executes the planning process for the energy modal C, a route such as that on the right side of FIG. 22 is set.

  Thereby, the action control part 64 can perform the following multistage action control processing.

  That is, the execution management unit 78 of the behavior control unit 64 acquires the cause modal and the cause node assigned to each transition on the route from the start node to the goal node from the causal estimation unit 77 of the causal unit 63. That is, when the causal estimation unit 77 receives a notification of the predetermined transition from the execution management unit 78, the causal table storage unit 76 searches and extracts the cause modal and the cause node assigned to the predetermined transition, and the execution management unit 78.

  Here, when the acquired cause modal is an action modal, the execution management unit 78 can acquire a command corresponding to the cause node from the controller unit 79 and provide it to the action unit 65. Therefore, in this case, the execution management unit 78 may execute the behavior control process according to the flowchart of FIG.

  On the other hand, when the cause modal is not an action modal, it is necessary to change the current state of the cause modal to the cause node. For example, in the example of FIG. 22, the cause modal is the light modal B, and it is necessary to transition the current state from the current state node to the cause node as shown in the diagram on the left side of FIG. 22. Therefore, when the cause modal is modal L (L is any one of A to C), the execution management unit 78 requests the modeling unit 62L of the cause modal L to perform a planning process. The planning unit 74L of the modeling unit 62L executes the planning process from the current state node to the cause node, and notifies the execution management unit 78 of the execution result, that is, the route. For example, in the example of FIG. 22, the planning unit 74B of the light modal B modeling unit 62B executes the planning process from the current state node to the cause node, as shown in the left diagram of FIG. The result, that is, the route is notified to the execution management unit 78.

  The execution management unit 78 acquires the cause modal and the cause node assigned to each transition on the notified route from the cause and effect estimation unit 77 of the cause and effect unit 63.

  In this way, the execution management unit 78 calls the cause modal and the cause node recursively. Then, the execution management unit 78 determines an action (command) at that time and provides it to the action unit 65 when the action modal that the agent can directly output is reached.

  After that, the execution management unit 78 returns to the original modal and executes the behavior control process in the modal at the stage where the cause node is reached through the procedure of the action control process. That is, in the example of FIG. 22, as shown by the dotted line in the stage where the cause node in the left diagram of FIG. 22, that is, the cause node on the HMM of the light modal B that is the cause modal, is reached. Returning to the modal C of the energy shown in the right side of FIG. 2, a transition of the current state node on the HMM occurs. Finally, the goal is achieved when the goal node finally reached (in the example of FIG. 22, the goal node on the HMM of the energy in the right diagram) can be reached.

  In addition, there are cases where there are not a single cause modal and cause node, and there are a plurality of problems such as many real world problems. For example, in the example of FIG. 16, since there are a plurality of light sources (three points 87) on the maze, any one of them may be the cause. Further, since sufficient energy can be obtained in the vicinity of light, any node in the vicinity thereof may be the cause. In such a case, an appropriate cause node and its route can be selected by selecting a route that is first reached when a plan is established for the cause node. Specifically, the system of FIG. 15 first selects one cause modal. Next, the system of FIG. 15 plans a route from the current state node with all of the candidate cause nodes as goal nodes in the cause modal. This planning is basically realized by executing the planning process of FIG. However, in the goal node arrival determination process in step S45, determination is performed for all goal nodes. According to this method, it is possible to select a goal node that arrives first and its route.

  Furthermore, it will be described in detail with specific task examples.

  First, as shown in FIG. 19, when a certain state on the distance HMM is designated as a goal node, this agent (the system in FIG. 15) has to go to a specific place on the simulator in FIG. Most likely it means. In that case, the change in location is very likely due to the action of the agent. Accordingly, each transition is associated with a moving action corresponding to the direction of the node under the route shown in FIG. That is, the cause and effect of the transition can be attributed to the action.

  Next, as shown in FIG. 23, when the goal is a state near a certain light source (point 87 in the example of FIG. 16) on the light HMM, the system of FIG. Process), if the current state node is near the light and the brightness is detected, it can be seen in which direction the light brightness will change, so the transition to the surrounding nodes is the same as the distance HMM Is associated with action. However, if the current position is in a state where light cannot be seen, the robot 85 as an agent does not know in which direction the light can be seen.

  Suppose that a route is created on the light HMM, and a route that approaches the light by approaching from the S (south) side of the light is created. Since there are three light sources (points 87) on the simulator shown in FIG. 16, there are three possible locations where the light can come to the S (south) side of the light from a place that cannot be seen by the robot 85. Will exist. If causal estimation is successfully performed, the node on the S (south) end side of the light source (any one of the three points 87 in the example of FIG. 16) from the node that cannot be seen by the robot 85 on the light HMM. The transition to is highly correlated with each of the southern nodes of the three light sources (three points 87 in the example of FIG. 16) on the distance HMM. Therefore, in the optical HMM, the system in FIG. 15 sets each of the south end nodes as the cause nodes as described above, executes the planning process for the distance HMM, and executes the control process. A transition occurs. When the planning process is executed when there are a plurality of goal nodes as described above, the system shown in FIG. 15 can first reach the route that can be reached only by checking the arrival conditions for the plurality of goal nodes in step S45 in FIG. Can be calculated.

  As a result, the robot 85 as an agent first moves to the nearest light source (point 87 in the example of FIG. 16) by finding a path to the outer edge of the light on the HMM at a distance, and further within the radiation of the light source. When entering, it becomes possible to move to the relative position with the target light by the transition of the light in the HMM.

  Next, let us consider controlling the state of the HMM of energy arbitrarily. For energy HMMs, there is no direct relationship with action for any transition. Assuming that the causal estimation is successful, the transition in the direction in which the energy rises indicates that the light source represented by the HMM of the distance (the point 87 in the example of FIG. 16) and the light source represented by the distance HMM (FIG. 16). In the example of 16, the causality with the node near the position of the point 87) is high. Furthermore, since there are three light sources, the causal is distributed to three on the distance HMM, whereas the same expression is used for any light in the optical HMM. Causal is high.

  Therefore, for example, when the above-described causal estimation of FIG. 21 is performed, the processing as shown in FIG. 24 can be executed. That is, in step S81C, a target for increasing energy is given. Then, in step S82C, the system of FIG. 15 first generates a path in which the energy sequentially increases with the energy HMM.

  The system of FIG. 15 generates a path approaching the light source (point 87 in the example of FIG. 16) on the light HMM based on the cause of the transition of this energy on the HMM. If necessary, the system shown in FIG. 15 uses the HMM representation of the distance (the point in the example of FIG. 16 is a point) even if the light source (point 87 in the example of FIG. 16) is not visible to the robot 85 as an agent. 87) is generated. In other words, the following steps S83C and S84C are executed. That is, step S83C is a process for realizing the cause state (to cause a transition). Step S84C is a process of creating a route.

  Based on this path, the system of FIG. 15 can act to approach the light from a distance and stay there until the energy is in the target state. In other words, the following processes of steps S85C and S86C are executed. That is, step S85C is processing for realizing the cause state. Step S86C is a preparation process that can be executed immediately. On the other hand, when the task of how to reduce the energy is set, it is only necessary to take an action that leaves the light source (point 87 in the example of FIG. 16) and stops there.

  As described above, the system of FIG. 15 can be recursively controlled by reducing the problem of state transition for each independent modal (event) and its path control, and determining the causal relationship between modals. As a result, it becomes possible to handle complicated behavior control problems without relying on prerequisite knowledge for tasks.

  By the way, the series of processes described above can be executed by hardware, but can also be executed by software.

  In this case, for example, a personal computer shown in FIG. 25 may be used as at least a part of the system described above.

  25, a CPU (Central Processing Unit) 91 executes various processes according to a program recorded in a ROM (Read Only Memory 92) or a program loaded from a storage unit 98 to a RAM (Random Access Memory) 93. The RAM 93 also appropriately stores data necessary for the CPU 91 to execute various processes.

  The CPU 91, ROM 92, and RAM 93 are connected to each other via a bus 94. An input / output interface 95 is also connected to the bus 94.

  The input / output interface 95 includes an input unit 96 including a keyboard and a mouse, an output unit 97 including a display, a storage unit 98 including a hard disk, and a communication unit 99 including a modem and a terminal adapter. It is connected. The communication unit 99 controls communication with other devices (not shown) via a network including the Internet.

  A drive 100 is also connected to the input / output interface 95 as necessary, and a removable medium 101 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately mounted, and a computer program read from them is loaded. It is installed in the storage unit 98 as necessary.

  When a series of processing is executed by software, a program constituting the software executes various functions by installing a computer incorporated in dedicated hardware or various programs. For example, a general-purpose personal computer is installed from a network or a recording medium.

  As shown in FIG. 25, the recording medium including such a program is distributed to provide a program to the user separately from the apparatus main body, and a magnetic disk (including a floppy disk) on which the program is recorded. , Removable media (package media) consisting of optical disks (including compact disk-read only memory (CD-ROM), DVD (digital versatile disk)), magneto-optical disks (including MD (mini-disk)), or semiconductor memory ) 101 as well as a ROM 92 on which a program is recorded and a hard disk included in the storage unit 98 provided to the user in a state of being incorporated in the apparatus main body in advance.

  FIG. 26 is a diagram illustrating an outline of a configuration example of an embodiment of a data processing device to which the present invention has been applied.

  In FIG. 26, the data processing apparatus stores a state transition model having a state and a state transition. The data processing device is a learning device that performs learning for modeling the modeling target by the state transition model, that is, based on the sensor signal observed from the modeling target, the probability statistical dynamic characteristic of the modeling target is determined. It is a kind of learning device that learns a given state transition model, and can be applied to the learning unit 31 described above.

  From the modeling target, a sensor signal obtained by sensing the modeling target is observed in time series, for example.

  The data processing apparatus learns the state transition model, that is, here, estimates the parameters of the state transition model and determines the structure using the sensor signal observed from the modeling target.

  Here, as the state transition model, for example, HMM, Bayesian network, POMDP (Partially Observable Markov Decision Process) or the like can be adopted. Hereinafter, for example, an HMM is adopted as the state transition model.

  FIG. 27 shows an example of an HMM.

  The HMM is a state transition model having a state and a transition between states.

  FIG. 27 shows an example of a three-state HMM.

  In FIG. 27 (the same applies to the following drawings), a circle represents a state, and an arrow represents a state transition.

In FIG. 27, s _i (i = 1, 2, 3 in FIG. 27) represents a state, and a _ij represents a state transition probability from the state s _i to the state s _j . Further, b _j (x) represents an output probability density function in which the observation value x is observed at the time of state transition to the state s _j , and π _i represents an initial probability that the state s _i is an initial state.

As the output probability density function b _j (x), for example, a mixed normal probability distribution is used.

Here, the HMM (continuous HMM) is defined by the state transition probability a _ij , the output probability density function b _j (x), and the initial probability π _i . These state transition probabilities a _ij , output probability density function b _j (x), and initial probability π _i , HMM parameters λ = {a _ij , b _j (x), π _i , i = 1,2,.・ ・ N, j = 1,2, ..., N}. N represents the number of states of the HMM.

  As described above, the Baum-Welch re-estimation method is widely used as a method for estimating the HMM parameter λ. The Baum-Welch re-estimation method is a parameter estimation method based on an EM algorithm (EM (Expectation-Maximization) algorithm).

According to Baum-Welch's re-estimation method, based on the observed time series data x = x ₁ , x ₂ , ..., x _T , the occurrence probability is the probability that the time series data will be observed (occurred). The HMM parameter λ is estimated so as to maximize the likelihood to be obtained.

Here, x _t represents a signal (sample value) observed at time t, and T represents the length (number of samples) of time-series data.

  The Baum-Welch re-estimation method is a parameter estimation method based on likelihood maximization, but it does not guarantee optimality, and it converges to a local solution depending on the structure of the HMM and the initial value of the parameter λ. Sometimes. For details of HMM and Baum-Welch re-estimation methods, see, for example, Laurence Rabiner and Biing-Hwang Juang, “Basics of Speech Recognition (Up / Down)”, NTT Advanced Technology Co., Ltd. (hereinafter also referred to as Document A) Etc. are described.

  HMMs are widely used in speech recognition, but in the HMMs used in speech recognition, the number of states, the state transition method, and the like are generally determined in advance.

  FIG. 28 shows an example of an HMM used for speech recognition.

  The HMM in FIG. 28 is called a left-to-right type.

In Figure 28, the number of states has become a 3, state transition, a self-transition (the state transition from the state s _i to the state s _i), the structure allows only the state transition from the left to the right state It is constrained.

FIG. 27 shows an HMM with no state transition restrictions, that is, a state transition from an arbitrary state s _i to an arbitrary state s _{j as} compared to the HMM with a state transition restriction like the HMM in FIG. An HMM that can do this is called an Ergodic HMM.

  The ergodic HMM is an HMM having the highest degree of freedom as a structure. However, as the number of states increases, it is difficult to estimate the parameter λ.

  For example, when the number of states of the ergodic HMM is 1000, the number of state transitions is 1 million (= 1000 × 1000).

Therefore, in this case, for example, regarding the state transition probability a _ij in the parameter λ, it is necessary to estimate one million state transition probabilities a _ij .

  Depending on the modeling target, limited state transitions may be sufficient for the required state transitions, but if you do not know in advance how to constrain the state transitions, It is very difficult to appropriately estimate such a large number of parameters λ. In addition, when the appropriate number of states is not known in advance and the information for determining the structure of the HMM is not known in advance, it is more difficult to obtain the appropriate parameter λ.

  The data processing apparatus of FIG. 26 determines the structure of the HMM appropriate for the modeling target without giving constraints in advance on the structure of the HMM, that is, the number of states of the HMM and the state transition, and the HMM. Learning is performed to estimate the parameter λ.

  FIG. 29 is a block diagram illustrating a configuration example of the data processing apparatus of FIG.

  29, the data processing apparatus includes a time-series data input unit 111, a data adjustment unit 112, a parameter estimation unit 113, an evaluation unit 114, a model storage unit 115, an initial structure setting unit 116, and a structure adjustment unit 117.

The time-series data input unit 111 receives sensor signals observed from the modeling target. The time series data input unit 111 is based on the sensor signal observed from the modeling target, and the time series data observed from the modeling target (hereinafter also referred to as observation time series data) x = x ₁ , x ₂ ,. - the x _T, and outputs the data adjusting unit 112.

  That is, the time series data input unit 111 supplies, for example, a time series sensor signal observed from the modeling target to the data adjustment unit 112 as it is as observation time series data x.

  The time series data input unit 111 supplies the observation time series data x to the data adjustment unit 112 in response to a request from the evaluation unit 114.

The data adjustment unit 112 is time series data used for HMM learning, that is, an observation time series supplied from the time series data input unit 111 according to the progress of HMM learning stored in the model storage unit 115 described later. The data x is adjusted, and adjusted time-series data (hereinafter also referred to as adjusted time-series data) x ′ = x ₁ ′, x ₂ ′,..., X _{T ′} ′ is output.

That is, the data adjustment unit 112 performs, for example, a downsampling process (downsampling process) on the observed time series data x = x ₁ , x ₂ ,..., X _T from the time series data input unit 111. And output adjusted time series data x ′ = x ₁ ′, x ₂ ′,..., X _{T ′} ′ obtained by the downsampling process.

  For example, when the observation time series data x is time series data sampled at 1000 Hz, the data adjustment unit 112 adjusts the observation time series data x sampled at 1000 Hz by the downsampling process. Convert to post-time series data x ′.

  Here, according to the downsampling process, the high-frequency component included in the observation time series data x is removed, and the adjusted time series data x ′ is a macro feature of the observation time series data x, that is, the observation time series data x It becomes time series data including only the low frequency component.

  By learning the HMM that is the state transition model stored in the model storage unit 115 using the adjusted time series data x ′ including only the macro features of the observed time series data x, the HMM The macro features of the observation time series data x are acquired.

  How to adjust the observation time series data x in the data adjustment unit 112, that is, in this case, the observation time series data x is converted to the adjusted time series data x ′ of what sampling frequency Whether or not to do so is an important problem in learning to appropriately acquire the characteristics of the observation time series data x.

  As described above, the data adjustment unit 112 adjusts the observation time series data x according to the progress of the HMM learning. The adjustment is performed as the adjusted time series data x ′ increases as the HMM learning progresses. The time series data including only the macro features of the observation time series data x is changed to the time series data including the micro features, that is, the high frequency components of the observation time series data x.

  For example, as the HMM learning progresses, the data adjustment unit 112 gradually changes the sampling frequency of the adjusted time-series data x ′ from a small value to a large value.

  Specifically, for example, the data adjustment unit 112 sets the sampling frequency of the adjusted time-series data x ′ to 10 Hz at the initial stage of learning, and thereafter, as the learning proceeds, the sampling frequency of the adjusted time-series data x ′. Are sequentially changed to 50Hz, 100Hz, 500Hz, 1000Hz.

  In this case, the HMM acquires macro features of the observation time series data x in the initial stage of learning, and then acquires micro features of the observation time series data x as learning progresses.

  Note that progress information representing the progress of learning is supplied from the evaluation unit 114 to the data adjustment unit 112. The data adjustment unit 112 recognizes the progress of learning based on the progress status information from the evaluation unit 114 and changes the sampling frequency of the adjusted time-series data x ′.

  Further, in the data adjustment unit 112, as the learning progresses, the observation time series data x is changed from the adjusted time series data x ′ including the macro features to the adjusted time series data x ′ including the micro features. As the adjustment process, a filter bank process can be adopted in addition to a downsampling process (a process of thinning the observation time series data x in the time direction).

  In the case of adopting filter bank processing in the data adjustment unit 112, the observation time series data x is divided into frequency components of the predetermined division number by being filtered using the filter bank of the predetermined division number. The Then, the predetermined number of frequency components are output as adjusted time-series data x ′.

  In this case, in the data adjustment unit 112, as the learning progresses, the number of divisions of the filter bank is gradually changed to a large value.

As described above, the data adjustment unit 112 adjusts the observation time series data x and outputs the adjusted time series data x ′ = x ₁ ′, x ₂ ′,..., X _{T ′} ′. , T ′ represents the length of the adjusted time-series data x ′.

  The adjusted time series data x ′ that the data adjustment unit 112 adjusts and outputs the observation time series data x is supplied to the parameter estimation unit 113 and the structure adjustment unit 117.

  The parameter estimation unit 113 estimates the parameter λ of the HMM stored in the model storage unit 115 using the adjusted time series data x ′ from the data adjustment unit 112.

  That is, the parameter estimation unit 113 uses the adjusted time-series data x ′ from the data adjustment unit 112 to estimate the HMM parameter λ stored in the model storage unit 115 by, for example, the Baum-Welch re-estimation method. Do.

  The parameter estimation unit 113 supplies the new parameter λ obtained by the estimation of the parameter λ of the HMM to the model storage unit 115 and stores it in the form of overwriting.

  Note that the parameter estimation unit 113 uses the value stored in the model storage unit 115 as the initial value of the parameter λ when estimating the parameter λ of the HMM.

  Here, it is assumed that the process of estimating a new parameter λ in the parameter estimation unit 113 is counted as one learning.

  Each time the parameter estimation unit 113 performs a process of estimating a new parameter λ, the learning number is incremented by 1, and the learning number is supplied to the evaluation unit 114.

  Further, the parameter estimation unit 113 obtains the likelihood that the adjusted time series data x ′ from the data adjustment unit 112 is observed from the HMM defined by the new parameter λ, and supplies the likelihood to the evaluation unit 114.

  Note that the likelihood that the parameter estimation unit 113 supplies to the evaluation unit 114 can be obtained using the observation time series data x instead of the adjusted time series data x ′.

  The evaluation unit 114 evaluates the HMM that has been learned based on the likelihood from the parameter estimation unit 113 and the number of learnings, that is, the HMM in which the parameter λ is estimated by the parameter estimation unit 113, and evaluates the HMM. Based on the result, it is determined whether or not to end the HMM learning.

  That is, for example, the evaluation unit 114 evaluates that acquisition of the characteristics (time series pattern) of the observation time series data x by the HMM is insufficient until the number of learning from the parameter estimation unit 113 reaches a predetermined number. It is determined that the learning of the HMM is continued.

  Then, when the number of learning times from the parameter estimation unit 113 reaches a predetermined number, the evaluation unit 114 evaluates that the feature of the observation time series data x is sufficiently acquired by the HMM and ends the learning of the HMM. Judge that.

  In addition, the evaluation unit 114 evaluates that the feature (time-series pattern) of the observation time-series data x by the HMM is insufficient until the likelihood from the parameter estimation unit 113 reaches a predetermined value, for example. It is determined that the learning of the HMM is continued.

  Then, when the likelihood from the parameter estimation unit 113 reaches a predetermined value, the evaluation unit 114 evaluates that the feature of the observation time series data x is sufficiently acquired by the HMM and ends the learning of the HMM Judge that.

  When the evaluation unit 114 determines to continue learning of the HMM, the evaluation unit 114 requests the time-series data input unit 111, the data adjustment unit 112, and the structure adjustment unit 117 to perform predetermined processing.

  That is, the evaluation unit 114 requests the time series data input unit 111 to supply observation time series data.

  Furthermore, the evaluation unit 114 supplies the learning frequency and likelihood to the data adjustment unit 112 as progress status information indicating the progress of learning, thereby performing a downsampling process corresponding to the progress of learning. Request.

  In addition, the evaluation unit 114 requests the structure adjustment unit 117 to adjust the structure of the HMM stored in the model storage unit 115 as the learning progresses.

  The model storage unit 115 stores a state transition model, for example, an HMM.

  That is, when a new parameter of the HMM is supplied from the parameter estimation unit 113, the model storage unit 115 updates (overwrites) the stored value (the stored HMM parameter) with the new parameter.

  In addition, the model storage unit 115 is based on the HMM structure (initial structure) that is initialized by the initial structure setting unit 116 before the learning of the HMM is started, that is, based on restrictions on the number of states of the HMM and state transitions. The initial value of the HMM parameter to be determined is stored.

  Here, the estimation of the parameters of the HMM by the parameter estimation unit 113 is performed from the initial value determined by the initial structure setting unit 116.

  Further, the structure of the HMM stored in the model storage unit 115 is adjusted by the structure adjustment unit 117 as the learning progresses, but updating of the stored value in the model storage unit 115 is performed by the structure adjustment unit 117. This is also done by HMM parameters obtained by adjusting the structure.

  The initial structure setting unit 116 initializes the structure of the HMM before the learning of the HMM is started, and sets the parameters (initial parameters) of the HMM of the initialized structure (initial structure).

  That is, the initial structure setting unit 116 sets the initial structure of the HMM, that is, the number of states and state transition of the HMM.

  Here, predetermined restrictions can be imposed on the number of states and state transition of the HMM as the initial structure.

  For example, the initial structure setting unit 116 sets the number of states of the HMM to a predetermined number or less as a predetermined constraint.

  Specifically, for example, the initial structure setting unit 116 sets the number of states of the HMM to a small number such as 16 or 100.

  Furthermore, the initial structure setting unit 116 appropriately arranges the number of states set as the initial structure in an L-dimensional space of one dimension or more (L is a positive integer).

  For example, when the number of states set as the initial structure is 16, and the 16 states are arranged in the two-dimensional space, the initial structure setting unit 116 sets the 16 states in the two-dimensional space, for example, Arrange in a grid pattern.

  Thereafter, the initial structure setting unit 116 sets state transition, that is, self-transition and state transition to another state for the 16 states arranged in the two-dimensional space.

  Predetermined constraints such as a sparse structure can be applied to the state transition set for the state.

  Here, a sparse structure is not a dense state transition such as an ergodic HMM that can make a state transition from an arbitrary state to an arbitrary state, but the state that can make a state transition from a certain state is very limited It is a structure that has been.

  Note that here, even in a sparse structure, at least one state transition to another state exists, and a self-transition exists.

When the initial structure setting unit 116 obtains the initial structure by, for example, initializing the structure of the HMM to a sparse structure by applying predetermined restrictions as described above, the initial structure setting unit 116 converts the initial structure to the HMM of the initial structure. , Initial parameters, ie, state transition probability a _ij , output probability density function b _j (x), and initial value of initial probability π _i are set.

That is, for example, the initial structure setting unit 116 sets, for each state, the state transition probability a _ij of possible (valid) state transitions that can be made from that state (M is the number of possible state transitions). If there is, it is set to 1 / M), and the state transition probability a _ij of state transitions other than state transitions that cannot be performed, that is, state transitions set as sparse state transitions, is set to 0.

Further, for example, when a normal distribution is used as the output probability density function b _j (x), the initial structure setting unit 116 uses the observed time series data x = x ₁ , x obtained from the time series data input unit 111. _2, ..., the average value μ and variance sigma ² of x _T, calculated according to the following equation, a normal distribution defined by the average value μ and variance sigma ^2, the output probability density function b _j for each state s _j Set to (x).

μ ＝ (1 / T) Σx _t
σ ² = (1 / T) Σ (x _t -μ) ²

  Here, in the above equation, Σ means summation in which time t is changed from 1 to the length T of the observation time series data x.

Furthermore, the initial structure setting unit 116 sets the initial probability [pi _i in each state s _i to a uniform value. That is, when the number of states of the HMM of the initial structure is N, the initial structure setting unit 116, the N states s _i each initial probability [pi _i, is set to 1 / N.

In the initial structure setting unit 116, the initial structure and initial parameters λ = {a _ij , b _j (x), π _i , i = 1, 2,..., N, j = 1, 2,. } Are set and supplied to the model storage unit 115. The (initial) structure and (initial) parameter λ of the HMM stored in the model storage unit 115 are updated by learning.

  The structure adjustment unit 117 adjusts the structure of the HMM stored in the model storage unit 115 using the adjusted time series data x ′ from the data adjustment unit 112 in response to a request from the evaluation unit 114. The adjustment of the structure of the HMM performed by the structure adjustment unit 117 includes adjustment of the parameters of the HMM that are necessary according to the adjustment of the structure.

  There are six types of HMM structure adjustment performed by the structure adjustment unit 117: state division, state merging, state addition, state transition addition, state deletion, and state transition deletion. Details thereof will be described later.

  Next, the processing of the initial structure setting unit 116 in FIG. 29 will be further described with reference to FIG.

  In the initial structure setting unit 116, an ergodic structure can be set as the initial structure of the HMM, or a sparse structure can be set by applying predetermined restrictions.

  FIG. 30 shows an HMM having a sparse initial structure (state transition).

  Here, in FIG. 30 (the same applies to the following drawings), a circle represents a state, and an arrow represents a state transition. Further, in FIG. 30 (the same applies to the following drawings), a bidirectional arrow connecting two states represents a state transition from one of the two states to the other and a state transition from the other to the other. Further, in FIG. 30 (the same applies to the following drawings), each state can perform self-transition, and an arrow indicating the self-transition is not shown.

  In FIG. 30, 16 states are arranged in a lattice pattern on a two-dimensional space. That is, in FIG. 30, four states are arranged in the horizontal direction, and four states are arranged also in the vertical direction.

  Now, assuming that the distance between the adjacent states in the horizontal direction and the distance between the adjacent states in the vertical direction are both 1, the state transition to a state where the distance is 1 or less is possible. This shows an HMM with a structure in which the state transition to the state cannot be made.

  FIG. 30B shows an HMM having a structure in which a state transition to a state having a distance of √2 or less is possible and a state transition to another state is not possible.

  As described above, the method for setting a sparse initial structure is as follows. For a state arranged in the L-dimensional space, a state transition to a nearby state (self-transition) according to the distance between the states. It is not limited to the method of applying a restriction that enables only

  In other words, as a method of setting a sparse initial structure, for example, when a certain state is focused, a random number of states are randomly selected from all states, and the state is selected. It is possible to adopt a method of restricting state transitions only for state transitions (including self-transitions) to randomly selected states.

  Further, as a method of setting a sparse initial structure, it is possible to adopt an HMM structure as shown in FIG. The HMM in FIG. 10A shows an HMM due to a three-dimensional grid constraint. The HMM in FIG. 10B shows an HMM due to a two-dimensional random arrangement constraint. The HMM in FIG. 10C represents an HMM by a small world network.

  Next, with reference to FIG. 31, the division of the state performed by the structure adjustment unit 117 as adjustment of the structure of the HMM will be described.

As described above, in the drawing, a circle represents a state, but in the following, a circle with a number i is referred to as a state s _i .

  FIG. 31A shows the HMM before the state is divided.

In FIG. 31A, the HMM has six states s ₁ , s ₂ , s ₃ , s ₄ , s ₅ , s ₆ , between states s ₁ and s _2, and between states s ₁ and s ₄ . Between states s ₂ and s ₃ , between states s ₂ and s ₅ , between states s ₃ and s ₆ , between states s ₄ and s ₅ , and states s ₅ and s ₆ Both state transitions and self-transitions are possible.

  FIG. 31B illustrates the HMM after the state is divided for the HMM of FIG. 31A.

  The division of the state is performed in order to increase the scale of the HMM.

In FIG. 31B, among the states s ₁ to s ₆ of the HMM in FIG. 31A, for example, the state s ₅ is divided.

Division of a state s ₅ is capable of the same state transition and state s ₅ of the subject of the split, and, by adding a new state s ₇ can provide two-way state transitions in between the state s ₅ Done.

In FIG. 31A, since the state s ₅ can undergo state transition and self-transition with each of the states s ₂ , s ₄ , and s ₆ , the structure adjusting unit 117 performs the following for the new state s ₇ . Similarly to the state s ₅ , state transitions and self-transitions between the states s ₂ , s ₄ , and s ₆ are set as valid (possible) state transitions.

Furthermore, the structure adjustment unit 117 sets a state transition between the state s ₅ and the new state s ₇ as an effective state transition.

In addition, in the state division, the structure adjustment unit 117 sets a new state s ₇ parameter, for example, by taking over the parameter of the state s _{5 to be} divided.

That is, the structural adjustment unit 117, the initial probability [pi ₇ of the new state s _7, and sets the initial probability [pi ₅ of the subject's condition s ₅ split (π ₇ = π _5), the new state s ₇ The output probability density function b ₇ (x) is set to the output probability density function b ₅ (x) of the state s _{5 to be} divided (b ₇ (x) = b ₅ (x)).

Furthermore, the structure adjustment unit 117 changes the state transition probability a _i7 of the state transition from the state s _i (in FIG. 31, i = 1, 2, 3, 4, 6) to the new state s ₇ from the state s _i. The state transition probability a _i5 to state s ₅ to be divided is set to a _i5 (a _i7 = a _i5 ).

Further, the structure adjustment unit 117 sets the state transition probability a _7i of the state transition from the new state s ₇ to the state s _{i as} the state transition probability a _5i from the state s ₅ to the state s _i to be divided. (A _7i = a _5i ).

Then, the structure adjustment unit 117 changes the state transition probability a ₅₇ of the state transition from the division target state s ₅ to the new state s ₇ and the state transition from the new state s ₇ to the division target state s ₅ . and a state transition probability a ₇₅ of a suitable value, i.e., for example, set to a ₅₇ = a ₇₅ = 0.5 and the like.

  Further, the structure adjustment unit 117 performs normalization processing on necessary parameters of the HMM after the state division, and ends the state division processing.

That is, the structure adjustment unit 117 performs normalization processing satisfying the following expression on the initial probability π _i of the HMM after the state division and the state transition probability a _ij .

Σπ _j = 1
Σa _ij = 1 (i = 1,2, ..., N)

  Here, in the above formula, Σ means a summation in which the variable j representing the state is changed from 1 to the number N of states of the HMM after the state is divided. In FIG. 31, the state number N of the HMM after the state division is 7.

As a method for normalization process that satisfies the above formula, for example, the sum of the normalized pre-treatment of the initial probability _{π j Σπ j = π 1 +} π 2 + ··· + π N, normalization pretreatment There is a method of dividing the initial probability π _j of. The same applies to the normalization processing of the state transition probability a _ij .

  In the state division, the state to be divided is not limited to one state.

That is, as a state to be divided, for example, a predetermined number n of 1 to N is randomly selected from _N states s ₁ to s _N of the HMM before the state is divided. be able to.

In addition, as a state to be divided, for example, the variance σ ² defining the output probability density function b _j (x) is large from among the _N states s ₁ to s _N of the HMM before the state is divided It is possible to select the top n states, that is, the top n states having relatively large variations in the observed values observed from the states.

  Here, the number n of states to be divided can be set randomly, or can be set to a fixed value. In any case, by dividing the state, the structure of the HMM is updated to a structure in which the number of states is increased by n from the number of states before the division.

  Next, with reference to FIG. 32, the merging of states performed by the structure adjustment unit 117 as adjustment of the structure of the HMM will be described.

  FIG. 32A shows the HMM before the state merging is performed, and is the same HMM as in FIG. 31A described above.

  FIG. 32B shows the HMM after the state merging is performed on the HMM of FIG. 32A.

  A state merge is performed to degenerate a redundantly assigned state.

In FIG. 32B, for example, the state s ₅ of the states s ₁ to s ₆ of the HMM in FIG. 32A is merged, and the merged state s ₅ is merged with the merged state s _6. Yes.

State s _5, merge to state s ₆ is a state s ₅ of merged, deletes the state transition between the states s ₆ of the merged, the merged state s ₅ is a merged The state transition between the state s ₅ and the other states excluding the state s ₆ to be merged (hereinafter also referred to as a unique state transition) is inherited by the state s ₆ to be merged. It is done by deleting the state s ₅ of merged.

Therefore, the structure adjustment unit 117 deletes (invalidates) the state transition between the merged state s ₅ and the merged state s ₆ .

Furthermore, in FIG. 32A, since the unique state transition of the state s ₅ is a state transition between the states s ₂ and s ₄ , the structure adjustment unit 117 includes the state s ₆ to be merged, A state transition between each of the states s ₂ and s ₄ is added (set) as an effective state transition.

The restructuring unit 117 deletes the state s ₅ of merged.

Furthermore, the structure adjusting unit 117, the merging state, for example, among the state transitions of the state s ₆ of the merged, for inherited state transition from the state s ₅ of merged, the merged state s ₅ State transition probabilities a _i6 and a _6j are set so as to take over the state transition probabilities a _i5 and a _5j .

That is, in FIG. 32, since the state s _{6 to} be merged is inherited from the state s ₅ to be merged, it is a state transition between the states s ₂ and s _4. Then, the state transition probability a ₆₂ of the state transition from the merged target state s ₆ to the state s ₂ is set to the state transition probability a ₅₂ of the state transition from the target state s ₅ to the state s ₂ (a ₆₂ = a ₅₂ ).

Furthermore, the structure adjustment unit 117 sets the state transition probability a ₂₆ of the state transition from the state s ₂ to the merged state s ₆ and the state transition probability a of the state transition from the state s ₂ to the merged state s ₅ . Set to ₂₅ (a ₂₆ = a ₂₅ ).

Similarly, the structure adjustment unit 117 uses the state transition probability a ₆₄ of the state transition from the merged target state s ₆ to the state s _{4 as} the state transition probability of the state transition from the merge target state s ₅ to the state s ₄ . Set to a ₅₄ (a ₆₄ = a ₅₄ ).

Furthermore, the structure adjustment unit 117 sets the state transition probability a ₄₆ of the state transition from the state s ₄ to the merged state s ₆ and the state transition probability a of the state transition from the state s ₄ to the merged state s ₅ . Set to ₄₅ (a ₄₆ = a ₄₅ ).

  Then, the structure adjustment unit 117 performs normalization processing on necessary parameters of the HMM after the state merging, and ends the state merging processing.

That is, the structure adjustment unit 117 performs normalization processing similar to that in the case of state division on the initial probability π _i and state transition probability a _ij of the HMM after state merging.

  In the state merging, a set of a state to be merged and a state to be merged (hereinafter also referred to as a merge set) is not limited to one set.

That is, as a set of states (pairs) to be merged, from among a set of states capable of bidirectional state transition in the _N states s ₁ to s _N of the HMM before merging of states, for example, It is possible to select the top n (n is a value of 1 or more) states having a large correlation between the states.

  Note that the number n of sets in a merge set can be set randomly or can be set to a fixed value. In any case, by merging states, the structure of the HMM is updated to a structure in which the number of states is reduced by n from the number of states before merging.

  Here, the correlation between states used when selecting a set of states to be a merge set will be described.

  The correlation between states represents the degree of similarity between state transitions to other states (including self-transitions), state transitions from other states, and observations observed from the states, for example, It is obtained as follows.

  That is, as described with reference to FIG. 29, the adjusted time series data x ′ used for estimating the parameters of the HMM in the parameter estimating unit 113 is supplied from the data adjusting unit 112 to the structure adjusting unit 117.

  The structure adjustment unit 117 uses the adjusted time series data x ′ from the data adjustment unit 112 to obtain a correlation between the states of the HMM stored in the model storage unit 115.

That is, the structure adjustment unit 117 applies a forward-backward algorithm to the HMM stored in the model storage unit 115, and the adjusted time-series data x ′ = x from the data adjustment unit 112. Forward and backward probability p _i that is the probability of being in state s _i (probability that the state at time t is state s _i) at each time t of ₁ ′, x ₂ ′,..., X _{T ′} ′ Find (t).

Here, the forward backward algorithm is a forward probability α _i (t) obtained by propagating the probability to reach each state s _i in the time direction and a backward probability β obtained by propagating backward. _This is an algorithm for calculating a probability value integrated with _i (t).

  The forward backward algorithm is described in Document A above.

The structure adjustment unit 117 observes the data x ₁ ′, x ₂ ′,..., X _t ′ of the adjusted time series data x ′ with respect to the HMM stored in the model storage unit 115, and sets the time t Next, the forward probability α _i (t) in the state s _i is obtained. In addition, the structure adjustment unit 117 is in the state s _{i at} time t with respect to the HMM stored in the model storage unit 115, and thereafter, the data x _t ′, x _{t + 1 in the} adjusted time series data x ′. ', ..., x _T' 'observed (when data x _t ', x _{t + 1} ', ..., x _T' 'are observed after time t, the state at time t Find the backward probability β _i (t) in s _i .

The restructuring unit 117, forward probability alpha _i (t) and by using the backward probability beta _i (t), at time t, determining the forward backward probability being in a state s _i p _i (t).

The structure adjustment unit 117 calculates a forward / backward probability p _i (t) in each state s _i at each time t = 1, 2,..., T ′ of the adjusted time-series data x ′.

Here, the forward and backward probabilities p _i (1), p _i (2),..., P _i (T ′) of a state s _i are time series data having a length T ′, and this time series The data is also expressed as p _i (= p _i (1), p _i (2),..., P _i (T ′)).

Now, assuming that the correlation between a certain state s _i and another state s _j is represented as p _i * p _j , the structure adjustment unit 117 has a correlation p _i * between the certain state s _i and another state s _j . the p _j, for example, forward and backward probability p _i = p _i (1) of the state _{s i, p i (2)} , ···, and p _i (T '), forward backward probability of the state s _j p _j = Using p _j (1), p _j (2),..., p _j (T ′), the following equation is used.

p _i * p _j = Σp _i (t) p _j (t)

  Here, in the above formula, Σ means a summation in which the time t is changed from 1 to the length T ′ of the adjusted time-series data x ′.

Correlation p _i * p _j in the state s _i and s _j, if forward and backward probabilities p _i of the state s _i, a pattern of temporal changes in the forward backward probabilities p _j state s _j are similar, i.e. , Higher if one of the states s _i and s _j is redundant in addition to the other.

In this case, between the state s _i and s _j, the bidirectional state transition is present, the set of the state s _i and s _j are chosen as the merge set. Then, one of the redundant states s _i and s _j is set as a merge target, and the other is set as a merge target, and the merge target state is merged with the merge target state.

  The structure adjustment unit 117 can also obtain the correlation between the states of the HMM stored in the model storage unit 115 using the observed time series data x instead of the adjusted time series data x ′.

  Further, which of the two states selected as the merge set is to be merged or merged can be selected at random, for example.

  Next, with reference to FIG. 33, the addition of a state performed by the structure adjustment unit 117 as adjustment of the structure of the HMM will be described.

  FIG. 33A shows an HMM before a state is added, and is the same HMM as in FIG. 31A described above.

  FIG. 33B shows the HMM after the state is added to the HMM of FIG. 33A.

  The addition of the state is performed in order to increase the scale of the HMM, as in the state division described with reference to FIG.

In FIG. 33B, for example, the state s ₅ of the states s ₁ to s ₆ of the HMM of FIG. 33A is added, and a new state s ₇ is added to the state s ₅ . Yes.

Additional states are done by adding the self-transition, the two-way state transition and the new state s ₇ possible between the state s ₅ for which to add a condition.

Therefore, the structure adjustment unit 117 sets the self-transition and the state transition between the state s ₅ as a valid state transition for the new state s ₇ .

In addition, in the addition of the state, the structure adjustment unit 117 sets the parameter of the new state s _{7 in} a manner that, for example, takes over the parameter of the state s ₅ to which the state is to be added.

That is, the structural adjustment unit 117, the initial probability [pi ₇ of new state s _7, additional and sets the initial probability [pi ₅ of the target state _{s 5 (π 7 = π 5} ), the new state s ₇ The output probability density function b ₇ (x) is set to the output probability density function b ₅ (x) of the additional target state s ₅ (b ₇ (x) = b ₅ (x)).

Furthermore, the structure adjusting unit 117, the state s ₅ of the subject to be added with the state transition probability a ₅₇ of state transition from the state s ₅ of the subject to a new state s _7, a state from the new state s ₇ to add state The state transition probability a ₇₅ of the state transition to is set to an appropriate value, for example, a ₅₇ = a ₇₅ = 0.5 or the like.

  Furthermore, the structure adjustment unit 117 performs normalization processing on the necessary parameters of the HMM after the state addition, and ends the state addition processing.

That is, the structure adjustment unit 117 performs normalization processing similar to that in the case of state division on the initial probability π _i and state transition probability a _ij of the HMM after the state is added.

  In addition, in the addition of a state, a state to which a state is added is not limited to one state.

In other words, the states to be added as states are, for example, an arbitrary number n of 1 to N in a random manner from among the _N states s ₁ to s _N of the HMM before the addition of the states. Can be selected.

In addition, as a state to which a state is to be added, for example, a variance σ ² defining an output probability density function b _j (x) from among the _N states s ₁ to s _N of the HMM before the addition of the state The top n states with a large value, that is, the top n states with relatively large variations in observed values observed from the states can be selected.

  Here, the number n of states to which a state is to be added can be set at random or can be set to a fixed value. In any case, the addition of the state updates the structure of the HMM to a structure in which the number of states is increased by n from the number of states before the addition.

  Note that the addition of states and the division of states described in FIG. 31 are common in that the number of states of the HMM increases. However, the addition of the state is different from the division of the state in which the new state takes over the state transition of the state to be divided, in that the new state does not take over the state transition of the state to which the state is added.

  Therefore, regarding the state transition, in the HMM after the division of the state of FIG. 31, the new state is not only the state transition between the state to be divided but also the influence of other state transitions of the state to be divided. However, in the HMM after the state is added, the new state is directly affected only by the state transition between the state to which the state is added.

  As a result, in the addition of the state, the independence of the new state becomes higher than in the case of the division of the state.

  Next, with reference to FIG. 34, description will be given of addition of state transition performed by the structure adjustment unit 117 as adjustment of the structure of the HMM.

  FIG. 34A shows an HMM before a state transition is added, and is the same HMM as in FIG. 31A described above.

  FIG. 34B shows the HMM after the state transition is added for the HMM in FIG. 34A.

  The addition of the state transition is performed in order to solve the problem that the state transition is insufficient to appropriately represent the modeling target in the structure of the HMM stored in the model storage unit 115. In particular, when a sparse state transition is set as the initial structure of the HMM in the initial structure setting unit 116, it is important to add a state transition necessary for appropriate expression of the modeling target.

In Figure 34B, of the s ₆ to the absence s ₁ of the HMM of FIG. 34A, for example, the state s ₄ and s _6, as an additional subject of the state transition, the state s ₄ is an additional object of the state transition s ₆ A bidirectional state transition is added between the two.

In the addition of the state transition, the structure adjustment unit 117 sets an effective state transition between the states s ₄ and s _{6 to} which the state transition is added. Furthermore, the structure adjustment unit 117 is one of the states s ₄ and s ₆ to which state transition is added, for example, the state transition probability a from the state s ₄ to the other, for example, the state s _{6 46} and the state transition probability a ₆₄ from the other state s ₆ to the one state s ₄ are set to appropriate values, for example, a ₄₆ = a ₆₄ = 0.5.

  Then, the structure adjustment unit 117 performs normalization processing on the necessary parameters of the HMM after the addition of the state transition, and ends the state transition addition processing.

That is, the structure adjustment unit 117 performs normalization processing similar to that in the case of state division on the state transition probability a _ij of the HMM after the state transition is added.

  In addition, in the addition of a state transition, a set of two states to be added as a state transition (hereinafter also referred to as an addition target set) is not limited to one set.

That is, as a set of states to be added set, for example, from among a set of states where bidirectional state transition is not possible in the _N states s ₁ to s _N of the HMM before the addition of the state transition, the state It is possible to select the top n (n is a value of 1 or more) states having a large correlation between them.

  As described above, when a pair having a high correlation among the states where bidirectional state transition is not possible is selected as the addition target set, for example, there is no state transition (direct state transition is not possible). ) When one of the two states becomes redundant to the other, the two states are organically connected by the state transition.

  Note that the number n of sets in the state to be added can be set randomly or can be set to a fixed value. In any case, with the addition of the state transition, the structure of the HMM is updated to a slightly complicated structure with the number of state transitions increased by n, although the number of states does not change.

  Next, with reference to FIG. 35, deletion of a state performed by the structure adjustment unit 117 as adjustment of the structure of the HMM will be described.

  FIG. 35A shows the HMM before the state is deleted.

In FIG. 35A, the HMM has nine states s ₁ , s ₂ , s ₃ , s ₄ , s ₅ , s ₆ , s ₇ , s ₈ , s ₉ , and between states s ₁ and s _2. , Between states s ₁ and s ₄ , between states s ₂ and s ₃ , between states s ₂ and s ₅ , between states s ₃ and s ₆ , between states s ₄ and s ₅ , Between states s ₄ and s ₇ , between states s ₅ and s ₆ , between states s ₅ and s ₈ , between states s ₆ and s _9, and between states s ₇ and s _8. And bi-directional state transition between states s ₈ and s ₉ and self-transition are possible.

  FIG. 35B shows the HMM after the state is deleted for the HMM in FIG. 35A.

  The deletion of the state is performed in the HMM in order to delete a state unnecessary for appropriately expressing the modeling target.

In FIG. 35B, for example, the state s ₅ is deleted from the states s ₁ to s ₉ of the HMM in FIG. 35A.

Delete state, the state s ₅ of deletion is done by deleting the possible state transitions from the state s ₅ (including a state transition to state s _5).

In FIG. 35A, since the state s ₅ can undergo state transitions and self-transitions between the states s ₂ , s ₄ , s ₆ , and s ₈ , the structure adjustment unit 117 is subject to deletion. the state s _5, the state s _5, and deletes its state s _5, the state transitions between the respective states _{s 2, s 4, s 6} , s 5, and the self-transition of the state s ₅ .

  Furthermore, the structure adjustment unit 117 performs normalization processing on the necessary parameters of the HMM after the state deletion, and ends the state deletion processing.

That is, the structure adjustment unit 117 performs normalization processing similar to that in the case of state division on the initial probability π _i of the HMM after state deletion and the state transition probability a _ij .

  The structure adjustment unit 117 selects a state to be deleted as follows, for example.

  That is, as described with reference to FIG. 29, the adjusted time series data x ′ used for estimating the parameters of the HMM in the parameter estimating unit 113 is supplied from the data adjusting unit 112 to the structure adjusting unit 117.

The structure adjustment unit 117 applies the Viterbi method to the HMM stored in the model storage unit 115, and the adjusted time series data x ′ = x ₁ ′, x ₂ ′,... From the data adjustment unit 112. , x _{T ′} ′ finds the state transition process (state sequence) (path) (hereinafter also referred to as the maximum likelihood path) that maximizes the likelihood of being observed.

Here, the Viterbi method is the state transition probability a _ij that makes a state transition from state s _i to state s _j at time t in the state transition path starting from each state s _i , and the state transition , The probability that the sample value x ′ _t at time t of the adjusted time series data x ′ = x ₁ ′, x ₂ ′,..., X _{T ′} ′ is observed (output probability density function b _j (x ) Is an algorithm that determines a path (maximum likelihood path) that maximizes the value (occurrence probability) accumulated over the length T ′ of the adjusted time-series data x ′.

  The Viterbi method is described in Document A described above.

When the structure adjustment unit 117 determines the state sequence s ₁ ′, s ₂ ′,..., S _{T ′} ′ as the maximum likelihood path for the adjusted time series data x ′, A state that does not constitute the maximum likelihood path (a state that is not included in the maximum likelihood path) is detected.

The state that does not constitute the maximum likelihood path is not always necessary to express the characteristics (time-series pattern) of the adjusted time-series data x '= x ₁ ', x ₂ ', ..., x _T' ' Since it can be regarded as a state, the structure adjustment unit 117 selects a state that does not constitute the maximum likelihood path as a state to be deleted.

For example, in the HMM having the states s ₁ to s ₉ in FIG. 35A, the state series s ₁ , s ₂ , s ₃ , s ₆ , s _{9 for} the adjusted time series data x ′ having a length T ′ of 16 is shown. , s ₈ , s ₇ , s ₄ , s ₁ , s ₄ , s ₇ , s ₈ , s ₉ , s ₆ , s ₃ , s ₂ , s ₁ are determined as the maximum likelihood path, the structure The adjustment unit 117 selects the state s ₅ that does not constitute the maximum likelihood path among the states s ₁ to s ₉ that constitute the HMM as a state to be deleted.

Then, as described above, the structure adjustment unit 117 deletes the state s ₅ selected as the deletion target, thereby adjusting the structure in which the HMM shown in FIG. 35A becomes the HMM shown in FIG. 35B. Done.

  In addition, the structure adjustment unit 117 performs not only the state division, state merging, state addition, state addition, state transition addition, and state deletion described in FIGS. 31 to 35 but also the state transition deletion. Made as an adjustment.

  The state transition is deleted in the same manner as the state deletion.

That is, as described above, the structure adjustment unit 117 determines the state sequences s ₁ ′, s ₂ ′,..., S _{T ′} ′ as the maximum likelihood path for the adjusted time series data x ′, A state transition that does not constitute the maximum likelihood path is selected as a state transition to be deleted.

Furthermore, the structure adjustment unit 117 deletes the state transition selected as the state transition to be deleted, and normalizes the state transition probability a _ij of the HMM after the state transition is deleted as in the case of state division. To finish the state transition deletion process.

  Next, FIG. 36 is a flowchart for explaining processing (learning processing) of the data processing apparatus of FIG.

  When the sensor signal from the modeling target is supplied to the time series data input unit 111, the time series data input unit 111, for example, uses the sensor signal observed from the modeling target as it is as the observation time series data. Let x be.

Here, as described above, the observation time series data x is supplied from the time series data input unit 111 to the data adjustment unit 112 and also to the initial structure setting unit 116. In the initial structure setting unit 116, As described above, it is used for setting the output probability density function b _j (x).

  The initial structure setting unit 116 initializes the HMM in step S111.

  That is, the initial structure setting unit 116 initializes the structure of the HMM to the initial structure, and sets the parameters (initial parameters) of the HMM of the initial structure.

  Specifically, the initial structure setting unit 116 sets the number of states of the HMM as an initial structure of the HMM, and sets a sparse state transition in the HMM of the number of states.

Furthermore, the initial structure setting unit 116 sets the initial values of the state transition probability a _ij , the output probability density function b _j (x), and the initial probability π _i as initial parameters in the HMM having the initial structure.

As described above, in the initial structure setting unit 116, the initial structure and initial parameters λ = {a _ij , b _j (x), π _i , i = 1, 2,..., N, j = 1, 2, .., N} are set and supplied to the model storage unit 115 for storage.

  Thereafter, the process proceeds from step S111 to step S112, the time series data input unit 111 supplies the observation time series data x to the data adjustment unit 112, and the process proceeds to step S113.

  In step S113, the data adjustment unit 112 performs adjustment of the observation time series data x from the time series data input unit 111 as described with reference to FIG. 29, thereby obtaining adjusted time series data x ′. The data is supplied to the parameter estimation unit 113, and the process proceeds to step S114.

  The adjusted time series data x ′ is supplied from the data adjustment unit 112 to the parameter estimation unit 113 and also to the structure adjustment unit 117.

  In step S114, the parameter estimation unit 113 sets the HMM parameter stored in the model storage unit 115 as an initial value, and uses the adjusted time-series data x ′ from the data adjustment unit 112 to set a new parameter for the HMM, Estimated by Baum-Welch reestimation method.

  Further, the parameter estimation unit 113 supplies new parameters of the HMM to the model storage unit 115 and stores them in the form of overwriting.

  In addition, the parameter estimation unit 113 increments the learning count reset to 0 at the start of the learning process of FIG. 36 by one, and supplies the learning count to the evaluation unit 114.

  Further, the parameter estimation unit 113 obtains the likelihood that the adjusted time series data x ′ is observed from the HMM defined by the new parameter λ, and supplies the likelihood to the evaluation unit 114. The processing is performed from step S114 to step S115. Proceed to

  In step S115, the evaluation unit 114 evaluates the HMM for which learning has been performed, that is, the HMM for which the parameter λ has been estimated by the parameter estimation unit 113, based on the likelihood from the parameter estimation unit 113 and the number of learnings. It is determined whether or not the learning of the HMM is terminated based on the result of the evaluation.

  If it is determined in step S115 that the learning of the HMM is not terminated, the evaluation unit 114 requests the time series data input unit 111, the data adjustment unit 112, and the structure adjustment unit 117 to perform a predetermined process. Then, the process proceeds to step S116.

  In step S116, the structure adjustment unit 117 adjusts the structure of the HMM stored in the model storage unit 115 using the adjusted time series data x ′ from the data adjustment unit 112 in response to a request from the evaluation unit 114. The process is performed, and the process returns to step S112.

  In step S112, the time series data input unit 111 supplies the observation time series data x to the data adjustment unit 112 in response to a request from the evaluation unit 114, and the process proceeds to step S113.

  In step S113, the data adjustment unit 112 adjusts the observation time series data x from the time series data input unit 111 in response to a request from the evaluation unit 114 as described with reference to FIG. After obtaining the time series data x ′, the above-described processing is repeated.

  That is, it is repeated that the parameter estimation unit 113 estimates the parameters of the HMM and the structure adjustment unit 117 adjusts the structure of the HMM defined by the parameters after the estimation.

  In addition, the data adjustment unit 112 obtains adjusted time-series data x ′ by performing a down-sampling process on the observed time-series data x as described with reference to FIG. 29, for example. In the downsampling process, as the HMM learning progresses, the sampling frequency of the adjusted time-series data x ′ is gradually changed from a small value to a large value.

  On the other hand, if it is determined in step S115 that learning of the HMM is to be terminated, the learning process is terminated.

  As described above, the data processing apparatus of FIG. 29 initializes the structure of the HMM to a sparse structure, and then adjusts the observation time series data x used for learning according to the progress of learning, and after adjustment The time series data x ′ is output, the adjusted time series data x ′ is used, the HMM parameters are estimated, and the HMM structure is adjusted.

  As a result, it is possible to obtain an HMM that appropriately models the modeling target even if it is a complicated modeling target.

  That is, for modeling a complex modeling target, an HMM with a large number of states and state transitions is generally required, but a large-scale HMM with a large number of states and state transitions is required from the beginning. It is difficult to correctly estimate the parameters of the HMM.

  In the data processing device of FIG. 29, by initializing the structure of the HMM to a sparse structure, adjusting the observation time-series data x according to the progress of learning, and adjusting the structure of the HMM, Even if an HMM that appropriately represents a complicated modeling target is a large-scale HMM, parameters of such a large-scale HMM can be correctly estimated (parameters estimated to be correct).

  Furthermore, in the data processing device of FIG. 29, even if the modeling target is an unknown target and the initial structure of the HMM and the initial value (initial parameter) of the parameter cannot be determined (predicted) in advance, An HMM that appropriately represents the modeling target (an HMM having an appropriate structure and appropriate parameters) can be obtained.

  Next, FIG. 37 is a flowchart for explaining details of processing performed by the structure adjustment unit 117 in step S116 of FIG.

  In step S121, the structure adjustment unit 117 divides the HMM stored in the model storage unit 115 in the state described with reference to FIG. 31, and the process proceeds to step S122.

  In step S122, the structure adjustment unit 117 uses the adjusted time-series data x ′ supplied from the data adjustment unit 112 to obtain the correlation between the states constituting the HMM after the state division, and the processing is performed in step S123. Proceed to

  In step S123, the structure adjustment unit 117 performs the state merging described in FIG. 32 on the HMM after the state division based on the correlation obtained in step S122, and the process proceeds to step S124.

  In step S124, the structure adjustment unit 117 adds the state transition described in FIG. 34 to the HMM after the state merging based on the correlation obtained in step S122, and the process proceeds to step S125.

  In step S125, the structure adjustment unit 117 adds the state described in FIG. 33 to the HMM after the addition of the state transition, and the process proceeds to step S126.

  In step S126, the structure adjustment unit 117 obtains the maximum likelihood path for the adjusted time-series data x ′ from the data adjustment unit 112 using the HMM after the state is added, and the process proceeds to step S127.

  In step S127, the structure adjustment unit 117 detects a state that does not constitute the maximum likelihood path and a state transition. Further, in step S127, the structure adjustment unit 117 deletes the state that does not constitute the maximum likelihood path and the state transition as described with reference to FIG.

  Then, the structure adjustment unit 117 updates the stored value of the model storage unit 115 with the state and the parameter of the HMM after the state transition is deleted, and the process returns.

  As described above, the structure adjustment unit 117 refers to state division, state merge, state addition, state transition addition, state deletion, and state transition deletion for the HMM stored in the model storage unit 115. Adjust 6 types of structures.

  Here, in FIG. 36 and FIG. 37, the evaluation unit 114 requests the structure adjustment unit 117 to adjust the structure every time the number of learning increases by one.

  Therefore, the structure adjustment unit 117 adjusts the structure of the HMM every time the number of learnings is increased by one, but the adjustment of the structure of the HMM is performed according to the progress of learning other than the one increase in the number of learnings. Is possible.

  That is, the evaluation unit 114 supplies the number of learnings and the likelihood to the data adjustment unit 112 as the progress status information indicating the progress of the learning. The progress status information is also supplied to the structure adjustment unit 117. Can do.

  In this case, the structure adjustment unit 117 adjusts the structure of the HMM according to the progress status information from the evaluation unit 114.

  That is, for example, the structure adjustment unit 117 adjusts the structure when the number of times of learning as the progress status information becomes a value increased by a predetermined number of times from the number of times of adjustment of the previous structure. be able to.

  In addition, the structure adjustment unit 117, for example, when the likelihood as the progress status information has decreased from the value at the time of previous adjustment of the structure, or when the rate of increase in the likelihood is below a predetermined value, etc. The structure can be adjusted.

  Note that adjustment of the structure of the HMM by the structure adjustment unit 117 does not guarantee that the structure of the HMM converges to an optimal structure that represents the modeling target.

  However, according to the adjustment of the structure of the HMM by the structure adjustment unit 117, a state estimated to be appropriate for expressing the modeling target and a state transition are added, while the modeling target is added. By removing the states that are assumed to be unnecessary to express the state and state transitions, even for complex modeling targets, obtain a large-scale HMM that appropriately models the modeling target Can do.

  In FIG. 37, the structure is adjusted in the order of state division, state merging, state transition addition, state addition, state deletion, and state transition deletion. However, the structure adjustment is performed. The order is not limited to this.

  Next, with reference to FIGS. 38 to 40, the simulation performed for the data processing apparatus of FIG. 29 will be described.

  In the simulation, a sequence of coordinates (x, y) of the movement trajectory of the robot moving randomly in the two-dimensional space was used as the observation time series data x.

  In addition, the range of coordinates (x, y) in the two-dimensional space in which the robot can move is the following among the ranges represented by the expression -100 <x <+100 and the expression -100 <y <+100. A range excluding the areas of the four blocks # 1, # 2, # 3, and # 4 indicated by the range of the equation was used.

Block # 1: -70 <x <-20, -70 <y <-20
Block # 2: -70 <x <-20, +20 <y <+70
Block # 3: +20 <x <+70, -70 <y <-20
Block # 4: +20 <x <+70, +20 <y <+70

  The robot was moved by 10,000 steps (times) within a movable range while sequentially determining a small amount of movement (Δx, Δy) with the origin (0,0) as the start position.

  FIG. 38 shows the movement trajectory of the robot.

  That is, FIG. 38A shows a movement trajectory from the start position (origin) to movement by 200 steps, and FIG. 38B shows a movement trajectory from the start position to movement by 10000 steps. .

  In FIG. 38, the black circles represent the coordinates after moving by a minute movement amount (Δx, Δy). Moreover, in FIG. 38, the movement locus is shown by connecting the black circles with straight lines in time order.

  According to FIG. 38, it can be seen that the robot is moving randomly throughout the entire movable range.

  In the simulation, the sequence of coordinates (x, y) for 10,000 steps as described above was used as the observation time series data x, but the range in which the robot can move and the observation time series data x are in the two-dimensional space. The coordinates (x, y) are unknown.

  In other words, in the simulation, information about the robot where the coordinate (x, y) (movement trajectory) that is the observation time series data x is observed is not given in advance, and the two-dimensional observation time series data x is observed. Only that was known in advance.

In the simulation, an HMM having the 16 states shown in FIG. 30A is used as the HMM having the initial structure, and a normal distribution is used as the output probability density function b _j (x) of each state s _j of the HMM.

  In the simulation, the learning of the HMM was completed when the number of learning was 36.

  In the simulation, as described above, observation time series data that is a sequence of coordinates (x, y) for 10000 steps, that is, observation time series data consisting of 10000 samples is used for learning. In the first learning, the downsampling process is performed on the observation time-series data consisting of 10000 samples so that the sampling frequency is 1/10 of the original, and the result is 1000 samples. The adjusted time series data was used for estimation of HMM parameters.

  After that, every time the number of learning increases by 3 times, the adjusted time series data is sampled so that the sampling frequency becomes the original 1/9, 1/8, 1/7, ..., 1/1. The frequency was gradually increased. In this case, the adjusted time-series data becomes the observed time-series data itself when the number of learning times is 28th or more.

  FIG. 39 shows an HMM obtained as a result of learning.

  That is, FIG. 39A shows an HMM at a point in time immediately after learning is started (early learning stage), and FIG. Further, FIG. 39C shows the HMM after learning a sufficient number of times of learning (after completion of learning).

In FIG. 39, black circles represent the coordinates (x, y) indicated by the average vector of the output probability density function b _j () of the state s _j of the HMM, and correspond to the state s _j .

In FIG. 39, when the state transition probability a _ij of the state transition from the state s _i to the state s _j is greater than 0 (when the state transition from the state s _i to the state s _j is an effective state transition). ), A black circle corresponding to the state s _i and a black circle corresponding to the state s _j are connected by a straight line (line segment). Therefore, in FIG. 39, the straight line connecting the black circles corresponds to a (valid) state transition.

  In FIG. 39, an arrow indicating the direction of state transition is not shown.

  In FIG. 39, the states are uniformly arranged in a movable range, and moreover, between states corresponding to two positions (coordinates) that can be moved in a single (constant) manner of movement. There is a state transition. Therefore, it can be seen that an HMM that adequately expresses the characteristics (features) of the movement method for moving within the movable range of the two-dimensional space can be obtained.

  FIG. 40 shows the log likelihood (logarithm value of likelihood) obtained for the adjusted time-series data from the HMM obtained as a result of learning.

  As can be seen from FIG. 40, as the number of times of learning increases, the log likelihood obtained from the HMM tends to improve.

  That is, according to FIG. 40, it can be seen that as learning progresses, an HMM that appropriately represents the characteristics of the movement trajectory is obtained.

  As described above, in the data processing device of FIG. 29, learning is started from the coarse HMM configured by the sparse state transition given by the initial structure setting unit 116, and the structure adjustment unit 117 gradually increases as the learning progresses. HMM is refined. In parallel with this, learning is started from the macro features of the observation time-series data by the data adjustment unit 112, and adjustment is performed so as to gradually include micro features as the learning progresses.

  The functions of setting the sparse initial structure of the HMM by the initial structure setting unit 116, adjusting the structure of the HMM by the structure adjusting unit 117, and adjusting the observation time series data by the data adjusting unit 112 are as described above. Therefore, it is possible to determine the structure and parameters of a large-scale HMM that was difficult to handle in the past.

  Note that the data processing device in FIG. 29 is a system (a system is a device or a logical collection of a plurality of devices, and whether or not each component device is in the same housing). It can be applied to learning of state transition models used for identification, control, artificial intelligence, etc. In particular, the present invention can be applied to learning or the like for an autonomous agent such as an autonomous robot to recognize (recognize) the environment and its state and to perform a cognitive action that takes an action corresponding to the recognition result. 29 is intended for social systems such as traffic, finance and information, physical systems and chemical systems for physical phenomena and chemical reactions, and biological systems related to living things. Can be applied to network learning.

  In the above case, the initial structure setting unit 116 initializes the structure of the HMM to a sparse structure. However, the initial structure setting unit 116, for example, converts the HMM structure to an ergodic structure. It is possible to initialize to a new structure.

  Further, in the above-described case, the data adjustment unit 112 adjusts the observation time series data according to the progress of learning. However, the adjustment of the observation time series data can be prevented from being performed. In this case, it is not necessary to provide the data adjustment unit 112 in the data processing apparatus of FIG.

  In addition, it is possible to determine whether or not the structure adjustment by the structure adjustment unit 117 is performed or not according to the likelihood itself or the likelihood change ratio.

  Next, the series of processes described above can be performed by hardware or software. When a series of processing is performed by software, a program constituting the software is installed in a general-purpose computer or the like.

  Therefore, FIG. 41 shows a configuration example of an embodiment of a computer in which a program for executing the series of processes described above is installed.

  The program can be recorded in advance on a hard disk 155 or a ROM 153 as a recording medium built in the computer.

  Alternatively, the program is stored temporarily on a removable recording medium 161 such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, or a semiconductor memory. It can be stored permanently (recorded). Such a removable recording medium 161 can be provided as so-called package software.

  The program is installed on the computer from the removable recording medium 161 as described above, or transferred from the download site to the computer wirelessly via a digital satellite broadcasting artificial satellite, or a LAN (Local Area Network), The program can be transferred to a computer via a network such as the Internet, and the computer can receive the program transferred in this way by the communication unit 158 and install it in the built-in hard disk 155.

  The computer includes a CPU (Central Processing Unit) 152. An input / output interface 160 is connected to the CPU 152 via the bus 151, and the CPU 152 operates the input unit 157 including a keyboard, a mouse, a microphone, and the like by the user via the input / output interface 160. When a command is input by being equalized, a program stored in a ROM (Read Only Memory) 153 is executed accordingly. Alternatively, the CPU 152 also transfers from a program stored in the hard disk 155, a program transferred from a satellite or a network, received by the communication unit 158 and installed in the hard disk 155, or a removable recording medium 161 attached to the drive 159. The program read and installed in the hard disk 155 is loaded into a RAM (Random Access Memory) 154 and executed. Thereby, the CPU 152 performs processing according to the above-described flowchart or processing performed by the configuration of the above-described block diagram. Then, the CPU 152 outputs the processing result from the output unit 156 configured with an LCD (Liquid Crystal Display), a speaker, or the like, for example, via the input / output interface 160, or from the communication unit 158 as necessary. Transmission, and further recording on the hard disk 155 is performed.

  FIG. 42 is a diagram illustrating a functional configuration example of the information processing apparatus.

  The information processing apparatus shown in FIG. 42 includes a configuration relating to causal perception and a configuration for determining the behavior of the robot (agent) based on the causal relationship. The configuration related to the causal perception corresponds to the configuration of the causal unit 63 in FIG. 15, and the configuration for determining the behavior of the robot based on the causal relationship corresponds to the configuration of the behavior control unit 64 in FIG.

  As shown in FIG. 42, the information processing apparatus includes a causal learning processing unit 201, a causal estimation processing unit 202, a causal candidate list storage unit 203, a causal candidate list organization processing unit 204, and an action determination unit 205.

  The causal learning processing unit 201 acquires a plurality of modal HMMs such as the distance HMM, optical HMM, and energy HMM generated as described above, and performs causal learning. The causal learning processing unit 201 outputs data obtained by performing the causal learning to the causal estimation processing unit 202.

  Since only one HMM node (state) is always fired in the same HMM, the event is a set of mutually exclusive and collectively (MECE) events. Therefore, it is possible to cause a node transition in one HMM to be caused by node firing of one or more other HMMs. For example, the time change of each ignition node of N types of HMMs is recorded, and this is used for causal learning.

  The causal estimation processing unit 202 performs causal estimation using the data supplied from the causal learning processing unit 201. The causal estimation processing unit 202 causes the causal candidate list storage unit 203 to store a list representing the causal relationship obtained by performing the causal estimation.

  The causal relationship between events is expressed by a conditional probability as will be described later, and acquiring data used to obtain the conditional probability is called causal learning. In addition, obtaining a conditional probability using data acquired by causal learning and estimating a causal relationship is called causal estimation. Causal perception represents a state in which a causal relationship between events is perceived by causal estimation.

  The causal candidate list organization processing unit 204 appropriately organizes the causal candidate list stored in the causal candidate list storage unit 203.

  When the target value is given, the behavior determining unit 205 determines the behavior with reference to the causal candidate list stored in the causal candidate list storage unit 203. The behavior of the robot is controlled based on a command representing the behavior determined by the behavior determination unit 205.

  The causal estimation processing unit 202 basically performs causal estimation as follows. Each content will be described in detail later.

  That is, in the causal estimation of an event a1, all possible events are within the range of the robot's experience, and they are mutually exclusive and exhaustive events a1, a2, a3, etc. including the event a1. And a set B that is a set of other events.

  The causal relationship is represented by a conditional probability P (T: ak → a1 | ak, b) obtained for all the events b that have occurred simultaneously with the event ak. T: ak → a1 represents a transition from event ak to event a1. Hereinafter, T: ak → a1 is simply expressed as T as appropriate.

Since the conditional probability P (T | ak, b) is expressed by the following equation, the conditional probability is obtained from the values of N (T, ak, b) and N (ak, b).
P (T | ak, b) = P (T, ak, b) / P (ak, b) ≈N (T, ak, b) / N (ak, b)

  N (T, ak, b) represents the number of times event ak and event b occur simultaneously and event a1 occurs at the next time. N (ak, b) represents the number of times that event ak and event b occurred simultaneously.

  There is an error in the conditional probability obtained in this way, and its magnitude is expected to be inversely proportional to √N (T, ak, b). Therefore, if the event b is controlled by changing the granularity so as to keep N (T, ak, b) within a suitable range, the error can be reduced.

  Specifically, when the number of experienced robots is small, a rough expression such as “when bright” is used as a method of expressing the event b. Also, as the number of experiences increases, it is included in “when bright”, “when illuminance is between 700 and 800 lux”, “when bright light comes from behind”, and “when illuminated with a warning sound” By using a finer granularity expression such as "", it is possible to perform finer control and causal estimation while keeping the error according to the number of experiences small.

  When the causal relationship varies with time, the conditional probability follows the temporal variation of the causal relationship by decaying the number of concurrent occurrences of the event N (T, ak, b) and N (ak, b). Can be made.

  For example, in the past 1000 attempts (N (ak, b) = 1000) have been successful 500 times (N (T, ak, b) = 500) and P (T | ak, b) = 500/1000 Suppose = 0.5. Also, suppose we have tried 10 times and succeeded 8 times. If there is no time decay, including 10 trials now, P (T | ak, b) = 508/1010 = 0.503, so it is assumed that it succeeded with a high probability of 8 out of 10 compared to previous trials. Has little effect on the value of P (T | ak, b).

  On the other hand, if there is time decay, the numerator and denominator are multiplied by the decay rate, and the evaluation is received. Therefore, if the decay rate is 0.1, P (T | ak, b) obtained by past trials Becomes P (T | ak, b) = 50/100 = 0.5. When the latest trial result is reflected in this state, P (T | ak, b) = 58/110 = 0.527. For example, if the past trial is older and further attenuated, for example, P (T | ak, b) = 5/10 = 0.5, and if the latest trial result is reflected in this state, P (T | ak, b) = 13/20 = 0.65.

  In other words, P (T | ak, b) can be made to follow the recent trial result (the value of P (T | ak, b) is greatly influenced by the recent trial result. Can be). By attenuating the number of simultaneous occurrences N (T, ak, b) and N (ak, b) at the same attenuation rate, the probability estimate based on past experience can be used as it is even if there is no recent trial result. is there. In addition, the value of N (T, ak, b), which affects the calculation of the estimation error, decreases with time, so the estimation error gradually increases, and past experience is less than that of recent experience. The property of being definite can be expressed automatically.

  By the way, the conditional probability P (T | ak, b) is calculated from the viewpoint of model fitting under the observation T: ak → a1 (transition T from event ak to event a1), “(ak, b) Is the expression that gives the likelihood of the model that T: ak → a1 occurs.

  On the other hand, this can be regarded as the probability that transition T: ak → a1 will occur when (ak, b) occur simultaneously with this posterior probability, and thus the conditional probability for the control to cause transition T P (T | ak, b) can be used. Specifically, among all the events b, if the event b with the maximum P (T | ak, b) can be achieved together with the event ak, the transition T is most likely to occur. In order to make it happen, it is only necessary to search for such an event b and determine an action.

  A disadvantage of this behavior determination method is that it is susceptible to estimation errors due to variability in experience. For example, consider whether an event that succeeded 5 times out of 10 times or an event that succeeded 501 times out of 1000 times should be executed. The former event may only appear to have a success rate of 0.5 because the success rate was actually 0.55, but it happened to succeed only 5 times in this 10 trials. If you try one more time, it may be 6/11 = 0.545 or 5/11 = 0.455. Under such circumstances, simply selecting the event b that maximizes P (T | ak, b) does not work, and this point has a drawback. The reason is that the option that happened to be unlucky only by chance was overwhelmingly unfavorable and never had the chance to recover again.

  Therefore, in order to eliminate these disadvantages, the expected value of the error is calculated using the number of trials so far and the conditional probability at the present time, and only the expected value is considered optimistically. The conditional probability is increased by the value and used for action determination. In the above example, this means that the success rate obtained when succeeding in the first event in the former event is 6/11 = 0.545, and it is obtained when succeeding in the latter event in the next one. Since the success rate is higher than 502/1001 = 0.501, it is close to the idea of selecting the former. As a result, the number of trials of the former increases and the estimation error also decreases, leading to an improvement in the accuracy of action determination.

  With reference to the flowchart of FIG. 43, the process regarding the causal perception performed by the information processing apparatus of FIG. 42 will be described.

  In step S201, the causal learning processing unit 201 acquires a plurality of modal HMMs and performs causal learning. The causal learning processing unit 201 outputs data obtained by performing the causal learning to the causal estimation processing unit 202.

  In step S <b> 202, the causal estimation processing unit 202 performs causal estimation using the data supplied from the causal learning processing unit 201. The causal estimation processing unit 202 causes the causal candidate list storage unit 203 to store a causal candidate list representing a causal relationship obtained by performing causal estimation.

  In step S203, the causal candidate list organization processing unit 204 arranges the causal candidate list stored in the causal candidate list storage unit 203, and ends the processing.

  Hereinafter, each process is demonstrated in order.

In the following, it is assumed that there are a total of M modals as target modals, and the i (i = 1, 2,..., M) -th modal has n _m states. In addition, the i-th modal state j (j = 1, 2,..., N _m ) is denoted as S ⁱ _j as appropriate. For example, S ² ₅ indicates that the second modal is state 5. The state of the entire system at time t is represented by an M-dimensional state vector S _t = (S ¹ _j1 , S ² _j2 ,..., S ^M _jM ).

In the state vector, when it is clear which modal state number indicates each dimensional element, the state of the system is represented by a state vector having the state number as an element for the sake of simplicity. For example, when the states of modals 1, 2, and 3 are ₅ , ₇ , and ₁₁ , respectively, the state vector of the entire system including modals 1, 2, and 3 is S = (S ¹ ₅ , S ² ₇ , S ³ ₁₁ ) = (5,7,11) For example, when attention is paid to modals 2 and 3, the state vector is represented by S ^(2,3) = (S ² ₇ , S ³ ₁₁ ) = (7, 11).

  FIG. 44 is a diagram showing an example of a modal.

In the example of FIG. 44, three modals of modals 1 to 3 are shown. The value of M is 3. For example, modal 1 corresponds to an energy HMM, modal 2 corresponds to an optical HMM, and modal 3 corresponds to a distance HMM. Each S ⁱ _j corresponds to an HMM node.

  First, causal learning performed by the causal learning processing unit 201 will be described.

  At time t = 0, 0 is set as the value of all counters and is initialized. In causal learning, two counters, an event occurrence counter and a transition occurrence counter, are used. In the following, t ≧ 1.

  Further, every time a predetermined time elapses, the values of all the counters are attenuated according to a predetermined attenuation rate γ such as γ = 0.999.

Of the entire system at time t the state S _t and one time before the time of the state S _t-1 are compared, the change of state is a modal are listed.

  The following processing is performed paying attention to each L value of L = 1, 2,..., Min (M−1, MaxCombi). MaxCombi is a parameter that defines the complexity of modal combinations to be considered, and an arbitrary natural number can be set. min (M-1, MaxCombi) represents the smaller value of M-1 and MaxCombi.

An arbitrary _one of _M C _{L + 1} modal combinations when L + 1 modals are selected from _M modals is represented by cM (L + 1;). In addition, a state vector representing each modal state of any one combination at time t−1 is represented by ^{ScM (L + 1;)} _t−1 .

For each combination of cM (L + 1;), the event occurrence counter corresponding to S ^{cM (L + 1;)} _t−1 is incremented by one. The event occurrence counter is a counter that counts the number of times the event represented by the corresponding state vector has occurred.

  If the modal whose state has changed is assumed to be modal i, the following processing is performed while paying attention to each modal i.

CM (L; i) represents any _one of _M-1 C _L modal combinations when L modals are selected from M-1 modals other than modal i. . Further, a state vector representing each modal state of any one combination at time t−1 is represented by S ^{cM (L; i)} _t−1 .

For each combination of cM (L; i), S ^{cM (L; i)} _t-1 and modal i state transition T ⁱ _t-1 = (S ⁱ _{k (t-1)} → S ⁱ _{k (t ))} ), The transition occurrence counter corresponding to (S ^{cM (L; i)} _t-1 | T ⁱ ) is counted up by one. The transition occurrence counter is a counter that counts the number of times that an event represented by a corresponding state vector has occurred at a timing immediately before the occurrence of a state transition for which a causal relationship is to be obtained.

  A specific example of causal learning will be described.

Here, as shown in FIG. 45, the number of states of modal 1 is 2 of states 1 and 2 (S ¹ _1, S ¹ ₂ ), and the number of states of modal 2 is states 1, 2, 3, 4 ( S ² _1, S ² _2, S ² _3, S ² ₄ ), and the number of states of modal 3 is 3 in states _{1, 2,} 3 (S ³ _1, S ³ _2, S ³ ₃ ) Shall.

  Further, it is assumed that the state of the system changes with time as shown in FIG.

  FIG. 46 shows state vectors representing the states of modals 1 to 3 observed between t = 0 and t = 5. 1 on the leftmost state vector representing the state at t = 0 indicates that the state of modal 1 is state 1, and 1 at the center indicates that the state of modal 2 is state 1. The lower 1 indicates that the state of the modal 3 is the state 1.

  47A to 47D are diagrams illustrating examples of event occurrence counters.

If the value of L is 1 or 2, the combination of _M C _{L + 1} modals when L + 1 modals are selected from the three modals, as shown on the left side, {1, 2, }, {1,3}, {2,3}, {1,2,3}. A modal combination of {1, 2}, {1, 3}, {2, 3}, {1, 2, 3} corresponds to the above-described cM (L + 1;).

  When attention is paid to the modal combination of {1, 2}, the number of state vectors that can be taken is represented by “−” for elements of modal 3 not included in the combination of interest, as shown in FIG. 1 1 −], [1 2 −], [1 3 −], [1 4 −], [2 1 −], [2 2 −], [2 3 −], and [2 4 −] are 8 . Here, the state vectors are shown side by side.

  Similarly, since the number of possible state vectors is a number obtained by multiplying the number of modal states included in the combination of interest, the number of possible state vectors when focusing on the modal combination of {1, 3} is It becomes 6 as shown in FIG. 47B. Further, when focusing on the modal combination of {2, 3}, the number of possible state vectors is 12, as shown in FIG. 47C, and when focusing on the modal combination of {1, 2, 3}, The number of possible state vectors is 24 as shown in FIG. 47D.

  Since event occurrence counters are prepared corresponding to the respective state vectors, in this case, a total of 50 event occurrence counters are prepared.

  48A to 48C are diagrams showing examples of transition occurrence counters prepared corresponding to respective state transitions of the modal 1.

  The transition occurrence counters shown in FIGS. 48A to 48C are prepared corresponding to bidirectional state transitions between the states 1 and 2 of the modal 1, for example, as shown in FIG.

When the value of L is 1 or 2, any one of the _M-1 C _L modal combinations when L modals are selected from modals other than modal 1 is shown in FIG. As shown on the left side of C, {2}, {3}, and {2, 3}. Each modal combination corresponds to the above-described cM (L; i).

  When focusing on the modal combination of {2}, the possible states are expressed as “*” for the element of modal 1 that is excluded from the combination of modal, and “-” for the element of modal 3 that is not included in the combination of interest. As shown in FIG. 48A, the number of vectors is 4 of [* 1 −], [* 2 −], [* 3 −], and [* 4 −].

  Similarly, when attention is paid to the modal combination of {3}, the number of possible state vectors is 3, as shown in FIG. 48B. When attention is paid to the modal combination of {2, 3}, the number of possible state vectors is 12 as shown in FIG. 48C.

  In this example, a total of 19 transition occurrence counters are prepared corresponding to the respective state transitions of the modal 1.

  50A to 50C are diagrams showing examples of transition occurrence counters prepared corresponding to the respective state transitions of the modal 2.

  As shown in FIG. 51, the transition occurrence counters shown in FIGS. 50A to 50C are connected between states 1 and 2 of modal 2, between states 2 and 3, between states 3 and 4, between states 4 and 1, 3 is prepared corresponding to a bidirectional state transition between states 2 and 4.

When the value of L is 1 or 2, any one of the _M-1 C _L modal combinations when L modals are selected from modals other than modal 2 is shown in FIG. As shown on the left side of C, {1}, {3}, and {1, 3}.

  When focusing on the modal combination of {1}, the modal 2 element that is excluded from the modal combination is represented by “*”, and the modal 3 element that is not included in the focused combination is represented by “-”. As shown in FIG. 50A, the number of vectors is 2 of [1 * −] and [2 * −].

  Similarly, when attention is paid to the modal combination of {3}, the number of possible state vectors is 3, as shown in FIG. 50B. When attention is paid to the modal combination of {1, 3}, the number of possible state vectors is 6, as shown in FIG. 50C.

  In this example, a total of 11 transition occurrence counters are prepared corresponding to the respective state transitions of the modal 2.

  52A to 52C are diagrams showing examples of transition occurrence counters prepared corresponding to respective state transitions of the modal 3.

  The transition occurrence counters shown in FIGS. 52A to 52C correspond to bidirectional state transitions between the states 1 and 2 of the modal 3, between the states 2 and 3, and between the states 3 and 1, as shown in FIG. Prepared.

If the value of L is 1 or 2, any one of the _M-1 C _L modal combinations when L modals are selected from modals other than modal 3 is shown in FIGS. As shown on the left side of C, {1}, {2}, {1, 2}.

  When focusing on the modal combination of {1}, the modal 3 elements that are excluded from the modal combination are represented by “*”, and the modal 2 elements that are not included in the focused combination are represented by “-”. The number of vectors is 2, as shown in FIG. 52A, [1− *] and [2− *].

  Similarly, when attention is paid to a modal combination of {2}, the number of possible state vectors is 4, as shown in FIG. 52B. When attention is paid to the modal combination of {1, 2}, the number of possible state vectors is 8, as shown in FIG. 52C.

  In this example, a total of 14 transition occurrence counters are prepared corresponding to the respective state transitions of the modal 3.

  In this way, the transition occurrence counter is prepared in a form that associates each state transition of a modal with all combinations of other modal states.

  When such event occurrence counter and transition occurrence counter are prepared, t = 1, and when the system state transitions from [1 1 1] to [1 2 1] as shown in FIG. The state at t = 1 is compared with the state at t = 0, which is the immediately preceding time, and modal 2, which is a modal having a state change, is listed.

  In addition, the event occurrence counter is counted up.

Here, any one of _M C _{L + 1} modal combinations {1, 2}, {1, 3}, {2, 3}, {1, 2, 3} The event occurrence counter corresponding to the state vector representing the state one hour before the modal included in the noted combination is counted up. A state vector representing a state one hour before the modal included in the combination of interest corresponds to S ^{cM (L + 1;)} _t−1 described above.

  When attention is paid to the modal combination of {1, 2}, since the states of modals 1 and 2 at t = 0 one time before are 1 respectively, [1 of 8 state vectors shown in FIG. The event occurrence counter corresponding to 1-] is incremented by one.

  When attention is paid to the modal combination of {1, 3}, since the states of modals 1 and 3 at t = 0 one time before are 1 respectively, [1 of 6 state vectors shown in FIG. 47B] -The event occurrence counter corresponding to 1] is incremented by one.

  When attention is paid to the modal combination of {2, 3}, since the states of modals 2 and 3 at t = 0 one time before are 1 respectively, [− of the 12 state vectors shown in FIG. The event occurrence counter corresponding to 1 1] is incremented by 1.

  When attention is paid to the modal combination of {1, 2, 3}, since the states of modals 1, 2, and 3 at t = 0 one time before are 1 respectively, the 24 state vectors shown in FIG. The event occurrence counter corresponding to [1 1 1] is counted up by one.

  Thus, when the state of the system transitions from [1 1 1] to [1 2 1], as shown in FIG. 54A, [1 1-], [1-1], [-1 1], The event occurrence counter corresponding to [1 1 1] is incremented by one.

  Further, the transition occurrence counter is counted up.

First, {1}, {3} which is an arbitrary _one of _M-1 C _L modal combinations when L modals are selected from modals other than modal 2 in which the state has changed. , {1, 3} are noticed, and a set of a state vector representing a state one hour before the modal included in the noted combination and a state transition (1 → 2) of modal 2 is obtained.

The obtained set represents a state vector that is associated with the state transition (1 → 2) of the modal 2 and represents the state one hour before the modal included in the combination of interest. As described with reference to FIG. 50, each state transition of the modal 2 is associated with a state vector. The state vector representing the state one hour before the modal included in the combination of interest corresponds to the above-described S ^{cM (L; i)} _t−1 , and the set is (S ^{cM (L; i)} _t−1. | T ⁱ ).

  Further, the transition occurrence counter (FIGS. 50A to 50C) corresponding to the state vector representing the state one hour before the modal included in the combination of interest associated with the state transition (1 → 2) of modal 2 is 1. Will only be counted up.

  When attention is paid to the modal combination of {1}, since the state of modal 1 at t = 0 one time before is 1, it is associated with the state transition (1 → 2) of modal 2 shown in FIG. 50A. Of the two transition occurrence counters, the transition occurrence counter corresponding to [1 *-] is incremented by one.

  When attention is paid to the modal combination of {3}, since the state of modal 3 at t = 0 one time before is 1, it is associated with the state transition (1 → 2) of modal 2 shown in FIG. 50B. Among the three transition occurrence counters, the transition occurrence counter corresponding to [− * 1] is counted up by one.

  When attention is paid to the modal combination of {1, 3}, since the states of modals 1 and 3 at t = 0 one time before are 1 respectively, state transition of modal 2 (1 → 2) shown in FIG. 50C Among the six transition occurrence counters associated with), the transition occurrence counter corresponding to [1 * 1] is counted up by one.

  Thus, when the state of the system transitions from [1 1 1] to [1 2 1], as shown in FIG. 54B, it is associated with the state transition (1 → 2) of modal 2, The transition occurrence counter corresponding to 1 *-], [-* 1], and [1 * 1] is incremented by one.

  Similarly, when t = 2, the state at t = 2 is compared with the state at t = 1, which is the previous time, and it is determined that there is no modal change in the state. As shown in FIG. 46, the state of the system at t = 2 is [1 2 1] which is the same as the state of the system at t = 1.

  In addition, the event occurrence counter is counted up.

{1,2}, {1,3}, which is an arbitrary combination of _M C _{L + 1} modal combinations when L + 1 modals are selected from three modals Each of {2, 3} and {1, 2, 3} is noticed, and the event occurrence counter corresponding to the state vector representing the state one hour before the modal included in the noted combination is counted up by one.

  When attention is paid to the modal combination of {1, 2}, since the state of modal 1 is 1 and the state of modal 2 is 2 at t = 1 one time before, the eight state vectors shown in FIG. The event occurrence counter corresponding to [1 2 −] is counted up by one.

  When attention is paid to the modal combination of {1, 3}, since the states of modals 1 and 3 at t = 1 one time before are 1 respectively, [1 of 6 state vectors shown in FIG. -The event occurrence counter corresponding to 1] is incremented by one.

  When attention is paid to the modal combination of {2, 3}, the state of modal 2 is 2 and the state of modal 3 is 1 at t = 1 one time before, so the 12 state vectors shown in FIG. The event occurrence counter corresponding to [−2 1] is counted up by one.

  When attention is paid to the modal combination of {1, 2, 3}, the state of modal 1 is 1, the state of modal 2 is 2, and the state of modal 3 is 1 at t = 1 one time ago. The event occurrence counter corresponding to [1 2 1] out of the 24 state vectors shown in FIG.

  Thus, when the system state remains [1 2 1], as shown in FIG. 55, [1 2-], [1-1], [-2 1], [1 2 1] The event occurrence counter is incremented by one.

  When t = 3 and the system state transitions from [1 2 1] to [2 2 1] as shown in FIG. 46, the state at t = 3 and the state at t = 2 which is the previous time Are compared, and modal 1, which is a modal having a state change, is listed.

  In addition, the event occurrence counter is counted up.

{1,2}, {1,3}, which is an arbitrary combination of _M C _{L + 1} modal combinations when L + 1 modals are selected from three modals Each of {2, 3} and {1, 2, 3} is noticed, and the event occurrence counter corresponding to the state vector representing the state one hour before the modal included in the noted combination is counted up by one.

  When attention is paid to the modal combination of {1, 2}, the state of modal 1 is 1 and the state of modal 2 is 2 at t = 2 one time before, so the eight state vectors shown in FIG. The event occurrence counter corresponding to [1 2 −] is counted up by one.

  When attention is paid to the modal combination of {1, 3}, since the states of modals 1 and 3 at t = 2 one time before are 1 respectively, [1 of 6 state vectors shown in FIG. -The event occurrence counter corresponding to 1] is incremented by one.

  When attention is paid to the modal combination of {2, 3}, the state of modal 2 is 2 and the state of modal 3 is 1 at t = 2 one time before, so the 12 state vectors shown in FIG. The event occurrence counter corresponding to [−2 1] is counted up by one.

  When attention is paid to the modal combination of {1, 2, 3}, the state of modal 1 is 1, the state of modal 2 is 2, and the state of modal 3 is 1 at t = 2 one time before, so FIG. 47D The event occurrence counter corresponding to [1 2 1] out of the 24 state vectors shown in FIG.

  Thus, when the system state transitions from [1 2 1] to [2 2 1], as shown in FIG. 56A, [1 2-], [1-1], [-2 1], The event occurrence counter of [1 2 1] is incremented by 1.

  Further, the transition occurrence counter is counted up.

First, {2}, {3}, which is an arbitrary _one of _M-1 C _L modal combinations when L modals are selected from modals other than modal 1 whose state has changed. , {2, 3} are noticed, and a set of a state vector representing a state one hour before the modal included in the noted combination and a state transition (1 → 2) of modal 1 is obtained.

  Further, the transition occurrence counter (FIGS. 48A to 48C) corresponding to the state vector representing the state one hour before the modal included in the combination of interest associated with the state transition (1 → 2) of modal 1 is 1. Will only be counted up.

  When attention is paid to the modal combination of {2}, since the state of modal 2 at t = 2 one time before is 2, it is associated with the state transition (1 → 2) of modal 1 shown in FIG. 48A. Of the four transition occurrence counters, the transition occurrence counter corresponding to [* 2-] is incremented by one.

  When attention is paid to the modal combination of {3}, since the state of modal 3 at t = 2 one time before is 1, it is associated with the state transition (1 → 2) of modal 1 shown in FIG. 48B. Among the three transition occurrence counters, the transition occurrence counter corresponding to [* -1] is counted up by one.

  When attention is paid to the modal combination of {2, 3}, the state of modal 2 is 2 and the state of modal 3 is 1 at t = 2 one time before, so the state transition of modal 1 shown in FIG. 48C Of the 12 transition occurrence counters associated with (1 → 2), the transition occurrence counter corresponding to [* 2 1] is incremented by one.

  Thus, when the state of the system transitions from [1 2 1] to [2 2 1], as shown in FIG. 56B, it is associated with the state transition of modal 1 (1 → 2), The transition occurrence counter corresponding to [* 2−], [* −1], and [* 2 1] is incremented by one.

  When t = 4 and the system state transitions from [2 2 1] to [2 4 3] as shown in FIG. 46, the state at t = 4 and the state at t = 3, which is the previous time Are compared, and modal 2 and modal 3, which are modals whose states have changed, are listed.

  In addition, the event occurrence counter is counted up.

{1,2}, {1,3}, which is an arbitrary combination of _M C _{L + 1} modal combinations when L + 1 modals are selected from three modals Each of {2, 3} and {1, 2, 3} is noticed, and the event occurrence counter corresponding to the state vector representing the state one hour before the modal included in the noted combination is counted up by one.

  When attention is paid to the modal combination of {1, 2}, since the states of modals 1 and 2 at t = 3 one time before are 2 respectively, [2 of 8 state vectors shown in FIG. 47A] The event occurrence counter corresponding to 2−] is incremented by one.

  When attention is paid to the modal combination of {1, 3}, the state of modal 1 is 2 and the state of modal 3 is 1 at t = 3 one hour before, so the six state vectors shown in FIG. The event occurrence counter corresponding to [2-1] is counted up by one.

  When attention is paid to the modal combination of {2, 3}, the state of modal 2 is 2 and the state of modal 3 is 1 at t = 3 one hour before, so the 12 state vectors shown in FIG. The event occurrence counter corresponding to [−2 1] is counted up by one.

  When attention is paid to the modal combination of {1, 2, 3}, the state of modal 1 is 2, the state of modal 2 is 2, and the state of modal 3 is 1 at t = 3 one time before, so FIG. 47D The event occurrence counter corresponding to [2 2 1] out of the 24 state vectors shown in FIG.

  Thus, when the state of the system transitions from [2 2 1] to [2 4 3], as shown in FIG. 57A, [2 2 −], [2-1], [−2 1], The event occurrence counter corresponding to [2 2 1] is incremented by one.

  Further, the transition occurrence counter is counted up. When two modals are listed, the same process is repeated for each modal.

First, {1}, {3} which is an arbitrary _one of _M-1 C _L modal combinations when L modals are selected from modals other than modal 2 in which the state has changed. , {1, 3} are noticed, and a set of a state vector representing a state one hour before the modal included in the noted combination and a state transition (2 → 4) of modal 2 is obtained.

  Also, the transition occurrence counter (FIGS. 50A to 50C) corresponding to the state vector representing the state one hour before the modal included in the combination of interest associated with the state transition (2 → 4) of modal 2 is 1. Will only be counted up.

  When attention is paid to the modal combination of {1}, since the state of modal 1 at t = 3 one time before is 2, it is associated with the state transition (2 → 4) of modal 2 shown in FIG. 50A. Of the two transition occurrence counters, the transition occurrence counter corresponding to [2 *-] is incremented by one.

  When attention is paid to the modal combination of {3}, since the state of modal 3 at t = 3 one time before is 1, it is associated with the state transition (2 → 4) of modal 2 shown in FIG. 50B. Among the three transition occurrence counters, the transition occurrence counter corresponding to [− * 1] is counted up by one.

  When attention is paid to the modal combination of {1, 3}, since the state of modal 1 is 2 and the state of modal 3 is 1 at t = 3 one hour before, the state transition of modal 2 shown in FIG. 50C Of the six transition occurrence counters associated with (2 → 4), the transition occurrence counter corresponding to [2 * 1] is incremented by one.

Next, {1}, {2 which is an arbitrary _one of _M-1 C _L modal combinations when L modals are selected from modals other than modal 3 in which the state has changed. }, {1, 2} are noticed, and a set of a state vector representing a state one hour before the modal included in the noted combination and a state transition (1 → 3) of modal 3 is obtained.

  In addition, the transition occurrence counter (FIGS. 52A to 52C) corresponding to the state vector representing the state one hour before the modal included in the combination of interest associated with the state transition (1 → 3) of modal 3 is 1. Will only be counted up.

  When attention is paid to the modal combination of {1}, since the state of modal 1 at t = 3 one time before is 2, it is associated with the state transition (1 → 3) of modal 3 shown in FIG. 52A. Of the two transition occurrence counters, the transition occurrence counter corresponding to [2 *-] is incremented by one.

  When attention is paid to the modal combination of {2}, since the state of modal 2 at t = 3 one time before is 2, it is associated with the state transition (1 → 3) of modal 3 shown in FIG. 52B. Of the four transition occurrence counters, the transition occurrence counter corresponding to [* 2-] is incremented by one.

  When attention is paid to the modal combination of {1, 2}, since the states of modals 1 and 2 at t = 3 one time before are 2 respectively, the state transition (1 → 3) of modal 3 shown in FIG. 52C is shown. Among the eight transition occurrence counters associated with), the transition occurrence counter corresponding to [2 2 −] is counted up by one.

  Thus, when the system state transitions from [2 2 1] to [2 4 3], as shown on the left side of FIG. 57B, the state transition of modal 2 (2 → 4) The transition occurrence counter corresponding to [2 * −], [− * 1], and [2 * 1] associated with the state transition (2 → 4) is incremented by one. As shown on the right side of FIG. 57B, the modal 3 state transition (1 → 3) is associated with the modal 3 state transition (1 → 3), [2− *], [− The transition occurrence counter corresponding to 2 *] and [2 2 −] is incremented by one.

  When t = 5 and the system state transitions from [2 4 3] to [3 4 3] as shown in FIG. 46, the state at t = 5 and the state at t = 4, which is the previous time Are compared, and modal 1, which is a modal having a state change, is listed.

  In addition, the event occurrence counter is counted up.

{1,2}, {1,3}, which is an arbitrary combination of _M C _{L + 1} modal combinations when L + 1 modals are selected from three modals Each of {2, 3} and {1, 2, 3} is noticed, and the event occurrence counter corresponding to the state vector representing the state one hour before the modal included in the noted combination is counted up by one.

  When attention is paid to the modal combination of {1, 2}, the state of modal 1 is 2 and the state of modal 2 is 4 at t = 4 one time before, so the eight state vectors shown in FIG. The event occurrence counter corresponding to [2 4 −] is counted up by one.

  When attention is paid to the modal combination of {1, 3}, the state of modal 1 is 2 and the state of modal 3 is 3 at t = 4 one hour before, so the six state vectors shown in FIG. The event occurrence counter corresponding to [2-3] is incremented by one.

  When attention is paid to the modal combination of {2, 3}, the state of modal 2 is 4 and the state of modal 3 is 3 at t = 4 one time before, so the 12 state vectors shown in FIG. The event occurrence counter corresponding to [−43] is counted up by one.

  When attention is paid to the modal combination of {1, 2, 3}, the state of modal 1 is 2, the state of modal 2 is 4, and the state of modal 3 is 3 at t = 4 one time before, so FIG. 47D The event occurrence counter corresponding to [2 4 3] out of the 24 state vectors shown in FIG.

  Thus, when the state of the system transitions from [2 4 3] to [3 4 3], as shown in FIG. 58A, [2 4 −], [2 −3], [− 4 3], The event occurrence counter corresponding to [2 4 3] is incremented by one.

  Further, the transition occurrence counter is counted up.

First, {2}, {3}, which is an arbitrary _one of _M-1 C _L modal combinations when L modals are selected from modals other than modal 1 whose state has changed. , {2, 3} are noticed, and a set of a state vector representing a state one hour before the modal included in the noted combination and a state transition (2 → 3) of modal 1 is obtained.

  Further, the transition occurrence counter (FIGS. 48A to 48C) corresponding to the state vector representing the state one hour before the modal included in the combination of interest associated with the state transition (2 → 3) of modal 1 is 1. Will only be counted up.

  When attention is paid to the modal combination of {2}, since the state of modal 2 at t = 4 one time ago is 4, it is associated with the state transition (2 → 3) of modal 1 shown in FIG. 48A. Of the four transition occurrence counters, the transition occurrence counter corresponding to [* 4 −] is incremented by one.

  When attention is paid to the modal combination of {3}, since the state of modal 3 at t = 4 one time before is 3, it is associated with the state transition (2 → 3) of modal 1 shown in FIG. 48B. Of the three transition occurrence counters, the transition occurrence counter corresponding to [* -3] is incremented by one.

  When attention is paid to the modal combination of {2, 3}, since the state of modal 2 is 4 and the state of modal 3 is 3 at t = 4 one hour before, the state transition of modal 1 shown in FIG. 48C Of the twelve transition occurrence counters associated with (2 → 3), the transition occurrence counter corresponding to [* 4 3] is incremented by one.

  Thus, when the state of the system transitions from [2 4 3] to [3 4 3], as shown in FIG. 58B, it is associated with the state transition (2 → 3) of modal 1, The transition occurrence counters corresponding to * 4−], [* −3], and [* 4 3] are incremented by one.

  Causal learning proceeds by repeating the above process. The value of the event occurrence counter obtained by the causal learning and the information indicating the value of the transition occurrence counter are supplied from the causal learning processing unit 201 to the causal estimation processing unit 202 and used for causal estimation.

  Next, causal estimation performed by the causal estimation processing unit 202 will be described.

When estimating the causal relationship of state transition T ⁱ = (S ⁱ _k → S ⁱ _{k ′} ) in modal i, each L value of L = 1,2, ..., min (M-1, MaxCombi) The following processing is performed paying attention to.

CM (L; i) represents any _one of _M-1 C _L modal combinations when L modals are selected from M-1 modals other than modal i. . A state vector pattern corresponding to ^{cM (L; i)} is represented by S ^{cM (L; i)} .

When expressed ^{by; (i L)} _j, each state vector S ^cM each state vector ^{of; (i L)} S ^cM state vector pattern S ^cM; relative ^{(L i)} _j, the state transition T ⁱ conditional probability P ^{(T i | S cM (L} ; i) j) is required.

Specifically, the state vector S ^{cM (L; i)} a transition source state of the modal i as an element of _j S ⁱ _k is the state vector (S ⁱ _k by being ^{supplemented, S cM (L; i)} j ) Is generated, and the value N _S of the event occurrence counter corresponding to the generated state vector (S ⁱ _k , S ^{cM (L; i)} _j ) is acquired.

If the value N _S of the corresponding event occurrence counter is 0, the state vector S ^cM; state vector ^{_{(i) j S i k,}} S cM (L) (L; i) conditional state transition T ⁱ for _j The probability P (T ⁱ | S ^{cM (L; i)} _j ) is set as σ ₀ . σ ₀ is a fixed value between 0 and 1 that gives the lowest probability.

On the other hand, the state vector _{^{(S i k, S cM (}} L; i) j) when the value N _S of event occurrence counter corresponding to is 1 or ^{more, (T i, S cM (} L; i) j) correspond to the The value _NT of the transition occurrence counter to be acquired is acquired. (T ⁱ , S ^{cM (L; i)} _j ) represents the state vector S ^{cM (L; i)} _j associated with the state transition T ⁱ .

By dividing the value N _T of transition occurrence counter value N _S of event occurrence counter, provisional probability value p ₀ = N _T / N _S is obtained.

When using the value N _S of provisional probability value p ₀ and event occurrence counter value σ and represented by the following formula (2), the state vector S ^{cM (L; i)} the conditional probability of the state transition T ⁱ for _j P (T ⁱ | S ^{cM (L; i)} _j ) is expressed by the following equation (3). In Expression (3), min represents that the smaller one of 1 and p ₀ + σ is set as the conditional probability P.

Adding the value σ to the provisional probability value p ₀ indicates that the value obtained by optimizing the estimation error of the probability based on experience is made the final conditional probability P.

In addition to this, the conditional probability P can be corrected to 0.5 according to the following equation (4). Further, it is possible to make a correction such that p + ασ is a conditional probability P.

Since the event of state transition that is targeted at this time is an event having a binary value indicating whether it occurs or not, it can be modeled by a Bernoulli trial with an occurrence probability p. For example, the occurrence probability p = N (X, T) / N (X) obtained from N (X) samples has an estimation error of expected value √p (1-p) / N (X) Therefore, the final conditional probability P is obtained by increasing the probability by the value σ obtained in the same manner. However, since the estimation error is 0 at P = 0 or P = 1, the value σ as the estimation error is calculated using an appropriate parameter σ ₀ in practice.

When the conditional probability P (T ⁱ | S ^{cM (L; i)} _j ) equal to or greater than the threshold is obtained, the target state vector S ^{cM (L; i)} _j becomes the conditional probability P (T ⁱ | S ^{cM (L; i)} _j ) is registered in the causal candidate list. Causality candidate list, the state vector of the state transition T ⁱ and causality S ^cM; a list of ^{(L i)} _j, for each state transition T ^i, the conditional probability ^{P (T i | S cM (} L; i ⁾ State vectors S ^{cM (L; i)} _j are associated in descending order of _j ). The state represented by the state vector S ^{cM (L; i)} _j is a causal candidate for the state transition T ⁱ .

  A specific example of causal estimation will be described.

  Here, a case where a state vector having a causal relationship with state transition (1 → 2) of modal 2 is estimated will be described.

The state vector that is causally related to the state transition (1 → 2) of modal 2 is a state vector that represents both states of modal 1 and modal 3, or a state vector that represents one of modal 1 and modal 3. is there. Therefore, given an L value of 1 or 2, and considering any one of the _M-1 C _L modal combinations when L modals are selected from modals other than modal 2, The combination is {1}, {3}, {1, 3}. Each modal combination corresponds to the above-described cM (L; i).

59A to 59C, two state vector patterns corresponding to {1}, three state vector patterns corresponding to {3}, and six corresponding to {1, 3} Each of the state vector patterns corresponds to the above-described state vector pattern S ^{cM (L; i)} corresponding to ^{cM (L; i)} . The state vectors shown in FIGS. 59A-C are the same as those shown in FIGS. 50A-C.

For example, among the two state vector patterns [1 * −] and [2 * −] corresponding to {1}, [1 * −] or [2 * −] is the state vector S ^{cM ( L; i)} corresponds to _j . Subsequent processing is performed on each of the 11 state vectors S ^{cM (L; i)} _j shown in FIGS. 59A to 59C, and a conditional expression representing a causal relationship with the state transition (1 → 2) of modal 2 is performed. Probability is required.

  For example, among the 11 state vectors shown in FIGS. 59A to 59C, the state transition of modal 2 (1 → 2) with respect to the respective state vectors [1 * −] and [1 * 1] shown in FIG. 60A The case of obtaining the conditional probability will be described.

When attention is paid to [1 *-], as shown on the left side of FIG. 60B, [1 *-] is generated by supplementing 1 that is the state of the transition source of modal 2 as the element of [1 *-]. Is done. [1 1 −] corresponds to (S ⁱ _k , S ^{cM (L; i)} _j ) described above.

The value N _S (FIG. 47) of the event occurrence counter corresponding to [1 1 −] is acquired. The value N _S represents the number of times the state 1 of the state 1 and the modal 2 modal 1 has occurred simultaneously, are acquired by the causal learning.

[1 1 -] After being acquired value N _S of event occurrence counter corresponding within, are prepared in association with the state transition of the modal 2 (1 → 2), [ 1 * -] corresponding to the transition occurs The counter value N _T (left side of FIG. 60C, FIG. 50A) is acquired. The value _NT of the transition occurrence counter represents the number of times that the state 1 of the modal 1 and the state 1 of the modal 2 occurred simultaneously one time before the state transition (1 → 2) of the modal 2 occurs, and is acquired by causal learning. Has been.

Based value N _S of event occurrence counter value N _T of transition occurrence counter, [1 * -] for, conditional probability of the state transition of the modal 2 (1 → 2) is obtained. In other words, a demand is sigma ₀ as the conditional probability when the value N _S of event occurrence counter is 0, the conditional probability is calculated according to the above equation (3) when the value N _S of event occurrence counter is 1 or more.

  Similarly, when attention is paid to [1 * 1], as shown in the right side of FIG. 60B, [1 * 1] is complemented by 1 which is the state of the transition source of modal 2 as [1 * 1]. ] Is generated.

[1 * 1] (right side of FIG. 60C, FIG. 50C) prepared in association with the value N _S of the event occurrence counter corresponding to [1 1 1] and the state transition (1 → 2) of modal 2 A transition occurrence counter value _NT corresponding to is obtained.

Based on the value N _S of the event occurrence counter and the value N _T of the transition occurrence counter, the conditional probability of the modal 2 state transition (1 → 2) with respect to [1 * 1] is obtained.

  The conditional probabilities obtained as described above are appropriately registered in the causal candidate list in association with the state vector, and stored in the causal candidate list storage unit 203.

  Next, the arrangement of the causal candidate list performed by the causal candidate list arrangement processing unit 204 will be described.

  The causal candidate list is organized by merging the state vectors registered in the causal candidate list. For example, the granularity is changed so that N (T, ak, b) is kept within a suitable range. This corresponds to controlling the event b. The arrangement of the causal candidate list is performed at a predetermined timing.

Consider a state vector S ^{cM (L;)} _k defined as a particular set of states in ^{L modals} .

Whether merge state vector S ^{cM (L;)} _k and, the L-number of a certain one modal not included in modal modal i of the particular condition S ⁱ _j the state vector S ^{cM (L;)} _k And the state vector (S ^{cM (L;)} _k , S ⁱ _j ) added to The state vector S ^{cM (L;)} _k and the state vector (S ^{cM (L;)} _k , S ⁱ _j ) are respectively registered in the causal candidate list in association with the conditional probabilities of the same state transition. Is a vector.

Since the state vector (S ^{cM (L;)} _k , S ⁱ _j ) is obtained by adding S ⁱ _j to the state vector S ^{cM (L;)} _k , conceptually, the state vector S ^{cM (L;)} _{It can be said that k} is a state vector higher than the state vector (S ^{cM (L;)} _k , S ⁱ _j ). The determination of whether or not merging is possible is a determination of whether or not the lower state vector is included in the upper state vector and considered the same.

The conditional probability P of the target state transition for the state vector S ^{cM (L;)} _k is expressed by the following equation (5), and the same state transition conditional condition for the state vector (S ^{cM (L;)} _k , S ⁱ _j ) The probability P ′ is expressed by the following formula (6).

In this case, whether or not merging is possible is determined according to the following equation (7). α is an appropriate merge coefficient.

Such determination of whether or not merging is possible is performed by adding the state vector S ^{cM (L;)} _k and the specific state S ⁱ _j to all the n _i state vectors (S ^{cM (L; )} _k , S ⁱ _j ).

If any one of the n _i state vectors (S ^{cM (L;)} _k , S ⁱ _j ) can be determined to be ^{unmerged, the} state vector S ^{cM (L;)} _k is determined from the causal candidate list. Is deleted. Conceptually lower state vectors remain in the causal candidate list.

On the other hand, merge not a determination can state vector n _i number of state vectors _{^{(S cM (L;) k}} , S i j) if no in, n _i number of state vectors ^{(S cM (L;)} _k , S ⁱ _j ) and the state vector (higher-order state vector) including it as a part are all deleted. Conceptually, lower state vectors are collectively handled by upper state vectors.

  A specific example of the arrangement of the causal candidate list will be described.

Consider the state vector [1 *-] shown in FIG. 61A. [1 * −] is a state vector registered in the causal candidate list as representing the state of a causal candidate of a state transition with modal 2, and corresponds to the above-described S ^{cM (L;)} _k .

In this case, the determination of whether or not merging is possible is [1 * −] and a state vector [1 * 1] obtained by adding a state of modal 3 that is one modal not included in modal 2 shown in FIG. It is judged between [1 * 2] and [1 * 3]. [1 * 1], [1 * 2], and [1 * 3] correspond to the above-described (S ^{cM (L;)} _k , S ⁱ _j ).

  The conditional probability of a state transition with modal 2 for [1 *-] is calculated according to equation (5) above, and for each of [1 * 1], [1 * 2], [1 * 3] The conditional probability of the same state transition is calculated according to the above equation (6).

  In addition, according to the above equation (7), it is determined whether [1 * −] and [1 * 1], [1 * 2], [1 * 3] can be merged.

  When it is determined that [1 * 1], [1 * 2], and [1 * 3] can all be merged into [1 *-], [1 * 1], [1 * 2], [1 * 3] is deleted from the causality list, leaving only [1 *-].

  On the other hand, when it is determined that any one of [1 * 1], [1 * 2], and [1 * 3] cannot be merged, [1 *-] is obtained from the causality list. Deleted, leaving [1 * 1], [1 * 2], [1 * 3].

  Here, taking the Simpson paradox as an example, the concept of organizing the causal candidate list will be described.

  The Simpson paradox is as follows.

  There were 80 sick patients. Half of them, 40 were treated, and 20 were cured. On the other hand, 16 people healed when they did nothing. The 50% cure rate when treated is higher than the 40% spontaneous cure rate when doing nothing, so this treatment is considered effective.

  The number of patients was 40 for both men and women, but there were differences in the number of participants in the treatment.

  30 men were treated and 18 of them were cured. The remaining 10 were not treated, and 7 of them were naturally cured. In other words, for men only, the cure rate of 60% when treated is lower than the spontaneous cure rate of 70% when nothing is done, and it may be better not to treat.

  Only 10 women participated in treatment, only 2 of them were cured. Of the remaining 30 people, 9 were naturally cured. In other words, in the case of women, the cure rate when treated is 20% lower than the natural cure rate when doing nothing is 30%, and in this case too, it may be better not to treat .

  Is this treatment effective or not? Rather there are side effects?

There are two modals that should be considered for the transition of healing = (disease → health): "treatment" and "man and woman". These relationships are summarized as follows.
P (healing | treating) = 0.5 σ = 0.079
P (healing | no treatment) = 0.4 σ = 0.078
P (healing | treating, male) = 0.6 σ = 0.089
P (healing | treating, female) = 0.2 σ = 0.13
P (healing | no treatment, male) = 0.7 σ = 0.14
P (healing | no treatment, female) = 0.3 σ = 0.084
P (healing | male) = 0.63 σ = 0.077
P (healing | female) = 0.28 σ = 0.071

  The question is where are the causal relationships that should really be considered? Here, the above-described causal relation rearrangement method (rearrangement of state vectors of the causal candidate list) is applied. Hereinafter, the merge coefficient α = 1.

The causal relationship regarding P (healing | treating) is summarized as follows, so P (healing | treating, male) and P (healing | treating, female) cannot be merged and P (healing) It will be erased.
| P (Cure | Care) -P (Cure | Cure, Men) | = 0.1 <(0.079 + 0.089) = 0.17
| P (Cure | Care) -P (Cure | Cure, Female) | = 0.3> (0.079 + 0.13) = 0.21

Similarly, with respect to P (healing | no treatment), P (healing | no treatment, male) and P (healing | no treatment, female) cannot merge and P (healing | Without treatment).
| P (Healing | No Treatment) -P (Healing | No Treatment, Male) | = 0.3> (0.078 + 0.14) = 0.22
| P (Cure | Untreated) -P (Cure | Untreated, Female) | = 0.1 <(0.078 + 0.084) = 0.16

On the other hand, when examining P (healing | male), P (healing | treating, male) and P (healing | not treating, male) can be merged, so P (healing | treating, male) And P (healing | no treatment, male) are erased together, leaving only P (healing | male).
| P (Healing | Male) -P (Healing | Care, Male) | = 0.03 <(0.077 + 0.089) = 0.17
| P (Healing | Male) -P (Healing | No treatment, Male) | = 0.07 <(0.077 + 0.14) = 0.22

Similarly, when examining P (healing | female), P (healing | treating, female) and P (healing | not treating, female) can still merge, so P (healing | treating, female) And P (healing | no treatment, female) are deleted and only P (healing | female) remains.
| P (Healing | Women) -P (Healing | Care, Women) | = 0.08 <(0.071 + 0.13) = 0.20
| P (Healing | Female) -P (Healing | Untreated, Female) | = 0.02 <(0.071 + 0.084) = 0.15

  Based on the above, the only causal relationship that is effective at present is only “gender difference → presence / absence of cure”, and it cannot be determined whether treatment is effective or has side effects. In other words, a causal analysis of L = 2 that considers both “gender-specific” and “presence / absence of treatment” at the same time is unnecessary, and it is considered sufficient to consider within the range of L = 1.

  After that, the experiment was repeated and the number of subjects was increased 100 times, but the paradoxical situation was still unchanged as follows.

  There were 8,000 sick patients. Half of them, 4,000, were treated, and 2000 were cured. On the other hand, 1600 people were healed when nothing was done.

  There were 4,000 patients for both men and women, but there were differences in the number of participants in the treatment.

  3,000 men were treated and 1800 of them were cured. The remaining 1000 were not treated, and 700 of them were spontaneously cured.

  Only 1000 women participated in treatment, only 200 of them were cured. Of the remaining 3,000, 900 healed spontaneously.

  Thinking about this situation, let's consider what can be said from here.

  In such a situation, since the probability itself is the same as described above, and the number of samples N is 100 times, only the expected value σ of the magnitude of error is 1/10.

Based on the same calculation as before, the causal candidates for P (healing | treating) are organized as follows, so P (healing | treating, male) and P (healing | treating, female) Will be unable to merge and will delete P (healing | treating).
| P (Cure | Care) -P (Cure | Cure, Men) | = 0.1> (0.0079 + 0.0089) = 0.017
| P (Cure | Care) -P (Cure | Cure, Female) | = 0.3> (0.0079 + 0.013) = 0.021

Similarly, with respect to P (healing | no treatment), P (healing | no treatment, male) and P (healing | no treatment, female) cannot merge and P (healing | Without treatment).
| P (Healing | No Treatment) -P (Healing | No Treatment, Male) | = 0.3> (0.0078 + 0.014) = 0.022
| P (Cure | Untreated) -P (Cure | Untreated, Female) | = 0.1> (0.0078 + 0.0084) = 0.016

Next, when examining P (healing | male), P (healing | treating, male) and P (healing | not treating, male) cannot be merged, so P (healing | male) It will be erased.
| P (Healing | Male) -P (Healing | Care, Male) | = 0.03> (0.0077 + 0.0089) = 0.017
| P (Healing | Male) -P (Healing | No treatment, Male) | = 0.07> (0.0077 + 0.014) = 0.022

Similarly, for P (healing | female), P (healing | treating, female) and P (healing | not treating, female) cannot be merged, so P (healing | female) is deleted. .
| P (Healing | Female) -P (Healing | Care, Female) | = 0.08> (0.0071 + 0.013) = 0.020
| P (Healing | Female) -P (Healing | Untreated, Female) | = 0.02> (0.0071 + 0.0084) = 0.015

From the above, in this case, a causal analysis of L = 2 that considers both “gender difference” and “presence / absence of treatment” is appropriate. In other words, the causal relationships that should be considered are summarized as follows.
P (healing | treating, male) = 0.6 σ = 0.0089
P (healing | treating, female) = 0.2 σ = 0.013
P (healing | no treatment, male) = 0.7 σ = 0.014
P (healing | no treatment, female) = 0.3 σ = 0.0084

  It is misjudged to focus on only one of the factors of "gender difference" and "presence of treatment". And, as is clear from the above, in this case, it can be concluded that the treatment results are better if treatment is not performed regardless of gender. That is, this treatment has more side effects.

  The reason for the need for causal estimation for L = 2 rather than causal estimation for L = 1 focusing only on the presence or absence of treatment is clearly significant because of gender differences in the specific values of the healing outcome. To deal with the need to take gender differences into account. In fact, this stratified need is the solution to the paradox.

  The causal candidate list organization processing unit 204 merges the state vectors as necessary, so that a state vector that is necessary in terms of representing a causal relationship with a certain state transition can be left.

  Next, with reference to the flowchart of FIG. 62, description will be given of the action determination process of the action determination unit 205 performed using the causal candidate list that is appropriately arranged as described above and stored in the causal candidate list storage unit 203. To do.

  In step S211, the action determining unit 205 acquires a target value. The target value is, for example, a value representing one state of a target modal.

  In step S212, the behavior determining unit 205 reads the causal candidate list stored in the causal candidate list storage unit 203, and determines an action for transitioning the modal state to the state represented by the target value. For example, the action determination unit 205 determines the transition from the current state of the modal to the state of the target value, and the cause-and-effect candidates of each transition are registered in the cause-and-effect candidate list in descending order of conditional probability. A predetermined number is acquired from. The action determining unit 205 selects other modal states from the acquired causal candidates to the state represented by the state vector that is one of the causal candidates having the highest conditional probability or higher than a certain level. Take action.

When causal estimation is performed appropriately, as shown in FIG. 63, the state of energy of the robot is changed from the current state S ₁ to the state S ₂ to increase the energy. This can be realized by changing the sensor state and the distance sensor state to predetermined states. In the example of FIG. 63, it is understood that the energy can be increased by changing the state of the optical sensor to the state when the robot is around the light. In the energy graph of FIG. 63, the horizontal axis represents energy.

Further, as shown in FIG. 64, the state of the optical sensor, the state of the distance sensor, etc. are changed by changing the energy state of the robot from the current state S ₁₁ to the state S ₁₂ and decreasing the energy. Each can be realized by transitioning to a predetermined state. In the example of FIG. 64, it is understood that the energy can be lowered by changing the state of the optical sensor to the state when the robot is in a position where light does not reach.

  It is also possible to perform causal estimation at the time of action determination.

  As described above, the causal relationship in the form of “occurrence of an event b” causes a state transition from an event ak to an event a1 of another modal ai (however, events a1 and ak are mutually exclusive) By formulating, it is possible to narrow down candidates for causal events and to improve learning stability. Since events a1 and ak are mutually exclusive events, the event that causes state transition ak → a1 is not in modal ai, and events in this modal ai are excluded as causal estimation candidates. Can do.

  There can be multiple mechanisms for the occurrence of the event a1, but the problem complexity can be reduced by using a causal estimation problem of the state transition ak → a1.

  In causal learning, the number of simultaneous occurrences of events counted by the counter is gradually attenuated over time, so that the state transition probability representing the time-dependent causality can be dynamically followed. Furthermore, it is possible to balance the use of causality and search in consideration of the possibility of time fluctuation.

  The causal relationship between state transition T: ak → a1 and event b is formulated in the form of conditional probability P (T | ak, b), and the estimated error estimated from the number of simultaneous occurrences of these events and the conditional probability By making an action decision in consideration of the expected value σ, the trade-off between use of causality and search can be solved. That is, it is possible to realize an appropriate action regardless of whether there are many data samples or few data samples.

  FIG. 65 shows a method for solving the trade-off between the use of causal relationship and search, the above-described method using the conditional probability increased by the expected value σ, the conventional random method, ε− It is a figure which shows the result of having calculated | required the optimal degree of the action of the robot using the greedy method and the Soft-max method.

The horizontal axis in FIG. 65 represents the number of experiences, and the vertical axis represents the optimum degree of behavior. A curve L ₁ represents a result when the above-described method is used, and a curve L ₂ represents a result when the Soft-max method is used. A curve L ₃ represents a result in the case of using the ε-greedy method in which the parameter ε is decreased with time, and a curve L ₄ represents a result in the case of using a variant of the ε-greedy method. A curve L ₅ represents the result when the random method is used. As shown in FIG. 65, according to the above-described method, a better result than other methods can be obtained.

  While the other conventional methods require parameter tuning, the above-described method using the conditional probability with the probability increased by the expected value σ is not necessary, and can be said to be practical in that respect as well. .

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software executes various functions by installing a computer incorporated in dedicated hardware or various programs. The program is installed from a program recording medium on a general-purpose personal computer capable of processing.

  FIG. 66 is a block diagram illustrating an example of a hardware configuration of a computer that executes the above-described series of processes using a program.

  A CPU (Central Processing Unit) 211, a ROM (Read Only Memory) 212, and a RAM (Random Access Memory) 213 are connected to each other via a bus 214.

  An input / output interface 215 is further connected to the bus 214. The input / output interface 215 includes an input unit 216 including a keyboard, a mouse, and a microphone, an output unit 217 including a display and a speaker, a storage unit 218 including a hard disk and a nonvolatile memory, and a communication unit 219 including a network interface. A drive 220 for driving a removable medium 221 such as an optical disk or a semiconductor memory is connected.

  In the computer configured as described above, the CPU 211 loads, for example, the program stored in the storage unit 218 to the RAM 213 via the input / output interface 215 and the bus 214, and executes the above-described series of processing. Is done.

  A program executed by the CPU 211 is recorded in the removable medium 221 or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting, and is installed in the storage unit 218.

  The program executed by the computer may be a program that is processed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.

  The embodiments of the present invention are not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present invention.

It is a figure explaining the outline | summary of the process to which this invention is applied. An outline of processing to which the present invention is applied will be described. It is a functional block diagram of one embodiment of an information processing system to which the present invention is applied. It is a figure explaining a simple pendulum task. It is a flowchart explaining an example of the control processing of a simple pendulum task. It is a figure which shows an example of a time series observation signal. It is a figure which shows an example of HMM. It is a figure which shows an example of HMM. It is a figure which shows an example of HMM. It is a figure which shows an example of HMM. An example of HMM learning results in a simple pendulum task is shown. It is a flowchart explaining the detailed example of the recognition process of FIG. It is a flowchart explaining the detailed example of the recognition process of FIG. It is a flowchart explaining the detailed example of the recognition process of FIG. It is a functional block diagram of one embodiment of an information processing system to which the present invention is applied. It is a figure which shows the example of 1 display of the simulator applicable to a multimodal task. It is a figure which shows an example of the observation signal of a multimodal sensor. An example of the learning result of each modal HMM in a multimodal task is shown. It is a figure explaining an example of the path | route of HMM of distance, and control. It is a figure explaining the outline of causal estimation, It is a figure explaining the outline of causal estimation, It is a figure explaining an example of multistage action control in a multimodal task. It is a figure explaining a path | route and control of an optical HMM. It is a figure explaining an example of the causal multistage action control in a multimodal task. It is a block diagram which shows the structural example of the personal computer as an information processing apparatus with which this invention is applied. It is a figure explaining the outline | summary of the structural example of one embodiment of a data processor. It is a figure which shows the example of ergodic HMM. It is a figure which shows the example of left-to-right type HMM. It is a block diagram which shows the detailed structural example of a data processor. It is a figure which shows the example of the initial structure of HMM which the initial structure setting part 116 sets. It is a figure explaining the division | segmentation of a state. It is a figure explaining the merge of a state. It is a figure explaining addition of a state. It is a figure explaining addition of a state transition. It is a figure explaining deletion of a state. It is a flowchart explaining the learning process of a data processor. 10 is a flowchart illustrating processing of a structure adjustment unit 117. It is a figure which shows the movement locus | trajectory used by simulation. It is a figure which shows HMM obtained as a result of learning. It is a figure which shows the log likelihood calculated | required from HMM obtained as a result of learning. It is a block diagram which shows the structural example of one Embodiment of the computer to which this invention is applied. It is a figure which shows the function structural example of information processing apparatus. It is a flowchart explaining the process regarding causal perception of an information processing apparatus. It is a figure which shows the example of a modal. It is a figure which shows the specific example of a modal. It is a figure which shows the example of the time change of the state of a system. It is a figure which shows the example of an event occurrence counter. It is a figure which shows the example of the transition occurrence counter prepared corresponding to each state transition of the modal 1. It is a figure which shows the example of the state transition of the modal 1. It is a figure which shows the example of the transition occurrence counter prepared corresponding to each state transition of the modal 2. FIG. It is a figure which shows the example of the state transition of the modal 2. It is a figure which shows the example of the transition occurrence counter prepared corresponding to each state transition of the modal 3. It is a figure which shows the example of the state transition of the modal 3. It is a figure which shows the example of the event occurrence counter and the transition occurrence counter which count up. It is a figure which shows the example of the event occurrence counter which counts up. It is a figure which shows the other example of the event occurrence counter and the transition occurrence counter which count up. It is a figure which shows the further another example of the event occurrence counter which counts up, and a transition occurrence counter. It is a figure which shows the example of the event occurrence counter and the transition occurrence counter which count up. It is a figure which shows the example of a state vector pattern. It is a figure which shows the example of a state vector. It is a figure which shows the other example of a state vector. It is a flowchart explaining the action determination process of information processing apparatus. It is a figure which shows the example of the action based on a causal relationship. It is a figure which shows the other example of the action based on a causal relationship. It is a figure which shows the example of a measurement result. It is a figure which shows the structural example of a computer.

Explanation of symbols

  21 sensor units, 22 modeling units, 23 innate controllers, 24 behavior control units, 25 action units, 31 learning units, 32 HMM storage units, 33 recognition units, 34 planning units, 41 learning units, 42 controller table storage units, 43 Controller storage unit, 44 execution management unit, 61 sensor unit, 62A to 62C modeling unit, 63 causal unit, 64 action control unit, 65 action unit, 71A learning unit, 72A HMM storage unit, 73A recognition unit, 74A planning unit, 75 causal learning unit, 76 causal table storage unit, 77 causal estimation unit, 78 execution management unit, 79 controller, 80 controller table storage unit, 81 controller storage unit, 91 CPU, 98 storage unit, 100 drive, 101 remover 111, time series data input unit, 112 data adjustment unit, 113 parameter estimation unit, 114 evaluation unit, 115 model storage unit, 116 initial structure setting unit, 117 structure adjustment unit, 151 bus, 152 CPU, 153 ROM, 154 RAM, 155 hard disk, 156 output unit, 157 input unit, 158 communication unit, 159 drive, 160 input / output interface, 161 removable recording medium, 201 causal learning processing unit, 202 causal estimation processing unit, 203 causal candidate list storage unit, 204 Causal candidate list organization processing unit, 205 action decision unit

Claims (7)

Number of states that are HMM (Hidden Markov Model) structures that are trained by the Baum-Welch re-estimation method using time series data that are sensor signals observed in time series by sensing the modeling target The state transition is initialized to a predetermined initial structure in which a state restricted to a predetermined number is arranged in a space of a predetermined dimension, and the state transition is restricted to a state transition to a neighboring state. Initial structure setting means for setting a state transition probability, an output probability density function, and an initial probability as parameters;
Before estimating new parameters of the HMM in the learning of the HMM, sampling of adjusted time-series data adjusted the time-series data with respect to an increase in the number of learning times estimated and updated the parameters of the HMM Data adjustment means for adjusting the time series data by down-sampling so that the frequency increases, and outputting the adjusted time series data;
Using the time series data after the adjustment, parameter estimation means for estimating a new parameter of the HMM by a Baum-Welch re-estimation method;
Structure adjustment means for adjusting the structure of the HMM to increase or decrease the state and state transition of the HMM using the HMM of the new parameter and the time series data or the time series data after adjustment A learning device comprising: The structure adjustment means adjusts the structure of the HMM by dividing the state of the HMM, merging states, adding states, adding state transitions, deleting states, or deleting state transitions,
In the division of the state,
Among the states of the HMM, a predetermined number of states having a large variance defining an output probability density function that is a parameter of the HMM is selected as a state division target that is a target of the state division,
By adding a new state capable of the same state transition as the state division target and a state transition between the state division target, the state division target is divided,
In the state merge,
Among the states of the HMM, select one state of a set of a predetermined number of states with high correlation as a state to be merged, and select the other state as a state to be merged.
Add the state transition of the merge target state as a state transition between the merge target states,
By deleting the merge target status, the merge target status is merged with the merge target status,
In the addition of the state,
A predetermined number of states having a large variance that defines an output probability density function that is a parameter of the HMM is selected as a target for adding states,
Add a state to the target to which the state is to be added by adding a new state capable of only a self-transition and a state transition between the target to which the state is to be added,
In the addition of the state transition,
Among the set of states of the HMM, a set of a predetermined number of states having a large correlation among the sets that are not capable of state transition is selected as a target for adding the state transition,
By adding a state transition between the states to which the state transition is to be added, a state transition is added,
In the deletion of the state,
In the HMM, the time series data or the maximum likelihood path that is a state series in which the adjusted time series data is observed is obtained by the Viterbi method,
Of the states of the HMM, select a state that does not constitute the maximum likelihood path as a state deletion target,
Delete the state by deleting the state deletion target,
In the deletion of the state transition,
Of the state transitions of the HMM, select a state transition that does not constitute the maximum likelihood path as a target of state transition deletion,
The learning device according to claim 1, wherein the state transition is deleted by deleting the state transition deletion target. In the division of the state,
The initial probability of a new state obtained by dividing the state division target is set as the initial probability of the state division target.
The output probability density function of the new state obtained by dividing the state division target is changed to the output probability density function of the state division target,
Of the state transitions between the new states obtained by dividing the state division target, the state transition probability of the same state transition as the state division target is the state transition probability of the corresponding state transition of the state division target,
The state transition probability of the state transition between the new state obtained by dividing the state division target and the state division target is set to a predetermined value.
Set each one
Normalizing the initial probability and the state transition probability;
In the state merge,
Of the state transitions between the merge target states, the state transition probability of the state transition added from the merge target state is set to the state transition probability of the corresponding state transition of the merge target state. ,
Normalizing the initial probability and the state transition probability;
In addition of the state,
The initial probability of a new state added to the target to which the state is added is changed to the initial probability of the target to which the state is added.
The output probability density function of the new state added to the target to which the state is added is changed to the output probability density function of the target to which the state is added,
The state transition probability of a state transition between a new state added to the target to which the state is added is set to a predetermined value,
Set each one
Normalizing the initial probability and the state transition probability;
In addition of the state transition,
The state transition probability of the state transition added between the states to which the state transition is to be added is set to a predetermined value,
Normalizing the state transition probabilities;
In the deletion of the state,
After the deletion of the state deletion target, normalize the initial probability and the state transition probability,
In the deletion of the state transition,
The learning device according to claim 2, wherein the state transition probability is normalized after deletion of the state transition deletion target. The learning apparatus according to claim 1, further comprising an evaluation unit that evaluates the HMM that has been subjected to the learning and determines whether or not to end the learning based on a result of the evaluation of the HMM. The HMM of the initial structure is an HMM in which a predetermined number of states are arranged in a space of a predetermined dimension, and is a self-transition and a state transition between states where the distance in the space is equal to or less than a predetermined distance. The learning device according to claim 1, wherein the learning device is an HMM. A learning device that performs learning using the Baum-Welch re-estimation method of HMM (Hidden Markov Model) using time series data that is a sensor signal observed in time series by sensing the modeling target ,
The number of states and state transitions that are the structure of the HMM are changed to a predetermined initial structure in which a state constrained to a predetermined number is arranged in a space of a predetermined dimension and the state transition is constrained to a state transition to a neighboring state. Initialization, setting the state transition probability, output probability density function, and initial probability that are parameters of the HMM of the initial structure,
Before estimating new parameters of the HMM in the learning of the HMM, sampling of adjusted time-series data adjusted the time-series data with respect to an increase in the number of learning times estimated and updated the parameters of the HMM Adjust by downsampling the time series data so that the frequency increases, and output the adjusted time series data,
Using the adjusted time series data, the new parameters of the HMM are estimated by the Baum-Welch re-estimation method,
Using the new parameter HMM and the time-series data or the adjusted time-series data, the structure of the HMM is adjusted so as to increase or decrease the state and state transition of the HMM. Learning method including steps. Number of states that are HMM (Hidden Markov Model) structures that are trained by the Baum-Welch re-estimation method using time series data that are sensor signals observed in time series by sensing the modeling target The state transition is initialized to a predetermined initial structure in which a state restricted to a predetermined number is arranged in a space of a predetermined dimension, and the state transition is restricted to a state transition to a neighboring state. Initial structure setting means for setting a state transition probability, an output probability density function, and an initial probability as parameters;
Before estimating new parameters of the HMM in the learning of the HMM, sampling of adjusted time-series data adjusted the time-series data with respect to an increase in the number of learning times estimated and updated the parameters of the HMM Data adjustment means for adjusting the time series data by down-sampling so that the frequency increases, and outputting the adjusted time series data;
Using the time series data after the adjustment, parameter estimation means for estimating a new parameter of the HMM by a Baum-Welch re-estimation method;
Structure adjustment means for adjusting the structure of the HMM to increase or decrease the state and state transition of the HMM using the HMM of the new parameter and the time series data or the time series data after adjustment As a program to make the computer function.

Images