JP5361615B2 - Behavior control learning method, behavior control learning device, behavior control learning program - Google Patents

Description

  The present invention relates to a behavior control learning method and a behavior control learning device for determining what action the system will take next in response to a user action in a system in which the system and the user interact with each other (such as a dialogue system). It relates to a behavior control learning program.

  Non-patent documents 1, 2 and 3 are known as behavior control techniques using a partially observable Markov decision process (hereinafter referred to as “POMDP”).

  Non-Patent Document 1 targets the task of buying tickets between six cities. Non-Patent Document 2 is directed to a task of troubleshooting DSL (Digital Subscriber Line). In these behavior control technologies, the types of tasks (the types of actions that can be taken) and the state transition method (in which order the behaviors are performed) are known. In Non-Patent Document 3, the system behavior is determined from a large amount of data. When POMDP is obtained, the task is known as in other non-patent documents.

J. Williams, P. Poupart, S. Young, "Partially Observable Markov Decision Processes with Continuous Observations for Dialogue Management", Recent Trends in Discourse and Dialogue, Springer Netherlands, 2008, Volume 39, p.191-217 Jason D. Williams, "Applying POMDPs to Dialog Systems in the Troubleshooting Domain", Bridging the Gap: Academic and Industrial Research in Dialog Technologies, 2007.4, p.1-8 K. Kim, C. Lee, S. Jung, G. G. Lee, “A Frame-Based Probabilistic Framework for Spoken Dialog Management Using Dialog Examples”, Proceedings of the 9th SIGdial Workshop on Discourse and Dialogue, 2008.6, p.120-127

  However, since all conventional technologies target tasks whose task types and state transitions are known, task types (greeting, handshake, fun conversations, chats, etc.) and task state transitions such as dialogue There are various methods, and there is a problem that it is not possible to perform behavior control on a system that cannot be predicted in advance by a system designer.

The behavior control learning device of the present invention is a device that generates learning data for performing person-to-person behavior in a person-to-person system. In addition, in the data indicating person-to-person behavior, one person is assigned as a user, the other person is assigned as a system, the user's action is taken as an observation value, the system action is taken as an action, and a series of actions consisting of observation values and actions. An evaluation value is obtained by evaluating whether or not the series is a desired action series. The behavior control learning device of the present invention includes a behavior data recording unit that stores observation values, actions, and evaluation values, a DBN generation unit, a DBN-POMDP conversion unit, and a reinforcement learning unit. DBN generation unit, observations, using the action and the evaluation value to generate a dynamic Bayesian network (hereinafter referred to as "DBN"), the compensation when executing an action a _t in state s _t probability P (r _t | s _{t, a t),} the probability state by the action _{a t} changes from _{s t} to _{s t + 1 P (s t} + 1 | s t, a t), the probability that the observed value _{o t + 1} in the state _{s t + 1} by the action _{a t} is observed P (o _{t + 1} | s _{t + 1} , a _t ) is estimated. The DBN-POMDP converter uses the probabilities P (r _t | s _t , a _t ), P (s _{t + 1} | s _t , a _t ), and P (o _{t + 1} | s _{t + 1} , a _t ) by action a. Probability P (s ′ | s, a) that state changes from s to s ′, probability P (o ′ | s ′, a _t ) that outputs observation value o ′ in state s ′ by action a, and action in state s A reward r (s, a) when a is executed is generated. The reinforcement learning unit uses the probabilities P (s ′ | s, a), P (o ′ | s ′, a _t ) and the reward r (s, a), and the system uses the probability distribution of the current state as an argument. Generate a function that outputs one action to take.

Further, the present invention, the state _{s t} a _{s a} representative of the internal state of the _{s o} and actions representing the internal state of the observed values set _{s t = (s o, s} a) ( _Note, s o, notation _{s a} In this case, the DBN generation unit generates a DBN with P (a | s _a ) = 1 only when a = s _a , and the DBN-POMDP conversion unit generates a reward ^ r ( (*, S _a ), a) [where, * represents an arbitrary s _o ] is a reward ^ r ((*,), so that 1 is taken when a = s _a and 0 is taken otherwise. s _a ), a) are determined, and a final objective function V _t is obtained by the following equation, which is replaced with a linear sum αr + β ^ r of a reward r for a desired action sequence and a reward r for a statistical action sequence.

  According to the behavior control learning device of the present invention, a function other than a desired behavior series is modeled and an action is determined. Therefore, if the system uses the function generated by the behavior control learning apparatus of the present invention, it is possible to perform statistically natural behavior even for user behavior other than the desired behavior series.

The figure which shows the structural example of the action control learning apparatus 100 of Example 1. FIG. The figure which shows the example of data memorize | stored in an action data memory | storage part. The figure which shows the relationship between s _a , a, and P (a | s _a ). The figure which shows the structure and variable of POMDP. The figure which shows a simulation result. The block diagram which illustrated the hardware constitutions of action control learning device 100.

  Hereinafter, embodiments of the present invention will be described in detail.

[Behavioral control learning device 100]
The behavior control learning device 100 generates learning data for performing a person-to-person action in a person-to-system. FIG. 1 shows a configuration example of the behavior control learning apparatus 100 according to the first embodiment. A behavior control learning apparatus 100 according to the first embodiment will be described with reference to FIG.

  The behavior control learning apparatus 100 includes a behavior data storage unit 101, a dynamic Bayesian network (hereinafter referred to as “DBN”) generation unit 103, a DBN probability table storage unit 105, a DBN-POMDP conversion unit 107, and a POMDP probability / reward table. A storage unit 109, a reinforcement learning unit 111, a POMDP policy storage unit 113, a state distribution update unit 115, a state probability table storage unit 117, and an action determination unit 119 are provided.

[Behavior data storage unit 101]
In data indicating person-to-person actions (for example, voice data or image data recording a dialogue), one person is assigned as a user, the other person is assigned as a system, and the user action is set as an observation value o. Is an action a, and an evaluation value r is an evaluation of whether or not a series of behavior sequences composed of observed values and actions is a desired behavior sequence.
The behavior data recording unit 101 stores an observation value o, an action a, and an evaluation value r. FIG. 2 shows an example of data stored in the behavior data storage unit.

  For example, handshake, greeting, laughter, movement, chatter, nod, swing, no action, and 8 action labels are prepared. Each label is associated with a numerical value of 0-7. Are stored in the behavior data storage unit 101 as behavior data. In this embodiment, the observation value and the action are stored as a pair. Further, it is evaluated whether or not a series of action series (one or more user / system action pairs) is a desired action series, and when the evaluation value is a desired action series, 1 is set. Is stored as 0.

  For example, did the user enjoy the desired action sequence? Is it a typical action sequence? Was it helpful to users? Etc. A typical sequence of actions is “shake each other, greet each other, then repeat laughter, chatter and nod several times at random, greet each other at the end, and shake hands” is there.

One evaluation is given for each action series. Here, in order to learn this value statistically, this value is distributed to each time. One of the following is used as a method for this distribution.
(Distributing means 1) If the observed action series is evaluated as 1, all values are set to 1. If the evaluation is 0, all values are set to 0.
(Distributing means 2) Evaluates only a part of the observed action sequence. If the evaluation of the part is 1, only 1 between the start and end of the part is set to 1. All other parts are set to zero.
(Distributing means 3) When the starting and ending points are known as in (distributing means 2), the other values that give 1 only to the last data of that portion are set to 0.

  Note that this evaluation value may be a multi-value instead of a binary value of 0 and 1, or may be a continuous value. In addition, although the discussion proceeds with the evaluation of one person here, the average of a large number of people may be used as the evaluation. A plurality of desired action sequences may be prepared, and an action sequence label may be provided for each desired action sequence. Evaluation may be given to each action series, and the action series label and the evaluation may be combined and stored. The person-to-person behavior data may be collected based on a plurality of person data instead of one-to-one data. In this case, there are a plurality of users and systems.

  The action label may be assigned manually, or it may be automatically assigned by recognizing which action corresponds to using voice recognition software or image recognition software. Moreover, what is necessary is just to select manual and automatic as evaluation according to evaluation object. For example, was it useful for the user as appropriate? Is difficult to recognize using voice recognition software or image recognition software, and is therefore given manually. Whether or not a typical action sequence is performed may be automatically given. The behavior control learning device 100 may be configured to include a recognition unit and a label providing unit, and generate observation values, actions, and evaluation values internally using conversation data or video data itself as input values.

[DBN generation unit 103 and DBN probability table storage unit 105]
DBN generating unit 103, by using the observed value o, action a and the evaluation value r, to generate a DBN, of reward when you perform an action _{a t} in state _{s t} probability _{P (r t | s t,} a t ), the action _{a t} probability state changes from _{s t} to _{s t + 1} by _{P (s t + 1 | s} t, a t), the action _{a t} probability the observed value _{o t + 1} in the state _{s t + 1} is observed by _{P (o t +} 1 | s _{t + 1} , a _t ). It should be noted, s hidden state between the user system (hereinafter referred to as "state"), and the state s is composed of a combination of a hidden state s _a for a hidden state s _o of the user-system behavior generation, t Represents time, and the evaluation value r is treated as a reward r which is a random variable. Here, t is a symbol used to clarify the relative time relationship of the variables, and does not assume a specific time. That is, the probabilities shown here and the calculations using them are independent of time.

For example, the DBN generation unit 103 performs likelihood maximization by using an EM algorithm, a junction tree algorithm, a sampling method, and the like using a time series of the observation value o _t , the action a _t , and the evaluation value r _t , and generates an action generation model Learning and generating DBN for. Further, the internal state of the system and the user is separated into the internal state of the system and the state corresponding to the action as s = (s _o , s _a ). In order to make one-to-one correspondence between s _a and a, a DBN is created with P (a | s _a ) = 1 only when a = s _a . FIG. 3 shows the relationship between _sa , a, and P (a | s _a ).

The probability estimated by the DBN generation unit 103 is stored in the DBN probability table storage unit 105.
[DBN-POMDP conversion unit 107 and POMDP probability / reward table storage unit 109]
The DBN-POMDP conversion unit 107 uses the probabilities P (r _t | s _t , a _t ), P (s _{t + 1} | s _t , a _t ), and P (o _{t + 1} | s _{t + 1} , a _t ) to take action a Is the probability that the state will change from s to s ′ (state transition probability) P (s ′ | s, a), and the probability that the observed value o ′ will be output in state s ′ by action a (output probability) P (o ′ | s ', A _t ) and a reward r (s, a) when the action a is executed in the state s.

Here, a probability model called POMDP will be described. Action generation is realized by this POMDP. FIG. 4 shows the structure and variables of POMDP. This model defines a state s that describes the state of the system and the psychological state of the user. s is expressed as a combination of a plurality of states, such as s = (s ₁ , s ₂ , s ₃ ,..., s _N ). o represents an observed value, and a represents an action that acts on the user from the system side. In this case, the probability between these variables P (s' | s, a ), the probability P (o '| s', a t) and reward r (s, a) has been set.

  The DBN-POMDP conversion unit 107 converts the probability estimated by the DBN generation unit 103 into a POMDP probability / reward according to the following equation. It is assumed that symbols with the same definition are used as observation values and actions.

Since the structures of DBN and POMDP are almost the same, regarding the state transition probability P (s ′ | s, a) and the output probability P (o ′ | s ′, a _t ), values may be substituted for the corresponding probabilities. Since the reward is handled as a random variable in the DBN, the random variable obtained by the DBN is averaged to be converted into a real number. For example, it is obtained from the probability distribution of r by equation (1). This setting is a method unique to the present invention that is not present in the prior art.

The POMDP probability / reward table storage unit 109 includes the probabilities P (s ′ | s, a), P (o ′ | s ′, a _t ) and the reward r (s, converted and obtained by the DBN-POMDP conversion unit 107. Store a).

[Reinforcement learning unit 111 and POMDP policy storage unit 113]
The reinforcement learning unit 111 uses the probabilities P (s ′ | s, a), P (o ′ | s ′, a _t ) and the reward r (s, a) to perform probability distribution of the current state through reinforcement learning. A function that outputs one action to be taken by the system (hereinafter referred to as “policy”) is generated.

  The POMDP policy storage unit 113 stores the policy generated by the reinforcement learning unit 111.

Next, a policy calculation method will be described. First, equation (4) shows a reward that can be acquired in the future when the action sequence a _{τ + t} is known.

Here, b _{τ + t} (s) is the distribution of the state at time τ + t. Further, the contribution of the future reward is reduced by the positive constant γ (<1). The policy is a function that calculates the current action a that maximizes Equation (4) from the current state distribution b _t (s).
[Method to select action according to statistical information appearing in data]
First, the following expression is obtained from the definition of the probability distribution b _t (s) of the current state.

This is, past o _1, a _{1, ...,} series that a _t-1, o _t, that is, after the history of observations and actions of the user and the system has been executed, represents the probability that the state is s _a Yes.
P at _{a t} = _s a _| because you are (a _s a) = _1, to obtain the following expression when _a t = _s a.

This is, past _{o 1,} a _{1, ...,} represents the probability that the action _{a t} happens next _{when a t-1, o t} is observed. In other words, it is made up of data of until now as the probability of indicating whether a _t how much nature. That is, if the POMDP reward is determined so as to maximize the expression (7), a natural action is generated according to the policy. To achieve this, rewards

Can be set as However, it is necessary to satisfy a = s _a . In order to determine the reward in this way, here, reward {circumflex over (r)} ((*, s _a ), a) is determined so that 1 is taken when a = s _a and 0 is taken otherwise.

Here, * refers to any of the _{s o.} If r is replaced with ^ r using this value, a natural dialogue can be realized. Here, in order to realize a conventional desired action sequence, a linear sum of rewards of the conventional method is taken. In order to do this, the final objective function V _t is obtained by the following equation (10) in which r in equation (4) is replaced by αr + β ^ r.

Here, α and β are arbitrary real numbers. By changing [alpha] and [beta], it is possible to weight the degree of priority whether a desired action is realized (when [alpha] is large) or a statistical action is prioritized ([beta] is large). Note that α and β can be set to 0.

  Normally, when learning of action generation by POMDP is performed on a target desired action sequence, the system tries to realize only the desired action sequence. For this reason, not only the desired behavior sequence but also various sequences are included in the record of the human behavior, the behavior other than the desired behavior sequence is not selected. Therefore, while reproducing the exchange between people. When creating a system that attracts users to a desired action sequence, it is not sufficient to learn only the desired action sequence. According to the present invention, when configuring a system, the behavioral control of the system is learned including statistical information of these behavioral sequences, so that natural behavioral control can be performed while pulling in the desired behavioral sequence. it can.

[Action control using policies]
Hereinafter, a method for controlling behavior using a policy will be described [state distribution update unit 115 and state probability table storage unit 117].
The state probability table storage unit 117 stores the probability distribution b _{t−1 of the} previous state. When the observation value o _t ′ is input, the state distribution update unit 115 queries the POMDP probability / reward table storage unit 109 from the system action a performed immediately before, and determines the state transition probability P from the stored statistics. (S ′ | s, a) is obtained. Further, the POMDP probability / reward table storage unit 109 is inquired from the observed value o _t ′, and the output probability P (o ′ | s ′, a) is obtained from the stored statistics. Further, the state probability table storage unit 117 is queried, the probability distribution b _t-1 of the previous state is received, and the probability distribution b _t of the current state is obtained by the following equation.

Note that η is a normalization constant for making the sum of all 1s. The obtained probability distribution b _t of the current state is stored in the state probability table storage unit 117 and output to the action determination unit 119.

[Action decision unit 119]
The action determination unit 119 acquires a policy from the POMDP policy storage unit 113 and stores it prior to behavior control. Further, upon receiving a probability distribution b _t of the current state, which as an argument to the policy f (), the system determines and outputs an action a _t should take.
By adopting such a configuration, it is possible to model a function other than a desired action sequence and generate a function for determining an action. If the system uses the function generated by the action control learning device of the present invention, any desired system can be used. It is possible to perform statistically natural behavior even for user behaviors other than the behavior series.

  The behavior control learning device 100 includes a state distribution update unit 115, a state probability table 117, and an action determination unit. However, these configurations are configured as separate devices, and state transition is performed in response to an inquiry from the separate device. It is good also as a structure which outputs a probability, an output probability, and a policy.

[simulation result]
Experiments were performed by simulation of behavior control assuming one-on-one behavior record data. FIG. 5 shows the simulation results. There are eight types of actions: handshake, greeting, laughter, movement, chatter, nod, pretend, and no action. Similarly, the above eight types of observation values were used. In general, it is assumed that there is a misrecognition of the observed value, but here it is a definite value. However, the hidden state represents the user's intention, and this part cannot be observed. This part was hidden. The number of hidden state _{s o} is 16. At the set hidden state s _a corresponding one-to-one separately to the system action was the number of its state 8. Two types of sequences were prepared as desired behavior sequences for labeling. These series were automatically labeled by the computer, and 1 was assigned to those judged to be the desired action series. One of these is a series of actions that shake hands with each other, greet each other, then repeat laughter, chatter and nod several times at random, greet each other at the end, and shake hands. . The other one moves, the other is inactive, then greets each other, repeats laughter, chatting and nodding several times at random, greets each other, and finally one does nothing, It is an action sequence of moving. (Distributing means 2) was used as a method of giving a reward for this action series. That is, it is assumed that the time from the start time to the end time of this action sequence is known, and 1 is added as a reward at each time from the start time to the end time. These action sequences were set to 1/10 of the total number of learning data. In the rest of the data, the user's observations and system behavior pairs are handshake-shake, greeting-greeting, laugh-laugh, move-move, talk-talk, nodding-talk, neck-shake, no action-no action Samples were created so that the probability of occurrence was statistically increased. If the user wants a desired action sequence, the system is learned so that the system operates to approach this desired sequence. However, if the user is not willing, he learns to act to show the statistical behavior of the remaining samples. A total of 10,000 samples of data were created as learning data. A dynamic Bayesian network was created from this data using the proposed method, converted into a POMDP probability / reward table, and a policy that was a behavior selection method was obtained by reinforcement learning. For comparison, a method of rewarding only a desired sequence in POMDP was used. For the evaluation, data of 2000 samples were used. Only the observation values of the user were generated according to the method of generating the desired series of learning data and the method of generating the other series of learning data. In the experiment, when the user wanted a desired series, the behavior of the desired series was performed, and whether the behavior was selected according to the statistics of the data for other data was examined.

  The method of rewarding only the desired series generated all the actions correctly for 200 samples of the desired series. The proposed method also showed correct behavior for all desired sequences. This confirms that both methods generate the correct action for the desired sequence.

  Observed and generated observation / action pairs using a method that rewards only the desired action sequence, observation / action pairs generated by the proposed method, observed / action pairs existing in the learning data The main frequencies are shown in FIG. As shown in this figure, it can be seen that in POMDP that rewards only a desired action sequence, an action is frequently selected for the observed value / action pair included in the learned desired sequence. However, it can be seen that it is far from the statistical pattern of the observed value / action pair of the learning data shown on the far right. This is because the method of giving a reward only to a desired behavior sequence determines the behavior so that the desired sequence is obtained no matter what the observed value is observed. On the other hand, it can be seen that the method of introducing the occurrence probability of the action proposed this time to the reward approaches the statistic amount of learning data other than the desired series which is 9 times.

  In the method of rewarding only a desired behavior sequence, an action for the desired behavior sequence is generated even when the user does not execute the desired behavior. This is because the system has learned only the desired behavior. In a system where tasks are determined from the beginning, this behavior is natural. However, while reproducing the interaction between people. When creating a system that attracts users to a desired action sequence, it is not sufficient to learn only the desired action sequence. Therefore, according to the present invention, when the user does not execute the desired action, the system operates according to the statistical amount in the learning data so that the user can perform a natural action even when the user does not execute the desired action. There is an effect that it can be controlled.

<Hardware configuration>
FIG. 6 is a block diagram illustrating a hardware configuration of the behavior control learning apparatus 100 according to the present embodiment. As illustrated in FIG. 6, the behavior control learning device 100 of this example includes a CPU (Central Processing Unit) 11, an input unit 12, an output unit 13, an auxiliary storage device 14, a ROM (Read Only Memory) 15, a RAM ( Random Access Memory) 16 and a bus 17.

  The CPU 11 in this example includes a control unit 11a, a calculation unit 11b, and a register 11c, and executes various calculation processes according to various programs read into the register 11c. The input unit 12 is an input interface for inputting data, a keyboard, a mouse, and the like, and the output unit 13 is an output interface for outputting data. The auxiliary storage device 14 is, for example, a hard disk, a semiconductor memory, or the like, and stores a program for causing the computer to function as the behavior control learning device 100 and various data. Further, the above-mentioned program and various data are expanded in the RAM 16 and used from the CUP 11 or the like. The bus 17 connects the CPU 11, the input unit 12, the output unit 13, the auxiliary storage device 14, the ROM 15, and the RAM 16 so that they can communicate with each other. In addition, as a specific example of such hardware, a server apparatus, a workstation, etc. other than a personal computer can be illustrated, for example.

<Program structure>
As described above, each program for executing each process of the behavior control learning apparatus 100 according to the present embodiment is stored in the auxiliary storage device 14. Each program constituting the license management program may be described as a single program sequence, or at least a part of the program may be stored in the library as a separate module.
<Cooperation between hardware and program>
The CPU 11 expands the above-described program and various data stored in the auxiliary storage device 14 in the RAM 16 according to the read OS program. The address on the RAM 16 where the program and data are written is stored in the register 11c of the CPU 11. The control unit 11a of the CPU 11 sequentially reads these addresses stored in the register 11c, reads a program and data from the area on the RAM 16 indicated by the read address, causes the calculation unit 11b to sequentially execute the operation indicated by the program, The calculation result is stored in the register 11c.

  FIG. 1 is a block diagram illustrating a functional configuration of the behavior control learning apparatus 100 configured by reading and executing the above-described program in the CPU 11 as described above.

  Here, the behavior data storage unit 101, the DBN probability table storage unit 105, the POMDP probability / reward table storage unit 109, the POMDP policy storage unit 113, and the state probability table storage unit 117 are the auxiliary storage device 14, RAM 16, register 11c, and others. It corresponds to any one of the buffer memory, the cache memory, etc., or a storage area using these together. The DBN generation unit 103, the DBN-POMD conversion unit P107, the reinforcement learning unit 111, the state distribution update unit 115, and the action determination unit 119 are configured by causing the CPU 11 to execute a license management program.

100 behavior control learning device 101 behavior data storage unit 103 DBN generation unit 105 DBN probability table storage unit 107 DBN-POMDP conversion unit 109 POMDP probability / reward table storage unit 111 reinforcement learning unit 113 POMDP policy storage unit 115 state distribution update unit 117 state Probability table storage unit 119 Action determination unit

Claims (5)

A behavior control learning device that generates learning data for performing a person-to-person action in a person-to-person system,
In data showing person-to-person actions, one person is assigned as a user, the other person is assigned as a system, the user action is taken as an observed value, the system action is taken as an action, and a series of actions consisting of an observed value o and an action a An evaluation value r is obtained by evaluating whether or not the series is a desired action series.
An action data recording unit for storing the observed value o , the action a, and the evaluation value r ;
t is assumed to represent a time, using the observation value o, action a and evaluation value r, to generate a dynamic Bayesian network (hereinafter referred to as "DBN"), reward when executing the action a _t in state s _t r _t of the probability _{P (r t | s t,} a t), the probability state by the action _{a t} changes from _{s t} to _{s t + 1 P (s t} + 1 | s t, a t), by the action _{a t} in state _{s t + 1} A DBN generator that estimates a probability P (o _{t + 1} | s _{t + 1} , a _t ) that the observed value o _{t + 1} is observed;
Using the probabilities P (r _t | s _t , a _t ), P (s _{t + 1} | s _t , a _t ), P (o _{t + 1} | s _{t + 1} , a _t ), the state is changed from s to s ′ by action a P (s ′ | s, a), the probability P (o ′ | s ′, a _t ) of outputting the observation value o ′ in the state s ′ by the action a, and the action a in the state s A DBN-POMDP conversion unit for generating a reward r (s, a);
Using the probabilities P (s ′ | s, a), P (o ′ | s ′, a _t ) and the reward r (s, a), an action to be taken by the system using the probability distribution of the current state as an argument for example Bei and reinforcement learning unit for generating a function to one output, the,
The reward r (s, a) is a product sum of r _t and P (r _t | s _t , a _t ).
A behavior control learning apparatus characterized by that. The behavior control learning device according to claim 1,
The state _{s t} a _{s a} representative of the internal state of the _{s o} and actions representing the internal state of the observed values set _{s t = (s o, s} a) and,
The DBN generation unit generates a DBN with P (a | s _a ) = 1 only when a = s _a ,
The DBN-POMDP conversion unit, reward _{^ r ((*, s a} ), a) [ where * represents any _{s o]} the

As sought
The reinforcement learning unit uses αr (s, a) + β ^ r ((*, s _a ), a) [where α and β are arbitrary real numbers] instead of the reward r (s, a). Using the current state probability distribution as an argument, generate a function that outputs one action that the system should take,
A behavior control learning apparatus characterized by that. A behavior control learning method for generating learning data for performing a person-to-person action in a person-to-system,
In data showing person-to-person actions, one person is assigned as a user, the other person is assigned as a system, the user action is taken as an observed value, the system action is taken as an action, and a series of actions consisting of an observed value o and an action a An evaluation value r is obtained by evaluating whether or not the series is a desired action series.
t is assumed to represent a time, using the observation value o, action a and evaluation value r, to generate a dynamic Bayesian network (hereinafter referred to as "DBN"), reward when executing the action a _t in state s _t r _t of the probability _{P (r t | s t,} a t), the probability state by the action _{a t} changes from _{s t} to _{s t + 1 P (s t} + 1 | s t, a t), by the action _{a t} in state _{s t + 1} A DBN generation step of estimating a probability P (o _{t + 1} | s _{t + 1} , a _t ) that the observed value o _{t + 1} is observed;
Using the probabilities P (r _t | s _t , a _t ), P (s _{t + 1} | s _t , a _t ), P (o _{t + 1} | s _{t + 1} , a _t ), the state is changed from s to s ′ by action a P (s ′ | s, a), the probability P (o ′ | s ′, a _t ) of outputting the observation value o ′ in the state s ′ by the action a, and the action a in the state s A DBN-POMDP conversion step for generating a reward r (s, a);
Using the probabilities P (s ′ | s, a), P (o ′ | s ′, a _t ) and the reward r (s, a), an action to be taken by the system using the probability distribution of the current state as an argument for example Bei and reinforcement learning step of generating a function to one output, the,
The reward r (s, a) is a product sum of r _t and P (r _t | s _t , a _t ).
A behavior control learning method characterized by that. The behavior control learning method according to claim 3,
The state _{s t} a _{s a} representative of the internal state of the _{s o} and actions representing the internal state of the observed values set _{s t = (s o, s} a) and,
The DBN generation step generates a DBN with P (a | s _a ) = 1 only when a = s _a ,
The DBN-POMDP conversion step, reward _{^ r ((*, s a} ), a) [ where * represents any _{s o]} the

As sought
In the reinforcement learning step, αr (s, a) + β ^ r ((*, s _a ), a) [where α and β are arbitrary real numbers] instead of the reward r (s, a). Using the current state probability distribution as an argument, generate a function that outputs one action that the system should take,
A behavior control learning method characterized by that.   A program for causing a computer to function as the behavior control learning device according to claim 1.

Images