KR102190584B1 - System and method for predicting human choice behavior and underlying strategy using meta-reinforcement learning - Google Patents

Abstract

A system and method for estimating human behavior patterns and behavior strategies using meta-reinforcement learning are presented. A system for estimating human behavior patterns and behavior strategies using meta-reinforcement learning according to an embodiment includes: a model-free (MF) reinforcement learning unit for learning a reward function; A model-based (MB) reinforcement learning unit that learns the reward function and a state-transition function and dynamically interacts with the model-free (MF) reinforcement learning unit; And a hierarchical control unit for predicting human decision-making by hierarchically controlling weights by dynamically assigning weights to the model-free (MF) reinforcement learning unit and the model-based (MB) reinforcement learning unit. have.

Description

Human behavior pattern and behavioral strategy estimation system and method using meta-reinforcement learning {SYSTEM AND METHOD FOR PREDICTING HUMAN CHOICE BEHAVIOR AND UNDERLYING STRATEGY USING META-REINFORCEMENT LEARNING}

The following embodiments relate to a system and method for estimating human behavior patterns and behavior strategies using meta-reinforcement learning.

The reinforcement learning (RL) algorithm has been used to solve the adaptive control problem because the value is updated in the direction of reducing the degree of prediction error calculated through compensation in order to find the optimal policy. . Reinforcement learning (RL) is a model-free (MF) reinforcement learning (RL) model that learns a reward function, and a model that learns both a reward function and a state-transition function. It can be classified as model-based (MB) reinforcement learning (RL). In most practical problems, the model-free (MF) reinforcement learning (RL) agent is suitable for general purposes because the state-space representation corresponding to the model is unclear. However, the model-based (MB) reinforcement learning (RL) agent has the advantage of quickly adapting by learning the state-transition function itself, which is utilized in a fully exploited state-space problem. .

Numerous studies on human reinforcement learning (RL) have shown that humans are using model-free (MF) reinforcement learning (RL) and model-based (MB) reinforcement learning (RL) together, and the uncertainty of state-transition. It has been suggested that it is hierarchically controlled by (Non-Patent Document 1). This study proposed a computer model for hierarchical control over two reinforcement learning (RL) to perform tasks with varying degrees of state-transition uncertainty.

However, most human reinforcement learning (RL) studies, including the above studies, assumed a simplified Markov decision process (MDP) environment that places major limitations on predicting real human selection behavior. The traditional approach to understanding human reinforcement learning (RL) is human learning because the real-life environment has a number of factors that influence the reinforcement learning (RL) process and hierarchical control over reinforcement learning (RL). And the impact of reinforcement learning (RL) on behavior. This problem can be solved by testing the Markov Decision Process (MDP), which takes into account realistic environmental factors.

S. W. Lee, S. Shimojo, and J. P. ODoherty, Neural Computations Underlying Arbitration between Model-Based and Model-free Learning, Neuron, vol. 81, no. 3, pp. 687-699, Feb. 2014. P. Dayan and L. F. Abbott, Theoretical neuroscience: computational and mathematical modeling of neural systems. Massachusetts Institute of Technology Press, 2001.

The embodiments describe a system and method for estimating human behavior patterns and behavior strategies using meta-reinforcement learning, and more specifically, a model-based (MB) reinforcement learning (RL) method and a number of models related to various task goals. It is to provide a system and method for estimating human behavior patterns and behavior strategies using meta-reinforcement learning that dynamically applies control weights to a model-free reinforcement learning (RL) method.

A system for estimating human behavior patterns and behavior strategies using meta-reinforcement learning according to an embodiment includes: a model-free (MF) reinforcement learning unit for learning a reward function; A model-based (MB) reinforcement learning unit that learns the reward function and a state-transition function and dynamically interacts with the model-free (MF) reinforcement learning unit; And a hierarchical control unit for predicting human decision-making by hierarchically controlling weights by dynamically assigning weights to the model-free (MF) reinforcement learning unit and the model-based (MB) reinforcement learning unit. have.

The hierarchical control unit includes the model-free (MF) reinforcement learning unit and the model-based (MB) reinforcement learning unit through the transition rates in two directions, which are functions of reliability. ) The weights of the reinforcement learning unit and the model-based (MB) reinforcement learning unit may be updated.

The hierarchical control unit may predict the human decision making and estimate a behavioral strategy inherent in the human decision making.

The hierarchical control unit includes the model-free (MF) reinforcement learning unit and the model-based (MB) reinforcement learning unit among a target value, which is a variable task variable, a state-space size, and a state-transition uncertainty. By hierarchical control by at least one or more, human decision-making can be predicted.

The model-free (MF) reinforcement learning unit may reflect a target value composed of target-dependent multiple model-free (MF) reinforcement learning agents in order to provide a plurality of policies according to a plurality of goals.

The model-free (MF) reinforcement learning unit or the model-based (MB) reinforcement learning unit, the variable task variable target value (goal value), state-space size, and state-transition uncertainty from a variable environment. Received and calculate a transition probability between the model-free (MF) reinforcement learning unit and the model-based (MB) reinforcement learning unit, the hierarchical control unit may determine a model selection probability through the transition probability.

The hierarchical control unit calculates an operation value by integrating the model selection probability of the model-free (MF) reinforcement learning unit and the model-based (MB) reinforcement learning unit, and provides an operation selection probability through the operation value. You can convert it into an action.

According to another embodiment, a method for estimating human behavior patterns and behavior strategies using meta-reinforcement learning may include dynamically assigning weights to a model-free (MF) reinforcement learning unit and a model-based (MB) reinforcement learning unit; And predicting human decision making by hierarchically controlling the model-free (MF) reinforcement learning unit and the model-based (MB) reinforcement learning unit according to the assigned weight.

The model-free (MF) reinforcement learning section and the model-free (MF) reinforcement learning section and the model-free (MF) reinforcement learning section and the It may further include the step of updating the weight of the model-based (MB) reinforcement learning unit.

In the hierarchical control to predict human decision-making, the human decision-making may be predicted, and a behavioral strategy inherent in the human decision-making may be estimated.

In the hierarchical control of predicting human decision making, the model-free (MF) reinforcement learning unit and the model-based (MB) reinforcement learning unit are variable task variables, such as a goal value and a state- Human decision making can be predicted by hierarchically controlling at least one or more of the size of the space and the uncertainty of the state-transition.

The model-free (MF) reinforcement learning unit may reflect a target value composed of target-dependent multiple model-free (MF) reinforcement learning agents in order to provide a plurality of policies according to a plurality of goals.

The step of predicting human decision-making by controlling hierarchically may include, in the model-free (MF) reinforcement learning unit or the model-based (MB) reinforcement learning unit, a target value that is the variable task variable from a variable environment ( goal value), the size of the state-space, and the uncertainty of the state-transition, using the result of calculating the transition probability between the model-free (MF) reinforcement learning unit and the model-based (MB) reinforcement learning unit , The model selection probability can be determined.

The hierarchically controlling and predicting human decision-making may include calculating an operation value by integrating the model selection probability of the model-free (MF) reinforcement learning unit and the model-based (MB) reinforcement learning unit, An operation selection probability may be provided through the operation value and converted into an operation.

According to embodiments, meta-reinforcement that dynamically applies control weights to a model-based (MB) reinforcement learning (RL) method related to various task goals and a plurality of model-free reinforcement learning (RL) methods. A system and method for estimating human behavior patterns and behavior strategies using learning can be provided.

According to embodiments, human behavior patterns and behavior strategies can be estimated through a model that implements competition between a model-based reinforcement learning (RL) method and a plurality of goal-dependent model-free reinforcement learning (RL) methods. In real life, you can apply goal-driven reinforcement learning (RL) control to various human-robot interaction scenarios.

1 is a diagram for explaining inference of a behavioral strategy and decision-making of a human according to an exemplary embodiment.
FIG. 2 is a diagram illustrating a two-stage Markov decision-making task of varying complexity, according to an exemplary embodiment.
3 is a diagram showing the configuration of the upper two models according to an embodiment.
4 is a diagram showing an excess probability for 57 models according to an embodiment.
5 is a block diagram illustrating a system for estimating human behavior patterns and behavior strategies using meta-reinforcement learning according to an exemplary embodiment.
6 is a flowchart illustrating a method of estimating human behavior patterns and behavior strategies using meta-reinforcement learning according to an exemplary embodiment.
7 is a diagram for describing a task design according to an embodiment.
8 is a diagram for explaining a computer model of arbitration control incorporating uncertainty and complexity according to an embodiment.

Hereinafter, embodiments will be described with reference to the accompanying drawings. However, the described embodiments may be modified in various forms, and the scope of the present invention is not limited by the embodiments described below. In addition, various embodiments are provided to more completely explain the present invention to those of ordinary skill in the art. In the drawings, the shapes and sizes of elements may be exaggerated for clearer explanation.

Recent evidence in the fields of neuroscience and psychology is that in an uncertain and dynamic environment where the uncertainty of the state-action-state transition changes over time, a single reinforcement learning algorithm is not limited to the variance of human selection behavior. This suggests that it only accounts for less than 60%. The prediction performance further decreases as the size of the state-space increases.

The present invention is a hierarchical context-dependent method that dynamically applies control weights to a model-based (MB) reinforcement learning (RL) method and a number of model-free reinforcement learning (RL) methods related to various task goals. We propose a (dependent) reinforcement learning control framework (hierarchical context-dependent RL control framework). In order to properly evaluate the effectiveness of the proposed method, we consider a two-stage Markov decision task (MDT) in which three different types of context change over time. Here, we trained 57 different reinforcement learning (RL) control models with the Caltech decision-making task (MDT) dataset and then compared their prediction performance using Bayesian model comparison, a model that provides accurate predictions. Is a model that implements competition between model-based reinforcement learning (RL) methods and multiple goal-dependent model-free reinforcement learning (RL) methods. According to embodiments, it is possible to apply goal-driven reinforcement learning (RL) control to various interaction scenarios between humans and robots in real life.

1 is a diagram for explaining inference of a behavioral strategy and decision-making of a human according to an exemplary embodiment.

Referring to FIG. 1, a system for estimating a human behavior pattern and a behavior strategy using meta-reinforcement learning according to an embodiment may estimate 140 a behavior strategy and a decision-making inherent in human decision-making. Embodiments describe the human behavior 110 through a plurality of target-dependent addictive behavior agents, so that the human behavior 110 may be predicted in accordance with various changes in circumstances. In predicting the human behavior 110, it is important how much the environment is assumed to be real, and since the environment includes various changes in circumstances, it is more applicable.

Reinforcement learning can be used in the human decision-making process. Reinforcement learning has an advantage in inferring human behavior 110 in an uncertain environment. Many studies aim to observe human reinforcement learning and predict behavior in most cases in simple environments. However, in real life, the environment in which humans have to make choices is not simple, but very complex, and undergoes constant environmental changes (variable environment). There are three major changes that can be predicted within a variable environment. The first is a change in the complexity (complexity) of the situation, and the number of possible choices changes. The second is the uncertainty change, and the prediction of the outcome of the choice becomes inaccurate. The third is goal change, and the goal you want to take is different as a result of your choice.

Examples are humans using meta-reinforcement learning that not only predicts human decision making using a control unit (meta-controller) that comprehensively controls various reinforcement learning agents, but also estimates 140 action strategies involved in the decision making process. A system for estimating behavior patterns and behavior strategies can be provided, and a system for estimating human behavior patterns and behavior strategies using meta-reinforcement learning can cope with various environmental changes and can be adaptively changed by a number of reinforcement learning agents. .

The reinforcement learning model is a model that is the basis of human decision-making, and it has one goal-directed behavior agent 120 and a number of goal-dependent habits to respond to various goals. It may be made of an action agent 130. The reason why the goal-dependent addictive behavior agent 130 is composed of a plurality of addictive behavior agents 131 is to easily and efficiently respond to the purpose that the user seeks. This is particularly useful in a variable environment.

The control unit can predict human decision-making behavior by determining the number of agents to increase the utilization of each agent based on the performance of each agent.

A system and method for estimating human behavior patterns and behavior strategies using meta-reinforcement learning according to an exemplary embodiment will be described in more detail below.

The present invention is a Caltech acquired through a more complex decision-making task (MDT) using three varying task variables, the goal value, the size of the state-space, and the degree of uncertainty of the state-transition. (Caltech) Consider the reinforcement learning dataset. In particular, embodiments study the influence of changes in the size and target value of the state-space in the hierarchical control reinforcement learning (RL) model. This effect has not been sufficiently studied in previous studies (Non-Patent Document 1). In order to prove the influence of factors that can affect hierarchical control in the complex decision making process (MDP) setting, a total of 57 hierarchical control reinforcement learning (RL) models were compared, each combining different hypotheses. Implemented. As a result, we discovered a new hierarchical controlled reinforcement learning (RL) model that provides highly accurate predictions of choice behavior in a realistic decision-making process (MDP) environment. Simulation results show that there is a large amount of hierarchical control over model-based (MB) reinforcement learning (RL) agents and goal-dependent model-free (MF) reinforcement learning (RL) agents in explaining human selection behavior. Show the advantages.

Below, we provide a hierarchical control architecture that integrates the three variable environment variables described above.

FIG. 2 is a diagram illustrating a two-stage Markov decision-making task of varying complexity, according to an exemplary embodiment.

Referring to FIG. 2, a two-step Markov decision-making task (MDT) using three variable task variables of a target value 210, a state-space size 220, and a state-transition uncertainty degree 230 is described. can do. Examples use a Caltech reinforcement learning dataset obtained through a two-step Markov decision-making task. The dataset consists of data from 22 participants (10 females, 19-55 years old). Participants can make two sequential choices 201, 202 while the three task variables 210, 220, 230 change. Participants are informed about the target value 210 of the color coin and can choose 201, 202 from two possible options to maximize the cumulative reward. The state transition occurs according to the participant's choices 201 and 202, and the degree of state-transition uncertainty 230 may vary between a low state and a high state. As shown in Stage 2 of FIG. 1, the number of possible options, meaning the size 220 of the state-space, may vary between two and four. The task's reward policy can encourage participants to learn the optimal policy.

Below, a hierarchical controlled reinforcement learning (RL) model will be described. Here, the hierarchical control reinforcement learning (RL) model may be a system for estimating human behavior patterns and behavior strategies using meta-reinforcement learning according to an embodiment. In addition, the hierarchical controlled reinforcement learning (RL) model may be included in a system for estimating human behavior patterns and behavior strategies using meta-reinforcement learning.

The hierarchical control reinforcement learning (RL) model can hierarchically control model-based (MB) agents and model-free (MF) agents. A major component of a hierarchical controlled reinforcement learning (RL) model is a reliability-based competition between model-based (MB) reinforcement learning (RL) and model-free (MF) reinforcement learning (RL). In this hierarchical controlled reinforcement learning (RL) model, the control unit reinforces each reinforcement learning (RL) through a transition rate in two directions (MB to MF, MF to MB) that is a function of the reliability of each agent. The weight of the learning (RL) agent can be updated.

Table 1 is Algorithm 1 representing a hierarchical controlled reinforcement learning (RL) model.

[Table 1]

Table 2 is Algorithm 2 indicating weight update for hierarchical control.

[Table 2]

This dynamic two-stage transition model was borrowed from biophysics to realize competition between two types of reinforcement learning (RL) (Non-Patent Document 2).

However, since we used a two-stage decision-making task (MDT) with three changing task variables to interpret the general human reinforcement learning (RL) in complex situations, the model is model-based (MB) reinforcement. It will require more than a control over learning (RL) and model-free (MF) reinforcement learning (RL). There are two key component complexity and currency associated with task variables, and there are several hypotheses for each component.

First, complexity (the size of the state-space) represents a binary change to the size of the state space, and can be used to formulate the influence of the current state space size in hierarchical control. As the state space changes, the weights of each reinforcement learning (RL) system and the corresponding optimal policy must also be updated.

는 복잡성에 대한 기여, 다시 말해 이전 연구에서 언급된 천이 확률(transition rate)이 없다는 것을 의미한다(비특허문헌 1). 다른 3가지의 천이 확률(알고리즘 2의

-

)들은 알고리즘 2의 line 3 - 9에 기재된 바와 같이 분자 또는 분모 상에서 복잡성에 대한 기여를 한다. 이 3가지 천이 확률에서, 각각에 대한 6개의 변형이 있으며, 이는 3가지 방법(양방향 복잡성 기여 / MB에서 MF 방향 기여/ MF에서 MB 방향 기여) 및 2가지 방향(순/역방향 복잡성, 즉 상태 공간이 확장할 때 복잡성 항 z가 증가/감소하는 것)으로 적용될 수 있다. Four different complexity effects are assumed here on hierarchical control (Algorithm 2). In this algorithm 2,

Means that there is no contribution to complexity, that is, the transition rate mentioned in the previous study (Non-Patent Document 1). The other 3 transition probabilities (Algorithm 2's

-

) Contribute to the complexity on the numerator or denominator as described in lines 3-9 of Algorithm 2. In these 3 transition probabilities, there are 6 variants for each, which are 3 methods (bidirectional complexity contribution / MB to MF direction contribution / MF to MB direction contribution) and 2 directions (forward/reverse complexity, i.e. state space). This can be applied as the complexity term z increases/decreases when expanding.

3 is a diagram showing the configuration of the upper two models according to an embodiment.

As shown in FIG. 3, for example, the upper model has only a complexity effect on the MF -> MB transition probability, which has a forward complexity term.

Next, the currency (goal value) represents the current target value of the coin, and should be reflected in the hierarchical control reinforcement learning (RL) model. Here, this concept is borrowed from multi-agent reinforcement learning (RL), which is frequently used in environments that require multiple optimal policies due to multiple goals. Since the target value in a task according to an embodiment changes, it is more target dependent than a single model-based (MB) reinforcement learning (RL) that already has a backward planning process and is less related to target change. It is intuitive to use model-free (MF) reinforcement learning (RL).

Therefore, we test two hypotheses for goal-dependent agents (3MF model and 3Q model), and one null hypothesis with only one model-free (MF) (1MF model). ) Has been verified. In the 3MF model, unlike the 3Q model, there are three separate model-free (MF) agents for three coins, and these are each action value for each coin and a single model-free (MF) reinforcement learning ( RL) can share agents.

Thus, in Algorithm 1, the 3MF model has 3 model-free (MF) reinforcement learning (RL) agents (line 5-7), and the 3Q model can have 3 Q _MFs in lines 5 and 6. Two target dependent agent models can only update one model-free (MF) reinforcement learning (RL) agent (or action value). Before line 5 starts, the target-dependent agent model can selectively update only the action value (3Q model), or update the action value and the corresponding reliability of each goal-dependent model-free (MF) agent according to the maximum call of each attempt. Yes (3MF model).

4 is a diagram illustrating an excedance probability for 57 models according to an exemplary embodiment.

(양방향 / MB에서 MF 방향 / MF에서 MB 방향)에 대한 기여의 개수, 목표 의존적 모델-프리(MF) 에이전트 구성의 개수(3MF / 3Q / 1MF 모델), 천이 확률에 대한 가설의 개수(

/

/

) 및 복잡성 항 방향의 개수(순방향 / 역방향)의 곱이다. 또한, 3개의 null 모델이 있으며, 이는 오직 목표 의존적 모델-프리(MF) 에이전트의 구성만을 구비하고 있다(1MF-null 모델은 이전 연구(비특허문헌 1)의 모델과 동일하다). 향후, 모델의 세부 사항은 목표 의존적 모델-프리(MF) 에이전트의 구성 및 복잡성 항

에 대한 기여를 제외하고는, 도 4(m1에서 m9)에 도시된 바와 같이 축약 표현될 것이다. 예를 들어, 역방향 복잡성과 3MF 모델을 갖는 m9는 m9-reverse 3MF로 축약될 것이다.3*3*3*2 = There are 54 hierarchical controlled reinforcement learning (RL) models, which in terms of complexity

(Bidirectional / MB to MF direction / MF to MB direction) number of contributions, number of target-dependent model-free (MF) agent configurations (3MF / 3Q / 1MF model), number of hypotheses for transition probability (

/

/

) And the number of directions in the complexity term (forward/reverse). In addition, there are three null models, which have only the configuration of target dependent model-free (MF) agents (the 1MF-null model is the same as the model of the previous study (Non-Patent Document 1)). In the future, the details of the model will be in terms of the composition and complexity of the target-dependent model-free (MF) agent.

Except for the contribution to, it will be abbreviated as shown in Fig. 4 (m1 to m9). For example, m9 with reverse complexity and 3MF model would be abbreviated to m9-reverse 3MF.

5 is a block diagram illustrating a system for estimating human behavior patterns and behavior strategies using meta-reinforcement learning according to an exemplary embodiment.

Referring to FIG. 5, a system 500 for estimating human behavior patterns and behavior strategies using meta-reinforcement learning according to an embodiment includes a model-free (MF) reinforcement learning unit 510, a model-based -based, MB) It may include a reinforcement learning unit 520 and a hierarchical control unit 530.

The model-free (MF) reinforcement learning unit learns a reward function, and can dynamically interact with the model-based (MB) reinforcement learning unit. In particular, the model-free (MF) reinforcement learning unit may reflect a target value composed of target-dependent multi-model-free (MF) reinforcement learning agents in order to provide a plurality of policies according to a plurality of goals.

The model-based (MB) reinforcement learning unit learns a reward function and a state-transition function, and can dynamically interact with the model-free (MF) reinforcement learning unit.

The hierarchical control unit can predict human decision making by hierarchically controlling the weights dynamically to the model-free (MF) reinforcement learning unit and the model-based (MB) reinforcement learning unit. In addition, the hierarchical control unit can estimate the behavioral strategy inherent in human decision-making.

This hierarchical control unit includes at least one of a model-free (MF) reinforcement learning unit and a model-based (MB) reinforcement learning unit, which are variable task variables, a goal value, a state-space size, and a state-transition uncertainty. By controlling hierarchically by the above, human decision making can be predicted.

And, the hierarchical control unit is model-free (MF) reinforcement learning through transition rates in two directions that are functions of the reliability of the model-free (MF) reinforcement learning unit and the model-based (MB) reinforcement learning unit. The weights of the sub and model-based (MB) reinforcement learning units can be updated.

The model-free (MF) reinforcement learning unit or the model-based (MB) reinforcement learning unit receives the variable task variable target value, state-space size, and state-transition uncertainty from a variable environment. The transition probability between the free (MF) reinforcement learning unit and the model-based (MB) reinforcement learning unit may be calculated, and the hierarchical control unit may determine a model selection probability through the transition probability.

Thereafter, the hierarchical control unit calculates the motion value by integrating the model selection probability of the model-free (MF) reinforcement learning unit and the model-based (MB) reinforcement learning unit, and converts it into an action by providing the motion selection probability through the motion value. can do.

6 is a flowchart illustrating a method of estimating a human behavior pattern and a behavior strategy using meta-reinforcement learning according to an exemplary embodiment.

6, a method for estimating a human behavior pattern and a behavior strategy using meta-reinforcement learning according to an exemplary embodiment dynamically applies weights to a model-free (MF) reinforcement learning unit and a model-based (MB) reinforcement learning unit. Including the step of giving (S110), and predicting human decision making by hierarchically controlling the model-free (MF) reinforcement learning unit and the model-based (MB) reinforcement learning unit according to the assigned weight (S120). It can be done by doing.

The model-free (MF) reinforcement learning unit and the model-based (model-free (MF) reinforcement learning unit and model-based ( MB) The step of updating the weight of the reinforcement learning unit (S130) may be further included.

Each step of a method of estimating a human behavior pattern and a behavior strategy using meta-reinforcement learning according to an exemplary embodiment will be described below. The method of estimating human behavior patterns and behavior strategies using meta-reinforcement learning according to an embodiment may be described in more detail through the system for estimating human behavior patterns and behavior strategies using meta-reinforcement learning according to the embodiment described with reference to FIG. 5. The system for estimating human behavior patterns and behavior strategies using meta-reinforcement learning according to an embodiment may include a hierarchical control unit, and according to an embodiment, a model-free (MF) reinforcement learning unit, a model-based (model-based, MB) may be further included in the reinforcement learning unit.

In step S110, the hierarchical control unit may dynamically assign a weight to the model-free (MF) reinforcement learning unit and the model-based (MB) reinforcement learning unit. On the other hand, the model-free (MF) reinforcement learning unit may reflect a target value composed of target-dependent multiple model-free (MF) reinforcement learning agents in order to provide a plurality of policies according to a plurality of goals.

In step S120, the hierarchical control unit may predict human decision making by hierarchically controlling a model-free (MF) reinforcement learning unit and a model-based (MB) reinforcement learning unit according to the assigned weight. In addition, the hierarchical control unit can estimate the behavioral strategy inherent in human decision-making.

This hierarchical control unit includes at least one of a model-free (MF) reinforcement learning unit and a model-based (MB) reinforcement learning unit, which are variable task variables, a goal value, a state-space size, and a state-transition uncertainty. By controlling hierarchically by the above, human decision making can be predicted.

The hierarchical control unit is a model-free (MF) reinforcement learning unit or a model-based (MB) reinforcement learning unit, which is a variable task variable from a variable environment, the goal value, the size of the state-space and the uncertainty of the state-transition. The model selection probability may be determined by using the result of calculating the transition probability between the model-free (MF) reinforcement learning unit and the model-based (MB) reinforcement learning unit by receiving the received.

In addition, the hierarchical control unit calculates the motion value by integrating the model selection probability of the model-free (MF) reinforcement learning unit and the model-based (MB) reinforcement learning unit, and provides the motion selection probability through the motion value to convert it I can.

In step S130, the hierarchical control unit uses the model-free (MF) reinforcement learning unit and the model-based (MB) reinforcement learning unit through a transition rate in two directions that is a function of the reliability. MF) The weights of the reinforcement learning unit and the model-based (MB) reinforcement learning unit can be updated.

Hereinafter, a system and method for estimating human behavior patterns and behavior strategies using meta-reinforcement learning according to an embodiment will be described in more detail.

7 is a diagram for describing a task design according to an embodiment.

(가장 높은 엔트로피 케이스)일 때, 원하는 상태로의 성공적인 천이 확률은 0.5를 초과할 수 없다.7A shows a second-stage Markov decision making task. More specifically, after a transition by a specific state-transition probability p, participants make a choice from 2 to 4 options (e.g., the participants' left (L) or right (R) behavior choices). Through this, participants move from one state 710 to another state 711. The probability of a successful transition to the desired state is proportional to the estimated accuracy of the state-transition probability, which is limited by the actual probability distribution entropy of the state-transition. For example,

In the (highest entropy case), the probability of a successful transition to the desired state cannot exceed 0.5.

및

에 대응된다. 721은 선택 이용 가능성을 나타내며, 낮은 상태-공간 복잡성 조건 및 높은 상태-공간 복잡성 조건은 각각 2개의 선택 및 4개의 선택이 이용 가능한 케이스에 대응한다. 첫 번째 상태에서 오직 2개의 선택만이 항상 이용 가능하며, 다음 상태에서 복잡성 조건에 따라 2개 또는 4개의 옵션이 이용 가능하다.7(b) describes the experimental conditions. 720 represents the state-transition probability, and the low state-transition uncertainty condition and the high state-transition uncertainty condition are the state-transition probability, respectively.

And

Corresponds to 721 indicates the availability of a selection, and the low state-space complexity condition and the high state-space complexity condition correspond to cases in which two and four options are available, respectively. In the first state only 2 options are always available, in the next state 2 or 4 options are available depending on the complexity condition.

Referring to (c) of FIG. 7, participants can make two sequential choices to obtain different color coins (silver, blue, and red) 730 whose values change over all trials. In each trial, participants are communicated for the "currency", that is, the current value of each coin 730. In each of the next two states (represented by a fractal image), they can make a selection by pressing one of the available buttons (L1, L2, R1, R2). Optional availability information 731 may be presented at the bottom of the screen. Here, a bold gray circle 732 and a light gray circle 733 represent an available choice and an unavailable choice, respectively.

7D illustrates a task, and each gray circle represents a state. Each of the dark arrows and solid lines represents the participants' choices (L ₁ , L ₂ , R ₁ , R ₂ ) and the subsequent state-transition according to the state-transition probability. Each result state (states 4-11) is associated with a reward (no coins represented by color coins or gray mosaic images). The reward probability is 0.8.

In order to examine the role of uncertainty and complexity in arbitration control, a new two-stage Markov decision-making task in which the degree of uncertainty and complexity is changed as shown in Fig.7(a). (MDP), where, as shown in Fig.7(b), the two task variables are systematically manipulated across state-transition uncertainty and state-space complexity in blocks of trial. I can. State-transition uncertainty can be controlled using state-action-state transition probabilities. The state-action-state transition probability can vary between the two states: high uncertainty (0.5 vs 0.5) and low uncertainty (0.9 vs 0.1).

The transition between these two uncertain states can be designed to change the average amount of state prediction errors (SPEs) and then effectively transform them into the reliability of a model-based (MB) system. For example, a high uncertainty state will lead to a large amount of state prediction error (SPE), which essentially leads to a decrease in model-based (MB) prediction performance. In a state of low uncertainty, the state prediction error (SPE) will decrease or remain low on average because the model-based (MB) improves the state-action-state transition probability estimation. On the other hand, the performance of model-free (MF) is less affected by the degree of state-transition uncertainty.

The second variable, the number of available choices, is intended to manipulate task complexity. The total number of available choices is 2 and 4 in the low complexity state and the high complexity state, respectively. To prevent the state-space expression from becoming too complex, here we limit the number of available choices to two in the first step of each trial, and to two or four in the second step. Manipulating the availability of choices increases the number of ways to achieve each goal, which makes the difficulty of each attempt range from easy to very difficult. Thus, this design can provide four different state types (low/high x uncertainty/complexity). Participants can make sequential choices to earn different color coins.

As shown in (c) of FIG. 7, another feature of the decision making process (MDP) is that in each attempt, participants can take action to obtain three different coins, silver, red or blue coins. In each trial, the relative value of the coin was flexibly allocated with respect to each coin's exchange rate against the real life currency (US cent), which was revealed at the beginning of each trial.

For example, if you win a particular attempt, a silver coin will yield 1 US cents, a red coin will yield 9 US cents, and a blue coin will yield 3 cents, and this allocation may vary in the next attempt. The characteristic of this design is to induce the change of the target value for each trial, to induce the deviation of the compensation prediction error, and to induce the reliability of the model-free (MF) in several trials.

Twenty-four adult participants (20 females, 19-55 years old) performed the task, and 22 of them had functional magnetic resonance imaging (fMRI) scans. The subject's task ability in terms of the total amount of rewards obtained and the ratio of optimal choices is significantly greater than the chance level under all conditions (t-test; p<1e-5).

8 is a diagram for describing a computer model of arbitration control incorporating uncertainty and complexity according to an embodiment.

Referring to FIG. 8, a computer model of interventional control incorporating uncertainty and complexity can be described, wherein the computer model of interventional control incorporating uncertainty and complexity is a human behavior pattern using meta-reinforcement learning according to an embodiment, and It may be a behavioral strategy estimation system. In addition, a computer model of interventional control that integrates uncertainty and complexity can be included in a system for estimating human behavior patterns and behavior strategies using meta-reinforcement learning.

The circle and arrow shapes represent a two-state dynamic transition model, where the current state is dependent on the previous state (endogenous variable) and the input from the environment (exogenous variables). This environmental input includes a state-transition that results in a state prediction error (SPE), and a compensation and task complexity that results in a reward prediction error (RPE). The arrows represent the transition probability from model-based (MB) reinforcement learning (RL) to model-free (MF) reinforcement learning (RL) or vice versa, which are state prediction error (SPE), compensation prediction error ( RPE) and task complexity. The circle shape may represent a state defined by the probability (P _MB ) of choosing model-based (MB) reinforcement learning (RL). Q(s,a) may represent a value of an operation (a) currently available in the current state (s). This value is converted to a subsequent operation, and can be expressed as an operation selection probability P(a|s).

In a computer model for interventional control incorporating uncertainty and complexity according to an embodiment, the dynamic interaction between model-free (MF) reinforcement learning (RL) and model-free (MF) reinforcement learning (RL) is a two-state It is based on the transition model. This type of regression structure has already been reviewed to better explain the intervention process between model-based (MB) reinforcement learning (RL) and model-free (MF) reinforcement learning (RL) in action and fMRI data.

In this model, each state is dependent on a previous state (endogenous input) and an input from the environment (exogenous variables) such as state, reward, and perceived task complexity. In this model, the preference for model-based (MB) reinforcement learning (RL) and model-free (MF) reinforcement learning (RL), expressed as model selection probabilities (P _MB ), is a function of prediction uncertainty and task complexity. . The prediction uncertainty represents the estimated uncertainty for the state-action-state transition and complexity.

It is calculated based on state prediction error (SPE) and compensation prediction error (RPE), which are key variables for model-based (MB) training and model-free (MF) training, respectively. It is specifically assumed here that task complexity affects the transition between model-free (MF) and model-free (MF). When the environment is completely stable (i.e., fixed state-transition uncertainty and a fixed level of task complexity), the details of this model are model-based (MB) reinforcement learning (RL) and model-free (MF) reinforcement. It can converge with a stable mix of learning (RL).

The process of the computer model according to the embodiment is described as follows.

MF, MF

MB)을 계산하는데 사용되며, 차후에 모델 선택 확률 P_MB을 결정할 수 있다.First, depending on the agent's behavior at each attempt, the environment can provide the model with state-action-state transitions, coin values, and task complexity. These are the transition probabilities (MB

MF, MF

MB), and can later determine the model selection probability P _MB .

Second, the model can calculate the overall integrated motion value (Q(s,a) in Fig. 8) by integrating the model-based (MB) value estimation and the model-free (MF) value estimation, which is a later operation. Can be converted (operation selection probability P(a|s) in FIG. 8). This framework can be used to formally implement various hypotheses about the impact of uncertainty and complexity on reinforcement learning (RL). For example, the construction of a model that well describes the subject's choice behavior is that people use model-based (MB) reinforcement learning (RL) and model-free (MF) reinforcement learning (RL) to account for the uncertainty and complexity of the environment. You can combine them to specify how to coordinate their behavior.

As described above, the variable environment referred to in the present invention reflects the real life of humans, and can be widely applied to predicting human decision-making that occurs accordingly. In particular, predicting human behavior is helpful in improving UX (user experience) for human convenience in terms of software. The embodiments can be applied in various ways as follows.

For example, the navigation system in the first city visited may provide additional information by estimating the user's behavioral intention. As another example, it can be used to help users adapt in situations such as an unattended system replacing an existing administrative process, such as a document issuing procedure, or increasing functions. In addition, when new IoT devices are added or functions of existing IoT devices are added in the Internet of things (IoT) infrastructure, human behavior can be predicted to give intuition in adaptation and utilization. In addition, when a user with an existing product purchases the latest product, it is possible to quickly learn about additional functions in the new product.

The apparatus described above may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the devices and components described in the embodiments include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It can be implemented using one or more general purpose computers or special purpose computers, such as a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For the convenience of understanding, although it is sometimes described that one processing device is used, one of ordinary skill in the art, the processing device is a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to behave as desired or processed independently or collectively. You can command the device. Software and/or data may be interpreted by a processing device or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. Can be embodyed in The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of the program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

As described above, although the embodiments have been described by the limited embodiments and drawings, various modifications and variations are possible from the above description by those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as a system, structure, device, circuit, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and claims and equivalents fall within the scope of the claims to be described later.

Claims (14)

A model-free (MF) reinforcement learning unit that learns a reward function;
A model-based (MB) reinforcement learning unit that learns the reward function and a state-transition function and dynamically interacts with the model-free (MF) reinforcement learning unit; And
A hierarchical control unit that predicts human decision-making by controlling hierarchically by dynamically assigning weights to the model-free (MF) reinforcement learning unit and the model-based (MB) reinforcement learning unit
Containing, human behavioral pattern and behavioral strategy estimation system using meta-reinforcement learning. The method of claim 1,
The hierarchical control unit,
The model-free (MF) reinforcement learning unit and the model-free (MF) reinforcement learning unit and the model-free (MF) reinforcement learning unit through transition rates in two directions that are functions of reliability of the model-based (MB) reinforcement learning unit Updating the weights of a model-based (MB) reinforcement learning unit
Characterized in, a system for estimating human behavior patterns and behavior strategies using meta-reinforcement learning. The method of claim 1,
The hierarchical control unit,
Predicting the human decision making, and estimating the behavioral strategy inherent in the human decision making
Characterized in, a system for estimating human behavior patterns and behavior strategies using meta-reinforcement learning. The method of claim 1,
The hierarchical control unit,
The model-free (MF) reinforcement learning unit and the model-based (MB) reinforcement learning unit are determined by at least one of a variable task variable, a goal value, a state-space size, and a state-transition uncertainty. Hierarchical control to predict human decision-making
Characterized in, a system for estimating human behavior patterns and behavior strategies using meta-reinforcement learning. The method of claim 4,
The model-free (MF) reinforcement learning unit,
Reflecting the target value, consisting of target-dependent multiple model-free (MF) reinforcement learning agents to provide multiple policies according to multiple goals
Characterized in, a system for estimating human behavior patterns and behavior strategies using meta-reinforcement learning. The method of claim 4,
The model-free (MF) reinforcement learning unit or the model-based (MB) reinforcement learning unit,
The model-free (MF) reinforcement learning unit and the model-based (MB) reinforcement learning by receiving the variable task variables target value, state-space size, and state-transition uncertainty from a variable environment Calculate the probability of transition between negatives,
The hierarchical control unit,
Determining the model selection probability through the transition probability
Characterized in, a system for estimating human behavior patterns and behavior strategies using meta-reinforcement learning. The method of claim 6,
The hierarchical control unit,
Integrating the model selection probabilities of the model-free (MF) reinforcement learning unit and the model-based (MB) reinforcement learning unit to calculate a motion value, and converting into motion by providing a motion selection probability through the motion value
Characterized in, a system for estimating human behavior patterns and behavior strategies using meta-reinforcement learning. Dynamically assigning weights to a model-free (MF) reinforcement learning unit and a model-based (MB) reinforcement learning unit in a hierarchical control unit of a human behavior pattern and behavior strategy estimation system using meta-reinforcement learning; And
Predicting human decision making by hierarchically controlling the model-free (MF) reinforcement learning unit and the model-based (MB) reinforcement learning unit according to the weights given by the hierarchical control unit
Containing, a method of estimating human behavior patterns and behavior strategies using meta-reinforcement learning. The method of claim 8,
The model-free (MF) reinforcement learning unit and the model-free (MF) reinforcement learning unit and the model-free (MF) reinforcement learning unit through transition rates in two directions that are functions of reliability of the model-based (MB) reinforcement learning unit Updating the weight of a model-based (MB) reinforcement learning unit
A method for estimating human behavior patterns and behavior strategies using meta-reinforcement learning further comprising a. The method of claim 8,
The hierarchical control to predict human decision making,
Predicting the human decision making, and estimating the behavioral strategy inherent in the human decision making
Characterized in, a method of estimating human behavior patterns and behavior strategies using meta-reinforcement learning. The method of claim 8,
The hierarchical control to predict human decision making,
The model-free (MF) reinforcement learning unit and the model-based (MB) reinforcement learning unit are determined by at least one of a variable task variable, a goal value, a state-space size, and a state-transition uncertainty. Hierarchical control to predict human decision-making
Characterized in, a method of estimating human behavior patterns and behavior strategies using meta-reinforcement learning. The method of claim 11,
The model-free (MF) reinforcement learning unit,
Reflecting the target value, consisting of target-dependent multiple model-free (MF) reinforcement learning agents to provide multiple policies according to multiple goals
Characterized in, a method of estimating human behavior patterns and behavior strategies using meta-reinforcement learning. The method of claim 11,
The hierarchical control to predict human decision making,
In the model-free (MF) reinforcement learning unit or the model-based (MB) reinforcement learning unit, the variable task variable target value, state-space size, and state-transition uncertainty are determined from a variable environment. Determining the model selection probability by using the result of calculating the transition probability between the model-free (MF) reinforcement learning unit and the model-based (MB) reinforcement learning unit received
Characterized in, a method of estimating human behavior patterns and behavior strategies using meta-reinforcement learning. The method of claim 13,
The hierarchical control to predict human decision making,
Integrating the model selection probabilities of the model-free (MF) reinforcement learning unit and the model-based (MB) reinforcement learning unit to calculate a motion value, and converting into motion by providing a motion selection probability through the motion value
Characterized in, a method of estimating human behavior patterns and behavior strategies using meta-reinforcement learning.

Images