JP2022182593A - Reverse reinforcement learning device, method and program - Google Patents

Abstract

To provide a reverse reinforcement learning device, a method and a program for achieving efficiency of learning of a compensation scheme.SOLUTION: A reverse reinforcement learning device (10) comprises: a measures determination part (111) for determining measures (π) of a behavior on the basis of a locus (Sg) of a given behavior; and a compensation determination part (112) for determining a compensation (r) applied from an environment with respect to a behavior of an agent, so as to maximize an expected compensation (J) calculated from the measures (π) and a state worth (V). The compensation determination part (112) mixes the state worth (Vt) with an additional worth (Rt) being calculated using the compensation (r) so as to approximate the state worth (Vt), for calculating the state worth (V), and updates the compensation (r) so as to maximize the expected compensation (J) with respect to the calculated state worth (V).SELECTED DRAWING: Figure 1

Description

The present invention relates to an inverse reinforcement learning device, an inverse reinforcement learning method, and a program.

Conventionally, reinforcement learning is used to accomplish a given task. Reinforcement learning is a method of evaluating an agent's actions in an environment given a task based on the reward given from the environment, and learning a policy so as to maximize the cumulative reward of a series of actions. For example, reinforcement learning is applied to games, motor control, automatic driving control of vehicles, and the like (see Patent Documents 1 and 2).

Japanese Patent Application Laid-Open No. 2018-63602 JP 2020-144483 A

On the other hand, inverse reinforcement learning is used to model the skill of experts. Inverse reinforcement learning learns the reward system from the trajectory of the expert's actions, which action is evaluated and rewarded. However, reward systems are generally complex. There is a demand for a method that can efficiently learn a reward system that approximates the reward system of an expert.

An object of the present invention is to improve the efficiency of learning a reward system.

〔Ｅは、期待値を出力する関数を表す。π（ｓ｜ａ）は、状態（ｓ）における行動（ａ）を選択する方策（π）を表す。Ｖ（ｓ）は、状態（ｓ）を評価する状態価値（Ｖ）を表す。Ｖ_ｔは時間ｔのときの状態（ｓ）を評価する状態価値（Ｖ）を表す。Ｒ_ｔは、時間ｔのときの付加価値を表す。τはＲ_ｔを混合する割合を表し、０≦τ≦１を満たす。〕 One aspect of the present invention is an inverse reinforcement learning device (10) that determines a reward (r) given from the environment for an agent's action based on a given action trajectory (Sg). The inverse reinforcement learning device (10) includes a policy decision unit (111) that decides a course of action (π) from the trajectory (Sg), and a policy (π) and the environment a remuneration determination unit (112) that determines the remuneration (r) so as to maximize the expected remuneration (J) calculated by the state value (V) of . _The remuneration determination unit (112) approximates the state value (V t ) to the state value (V _t ) for evaluating the state (s) at time t, as shown in the following formula (2): and _the expected reward ( Update the reward (r) such that J) is maximized.

[E represents a function that outputs an expected value. π(s|a) represents a policy (π) that selects action (a) in state (s). V(s) represents a state value (V) that evaluates state (s). _Vt represents the state value (V) that evaluates the state (s) at time t. _Rt represents the added value at time t. τ represents the mixing ratio of _Rt and satisfies 0≦τ≦1. ]

Another aspect of the present invention is an inverse reinforcement learning method that determines a reward (r) given from the environment for an agent's action based on a given action trajectory (Sg). The inverse reinforcement learning method includes a step of determining a behavior policy (π) from the trajectory (Sg), and calculating the policy (π) and the state value (V) of the environment as shown in the above formula (1). determining said reward (r) such that it maximizes the expected reward (J) calculated by . The step of determining the reward (r) includes approximating the state value (V t ₎ to a state value (V _t ) that evaluates the state (s) at time t, as shown in equation (2) above. calculating the state value (V) by mixing the value added (R _t ) calculated with the reward (r) so that for the calculated state value (V) and updating said reward (r) such that said expected reward (J) is maximized.

Another aspect of the present invention causes a computer to execute an inverse reinforcement learning method for determining a reward (r) given from the environment for an agent's action based on a given action trajectory (Sg). It is a program for The inverse reinforcement learning method includes a step of determining a behavior policy (π) from the trajectory (Sg), and calculating the policy (π) and the state value (V) of the environment as shown in the above formula (1). determining said reward (r) such that it maximizes the expected reward (J) calculated by . The step of determining the reward (r) includes approximating the state value (V t ₎ to a state value (V _t ) that evaluates the state (s) at time t, as shown in equation (2) above. calculating the state value (V) by mixing the value added (R _t ) calculated with the reward (r) so that for the calculated state value (V) and updating said reward (r) such that said expected reward (J) is maximized.

ADVANTAGE OF THE INVENTION According to this invention, learning of a reward system can be made efficient.

It is a block diagram which shows the structure of the inverse reinforcement learning apparatus of this embodiment. It is a flow chart which shows inverse reinforcement learning processing. It is a figure which shows an example of the locus|trajectory of an expert's action. It is a figure explaining a state transition model.

An embodiment of an inverse reinforcement learning device, an inverse reinforcement learning method, and a program according to the present invention will be described below with reference to the drawings. The following description is an example (representative example) of the present invention, and the present invention is not limited thereto.

FIG. 1 shows the configuration of an inverse reinforcement learning device 10 according to one embodiment of the present invention.
The inverse reinforcement learning device 10 includes a CPU (Central Processing Unit) 11 and a storage unit 12 . The inverse reinforcement learning device 10 may further include an operation unit 13 , a display unit 14 and a communication unit 15 .

The CPU 11 reads out a program from the storage unit 12 and executes it, thereby executing reverse reinforcement learning processing, which will be described later. In the inverse reinforcement learning process, the CPU 11 functions as a policy determination section 111 and a reward determination section 112 .

The policy decision unit 111 decides a course of action from the given trajectory of the action of the expert. The remuneration determination unit 112 determines a remuneration based on the policy determined by the policy determination unit 111 and the state value of the environment so as to maximize the expected remuneration for a series of actions.

The storage unit 12 stores programs readable by the CPU 11, data used for executing the programs, and the like. As the storage unit 12, for example, a recording medium such as a hard disk can be used.

The operation unit 13 is a keyboard, mouse, or the like. The operation unit 13 receives a user's operation and outputs the content of the operation to the CPU 11 .

The display unit 14 is a display or the like. The display unit 14 displays an operation screen, a processing result of the CPU 11, and the like according to a display instruction from the CPU 11. FIG.

The communication unit 15 is an interface that communicates with an external computer via a network.

The inverse reinforcement learning device 10 can determine the reward (r) given from the environment by inverse reinforcement learning processing from the trajectory of the action of the expert to be imitated.
In this embodiment, reward (r) is defined as a neural network with parameters (θ) as shown in equation (4). Parameters are weights or biases set in the neural network.
(4) r=r(θ)

FIG. 2 is a flowchart of inverse reinforcement learning processing.
First, the policy determination unit 111 acquires a group of expert action trajectories (Sg) given along with the environment (step S1). The policy determination unit 111 may acquire the group of trajectories (Sg) from the storage unit 12 or from an external device on the network. A trajectory (Sg) is expressed as a set of environmental states (s) transitioned by a series of actions, as shown in Equation (5).
(5) Sg={( _s0 , _s1 ,..., _sn )}

FIG. 3 shows an example of an action trajectory (Sg).
A trajectory L1 exemplified in FIG. 3 is a route when the expert moves in the maze from the start point Ps to the goal point Pg. The maze consists of a multi-block area 30 in which one can move one block at a time. Movement can be blocked by walls placed between blocks. Here, area 30 is the given environment and each block corresponds to a state (s) of the environment.

Policy determination unit 111 generates a state transition model from this trajectory (Sg). A state transition model is the distribution of transition probabilities from one state(s) of the environment to the next state(s). For example, the state transition model is generated as a state transition matrix in which transition probabilities are tabulated. The policy determination unit 111 determines a policy (π) based on this state transition model (step S2). A policy (π) is the probability distribution of actions (a) that are chosen in each state (s).

FIG. 4 is a diagram for explaining the state transition model.
Blocks on trajectory L1 in the maze described above are worth state(s). In FIG. 4, a circle arranged in each block represents the value of the state (s), and the higher the density of the circle, the higher the value of the state (s). The policy determination unit 111 can determine the transition probability from each block (state) to the next block (state) so that the probability of transition to the block on the trajectory L1 is high.

Next, the reward determining unit 112 determines the reward (r) from the determined policy (π) so as to maximize the expected reward (J). Expected reward (J) refers to cumulative reward that can be expected to be obtained in one episode. An episode refers to a series of actions that transition from the initial state (s ₀ ) of the environment to the final state (s _e ). The expected reward (J) is calculated by policy (π) and state value (V) as shown in equation (1).

The above E[] represents a function that outputs the expected value in []. π(s|a) represents a policy (π) that selects action (a) in state (s). V(s) represents a state value (V) that evaluates state (s).

In this embodiment, the state value (V) is defined as shown in Equation (2). As shown in equation (2), the state value (V) is calculated by mixing the state value (V _t ), which evaluates the state (s) at time t, with the added value (R _t ). Value added (R _t ) is computed using reward (r) to approximate state value (V _t ) at time t, as shown in equation (3).

The above V _t represents the state value (V) that evaluates the state (s) at time t. _Rt represents the added value at time t. τ, τ _P , τ _D and τ _I each represent a coefficient of 0 or more and 1 or less. γ _E represents the discount rate of the reward (r) given to each action (a) and satisfies 0<γ _E ≦1. t _e represents the time in the final state (s _e ). In equation (3), the term including the coefficient _τP is called the proportional term, the term including the coefficient _τD is called the differential term, and the term including the coefficient _τI is called the integral term.

The r ^* multiplied by the factor τ _P in the proportional term represents the reward (r) given for the action (a) in the state (s) at time t during one episode. For example, when t=5, the reward determining unit 112 can use the reward (r) given for the action (a) in the state (s ₅ ) as the proportional term.

dr / dt multiplied by the coefficient τ _D in the differential term represents the differential value of the reward (r) given within a certain time from the state before time t to the state (s) at time t in one episode . By adding the differential term, the state value (V) can be determined considering the time variation of the reward (r). For example, when t=5, the reward determining unit 112 uses the differential value of the reward (r) given from the state (s ₃ ) to the state (s ₅ ) in steps 3 to 5 as the differential term. can be done.

The integral value of r multiplied by the coefficient _τI in the integral term represents the accumulated reward (r) given during one episode. In this accumulated value, the reward (r) for the action (a) in each state is discounted by the discount rate _γE . The discount rate γ _E is multiplied by (t _e −t), and the closer to the final state (s _e ), the smaller the discount rate of the reward (r).

When determining the reward (r), first, the reward determination unit 112 optimizes the added value (R _t ) in Equation (2) (step S3). Specifically, the remuneration determination unit 112 updates the remuneration (r) shown in Equation (4) by updating the parameter (θ) so as to maximize the expected remuneration (J). Optimizing added value (R _t ) is obtained by calculating added value (R _t ) using updated reward (r).

Next, the remuneration determining unit 112 optimizes the state value (V _t ) in Equation (2) with respect to the optimized added value (R _t ) (step S4). This optimization is done by using the updated reward (r) to update the state value (V _t ) as shown in equation (6).

s _t represents the state of the environment (s) at time t. s _t+1 represents the state ( _s ) one step after transitioning from the state (s t ). r _t+1 represents the reward (r) given from the environment according to the action (a) in the state (s _t ). α represents the learning rate and satisfies 0<α≦1. γ represents a discount rate and satisfies 0<γ≦1. max represents a function that outputs the maximum value among the state values (V) of the next state that can transition from the state (s _t+1 ).

If the expected reward (J) given in equation (1) does not converge after optimizing the state value (V _t ), then neither does the reward (r). In this case (step S5: NO), the remuneration determination unit 112 alternately repeats the optimization of the added value (R _t ) (step S3) and the optimization of the state value (V _t ) (step S4). Thereby, the reward (r) is optimized so that the expected reward (J) is maximized. When the expected reward (J) converges, so does the reward (r). In this case (step S5: YES), the inverse reinforcement learning process ends.

In this way, the reward determination unit 112 repeats updating the added value (R _t ), that is, updating the reward (r), and updating the state value (V _t ) using the updated reward (r). to optimize the reward (r). This iteration is possible because the state value (V _t ) term at time t and the reward (r) are used to approximate the state value (V _t ), as shown in equation (2). This is because the state value (V) is defined separately from the added value (R _t ) term. According to Equations (2) and (3), it is possible to easily calculate the reward (r) that maximizes the expected reward (J) and improve the efficiency of inverse reinforcement learning.

In the inverse reinforcement learning process, the reward determining unit 112 can adjust the ratio (τ) of mixing the added value (R _t ). For example, for equal mixing, τ=0.5 can be adjusted. To increase the ratio of the added value (R _t ) over the original state value (V _t ), adjustment should be made such that τ=0.7.

Preferably, the remuneration determination unit 112 reduces the ratio (τ) of mixing the added value (R _t ) as the number of updates of the remuneration (r) increases. The reward determination unit 112 can eventually reduce the ratio (τ) to zero. By reducing the ratio (τ), it is possible to shorten the learning time of the reward (r) and converge to the same result as when the added value (R _t ) is not used.

The remuneration determination unit 112 may monotonically decrease the ratio (τ), or may arbitrarily determine the extent to which the ratio (τ) is reduced with respect to the number of updates. Also, the remuneration determination unit 112 may temporarily increase the ratio (τ) in the process of decreasing it.

In addition, the remuneration determining unit 112 can adjust the proportions of the proportional term, the differential term, and the integral term by adjusting the respective coefficients τ _P , τ _D , and τ _I , and can set the proportion to zero. For example, by setting τ _P =0, τ _D =0, τ _I =1, only the integral term, ie the cumulative reward, can be blended into the state value (V _t ). If you want to emphasize the state (s) at time t, set τ _P =1 to add the proportional term, and if you want to consider the time change, set τ _D =1 to add the differential term. Just do it.

As described above, according to the present embodiment, the policy (π) is determined from the given trajectory (Sg), and the state value Update (V). At this time, as shown in formula (2), the state value (V _t ) at time t is mixed with the added value (R _t ) calculated using the reward (r) at a predetermined ratio (τ) calculates the state value (V).

This allows iterative updating of added value (R _t ) and updating of state value (V _t ) such that expected reward (J) is maximized. Since reward (r) is updated by updating value added (R _t ), reward (r) can be optimized by maximizing expected reward (J). Equation (2) facilitates calculation, and complex rewards (r) can be easily optimized, so that inverse reinforcement learning can be made more efficient.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments. Various modifications are possible within the scope of the invention.
For example, the added value (R _t ) is defined by Equation (3), but it is not limited to this as long as the state value (V _t ) can be approximated using the reward (r).

Since the state value (V) is calculated by the reward (r), dV/dt or V(s _t )−V(s) may be used instead of dr/dt in equation (3). dV/dt represents the differential value of the state value (V) within a certain period of time from the state before time t to the state (s) at time t. Also, V(s _t )−V(s) represents the change in state value (V) from the state before time t to the state (s) at time t.

Also, the reward (r) is defined by a neural network, but may be defined as a reward function approximated by a linear function.

The inverse reinforcement learning device 10 can be used in various technical fields, and the technical field is not particularly limited. For example, the reverse reinforcement learning device 10 can be used for automatic driving control for determining the vehicle's travel route while avoiding dangerous objects, motor drive control, game character control, and the like.

Also, a recording medium recording a program for causing a computer to execute the inverse reinforcement learning method of the present invention may be provided. The recording medium is not particularly limited as long as it can be read by a computer such as a CPU, and semiconductor memories, magnetic disks, optical disks, and the like can be used.

10... Reinforcement learning device, 11... CPU, 111... Policy determination unit, 112... Reward determination unit, 12... Storage unit

Claims (8)

In an inverse reinforcement learning device (10) that determines a reward (r) given from the environment for an agent's action based on a given action trajectory (Sg),
A policy decision unit (111) that decides a course of action (π) from the trajectory (Sg);
A reward that determines the reward (r) so as to maximize the expected reward (J) calculated by the policy (π) and the state value (V) of the environment, as shown in the following formula (1) a determining unit (112),
The remuneration determination unit (112)
Using the reward (r) so as to approximate the state value (V _t ) to the state value (V _t ) for evaluating the state (s) at time t, as shown in the following formula (2) calculating said state value (V) by mixing the calculated added value (R _t );
An inverse reinforcement learning device (10) for updating the reward (r) so as to maximize the expected reward (J) with respect to the calculated state value (V).

[E represents a function that outputs an expected value. π(s|a) represents a policy (π) that selects action (a) in state (s). V(s) represents a state value (V) that evaluates state (s). _Vt represents the state value (V) that evaluates the state (s) at time t. _Rt represents the added value at time t. τ represents the mixing ratio of _Rt and satisfies 0≦τ≦1. ] The reward determination unit (112), as shown in the following formula (3), provides the reward (r) given to the action (a) in the state (s) at time t in one episode, and the 1 The differential value of the reward (r) given from the state before the time t of the episode to the state (s) at the time t, and the accumulated value of the reward (r) given during the one episode The inverse reinforcement learning device (10) of claim 1, wherein the added value (R _{t ) is calculated by adding R t} .

[τ _P , τ _D and τ _I represent coefficients of 0 or more and 1 or less. r ^* represents the reward (r) given for action (a) in state (s) at time t. dr/dt represents the differential value of the reward (r) given from the state before time t to the state (s) at time t. γ _E represents a discount rate and satisfies 0<γ _E ≦1. t _e represents the time in the final state of one episode. ] The remuneration determination unit (112)
optimizing the added value (R _t ) by updating the reward (r) to maximize the expected reward (J);
optimizing the state value (V _t ) by updating the state value (V _t ) with the updated reward (r);
The inverse reinforcement learning device (10) according to claim 1 or 2, wherein updating the added value (R _t ) and updating the state value (V _t ) are repeated until the reward (r) converges. The inverse reinforcement learning device (10) according to any one of claims 1 to 3, wherein the reward determination unit (112) adjusts a ratio (τ) of mixing the added value (R _t ). The remuneration determination unit (112) reduces the ratio (τ) of mixing the added value (R _t ) as the number of updates of the remuneration (r) increases. An inverse reinforcement learning device (10). The remuneration determination unit (112)
defining the reward (r) as a neural network with parameters (θ),
The inverse reinforcement learning device (10) according to any one of claims 1 to 5, wherein the reward (r) is updated by updating the parameter (θ) so that the expected reward (J) is maximized ). In an inverse reinforcement learning method that determines the reward (r) given from the environment for the action of the agent based on the given action trajectory (Sg),
determining a course of action (π) from the trajectory (Sg);
A step of determining the reward (r) so as to maximize the expected reward (J) calculated by the policy (π) and the state value (V) of the environment, as shown in the following formula (1): and including
The step of determining the reward (r) comprises:
Using the reward (r) so as to approximate the state value (V _t ) to the state value (V _t ) for evaluating the state (s) at time t, as shown in the following formula (2) calculating said state value (V) by mixing the calculated value added (R _t );
updating said reward (r) such that said expected reward (J) is maximized with respect to said calculated state value (V).

[E represents a function that outputs an expected value. π(s|a) represents a policy (π) that selects action (a) in state (s). V(s) represents a state value (V) that evaluates state (s). _Vt represents the state value (V) that evaluates the state (s) at time t. _Rt represents the added value at time t. τ represents the mixing ratio of _Rt and satisfies 0≦τ≦1. ] A program for causing a computer to execute an inverse reinforcement learning method for determining a reward (r) given from the environment for an action of an agent based on a given action trajectory (Sg),
The inverse reinforcement learning method includes:
determining a course of action (π) from the trajectory (Sg);
A step of determining the reward (r) so as to maximize the expected reward (J) calculated by the policy (π) and the state value (V) of the environment, as shown in the following formula (1): and including
The step of determining the reward (r) comprises:

Using the reward (r) so as to approximate the state value (V _t ) to the state value (V _t ) for evaluating the state (s) at time t, as shown in the following formula (2) calculating said state value (V) by mixing the calculated value added (R _t );
updating said reward (r) such that said expected reward (J) is maximized with respect to said calculated state value (V).

[E represents a function that outputs an expected value. π(s|a) represents a policy (π) that selects action (a) in state (s). V(s) represents a state value (V) that evaluates state (s). _Vt represents the state value (V) that evaluates the state (s) at time t. _Rt represents the added value at time t. τ represents the mixing ratio of _Rt and satisfies 0≦τ≦1. ]

Images