JP7421391B2 - Learning methods and programs - Google Patents

Description

The present invention relates to a learning method and a program.

As a learning method for controlling an agent, there is a method that employs a dynamics model that takes uncertainty into consideration (see Non-Patent Document 1). Here, an agent is a behavioral entity that takes action against the environment.

K. Chua, R. Calandra, R. McAllister, and S. Levine. “Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In NeurIPS, 2018.”

However, there is room for improvement in learning methods for controlling agent behavior.

Therefore, the present invention provides a learning method for improving the behavior of an agent.

A learning method according to one aspect of the present invention is a learning method of agent behavior using model-based reinforcement learning, the learning method acquiring time-series data indicating the state and behavior of the agent when the agent acts. Then, a dynamics model is constructed by performing supervised learning using the acquired time series data, and based on the dynamics model, the action sequence of the agent is determined by variational inference using a mixture model as a variational distribution. A plurality of candidates are derived, and one candidate selected from the plurality of derived candidates is output as an action sequence of the agent.

Note that these comprehensive or specific aspects may be realized by a system, a device, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, and the system, device, integrated circuit, computer program and a recording medium may be used in any combination.

The learning method of the present invention can improve the learning method for controlling the behavior of an agent.

FIG. 1 is a block diagram showing the functional configuration of a learning device in an embodiment. FIG. 2 is an explanatory diagram showing a time series of agent states and actions in the embodiment. FIG. 3 is an explanatory diagram showing a variational distribution in the embodiment. FIG. 4 is an explanatory diagram showing the concept of variational inference performed by the inference unit in the embodiment. FIG. 5 is an explanatory diagram showing the concept of variational inference performed by the inference unit in the related technology. FIG. 6 is an explanatory diagram illustrating a plurality of candidates determined by the inference unit in the embodiment in comparison with a case in related technology. FIG. 7 is a flow diagram showing the learning method in the embodiment. FIG. 8 is an explanatory diagram illustrating temporal changes in cumulative rewards in the learning method according to the embodiment in comparison with related techniques. FIG. 9 is an explanatory diagram illustrating convergence values of cumulative rewards in the learning method according to the embodiment in comparison with related techniques.

A learning method according to one aspect of the present invention is a learning method of agent behavior using model-based reinforcement learning, the learning method acquiring time-series data indicating the state and behavior of the agent when the agent acts. Then, a dynamics model is constructed by performing supervised learning using the acquired time series data, and based on the dynamics model, the action sequence of the agent is determined by variational inference using a mixture model as a variational distribution. A plurality of candidates are derived, and one candidate selected from the plurality of derived candidates is output as an action sequence of the agent.

According to the above aspect, since variational inference is performed using a mixture model as a variational distribution, it is possible to derive a plurality of candidates for the action sequence of the agent. Each of the multiple behavioral sequence candidates derived in this way has a smaller difference from the optimal behavioral sequence than when performing variational inference using a single model as a variational distribution, and Convergence speed is fast when deriving multiple candidates. It is assumed that one of the plurality of derived behavioral sequence candidates is used as the agent's behavioral sequence. By doing so, a more appropriate action sequence can be output in a shorter time. In this way, according to the above aspect, it is possible to improve the learning method for controlling the agent.

For example, new time-series data indicating the state and behavior of the agent when the agent acts according to the output action sequence may be acquired as the time-series data.

According to the above aspect, since the state and behavior of the agent controlled using the action sequence output as a result of learning are used for new learning, the action sequence constructed as a result of learning is brought closer to an appropriate one. be able to. In this way, the learning method for controlling the agent can be further improved.

For example, when deriving the plurality of candidates, the mixture proportion of the probability distribution corresponding to each of the plurality of candidates in the entire mixture model is derived together with the plurality of candidates, and when selecting the one candidate, Of the plurality of derived candidates, a candidate corresponding to a probability distribution with the highest mixing ratio may be selected as the one candidate.

According to the above aspect, since the candidate corresponding to the probability distribution with the maximum mixing ratio in the mixture model is used as the agent's action sequence, the difference from the optimal action sequence can be further reduced. Therefore, according to the above aspect, it is possible to improve the learning method for controlling the agent so as to output an action sequence closer to the optimal action sequence.

For example, the mixture model may be a Gaussian mixture distribution.

According to the above aspect, since a Gaussian mixture distribution is specifically used as the mixture model, it is possible to more easily derive a plurality of candidates by variational inference. Therefore, the learning method for controlling the agent can be improved more easily.

For example, the dynamics model may be an ensemble of multiple neural networks.

According to the above aspect, since an ensemble of a plurality of neural networks is used as the dynamics model, it is possible to more easily construct a dynamics model with relatively high estimation accuracy. Therefore, the learning method for controlling the agent can be improved more easily.

Further, a program according to one aspect of the present invention is a program for causing a computer to execute the above learning method.

The above aspect provides the same effects as the learning method described above.

Note that these comprehensive or specific aspects may be realized by a system, a device, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, and the system, device, integrated circuit, computer program Alternatively, it may be realized using any combination of recording media.

Hereinafter, embodiments will be specifically described with reference to the drawings.

Note that the embodiments described below are all inclusive or specific examples. The numerical values, shapes, materials, components, arrangement positions and connection forms of the components, steps, order of steps, etc. shown in the following embodiments are merely examples, and do not limit the present invention. Further, among the constituent elements in the following embodiments, constituent elements that are not described in the independent claims indicating the most significant concept will be described as arbitrary constituent elements.

(Embodiment)
In this embodiment, a learning device and a learning method for improving the learning method for controlling agent behavior will be described.

FIG. 1 is a block diagram showing the functional configuration of a learning device 10 in this embodiment.

A learning device 10 shown in FIG. 1 is a device that uses model-based reinforcement learning to learn the behavior of an agent 20. The learning device 10 can acquire information when the agent 20 behaves, perform model-based reinforcement learning using the acquired information, and control the behavior of the agent 20.

The agent 20 is a device that selectively and sequentially assumes one of a plurality of states and selectively and sequentially performs one of a plurality of actions. The agent 20 is, for example, an industrial machine that is equipped with a robot arm and processes objects. In this case, the coordinate values of the robot arm, information regarding the target object, information indicating the processing status, etc. correspond to the above-mentioned "state". Further, information for operating the robot arm (coordinate values of the target position of the robot arm), information for changing the state of the industrial machine, etc. correspond to the above-mentioned "action".

The agent 20 provides the learning device 10 with time-series data indicating the state and behavior of the agent 20 when performing a series of actions. Further, the agent 20 can act according to the action sequence output by the learning device 10.

As shown in FIG. 1, the learning device 10 includes an acquisition section 11, a learning section 12, a storage section 13, an inference section 14, and an output section 15. Each functional unit included in the learning device 10 can be realized by a CPU (Central Processing Unit) (not shown) included in the learning device 10 executing a predetermined program using a memory (not shown).

The acquisition unit 11 is a functional unit that acquires time-series data indicating information about the agent 20 when the agent 20 acts. The information on the agent 20 that the acquisition unit 11 acquires includes at least the state and behavior of the agent 20. The sequence of actions of the agent 20 that the acquisition unit 11 acquires may be of any type, such as a randomly set sequence of actions, or a sequence of actions output by the output unit 15 (described later). Good too.

Then, the acquisition unit 11 acquires new time-series data indicating the state and behavior of the agent 20 as the above-mentioned time-series data by controlling the agent 20 according to the action sequence output by the output unit 15 (described later). Good too. The acquisition unit 11 acquires one or more of the above time series data.

The learning unit 12 is a functional unit that constructs the dynamics model 17 by performing supervised learning using the time series data acquired by the acquisition unit 11. The learning unit 12 stores the constructed dynamics model 17 in the storage unit 13.

The learning unit 12 uses a reward value determined according to the action sequence of the agent 20 when performing supervised learning. For example, if the agent 20 is an industrial machine equipped with a robot arm, a higher reward value is determined if the series of actions of the robot arm is more appropriate. Furthermore, the higher the quality of the object processed by the industrial machine, the higher the reward value is determined.

An example of the dynamics model 17 is an ensemble of multiple neural networks. In other words, the learning unit 12 acquires the same sequence of actions as the sequence of actions of the agent 20 acquired by the acquisition unit 11, or a part of the sequence of actions included in the sequence of actions of the agent 20 acquired by the acquisition unit 11. Coefficients (weights) of filters in each layer included in the plurality of neural networks are determined based on mutually different action sequences. In this case, the output of the ensemble of the plurality of neural networks is the result of aggregating the output data output by each of the plurality of neural networks with respect to the input data using an aggregation function. The aggregation function can include various operations, such as an operation that takes the average value of the output data of each neural network, or an operation that takes a majority vote of each output data.

The storage unit 13 is a storage device that stores the dynamics model 17 constructed by the learning unit 12. The dynamics model 17 is stored by the learning section 12 and read out by the inference section 14. The storage unit 13 is realized by a memory or a storage device.

The inference unit 14 is a functional unit that derives a plurality of candidates for the action sequence of the agent 20 based on the dynamics model 17. The inference unit 14 uses variational inference using a mixture model as a variational distribution when deriving a plurality of candidates for the action sequence of the agent 20. Thereby, the inference unit 14 derives a plurality of action sequences as the action sequences of the agent 20.

The mixture model that the inference unit 14 uses for variational inference is, for example, a Gaussian mixture distribution, and although this case will be described as an example, it is not limited to this. For example, a mixed categorical distribution is an example of a mixed model when the behavior sequence is a vector that takes discrete values.

The output unit 15 is a functional unit that outputs the action sequence of the agent 20. The output unit 15 outputs one candidate selected from among the plurality of candidates derived by the inference unit 14 as the action sequence of the agent 20. For example, it is assumed that the action sequence output by the output unit 15 is input to the agent 20, and the agent 20 acts according to this action sequence.

Note that the action series output by the output unit 15 may be managed as numerical data by another device (not shown).

Note that when deriving a plurality of candidates, the inference unit 14 may derive a mixture ratio of the probability distributions corresponding to each of the plurality of candidates in the entire mixture model together with the plurality of candidates. Then, when selecting the one candidate, the output unit 15 selects the candidate corresponding to the probability distribution with the maximum mixing ratio in the entire mixture model from among the derived candidates as the one candidate. You may.

FIG. 2 is an explanatory diagram showing an example of a time series of states of the agent 20 in this embodiment.

In FIG. 2, states S _t and S _t+1 are shown as an example of a series of states of the agent 20. Here, the state of the agent 20 at time t is defined as state S _t , and the state of the agent 20 at time t+1 is defined as S _t+1 .

Further, the action a _t shown in FIG. 2 indicates the action taken by the agent 20 in the state S _t . The reward r _t shown in FIG. 2 indicates the reward obtained by the agent 20 for transitioning from the state S _t to S _t+1 .

In this way, the state and behavior of agent 20 and the reward value that agent 20 receives can be diagrammed. The inference unit 14 derives an action sequence that can maximize the expected value of the cumulative value (also referred to as cumulative reward) of the reward r _t when the agent 20 takes a series of actions a _t .

Hereinafter, a method for deriving a behavior sequence by the inference unit 14 will be explained.

FIG. 3 is an explanatory diagram showing cumulative rewards for each action sequence in this embodiment. Here, we will explain a case in which the action sequence is a vector whose elements are continuous values, but the same explanation holds true even if it is a discrete value.

FIG. 3 is a graph in which the horizontal axis represents the action sequence and the vertical axis represents the cumulative reward for each action sequence. As shown in FIG. 3, the cumulative reward generally takes multiple maximum values for the action sequence.

FIG. 4 is an explanatory diagram showing the concept of variational inference performed by the inference unit 14 in this embodiment.

FIG. 4 is a graph in which the horizontal axis represents the action sequence and the vertical axis represents the probability distribution corresponding to each action sequence. Note that the probability distribution, which is the vertical axis in FIG. 4, can be converted from the cumulative reward, which is the vertical axis in FIG. 3, by a predetermined calculation.

The inference unit 14 derives candidates for the optimal action sequence of the agent 20 by performing variational inference using a Gaussian Mixture Model (GMM) as a variational distribution.

The solid line shown in FIG. 4 is the cumulative reward for the action sequence shown in FIG. 3 converted into a probability distribution. Examples of means for converting the cumulative reward into a probability distribution include a method of mapping the cumulative reward with an exponential function.

The broken line shown in FIG. 4 shows a mixed Gaussian distribution, that is, a probability distribution that is a mixture (synthesis) of a plurality of Gaussian distributions. The Gaussian mixture distribution is specified by parameters including a set of mean value, variance, and peak value of each Gaussian distribution that constitutes the Gaussian mixture distribution.

The inference unit 14 determines the plurality of parameters of the Gaussian mixture distribution by variational inference.

Specifically, the inference unit 14 performs a calculation to determine multiple parameters of the Gaussian mixture distribution so as to reduce the difference between the probability distribution (Gaussian mixture distribution) indicated by the broken line in FIG. 4 and the probability distribution indicated by the solid line. Repeat. The number of times the calculation is performed may be arbitrarily determined, and the repetition may be terminated when the improvement rate of the cumulative reward for each repetition falls within a certain value (for example, ±1%).

The inference unit 14 determines the parameters of each of the plurality of Gaussian distributions included in the Gaussian mixture distribution by performing variational inference as described above. Since each of the plurality of determined Gaussian distributions corresponds to an action sequence, the determination of the parameters corresponds to the inference unit 14 deriving a plurality of candidates for the action sequence of the agent 20.

In the example of FIG. 4, the inference unit 14 obtains mean values μ1 and μ2, variances σ1 and σ2, and mixing ratios π1 and π2 as parameters for each of the two Gaussian distributions. The parameters of the action sequence determined in this way become a plurality of candidates for the action sequence of the agent 20.

Hereinafter, the variational inference performed by the inference unit 14 and the variational inference in related technology will be explained while being compared. In variational inference in related art, a single model (eg, a single Gaussian distribution) is used instead of a mixed model (Gaussian mixture distribution).

FIG. 5 is an explanatory diagram showing the concept of variational inference in related technology. The vertical and horizontal axes in FIG. 5 are displayed in the same way as in FIG. 4 .

The inference unit in the related technology derives candidates for the optimal action sequence of the agent 20 by performing variational inference using a Gaussian distribution (that is, a single Gaussian distribution, not a mixed Gaussian distribution) as a variational distribution.

The dashed line shown in FIG. 5 indicates a Gaussian distribution. A Gaussian distribution is specified by a set of mean, variance, and peak value of the probability distribution.

The inference unit determines the parameters of the Gaussian distribution by variational inference.

Specifically, the inference unit repeats calculations to determine multiple parameters of the Gaussian mixture distribution so as to reduce the difference between the probability distribution (Gaussian distribution) indicated by the broken line in FIG. 5 and the probability distribution indicated by the solid line. conduct. The number of times the calculation is executed is the same as in the inference unit 14.

Thereby, the inference unit obtains only one action sequence μ0.

As shown in FIGS. 4 and 5, the variational distribution shown by the broken line (corresponding to the Gaussian mixture distribution in FIG. 4 and the Gaussian distribution in FIG. 5) and the probability distribution shown by the solid line (corresponding to the behavioral sequence) The magnitude of the difference from the probability distribution (probability distribution) tends to be smaller when a Gaussian mixture distribution is used (see FIG. 4). This is because when a Gaussian mixture distribution is used, by adjusting the parameters of each Gaussian distribution, the difference from the probability distribution indicated by the solid line can be reduced.

FIG. 6 is an explanatory diagram conceptually showing a plurality of candidates determined by the inference unit 14 in this embodiment.

Here, the route search problem will be explained as an example. This problem consists of moving within a rectangular area while avoiding obstacles indicated by black circles (●) in Figure 7, and reaching from the start position ("S" in the diagram) to the goal position ("G" in the diagram). This is a problem of deriving a route. The direction of travel at each location on the route corresponds to an "action."

In FIG. 6, a plurality of routes derived by the inference unit 14 will be explained while being compared with one route derived by the inference unit in related technology.

(a) of FIG. 6 shows one route 30 derived by the inference unit according to the related technology. One route 30 shown in FIG. 7(a) corresponds to one action sequence μ0 shown in FIG. 5.

FIG. 6B shows a plurality of routes 31, 32, 33, 34, and 35 derived by the inference unit 14 in this embodiment. Here, a case is illustrated in which the inference unit 14 uses a mixed Gaussian distribution, which is a mixture of five Gaussian distributions, as the variational distribution. In this case, the number of routes derived is five.

Further, in FIG. 6B, the width of the line indicating the path 31 etc. indicates the mixing ratio of the Gaussian distribution corresponding to the path in the mixed Gaussian distribution. In this example, the line width of the path 31 is drawn thicker than the others, indicating that the mixing ratio of the Gaussian distribution corresponding to the path 31 is higher than the other paths 32 to 35.

The output unit 15 can output one route selected from the routes shown in FIG. 6(b), for example, route 31.

FIG. 7 is a flow diagram showing a learning method executed by the learning device 10 in this embodiment. This learning method is a learning method for the behavior of the agent 20 using model-based reinforcement learning.

As shown in FIG. 7, in step S101, time-series data indicating the state and behavior of the agent 20 when the agent 20 acts is acquired.

In step S102, the dynamics model 17 is constructed by performing supervised learning using the time series data acquired in step S101.

In step S103, based on the dynamics model 17, a plurality of candidates for the action sequence of the agent 20 are derived by variational inference using a mixture model as a variational distribution.

In step S104, one candidate selected from the plurality of derived candidates is output as the action sequence of the agent 20.

Through the series of processes described above, the learning device 10 improves the learning method for controlling the behavior of the agent 20.

Hereinafter, the effects of the learning method in this embodiment will be explained.

FIG. 8 is an explanatory diagram showing temporal changes in cumulative rewards in the learning method according to the present embodiment in comparison with related techniques.

An example of a simulation result using an existing simulator will be shown regarding cumulative rewards when learning is performed by the learning device 10 according to the present embodiment. An existing simulator is the physics engine MuJoCo. Figures (a) to (d) show the results for four different tasks (i.e., HalfCheetah, Ant, Hopper, and Walker2d) provided by OpenAI Gym, an evaluation platform for reinforcement learning that runs on the above simulator. It is.

The thick solid line shown in FIG. 8 indicates the change in cumulative reward related to the learning device 10 according to the present embodiment. Moreover, the thin solid line shown in FIG. 8 shows the change in the cumulative reward related to the learning device related to the related technology. Furthermore, the convergence values for each of the changes in the cumulative reward described above are indicated by dashed lines.

As shown in FIG. 8, in all the tasks shown in (a) to (d) of FIG. The cumulative reward starts increasing earlier and converges to the convergence value in a shorter time (i.e., the speed of convergence is faster). Furthermore, the learning device 10 according to the present embodiment has a larger convergence value of cumulative reward than learning devices according to related technologies.

This indicates that the learning device 10 according to the present embodiment has improved learning performance compared to learning devices in related technology.

FIG. 9 is an explanatory diagram illustrating the convergence value of the cumulative reward in the learning method according to the present embodiment in comparison with related techniques. FIG. 9 shows the convergence value of the cumulative reward for each number M of Gaussian distributions included in the Gaussian distribution and Gaussian mixture distribution in the HalfCheetah task.

Note that the case of M=1 corresponds to a single Gaussian distribution, that is, corresponds to the learning method in the related technology. Moreover, the case where M=3, 5, or 7 corresponds to a Gaussian mixture distribution, that is, corresponds to the learning method in this embodiment.

As shown in Figure 9, compared to the case where variational inference is performed using a single Gaussian distribution (when M = 1), the case where variational inference is performed using a mixed Gaussian distribution (when M = 3, 5, or 7), the convergence value of the cumulative reward is larger. Further, the convergence value of the cumulative reward changes depending on the different value of M. Note that the convergence value of the cumulative reward has a property that it differs depending on the method of defining the reward, the type of task, etc.

As described above, according to the learning method according to the present embodiment, variational inference is performed using a mixture model as a variational distribution, so a plurality of candidates for the action sequence of the agent can be derived. Each of the multiple behavioral sequence candidates derived in this way has a smaller difference from the optimal behavioral sequence than when performing variational inference using a single model as a variational distribution, and Convergence speed is fast when deriving multiple candidates. It is assumed that one of the plurality of derived behavioral sequence candidates is used as the agent's behavioral sequence. By doing so, a more appropriate action sequence can be output in a shorter time. In this way, according to the above aspect, it is possible to improve the learning method for controlling the agent.

Furthermore, since the state and behavior of the agent controlled using the action sequence output as a result of learning are used for new learning, the action sequence constructed as a result of learning can be brought closer to an appropriate one. In this way, the learning method for controlling the agent can be further improved.

Furthermore, since the candidate corresponding to the probability distribution with the maximum mixing ratio in the mixture model is used as the agent's action sequence, the difference from the optimal action sequence can be further reduced. Therefore, according to the above aspect, it is possible to improve the learning method for controlling the agent so as to output an action sequence closer to the optimal action sequence.

Further, since a Gaussian mixture distribution is specifically used as the mixture model, a plurality of candidates can be more easily derived by variational inference. Therefore, the learning method for controlling the agent can be improved more easily.

Furthermore, since an ensemble of a plurality of neural networks is used as the dynamics model, it is possible to more easily construct a dynamics model with relatively high estimation accuracy. Therefore, the learning method for controlling the agent can be improved more easily.

Note that in the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. Here, the software that implements the content management system of the above embodiment is the following program.

That is, this program is a method of learning agent behavior using model-based reinforcement learning in a computer, and acquires time-series data indicating the state and behavior of the agent when the agent acts, A dynamics model is constructed by performing supervised learning using the acquired time series data, and based on the dynamics model, multiple behavioral sequences of the agent are determined by variational inference using a mixture model as a variational distribution. This program executes a learning method for deriving candidates, and outputting one candidate selected from the plurality of derived candidates as an action sequence of the agent.

Although the learning method and the like according to one or more aspects have been described above based on the embodiments, the present invention is not limited to these embodiments. Unless departing from the spirit of the present invention, various modifications that can be thought of by those skilled in the art to this embodiment, and embodiments constructed by combining components of different embodiments are within the scope of one or more embodiments. may be included within.

INDUSTRIAL APPLICATION This invention can be utilized for the learning device which controls the behavior of an agent.

10 learning device 11 acquisition unit 12 learning unit 13 storage unit 14 inference unit 15 output unit 17 dynamics model 20 agent 30, 31, 32, 33, 34, 35 route

Claims (6)

A method for learning agent behavior using model-based reinforcement learning, the method comprising:
obtaining time series data indicating the state and behavior of the agent when the agent acts;
Build a dynamics model by performing supervised learning using the acquired time series data,
Based on the dynamics model, derive a plurality of candidates for the action sequence of the agent by variational inference using a mixture model as a variational distribution;
A learning method in which one candidate selected from the plurality of derived candidates is output as an action sequence of the agent. moreover,
The learning method according to claim 1, wherein new time-series data indicating the state and behavior of the agent when the agent acts according to the outputted action sequence is acquired as the time-series data. When deriving the plurality of candidates,
Deriving the mixture ratio of the probability distribution corresponding to each of the plurality of candidates in the entire mixture model along with the plurality of candidates,
When selecting the first candidate,
The learning method according to claim 1 or 2, wherein, among the plurality of derived candidates, a candidate corresponding to a probability distribution in which the mixing ratio is maximum is selected as the one candidate. The learning method according to claim 1, wherein the mixture model is a Gaussian mixture distribution. The learning method according to claim 1, wherein the dynamics model is an ensemble of a plurality of neural networks. A program for causing a computer to execute the learning method according to any one of claims 1 to 5.

Images