CN116757272A - A continuous action control reinforcement learning framework and learning method - Google Patents

Abstract

The application discloses a continuous action control reinforcement learning framework and a learning method, and relates to the technical field of artificial intelligence. The learning framework includes: the multi-step state transfer learning module is used for learning multi-step state transfer by adopting a convolutional neural network and updating a strategy; the expected estimation module is used for estimating the expected of multi-step accumulated returns by adopting a multi-step time sequence difference algorithm; and the sample clustering module is used for clustering different types of state transition samples so that each sample is uniformly sampled. The application combines convolutional neural network, multi-step time sequence differential estimation and state transfer clustering, effectively improves learning efficiency and accuracy, and makes the sample more fully utilized.

Description

A continuous action control reinforcement learning framework and learning method

Technical field

The present invention relates to the field of artificial intelligence technology, and specifically to a continuous action control reinforcement learning framework and a learning method.

Background technique

Currently, some effective deep reinforcement learning algorithms have been proposed for optimizing continuous control. The most representative one is DDPG, which works based on the actor critic method. Define ρ _t as the state at time t, α _t as the action at time t, and define the deterministic strategy as follows:

α _t =π _θ (ρ _t )

Existing actor-critic frameworks train agents by cyclically updating an estimated function of cumulative rewards and a policy that maximizes this function. An estimate of the cumulative return can be obtained by minimizing the following objective function.

Where B is the set of sampled state transitions, returns and actions.

The objective function that needs to be maximized when updating the strategy is as follows:

Based on the actor-critic framework, DDPG mainly learns the single-step state transition through a fully connected neural network and then estimates the expectation of the cumulative reward function through the cumulative reward of the single step. TD3 and SAC are two improved algorithms based on DDPG. TD3 improves overestimation, policy updating and exploration in DDPG through dual critic network, temporal difference estimation and Gaussian noise. SAC improves exploration in DDPG mainly by improving the objective function. It also uses dual critic network and temporal difference estimation.

However, the existing technology has the following shortcomings:

1. Only considering single-step state transfer will lead to insufficient learning efficiency.

2. Estimating the expectation of cumulative returns by considering only single-step returns will lead to inaccurate estimates.

3. Using randomly sampled state transitions to update the neural network can easily lead to insufficient sample utilization.

Contents of the invention

In order to overcome the above problems or at least partially solve the above problems, the present invention provides a continuous action control reinforcement learning framework and learning method, which combines a convolutional neural network, multi-step temporal difference estimation and state transfer clustering, effectively improving learning efficiency. and accuracy, and make full use of samples.

In order to solve the above technical problems, the technical solutions adopted by the present invention are:

In the first aspect, the present invention provides a continuous action control reinforcement learning framework, including a multi-step state transfer learning module, an expectation estimation module and a sample clustering module, wherein:

The multi-step state transfer learning module is used to learn multi-step state transfer and update strategies using convolutional neural networks;

The expectation estimation module is used to estimate the expectation of multi-step cumulative returns using a multi-step temporal difference algorithm;

The sample clustering module is used to cluster different types of state transition samples so that each sample is evenly sampled.

This framework combines convolutional neural networks, multi-step temporal difference estimation and state transition clustering for the first time. It has the following characteristics: it uses a convolutional neural network to consider multi-step state transitions to update the policy; it uses a multi-step temporal difference algorithm to estimate multi-step Expectation of cumulative returns; by clustering existing state transition samples, each sample is fully sampled. This invention uses a convolutional neural network to learn multi-step state transitions in reinforcement learning for continuous control, which improves learning efficiency; based on the previous step, it estimates the expectation of cumulative returns through multi-step returns, making the estimation more accurate; this invention The invention also enables different types of state transition samples to be evenly sampled through clustering, thereby making full use of the samples.

Based on the first aspect, further, the above strategy is in, α is the action, ρ refers to the state, π is the strategy, θ ^c is the parameter of the convolutional neural network, t is the current moment and n _p is the number of state transition steps.

Based on the first aspect, further, the following objective function is minimized to obtain the expectation of estimating the multi-step cumulative return,

The objective function is:

Among them, n _p is the number of state transition steps, n _q is the number of reward steps, B _n is the set of sampled multi-step state transitions, multi-step rewards and actions, E is the expectation, and Q is a function that estimates the cumulative reward expectation. , are the parameters for estimating Q and

Based on the first aspect, further, using the function Update strategy.

Based on the first aspect, further, when performing clustering, the total number of training steps is evenly distributed to different time periods, and the samples in each time period are clustered; the above clustering method uses the k-means algorithm.

Based on the first aspect, further, when sampling the state transition update function, samples in each cluster are uniformly sampled.

The present invention has at least the following advantages or beneficial effects:

The present invention provides a continuous action control reinforcement learning framework and learning method, which combines a convolutional neural network, multi-step temporal difference estimation and state transfer clustering, and learns multi-steps through the convolutional neural network in reinforcement learning for continuous control. State transfer improves learning efficiency; on the basis of the previous step, the expectation of cumulative return is estimated through multi-step returns, making the estimate more accurate; the present invention also uses clustering to evenly sample different types of state transition samples, so that Samples are more fully utilized.

Description of the drawings

In order to explain the technical solutions of the embodiments of the present invention more clearly, the drawings required to be used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and therefore do not It should be regarded as a limitation of the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without exerting creative efforts.

Figure 1 is a schematic diagram of a continuous action control reinforcement learning framework according to an embodiment of the present invention;

Figure 2 is a schematic diagram of experimental training of an agent in different virtual environments in an embodiment of the present invention;

Figure 3 is a schematic diagram of the sampling pool obtained after sample clustering in the embodiment of the present invention;

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, rather than all embodiments. The components of the embodiments of the invention generally described and illustrated in the figures herein may be arranged and designed in a variety of different configurations.

Therefore, the following detailed description of the embodiments of the invention provided in the appended drawings is not intended to limit the scope of the claimed invention, but rather to represent selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of protection of the present invention.

It should be noted that similar reference numerals and letters represent similar items in the following figures, therefore, once an item is defined in one figure, it does not need further definition and explanation in subsequent figures.

It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that these entities or operations are mutually exclusive. any such actual relationship or sequence exists between them. Furthermore, the term "comprising" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article or apparatus including a list of elements includes not only those elements but also other elements not expressly listed, Or it also includes elements inherent to the process, method, article or equipment. Without further limitation, an element defined by the statement "comprises a..." does not exclude the presence of additional identical elements in a process, method, article, or apparatus that includes the stated element.

In the description of the embodiments of the present invention, “plurality” represents at least two.

Example:

As shown in Figure 1, in the first aspect, embodiments of the present invention provide a continuous action control reinforcement learning framework, including a multi-step state transfer learning module 100, an expectation estimation module 200 and a sample clustering module 300, wherein:

The multi-step state transfer learning module 100 is used to learn multi-step state transfer and update strategies using convolutional neural networks;

The expectation estimation module 200 is used to estimate the expectation of multi-step cumulative returns using a multi-step temporal difference algorithm;

The sample clustering module 300 is used to cluster different types of state transition samples so that each sample is uniformly sampled.

This framework combines the convolutional neural network, multi-step temporal difference estimation and state transition clustering through the cooperation of the multi-step state transfer learning module 100, the expectation estimation module 200 and the sample clustering module 300. It has the following characteristics: Use convolution The neural network considers multi-step state transitions to update the strategy; uses a multi-step temporal difference algorithm to estimate the expectation of multi-step cumulative returns; and clusters existing state transition samples so that each sample is fully sampled. This invention uses a convolutional neural network to learn multi-step state transitions in reinforcement learning for continuous control, which improves learning efficiency; based on the previous step, it estimates the expectation of cumulative returns through multi-step returns, making the estimation more accurate; this invention The invention also enables different types of state transition samples to be evenly sampled through clustering, thereby making full use of the samples.

Based on the first aspect, further, the above strategy is in, α is the action, ρ refers to the state, π is the strategy, θ ^c is the parameter of the convolutional neural network, t is the current moment and n _p is the number of state transition steps.

Based on the first aspect, further, the following objective function is minimized to obtain the expectation of estimating the multi-step cumulative return,

The objective function is:

Among them, n _p is the number of state transition steps, n _q is the number of reward steps, B ⁿ is the set of sampled multi-step state transitions, multi-step rewards and actions, E is the expectation, and Q is a function that estimates the cumulative reward expectation. , are the parameters for estimating Q and

Based on the above strategy, in the newly defined framework, the estimate of the cumulative return can be obtained by minimizing the above objective function. above with/> correspond.

Based on the first aspect, further, using the function Update strategy.

The objective function that needs to be maximized when updating the strategy is as above. This step is also completed by updating the defined convolutional neural network.

Based on the first aspect, further, when performing clustering, the total number of training steps is evenly distributed to different time periods, and the samples in each time period are clustered. The above clustering method uses the k-means algorithm.

Based on the first aspect, further, when sampling the state transition update function, samples in each cluster are uniformly sampled.

In some embodiments of the present invention, it is also necessary to divide the total number of training steps evenly into different time periods, and then cluster the samples in each time period. The clustering method uses k-means. The resulting sampling pool is shown in Figure 3. During sampling, it is assumed that the current period is p, and the number of clusters in each period is k. The probability that a sample in each cluster is sampled every time the neural network is updated is The probability of a sample being sampled in the current period is 0.2.

In some embodiments of the present invention, the algorithm flow for learning based on this framework is as follows:

np is the number of defined time periods, pt is the number of steps in each time period, and the algorithm is as follows:

Initialize neural network parameters

Initialize the sampling space

Initialize exploration noise

Fore＝1:np

Fort＝1:pt

Choose actions through strategies

Add exploration noise to actions

Executing an action gets the reward rt and status

Store actions, states and returns in the sampling space

Select samples from each existing cluster in the sampling space and from the state transitions generated during the current period

Update the neural network with selected samples

Endfor

Cluster samples from the previous time period

Endfor

Outputs a neural network-based policy model.

In some embodiments of the present invention, the TD3 algorithm is improved using the framework proposed by the present invention, and a new algorithm TD3+ is obtained. Experiments were conducted on the virtual robot control environment Mujoco, and the experimental tasks included HafCheetal, Walker2d, and Hopper. As shown in Figure 2, they are agents in the HafCheeta, Walker2d and Hopper environments. In the first two environments, reinforcement learning is needed to make the agent walk farther within a fixed number of steps, while the last one requires training a single-legged agent to jump as far as possible.

Comparison algorithms include DDPG, SAC, and TD3. For all methods, each task runs for 2*10^6 time steps. The cumulative returns obtained by different algorithms on different tasks are shown in Table 1. It can be seen that the algorithm TD+3 implemented using the proposed framework is better than the existing algorithm.

Table 1:

Operating environment TD3+ TD3 SAC DDPG HafCheetal 13589.17 10032.66 9643.93 9453.22 Walker2d 6167.26 4471.43 4971.42 3804.91 Hopper 3812.30 3472.65 3531.77 3736.21

In some embodiments of the present invention, ablation experiments are performed, and the results are shown in Table 2, which compares the new method without clustering (TD3+woC), without using convolutional neural networks (TD3+woS), and without using Results of the multi-step temporal difference algorithm (TD3+woQ). It can be seen from this that each part of the present invention (convolutional neural network, multi-step temporal difference estimation and clustering) can effectively improve the effect of reinforcement learning.

Table 2:

Operating environment TD3+ TD3+woC TD3+woS TD3+woQ HafCheetal 13589.17 12824.48 12654.81 12051.56 Walker2d 6267.26 6056.13 5401.72 5737.12 Hopper 3812.30 3758.30 3713.23 3762.34

In a second aspect, embodiments of the present invention provide a continuous action control reinforcement learning method based on the continuous action control reinforcement learning framework as described in any one of the above first aspects, including the following steps:

Use convolutional neural networks to learn multi-step state transitions and update learning strategies;

A multi-step temporal difference algorithm is used to estimate the expectation of multi-step cumulative returns;

Cluster different types of state transition samples so that each sample is evenly sampled.

This invention uses a convolutional neural network to learn multi-step state transitions in reinforcement learning for continuous control, which improves learning efficiency; based on the previous step, it estimates the expectation of cumulative returns through multi-step returns, making the estimation more accurate; this invention The invention also enables different types of state transition samples to be evenly sampled through clustering, thereby making full use of the samples.

The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

It is obvious to those skilled in the art that the present application is not limited to the details of the above-described exemplary embodiments, and that the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be regarded as illustrative and non-restrictive from any point of view, and the scope of the present application is defined by the appended claims rather than the above description, and it is therefore intended that all claims falling within the claims All changes within the meaning and scope of the equivalent elements are included in this application. Any reference signs in the claims shall not be construed as limiting the claim in question.

Claims (7)

1. A continuous action control reinforcement learning framework, characterized by including a multi-step state transfer learning module, an expectation estimation module and a sample clustering module, wherein: The multi-step state transfer learning module is used to learn multi-step state transfer and update strategies using convolutional neural networks; The expectation estimation module is used to estimate the expectation of multi-step cumulative returns using a multi-step temporal difference algorithm; The sample clustering module is used to cluster different types of state transition samples so that each sample is evenly sampled. 2. A continuous action control reinforcement learning framework according to claim 1, characterized in that the strategy is Among them,/> α is the action, ρ refers to the state, π is the strategy, θ ^c is the parameter of the convolutional neural network, t is the current moment and n _p is the number of state transition steps. 3. A continuous action control reinforcement learning framework according to claim 1, characterized in that the following objective function is minimized to obtain the expectation of estimating the multi-step cumulative return, The objective function is: Among them, n _p is the number of state transition steps, n _q is the number of reward steps, B ⁿ is the set of sampled multi-step state transitions, multi-step rewards and actions, E is the expectation, and Q is a function that estimates the cumulative reward expectation. , are the parameters for estimating Q and 4. A continuous action control reinforcement learning framework according to claim 3, characterized in that using a function Update strategy. 5. A continuous action control reinforcement learning framework according to claim 1, characterized in that when performing clustering, the total number of training steps is evenly distributed to different time periods, and the samples in each time period are Clustering; the clustering method adopts k-means algorithm. 6. A continuous action control reinforcement learning framework according to claim 1, characterized in that when sampling the state transfer update function, samples in each cluster are uniformly sampled. 7. A continuous action control reinforcement learning method based on the continuous action control reinforcement learning framework according to any one of claims 1 to 6, characterized in that it includes the following operations Use convolutional neural networks to learn multi-step state transitions and update learning strategies; A multi-step temporal difference algorithm is used to estimate the expectation of multi-step cumulative returns; Cluster different types of state transition samples so that each sample is evenly sampled.