CN112100834A - Underwater glider attitude control method based on deep reinforcement learning - Google Patents

Abstract

The present invention proposes an underwater glider attitude control method based on deep reinforcement learning, which includes a learning stage and an application stage. network, current evaluation neural network, target decision neural network and target evaluation neural network parameters; after obtaining the trained deep reinforcement learning neural network model, it is applied to the actual underwater glider in the vertical plane gliding motion, given the target pitch angle θ _d , the state value of the underwater glider is collected and input to the deep reinforcement learning neural network model to obtain the control amount to realize the attitude control of the underwater glider. The invention performs learning based on simulation model data or artificial experimental data, realizes the attitude control of the underwater glider, and has a simple learning method; and does not need to obtain an accurate mathematical model of the underwater glider, and is also applicable in complex environments.

Description

An Attitude Control Method for Underwater Glider Based on Deep Reinforcement Learning

technical field

The invention relates to a control technology of an underwater robot, in particular to an attitude control method of an underwater glider based on deep reinforcement learning.

Background technique

Underwater glider is a new type of underwater vehicle developed by combining buoy, submersible buoy technology and underwater robot technology without external hangers and driven by its own gravity. Its main features are: the motion control does not rely on the propeller propulsion system, but by adjusting the net buoyancy of the glider to achieve up and down movements, and use the horizontal wings attached to the fuselage to generate oblique upward or oblique downward lift to control the glider forward. gliding. The underwater glider overcomes the shortcomings of high power and short sailing time of underwater vehicles, greatly reduces the operating cost and manufacturing cost, and improves the endurance time. It is of great practical value in military and marine exploration and research.

The motion attitude of the underwater glider is easily affected by ocean currents and waves. At the same time, the structure of the underwater glider is complex, the dynamic mode is single, and the dynamic model shows strong nonlinearity. Accurate model parameters are not easy to obtain and are constructed in different water environments. Models also lack universality. Although many traditional control methods can realize the attitude control of underwater glider and can achieve a certain control precision, they still cannot meet the requirements of high precision, and the control process is more complicated.

SUMMARY OF THE INVENTION

technical problem to be solved

The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, provide an underwater glider attitude control method based on deep reinforcement learning, establish a deep reinforcement learning neural network model, and learn the simulation model data or artificial experimental data. Realize precise control of underwater glider attitude.

Technical solutions

The underwater glider attitude control method based on deep reinforcement learning proposed by the present invention includes a learning stage and an application stage. The parameters of the current evaluation neural network, target decision neural network and target evaluation neural network are as follows:

Step 1: Establish 4 BP neural networks, namely the current decision-making neural network, the current evaluation neural network, the target decision-making neural network and the target evaluation neural network. The current decision neural network and the target decision neural network are called decision neural networks, and the current evaluation neural network and the target evaluation neural network are called evaluation neural networks. The decision neural network uses the state value of the underwater glider as the input, and uses the control value a of the underwater glider as the output action. The evaluation neural network has the state value and control quantity of the underwater glider as the input, and the evaluation value as the output;

After building the neural network, initialize the parameters of the 4 neural networks, initialize the memory bank and the size of the data buffer.

Step 2: Obtain the state value s _t of the underwater glider at the current moment, input the state value into the current decision-making neural network to calculate the output action a _t of the attitude controller at the current moment _, and apply the output action at to the underwater glider simulator, Obtain the state value s _t+1 of the underwater glider at the next moment. The reward value _rt at the current moment is calculated according to the state s _t at the current moment, the action at at the current moment, the target pitch angle θ _d and the state _{st +1} at the next moment.

The preferred value of r _t is:

r _t =r ₁ +r ₂ +r ₃

Among them, r ₁ =-λ ₁ (d _t -d _t-1 ), r ₁ represents the reward value obtained when the current pitch angle is far away from or close to the desired angle, r ₂ =-λ ₂ (w _t -w _t-1 ), r ₂ represents the reward value obtained from the change of angular velocity, and r ₃ takes the value according to the current pitch angle, if: |d _t -d _t-1 |＜0.1°, r ₃ is a positive "reward", if θ＜ When -90° or θ>90°, r ₃ is a negative number, indicating punishment, d _t is the difference between the current pitch angle θ and the target pitch angle θ _d at time t, and w _t is the pitch at time t The size of the angular rate, λ ₁ and λ ₂ are set coefficients.

Step 3: Store the state (s _t , at , r _t , s _{t +1} ) obtained in step 2 as a set of empirical data units in the memory bank, and increment t by 1; judge t and the set memory The relationship between the library size n, if t<n, use the updated s _t to return to step 2, until the number of experience data units stored in the memory library meets the requirement of n, then enter step 4.

Step 4: Sampling a specified number of N empirical data units from the memory bank and storing them in the buffer.

Preferably, the N experience data units in the buffer are prioritized through a priority-based experience playback mechanism and stored in the SumTree data structure; the value of the head node in the SumTree data structure is the experience data represented by all nodes The sum of unit priorities, denoted as p _total .

Step 5: For the N experience data units stored in the SumTree data structure obtained in step 4, and then sample m experience data units from the N experience data units. In this embodiment, in order to ensure that various experiences may be If selected, m empirical data units are sampled through the following process;

Divide [0,p _total ] into equal numbers of m cells, and perform random and uniform sampling in each cell to obtain experience data units corresponding to the sampling priorities, and obtain m experience data units in total.

The sampled m empirical data units are processed one by one according to the following process, and the gradient value of the current evaluation neural network is obtained.

For a certain empirical data unit (s _t , at _t , r _t , s _{t +1} ), input the state s _t and the action signal at into the current evaluation neural network to obtain the evaluation value Q of the current evaluation neural network; The state s _t+1 at one moment is input into the target decision-making neural network, and the action signal μ' of the actuator output by the target decision-making neural network is obtained; the next moment state s _t+1 and the action value μ' output by the target decision-making neural network are input Go to the target evaluation neural network to obtain the evaluation value Q' of the target evaluation neural network;

Using the evaluation value Q of the current evaluation neural network, the evaluation value Q' of the target evaluation neural network and the loss function L of the evaluation neural network, the gradient value of the current evaluation neural network is calculated

Specifically, the loss function L of the current evaluation neural network is:

Among them, ω _i is the importance sampling weight of the i-th empirical data unit, and δ _i is:

δ _i =r _i +γQ'(s _i+1 ,μ'(s _i+1 |σ ^μ' )σ ^Q' )-Q(s _i ,a _i |σ ^Q )

ri is the reward value in the _i -th empirical data unit, s _i+1 is the next moment state in the i-th empirical data unit, σ ^μ' is the parameter of the target decision-making neural network, μ'(s _i+1 |σ ^μ' ) is the action signal of the actuator output by _si+1 through the target decision-making neural network. σ ^Q' is the parameter of the target evaluation neural network, Q'(s _i+1 ,μ'(s _i+1 |σ ^μ' )σ ^Q' ) is the combination of μ'(s _i+1 |σ ^μ' ) and s _i+1 is the size of the evaluation value obtained by the target evaluation neural network. s _i is the current moment state in the ith experience data unit, a _i is the action signal at the current moment state in the ith experience data unit, σ ^Q is the parameter of the current evaluation neural network, Q(s _i , a _i |σ ^Q ) is the size of the evaluation value of the current evaluation neural network. γ is the discount coefficient, and the value of γ is 0.99.

The gradient of the loss function L of the current evaluation neural network is:

对当前评价神经网络参数σ^Q自增

进行更新，α为评价神经网络的学习率，取值为0.001。Step 6: Update the current evaluation neural network: according to the gradient of the current evaluation neural network

Self-increase for the current evaluation neural network parameter σ ^Q

Update, α is the learning rate of the evaluation neural network, and the value is 0.001.

具体梯度大小计算公式为：Step 7: Calculate the gradient of the current decision neural network

The specific gradient size calculation formula is:

表示当前评价神经网络的评价值

对参数α的梯度，μ(s_i)是s_i通过当前决策神经网络输出的执行机构的动作信号；

表示当前决策神经网络动作

对当前决策神经网络参数σ^μ的梯度。in

Indicates the evaluation value of the current evaluation neural network

For the gradient of the parameter α, μ(s _i ) is the action signal of the actuator output by s _i through the current decision-making neural network;

Represents the current decision neural network action

Gradient of the current decision neural network parameter σ ^μ .

Step 8: Current Decision Neural Network Update:

对当前决策神经网络参数σ^μ自增

进行更新，β为决策神经网络的学习率，取值为0.001。According to the gradient of the current decision neural network

Self-increase for the current decision-making neural network parameter σ ^μ

To update, β is the learning rate of the decision neural network, and the value is 0.001.

Step 9: update the target evaluation neural network and the target decision neural network: update the target evaluation neural network parameters according to the updated current evaluation neural network parameters, and update the target decision neural network parameters according to the updated current decision neural network parameters, The update formula is

Among them, σ _t+1 ^Q represents the updated current evaluation neural network parameters, σ _t ^Q' represents the target evaluation neural network parameters to be updated, σ _t+1 ^Q' represents the updated target evaluation neural network parameters; σ _{t+ 1} ^μ represents the updated current decision-making neural network parameters, σ _t ^μ' represents the target decision-making neural network parameters to be updated, σ _t+1 ^μ' represents the updated target decision-making neural network parameters; τ ₁ is the target evaluation neural network parameter. The update rate is 0.001, and τ ₂ is the update rate of the target decision-making neural network, which is 0.0001.

Step 10: Determine whether the training times exceed the set training times. If it exceeds the set training times, stop the training and save the parameter values of the 4 neural networks. If it does not exceed the set training times, go back to step 4 and re-store in the memory. The empirical data units with a specified number of N samples in the library are stored in the buffer.

Step 11: After obtaining the trained deep reinforcement learning neural network model, apply it to the actual underwater glider in the vertical plane gliding motion, given the target pitch angle θ _d , collect the state value of the underwater glider and input it to the deep reinforcement learning neural network The model gets the control amount to realize the attitude control of the underwater glider.

beneficial effect

Compared with the prior art, the beneficial effects brought by the technical solution of the present invention are:

1. Learning based on simulation model data or artificial experimental data to realize the attitude control of the underwater glider, and the learning method is simple.

2. There is no need to obtain an accurate mathematical model of the underwater glider, and it is also applicable in complex environments.

Additional aspects and advantages of the present invention will be set forth, in part, from the following description, and in part will be apparent from the following description, or may be learned by practice of the invention.

Description of drawings

The above and/or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of embodiments taken in conjunction with the accompanying drawings, wherein:

Figure 1: Schematic diagram of the deep reinforcement learning algorithm framework;

Figure 2: The longitudinal gliding motion control diagram of the underwater glider;

Figure 3: Schematic diagram of SumTree structure;

Figure 4: Schematic diagram of the size of the round reward value during the attitude control training process of the underwater glider;

Figure 5: The change process of the current pitch angle under the action of the attitude adjustment unit when the desired pitch angle of the glide is set to -25° in the application stage.

Detailed ways

The example of the present invention will be described in detail below. In this example, taking the pitch angle control of an actual underwater glider in the vertical plane as an example, first determine the basic characteristic parameters of the underwater glider, and obtain the mathematics of the underwater glider when it glides in the vertical plane. The model, where the input control quantity is the speed command a _t of the moving mass on the x-axis, and the state of the glider change is s: {v ₁ , v ₃ , ω ₂ , θ}, v ₁ , v ₃ , ω ₂ , θ They are the speed of the underwater glider in the x and z-axis directions (the x-axis is the forward axis of the body coordinate system, and the z-axis is the coordinate axis perpendicular to the body plane), the pitch angular velocity, and the pitch angle. The block diagram of longitudinal gliding motion control with slider speed as input is shown in Figure 2.

The principle of this example method is to use the deep reinforcement learning method based on priority sampling to realize the control of the pitch angle of the underwater glider in the vertical plane. The specific steps are as follows:

Step 1: Establish 4 BP neural networks, namely the current decision-making neural network, the current evaluation neural network, the target decision-making neural network and the target evaluation neural network. The current decision neural network and the target decision neural network are called decision neural networks, and the current evaluation neural network and the target evaluation neural network are called evaluation neural networks. The decision neural network has 4 inputs, namely the state values v ₁ , v ₃ , ω ₂ , θ of the underwater glider at the current moment, and 1 output, namely the action value a. The evaluation neural network has five inputs v ₁ , v ₃ , ω ₂ , θ, a, and one output is the evaluation value. In this embodiment, the four BP neural networks have two hidden layers, the first hidden layer has 400 nodes, the second hidden layer has 300 nodes, and the four BP neural networks have one output node .

The mapping process of BP neural network is as follows:

Input layer information - hidden layer activation function - hidden layer output of the first layer:

Among them, x _i is the ith input quantity, and ε is the total number of input quantities. v _ki represents the weight from the input layer to the first hidden layer, b _k is the bias threshold, z _1k is the kth node of the first hidden layer, h ₁ is taken as 400, and f ₁ (s) is Hidden layer activation function, selected as Relu function.

The output of the hidden layer of the first layer - the activation function of the hidden layer - the output of the hidden layer of the second layer:

Among them, w _jk represents the weight between the first hidden layer and the second hidden layer, b _j is the bias threshold, z _2j is the jth output of the second hidden layer, and h ₂ is taken as 300 , f ₂ (s) is the activation function of the output layer, which is selected as the Relu function.

Layer 2 hidden layer output - output layer activation function - output layer output:

Among them, y is the output, where the output is all a value, w _j represents the weight between the second hidden layer and the output layer, b _l is the bias threshold, f ₃ (s) is the activation function of the output layer, The activation function of the output layer of the decision neural network is the Tanh function, and the activation function of the output layer of the evaluation neural network is the Relu function.

The parameters of the 4 neural networks are initialized, the size of the initialized memory bank is n=10000, and the size of the data buffer is 64.

Step 2: Obtain the state value s _t of the underwater glider at the current moment: {v ₁ , v ₃ , ω ₂ , θ}, input the state value into the current decision-making neural network to calculate the output action a _t of the attitude controller at the current moment, The output action at is applied to the simulator constructed based on the mathematical model of the underwater glider gliding in the longitudinal plane, and the state value s _{t +1} of the underwater glider at the next moment is obtained. The reward value _rt at the current moment is calculated according to the state s _t at the current moment, the action at at the current moment, the target pitch angle θ _d and the state _{st +1} at the next moment. In this embodiment, the value of r _t is:

r _t =r ₁ +r ₂ +r ₃

Among them, r ₁ =-λ ₁ (d _t -d _t-1 ), r ₁ represents the reward value obtained when the current pitch angle is far away from or close to the desired angle, r ₂ =-λ ₂ (w _t -w _t-1 ), r ₂ represents the reward value obtained from the change of angular velocity, and r ₃ takes the value according to the current pitch angle, if: |d _t -d _t-1 |＜0.1°, r ₃ is a positive "reward", if θ＜ When -90° or θ>90°, r ₃ is a negative number, indicating punishment, d _t is the difference between the current pitch angle θ and the target pitch angle θ _d at time t, and w _t is the pitch at time t The size of the angular rate, λ ₁ and λ ₂ are set coefficients.

Step 3: Store the state (s _t , at , r _t , s _{t +1} ) obtained in step 2 as a set of empirical data units in the memory bank, and increment t by 1; judge t and the set memory The relationship between the library size n, if t<n, use the updated s _t to return to step 2, until the number of experience data units stored in the memory library meets the requirement of n, then enter step 4.

Step 4: Sampling a specified number of N experience data units from the memory bank and storing them in the buffer; prioritizing the N experience data units in the buffer through the priority-based experience playback mechanism, and storing them in the SumTree data In the structure; the value of the head node in the SumTree data structure is the sum of the priorities of the experience data units represented by all nodes, denoted as p _total .

The priority-based experience playback is as follows: P(i) is defined as the sampling probability of the ith experience, and the calculation of P(i) is:

代表第i次经验的优先级，η用来设定使用优先级的程度，当η＝0时，为均匀经验采样。p_i＝|δ_i|+ε，δ_i为TD误差，ε为一个正的常数。为了不使算法的复杂性依赖经验池的容池大小，将得出的P(i)数据采用一种SumTree数据结构进行整理。SumTree是一种特殊的二叉树结构，该结构的所有叶子节点存储着对应经验的优先级，父节点的值为对应子节点的和，这样头结点的值为所有优先级的和，记为p_total。SumTree原理如图3所示：SumTree中的叶节点保存着各个经验的优先级，叶节点的序号与记忆库的经验序号一一对应，当选中该优先级的经验时，按照选中的当前优先级的序号去缓冲区中取出相应的经验。in,

Represents the priority of the i-th experience, and η is used to set the degree of use priority. When η=0, it is a uniform experience sampling. p _i =|δ _i |+ε, δ _i is the TD error, and ε is a positive constant. In order not to make the complexity of the algorithm depend on the size of the experience pool, the obtained P(i) data is organized using a SumTree data structure. SumTree is a special binary tree structure. All leaf nodes of the structure store the priority of the corresponding experience. The value of the parent node is the sum of the corresponding child nodes, so the value of the head node is the sum of all priorities, denoted as p _total . The principle of SumTree is shown in Figure 3: The leaf nodes in SumTree store the priority of each experience, and the serial number of the leaf node corresponds to the experience serial number of the memory bank. When the experience of this priority is selected, the current priority is selected The sequence number goes to the buffer to take out the corresponding experience.

Step 5: For the N experience data units stored in the SumTree data structure obtained in step 4, and then sample m experience data units from the N experience data units. In this embodiment, in order to ensure that various experiences may be If selected, m empirical data units are sampled through the following process;

Divide [0,p _total ] into equal numbers of m cells, and perform random and uniform sampling in each cell to obtain experience data units corresponding to the sampling priorities, and obtain m experience data units in total.

The sampled m empirical data units are processed one by one according to the following process, and the gradient value of the current evaluation neural network is obtained.

For a certain empirical data unit (s _t , at _t , r _t , s _{t +1} ), input the state s _t and the action signal at into the current evaluation neural network to obtain the evaluation value Q of the current evaluation neural network; The state s _t+1 at one moment is input into the target decision-making neural network, and the action signal μ' of the actuator output by the target decision-making neural network is obtained; the next moment state s _t+1 and the action value μ' output by the target decision-making neural network are input Go to the target evaluation neural network to obtain the evaluation value Q' of the target evaluation neural network;

Using the evaluation value Q of the current evaluation neural network, the evaluation value Q' of the target evaluation neural network and the loss function L of the evaluation neural network, the gradient value of the current evaluation neural network is calculated

The loss function L of the current evaluation neural network is:

Among them, ω _i is the importance sampling weight of the i-th empirical data unit, and δ _i is:

δ _i =r _i +γQ'(s _i+1 ,μ'(s _i+1 |σ ^μ' )σ ^Q' )-Q(s _i ,a _i |σ ^Q )

ri is the reward value in the _i -th empirical data unit, s _i+1 is the next moment state in the i-th empirical data unit, σ ^μ' is the parameter of the target decision-making neural network, μ'(s _i+1 |σ ^μ' ) is the action signal of the actuator output by _si+1 through the target decision-making neural network. σ ^Q' is the parameter of the target evaluation neural network, Q'(s _i+1 ,μ'(s _i+1 |σ ^μ' )σ ^Q' ) is the combination of μ'(s _i+1 |σ ^μ' ) and s _i+1 is the size of the evaluation value obtained by the target evaluation neural network. s _i is the current moment state in the ith experience data unit, a _i is the action signal at the current moment state in the ith experience data unit, σ ^Q is the parameter of the current evaluation neural network, Q(s _i , a _i |σ ^Q ) is the size of the evaluation value of the current evaluation neural network. γ is the discount coefficient, and the value of γ is 0.99.

The gradient of the loss function L of the current evaluation neural network is:

对当前评价神经网络参数σ^Q自增

进行更新，α为评价神经网络的学习率，取值为0.001。Step 6: Update the current evaluation neural network: according to the gradient of the current evaluation neural network

Self-increase for the current evaluation neural network parameter σ ^Q

Update, α is the learning rate of the evaluation neural network, and the value is 0.001.

Step 7: Calculate the gradient of the current decision-making neural network. The formula for calculating the gradient size is:

表示当前评价神经网络的评价值

对参数α的梯度，μ(s_i)是s_i通过当前决策神经网络输出的执行机构的动作信号；

表示当前决策神经网络动作

对当前决策神经网络参数σ^μ的梯度。in

Indicates the evaluation value of the current evaluation neural network

For the gradient of the parameter α, μ(s _i ) is the action signal of the actuator output by s _i through the current decision-making neural network;

Represents the current decision neural network action

Gradient of the current decision neural network parameter σ ^μ .

Step 8: Current Decision Neural Network Update:

对当前决策神经网络参数σ^μ自增

进行更新，β为决策神经网络的学习率，取值为0.001。According to the gradient of the current decision neural network

Self-increase for the current decision-making neural network parameter σ ^μ

To update, β is the learning rate of the decision neural network, and the value is 0.001.

Step 9: update the target evaluation neural network and the target decision neural network: update the target evaluation neural network parameters according to the updated current evaluation neural network parameters, and update the target decision neural network parameters according to the updated current decision neural network parameters, The update formula is

Among them, σ _t+1 ^Q represents the updated current evaluation neural network parameters, σ _t ^Q' represents the target evaluation neural network parameters to be updated, σ _t+1 ^Q' represents the updated target evaluation neural network parameters; σ _{t+ 1} ^μ represents the updated current decision-making neural network parameters, σ _t ^μ' represents the target decision-making neural network parameters to be updated, σ _t+1 ^μ' represents the updated target decision-making neural network parameters; τ ₁ is the target evaluation neural network parameter. The update rate is 0.001, and τ ₂ is the update rate of the target decision-making neural network, which is 0.0001.

Step 10: Determine whether the training times exceed the set training times. If it exceeds the set training times, stop the training and save the parameter values of the 4 neural networks. If it does not exceed the set training times, go back to step 4 and re-store in the memory. The empirical data units with a specified number of N samples in the library are stored in the buffer.

Step 11: After obtaining the trained deep reinforcement learning neural network model, apply it to the actual underwater glider in the vertical plane gliding motion. Given the target pitch angle θ _d , collect the state values of the underwater glider {v ₁ , v ₃ , ω ₂ , θ} is input to the deep reinforcement learning neural network model to obtain the control quantity (the speed command at _t of the moving mass block on the x-axis) to realize the attitude control of the underwater glider.

Figure 4 is the episode reward of the training process of the example attitude controller. The evaluation of the training effect is mainly measured by the episode reward value. After a certain period of training, the larger the average reward value is, the better the training effect is. As shown in Figure 4, the round reward continues to rise. After 900 rounds of training, the round reward value is basically stable at around 850, indicating that the controller has learned a good strategy.

Fig. 5 is an example of the application stage, which shows the change process of the pitch angle during the gliding process. Set the initial pitch angle to 0° to start gliding, and the desired pitch angle is -25°. After 16s, the pitch angle reaches the desired value, and the steady-state error is 0.06° during stable gliding. It can be seen that the error between the pitch angle and the desired angle during stable gliding is small, and it can be considered that the glider can glide continuously according to the desired trajectory.

Although the embodiments of the present invention have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present invention, and those of ordinary skill in the art will not depart from the principles and spirit of the present invention Variations, modifications, substitutions, and alterations to the above-described embodiments are possible within the scope of the present invention without departing from the scope of the present invention.

Claims (8)

1. an underwater glider attitude control method based on deep reinforcement learning, is characterized in that: comprise the following steps: Step 1: establish the current decision-making neural network, the current evaluation neural network, the target decision-making neural network and the target evaluation neural network; the decision-making neural network uses the state value of the underwater glider as the input, and the control value a of the underwater glider as the output action; The evaluation neural network takes the state value and control quantity of the underwater glider as the input, and the evaluation value is the output; initializes the parameters of 4 neural networks, initializes the memory bank and the data buffer; Step 2: Obtain the state value s _t of the underwater glider at the current moment, input the state value into the current decision-making neural network to calculate the output action a _t of the attitude controller at the current moment _, and apply the output action at to the underwater glider simulator, Obtain the state value s _t+1 of the underwater glider at the next moment; according to the state s _t at the current moment, the action a _t at the current moment, the target pitch angle θ _d and the state s _t+1 at the next moment, calculate the current moment’s reward value r _t ; Step 3: Store the state (s _t , at , r _t , s _{t +1} ) obtained in step 2 as a set of empirical data units in the memory bank, and increment t by 1; judge t and the set memory The relationship between the library size n, if t<n, use the updated s _t to return to step 2, until the number of empirical data units stored in the memory library meets the requirement of n, then enter step 4; Step 4: Sampling a specified number of N empirical data units from the memory bank and storing them in the buffer; Step 5: Sample m experience data units from N experience data units in the buffer; process the sampled m experience data units one by one according to the following process: For a certain empirical data unit (s _t , at _t , r _t , s _{t +1} ), input the state s _t and the action signal at into the current evaluation neural network to obtain the evaluation value Q of the current evaluation neural network; The state s _t+1 at one moment is input into the target decision-making neural network, and the action signal μ' of the actuator output by the target decision-making neural network is obtained; the next moment state s _t+1 and the action value μ' output by the target decision-making neural network are input Go to the target evaluation neural network to obtain the evaluation value Q' of the target evaluation neural network; Using the evaluation value Q of the current evaluation neural network, the evaluation value Q' of the target evaluation neural network, and the loss function L of the evaluation neural network, the gradient value of the current evaluation neural network is calculated

Step 6: Update the current evaluation neural network: according to the gradient of the current evaluation neural network

Self-increase for the current evaluation neural network parameter σ ^Q

Update, α is the learning rate of the evaluation neural network; Step 7: Calculate the gradient of the current decision neural network

Step 8: Current decision neural network update: according to the gradient of the current decision neural network

Self-increase for the current decision-making neural network parameter σ ^μ

Update, β is the learning rate of the decision neural network; Step 9: update the target evaluation neural network and the target decision neural network: update the target evaluation neural network parameters according to the updated current evaluation neural network parameters, and update the target decision neural network parameters according to the updated current decision neural network parameters; Step 10: Determine whether the training times exceed the set training times. If it exceeds the set training times, stop the training and save the parameter values of the 4 neural networks. If it does not exceed the set training times, go back to step 4 and re-store in the memory. The sampled number N of empirical data units in the library are stored in the buffer; Step 11: After obtaining the trained deep reinforcement learning neural network model, apply it to the actual underwater glider in the vertical plane gliding motion, given the target pitch angle, collect the state value of the underwater glider and input it into the deep reinforcement learning neural network model to obtain The control quantity realizes the attitude control of the underwater glider. 2. a kind of underwater glider attitude control method based on deep reinforcement learning according to claim 1, is characterized in that: reward value r _t is taken as: r _t =r ₁ +r ₂ +r ₃ Among them, r ₁ =-λ ₁ (d _t -d _t-1 ), r ₁ represents the reward value obtained when the current pitch angle is far away from or close to the target pitch angle; r ₂ =-λ ₂ (w _t -w _t-1 ) , r ₂ represents the reward value obtained from the change of angular velocity; r ₃ takes the value according to the current pitch angle, if: |d _t -d _t-1 |＜0.1°, r ₃ is a positive reward, if the current time t When the pitch angle θ<-90° or θ>90°, r ₃ is a negative number, indicating punishment, d _t is the difference between the pitch angle θ and the target pitch angle θ _d at time t, and w _t is the The magnitude of the pitch rate at time t, λ ₁ and λ ₂ are set coefficients. 3. a kind of underwater glider attitude control method based on deep reinforcement learning according to claim 1 is characterized in that: in step 4, by the experience replay mechanism based on priority, the N experience data units in the buffer zone are prioritized The level is sorted and stored in the SumTree data structure; the value of the head node in the SumTree data structure is the sum of the priorities of the experience data units represented by all nodes, denoted as p _total . 4. a kind of underwater glider attitude control method based on deep reinforcement learning according to claim 3, is characterized in that: in step 5, sample m empirical data units by following process, [0, p _total ] is equally divided into number is a small area of m, random and uniform sampling is performed in each small area, and the experience data units corresponding to the sampling obtained priorities are obtained, and m experience data units are obtained in total. 5. a kind of underwater glider attitude control method based on deep reinforcement learning according to claim 1 is characterized in that: in step 5, the gradient value of current evaluation neural network is obtained

The process is: The loss function L of the current evaluation neural network is:

Among them, ω _i is the importance sampling weight of the i-th empirical data unit, and δ _i is: δ _i =r _i +γQ'(s _i+1 ,μ'(s _i+1 |σ ^μ' )σ ^Q' )-Q(s _i ,a _i |σ ^Q ) ri is the reward value in the _i -th empirical data unit, s _i+1 is the next moment state in the i-th empirical data unit, σ ^μ' is the parameter of the target decision-making neural network, μ'(s _i+1 |σ ^μ' ) is the action signal of the actuator output by s _i+1 through the target decision-making neural network; σ ^Q' is the parameter of the target evaluation neural network, Q'(s _i+1 ,μ'(s _i+1 | σ ^μ' )σ ^Q' ) is the size of the evaluation value obtained by passing μ'(s _i+1 |σ ^μ' ) and s _i+1 through the target evaluation neural network; s _i is the current value in the i-th empirical data unit. Time state, a _i is the action signal at the current time state in the i-th empirical data unit, σ ^Q is the parameter of the current evaluation neural network, Q(s _i , a _i |σ ^Q ) is the evaluation of the current evaluation neural network. value; γ is the discount factor; The gradient of the loss function L of the current evaluation neural network is obtained as:

6. a kind of underwater glider attitude control method based on deep reinforcement learning according to claim 1 is characterized in that: in step 7, the gradient of the current decision-making neural network

The calculation formula is:

in

Indicates the evaluation value of the current evaluation neural network

For the gradient of the parameter α, μ(s _i ) is the action signal of the actuator output by s _i through the current decision-making neural network;

Represents the current decision neural network action

Gradient of the current decision neural network parameter σ ^μ . 7. a kind of underwater glider attitude control method based on deep reinforcement learning according to claim 1 is characterized in that: in step 9, target evaluation neural network and target decision neural network update formula are:

Among them, σ _t+1 ^Q represents the updated current evaluation neural network parameters, σ _t ^Q' represents the target evaluation neural network parameters to be updated, σ _t+1 ^Q' represents the updated target evaluation neural network parameters; σ _{t+ 1} ^μ represents the updated current decision-making neural network parameters, σ _t ^μ' represents the target decision-making neural network parameters to be updated, σ _t+1 ^μ' represents the updated target decision-making neural network parameters; τ ₁ is the target evaluation neural network parameter. update rate, τ ₂ is the update rate of the target decision-making neural network. 8. A kind of underwater glider attitude control method based on deep reinforcement learning according to claim 1, is characterized in that: decision-making neural network adopts the state values v ₁ , v ₃ , ω ₂ , θ of the underwater glider as input, v ₁ , v ₃ , ω ₂ , θ are the speed of the underwater glider in the x and z-axis directions, the pitch angular velocity and the pitch angle, respectively, where the x-axis is the forward axis of the body coordinate system, and the z-axis is the coordinate axis perpendicular to the body plane; The control quantity a of the underwater glider is the speed command of the moving mass on the x-axis.

Images