CN114296350A - A fault-tolerant control method for unmanned ships based on model reference reinforcement learning - Google Patents

Abstract

The invention discloses a fault-tolerant control method for an unmanned ship based on model reference reinforcement learning. The method includes: analyzing the uncertainty factors of the unmanned ship, and constructing a nominal dynamic model of the unmanned ship; based on the maximum entropy Actor-Critic method, according to the actual unmanned ship system, the state variable difference of the nominal dynamic model of the unmanned ship and the output of the nominal controller of the unmanned ship , build a fault-tolerant controller based on model reference reinforcement learning; according to the control task requirements, build a reinforcement learning evaluation function and a control strategy model and train the fault-tolerant controller to obtain the trained control strategy. By using the present invention, the safety and reliability of the unmanned ship system can be significantly improved. As a fault-tolerant control method for an unmanned ship based on model reference reinforcement learning, the present invention can be widely used in the field of unmanned ship control.

Description

A fault-tolerant control method for unmanned ships based on model reference reinforcement learning

technical field

The invention relates to the field of unmanned ship control, in particular to an unmanned ship fault-tolerant control method based on model reference reinforcement learning.

Background technique

With significant advancements in guidance, navigation, and control technologies, the application of autonomous surface vehicles (ASVs) has taken over a pivotal part of aviation. In most applications, unmanned ships are expected to operate safely without human intervention for extended periods of time. Therefore, unmanned ships are required to have sufficient safety and reliability attributes to provide normal operation and avoid catastrophic consequences. However, unmanned ships are prone to problems such as failure, system component degradation, sensor failure, etc., and thus experience performance degradation, instability, and even catastrophic losses.

SUMMARY OF THE INVENTION

In order to solve the above technical problems, the purpose of the present invention is to provide a fault-tolerant control method for unmanned ships based on model reference reinforcement learning, which can restore the system performance or keep the system running after encountering a fault, thereby significantly improving the safety and reliability of the system sex.

The first technical solution adopted by the present invention is: a fault-tolerant control method for an unmanned ship based on model reference reinforcement learning, comprising the following steps:

S1. Analyze the uncertainty factors of the unmanned ship, and construct a nominal dynamic model of the unmanned ship;

S2. Based on the nominal dynamics model of the unmanned ship, design the nominal controller of the unmanned ship;

S3. Actor-Critic method based on maximum entropy, according to the actual unmanned ship system, the state variable difference of the unmanned ship nominal dynamic model and the output of the unmanned ship nominal controller, construct fault-tolerant control based on model reference reinforcement learning device;

S4. According to the control task requirements, build a reinforcement learning evaluation function and a control strategy model and train a fault-tolerant controller to obtain a trained control strategy.

Further, the formula of the nominal dynamic model of the unmanned ship is expressed as follows:

表示广义坐标向量，v表示广义速度向量，u表示控制力和力矩，M表示惯性矩阵，C(v)包括科氏力和向心力，D(v)表示阻尼矩阵，G(v)表示由于重力和浮力及力矩而产生的未建模动力学，B表示预设的输入矩阵

In the above formula,

Represents generalized coordinate vector, v represents generalized velocity vector, u represents control force and moment, M represents inertia matrix, C(v) includes Coriolis force and centripetal force, D(v) represents damping matrix, G(v) represents due to gravity and Unmodeled dynamics due to buoyancy and moment, B is a preset input matrix

Further, the formula of the nominal controller of the unmanned ship is expressed as follows:

In the above formula, N _m and H _m contain all known constant parameters of the dynamic model of the unmanned ship, η _m represents the generalized coordinate vector of the nominal model, u _m represents the control law, and x _m represents the state of the reference model.

Further, the formula of the fault-tolerant controller is expressed as follows:

u_l表示来自深度学习模块的控制策略，β(v)表示内环动力学中所有模型不确定性的集合，n_v表示广义速度测量值上的噪声矢量，f_v表示作用于广义速度矢量的传感器故障。In the above formula, H _m -L represents the Hurwitz matrix,

u _l represents the control strategy from the deep learning module, β(v) represents the set of all model uncertainties in the inner loop dynamics, n _v represents the noise vector on the generalized velocity measurement, f _v represents the action on the generalized velocity vector Sensor failure.

Further, the formula of the reinforcement learning evaluation function is expressed as follows:

Q _π (s _t ,u _l,t )=T ^π Q _π (s _t ,u _l,t )

In the above formula, u _l,t represents the control excitation from RL, s _t represents the state signal at time step t, T ^π represents the fixed policy, E _π represents the expectation operator, γ represents the discount factor, α represents the temperature coefficient, Q _π (s _t , u _{l, t} ) represents the reinforcement learning evaluation function.

Further, the formula of the control strategy model is expressed as follows:

表示π^old的Q值，D_KL表示KL散度，

表示归一化因子，(s_t,·)表示控制策略，点表示省去自变量的写法。In the above formula, Π represents the strategy set, π ^old represents the strategy of the previous update,

represents the Q value of π ^old , D _KL represents the KL divergence,

represents the normalization factor, (s _t , ) represents the control strategy, and the dot represents the writing without independent variables.

Further, according to the control task requirements, the step of building a reinforcement learning evaluation function and a control strategy model and training a fault-tolerant controller to obtain a trained control strategy specifically includes:

S41 , building a reinforcement learning rating function and a model policy model for the fault-tolerant controller based on model reference reinforcement learning according to the requirements of the control task.

S42, train the fault-tolerant controller based on model reference reinforcement learning to obtain an initial control strategy;

S43, inject a fault into the unmanned ship system, retrain the initial control strategy, and return to step S41, until the evaluation function network model and the control strategy model of the reinforcement learning converge.

Further, it also includes:

A dual evaluation function model is introduced, and the entropy value of the strategy is added to the expected return function of the control strategy, where R _t is the reward function, and R _t =R(s _t ,u _l,t ).

The beneficial effects of the method of the invention are as follows: the invention proposes a fault-tolerant control algorithm based on reinforcement learning that combines model reference reinforcement learning with fault diagnosis and estimation mechanisms for the unmanned ship system with model uncertainty and sensor faults , considering the low efficiency of Monte Carlo sampling, using the Actor-Critic model to replace the cumulative revenue with the Q function, through the new fault-tolerant control based on reinforcement learning, we ensure that the unmanned ship can learn to adapt to different sensor failures, and in the failure Recover trajectory tracking performance under conditions.

Description of drawings

Fig. 1 is the step flow chart of a kind of unmanned ship fault-tolerant control method based on model reference reinforcement learning of the present invention;

FIG. 2 is a structural block diagram of an Actor-Critic network according to a specific embodiment of the present invention.

Detailed ways

The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The numbers of the steps in the following embodiments are only set for the convenience of description, and the sequence between the steps is not limited in any way, and the execution sequence of each step in the embodiments can be adapted according to the understanding of those skilled in the art Sexual adjustment.

As shown in FIG. 1 , the present invention provides a fault-tolerant control method for an unmanned ship based on model reference reinforcement learning (RL), and the method includes the following steps:

S1. Analyze the inherent uncertainty factors of the unmanned ship, ignore all nonlinear terms in the inner loop dynamics, obtain the linear and decoupled models of the dynamic equation of the generalized velocity vector, and establish the nominal dynamics of the unmanned ship Model;

The dynamic model is specifically:

是一个广义坐标向量，x_p和y_p表示ASV在惯性系中的水平坐标，

是航向角。v＝[u_p,v_p,r_p]^T∈R³是广义速度向量，u_p和v_p分别为x轴和y轴方向上的线速度，r_p为航向角速率。u＝[τ_u,τ_r]∈R³控制力和力矩，G(v)＝[g₁(v),g₂(v),g₃(v)]^T∈R³是由于重力和浮力及力矩而产生的未建模动力学，M∈R^3×3是带有M＝M^T＞0的惯性矩阵且in

is a generalized coordinate vector, x _p and y _p represent the horizontal coordinates of the ASV in the inertial frame,

is the heading angle. v ₌ [up _p , v _p , rp ] ^T _∈ R ³ is a generalized velocity vector, up and v _p are the linear velocities in the x-axis and y-axis directions, respectively, and _rp is the heading angular rate. u=[τ _u ,τ _r ]∈R ³ controls forces and moments, G(v)=[g ₁ (v),g ₂ (v),g ₃ (v)] ^T ∈ R ³ is due to gravity and buoyancy and the unmodeled dynamics resulting from the moment, M∈R ^3×3 is the inertia matrix with M=M ^T > 0 and

矩阵C(v)＝-C^T(v)包含科氏力和向心力，由下式给出：in

The matrix C(v)=- ^CT (v) contains the Coriolis and centripetal forces and is given by:

where C ₁₃ (v)=-M ₂₂ vM ₂₃ r, C ₂₃ (v)=M ₁₁ u. Damping matrix

输入矩阵

where D ₁₁ (v)=-X _u -X _|u|u |u|-X _uuu u ² , D ₂₂ (v)=-Y _v -Y _|v|v |v|-Y _|r|v | r|, D ₂₃ (v)=-Y _r -Y _|v|r |v|-Y _|r|r |r|, D ₃₂ (v)=-N _v -N _|v|v |v|- N _|r|v |r|, D ₃₃ (v)=-N _r -N _|v|r |v|-N _|r|r |r|, X(·), Y(·), N(· ) is the hydrodynamic coefficient, the definition is detailed in the Manual of Ship Hydrodynamics and Motion Control. rotation matrix

input matrix

Define x=[η ^T v ^T ] ^T , we have

where H(v)=-M ^-1 (C(v)+D(v)) and N=-M ^- 1B.

The state measurements of the ASV system (1) are corrupted by noise and sensor failures and are therefore denoted as y=x+n+f(t), where ^n∈R6 is the measurement noise vector and f(t) ^∈R6 denotes the possible sensor fault vector. In the present invention, we only consider the measurement of the yaw rate r _p due to sensor failure, so f(t)=[0,0,0,0,0,f _r (t)] ^T . The sensor fault f _r (t) is given by:

(k是故障的演化速率)的时间剖面。注意如果传感器故障的发生是突然的，例如偏置故障,k→∞。本发明的目的便是设计一个控制器，允许状态x在存在模型不确定性、可能的传感器故障和测量噪声的情况下跟踪由x_r表示的参考状态轨迹。f _r (t) = β(tT _f )φ(tT _f ), where φ(tT _f ) is the unknown function of sensor failure at instant T and β(tT _f ) is β(tT for t < T _f ) _f )=0 and t>T _f

(k is the evolution rate of the fault) time profile. Note that if the sensor failure occurs suddenly, such as a bias failure, k→∞. The purpose of the present invention is to design a controller that allows state x to track a reference state trajectory represented by _xr in the presence of model uncertainty, possible sensor failures and measurement noise.

S2. Based on the nominal dynamics model, design the nominal controller of the unmanned ship to ensure the basic stability of the unmanned ship system under the premise of no faults. Analyze the nominal model of the unmanned ship.

The nominal controller design process is:

The proposed RL-based FTC algorithm follows a model reference control structure. For most ASV systems, accurate nonlinear dynamic models are rarely available, and the main uncertainties come from M, C(v) and D(v) due to fluid mechanics, and G(v) due to gravity and buoyancy and moments ). Despite the uncertainties in ASV dynamics, based on known information on ASV dynamics, a nominal model can still be used (5). The nominal model of the uncertain ASV model (5) is as follows:

是标称模型的广义坐标向量。本发明中，M_m是由M_m＝diag{M₁₁,M₂₂,M₃₃}得出的，H_m＝M_m ^-1D_m由D_m＝diag{-X_u,-Y_v,-N_r}和N_m＝M_m ^-1B得到。因此在标称模型中，忽略了内环动力学中的所有非线性项，因此最终得到了广义速度矢量v动力学方程的线性解耦模型。由于已知标称模型(6)的动力学，因此可以设计控制律u_m，以允许标称系统(6)的状态收敛到参考信号x_r，如当t→∞时||x_m-x_r||₂→0。这种控制律u_m也可被整个ASV动力学(5)用作标称控制器。where N _m and H _m contain all known constant parameters of ASV kinetics (5),

is the generalized coordinate vector of the nominal model. In the present invention, M _m is derived from M _m =diag{M ₁₁ , M ₂₂ , M ₃₃ }, and H _m =M _m ^-1 D _m is derived from D _m =diag{-X _u ,-Y _v ,- N _r } and N _m =M _m ^-1 B are obtained. Therefore, in the nominal model, all nonlinear terms in the dynamics of the inner loop are ignored, and thus a linearly decoupled model of the dynamics equation of the generalized velocity vector v is finally obtained. Since the dynamics of the nominal model (6) are known, the control law _um can be designed to allow the state of the nominal system (6) to converge to the reference signal x _r as ||x _m -x as t→∞ _r || ₂ → 0. This control law _um can also be used as a nominal controller by the entire ASV dynamics (5).

In a model-referenced control structure, the goal is to design a control law that allows the state of (5) to track the state trajectory of the nominal model (6). The overall control law of the ASV system (5) has the following expression:

u=u _b +u _l (7)

where u _b is the nominal of the model-based approach and u _l is the control policy from the deep learning module. The baseline control u _b is used to ensure some basic performance (ie local stability), while u _l is used to compensate for all system uncertainties and sensor failures.

S3. The Actor-Critic method based on maximum entropy takes the difference between the state variables of the actual unmanned ship system and the nominal model and the output of the nominal controller as input to construct a fault-tolerant controller based on model reference reinforcement learning.

Refer to Figure 2 for the network block diagram of Actor-Critic. The specific derivation process of the fault-tolerant controller is as follows:

The formulation of RL is based on a Markov Decision Process MDP:=<S,U,P,R,γ> represented by a tuple, where S is the state space, U specifies the operation/input space, and P:S×U×S →R defines the transition probability, R:S×U→R is a reward function, and γ∈[0,1) is a discount coefficient. In MDP, the state vector s ∈ S contains all available signals that affect the RL control u _l ∈ U. For the tracking control of the ASV system in the present invention, the transition probability is characterized by the ASV dynamics in (1) and the reference signal _xr . In RL, control policies are learned using data samples collected in the discrete time domain. Let s _t be the state signal s at time step t, and accordingly, u _l,t be the input to the RL-based control at time step t. The RL algorithm in the present invention aims to maximize an action value function, also known as the Q-function, as follows:

且V_π(s_t+1)称为策略π下s_t+1的状态值函数，其中where R _t is the reward function, R _t =R(s _t ,u _l,t ),

And V _π (s _t +1) is called the state value function of s _t +1 under policy π, where

是策略的熵，α是温度参数。RL中的控制策略π(u_l,t|s_t)是选择行动u_l,t∈U在状态s_t∈S下的概率。在本发明中，采用满足高斯分布的控制策略，即where π(u _l,t |s _t ) is the control strategy,

is the entropy of the policy and α is the temperature parameter. The control policy π(u _l,t |s _t ) in RL is the probability of choosing action u _l,t ∈ U in state s _t ∈ S. In the present invention, a control strategy that satisfies the Gaussian distribution is adopted, that is,

π(u _l |s)=N(u _l (s),σ) (10)

where N(·,·) represents the Gaussian distribution, u _l (s) is the mean, and σ is the covariance matrix. The covariance matrix σ controls the exploration performance in the learning phase.

The goal of RL is to find an optimal control policy π ^* that maximizes Q _π (s _t ,u _l,t ) in (8), i.e.

π ^* = argmaxQ _π (s _t ,u _l,t ) (11)

Note that the variance σ ^* will converge to 0, and once the optimal policy π ^* (u _l ^* |s) = N(u _l ^* (s),σ ^* ), the mean function u _l ^* (s) will be The learned optimal control law deep neural network Q _θ (s _t , u _l , t) is called critic, and the control strategy π _φ (u _{l, t} |s _t ) is called actor. The uncertain inner loop dynamics of the ASV model is rewritten as:

where β(v) is the set of all model uncertainties in the inner loop dynamics. Assume that the uncertainty term β(v) is bounded. Let e _v = _vvm , according to (6) and (12), the error dynamics are:

Under healthy conditions, the model uncertainty term β(v) can be fully compensated using the learning-based control _ul . This means that ||e _v (t)|| ₂ ≤ ε as t→∞, where ε is some positive small constant. In the event of a sensor failure, the error signal _ev will be greater than ε. An inexperienced idea of learning-based fault tolerant control (FTC) is to treat sensor failures as part of external disturbances. However, treating sensor failures as disturbances will lead to conservative learning-based control, such as robust control. Therefore, we introduce a fault diagnosis and estimation mechanism that allows learning-based control to adapt to different scenarios: healthy and unhealthy conditions.

作为故障跟踪误差向量。在实际应用中，

是可测量的，而不是e_v。最后，介绍了以下故障诊断和估计机制：Let y _v =v+n _v +f _v , where n _v represents the noise vector on the generalized velocity measurement, and correspondingly, f _v is the sensor fault acting on the generalized velocity vector. Furthermore, we define

as the fault tracking error vector. In practical applications,

is measurable, not e _v . Finally, the following fault diagnosis and estimation mechanisms are introduced:

作为传感器故障发生和强度的指示器。设

得到where L is chosen to be H _m -L Hurwitz. Signal

As an indicator of the occurrence and intensity of sensor failures. Assume

get

u_l表示来自深度学习模块的控制策略，β(v)表示内环动力学中所有模型不确定性的集合，n_v表示广义速度测量值上的噪声矢量，f_v表示作用于广义速度矢量的传感器故障。In the above formula, H _m -L represents the Hurwitz matrix,

u _l represents the control strategy from the deep learning module, β(v) represents the set of all model uncertainties in the inner loop dynamics, n _v represents the noise vector on the generalized velocity measurement, f _v represents the action on the generalized velocity vector Sensor failure.

S4. Design a corresponding callback function according to the control task requirements, and use a fully connected network to build a reinforcement learning evaluation function model (Q-value) and a control strategy model.

The callback function, the learning evaluation function and the control strategy model are derived as follows:

分别代表ASV状态、标称控制器激发、来自RL的控制激发以及时间步长t处故障诊断和估计机制的输出。因此，在时间步长t处的状态信号s表示为：

RL的训练学习过程将重复执行策略评估和策略改进。在策略评估中，Q-value是通过Bellman操作Q_π(s_t,u_l,t)＝T^πQ_π(s_t,u_l,t)得到的，其中RL-based fault-tolerant control is obtained using the outputs of fault diagnosis and estimation mechanisms. RL uses data samples (including input and state data) to learn control policies at discrete time steps. The sampling time step is assumed to be fixed, denoted by δt. Without loss of generality, let y _t , u _b,t , u _l,t , and

represent the ASV state, the nominal controller excitation, the control excitation from the RL, and the output of the fault diagnosis and estimation mechanism at time step t, respectively. Therefore, the state signal s at time step t is expressed as:

The training learning process of RL will repeatedly perform policy evaluation and policy improvement. In policy evaluation, Q-value is obtained by Bellman operation Q _π (s _t ,u _l,t )=T ^π Q _π (s _t ,u _l,t ), where

In the above formula, u _l,t represents the control excitation from RL, s _t represents the state signal at time step t, T ^π represents the fixed policy, E _π represents the expectation operator, γ represents the discount factor, α represents the temperature coefficient, Q _π (s _t , u _{l, t} ) represents the reinforcement learning evaluation function.

In policy refinement, the policy is updated by:

表示π^old的Q值，D_KL表示Kullback-Leibler(KL)散度，

表示归一化因子。通过数学运算，目标被转化为where Π represents the policy set, π ^old represents the last updated policy,

represents the Q value of π ^old , D _KL represents the Kullback-Leibler (KL) divergence,

represents the normalization factor. Through mathematical operations, the goal is transformed into

S5. Introduce the idea of a dual-evaluation function model into the evaluation function training framework, and at the same time add the entropy value of the strategy to the expected return function of the control strategy to improve the training efficiency of reinforcement learning.

The derivation process of the double evaluation function model:

θ_K∈R^1×(L+1)，θ_i∈R^(L)×(L+1)，其中dim_s表示状态s的尺寸，dim_u表示输入u_l的尺寸。The Q function is parameterized by θ and denoted by Q _θ (s _t , u _l,t ). The parameterized policy is denoted by π _φ (u _l,t |s _t ), where φ is the set of parameters to be trained. Note that both θ and φ are a set of parameters whose dimensions are determined by the deep neural network settings. For example, if Q _θ is represented by an MLP with K hidden layers and L neurons in each hidden layer, the parameter set θ is θ = {θ ₀ , θ ₁ ,...,θ _K } and in 1≤ i≤K-1 on

θ _K ∈ R ^1×(L+1) , θ _i ∈ R ^(L)×(L+1) , where dim _s represents the size of state s, and dim _u represents the size of input u _l .

The whole training process is offline, and data samples are collected at each time step t+1, such as the input u _l,t from the previous time step, the state of the previous time step s _t , the reward R _t and the current state s _t+1 . These historical data will be stored in memory pool D as a tuple (s _t , u _l,t , R _t , s _t+1 ). At each policy evaluation or improvement step, we randomly sample a batch of historical data B from memory pool D for training parameters θ and φ. When starting training, we apply the nominal control strategy _ub to the ASV system to collect initial data D ₀ , as shown in Algorithm 1. The initial data set D ₀ is used for the initial fitting of the Q-function. After initialization, u _b and the newly updated reinforcement learning policy π _φ (u _l,t |s _t ) are executed to run the ASV system.

The parameter θ of the Q-function is trained to minimize the Bellman residual:

其中

是将缓慢更新的目标参数。DNN参数θ是通过将随机梯度下降法应用于修正数据批次B上的(15)而获得的，数据批次B的大小由|B|表示。本发明中使用了两个分别由θ₁和θ₂参数化的评价。引入这两个评价是为了减少评价神经网络训练中的高估问题。在双评价函数下，目标值Y_target为：where (s _t , u _{l, t} ) ~ D means a sample (s _t , u _{l, t} ) that we randomly select from memory pool D, and

in

is the target parameter that will be updated slowly. The DNN parameters θ are obtained by applying stochastic gradient descent to (15) on the revised data batch B, the size of which is denoted by |B|. Two evaluations parameterized by θ ₁ and θ ₂ , respectively, are used in the present invention. These two evaluations are introduced to reduce the overestimation problem in evaluating neural network training. Under the double evaluation function, the target value Y _target is:

The policy improvement step uses the data samples in memory pool D to minimize the following parameterized objective function:

The parameter φ is trained to minimize using stochastic gradient descent. In the training phase, the actor neural network is expressed as:

是要学习的参数化控制律，

是探测噪声标准偏差，ξ～N(0,I)是探测噪声，“⊙”是哈达玛积。注意，探测噪声ξ只适用于训练阶段，一旦训练完成，只需要在运用中的

因此，在训练阶段的u_l等价于u_l,φ。一旦训练结束，得到

in

is the parametric control law to be learned,

is the standard deviation of the detection noise, ξ～N(0,I) is the detection noise, and “⊙” is the Hadamard product. Note that the detection noise ξ is only applicable in the training phase, once the training is completed, only the

Therefore, u _l in the training phase is equivalent to u _l,φ . Once the training is over, get

The temperature parameter α is also updated during the training phase. Its update is obtained by minimizing the following objective function:

为策略的熵值。本发明中设置

其中“2”表示动作维度。in

is the entropy value of the strategy. set in the present invention

where "2" represents the action dimension.

S6. In the case of no fault, the controller based on model reference reinforcement learning is trained to obtain an initial control strategy, so as to ensure the robustness of the overall controller to model uncertainty.

S7. Inject faults into the unmanned ship system, and retrain the obtained initial control strategy based on model reference reinforcement learning to realize the adaptability of the overall controller to some sensor faults.

S8. Repeat step S6 and step S7 continuously under different initial state conditions until the evaluation function network model and the control strategy model of the reinforcement learning converge.

和

分别初始化参数θ₁，θ₂，用φ表示actor网络；2)为目标参数指定值：

3)运行u_l＝0时公式(5)中的u_b，得到数据集D₀；4)结束学习阶段的探索，使用数据集D₀训练初始critic参数θ₁ ⁰，

5)初始化记忆池D←D₀；6)为critic参数与其目标指定初始值：θ₁←θ₁ ⁰，

7)重复；8)开始循环，每个数据收集步骤执行操作；9)根据π_φ(u_l,t|s_t)选择一个动作u_l,t；10)运行标称系统(6)和整个系统(5)以及故障诊断和估计机制(14)&收集s_t+1＝{x_t+1,x_m,t+1,u_b,t+1}；11)D←D∪{s_t,u_l,t,R(s_t,u_l,t),s_t+1}；12)结束循环；13)开始循环，每个梯度更新步骤执行动作；14)从D中抽取一批数据B；15)θ_j←θ_j-ι_Q▽_θJ_Q(θ_j)，且j＝1,2；16)φ←φ-ι_π▽_φJ_π(φ)；17)α←α-ι_α▽_αJ_α(α)；18)

且j＝1,2；19)结束循环；20)直至收敛(如J_Q(θ)＜一个小阈值)。在该算法中，ι_Q，ι_π和ι_α是正学习率(标量)，κ＞0是常数标量。Specifically, the training process of steps S6-S8 is as follows: 1) is

and

Initialize the parameters θ ₁ and θ ₂ respectively, and use φ to represent the actor network; 2) Specify values for the target parameters:

3) Run u _b in formula (5) when u _l = 0 to obtain the data set D ₀ ; 4) End the exploration of the learning phase, use the data set D ₀ to train the initial critic parameter θ ₁ ⁰ ,

5) Initialize memory pool D←D ₀ ; 6) Specify initial values for critic parameters and their targets: θ ₁ ←θ ₁ ⁰ ,

7) Repeat; 8) Start the loop, performing actions at each data collection step; 9) Choose an action u _l,t according to _πφ (u _l,t |s _t ); 10) Run the nominal system (6) and the entire System (5) and Fault Diagnosis and Estimation Mechanism (14) & Collection s _t +1 = {x _t+1 , x _{m, t+1} , u _{b, t+1} }; 11) D←D∪{s _t ,u _l,t ,R(s _t ,u _l,t ),s _t+1 }; 12) End the loop; 13) Start the loop, each gradient update step performs actions; 14) Extract a batch of data from D B; 15) θ _j ←θ _j -ι _Q ▽ _θ J _Q (θ _j ), and j=1,2; 16) φ←φ-ι _π ▽ _φ J _π (φ); 17) α←α- ι _α ▽ _α J _α (α); 18)

and j=1,2; 19) end the loop; 20) until convergence (eg J _Q (θ) < a small threshold). In this algorithm, ι _Q , ι _π and ι _α are positive learning rates (scalars), and κ > 0 is a constant scalar.

A fault-tolerant control system for unmanned ships based on model reference reinforcement learning, including:

The dynamic model building module is used to analyze the uncertain factors of the unmanned ship and construct the nominal dynamic model of the unmanned ship;

The controller design module, based on the nominal dynamics model of the unmanned ship, designs the nominal controller of the unmanned ship;

Fault-tolerant controller building module, based on the Actor-Critic method of maximum entropy, according to the actual unmanned ship system, the state variable difference of the unmanned ship's nominal dynamic model and the output of the unmanned ship's nominal controller, the model-based reference enhancement is constructed. Learned fault-tolerant controllers;

The training module is used to build a reinforcement learning evaluation function and a control strategy model and train a fault-tolerant controller according to the requirements of the control task, and obtain the trained control strategy.

The contents in the above method embodiments are all applicable to the present system embodiments, the specific functions implemented by the present system embodiments are the same as the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

A fault-tolerant control device for unmanned ships based on model reference reinforcement learning:

at least one processor;

at least one memory for storing at least one program;

When the at least one program is executed by the at least one processor, the at least one processor implements the above-mentioned method for fault-tolerant control of an unmanned ship based on model reference reinforcement learning.

The contents in the above method embodiments are all applicable to the present device embodiments, the specific functions implemented by the present device embodiments are the same as the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

A storage medium storing processor-executable instructions, wherein the processor-executable instructions, when executed by the processor, are used to implement the above-mentioned model reference reinforcement learning-based unmanned ship Fault-tolerant control methods.

The contents in the foregoing method embodiments are all applicable to this storage medium embodiment, and the specific functions implemented by this storage medium embodiment are the same as those of the foregoing method embodiments, and the beneficial effects achieved are also the same as those achieved by the foregoing method embodiments. same.

The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments, and those skilled in the art can make various equivalent deformations or replacements without departing from the spirit of the present invention. , these equivalent modifications or substitutions are all included within the scope defined by the claims of the present application.

Claims (8)

1. A fault-tolerant control method of an unmanned ship based on model reference reinforcement learning is characterized by comprising the following steps:

s1, analyzing uncertainty factors of the unmanned ship and constructing a nominal dynamics model of the unmanned ship;

s2, designing a nominal controller of the unmanned ship based on the name meaning dynamic model of the unmanned ship;

s3, constructing a fault-tolerant controller based on model reference reinforcement learning according to the actual unmanned ship system, the state variable difference value of the unmanned ship name-sense dynamic model and the output of the unmanned ship nominal controller by an Actor-criticic method based on the maximum entropy;

and S4, building a reinforcement learning evaluation function and a control strategy model according to the control task requirements, and training a fault-tolerant controller to obtain a trained control strategy.

2. The unmanned ship fault-tolerant control method based on model reference reinforcement learning of claim 1, wherein the formula of the unmanned ship nominal dynamics model is as follows:

in the above formula, the first and second carbon atoms are,

representing a generalized coordinate vector, v representing a generalized velocity vector, u representing a control force and moment, M representing an inertia matrix, c (v) comprising coriolis force and centripetal force, d (v) representing a damping matrix, g (v) representing unmodeled dynamics due to gravity and buoyancy and moment, and B representing a preset input matrix.

3. The unmanned ship fault-tolerant control method based on model reference reinforcement learning as claimed in claim 2, wherein the formula of the unmanned ship nominal controller is as follows:

in the above formula, N_mAnd H_mComprising all known constant parameters, η, of the unmanned ship dynamics model_mGeneralized coordinate vector, u, representing a nominal model_mRepresenting the control law, x_mRepresenting the state of the reference model.

4. The unmanned ship fault-tolerant control method based on model reference reinforcement learning as claimed in claim 3, wherein the formula of the fault-tolerant controller is as follows:

in the above formula, H_mL represents a Hurwitz matrix,

u_lrepresents the control strategy from the deep learning module, β (v) represents the set of all model uncertainties in the inner-loop dynamics, n_vRepresenting a noise vector on the generalized velocity measurement, f_vIndicating a sensor fault acting on the generalized velocity vector.

5. The unmanned ship fault-tolerant control method based on model reference reinforcement learning as claimed in claim 4, wherein the formula of the reinforcement learning evaluation function is expressed as follows:

Q_π(s_t,u_l,t)＝T^πQ_π(s_t,u_l,t)

in the above formula, u_l,tIndicating the control excitation, s, from the RL_tRepresenting the state signal at a time step T, T^πDenotes a fixed policy, E_πRepresenting the desired operator, gamma representing the discount factor, alpha representing the temperature coefficient, Q_π(s_t,u_l,t) Representing a reinforcement learning evaluation function.

6. The unmanned ship fault-tolerant control method based on model reference reinforcement learning as claimed in claim 4, wherein the formula of the control strategy model is as follows:

in the above formula, pi represents a policy set, pi^oldRepresents the previous onePolicy of secondary update, Q^πoldDenotes pi^oldQ value of (D)_KLThe dispersion of the KL is expressed,

represents a normalization factor,(s)_tAnd) represents a control strategy.

7. The unmanned ship fault-tolerant control method based on model reference reinforcement learning according to claim 1, wherein the step of constructing a reinforcement learning evaluation function and a control strategy model and training a fault-tolerant controller to obtain a trained control strategy according to control task requirements specifically comprises:

and S41, building a reinforcement learning rating function and a model strategy model for the fault-tolerant controller based on model reference reinforcement learning according to the control task requirements.

S42, training the fault-tolerant controller based on model reference reinforcement learning to obtain an initial control strategy;

and S43, injecting faults into the unmanned ship system, retraining the initial control strategy and returning to the step S41 until the reinforcement learning evaluation function network model and the control strategy model converge.

8. The unmanned ship fault-tolerant control method based on model reference reinforcement learning according to claim 7, further comprising:

and introducing a double-evaluation function model, and adding an entropy value of the strategy into the expected return function of the control strategy.

Images