CN115524964A - A real-time robust guidance method and system for rocket landing based on reinforcement learning - Google Patents

Abstract

The present invention provides a real-time robust guidance method and system for rocket landing based on reinforcement learning. The three-degree-of-freedom motion model of the rocket is constructed according to the force of the rocket's earth landing dynamic descent stage flight, and the three-degree-of-freedom motion model of the rocket is Construct the rocket landing Markov decision-making process model, and after constructing the intelligent agent Agent according to the rocket landing Markov decision-making process model, conduct interactive simulation training between the intelligent agent Agent and the pre-built rocket landing flight simulation environment to obtain the landing control Agent, and The method of guiding the rocket to land and fly according to the real-time control instructions generated by the landing control agent not only has extremely high real-time performance, but also has strong algorithm robustness, and can adapt to a wide range of modeling deviations. It can still guide the rocket to perform a high-precision fixed-point soft landing under certain conditions, and has high application value.

Description

A real-time robust guidance method and system for rocket landing based on reinforcement learning

technical field

The invention relates to the technical field of vertical take-off and landing rocket earth landing guidance, in particular to a method and system for real-time robust guidance of rocket landing based on reinforcement learning.

Background technique

The vertical take-off and landing reusable launch vehicle is a new type of launch vehicle, and it is an effective tool to reduce the cost of space delivery tasks and improve the efficiency of entering space. Rocket sub-stage earth landing guidance refers to the control of the position and velocity of the launch vehicle’s center of mass during the three-degree-of-freedom flight process of the rocket’s return to the earth’s landing center of mass, that is, according to a certain principle or strategy, generate instructions to guide the movement of the rocket’s center of mass, so that the movement process It is a key technology to meet the constraints and make the terminal state meet the predetermined goals to ensure the recovery accuracy of the launch vehicle, reduce fuel consumption and achieve reliable reuse.

The existing rocket sub-stage earth landing guidance methods mainly include establishing the target rocket dynamics model and the corresponding trajectory optimization problem model, using the indirect method or the direct method to solve the online trajectory optimization guidance method of the optimal flight trajectory online, and using the "offline A deep learning-guided method for the strategy of "training + online application". Although the existing guidance methods have certain real-time and optimality, and can realize the recovery and use of launch vehicles to a certain extent, these methods are all model-based guidance methods, and their algorithm efficiency and availability of solution results are heavily dependent on The precision and accuracy of modeling are poor, and once there are unknown factors that cannot be modeled in the environment or the model has deviation and uncertainty interference, the performance of the algorithm and the availability of the solution results will be seriously affected, which will lead to the guidance invalidated.

Contents of the invention

The purpose of the present invention is to provide a real-time robust guidance method for rocket landing based on reinforcement learning, by constructing a three-degree-of-freedom motion model of a rocket based on the force analysis of the rocket's earth landing dynamic descent stage flight, and constructing a rocket landing Marco by combining gaze heuristics. The model of the decision-making process is made, and the intelligent agent agent based on the value function neural network and the strategy neural network is used for interactive simulation training with the rocket landing flight simulation environment, and the landing control agent used to generate the rocket landing guidance control strategy is obtained, and the rocket landing guidance control agent is obtained according to the rocket landing guidance The control strategy generates real-time control instructions and guides the rocket landing flight method. On the basis of effectively solving the application defects of the existing rocket sub-stage earth landing guidance method, it can not only effectively deal with the large deviation of the rocket dynamics model during the return and landing flight of the rocket sub-stage. It has strong robustness to the existing environmental interference, and has high real-time performance, and can guide the rocket sub-stage to perform high-precision fixed-point soft landing under complex and uncertain working conditions.

In order to achieve the above objectives, it is necessary to provide a real-time robust guidance method and system for rocket landing based on reinforcement learning for the above technical problems.

In a first aspect, an embodiment of the present invention provides a method for real-time robust guidance of a rocket landing based on reinforcement learning, the method comprising the following steps:

According to the force of the rocket's earth landing dynamic descent stage flight, a three-degree-of-freedom motion model of the rocket is constructed;

According to the three-degree-of-freedom motion model of the rocket, a Markov decision process model for rocket landing is constructed;

According to the Markov decision process model of the rocket landing, an intelligent agent Agent is constructed, and the intelligent agent Agent is interactively trained with the pre-built rocket landing flight simulation environment to obtain a landing control Agent; the intelligent agent Agent includes a value-based Functional neural network and policy-based neural network;

Generate real-time control instructions according to the landing control Agent, and guide the rocket to land and fly according to the real-time control instructions.

Further, the step of constructing a three-degree-of-freedom motion model of the rocket according to the acting force of the rocket's earth landing power descent stage flight includes:

Taking the landing point of the rocket sub-stage target as the origin, the landing point coordinate system is established; the landing point coordinate system is that the target landing point of the rocket sub-stage landing is the coordinate origin O, and the vertical upward direction of the center of the earth is the coordinate axis Oz, with The main flight direction of the rocket during landing is the coordinate axis Ox, and the coordinate system of the coordinate axis Oy is the direction perpendicular to the xOz plane and forming a right-handed rectangular coordinate system with the coordinate axis Ox and the coordinate axis Oz;

Based on the coordinate system of the landing point, the force analysis is carried out on the rocket flying in the dynamic descent stage of the earth landing, and the corresponding earth gravity, aerodynamic drag and engine thrust are determined;

According to the gravitational force of the earth, aerodynamic drag and engine thrust, the three-degree-of-freedom motion model of the rocket is constructed; the three-degree-of-freedom motion model of the rocket is expressed as:

In the formula,

为发动机开机后的推进剂秒耗量；C_D表示阻力系数；S_ref表示火箭子级的参考面积；ρ₀表示地球海平面处的参考大气密度；h表示火箭子级的飞行高度；h_ref表示参考高度。Among them, r represents the rocket position vector; v represents the rocket velocity vector; m represents the rocket mass; g(r) represents the gravitational acceleration vector received by the rocket; T represents the engine thrust vector; D represents the aerodynamic drag vector; I _sp represents the fuel specific impulse; g ₀ represents the average gravitational acceleration at the sea level of the earth;

C _D represents the drag coefficient; S _ref represents the reference area of the rocket sub-stage; ρ ₀ represents the reference atmospheric density at the earth’s sea level; h represents the flight height of the rocket sub-stage; h _ref Indicates the reference height.

Further, according to the rocket three-degree-of-freedom motion model, the step of constructing the rocket landing Markov decision process model includes:

Based on the idea of gaze heuristic, the state variable of the rocket is converted to obtain the state quantity of the rocket landing Markov decision process model; the state quantity is expressed as:

In the formula,

V _error = VV _sight

Among them, S represents the state quantity of the rocket landing Markov decision process model; r, V and V ₀ represent the rocket position vector, rocket velocity vector and rocket initial velocity respectively; t _go represents the remaining flight time of the rocket; r _z represents the rocket position The Z-axis component of the vector; V _sight represents the sight vector; V _error represents the error between the rocket velocity vector and the sight vector; λ represents the parameter to adjust the size of the sight vector to change with time;

According to the control instruction of the rocket, the action amount of the Markov decision process model of the rocket landing is obtained; the action amount is expressed as:

Among them, A represents the action amount of the rocket landing Markov decision process model; T represents the engine thrust vector; T _x , _Ty and T _z represent the X-axis, Y-axis and Z-axis components of the engine thrust, respectively;

Determine the design principle of the reward function according to the fixed-point soft landing requirement of the rocket, and obtain the reward function of the Markov decision process model of the rocket landing according to the design principle of the reward function;

The continuous rocket landing process is discretized according to a preset period, and the state transition probability of the rocket landing Markov decision process model is determined according to the integral dynamics of the rocket.

Further, the step of constructing an intelligent agent Agent according to the Markov decision process model of the rocket landing includes:

According to the Markov decision process model of the rocket landing, select the proximal strategy optimization algorithm as the reinforcement learning algorithm of the intelligent agent Agent;

Based on the proximal policy optimization algorithm and a multi-layer perceptron model, the value function-based neural network and the policy-based neural network are constructed.

Further, the construction steps of the rocket landing flight simulation environment include:

Based on the rocket three-degree-of-freedom motion model, a rocket landing operation environment is constructed, and corresponding initial value condition generators and flight termination determiners are simultaneously constructed to obtain the rocket landing flight simulation environment.

Further, the described intelligent agent Agent is interactively trained with the pre-built rocket landing flight simulation environment, and the step of obtaining the landing control Agent includes:

Through interactive simulation between the intelligent agent Agent and the rocket landing flight simulation environment, the strategy-based neural network of the intelligent agent Agent is trained until convergence, and the landing control Agent is obtained.

Further, the step of training the strategy-based neural network of the intelligent agent Agent until convergence through the interactive simulation of the intelligent agent Agent and the rocket landing flight simulation environment, and obtaining the landing control Agent includes:

Randomly selecting an initial state to be simulated from a preset initial state space according to the initial value condition generator;

According to the initial state to be simulated, execute the interactive simulation of the intelligent agent Agent and the flight simulation environment, and when the simulation termination condition preset by the flight termination determiner is reached, terminate the simulation flight of the current wheel, and according to The reward function evaluation obtains the cumulative reward value of each state point in the current simulated flight track, and updates the parameters based on the value function neural network of the intelligent agent Agent according to the cumulative reward value;

Predict the expected cumulative return value of each state point in the current simulated flight trajectory according to the value function-based neural network of the updated intelligent agent Agent, and calculate the advantage function according to the cumulative return value and the expected return value, and According to the advantage function, update the parameters based on the policy neural network of the intelligent agent Agent;

Judging whether the policy-based neural network of the intelligent agent Agent reaches the preset convergence condition, if it is reached, then stop the simulation training to obtain the landing control Agent, otherwise, reselect the initial value to be simulated according to the initial value condition generator. state, and start the next round of interactive simulation training.

In the second aspect, the embodiment of the present invention provides a real-time robust guidance system for rocket landing based on reinforcement learning, the system comprising:

The motion model building block is used to construct a three-degree-of-freedom motion model of the rocket according to the force of the rocket's earth landing dynamic descent stage flight;

An optimization model construction module is used to construct a Markov decision process model for rocket landing according to the three-degree-of-freedom motion model of the rocket;

The control strategy training module is used to construct an intelligent agent Agent according to the Markov decision process model of the rocket landing, and carry out interactive training between the intelligent agent Agent and the pre-built rocket landing flight simulation environment to obtain the landing control Agent; Said intelligent agent Agent comprises based on value function neural network and based on strategy neural network;

The rocket landing guidance module is used to generate real-time control instructions according to the landing control agent, and guide the rocket to land and fly according to the real-time control instructions.

In a third aspect, an embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored on the memory and operable on the processor, and the above method is implemented when the processor executes the computer program A step of.

In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

The above-mentioned application provides a real-time robust guidance method and system for rocket landing based on reinforcement learning. Through the method, the three-degree-of-freedom motion model of the rocket is constructed according to the force of the rocket's earth landing dynamic descent stage flight, and Construct the rocket landing Markov decision process model according to the rocket three-degree-of-freedom motion model, and after constructing the intelligent agent Agent according to the rocket landing Markov decision process model, conduct interactive simulation between the intelligent agent Agent and the pre-built rocket landing flight simulation environment The landing control agent is obtained through training, and the technical plan for guiding the rocket to land and fly is based on the real-time control instructions generated by the landing control agent. Compared with the existing technology, the real-time robust guidance method for rocket landing based on reinforcement learning not only has extremely high real-time performance, but also has strong algorithm robustness, and can adapt to a wide range of modeling deviations. Under interference conditions, it can still guide the rocket to a high-precision fixed-point soft landing, which has high application value.

Description of drawings

1 is a schematic diagram of an application scenario of a real-time robust guidance method for rocket landing based on reinforcement learning in an embodiment of the present invention;

Fig. 2 is a schematic flow chart of a real-time robust guidance method for rocket landing based on reinforcement learning in an embodiment of the present invention;

Fig. 3 is a schematic diagram of the landing point coordinate system used to establish the rocket three-degree-of-freedom motion model in the embodiment of the present invention;

Fig. 4 is the structural representation based on the strategy neural network of intelligent agent Agent in the embodiment of the present invention;

Fig. 5 is the structural representation based on the value function neural network of intelligent agent Agent in the embodiment of the present invention;

Fig. 6 is a structural schematic diagram of a real-time robust guidance system for rocket landing based on reinforcement learning in an embodiment of the present invention;

Fig. 7 is an internal structure diagram of a computer device in an embodiment of the present invention.

detailed description

In order to make the purpose, technical solutions and beneficial effects of the present application clearer, the present invention will be described in further detail below in conjunction with the accompanying drawings and embodiments. Obviously, the embodiments described below are part of the embodiments of the present invention and are only used for The present invention is illustrated, but not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

The real-time robust guidance method for rocket landing based on reinforcement learning provided by the present invention can be applied to the earth-return landing guidance of vertical take-off and landing reusable launch vehicles. Based on the overall architecture shown in Figure 1, the engine thrust can be mapped based on the real-time state of the rocket Instructions, and the instructions given have the ability to adapt to large-scale rocket model deviations and environmental disturbances, ensuring that the rocket sub-stages are guided to perform high-precision fixed-point soft landings under complex uncertain working conditions, and by using deep neural networks as The policy network of reinforcement learning, combined with the improved PPO algorithm to learn the policy network, effectively fits the high-dimensional continuous action space instructions, and sets the state quantity guidance by using the gaze heuristic method, and designs different rewards for the terminal and process indicators of the rocket landing trajectory The discount rate is used to speed up the convergence speed of the strategy, thereby ensuring that the rocket landing guidance method has extremely high real-time performance, and the algorithm has strong robustness, which can not only adapt to a wide range of modeling deviations, but also effectively deal with uncertainties in the environment Interference provides a reliable guarantee for guiding the rocket to perform a high-precision fixed-point soft landing, and has high application value; it should be noted that the method of the present invention can be executed by a server that undertakes related functions, and the following embodiments all use the server as the execution body As an example, the real-time robust guidance method for rocket landing based on reinforcement learning of the present invention will be described in detail.

In one embodiment, as shown in FIG. 2 , a real-time robust guidance method for rocket landing based on reinforcement learning is provided, including the following steps:

S11. Construct a rocket three-degree-of-freedom motion model according to the force of the rocket's earth-landing dynamic descent stage flight; wherein, the rocket three-degree-of-freedom motion model can be understood as considering the actual flight conditions and mission goals, establishing a nonlinear, continuous The optimal landing trajectory optimization problem of the rocket fuel is a motion model based on the targeted improvement of the current rocket landing operation model; because the rocket sub-stage is mainly affected by the engine thrust, the earth's gravity and the dense atmospheric environment during the final landing process. In order to simplify the problem as much as possible on the basis of ensuring the reliability of the research problem, this embodiment mainly considers the force of the rocket's earth landing dynamic descent stage flight to construct a corresponding rocket three-degree-of-freedom motion model; specifically, The steps of constructing a three-degree-of-freedom motion model of the rocket according to the force of the rocket's earth landing power descent stage flight include:

The landing point coordinate system is established with the rocket sub-stage target landing point as the origin; the landing point coordinate system is shown in Figure 3, which is the coordinate origin O with the target landing point of the rocket sub-stage landing, and the vertical upward direction from the center of the earth is the coordinate axis Oz, the main flight direction of the rocket during landing is the coordinate axis Ox, and the direction perpendicular to the plane xOz plane and forming a right-handed rectangular coordinate system with the coordinate axis Ox and the coordinate axis Oz is the coordinate system of the coordinate axis Oy; it needs to be explained What is most important is that the final flight stage in the rocket sub-stage landing process considered by the present invention has the characteristics of short flight time and narrow flight space, the curvature of the earth's surface and the influence of the earth's rotation can be ignored, and the earth's surface is regarded as a plane, so In order to more intuitively describe the flight process of the rocket sub-stage and simplify the solution of the problem, this embodiment preferably establishes a landing point coordinate system for force analysis when building a rocket three-degree-of-freedom motion model;

Based on the coordinate system of the landing point, the force analysis is carried out on the rocket flying in the dynamic descent stage of the earth landing, and the corresponding earth gravity, aerodynamic drag and engine thrust are determined; wherein, the earth gravity is set as a constant value in the three-degree-of-freedom motion model of the rocket , and based on the short flight time (tens of seconds) of the above-mentioned dynamic descent section, the influence of the earth's rotation can be ignored, and the flight airspace is narrow (more than ten kilometers), and the plane landing field and constant gravitational field model can be used to meet the accuracy requirements and effectively simplify the problem. Solving; aerodynamic drag can be understood as the aerodynamic drag experienced by the rocket in a dense atmosphere, which can be expressed as:

D＝-C _D S _ref ρ||V|| ₂ V/2

In the formula,

Among them, C _D represents the drag coefficient; S _ref represents the reference area of the rocket sub-stage; ρ represents the atmospheric density in the earth’s landing environment, which is represented by an exponential atmospheric density model; V represents the velocity vector of the rocket _; Reference atmospheric density; h represents the flight height of the rocket sub-stage, that is, the rocket position component of the Z axis in the coordinate system of the landing point; h _ref represents the reference height;

The engine thrust can be understood as the engine thrust obtained by combining multiple engines equipped with rocket sub-stages into one engine to provide thrust for the rocket without considering the attitude change of the rocket, which is expressed as:

为发动机开机后的推进剂秒耗量。Among them, I _sp is the fuel specific impulse, g ₀ is the average gravitational acceleration at the sea level of the earth,

is the propellant consumption per second after the engine is turned on.

In addition, in the landing problem studied by the present invention, the influence of control mechanisms such as grid rudders and reaction control systems (ReactionControl System, RCS) on rocket adjustment is not considered, and the thrust produced by the rocket engine is used as the only control quantity; At the same time, regardless of the attitude motion of the rocket, the rocket landing motion is regarded as a center of mass motion, so the total thrust T of the engine can be decomposed in the three-axis direction in the established landing point coordinate system to obtain its The three-axis thrust component, that is, T=[T _x ,T _y ,T _z ] ^T , can effectively avoid complex trigonometric function thrust calculations, and directly use the three thrust components as rocket sub-components in the modeling and solution of subsequent problems level of control, and impose the following constraints on its form:

Among them, ||T|| is the thrust amplitude; due to the limitation of the current level of reusable engine technology, and to ensure the safety of the landing process, during the last landing flight, the engine will not shut down after it is ignited and turned on. That is to say, during the entire power descent flight, the rocket sub-stage will be subjected to a non-zero minimum thrust, and the corresponding engine thrust amplitude has the following constraints:

T _min ≤||T||≤T _max

Among them, T _max and T _min are the upper bound and lower bound of the thrust amplitude of the rocket engine, respectively;

According to the gravitational force of the earth, aerodynamic drag and engine thrust, the three-degree-of-freedom motion model of the rocket is constructed; the three-degree-of-freedom motion model of the rocket is expressed as:

In the formula,

v表示火箭速度矢量，

m表示火箭质量；g(r)表示火箭受到的引力加速度矢量，

本身是关于火箭位置r的函数，在本发明的问题求解中将其设为常值；T表示发动机推力矢量，

为本发明轨迹优化问题的控制变量；D表示气动阻力矢量，

I_sp表示燃料比冲；g₀表示地球海平面处的平均引力加速度；

为发动机开机后的推进剂秒耗量；C_D表示阻力系数；S_ref表示火箭子级的参考面积；ρ₀表示地球海平面处的参考大气密度；h表示火箭子级的飞行高度；h_ref表示参考高度。Among them, r represents the rocket position vector,

v represents the rocket velocity vector,

m represents the mass of the rocket; g(r) represents the gravitational acceleration vector received by the rocket,

Itself is the function about rocket position r, it is set as constant value in problem solving of the present invention; T represents engine thrust vector,

Be the control variable of trajectory optimization problem of the present invention; D represents aerodynamic drag vector,

I _sp represents the fuel specific impulse; g ₀ represents the average gravitational acceleration at the sea level of the earth;

C _D represents the drag coefficient; S _ref represents the reference area of the rocket sub-stage; ρ ₀ represents the reference atmospheric density at the earth’s sea level; h represents the flight height of the rocket sub-stage; h _ref Indicates the reference height.

For the rocket sub-stage landing process, its system state and system control can be expressed as:

Among them, the system state x of the rocket includes the position of the rocket, the speed and mass of the rocket;

Wherein, the system control u of the rocket is the above-mentioned equivalent rocket engine thrust;

S12. According to the rocket three-degree-of-freedom motion model, construct a rocket landing Markov decision-making process model; wherein, the rocket landing Markov decision-making process model includes five elements: state quantity S, action quantity A, reward function R, and state transition Probability P and discount factor γ; Specifically, according to the rocket three-degree-of-freedom motion model, the step of constructing the rocket landing Markov decision process model includes:

Based on the idea of gaze heuristic, the state variable of the rocket is converted to obtain the state quantity of the rocket landing Markov decision process model; the state quantity S does not directly adopt the state variable of the rocket, but the observed The rocket state adopts the idea of gaze inspiration to carry out conversion processing to speed up the convergence speed of the subsequent intelligent agent strategy in the early learning; it should be noted that the state of the rocket system involved in the following can be understood as the state quantity obtained through transformation processing; the state quantity S Can be expressed as:

In the formula,

V _error = VV _sight

Among them, S represents the state quantity of the rocket landing Markov decision process model; r, V and V ₀ represent the rocket position vector, rocket velocity vector and rocket initial velocity respectively; t _go represents the remaining flight time of the rocket; r _z represents the rocket position The Z-axis component of the vector; V _sight represents the sight vector; V _error represents the error between the rocket velocity vector and the sight vector; λ represents the parameter to adjust the size of the sight vector to change with time;

According to the control command of the rocket, the action amount of the Markov decision process model of the rocket landing is obtained; the action amount can be understood as directly selecting the control command of the rocket, that is, the engine thrust, expressed as:

Among them, A represents the action amount of the rocket landing Markov decision process model; T represents the engine thrust vector; T _x , _{Ty y} and T _z represent the X-axis, Y-axis and Z-axis components of the engine thrust respectively; in addition, considering the follow-up The control instruction given by the intelligent agent strategy cannot constrain the modulus of the output action. In order to ensure that the output control instruction meets the thrust amplitude constraint of the rocket engine, this embodiment preferably also needs to intercept the amplitude of the control instruction, so that It strictly meets the constraints of the engine thrust amplitude;

According to the fixed-point soft landing requirements of the rocket, the design principle of the reward function is determined, and according to the design principle of the reward function, the reward function of the Markov decision process model of the rocket landing is obtained; wherein, the design principle of the reward function can be understood as a fixed-point soft landing of the rocket Restrictions such as:

(1) The landing terminal position of the rocket reaches the landing point, that is, r _f =0;

(2) The landing terminal velocity of the rocket is zero, that is, V _f =0;

(3) The remaining mass m _f of the rocket landing terminal is as large as possible, that is, the fuel consumption is reduced as much as possible during the flight;

(4) The lateral maneuver of the rocket during landing and flight cannot be too large;

It should be noted that the design principles of the return function can include but not limited to the principles listed above according to the actual analysis situation. After the design principles are determined, the design of the trajectory return function can be divided into two parts by combining the gaze heuristic: the cumulative return of the process and the terminal reward. return, where the process cumulative return R _prog is expressed as:

R _prog ＝α||V _error ||+β||F _use ||+η·P _glide

stV _error = VV _sight

之间的比值gs；其余变量均为初始化设定参数，如α＝-0.01、β＝-0.05和η＝-100为累计回报R_prog中对应项的比例因子，gs_limit＝0.1和gs_τ＝0.05分别表示最小轨迹坡度和P_glide包络约束公式的比例因子；Among them, V _error is the error between the current rocket speed V and the "sight" vector V _sight , which is the rocket speed vector; F _use is the fuel consumption of the rocket at the current moment, which is related to the amplitude of the command adopted, where |A|| In order to control the amplitude of the command output thrust, T _max is the maximum thrust of the rocket; gs (glideslope) represents the trajectory slope, and P _glide is the envelope constraint, which limits the lateral maneuvering of the rocket. During the landing process of the rocket, whenever the two states When the altitude drops by more than 2m, calculate its longitudinal maneuver dr _z and lateral maneuver

The ratio gs between them; the rest of the variables are initialization setting parameters, such as α=-0.01, β=-0.05 and η=-100 are the proportional factors of the corresponding items in the cumulative return R _prog , gs _limit ＝0.1 and gs _τ ＝ 0.05 represents the scaling factor of the minimum trajectory slope and the P _glide envelope constraint formula, respectively;

The terminal reward return is expressed as:

R _term = reward _landing +P _term

Among them, the reward _term is the reward for the position and speed of the rocket landing terminal meeting the requirements, and the P _term is the penalty for the excessive lateral maneuvering of the rocket at the moment before landing; ||V _term || and ||r _term || represent the terminal The modulus of velocity and terminal position; gs _term is the ratio between the rocket’s longitudinal displacement and lateral displacement during landing, which is consistent with the calculation method of gs in the process constraints; other variables V _limit , r _limit and gs _limit are initialized settings parameter;

Through the above-mentioned process of accumulating rewards and terminal reward rewards, the intelligent agent Agent can be guided to control the rocket to complete the goal of vertical fixed-point soft landing.

The continuous rocket landing process is discretized according to the preset period, and the state transition probability of the rocket landing Markov decision process model is determined according to the rocket integral dynamics; specifically, the state transition probability P is expressed as:

P(s _τ+1 ＝f(s _τ ,a _τ )|s _τ ,a _τ )=1;

Among them, s _τ and a _τ respectively represent the current state of the system at time τ and the action currently taken by the system; s _τ+1 represents the state of the system at time τ+1; f(s,a) represents the dynamic equation of system state transition; P Indicates the probability of transitioning from the state s _τ at the time τ to the state s _τ +1 at the time τ+1 based on the state quantity s _τ and the action quantity a _τ ;

Correspondingly, the discount factor γ in the rocket landing Markov decision process model is used to decay the cumulative return of the future process in the trajectory along time, and the value is preferably 0.95.

S13. Construct an intelligent agent Agent according to the Markov decision process model of the rocket landing, and conduct interactive training between the intelligent agent Agent and the pre-built rocket landing flight simulation environment to obtain a landing control Agent; the intelligent agent Agent includes Based on the value function neural network and based on the policy neural network, specifically, according to the Markov decision process model of the rocket landing, the step of constructing an intelligent agent Agent includes:

According to the Markov decision process model of the rocket landing, the proximal strategy optimization algorithm is selected as the reinforcement learning algorithm of the intelligent agent Agent; wherein, the proximal strategy optimization algorithm can be understood as an improved PPO algorithm for training the intelligent agent Agent rocket sub-stage landing mission;

Based on the proximal policy optimization algorithm, according to the multi-layer perceptron model, construct the value function-based neural network and the policy-based neural network; wherein, the input of the policy-based neural network is the intelligent agent Agent observation value after processing The system state S, the corresponding output is the thrust vector control command value A of the rocket landing; the value-based function neural network is used to accelerate the convergence of the policy-based neural network, and the rocket state value x and its corresponding actual cumulative return in the simulation are based on the trajectory of the plot Q(s,a), training the value function-based network to predict the expected cumulative return V(s) of a certain state;

The above-mentioned policy-based neural network and value-based function-based neural network both adopt a three-hidden layer structure, the hidden layer activation function adopts the tanh function, the output layer activation function adopts a linear activation function, and the input of the policy-based neural network and the value-based function neural network The number n _in of layer neurons is 5, that is, the dimension of the state quantity S; the output layer of the policy-based neural network contains 3 neurons, corresponding to the three-dimensional thrust components of the rocket, while the output layer of the value-based neural network only contains one neuron, corresponding to the expected cumulative reward. Specifically, the specific structural parameters of the policy-based neural network shown in Figure 4 and the value function-based neural network shown in Figure 5 are shown in Table 1:

Table 1 Structural parameters of policy-based neural network and value function-based neural network

The above-mentioned rocket landing flight simulation environment can be understood as a simulation model for simulating rocket landing flight based on the rocket landing dynamics model; specifically, the construction steps of the rocket landing flight simulation environment include:

Based on the three-degree-of-freedom motion model of the rocket, the rocket landing operation environment is constructed, and the corresponding initial value condition generator and flight termination determiner are simultaneously constructed to obtain the rocket landing flight simulation environment; wherein, the initial value condition generator can understand It is the initial state selector used to randomly select the initial state from the set initial state space to start a round of trajectory simulation; the flight termination determiner can be understood as setting the rocket flight abnormality and normal termination criteria at the same time to detect the rocket in real time A motion state detector for whether the landing is terminated;

After the intelligent agent Agent and the rocket landing flight simulation environment are constructed through the steps of the above method, each round of the rocket landing operation environment can be simulated by the intelligent agent Agent interacting with the rocket landing operation environment according to the following method. The value condition generator first randomly selects the initial state of the rocket landing in the initial state space, and then the intelligent agent Agent, according to the observed system state, fits the corresponding control instructions based on the policy-based neural network to guide the rocket to land and fly. When the rocket lands successfully Or when the truncation condition is reached in advance and the flight is terminated, a round of scenario simulation ends accordingly, and a complete rocket landing flight trajectory is completed; and so on, in different initial states, after multiple rounds of scenario simulation, the corresponding enhancement can be completed study and training;

Specifically, the described intelligent agent Agent is interactively trained with the pre-built rocket landing flight simulation environment, and the step of obtaining the landing control Agent includes:

Through the interactive simulation of the intelligent agent Agent and the rocket landing flight simulation environment, the strategy-based neural network of the intelligent agent Agent is trained until convergence, and the landing control Agent is obtained; correspondingly, the landing control Agent training process is as follows:

Randomly selecting an initial state to be simulated from a preset initial state space according to the initial value condition generator;

According to the initial state to be simulated, execute the interactive simulation of the intelligent agent Agent and the flight simulation environment, and when the simulation termination condition preset by the flight termination determiner is reached, terminate the simulation flight of the current wheel, and according to The reward function evaluation obtains the cumulative reward value of each state point in the current simulated flight trajectory, and updates the parameters of the value-based function neural network of the intelligent agent Agent according to the cumulative reward value; in the PPO algorithm learning framework, each round of plot In the simulation, the intelligent agent Agent interacts with the flight simulation environment to obtain a complete trajectory (s _l , a _l , r _l ) consisting of observed states, actions and rewards, where s _l is the environment state observed by the intelligent agent, and a _l is the action taken by the intelligent agent according to the observed value, r _l is the return from the environment to the intelligent agent, and the return r _l is usually expressed as a function of s _l and a _l , then from time k to time T (episode termination time) can be expressed as (s _k ,a _k ,...,s _T ,a _T ), and the cumulative discounted return of this trajectory can be expressed as:

Among them, γ∈[0,1] represents the discount factor, which is used to represent the discount of the return of each time node in the trajectory. For a reinforcement learning algorithm, the goal is to find a set of strategies that can make the cumulative discounted return of the trajectory The expected value is as large as possible;

Predict the expected cumulative return value of each state point in the current simulated flight trajectory according to the value function-based neural network of the updated intelligent agent Agent, and calculate the advantage function according to the cumulative return value and the expected return value, and According to the advantage function, update the parameters based on the policy neural network of the intelligent agent Agent; wherein, the advantage function can be expressed as:

A(s,a)=Q(s,a)-V(s)

Among them, A(s,a), Q(s,a) and V(s) are respectively expressed as advantage function, cumulative return value and expected cumulative return value;

Judging whether the policy-based neural network of the intelligent agent Agent reaches the preset convergence condition, if it is reached, then stop the simulation training to obtain the landing control Agent, otherwise, reselect the initial value to be simulated according to the initial value condition generator. state, and start the next round of interactive simulation training.

It should be noted that, in the above reinforcement learning process, in order to increase the convergence rate of the network and try to avoid reaching the saturation region of the activation function of the hidden layer, preferably, the input of the network is normalized. For the input data of the network, count the mean and standard deviation of each dimension, and then scale according to the following formula:

At the same time, for the output of the strategy-based neural network, in order to satisfy the thrust constraint, preferably, the total amplitude of the output thrust command is limited, and the specific process is shown in the following formula:

分别表示限幅操作前后的推力指令，其他变量释义参见前文描述，此处不再赘述；Among them, a and

Indicates the thrust command before and after the clipping operation respectively. For the interpretation of other variables, please refer to the previous description, and will not repeat them here;

S14. Generate a real-time control instruction according to the landing control agent, and guide the rocket to land and fly according to the real-time control instruction; wherein, after the landing control agent is obtained through the above-mentioned interactive simulation training, the obtained policy-based neural network can be used for the rocket Online landing guidance, and no longer need the amplitude based on the value function neural network, can give corresponding control instructions in real time according to the state of the rocket during flight, and guide the rocket to complete high-precision landing in an environment with deviations.

In the embodiment of the present application, a three-degree-of-freedom motion model of the rocket is constructed according to the force of the rocket's earth landing dynamic descent stage flight, and a Markov decision process model of the rocket landing is constructed according to the three-degree-of-freedom motion model of the rocket, and a Markov decision process model is constructed according to the rocket landing Markov After the Kove decision-making process model builds the intelligent agent Agent, the intelligent agent Agent and the pre-built rocket landing flight simulation environment are interactively simulated and trained to obtain the landing control Agent, and the method of guiding the rocket to land and fly according to the real-time control instructions generated by the landing control Agent , not only can map the engine thrust command to deal with large-scale rocket model deviation and environmental interference based on the real-time state of the rocket, but also ensure that the rocket sub-stage can be guided to a high-precision fixed-point soft landing under complex uncertain working conditions, and through the use of depth As a strategy network for reinforcement learning, the neural network uses the improved PPO algorithm to carry out simulation training and learning on the strategy network, which can effectively fit the high-dimensional continuous action space instructions. It also uses the gaze heuristic method to set the state quantity guidance, and the terminal and process of the rocket landing trajectory Different reward discount rates are designed for indicators, speeding up the convergence speed of the strategy, and effectively improving the efficiency of strategy learning for fixed-point soft landing decisions; compared with the prior art, the method of the present invention has extremely high real-time performance, and the algorithm is extremely robust , can adapt to a wide range of modeling deviations, and can still guide the rocket to a high-precision fixed-point soft landing under the conditions of uncertain interference in the environment, which has high application value.

It should be noted that although the various steps in the above flow chart are displayed sequentially according to the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders.

In one embodiment, as shown in FIG. 6, a real-time robust guidance system for rocket landing based on reinforcement learning is provided, the system comprising:

The motion model building block 1 is used to construct a three-degree-of-freedom motion model of the rocket according to the force of the rocket's earth landing dynamic descent stage flight;

An optimization model construction module 2 is used to construct a Markov decision process model for rocket landing according to the three-degree-of-freedom motion model of the rocket;

The control strategy training module 3 is used to construct an intelligent agent Agent according to the Markov decision process model of the rocket landing, and carry out interactive training between the intelligent agent Agent and the pre-built rocket landing flight simulation environment to obtain the landing control Agent; The intelligent agent Agent includes a neural network based on a value function and a neural network based on a strategy;

The rocket landing guidance module 4 is configured to generate real-time control instructions according to the landing control Agent, and guide the rocket to land and fly according to the real-time control instructions.

For the specific definition of a real-time robust guidance system for rocket landing based on reinforcement learning, please refer to the above-mentioned definition of a real-time robust guidance method for rocket landing based on reinforcement learning, which will not be repeated here. Each module in the above-mentioned real-time robust guidance system for rocket landing based on reinforcement learning can be fully or partially realized by software, hardware and combinations thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, and can also be stored in the memory of the computer device in the form of software, so that the processor can invoke and execute the corresponding operations of the above-mentioned modules.

Fig. 7 shows an internal structural diagram of a computer device in an embodiment, and the computer device may specifically be a terminal or a server. As shown in FIG. 7, the computer device includes a processor, a memory, a network interface, a display and an input device connected through a system bus. Wherein, the processor of the computer device is used to provide calculation and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by a processor, a real-time robust guidance method for rocket landing based on reinforcement learning is realized. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a button, a trackball or a touch pad provided on the casing of the computer device , and can also be an external keyboard, touchpad or mouse.

Those of ordinary skill in the art can understand that the structure shown in Figure 7 is only a block diagram of a partial structure related to the solution of this application, and does not constitute a limitation to the computer equipment on which the solution of this application is applied. The specific computing equipment More or fewer components than shown in the figures may be included, or certain components may be combined, or have the same component arrangement.

In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and operable on the processor, and the steps of the above method are implemented when the processor executes the computer program.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

To sum up, the reinforcement learning-based real-time robust guidance method and system for rocket landing provided by the embodiment of the present invention, its real-time robust guidance method for rocket landing based on reinforcement learning realizes the force according to the flight force of the rocket earth landing dynamic descent stage. Construct the rocket three-degree-of-freedom motion model, and construct the rocket landing Markov decision-making process model according to the rocket three-degree-of-freedom motion model, and after constructing the intelligent agent Agent according to the rocket landing Markov decision-making process model, combine the intelligent agent Agent with the pre-built The landing control agent is obtained by interactive simulation training in the rocket landing and flight simulation environment, and the technical scheme of guiding the rocket landing and flight according to the real-time control instructions generated by the landing control agent can map the engine thrust instructions based on the real-time state of the rocket, and the instructions given have The ability to adapt to large-scale rocket model deviations and environmental disturbances ensures that the rocket sub-stages can be guided to perform high-precision fixed-point soft landings under complex and uncertain conditions. The PPO algorithm learns the policy network to effectively fit the high-dimensional continuous action space instructions. It also uses the gaze heuristic method to set the state quantity guidance, and designs different reward discount rates for the rocket landing trajectory terminal and process indicators to speed up the convergence of the strategy. , so as to ensure that the rocket landing guidance method has extremely high real-time performance, and the algorithm has strong robustness. It can not only adapt to a wide range of modeling deviations, but also effectively deal with uncertain disturbances in the environment. The fixed-point soft landing provides a reliable guarantee and has high application value.

Each embodiment in this specification is described in a progressive manner, and the same or similar parts of each embodiment can be directly referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for relevant parts, refer to part of the description of the method embodiment. It should be noted that the various technical features of the above-mentioned embodiments can be combined arbitrarily. For the sake of concise description, all possible combinations of the various technical features in the above-mentioned embodiments are not described. Where there is a contradiction, all should be deemed to be within the scope of this specification.

The above-mentioned embodiments only express several preferred implementation modes of the present application, and the description thereof is relatively specific and detailed, but it should not be construed as limiting the scope of the patent for the invention. It should be noted that those skilled in the art can make several improvements and substitutions without departing from the technical principle of the present invention, and these improvements and substitutions should also be regarded as the protection scope of the present application. Therefore, the scope of protection of the patent application should be based on the scope of protection of the claims.

Claims (10)

1. A rocket landing real-time robust guidance method based on reinforcement learning is characterized by comprising the following steps:

constructing a rocket three-degree-of-freedom motion model according to acting force borne by the rocket in the earth landing power descent stage;

constructing a rocket landing Markov decision process model according to the rocket three-degree-of-freedom motion model;

constructing an intelligent Agent according to the rocket landing Markov decision process model, and performing interactive training on the intelligent Agent and a pre-constructed rocket landing flight simulation environment to obtain a landing control Agent; the intelligent Agent comprises a value function-based neural network and a strategy-based neural network;

and generating a real-time control instruction according to the landing control Agent, and guiding the rocket to land and fly according to the real-time control instruction.

2. A rocket landing real-time robust guidance method based on reinforcement learning as claimed in claim 1, wherein said step of constructing a rocket three-degree-of-freedom motion model according to the acting force applied to the rocket in the landing power descent stage of the earth comprises:

establishing a landing point coordinate system by taking the rocket sub-level target landing point as an original point; the landing point coordinate system is a coordinate system which takes a target landing point of rocket sublevel landing as a coordinate origin O, takes the vertical upward direction of the geocenter as a coordinate axis Oz, takes the main flight direction of the rocket during landing as a coordinate axis Ox, and takes the direction which is perpendicular to the plane xOz and forms a right-hand rectangular coordinate system with the coordinate axis Ox and the coordinate axis Oz as a coordinate axis Oy;

based on the landing point coordinate system, carrying out stress analysis on the rocket flying in the earth landing power descent stage, and determining corresponding earth attraction, aerodynamic resistance and engine thrust;

constructing the rocket three-degree-of-freedom motion model according to the earth attraction, the pneumatic resistance and the engine thrust; the rocket three-degree-of-freedom motion model is expressed as follows:

in the formula (I), the compound is shown in the specification,

wherein r represents a rocket position vector; v represents a rocket velocity vector; m represents rocket mass; g (r) represents the gravitational acceleration vector received by the rocket; t represents an engine thrust vector; d represents an aerodynamic resistance vector; I.C. A _sp Represents the fuel specific impulse; g ₀ Representing the average gravitational acceleration at sea level of the earth;

the second consumption of the propellant after the engine is started; c _D Representing a drag coefficient; s _ref A reference area representing a rocket substage; rho ₀ Representing a reference atmospheric density at sea level of the earth; h represents the flight height of the rocket stage; h is _ref Indicating a reference height.

3. A rocket landing real-time robust guidance method based on reinforcement learning as claimed in claim 1, wherein said step of constructing a rocket landing markov decision process model according to said rocket three degrees of freedom motion model comprises:

based on the concept of staring inspiration, carrying out conversion processing on the state variable of the rocket to obtain the state quantity of the rocket landing Markov decision process model; the state quantities are expressed as:

in the formula (I), the compound is shown in the specification,

V _error ＝V-V _sight

wherein S represents the state quantity of the rocket landing Markov decision process model; r, V and V ₀ Respectively representing a rocket position vector, a rocket speed vector and a rocket initial speed; t is t _go Representing the remaining time of flight of the rocket; r is _z A Z-axis component representing a rocket position vector; v _sight Represents a line-of-sight vector; v _error Representing the error of the rocket velocity vector and the sight line vector; λ represents a parameter that adjusts the magnitude of the sight vector over time;

obtaining the action quantity of the rocket landing Markov decision process model according to the control instruction of the rocket; the action amount is expressed as:

wherein A represents the action amount of a rocket landing Markov decision process model; t represents an engine thrust vector; t is a unit of _x 、T _y And T _z X-axis, Y-axis and Z-axis components representing engine thrust, respectively;

determining a return function design principle according to rocket fixed point soft landing requirements, and obtaining a return function of the rocket landing Markov decision process model according to the return function design principle;

discretizing a continuous rocket landing process according to a preset period, and determining the state transition probability of the rocket landing Markov decision process model according to rocket integral dynamics.

4. A rocket landing real-time robust guidance method based on reinforcement learning according to claim 1, wherein said step of constructing an intelligent Agent according to said rocket landing markov decision process model comprises:

selecting a near-end strategy optimization algorithm as a reinforcement learning algorithm of the intelligent Agent according to the rocket landing Markov decision process model;

and constructing the value function-based neural network and the strategy-based neural network according to a multilayer perceptron model based on the near-end strategy optimization algorithm.

5. A rocket landing real-time robust guidance method based on reinforcement learning according to claim 1, wherein the rocket landing flight simulation environment construction step comprises:

and constructing a rocket landing operating environment based on the rocket three-degree-of-freedom motion model, and synchronously constructing a corresponding initial value condition generator and a corresponding flight termination determiner to obtain the rocket landing flight simulation environment.

6. A rocket landing real-time robust guidance method based on reinforcement learning as claimed in claim 5, wherein said step of interactively training said intelligent Agent and a pre-constructed rocket landing flight simulation environment to obtain a landing control Agent comprises:

and training a strategy-based neural network of the intelligent Agent until convergence through interactive simulation of the intelligent Agent and the rocket landing flight simulation environment to obtain the landing control Agent.

7. A rocket landing real-time robust guidance method based on reinforcement learning as claimed in claim 6, wherein said step of training strategy-based neural network of intelligent Agent through interactive simulation of said intelligent Agent and said rocket landing flight simulation environment until convergence, obtaining said landing control Agent comprises:

randomly selecting an initial state to be simulated from a preset initial state space according to the initial value condition generator;

according to the initial state to be simulated, executing interactive simulation of the intelligent Agent and the flight simulation environment, terminating the simulation flight of the current wheel when a simulation termination condition preset by the flight termination judger is reached, evaluating and obtaining an accumulated return value of each state point in the current simulation flight track according to a return function, and updating a value-function-based neural network parameter of the intelligent Agent according to the accumulated return value;

predicting expected accumulated return values of all state points in the current simulated flight trajectory according to the updated value-based function neural network of the intelligent Agent, calculating an advantage function according to the accumulated return values and the expected return values, and updating parameters of the intelligent Agent based on a strategy neural network according to the advantage function;

and judging whether the strategy-based neural network of the intelligent Agent reaches a preset convergence condition, if so, stopping simulation training to obtain the landing control Agent, otherwise, reselecting an initial state to be simulated according to the initial condition generator, and starting the next round of interactive simulation training.

8. A rocket landing real-time robust guidance system based on reinforcement learning, which is characterized in that the system comprises:

the motion model building module is used for building a rocket three-degree-of-freedom motion model according to acting force borne by the rocket in the earth landing power descent stage;

the optimization model building module is used for building a rocket landing Markov decision process model according to the rocket three-degree-of-freedom motion model;

the control strategy training module is used for constructing an intelligent Agent according to the rocket landing Markov decision process model and interactively training the intelligent Agent and a pre-constructed rocket landing flight simulation environment to obtain a landing control Agent; the intelligent Agent comprises a value function-based neural network and a strategy-based neural network;

and the rocket landing guidance module is used for generating a real-time control instruction according to the landing control Agent and guiding the rocket to land and fly according to the real-time control instruction.

9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method of any of claims 1 to 7 are implemented when the computer program is executed by the processor.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.

Images