CN115758706A - Modeling method, model and acquisition method of east-west retention strategy model of satellite - Google Patents
Modeling method, model and acquisition method of east-west retention strategy model of satellite Download PDFInfo
- Publication number
- CN115758706A CN115758706A CN202211408082.4A CN202211408082A CN115758706A CN 115758706 A CN115758706 A CN 115758706A CN 202211408082 A CN202211408082 A CN 202211408082A CN 115758706 A CN115758706 A CN 115758706A
- Authority
- CN
- China
- Prior art keywords
- satellite
- state
- moment
- west
- state data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 75
- 230000014759 maintenance of location Effects 0.000 title abstract description 16
- 230000009471 action Effects 0.000 claims abstract description 45
- 238000013528 artificial neural network Methods 0.000 claims abstract description 32
- 238000012549 training Methods 0.000 claims abstract description 26
- 230000006399 behavior Effects 0.000 claims abstract description 25
- 230000006870 function Effects 0.000 claims description 24
- 238000004321 preservation Methods 0.000 claims description 21
- 239000013598 vector Substances 0.000 claims description 21
- 238000011217 control strategy Methods 0.000 claims description 7
- 238000004590 computer program Methods 0.000 claims description 2
- 230000006996 mental state Effects 0.000 claims description 2
- 239000000446 fuel Substances 0.000 abstract description 9
- 230000002787 reinforcement Effects 0.000 description 14
- 230000000875 corresponding effect Effects 0.000 description 9
- 238000013135 deep learning Methods 0.000 description 9
- 230000008569 process Effects 0.000 description 8
- 238000004422 calculation algorithm Methods 0.000 description 7
- 238000012545 processing Methods 0.000 description 7
- 238000004364 calculation method Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 230000008901 benefit Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 3
- 230000008447 perception Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000004088 simulation Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 230000001276 controlling effect Effects 0.000 description 1
- 230000001351 cycling effect Effects 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 230000010355 oscillation Effects 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 239000003380 propellant Substances 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Landscapes
- Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)
Abstract
The invention relates to the field of aerospace, and provides a modeling method, a model, an obtaining method, equipment and a medium of a satellite east-west retention strategy model, wherein the modeling method comprises the following steps: s1: acquiring a plurality of satellite training state data sets; s2: obtaining all tangential control behaviors after the initial moment and the corresponding output Q values; s3: obtaining the satellite state at the current moment and the tangential control action executed by the satellite; s4: obtaining the reward and the satellite state at the next moment; s5: storing the satellite combination state data set into an experience pool; s6: calculating a target value; s7: calculating an error according to the loss function; s8: updating the Q value; taking the satellite state at the next moment as the satellite state at the current moment; s9: repeating the steps S3-S8 of the expected tracking times; after the steps S3-S8 of the appointed iteration times are executed repeatedly, updating the weight parameter of the target neural network; s10: steps S2-S9 are repeatedly performed. The scheme can obtain an optimal decision strategy and reduce the consumption of satellite fuel.
Description
Technical Field
The invention relates to the technical field of aerospace, in particular to a satellite east-west retention strategy model modeling method, a satellite east-west retention strategy model obtaining method, satellite east-west retention strategy model obtaining equipment and satellite east-west retention strategy model medium based on Nature DQN.
Background
With the continuous development of human aerospace activities, more and more remote sensing satellites provide help for the daily life of people.
The GEO satellite is influenced by day and month gravitation and earth non-spherical perturbation in the operation process, so that drifting occurs in the east-west direction, and the maintaining control of the east-west position of the GEO three-axis stable satellite plays an important role in the aerospace field.
The method comprises the steps of firstly analyzing that satellite east-west drift is caused by the influence of various shooting forces such as a spherical shape of a satellite and sunlight pressure on the satellite in the process of orbital operation, so that the flatness and eccentricity of the satellite are changed, then modeling according to the shooting forces, formulating an east-west keeping strategy, further optimizing keeping parameters and calculating the consumption of a propellant. In the prior art, various types of perturbation force applied to the satellite in the process of orbital operation are subjected to complex modeling, however, due to the complexity of space stress and the uncertainty of parameters of the satellite, the satellite cannot be subjected to accurate modeling, the number of parameters is large, the calculation is complex, the satellite east-west control precision is further influenced, and more fuel may be consumed.
Therefore, it is urgently needed to develop a modeling method, a model, an obtaining method, equipment and a medium of a satellite east-west retention strategy model based on a Nature DQN, reduce modeling difficulty and accurately calculate the east-west retention strategy.
Disclosure of Invention
The invention aims to provide a modeling method, a modeling model, an obtaining method, equipment and a medium of a satellite east-west keeping strategy model, which do not need to carry out complex modeling when carrying out east-west position keeping on a GEO triaxial stable satellite, do not need to consider the complexity of space stress and the uncertainty of the parameters of the satellite, have strong behavior decision-making capability in reinforcement learning, can obtain an optimal decision-making strategy and reduce the consumption of satellite fuel.
In order to solve the above technical problem, as an aspect of the present invention, there is provided a method for modeling a satellite east-west preservation policy model based on Nature DQN, comprising the steps of:
s1: initializing a model, and acquiring a plurality of groups of satellite training state data sets, wherein each group of satellite training state data sets comprises an initial state of a satellite, a plurality of expected orbit control moments and expected orbit control times; the initial state of each satellite comprises an initial time satellite state; satellite states include mean longitude and eccentricity vectors;
s2: inputting initial time satellite states of a group of satellite training state data sets into the model to obtain all tangential control behaviors after the initial time and corresponding output Q values;
s3: acquiring the state of the satellite at the current moment, and acquiring a tangential control behavior executed by the satellite according to a greedy strategy;
s4: executing a tangential control action to obtain the satellite state and the actual orbit control time at the next time, and obtaining a reward according to the actual orbit control time and an east-west keeping strategy reward function; the east-west retention policy reward function adopts formula 1:
wherein r is t Prizes derived from tangential control activity for satellites at the present timeExciting, wherein delta t is a time interval from the actual orbit control moment and is obtained according to the current moment and the actual orbit control moment; the longer Δ t, the longer the retention period, the higher the reward; t is t 0 The expected track control time closest to the current time is obtained; t is the current time;
s5: storing the satellite state at the current moment, the tangential control action executed by the satellite, the reward and the satellite state at the next moment into an experience pool as a group of satellite combination state data groups;
s6: taking out a plurality of satellite combination state data sets from the experience pool, and calculating the target value of each satellite combination state data set according to the target neural network weight parameter;
s7: calculating an error according to the loss function, and updating the weight parameter of the current neural network;
s8: updating the Q value according to the value function; taking the satellite state at the next moment as the satellite state at the current moment;
s9: repeating steps S3-S8, wherein the times for executing steps S3-S8 is equal to the expected orbit control times of the set of satellite training state data; after the steps S3-S8 of the appointed iteration times are repeatedly executed, updating the weight parameter of the target neural network according to the weight parameter of the current neural network;
s10: and repeatedly executing the steps S2-S9 until all the data of the satellite training state data set are input.
According to an exemplary embodiment of the invention, initializing the model in step S1 comprises defining a loss function.
According to an exemplary embodiment of the present invention, the input of the model is the satellite state, and the output is the return value (Q value) after the execution of the tangential control action by the satellite.
According to an example embodiment of the present invention, the satellite states include: mean longitude drift rate, eccentricity vector, tilt vector, and mean longitude. The mean longitude drift rate is denoted by L, the eccentricity vector by e, the inclination vector by i, and the mean longitude by R.
According to an exemplary embodiment of the present invention, in step S3, during the first loop, the current satellite state is the initial satellite state.
According to an exemplary embodiment of the present invention, in step S3, the method for obtaining the tangential control behavior executed by the satellite according to the greedy policy includes: the satellite randomly selects a tangential control behavior with a first specified probability or executes the tangential control behavior corresponding to the maximum Q value with a second specified probability; the sum of the first specified probability and the second specified probability equals 1.
According to an exemplary embodiment of the present invention, in step S6, the method for calculating the target value of each satellite combination state data set according to the target neural network weight parameter uses formula 2:
wherein, y j Representing the target value, gamma is the discount value, theta' is the target neural network weight parameter,
represents the maximum Q value, s, of the satellite performing the tangential control action a at the next moment in the set of satellite combined state data sets j+1 Representing the satellite state at the next moment in the set of satellite combined state data sets, a representing the tangential control action performed by the satellite at the current moment in the set of satellite combined state data sets, r j A reward is indicated in a set of satellite constellation state data sets.
According to an exemplary embodiment of the present invention, in step S7, the loss function adopts formula 3:
wherein, y j Representing the target value, theta is the current weight parameter of the neural network, Q(s) j ，a j (ii) a Theta) represents the current time satellite in a set of satellite combined state data sets performing the tangential control action a j Value of later Q, s j Representing the state of the satellite at the current time in a set of satellite constellation state data sets, a j Indicating satellite execution at the current timeM is the number of satellite combined state data sets.
According to an exemplary embodiment of the present invention, in step S8, the method for updating the Q value according to the value function uses formula 4:
Q(s t ，a t )←Q(s t ，a t )+α[r t +γmax Q(s t+1 ，a t )-Q(s t ，a t )] (4)；
wherein, Q(s) at the left side of the arrow t ，a t ) The satellite representing the updated current time performs a tangential control action a t The latter Q value, Q(s) on the right side of the arrow t ，a t ) The satellite representing the current moment before update performs a tangential control action a t Later Q value, Q(s) t+1 ，a t ) The satellite performs a tangential control action a at the next moment in time representing the current moment in time before updating t The latter Q value, alpha is the weight, gamma is the discount value, s t Representing the state of the satellite at the current time, a t Representing the tangential control action, s, performed by the satellite at the current moment t+1 The satellite state at the next moment in time, r, representing the current moment in time t Indicating a reward.
the time t is the current time, and the time t +1 is the next time of the current time.
The invention provides a satellite east-west conservation strategy model based on the Nature DQN, and a model is established by adopting the modeling method of the satellite east-west conservation strategy model based on the Nature DQN.
As a third aspect of the present invention, a method for obtaining a satellite east-west preservation optimal strategy is provided, wherein a satellite east-west preservation strategy model based on Nature DQN is established by using the modeling method of the satellite east-west preservation strategy model based on Nature DQN;
obtaining an optimal strategy according to the model;
the method for obtaining the optimal strategy according to the model adopts a formula 5:
wherein, pi represents the strategy of tangential control of the satellite, pi * Represents the optimal tangential control strategy learned by the model, i.e. the satellite passes through the strategy pi under the condition that the satellite state is s at the initial moment * The control action of (a) yields the greatest return.
As a fourth aspect of the present invention, there is provided an electronic apparatus comprising:
one or more processors;
storage means for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement the method for modeling a satellite east-west preservation policy model based on Nature DQN.
As a fifth aspect of the present invention, there is provided a computer readable medium having stored thereon a computer program which, when being executed by a processor, implements the modeling method of the satellite east-west preservation policy model based on Nature DQN.
The beneficial effects of the invention are:
according to the scheme, the modeling is carried out through the neural network, the deep reinforcement learning and decision making are carried out by using the current satellite state data, the complex modeling is carried out without using various perturbation forces received by the satellite in the orbital operation process, the optimal east-west control strategy can be obtained, the consumption of satellite fuel can be reduced, and the method has important significance and value for practical aerospace application.
Drawings
Fig. 1 schematically shows a step diagram of a modeling method of a satellite east-west preservation policy model based on Nature DQN.
Fig. 2 schematically shows a block diagram of an electronic device.
FIG. 3 schematically shows a block diagram of a computer-readable medium.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The same reference numerals denote the same or similar parts in the drawings, and thus, a repetitive description thereof will be omitted.
Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the application. One skilled in the relevant art will recognize, however, that the subject matter of the present application can be practiced without one or more of the specific details, or with other methods, components, devices, steps, and so forth. In other instances, well-known methods, devices, implementations, or operations have not been shown or described in detail to avoid obscuring aspects of the application.
The block diagrams shown in the figures are functional entities only and do not necessarily correspond to physically separate entities. I.e. these functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor means and/or microcontroller means.
The flowcharts shown in the figures are illustrative only and do not necessarily include all of the contents and operations/steps, nor do they necessarily have to be performed in the order described. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.
It will be understood that, although the terms first, second, third, etc. may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one element from another. Thus, a first component discussed below may be termed a second component without departing from the teachings of the present concepts. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.
It will be appreciated by those skilled in the art that the drawings are merely schematic representations of exemplary embodiments, and that the blocks or processes shown in the drawings are not necessarily required to practice the present application and are, therefore, not intended to limit the scope of the present application.
According to the scheme, observation information is obtained from the environment based on strong perception capability of deep learning, and the expected return value is obtained based on strong decision-making capability of reinforcement learning to evaluate the value of the footstock. The entire learning process can be described as: at a certain moment, the satellite interacts with the flying environment to acquire observation information, the current state information is mapped into corresponding actions (control actions) through the neural network, the environment reacts to the actions to acquire corresponding reward values and next observation information, and the complete interaction information is stored in an experience pool. By continuously cycling the processes, the optimal strategy for achieving the target can be finally obtained.
The satellite described in the scheme is a GEO triaxial stabilized satellite. Geosynchronous orbit (GEO), refers to a circular orbit of a satellite around the earth at approximately 36000 km above the earth's equator. Because the orbit of a satellite is synchronized with the earth rotation about the earth, and the satellite and the earth are in a relatively stationary state, the satellite traveling in such an orbit is referred to as a "geostationary satellite" for short, and is also referred to as a "geostationary satellite" or a "stationary satellite". The three-axis stability means that the satellite does not rotate, and the body is stable in the X direction, the Y direction and the Z direction, in other words, a certain attitude relation is kept between the body and the earth.
The Deep Q Network (DQN) algorithm is a network in Deep reinforcement learning, and is a combination of Deep learning and Q learning. The method integrates the advantages of reinforcement learning and deep learning, so that the method is widely applied to various fields at present.
Deep reinforcement learning is taken as a new research hotspot in the field of artificial intelligence, combines the deep learning with the reinforcement learning, and realizes direct control and decision from original input to output through an end-to-end learning mode. Because the deep learning is based on a neural network structure, the deep learning has stronger perception capability to the environment, but lacks certain decision control capability; and reinforcement learning has very strong behavior decision-making capability. Therefore, the perception capability of deep learning and the decision capability of the reinforcement learning are combined in the deep reinforcement learning, the advantages are complementary, and the control strategy can be directly learned from high-dimensional original data. Since the deep reinforcement learning method is provided, substantial breakthrough is achieved in a plurality of tasks requiring sensing of high-dimensional original input data and decision control, and due to the end-to-end learning advantage of deep learning, the problems of difficult modeling and difficult planning can be solved by deep reinforcement learning.
The DQN algorithm uses the same network for calculating the target value and the current value, i.e. the calculation of the target value uses the parameters in the Q network to be trained, and the target value is used to update the parameters of the network, so that the two depend on each other circularly, which is not favorable for the convergence of the algorithm. Compared with DQN, the Nature DQN increases a target network, reduces the dependency relationship between the calculation of a target Q value and the Q network parameter to be updated through a double-network structure, and integrates the advantages of reinforcement learning and deep learning, thereby greatly improving the stability of the DQN algorithm.
Nature DQN reduces the correlation between the target value of the computational target network and the current network parameters by using two independent but identical Q networks (one as the current Q network and the other as the target Q network). The target network is updated at intervals of a certain step length C by copying the weight parameters of the current network to the target network, and the target Q value is kept unchanged in a period of time by the double-network structure, so that the correlation between the calculated target Q value and the current network parameters is reduced, and the convergence and the stability of the algorithm are improved.
As a first embodiment of the present invention, there is provided a method for modeling a satellite east-west preservation policy model based on Nature DQN, as shown in fig. 1, including the steps of:
s1: initializing a model, and acquiring a plurality of groups of satellite training state data sets, wherein each group of satellite mental state data sets comprise an initial state of a satellite, a plurality of expected orbit control moments and expected orbit control times; the initial state of each satellite comprises an initial time satellite state; the satellite states include the mean longitude and the eccentricity vector.
The input of the model is the satellite state, and the output is the return value (Q value) after the execution of the tangential control action of the satellite.
The satellite states include: mean longitude drift rate, eccentricity vector, tilt vector, mean longitude. The mean longitude drift rate is denoted by L, the eccentricity vector by e, the inclination vector by i, and the mean longitude by R.
The method for initializing the model comprises the following steps: defining a loss function; initializing the capacity of an experience pool to be N, wherein the experience pool is used for storing training samples; initializing a current neural network weight parameter theta and a target neural network weight parameter theta ', theta' = theta of the network model; the input of the initialization network is the satellite state s, and the calculated network output is the return value O after the satellite executes the tangential control action.
The data set is composed of a plurality of groups of satellite training state data sets, the data of the satellite states in the data set is larger than or equal to 100 groups, and the more the data of the satellite states are, the more accurate the result trained by the model is.
The data of the multiple groups of satellite training state data sets are data of a training set, and simulation data can be adopted, or simulation data and real data can be combined. The time line within a time period comprises a plurality of time points, the state of the satellite at each time point is different, and different effects can be obtained when the orbit control strategy is executed at different time points. According to the scheme, through a plurality of groups of satellite training state data groups, the satellite state of each group of satellites at the initial moment corresponds to the satellite state of a time point, the time points corresponding to the initial moments of each group of satellite training state data groups are different, namely the initial moments of each group of satellite training state data groups are different.
S2: and inputting the initial time satellite states of a group of satellite training state data groups into the model to obtain all tangential control behaviors after the initial time and the corresponding output Q values.
And after the initial time satellite executes the tangential control action, the state of the next time satellite is obtained. And after the satellite executes the tangential control action at the next moment, obtaining the satellite state at the next moment. By analogy, the tangential control behaviors at the next multiple moments are obtained.
S3: and acquiring the state of the satellite at the current moment, and acquiring the tangential control behavior executed by the satellite according to a greedy strategy.
And in the initial circulation, the satellite state at the current moment is the satellite state at the initial moment.
The method for acquiring the tangential control behavior executed by the satellite according to the greedy strategy comprises the following steps: the satellite randomly selects a tangential control behavior according to a first specified probability or executes the tangential control behavior corresponding to the maximum Q value according to a second specified probability; the sum of the first specified probability and the second specified probability equals 1.
If the first specified probability is larger than the second specified probability, the method for obtaining the tangential control behavior executed by the satellite according to the greedy strategy adopts the following steps: randomly selecting a tangential control behavior by the satellite with a first specified probability;
if the second designated probability is greater than the first designated probability, the method for obtaining the tangential control behavior executed by the satellite according to the greedy strategy adopts the following steps: the satellite executes the tangential control action corresponding to the maximum Q value according to a second specified probability;
if the first specified probability is equal to the second specified probability, then one of the methods for obtaining the tangential control behavior executed by the satellite according to the greedy strategy is selected: the satellite randomly selects the tangential control action with a first specified probability or executes the tangential control action corresponding to the maximum Q value with a second specified probability.
The greedy policy is an epsilon-greedy policy.
The first assigned probability is ε, which decreases as the number of iterations increases.
The tangential control action performed by the satellite at the current moment is a t 。
S4: executing a tangential control action to obtain the satellite state and the actual orbit control moment at the next moment; and awarding according to the actual track control time and east-west keeping strategy awarding function.
The satellite drift in the east-west direction consists of two parts: one is spherical perturbation which causes the satellite longitude and latitude to drift, and the other is eccentricity perturbation generated by sunlight pressure which causes the satellite longitude and latitude periodic oscillation. So the east-west preservation of the satellite mainly includes the flatness preservation and the eccentricity preservation. The flatness maintenance mainly utilizes east-west maneuvering to correct the drift rate of the satellite, and one satellite is maintained in a narrow orbit window; in the satellite eccentricity vector keeping strategy, the sun pointing to the near place strategy is a more common method, and the method enables the mean eccentricity vector of the satellite to be always kept in an eccentricity control circle in a control period, and enables the average direction of a connecting line from the center of the control circle to the end point of the mean eccentricity vector to point to the sun direction.
In the two-body problem, the mean longitude (mean longitude) is a longitude value of an object moving on an imaginary circular orbit whose orbit inclination is 0.
Eccentricity (Eccentricity), i.e. Eccentricity, is a mathematical quantity used to describe the shape of a conic curve track. Defined as the ratio of the distance of the curve from the fixed point (focal point) to the fixed line (directrix). For an ellipse, eccentricity is the ratio of the distance between two foci (focal length) to the length of the major axis. Eccentricity is generally indicated by e.
The method for obtaining the satellite mean longitude adopts the following formula:
wherein R represents the mean longitude of the satellite, a s Is the semi-major axis of the geostationary orbit, a 0 Is the orbital semi-major axis of the satellite;
the method for obtaining the satellite eccentricity vector adopts the following formula:
wherein e represents the eccentricity of the satellite, and the two-dimensional eccentricity vector of the satellite on the orbit is (e) x ，e y ) Omega represents the right ascension of the satellite, and omega represents the argument of the perigee.
Both tangential and radial thrust belong to in-plane maneuvers. Maneuvers that employ tangential thrust, known as tangential control (also known as east-west control or longitude control), can alter the azimuth drift rate and eccentricity vectors of the satellite. The maneuvering of radial thrust only changes the eccentricity vector of the satellite, but the radial thrust of the same magnitude can only achieve half the effect of tangential thrust. Tangential control is more efficient than radial control. Thus, satellite-borne east keeping maneuvers are mainly achieved by means of tangential thrust, and radial thrust is rarely used.
Tangential control refers to the maneuvering of a satellite in the orbital plane in the direction of velocity.
Because the tangential maneuvering can not only cause the change of the eccentricity vector, but also cause the change of the semi-major axis of the orbit, in order to optimize the ground measurement and control work and reduce the fuel consumption, the ideal state of things is kept to ensure that the satellite mean longitude keeps the control period and the eccentricity keeping period the same, and the optimum criterion of the maneuvering fuel consumption in the plane is utilized to simultaneously keep and control the eccentricity vector and the mean longitude of the satellite.
The goal of the satellite east-west keeping strategy problem is to keep the satellite east-west position while minimizing the fuel consumption as much as possible. The velocity impulse required for east-west position holding control is mainly used for holding control of the state. Considering that each control of the satellite requires a lot of manual work, the fewer the number of controls, the lower the risk, which requires longer periods of east-west maintenance of the satellite.
On the premise that the fuel consumed by each control is equal, the control amount of the satellite at the time t determines the time for next orbit control. R 0 、e 0 Respectively mean longitude and eccentricity, Δ R, of the nominal orbit s 、Δe s And keeping the circle radius for the mean longitude and the eccentricity respectively, and controlling the satellite at the time t (a certain time), and then obtaining the time when the mean longitude or the eccentricity of the satellite exceeds the keeping range according to the extrapolated ephemeris. Based on the purpose that the satellite mean longitude and eccentricity retention period are long, a reward strategy at the moment t is designed. Thus, the east-west retention policy reward function employs equation 1:
wherein r is t Reward obtained by performing tangential control action on the satellite at the current moment, wherein delta t is a time interval from the actual orbit control moment and is obtained according to the current moment and the actual orbit control moment; t is t 0 The expected track control time closest to the current time is obtained; and t is the current moment. It is clear that the longer the interval time (Δ t), the longer the hold period, and the higher the reward.
Δ t = t1-t; t1 is the actual tracking instant (the instant at which the tangential control action is required next time). The actual tracking timing is the timing of the extrapolated tangential control to be performed, i.e., the timing at which the average longitude or eccentricity vector exceeds a predetermined range.
S5: and storing the satellite state at the current moment, the tangential control action executed by the satellite, the reward and the satellite state at the next moment into an experience pool as a group of satellite combination state data sets.
S6: and taking out a plurality of satellite combination state data sets from the experience pool, and calculating the target value of each satellite combination state data set according to the target neural network weight parameter.
The number of the satellite combination state data sets is m, m is a natural number greater than 0, and m is smaller than the number of the satellite training state data sets. The m groups of satellite combination state data groups are small-batch state data groups. The number of satellite constellation state data sets is determined from the data set of satellite training state data sets.
The method for calculating the target value of each satellite combination state data set according to the target neural network weight parameter adopts a formula 2:
wherein, y j Representing the target value, gamma is a discount value (attenuation factor), theta' is a target neural network weight parameter,
representing the next-in-time satellite in a set of satellite constellation state data setsMaximum Q value after line tangent control action a, s j+1 Representing the satellite state at the next moment in a set of satellite combined state data sets, a representing the tangential control action performed by the satellite at the current moment in a set of satellite combined state data sets, r j Representing a reward in a set of satellite constellation state data sets.
And ending the task to be the convergence of the model or the iteration. When s is j+1 For model convergence or iteration completion, y i Is equal to r j (ii) a When s is j+1 When model convergence or iteration is not complete, y i Is equal to
The conditions for model convergence are: the error calculated by the loss function is within a specified range.
The conditions for the iteration to be completed are: all steps are executed.
S7: and calculating errors according to the loss function, and updating the weight parameters of the current neural network.
An error is also calculated based on the target value.
The loss function uses equation 3:
wherein, y j Representing the target value, theta is the current weight parameter of the neural network, Q(s) j ，a j (ii) a Theta) represents the current time satellite in a set of satellite combined state data sets to perform the control action a j Value of later Q, s j Representing the satellite state at the current time in a set of satellite constellation state data sets, a j Representing the tangential control action performed by the satellite at the current moment, r j Representing a reward in a set of satellite constellation state data sets; and m is the number of the satellite combination state data sets.
The error is the calculation result of the loss function by using the formula 3.
The current neural network weight parameters are updated by a Stochastic Gradient Descent (SGD) method.
r t 、a t 、s t 、s t+1 Samples in the data set representing the satellite training state data set, r j 、a j 、s j 、s j+1 Representing samples in an experience pool.
S5-S7 adjust the parameters of the model, so that the calculation accuracy of the model can be higher.
S8: updating the Q value according to the value function; and taking the satellite state at the next moment as the satellite state at the current moment.
The method for updating the Q value according to the value function uses equation 4:
Q(s t ，a t )←Q(s t ，a t )+α[r t +γmax Q(s t+1 ，a t )-Q(s t ，a t )] (4)；
wherein, Q(s) at the left side of the arrow t ，a t ) The satellite representing the updated current time performs a tangential control action a t The latter Q value, Q(s) on the right side of the arrow t ，a t ) The satellite representing the current moment before update performs a tangential control action a t Later Q value, Q(s) t+1 ，a t ) The satellite representing the next moment in time to the current moment in time before the update performs a tangential control action a t The latter Q value, α is the weight, γ is the discount value (attenuation factor), s t Representing the state of the satellite at the current time, a t Representing the tangential control action performed by the satellite at the current moment, s t+1 The satellite state at the next moment in time, r, representing the current moment in time t Indicating a reward.
Wherein both α and γ range between 0 and 1.
S9: repeating steps S3-S8, wherein the times for executing steps S3-S8 is equal to the expected orbit control times of the set of satellite training state data; and after the steps S3-S8 of the appointed iteration times are repeatedly executed, updating the weight parameter of the target neural network according to the weight parameter of the current neural network.
And updating the target neural network weight parameter to the current neural network weight parameter after finishing the iteration of the specified iteration times.
S10: and repeatedly executing the steps S2-S9 until all the data of the satellite training state data set are input.
According to the modeling method, the satellite training state data is used as the input of the neural network model, the generated return value is used as the output, the Nature DQN neural network is adopted, complex modeling is carried out without various perturbation forces applied to the satellite in the orbital operation process, deep reinforcement learning is directly adopted for learning and decision making, improvement is carried out on the basis of the DQN algorithm, the method is suitable for training a large-scale neural network, the stability of the DQN algorithm is greatly improved, an optimal east-west control strategy can be obtained, and the consumption of satellite fuel can be reduced, so that the method has important significance and value for practical aerospace application.
According to a second embodiment of the invention, the invention provides a satellite east-west preservation strategy model based on the Nature DQN, and the model is established by adopting the modeling method of the satellite east-west preservation strategy model based on the Nature DQN of the first embodiment.
According to a third embodiment of the invention, the invention provides a method for acquiring satellite east-west retention optimal strategy, which comprises the steps of establishing a satellite east-west retention strategy model based on the Nature DQN by adopting the modeling method of the satellite east-west retention strategy model based on the Nature DQN of the first embodiment;
and obtaining an optimal strategy according to the model.
The method for obtaining the optimal strategy according to the model adopts a formula 5:
wherein, pi represents the strategy of tangential control of the satellite, pi * Represents the optimal tangential control strategy learned by the model, i.e. the satellite passes through the strategy pi under the condition that the satellite state is s at the initial moment * Produces the greatest return under tangential control behavior a.
According to a fourth specific embodiment of the present invention, there is provided an electronic device, as shown in fig. 2, where fig. 2 is a block diagram of an electronic device shown according to an exemplary embodiment.
An electronic device 200 according to this embodiment of the present application is described below with reference to fig. 2. The electronic device 200 shown in fig. 2 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.
As shown in fig. 2, electronic device 200 is in the form of a general purpose computing device. The components of the electronic device 200 may include, but are not limited to: at least one processing unit 210, at least one memory unit 220, a bus 230 connecting different system components (including the memory unit 220 and the processing unit 210), a display unit 240, and the like.
Wherein the storage unit stores program code executable by the processing unit 210 to cause the processing unit 210 to perform the steps according to various exemplary embodiments of the present application described in the present specification. For example, the processing unit 210 may perform the steps as shown in fig. 1.
The memory unit 220 may include readable media in the form of volatile memory units, such as a random access memory unit (RAM) 2201 and/or a cache memory unit 2202, and may further include a read only memory unit (ROM) 2203.
The storage unit 220 may also include a program/utility 2204 having a set (at least one) of program modules 2205, such program modules 2205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.
The electronic device 200 may also communicate with one or more external devices 200' (e.g., keyboard, pointing device, bluetooth device, etc.), devices that enable a user to interact with the electronic device 200, and/or any device (e.g., router, modem, etc.) that the electronic device 200 may communicate with one or more other computing devices. Such communication may occur through input/output (I/O) interfaces 250. Also, the electronic device 200 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via the network adapter 260. The network adapter 260 may communicate with other modules of the electronic device 200 via the bus 230. It should be understood that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the electronic device 200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, and may also be implemented by software in combination with necessary hardware.
Thus, according to a fifth embodiment of the present invention, there is provided a computer readable medium. As shown in fig. 3, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the embodiment of the present invention.
The software product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The computer readable storage medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable storage medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In situations involving remote computing devices, the remote computing devices may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to external computing devices (e.g., through the internet using an internet service provider).
The computer-readable medium carries one or more programs which, when executed by a device, cause the computer-readable medium to carry out the functions of the first embodiment.
Those skilled in the art will appreciate that the modules described above may be distributed in the apparatus according to the description of the embodiments, or may be modified accordingly in one or more apparatuses unique from the embodiments. The modules of the above embodiments may be combined into one module, or further split into multiple sub-modules.
Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiment of the present invention.
The present invention has been described in terms of the preferred embodiment, and it is not intended to be limited to the embodiment. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (9)
1. A modeling method of a satellite east-west preservation strategy model based on Nature DQN is characterized by comprising the following steps:
s1: initializing a model, and setting a plurality of groups of satellite training state data sets, wherein each group of satellite mental state data sets comprise an initial state of a satellite, a plurality of expected orbit control moments and expected orbit control times; the initial state of each satellite comprises an initial time satellite state; satellite states include mean longitude and eccentricity vectors;
s2: inputting initial time satellite states of a group of satellite training state data sets into the model to obtain all tangential control behaviors after the initial time and corresponding output Q values;
s3: acquiring the state of the satellite at the current moment, and acquiring a tangential control behavior executed by the satellite according to a greedy strategy;
s4: executing a tangential control action to obtain the satellite state and the actual orbit control moment at the next moment; awarding is obtained according to the actual track control and east-west keeping strategy awarding function; the east-west keeping policy reward function adopts formula 1:
wherein r is t The reward obtained by the tangential control action of the satellite at the current moment is obtained, and delta t is a time interval from the actual orbit control moment and is obtained according to the current moment and the actual orbit control moment; t is t 0 The expected track control time closest to the current time is obtained; t is the current time;
s5: storing the satellite state at the current moment, the tangential control action executed by the satellite, the reward and the satellite state at the next moment into an experience pool as a group of satellite combination state data groups;
s6: taking out a plurality of satellite combination state data sets from the experience pool, and calculating the target value of each satellite combination state data set according to the target neural network weight parameter;
s7: calculating an error according to the loss function, and updating the weight parameter of the current neural network;
s8: updating the Q value according to the value function; taking the satellite state at the next moment as the satellite state at the current moment;
s9: repeating steps S3-S8, wherein the times for executing steps S3-S8 is equal to the expected orbit control times of the set of satellite training state data; after the steps S3-S8 of the appointed iteration times are repeatedly executed, updating the weight parameter of the target neural network according to the weight parameter of the current neural network;
s10: and repeatedly executing the steps S2-S9 until all the data of the satellite training state data set are input.
2. The modeling method of the satellite east-west preservation policy model based on Nature DQN according to claim 1, wherein in step S3, the method of obtaining the tangential control behavior performed by the satellite according to the greedy policy comprises: the satellite randomly selects a tangential control behavior with a first specified probability or executes the tangential control behavior corresponding to the maximum Q value with a second specified probability; the sum of the first specified probability and the second specified probability equals 1.
3. The modeling method of satellite east-west preservation policy model based on Nature DQN according to claim 1, wherein in step S6, the method of calculating the target value of each satellite combination state data set according to the target neural network weight parameter employs formula 2:
wherein, y j Representing the target value, gamma is the discount value, theta' is the target neural network weight parameter,
represents the maximum Q value, s, of the combined state data set of the satellites at the next moment after the tangential control action a is executed by the satellite j+1 Representing the satellite state at the next moment in a set of satellite combined state data sets, a representing the tangential control action performed by the satellite, r j Representing a reward in a set of satellite constellation state data sets.
4. The modeling method of a satellite east-west preservation strategy model based on Nature DQN according to claim 1, wherein in step S7, the loss function adopts formula 3:
wherein, y j Representing the target value, theta is the current weight parameter of the neural network, Q(s) j ，a j (ii) a Theta) represents the current time satellite in a set of satellite combined state data sets performing a tangential control action a j Value of Q after, s j Representing the satellite state at the current time in a set of satellite constellation state data setsState a of j Representing the tangential control action performed by the satellite at the current moment, and m is the number of the satellite combination state data sets.
5. The modeling method of the satellite east-west preservation strategy model based on Nature DQN according to claim 1, wherein in step S8, the method for updating Q value according to value function adopts formula 4:
Q(s t ，a t )←Q(s t ，a t )+α[r t +γmax Q(s t+1 ，a t )-Q(s t ，a t )] (4)；
wherein, Q(s) at the left side of the arrow t ，a t ) Representing the updated current time satellite performing the tangential control action a t The latter Q value, Q(s) on the right side of the arrow t ，a t ) The satellite representing the current moment before update performs a tangential control action a t Later Q value, Q(s) t+1 ，a t ) The satellite executes the tangential control action a at the next moment of time representing the current moment before updating t The latter Q value, alpha is the weight, gamma is the discount value, s t Representing the state of the satellite at the current time, a t Representing the tangential control action, s, performed by the satellite at the current moment t+1 The satellite state at the next moment in time, r, representing the current moment in time t Indicating a reward.
6. A satellite east-west preservation strategy model based on Nature DQN, characterized in that the model is built using the modeling method of any of claims 1-5.
7. A method for acquiring satellite east-west preservation optimal strategies, which is characterized in that a satellite east-west preservation strategy model based on Nature DQN is established according to the modeling method of any one of claims 1-5;
obtaining an optimal strategy according to the model;
the method for obtaining the optimal strategy according to the model adopts a formula 5:
wherein, pi represents the strategy of tangential control of the satellite, pi * Represents the optimal tangential control strategy learned by the model, i.e. the satellite passes through the strategy pi under the condition that the satellite state is s at the initial moment * Produces the greatest return under tangential control behavior a.
8. An electronic device, comprising:
one or more processors;
storage means for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-5.
9. A computer-readable medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211408082.4A CN115758706B (en) | 2022-11-10 | 2022-11-10 | Modeling method, model and acquisition method of satellite east-west maintenance strategy model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211408082.4A CN115758706B (en) | 2022-11-10 | 2022-11-10 | Modeling method, model and acquisition method of satellite east-west maintenance strategy model |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115758706A true CN115758706A (en) | 2023-03-07 |
| CN115758706B CN115758706B (en) | 2023-04-25 |
Family
ID=85369194
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211408082.4A Active CN115758706B (en) | 2022-11-10 | 2022-11-10 | Modeling method, model and acquisition method of satellite east-west maintenance strategy model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115758706B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20210103841A1 (en) * | 2019-10-07 | 2021-04-08 | Intelligent Fusion Technology, Inc. | Method and apparatus for rapid discovery of satellite behavior |
| CN114090537A (en) * | 2022-01-20 | 2022-02-25 | 北京航天驭星科技有限公司 | Real-time analysis method, device, system, equipment and medium for satellite in-orbit state |
| CN114362810A (en) * | 2022-01-11 | 2022-04-15 | 重庆邮电大学 | Low-orbit satellite beam hopping optimization method based on migration depth reinforcement learning |
| CN114802817A (en) * | 2022-05-27 | 2022-07-29 | 中国科学院软件研究所 | Satellite attitude control method and device based on multi-flywheel array |
| CN114933028A (en) * | 2022-07-21 | 2022-08-23 | 北京航天驭星科技有限公司 | Dual-star-orbit control strategy control method and device, electronic equipment and storage medium |
| CN114967453A (en) * | 2022-05-25 | 2022-08-30 | 北京理工大学 | Satellite east-west coordination state initial value estimation method based on neural network |
-
2022
- 2022-11-10 CN CN202211408082.4A patent/CN115758706B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20210103841A1 (en) * | 2019-10-07 | 2021-04-08 | Intelligent Fusion Technology, Inc. | Method and apparatus for rapid discovery of satellite behavior |
| CN114362810A (en) * | 2022-01-11 | 2022-04-15 | 重庆邮电大学 | Low-orbit satellite beam hopping optimization method based on migration depth reinforcement learning |
| CN114090537A (en) * | 2022-01-20 | 2022-02-25 | 北京航天驭星科技有限公司 | Real-time analysis method, device, system, equipment and medium for satellite in-orbit state |
| CN114967453A (en) * | 2022-05-25 | 2022-08-30 | 北京理工大学 | Satellite east-west coordination state initial value estimation method based on neural network |
| CN114802817A (en) * | 2022-05-27 | 2022-07-29 | 中国科学院软件研究所 | Satellite attitude control method and device based on multi-flywheel array |
| CN114933028A (en) * | 2022-07-21 | 2022-08-23 | 北京航天驭星科技有限公司 | Dual-star-orbit control strategy control method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115758706B (en) | 2023-04-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Boain | AB-Cs of sun-synchronous orbit mission design | |
| US9798008B2 (en) | Method of guidance for placing a satellite on station | |
| JP7250221B2 (en) | quasi-satellite orbit | |
| Lagona et al. | Autonomous trajectory optimisation for intelligent satellite systems and space traffic management | |
| CN115795816B (en) | Modeling method, model and acquisition method of satellite east-west maintenance strategy model | |
| CN115758707B (en) | Modeling method, system and acquisition method of satellite east-west maintenance strategy model | |
| CN115795817B (en) | Modeling method, system and acquisition method of satellite east-west maintenance strategy model | |
| Parsay et al. | Drift-free solar sail formations in elliptical Sun-synchronous orbits | |
| Fraser | Adaptive extended Kalman filtering strategies for autonomous relative navigation of formation flying spacecraft | |
| Yang et al. | A station-keeping control method for GEO spacecraft based on autonomous control architecture | |
| Herrmann et al. | A comparison of deep reinforcement learning algorithms for earth-observing satellite scheduling | |
| CN115758706A (en) | Modeling method, model and acquisition method of east-west retention strategy model of satellite | |
| WO2023234420A1 (en) | Eigenmotion control for near rectilinear halo orbits | |
| Herrmann et al. | Autonomous spacecraft tasking using monte carlo tree search methods | |
| CN115806060B (en) | Modeling method, model and acquisition method of satellite relative phase maintaining strategy model | |
| CN115806062B (en) | Modeling method, system and acquisition method of satellite relative phase maintaining strategy model | |
| CN115758705B (en) | Modeling method, system and acquisition method for satellite north-south maintenance strategy model | |
| CN115892516B (en) | Modeling method, model and acquisition method of satellite relative phase maintaining strategy model | |
| CN115865166B (en) | Modeling method, system and acquisition method for satellite north-south maintenance strategy model | |
| Cianciolo et al. | Autonomous aerobraking development software: Phase 2 summary | |
| CN115865167A (en) | Modeling method, model and acquisition method of satellite north-south conservation strategy model | |
| CN115758704B (en) | Modeling method, system and acquisition method for satellite north-south maintenance strategy model | |
| CN115806061B (en) | Modeling method, model and acquisition method of satellite relative phase maintaining strategy model | |
| Zhang et al. | GTOC11: Methods and results from the team of Harbin Institute of Technology | |
| Surovik | Autonomous mission design in extreme orbit environments |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address |
Address after: Room 301, 3rd Floor, Building 2, Yard 1, Shangdi East Road, Haidian District, Beijing, 100085 Patentee after: Beijing Aerospace Yuxing Technology Co.,Ltd. Country or region after: China Address before: Room A601, 6th Floor, Building 1, Shengjing Entrepreneurship Park T01, Tujing Village, South of Dengzhuang South Road and West of Youyi Road, Xibeiwang Town, Haidian District, Beijing Patentee before: BEIJING AEROSPACE SATELLITEHERD SCIENCE AND TECHNOLOGY CO.,LTD. Country or region before: China |
|
| CP03 | Change of name, title or address |