CN115806062B - Modeling method, system and acquisition method of satellite relative phase maintaining strategy model - Google Patents
Modeling method, system and acquisition method of satellite relative phase maintaining strategy model Download PDFInfo
- Publication number
- CN115806062B CN115806062B CN202211410211.3A CN202211410211A CN115806062B CN 115806062 B CN115806062 B CN 115806062B CN 202211410211 A CN202211410211 A CN 202211410211A CN 115806062 B CN115806062 B CN 115806062B
- Authority
- CN
- China
- Prior art keywords
- satellite
- semi
- long axis
- axis control
- states
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 80
- 230000006399 behavior Effects 0.000 claims abstract description 40
- 238000013528 artificial neural network Methods 0.000 claims abstract description 32
- 238000012549 training Methods 0.000 claims abstract description 29
- 238000012423 maintenance Methods 0.000 claims abstract description 18
- 230000009471 action Effects 0.000 claims description 39
- 230000008859 change Effects 0.000 claims description 17
- 238000011217 control strategy Methods 0.000 claims description 6
- 230000014759 maintenance of location Effects 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 2
- 239000000446 fuel Substances 0.000 abstract description 8
- 230000006870 function Effects 0.000 description 18
- 230000007774 longterm Effects 0.000 description 16
- 230000002787 reinforcement Effects 0.000 description 14
- 230000000875 corresponding effect Effects 0.000 description 9
- 238000013135 deep learning Methods 0.000 description 9
- 230000008569 process Effects 0.000 description 9
- 238000004422 calculation algorithm Methods 0.000 description 7
- 238000012545 processing Methods 0.000 description 7
- 230000008901 benefit Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 6
- 238000004364 calculation method Methods 0.000 description 4
- 238000013461 design Methods 0.000 description 3
- 238000010295 mobile communication Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- PEDCQBHIVMGVHV-UHFFFAOYSA-N Glycerine Chemical compound OCC(O)CO PEDCQBHIVMGVHV-UHFFFAOYSA-N 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000005484 gravity Effects 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000008447 perception Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000004088 simulation Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000001174 ascending effect Effects 0.000 description 1
- 230000003542 behavioural effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 230000001351 cycling effect Effects 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013213 extrapolation Methods 0.000 description 1
- 238000011478 gradient descent method Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 239000002245 particle Substances 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 239000003380 propellant Substances 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
Images
Landscapes
- Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)
Abstract
The invention relates to the aerospace field, and provides a modeling method, a system, an acquisition method, equipment and a medium of a satellite relative phase maintenance strategy model, wherein the modeling method comprises the following steps: s1: acquiring a plurality of groups of satellite training state data groups; s2: obtaining all semi-long axis control behaviors and Q values corresponding to the semi-long axis control behaviors after the initial moment; s3: acquiring states of a first satellite and a second satellite at the current moment; obtaining a semi-long axis control behavior; s4: obtaining rewards and states of the first satellite and the second satellite at the next moment; s5: storing the satellite combination state data set in an experience pool; s6: calculating a target value; s7: calculating an error; s8: updating the Q value; taking the states of the first satellite and the second satellite at the next moment as the states of the first satellite and the second satellite at the current moment; s9: repeating S3-S8, and updating the weight parameters of the target neural network; s10: and repeating S2-S9 until the data input is completed. The scheme can obtain an optimal decision strategy and reduce the consumption of satellite fuel.
Description
Technical Field
The invention relates to the technical field of aerospace, in particular to a modeling method, a system, an acquisition method, equipment and a medium of a satellite relative phase maintaining strategy model based on Nature DQN.
Background
With the continuous development of human aerospace activities, more and more remote sensing satellites provide assistance for daily life of people.
The satellite constellation generally requires that each satellite in the constellation maintains a certain phase in the running process, and certain errors exist between the actual phase and the nominal phase of the satellite due to the influence of various perturbation factors in the orbit entering and running processes, and when the magnitude of the errors reduces the performance of the constellation, the phase of the satellite is controlled, so that the errors of the actual phase and the nominal phase of the satellite are eliminated.
The complete autonomous orbit maintenance can effectively reduce the running cost of the satellite and improve the capability of the satellite to cope with emergencies. If autonomous orbit maintenance of the MEO satellite can be achieved, the working capacity of the constellation can be greatly improved while the maintenance cost is reduced. Satellites with fully autonomous orbit maintenance capability must have fully autonomous navigation and orbit control. The lifetime of the satellite will be determined primarily by the fuel it carries and an effective phase control method will extend the lifetime of the satellite.
The prior art method firstly analyzes satellite phase change caused by the influence of various perturbation forces such as earth shape, sun-moon attraction and the like on satellites in a constellation in the orbit running process through a dynamic model, then obtains a conclusion that the phase deviation can be eliminated indirectly by adjusting a semi-long axis according to the relation between the phase deviation and the semi-long axis deviation, and then designs a strategy for maintaining relative phases, so as to optimize the maintenance parameters and calculate the consumption of the propellant. In the prior art, the satellite cannot be accurately modeled due to the complexity of space stress and uncertainty of parameters of the satellite, and the satellite cannot be accurately modeled due to multiple parameters and complex calculation, so that the accuracy of satellite phase maintenance is influenced, and more fuel can be consumed.
Therefore, development of a modeling method, a model, an acquisition method, equipment and a medium for a satellite relative phase holding strategy model based on Nature DQN is needed, modeling difficulty is reduced, and a relative phase holding strategy is accurately calculated.
Disclosure of Invention
The invention aims to provide a modeling method, a system, an acquisition method, equipment and a medium for a satellite relative phase maintenance strategy model, which do not need to carry out complex modeling when carrying out relative phase position maintenance on a MEO triaxial stabilized satellite, do not need to consider the complexity of space stress and the uncertainty of the satellite self parameters, have strong behavior decision-making capability in reinforcement learning, can obtain an optimal decision strategy and reduce the consumption of satellite fuel.
In order to solve the above technical problems, as one aspect of the present invention, there is provided a modeling method of a satellite relative phase maintaining strategy model based on Nature DQN, comprising the steps of:
s1: initializing a model to obtain a plurality of groups of satellite training state data sets, wherein each group of satellite training state data sets comprises states of a first satellite and a second satellite at an initial moment, a plurality of expected orbit control moments and expected orbit control times; the states of the first satellite and the second satellite comprise relative phase differences of the first satellite and the second satellite;
s2: inputting the states of a first satellite and a second satellite at the initial moment of a group of satellite training state data sets into the model to obtain all semi-long axis control behaviors and Q values which are output correspondingly after the initial moment;
s3: acquiring states of a first satellite and a second satellite at the current moment, and acquiring semi-long axis control behaviors executed by the first satellite or the second satellite according to a greedy strategy;
s4: executing a semi-long axis control action to obtain states of the first satellite and the second satellite at the next moment, and rewarding according to the states of the first satellite and the second satellite at the next moment and a relative phase maintaining strategy rewarding function; the relative phase-preserving strategy reward function employs equation 1:
Wherein r is t For the first satellite or the second satellite at the current momentRewards, Δλ, obtained by row semi-major axis control behavior 0 A relative phase difference of the first satellite and the second satellite which are in nominal orbit, delta lambda s Maintaining a threshold for the relative phase difference of the first satellite and the second satellite, Δλ t+1 For the relative phase difference between the first satellite and the second satellite at the next moment, |delta lambda t+l -Δλ 0 The I is the change of the relative phase difference between the first satellite and the second satellite relative to the nominal orbit at the next moment of the current moment after the semi-long axis control action is performed on the first satellite or the second satellite at the current moment, namely the influence of the semi-long axis control action performed on the first satellite or the second satellite at the current moment on the relative phase difference between the first satellite and the second satellite; t is the current moment; t is t 0 The expected track control time closest to the current time;
s5: storing the states of the first satellite and the second satellite at the current moment, the semi-long axis control behavior executed by the first satellite or the second satellite, rewards and the states of the first satellite and the second satellite at the next moment as a group of satellite combination state data sets into an experience pool;
s6: taking out a plurality of groups of satellite combined state data sets from the experience pool, and calculating a target value of each combined satellite state data set according to the target neural network weight parameters;
S7: calculating an error according to the loss function, and updating the weight parameter of the current neural network;
s8: updating the Q value according to the value function; taking the states of the first satellite and the second satellite at the next moment as the states of the first satellite and the second satellite at the current moment;
s9: repeating steps S3-S8, the number of times steps S3-S8 are performed being equal to the expected orbit control number of the set of satellite training state data sets; after each repetition of the steps S3-S8 of the appointed iteration times, updating the weight parameters of the target neural network according to the weight parameters of the current neural network;
s10: steps S2-S9 are repeated until all the data of the satellite training state data set has been entered.
According to an exemplary embodiment of the present invention, initializing the model includes defining a loss function in step S1.
According to an exemplary embodiment of the present invention, the input of the model is the states of the first satellite and the second satellite, and the output is the return value (Q value) after the first satellite or the second satellite performs the semi-long axis control action.
According to an exemplary embodiment of the present invention, in step S3, at the initial cycle, the states of the first satellite and the second satellite at the current time are the states of the first satellite and the second satellite at the initial time.
According to an exemplary embodiment of the present invention, in step S3, the method for obtaining the semi-long axis control behavior performed by the first satellite or the second satellite according to the greedy strategy includes: the first satellite or the second satellite randomly selects the semi-long axis control behavior according to the first specified probability or executes the semi-long axis control behavior corresponding to the maximum Q value according to the second specified probability; the sum of the first specified probability and the second specified probability is equal to 1.
According to an exemplary embodiment of the present invention, in step S6, the method for calculating the target value of each satellite combination status data set according to the target neural network weight parameter uses formula 3:
wherein y is j Representing the target value, gamma being the discount value, theta' being the target neural network weight parameter,
representing the maximum Q value, s, of a first satellite or a second satellite in a group of satellite combined state data sets after the next moment in time performs semi-long axis control action a j+1 Representing the states of the first satellite and the second satellite at the next moment in the set of satellite combination state data sets, a represents the semi-long axis control action executed by the first satellite or the second satellite at the current moment in the set of satellite combination state data sets, r j Representing rewards in a set of satellite combination status data sets.
According to an exemplary embodiment of the present invention, in step S7, the loss function uses formula 4:
wherein y is j Represents the target value, θ is the current neural network weight parameter, Q (s j ，a j The method comprises the steps of carrying out a first treatment on the surface of the θ) represents the semi-long axis control behavior a performed by the first satellite or the second satellite at the current time in the set of satellite combined state data sets j Q, s after j Representing the states of a first satellite and a second satellite at the current moment in a set of satellite combined state data sets, a j And representing semi-long axis control behaviors executed by the first satellite or the second satellite at the current moment, wherein m is the number of satellite combination state data sets.
According to an exemplary embodiment of the present invention, in step S8, the method for updating the Q value according to the value function uses formula 5:
Q(s t ，a t )←Q(s t ，a t )+α[r t +γmax Q(s t+1 ，a t )-Q(s t ，a t )] (5)；
wherein Q(s) on the left side of the arrow t ，a t ) The first satellite or the second satellite representing the updated current moment performs the semi-long axis control action a t The Q value at the rear, Q(s) at the right side of the arrow t ，a t ) The first satellite or the second satellite representing the current moment before updating executes the semi-long axis control action a t Q(s) t +l，a t ) The first satellite or the second satellite performs the semi-long axis control action a at the next time of the current time before the update t The Q value, alpha is weight, gamma is discount value, s t Representing the states of the first satellite and the second satellite at the current moment, a t Representing semi-long axis control behavior performed by the first satellite or the second satellite at the current moment s t+1 Representing the states of the first satellite and the second satellite at the next moment of the current moment, r t Indicating a reward.
the time t is the current time, and the time t+1 is the time next to the current time.
As a second aspect of the present invention, there is provided a system for modeling a satellite relative phase holding strategy based on Nature DQN, the system being constructed by using the modeling method of the satellite relative phase holding strategy model based on Nature DQN.
As a third aspect of the present invention, there is provided a method for obtaining a satellite relative phase maintenance optimum strategy, wherein a modeling method of the satellite relative phase maintenance strategy model based on Nature DQN is adopted to build a satellite relative phase maintenance strategy model based on Nature DQN;
obtaining an optimal strategy according to the model;
the method for obtaining the optimal strategy according to the model adopts a formula 6:
wherein pi represents a strategy of semi-long axis control of the first satellite or the second satellite, pi * Representing the optimal semi-long axis control strategy learned by the model, namely passing the strategy pi under the condition that the states of the first satellite and the second satellite at the initial moment are s * The maximum return is generated under the semi-long axis control behavior a.
As a fourth aspect of the present invention, there is provided an electronic apparatus comprising:
one or more processors;
a storage means for storing one or more programs;
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement a modeling method of the Nature DQN-based satellite relative phase maintenance policy model.
As a fifth aspect of the present invention, there is provided a computer-readable medium having stored thereon a computer program which, when executed by a processor, implements a modeling method of the Nature DQN-based satellite relative phase retention policy model.
The beneficial effects of the invention are as follows:
according to the scheme, modeling is performed through the neural network, the current first satellite state data and the current second satellite state data are utilized to perform deep reinforcement learning and decision making, complex modeling is not needed by utilizing various perturbation forces received by the satellite in the orbit running process, an optimal relative phase control strategy can be obtained, consumption of satellite fuel can be reduced, and the scheme has important significance and value for practical aerospace application.
Drawings
Fig. 1 schematically shows a step diagram of a modeling method of a satellite relative phase-maintaining strategy model based on Nature DQN.
Fig. 2 schematically shows a block diagram of an electronic device.
Fig. 3 schematically shows a block diagram of a computer readable medium.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments can be embodied in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and thus a repetitive description thereof will be omitted.
Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the present application. One skilled in the relevant art will recognize, however, that the aspects of the application can be practiced without one or more of the specific details, or with other methods, components, devices, steps, etc. In other instances, well-known methods, devices, implementations, or operations are not shown or described in detail to avoid obscuring aspects of the application.
The block diagrams depicted in the figures are merely functional entities and do not necessarily correspond to physically separate entities. That is, the functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.
The flow diagrams depicted in the figures are exemplary only, and do not necessarily include all of the elements and operations/steps, nor must they be performed in the order described. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the order of actual execution may be changed according to actual situations.
It will be understood that, although the terms first, second, third, etc. may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one element from another element. Thus, a first component discussed below could be termed a second component without departing from the teachings of the present application concept. As used herein, the term "and/or" includes any one of the associated listed items and all combinations of one or more.
Those skilled in the art will appreciate that the drawings are schematic representations of example embodiments, and that the modules or flows in the drawings are not necessarily required to practice the present application, and therefore, should not be taken to limit the scope of the present application.
The scheme obtains observation information from the environment based on the perception capability with strong deep learning, and obtains an expected return value to evaluate the footstock value based on the decision capability with strong reinforcement learning. The entire learning process can be described as: at a certain moment, the satellite interacts with the flying environment to acquire the observation information, the current state information is mapped into corresponding actions (control actions) through the neural network, the environment reacts to the actions to obtain corresponding reward values and next observation information, and the complete interaction information is stored in the experience pool. By continuously cycling the above processes, an optimal strategy for achieving the objective can be finally obtained.
The satellite in the scheme is an MEO satellite. Middle orbit (MEO) earth satellites mainly refer to earth satellites whose satellite orbits are 2000-20000 km from the earth's surface. The satellite belongs to a geosynchronous satellite, is mainly used as supplement and expansion of a land mobile communication system, and is organically combined with a ground public network to realize global personal mobile communication. May also be used as a satellite navigation system. Therefore, it has great advantages in global personal mobile communications and satellite navigation systems. The medium orbit satellite has the advantages of both stationary orbit and low orbit earth satellite, and can realize real global coverage and more effective frequency reuse.
The satellite is used for completing the tasks of global communication, global navigation, global environment monitoring and the like including two-pole areas, and any place on the earth must be covered by the satellite at any time. It is not enough to do this with a single satellite or one satellite ring, and several satellite rings need to be configured in a certain way to form a satellite network, i.e. a constellation. A satellite constellation is a collection of satellites that are launched into orbit to function properly, typically a satellite network consisting of a number of satellite rings configured in a certain manner. The main satellite constellation includes GPS satellite constellation, GLONASS satellite constellation, galileo satellite constellation, beidou satellite constellation, etc.
Deep Q Networks (DQN) algorithms are one type of network in Deep reinforcement learning, which is a combination of Deep learning and Q learning. Since it combines the advantages of reinforcement learning and deep learning, it has been widely used in various fields at present.
Deep reinforcement learning is used as a new research hotspot in the field of artificial intelligence, and combines deep learning and reinforcement learning, so that direct control and decision from original input to output are realized through an end-to-end learning mode. Because the deep learning is based on a neural network structure, the deep learning has stronger perceptibility to the environment, but lacks a certain decision control capability; whereas reinforcement learning happens to have very strong behavioural decision-making capability. Therefore, the deep reinforcement learning combines the perception capability of the deep learning and the decision capability of the reinforcement learning, has complementary advantages, and can directly learn the control strategy from the high-dimensional original data. Since the deep reinforcement learning method is proposed, a substantial breakthrough is made in a plurality of tasks requiring to perceive high-dimensional original input data and decision control, and the deep reinforcement learning can solve the problems of difficult modeling and difficult planning due to the end-to-end learning advantage of the deep learning.
The DQN algorithm uses the same network for calculating the target value and the current value, i.e. the target value is calculated by using parameters in the Q network to be trained currently, and the target value is used for updating the parameters of the network, so that the two are circularly dependent, and the convergence of the algorithm is not facilitated. Compared with the DQN, the Nature DQN is added with a target network, the dependency relationship between the target Q value calculation and the Q network parameters to be updated is reduced through a double-network structure, and the advantages of reinforcement learning and deep learning are integrated, so that the stability of the DQN algorithm is greatly improved.
Nature DQN reduces the correlation between the target value of the calculated target network and the current network parameters by using two independent but identical Q networks (one as the current Q network and the other as the target Q network). The current network updates the target network at intervals of a certain step length C by copying the weight parameters of the current network to the target network, and the target Q value is kept unchanged for a period of time by the double-network structure, so that the correlation between the calculated target Q value and the current network parameters is reduced, and the convergence and stability of the algorithm are improved.
As a first embodiment of the present invention, there is provided a modeling method of a satellite relative phase holding strategy model based on Nature DQN, as shown in fig. 1, comprising the steps of:
S1: initializing a model to obtain a plurality of groups of satellite training state data sets, wherein each group of satellite training state data sets comprises states of a first satellite and a second satellite at an initial moment, a plurality of expected orbit control moments and expected orbit control times; the states of the first satellite and the second satellite include a relative phase difference of the first satellite and the second satellite.
The input of the model is the states of the first satellite and the second satellite, and the output is the return value (Q value) after the first satellite or the second satellite executes the semi-long axis control action.
The method for initializing the model comprises the following steps: defining a loss function; initializing the capacity of an experience pool to be N, wherein the experience pool is used for storing training samples; initializing a current neural network weight parameter theta and a target neural network weight parameter theta ', theta' =theta of a network model; the input of the initialization network is the state s of the first satellite and the second satellite, and the calculated network output is the return value Q after the first satellite or the second satellite executes the semi-long axis control action.
The motion state of a satellite at a certain moment can be represented by six numbers of kepler orbits: semi-long axis a 1 Eccentricity e, ascending intersection point right angle omega, near-place amplitude angle omega and orbit inclination angle i o The orbit phase angle of the satellite is λ=ω+m. I.e. the state of motion of the satellite can be expressed as { a } 1 ，e，i o Omega, M }. The states of the first satellite and the second satellite may be obtained from the motion states of the satellites, the states of the first satellite and the second satellite including a relative phase difference of the first satellite and the second satellite.
The method for obtaining the relative phase difference of the first satellite and the second satellite through the motion state of the satellites is as follows:
in the constellation building process, satellites cannot accurately enter theoretical design orbits. There is often a bias (hereinafter referred to as an orbit bias) which will also affect the long-term variation of the satellite phase relative to the design phase. This long-term change is:
wherein δλ 1 Representing a long-term change in phase;
the deviation of the semi-long axis of the track is equal to the semi-long axis of the moment t minus the semi-long axis of the initial moment; n is the average angular velocity of motion of the satellite, +.>
Wherein G is a constant of gravitational force, M is the mass of the earth; lambda (lambda) 1 For the long-term rate of change of phase of the satellite due to J2 perturbation, +.>
A is the long-term phase change rate of the satellite due to the gravity of the sun and the moon 0 The method comprises the steps that the initial orbit semi-long axis of a satellite is defined, t is the current moment, and t is the initial moment; entry 1 of the right bracket of the equal sign +.>
Deviation +. >
Long-term phase changes due to induced changes in angular velocity of satellite motion, items 2 and 3->
Is->
The part of the change in phase long term perturbation that is caused is 3 orders of magnitude smaller than item 1 and is typically negligible. The satellite orbit phase due to J2 term perturbation, solar-lunar gravitational perturbation and orbit bias is then evolved to:
wherein Deltalambda is the deviation of the actual working phase of the satellite from the designed orbit under the condition of two bodies;
the track semi-long axis deviation is equal to the semi-long axis of the t moment minus the initial moment semi-long axis; n is the average angular velocity of motion of the satellite, +.>
Wherein G is a constant of gravitational force, M is the mass of the earth; lambda (lambda) 1 For the long-term rate of change of phase of the satellite due to J2 perturbation, +.>
Is a satellite guide due to the attraction of sun and moonLong-term variability of phase of onset, a 0 For the initial orbit semi-long axis of the satellite, t is the current moment (t moment), t Initially, the method comprises Is the initial time. />
J2 perturbation refers to a long-period change in orbit root number due to earth's non-spherical shape. The condition of two bodies refers to the problem of studying the dynamics of two celestial bodies that can be regarded as particles under the action of their mutual gravitation.
As can be seen from the above equation, for a class of satellites of the same orbital altitude, eccentricity and orbital tilt, the major part of their long-term phase drift due to orbital perturbation is the same and does not produce significant relative phase changes, but the long-term phase drift of each satellite varies due to the presence of orbital misalignment. Thus, the goal of the relative phase control is to eliminate the initial tracking semi-major axis deviation.
For a first satellite (denoted by i) and a second satellite (denoted by j), the semi-major axes thereof are a respectively i And a j Phase angles of lambda respectively i And lambda (lambda) j The relative deviation of the semi-long axis is
From the above analysis, the relative phase change between the constellation satellites is obtained by the following formula:
wherein Deltalambda ij A relative phase difference between the first satellite and the second satellite;
for the orbit semi-major axis deviation of the first satellite, < >>
The orbit semi-long axis deviation of the second satellite is that the relative deviation of the first satellite and the second satellite semi-long axis is +.>
t is the current time (t time), and t is the initial time; lambda (lambda) 1，i Lambda is the long-term rate of change of phase of the first satellite due to J2 perturbation 1，j For the second satellite the rate of long-term change of phase due to J2 perturbation,/for the second satellite>
For the first satellite due to the sun-moon attraction +.>
The resulting rate of long-term change of phase, +.>
The phase long-term change rate of the second satellite due to the gravity of the sun and the moon; n is the average angular velocity of motion of the satellite, +.>
Wherein G is a constant of gravitational force, M is the mass of the earth, a 0 An initial orbit semi-major axis for the satellite;
the initial orbit semi-long axis of the satellite is equal to the average motion angular velocity of the first satellite and the second satellite; the semi-long axis deviation is equal to the semi-long axis at time t (current time) minus the initial orbit semi-long axis of the satellite.
Wherein Deltalambda ij Is the relative phase change of the first satellite relative to the second satellite. Considering that the orbit semi-long axis, the eccentricity and the inclination angle of each satellite in the constellation are the same, the long-term cheaply-priced phases of the constellation satellites under the action of orbit perturbation can be considered to be the same, and then the above formula is further simplified into:
the relative phase evolution of the constellation is mainly caused by satellite orbit deviation, so the relative phase maintenance can be realized by adjusting the semi-long axis of the satellite.
In summary, the relative phase difference between the first satellite and the second satellite at the next moment is obtained by using equation 2:
wherein Deltalambda ij A relative phase difference between the first satellite and the second satellite at the next moment;
for the orbit semi-major axis deviation of the first satellite, < >>
The orbit semi-long axis deviation of the second satellite is that the relative deviation of the first satellite and the second satellite semi-long axis is
t 1 T is the next time to time t Initially, the method comprises Is the initial time; n is the average angular velocity of motion of the satellite, +.>
Wherein G is a constant of gravitational force, M is the mass of the earth; a, a 0 The initial orbit half long axis of the satellite is equal to the average motion angular velocity of the first satellite and the second satellite; the semi-major axis deviation is equal to the semi-major axis at the next time to time t minus the initial orbit semi-major axis of the satellite.
The satellite training state data sets form a data set, the data of the satellite training states in the data set is more than or equal to 100 groups, and the more the satellite state data is, the more accurate the model training result is.
The data of the satellite training state data sets are the data of the training set, and can be simulation data or combination of the simulation data and real data. The time line in a time period comprises a plurality of time points, the states of the satellites at each time point are different, and different effects can be obtained when the orbit control strategy is executed at different time points. According to the scheme, through the plurality of sets of satellite training state data sets, the satellite state of each set of satellite at the initial time corresponds to the satellite state of one time point, and the time points corresponding to the initial time of each set of satellite training state data sets are different, namely the initial time of each set of satellite training state data sets is different.
The orbit semi-major axis is one of orbit elements of an artificial satellite and indicates the size of the orbit. When the instantaneous orbit is elliptical, the semi-major axis refers to half of the major axis; when the track is round, the semi-major axis is the radius.
S2: and inputting states of the first satellite and the second satellite at the initial time of a group of satellite training state groups into the model to obtain all semi-long axis control behaviors and corresponding output Q values after the initial time.
The state of the first satellite and the second satellite at the current moment is s t 。
the time t is the current time, and the next time t is the time t+1.
After the first satellite or the second satellite at the current moment executes the semi-long axis control action, the state of the first satellite or the second satellite at the next moment (input of the next network) is obtained, namely s t+1 。
After the first satellite or the second satellite at the current moment executes the semi-long axis control action, the states of the first satellite and the second satellite at the next moment are obtained, namely s t+1 . Because the relative phase of the first satellite and the second satellite needs to be adjusted, only one satellite of the first satellite or the second satellite is required to control the semi-long axis.
S3: the states of the first satellite and the second satellite at the current moment are obtained, and the semi-long axis control behavior executed by the first satellite or the second satellite is obtained according to a greedy strategy.
In the initial cycle, the states of the first satellite and the second satellite at the current moment are the states of the first satellite and the second satellite at the initial moment.
The method for obtaining semi-long axis control actions executed by the first satellite or the second satellite according to the greedy strategy comprises the following steps: the first satellite or the second satellite randomly selects the semi-long axis control behavior according to the first specified probability or executes the semi-long axis control behavior corresponding to the maximum Q value according to the second specified probability; the sum of the first specified probability and the second specified probability is equal to l.
If the first specified probability is greater than the second specified probability, the method for obtaining the semi-long axis control behavior executed by the first satellite or the second satellite according to the greedy strategy is adopted by the method: the first satellite or the second satellite randomly selects semi-long axis control behaviors with a first specified probability;
if the second specified probability is greater than the first specified probability, the method for obtaining the semi-long axis control behavior executed by the first satellite or the second satellite according to the greedy strategy is adopted by the method: executing a semi-long axis control behavior corresponding to the maximum Q value by the first satellite or the second satellite according to the second specified probability;
if the first specified probability is equal to the second specified probability, selecting one of the methods for obtaining the semi-long axis control behavior executed by the first satellite or the second satellite according to the greedy strategy: the first satellite or the second satellite randomly selects the semi-long axis control behavior according to the first specified probability or executes the semi-long axis control behavior corresponding to the maximum Q value according to the second specified probability.
The greedy strategy is epsilon-greedy strategy.
The first specified probability is epsilon, which decreases with increasing iteration number.
The semi-long axis control action executed by the first satellite or the second satellite at the current moment is a t 。
S4: executing a semi-long axis control action to obtain states of the first satellite and the second satellite at the next moment; and rewarding the function according to the states of the first satellite and the second satellite at the next moment and the relative phase maintaining strategy.
During long-term in-orbit operation of satellites, when the relative phase deviation of the ith star and the jth star in the constellation exceeds a threshold value (|delta lambda) ij |＞Δλ max ) It is necessary to perform orbital control of one of the satellites by active control of fuel consumptionThe interference of various factors is eliminated, and the satellite phase adjustment time is small relative to the service life of the navigation satellite, so the performance index only requires the least fuel consumed by the whole process.
Assuming that the satellite control frequency is fixed (i.e., orbit control is performed after a period of time has elapsed), the current control amount is expected to ensure that the double-star phase is within the holding range and the control amount is as small as possible at the next control, so that the variation of the semi-long axis at the time t (current time) determines the extrapolated phase difference of the double-star at the time t+1 (time next to the current time). For this purpose, a bonus strategy at time t is designed.
Therefore, the relative phase hold strategy bonus function (the bonus strategy at time t) is the formula i:
wherein r is t Rewards, delta lambda, obtained for semi-long axis control actions for the first satellite or the second satellite at the current moment 0 A relative phase difference of the first satellite and the second satellite which are in nominal orbit, delta lambda s Maintaining a threshold for the relative phase difference of the first satellite and the second satellite, Δλ t+1 For the relative phase difference between the first satellite and the second satellite at the next moment, |delta lambda t+1 -Δλ 0 The I is the change of the relative phase difference between the first satellite and the second satellite relative to the nominal orbit (theoretical orbit) at the next moment from extrapolation to the current moment after performing semi-long axis control action on the first satellite or the second satellite at the current moment, namely the influence of the semi-long axis control action on the relative phase difference between the first satellite and the second satellite at the current moment; giving a penalty when the relative phase difference at time t+1 (the time next to time t) is not within the desired range; t is the current time, t 0 Is the expected orbit control moment closest to the current moment.
S5: the state of the first satellite and the second satellite at the current moment, the semi-long axis control action executed by the first satellite or the second satellite, rewards and the state of the first satellite and the second satellite at the next moment are stored as a group of satellite combination state data sets into an experience pool.
S6: and taking out a plurality of groups of satellite combination state data sets from the experience pool, and calculating the target value of each satellite combination state data set according to the target neural network weight parameter.
The number of satellite combined state data sets is m, m is a natural number greater than 0, and m is less than the number of satellite training state data sets. The m sets of satellite combination state data sets are small batches of satellite combination state data sets. The number of satellite combination state data sets is determined based on the number of satellite training state data sets.
The method for calculating the target value of each satellite state data set according to the target neural network weight parameter adopts a formula 3:
wherein y is j Representing the target value, gamma being the discount value (attenuation factor), theta' being the target neural network weight parameter,
representing the maximum Q value, s, of a first satellite or a second satellite in a group of satellite combined state data sets after the next moment in time performs semi-long axis control action a j+1 Representing the states of the first satellite and the second satellite at the next moment in the set of satellite combination state data sets, a represents the semi-long axis control action executed by the first satellite or the second satellite at the current moment in the set of satellite combination state data sets, r j Representing rewards in a set of satellite combination status data sets.
And stopping the task to obtain model convergence or iteration completion. When s is j+1 When model convergence or iteration is completed, y i Equal to r j The method comprises the steps of carrying out a first treatment on the surface of the When s is j+1 When model convergence or iteration is not completed, y i Equal to
The conditions for model convergence are: the error calculated by the loss function is within a specified range.
The iteration is completed under the following conditions: all steps are performed.
S7: and calculating errors according to the loss function, and updating the weight parameters of the current neural network.
The error is also calculated from the target value.
The loss function uses equation 4:
wherein y is j Represents the target value, θ is the current neural network weight parameter, Q (s j ，a j The method comprises the steps of carrying out a first treatment on the surface of the θ) represents the control behavior a performed by the first satellite or the second satellite at the current moment in the set of satellite combined state data sets j Q, s after j Representing the current states of a first satellite and a second satellite in a set of satellite combined state data sets, a j Representing semi-long axis control action executed by the first satellite or the second satellite at the current moment, r j Representing rewards in a set of satellite combination status data sets; m is the number of satellite combined state data sets.
The error is the result of the loss function calculation using equation 4.
The current neural network weight parameters are updated by a random gradient descent method (SGD).
r t 、a t 、s t 、s t+1 Samples in a dataset representing a satellite training state dataset, r j 、a j 、s j 、s j+1 Representing samples in the experience pool.
And the steps S5-S7 are used for adjusting the parameters of the model, so that the calculation accuracy of the model is higher.
S8: updating the Q value according to the value function; and taking the states of the first satellite and the second satellite at the next moment as the states of the first satellite and the second satellite at the current moment.
The method of updating the Q value according to the value function employs equation 5:
Q(s t ，a t )←Q(s t ，a t )+α[r t +γmax Q(s t+1 ，a t )-Q(s t ，a t )] (5)；
wherein Q(s) on the left side of the arrow t ，a t ) The first satellite or the second satellite representing the updated current moment performs the semi-long axis control action a t The Q value at the rear, Q(s) at the right side of the arrow t ，a t ) The first satellite or the second satellite representing the current moment before updating executes the semi-long axis control action a t Q(s) t+1 ，a t ) The first satellite or the second satellite representing the next time of the current time before updating executes the semi-long axis control action a t The Q value, alpha is weight, gamma is discount value (attenuation factor), s t Representing the states of the first satellite and the second satellite at the current moment, a t Representing semi-long axis control behavior performed by the first satellite or the second satellite at the current moment s t+1 Representing the states of the first satellite and the second satellite at the next moment of the current moment, r t Indicating a reward.
Wherein alpha and gamma are both in the range of 0 to 1.
S9: repeating steps S3-S8, the number of times steps S3-S8 are performed being equal to the expected orbit control number of the set of satellite training state data sets; and after each step S3-S8 of the appointed iteration times is repeatedly executed, updating the weight parameters of the target neural network according to the weight parameters of the current neural network.
After the iteration of the appointed iteration times is completed, the target neural network weight parameter is updated to the current neural network weight parameter.
S10: steps S2-S9 are repeated until all the data of the satellite training state data set has been entered.
According to the modeling method, the state data of the first satellite and the second satellite are used as the input of the neural network model, the generated return value is used as the output, the Nature DQN neural network is adopted, complex modeling is carried out without utilizing various power received by the satellites in the orbit running process, deep reinforcement learning is directly adopted for learning and decision making, the improvement is carried out based on the DQN algorithm, the method is suitable for training a large-scale neural network, the stability of the DQN algorithm is greatly improved, the consumption of satellite fuel can be reduced, and the method has important significance and value for practical aerospace application.
According to a second specific embodiment of the present invention, the present invention provides a system for establishing a satellite relative phase maintaining strategy based on Nature DQN, using the modeling method of the satellite relative phase maintaining strategy model based on Nature DQN of the first embodiment.
According to a third specific embodiment of the present invention, the present invention provides a method for obtaining a satellite relative phase maintenance optimal strategy, and the method for modeling a satellite relative phase maintenance strategy model based on Nature DQN of the first embodiment is used to build a satellite relative phase maintenance strategy model based on Nature DQN;
And obtaining an optimal strategy according to the model.
The method for obtaining the optimal strategy according to the model adopts the formula 5:
wherein pi represents a strategy of semi-long axis control of the satellite, pi * Representing the optimal semi-long axis control strategy learned by the model, namely passing the strategy pi under the condition that the states of the first satellite and the second satellite at the initial moment are s * The maximum return is generated under the semi-long axis control behavior a.
According to a fourth embodiment of the present invention, an electronic device is provided, as shown in fig. 2, and fig. 2 is a block diagram of an electronic device according to an exemplary embodiment.
An electronic device 200 according to this embodiment of the present application is described below with reference to fig. 2. The electronic device 200 shown in fig. 2 is only an example and should not be construed as limiting the functionality and scope of use of the embodiments herein.
As shown in fig. 2, the electronic device 200 is in the form of a general purpose computing device. The components of the electronic device 200 may include, but are not limited to: at least one processing unit 210, at least one memory unit 220, a bus 230 connecting the different system components (including the memory unit 220 and the processing unit 210), a display unit 240, and the like.
Wherein the storage unit stores program code that is executable by the processing unit 210 such that the processing unit 210 performs the steps described in the present specification according to various exemplary embodiments of the present application. For example, the processing unit 210 may perform the steps as shown in fig. 1.
The memory unit 220 may include readable media in the form of volatile memory units, such as Random Access Memory (RAM) 2201 and/or cache memory 2202, and may further include Read Only Memory (ROM) 2203.
The storage unit 220 may also include a program/utility 2204 having a set (at least one) of program modules 2205, such program modules 2205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.
The electronic device 200 may also communicate with one or more external devices 200' (e.g., keyboard, pointing device, bluetooth device, etc.), devices that enable a user to interact with the electronic device 200, and/or any devices (e.g., routers, modems, etc.) that the electronic device 200 can communicate with one or more other computing devices. Such communication may occur through an input/output (I/O) interface 250. Also, the electronic device 200 may communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet, through a network adapter 260. Network adapter 260 may communicate with other modules of electronic device 200 via bus 230. It should be appreciated that although not shown, other hardware and/or software modules may be used in connection with electronic device 200, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and the like.
From the above description of embodiments, those skilled in the art will readily appreciate that the example embodiments described herein may be implemented in software, or may be implemented in software in combination with the necessary hardware.
Thus, according to a fifth embodiment of the present invention, the present invention provides a computer readable medium. As shown in fig. 3, the technical solution according to the embodiment of the present invention may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (may be a CD-ROM, a U-disk, a mobile hard disk, etc.) or on a network, and includes several instructions to cause a computing device (may be a personal computer, a server, or a network device, etc.) to perform the above-described method according to the embodiment of the present invention.
The software product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, with readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A readable storage medium may also be any readable medium that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).
The computer-readable medium carries one or more programs, which when executed by one of the devices, cause the computer-readable medium to implement the functions of the first embodiment.
Those skilled in the art will appreciate that the modules may be distributed throughout several devices as described in the embodiments, and that corresponding variations may be implemented in one or more devices that are unique to the embodiments. The modules of the above embodiments may be combined into one module, or may be further split into a plurality of sub-modules.
From the above description of embodiments, those skilled in the art will readily appreciate that the example embodiments described herein may be implemented in software, or in combination with the necessary hardware. Thus, the technical solution according to the embodiments of the present invention may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (may be a CD-ROM, a U-disk, a mobile hard disk, etc.) or on a network, and includes several instructions to cause a computing device (may be a personal computer, a server, a mobile terminal, or a network device, etc.) to perform the method according to the embodiments of the present invention.
The above is only a preferred embodiment of the present invention, and is not intended to limit the present invention, but various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. The modeling method of the satellite relative phase maintaining strategy model based on the Nature DQN is characterized by comprising the following steps of:
s1: initializing a model to obtain a plurality of groups of satellite training state data sets, wherein each group of satellite training state data sets comprises states of a first satellite and a second satellite at an initial moment, a plurality of expected orbit control moments and expected orbit control times; the states of the first satellite and the second satellite comprise relative phase differences of the first satellite and the second satellite;
s2: inputting the states of a first satellite and a second satellite in an initial time of a group of satellite training state data sets into the model to obtain all semi-long axis control behaviors and Q values corresponding to the semi-long axis control behaviors after the initial time, namely return values;
s3: acquiring states of a first satellite and a second satellite at the current moment, and acquiring semi-long axis control behaviors executed by the first satellite or the second satellite according to a greedy strategy;
S4: executing a semi-long axis control action to obtain states of the first satellite and the second satellite at the next moment; obtaining rewards according to the states of the first satellite and the second satellite at the next moment and a relative phase maintaining strategy rewarding function; the relative phase-preserving strategy reward function employs equation 1:
wherein r is t Rewards, delta lambda, obtained for semi-long axis control actions for the first satellite or the second satellite at the current moment 0 A relative phase difference of the first satellite and the second satellite which are in nominal orbit, delta lambda s Maintaining a threshold for the relative phase difference of the first satellite and the second satellite, Δλ t+1 For the relative phase difference between the first satellite and the second satellite at the next moment, |delta lambda t+1 -Δλ 0 The I is the change of the relative phase difference between the first satellite and the second satellite relative to the nominal orbit at the next moment after the semi-long axis control action is performed on the first satellite or the second satellite at the current moment, namely the influence of the semi-long axis control action performed on the first satellite or the second satellite at the current moment on the relative phase difference between the first satellite and the second satellite; t is the current moment; t is t 0 The expected track control time closest to the current time;
s5: storing the states of the first satellite and the second satellite at the current moment, the semi-long axis control behavior executed by the first satellite or the second satellite, rewards and the states of the first satellite and the second satellite at the next moment as a group of satellite combination state data sets into an experience pool;
S6: taking out a plurality of groups of satellite combination state data sets from the experience pool, and calculating a target value of each satellite combination state data set according to the target neural network weight parameters;
s7: calculating an error according to the loss function, and updating the weight parameter of the current neural network;
s8: updating the Q value according to the value function; taking the states of the first satellite and the second satellite at the next moment as the states of the first satellite and the second satellite at the current moment;
s9: repeating steps S3-S8, the number of times steps S3-S8 are performed being equal to the expected orbit control number of the set of satellite training state data sets; after each repetition of the steps S3-S8 of the appointed iteration times, updating the weight parameters of the target neural network according to the weight parameters of the current neural network;
s10: steps S2-S9 are repeated until all the data of the satellite training state data set has been entered.
2. The modeling method of a satellite relative phase retention policy model according to claim 1, wherein in step S3, at the initial cycle, the states of the first satellite and the second satellite at the current time are the states of the first satellite and the second satellite at the initial time.
3. The modeling method of a Nature DQN-based satellite relative phase retention strategy model according to claim 1, wherein in step S3, the method of obtaining semi-long axis control behavior performed by the first satellite or the second satellite according to a greedy strategy comprises: the first satellite or the second satellite randomly selects the semi-long axis control behavior according to the first specified probability or executes the semi-long axis control behavior corresponding to the maximum Q value according to the second specified probability; the sum of the first specified probability and the second specified probability is equal to 1.
4. The modeling method of a Nature DQN-based satellite relative phase retention policy model according to claim 1, wherein in step S6, the method of calculating the target value of each satellite state data set according to the target neural network weight parameter uses formula 3:
wherein y is j Representing the target value, gamma being the discount value, theta' being the target neural network weight parameter,
representing the maximum Q value, s, of the first satellite or the second satellite in the satellite combination state data set after the next moment in time executes the semi-long axis control action a j+1 Representing the states of the first satellite and the second satellite at the next moment in the satellite combination state data set, a represents the semi-long axis control action executed by the first satellite or the second satellite, and r j Representing a set of satellite assembliesRewards in the status data set.
5. The modeling method of a Nature DQN-based satellite relative phase retention strategy model according to claim 1, wherein in step S7, the loss function uses formula 4:
wherein y is j Represents the target value, θ is the current neural network weight parameter, Q (s j ，a j The method comprises the steps of carrying out a first treatment on the surface of the θ) represents the semi-long axis control behavior a performed by the first satellite or the second satellite at the current time in the set of satellite combined state data sets j Q, s after j Representing the states of a first satellite and a second satellite at the current moment in a set of satellite combined state data sets, a j And representing semi-long axis control behaviors executed by the first satellite or the second satellite at the current moment, wherein m is the number of satellite combination state data sets.
6. The modeling method of a Nature DQN-based satellite relative phase retention policy model according to claim 1, wherein in step S8, the method of updating Q value according to a value function uses formula 5:
Q(s t ，a t )←Q(s t ，a t )+α[r t +γmaxQ(s t+1 ，a t )-Q(s t ，a t )] (5)；
wherein Q(s) on the left side of the arrow t ，a t ) Representing the updated current time when the first satellite or the second satellite executes the semi-long axis control action a t The Q value at the rear, Q(s) at the right side of the arrow t ，a t ) The first satellite or the second satellite representing the current moment before updating executes the semi-long axis control action a t Q(s) t+1 ，a t ) The first satellite or the second satellite performs the semi-long axis control action a at the next time of the current time before the update t The Q value, alpha is weight, gamma is discount value,s t representing the states of the first satellite and the second satellite at the current moment, a t Representing semi-long axis control behavior performed by the first satellite or the second satellite at the current moment s t+1 Representing the states of the first satellite and the second satellite at the next moment of the current moment, r t Indicating a reward.
7. A Nature DQN-based satellite relative phase maintenance strategy system, characterized in that the system is built using the modeling method of any of claims 1-6.
8. A method for acquiring a satellite relative phase maintaining optimal strategy, which is characterized in that a satellite relative phase maintaining strategy model based on Nature DQN is established according to the modeling method of any one of claims 1-6;
obtaining an optimal strategy according to the model;
the method for obtaining the optimal strategy according to the model adopts a formula 6:
wherein pi represents a strategy of semi-long axis control of the satellite, pi * Representing the optimal semi-long axis control strategy learned by the model, namely passing the strategy pi under the condition that the states of the first satellite and the second satellite at the initial moment are s * The maximum return is generated under the semi-long axis control behavior a.
9. An electronic device, comprising:
one or more processors;
a storage means for storing one or more programs;
the one or more programs are executed by the one or more processors to cause the one or more processors to implement the method of any of claims 1-6.
10. A computer readable medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211410211.3A CN115806062B (en) | 2022-11-10 | 2022-11-10 | Modeling method, system and acquisition method of satellite relative phase maintaining strategy model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211410211.3A CN115806062B (en) | 2022-11-10 | 2022-11-10 | Modeling method, system and acquisition method of satellite relative phase maintaining strategy model |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115806062A CN115806062A (en) | 2023-03-17 |
| CN115806062B true CN115806062B (en) | 2023-05-09 |
Family
ID=85483371
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211410211.3A Active CN115806062B (en) | 2022-11-10 | 2022-11-10 | Modeling method, system and acquisition method of satellite relative phase maintaining strategy model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115806062B (en) |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| AU2017101165A4 (en) * | 2017-08-25 | 2017-11-02 | Liu, Yichen MR | Method of Structural Improvement of Game Training Deep Q-Network |
| WO2019190476A1 (en) * | 2018-03-27 | 2019-10-03 | Nokia Solutions And Networks Oy | Method and apparatus for facilitating resource pairing using a deep q-network |
| CN110378382A (en) * | 2019-06-18 | 2019-10-25 | 华南师范大学 | Novel quantization transaction system and its implementation based on deeply study |
| CN112173174A (en) * | 2020-10-15 | 2021-01-05 | 中国西安卫星测控中心 | MEO constellation phase control method |
| CN112946698B (en) * | 2021-01-29 | 2023-08-29 | 合肥工业大学智能制造技术研究院 | Satellite signal cycle slip detection method based on reinforcement learning |
-
2022
- 2022-11-10 CN CN202211410211.3A patent/CN115806062B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN115806062A (en) | 2023-03-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Boain | AB-Cs of sun-synchronous orbit mission design | |
| Sullivan et al. | Using reinforcement learning to design a low-thrust approach into a periodic orbit in a multi-body system | |
| Kiesbye et al. | Hardware-in-the-loop and software-in-the-loop testing of the move-ii cubesat | |
| Tipaldi et al. | Reinforcement learning in spacecraft control applications: Advances, prospects, and challenges | |
| Abdelkhalik et al. | Optimization of space orbits design for Earth orbiting missions | |
| Zuo et al. | A case learning-based differential evolution algorithm for global optimization of interplanetary trajectory design | |
| Singh et al. | Feasibility of quasi-frozen, near-polar and extremely low-altitude lunar orbits | |
| CN115795816B (en) | Modeling method, model and acquisition method of satellite east-west maintenance strategy model | |
| CN115758707B (en) | Modeling method, system and acquisition method of satellite east-west maintenance strategy model | |
| CN115795817B (en) | Modeling method, system and acquisition method of satellite east-west maintenance strategy model | |
| Lou et al. | A consider unscented particle filter with genetic algorithm for UAV multi-source integrated navigation | |
| Fraser | Adaptive extended Kalman filtering strategies for autonomous relative navigation of formation flying spacecraft | |
| CN115806062B (en) | Modeling method, system and acquisition method of satellite relative phase maintaining strategy model | |
| Yang et al. | A station-keeping control method for GEO spacecraft based on autonomous control architecture | |
| CN115892516B (en) | Modeling method, model and acquisition method of satellite relative phase maintaining strategy model | |
| CN115806060B (en) | Modeling method, model and acquisition method of satellite relative phase maintaining strategy model | |
| Herrmann et al. | A comparison of deep reinforcement learning algorithms for earth-observing satellite scheduling | |
| CN115806061B (en) | Modeling method, model and acquisition method of satellite relative phase maintaining strategy model | |
| CN115865166B (en) | Modeling method, system and acquisition method for satellite north-south maintenance strategy model | |
| CN115758705B (en) | Modeling method, system and acquisition method for satellite north-south maintenance strategy model | |
| Herrmann et al. | Autonomous spacecraft tasking using monte carlo tree search methods | |
| CN115758704B (en) | Modeling method, system and acquisition method for satellite north-south maintenance strategy model | |
| Adams | Theory and Applications of Gram-Scale Spacecraft | |
| CN115758706B (en) | Modeling method, model and acquisition method of satellite east-west maintenance strategy model | |
| CN115865167B (en) | Modeling method, system and acquisition method for satellite north-south maintenance strategy model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |