CN115514769A - Satellite elastic internet resource scheduling method, system, computer equipment and medium - Google Patents
Satellite elastic internet resource scheduling method, system, computer equipment and medium Download PDFInfo
- Publication number
- CN115514769A CN115514769A CN202211125448.7A CN202211125448A CN115514769A CN 115514769 A CN115514769 A CN 115514769A CN 202211125448 A CN202211125448 A CN 202211125448A CN 115514769 A CN115514769 A CN 115514769A
- Authority
- CN
- China
- Prior art keywords
- satellite
- task
- model
- delay
- resource scheduling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 58
- 238000005457 optimization Methods 0.000 claims abstract description 63
- 238000004422 calculation algorithm Methods 0.000 claims description 85
- 238000004364 calculation method Methods 0.000 claims description 31
- 101000919040 Klebsiella oxytoca (strain ATCC 8724 / DSM 4798 / JCM 20051 / NBRC 3318 / NRRL B-199 / KCTC 1686) Diol dehydratase-reactivating factor large subunit Proteins 0.000 claims description 28
- 230000005540 biological transmission Effects 0.000 claims description 25
- 230000009471 action Effects 0.000 claims description 24
- 238000012545 processing Methods 0.000 claims description 24
- 238000004891 communication Methods 0.000 claims description 20
- 230000006870 function Effects 0.000 claims description 20
- 238000005265 energy consumption Methods 0.000 claims description 15
- 230000002787 reinforcement Effects 0.000 claims description 15
- 238000004590 computer program Methods 0.000 claims description 14
- 238000005516 engineering process Methods 0.000 claims description 7
- 238000010276 construction Methods 0.000 claims description 5
- 230000007613 environmental effect Effects 0.000 claims description 4
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 238000007726 management method Methods 0.000 claims description 3
- 238000012935 Averaging Methods 0.000 claims description 2
- 239000011159 matrix material Substances 0.000 claims description 2
- 230000005251 gamma ray Effects 0.000 claims 1
- 238000004088 simulation Methods 0.000 description 20
- 239000003795 chemical substances by application Substances 0.000 description 19
- 238000009826 distribution Methods 0.000 description 11
- 238000012549 training Methods 0.000 description 11
- 230000008569 process Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 238000013468 resource allocation Methods 0.000 description 5
- 238000004458 analytical method Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 238000009827 uniform distribution Methods 0.000 description 4
- 238000013528 artificial neural network Methods 0.000 description 3
- 230000007547 defect Effects 0.000 description 3
- 230000006872 improvement Effects 0.000 description 3
- 230000003993 interaction Effects 0.000 description 2
- 210000002569 neuron Anatomy 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000000630 rising effect Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000001186 cumulative effect Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000005562 fading Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/104—Peer-to-peer [P2P] networks
- H04L67/1074—Peer-to-peer [P2P] networks for supporting data block transmission mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W28/00—Network traffic management; Network resource management
- H04W28/02—Traffic management, e.g. flow control or congestion control
- H04W28/08—Load balancing or load distribution
- H04W28/09—Management thereof
- H04W28/0958—Management thereof based on metrics or performance parameters
- H04W28/0967—Quality of Service [QoS] parameters
- H04W28/0975—Quality of Service [QoS] parameters for reducing delays
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D30/00—Reducing energy consumption in communication networks
- Y02D30/70—Reducing energy consumption in communication networks in wireless communication networks
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Molecular Biology (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Health & Medical Sciences (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Quality & Reliability (AREA)
- Radio Relay Systems (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention provides a method, a system, computer equipment and a medium for scheduling satellite elastic internet resources, which are characterized in that a time delay sensitive satellite elastic internet framework is established through a multi-to-multi mode based on an LEO satellite and a user terminal, a minimized time delay optimization model is established according to the time delay sensitive satellite elastic internet framework and a satellite elastic internet resource scheduling model, the minimized time delay optimization model is converted into a corresponding Markov decision model, and the Markov decision model is solved to obtain a resource scheduling strategy.
Description
Technical Field
The invention relates to the technical field of satellite resource scheduling, in particular to a satellite elastic internet resource scheduling method and system based on deep reinforcement learning, computer equipment and a storage medium.
Background
With the continuous expansion of internet users, user equipment accessing a network is also increasing, which brings about greater performance requirements on time delay and energy consumption balance. However, due to the limitations of cost and technical conditions, in many global areas, such as deserts, deep seas, forests and other complex natural geographic environments, regional network coverage cannot be realized by deploying ground base stations in a large scale, and the satellite internet becomes a reliable way for ensuring efficient communication in such areas.
The traditional satellite network systems are independent, the networking mechanism and related protocols have remarkable heterogeneous characteristics, so that a serious chimney forest phenomenon is caused, and the utilization efficiency of network space resources is limited to a great extent. Therefore, how to establish an efficient, flexible and agile satellite internet architecture, how to efficiently unload tasks and how to reasonably schedule and allocate limited resources to different task demands becomes an important research direction for improving the utilization efficiency of world-wide converged network resources.
The existing satellite internet resource scheduling method mainly sinks the computing capacity to an LEO satellite, arranges an edge computing technology of an MEC server on the LEO satellite, and draws the physical distance between the user and the MEC server to achieve better time delay and energy consumption balance; however, the existing methods for solving the optimization problem are mainly divided into two types: 1) The problem of task scheduling is solved by using a Hungarian method; 2) The optimization problem is proved to be a convex optimization problem and solved by KKT conditions. Although the two methods can solve the problem of internet dangerous resource scheduling to a certain extent, each method has corresponding application defects: when the Hungarian method is used for solving the task scheduling problem, although the time delay is reduced compared with the traditional algorithm, the method only considers the application scene that one server is responsible for calculating tasks, does not consider the problem that under the satellite communication scene, calculation and storage resources are scarce, and the load is unbalanced easily; the optimization problem is proved to be a convex optimization problem and solved by using the KKT condition, only the high efficiency of the solving process is considered, and the influence caused by task queuing time delay due to limited resources and large task quantity is not considered; that is, the existing solution does not consider the application difference between the satellite communication scene and the ground internet scene, and cannot really and effectively schedule resources based on the actual scenes of very scarce resources, limited calculation and storage resources, large task amount, long queuing delay and the like, and the practicability is low.
Therefore, it is urgently needed to provide a reasonable resource scheduling method which can fully utilize the limited resources on the satellite, provide low-delay, high-quality, high-security and high-security services for users in various regions around the world, and solve the problem of high delay sensitivity.
Disclosure of Invention
The invention aims to provide a satellite elastic internet resource scheduling method, which considers the problem of resource scheduling among edge servers under the edge calculation of a satellite network in a plurality of modes, completes the construction of a delay sensitive satellite elastic internet framework based on an SDN/NFV technology and a TSN (time series network) security protocol, obtains a corresponding delay optimization model, solves a resource scheduling strategy by adopting a delay optimization algorithm constructed based on a deep reinforcement learning algorithm framework, effectively makes up for the application defect of the conventional satellite resource scheduling scheme, can fully utilize the calculation and storage resources of a satellite, provides services with priority, optimizes the average delay performance and realizes the real and effective load balance.
In order to achieve the above objects, it is necessary to provide a satellite flexible internet resource scheduling method, system, computer device and storage medium for solving the above technical problems.
In a first aspect, an embodiment of the present invention provides a method for scheduling satellite flexible internet resources, where the method includes the following steps:
establishing a time delay sensitive satellite elastic Internet architecture based on a many-to-many mode of an LEO satellite and a user side; the LEO satellite corresponds to an MEC server;
establishing a satellite elastic internet resource scheduling model according to the time delay sensitive satellite elastic internet architecture;
establishing a minimum time delay optimization model according to the satellite elastic internet resource scheduling model;
converting the minimized time delay optimization model into a corresponding Markov decision model;
and solving the Markov decision model to obtain a resource scheduling strategy.
In a second aspect, an embodiment of the present invention provides a satellite flexible internet resource scheduling system, where the system includes:
the architecture construction module is used for establishing a time delay sensitive satellite elastic Internet architecture based on a many-to-many mode of an LEO satellite and a user side; the LEO satellite corresponds to an MEC server;
the first modeling module is used for establishing a satellite elastic internet resource scheduling model according to the time delay sensitive satellite elastic internet architecture;
the second modeling module is used for establishing a minimized time delay optimization model according to the satellite elastic internet resource scheduling model;
the model conversion module is used for converting the minimized time delay optimization model into a corresponding Markov decision model;
and the strategy solving module is used for solving the Markov decision model to obtain a resource scheduling strategy.
In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the steps of the method when executing the computer program.
In a fourth aspect, the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to implement the steps of the above method.
The application provides a satellite flexible internet resource scheduling method, a satellite flexible internet resource scheduling system, computer equipment and a storage medium, and the method realizes the technical scheme of establishing a delay sensitive satellite flexible internet framework based on a multi-to-multi mode of an LEO satellite and a user terminal, establishing a satellite flexible internet resource scheduling model according to the delay sensitive satellite flexible internet framework, establishing a minimized delay optimization model according to the satellite flexible internet resource scheduling model, converting the minimized delay optimization model into a corresponding Markov decision model, and solving the Markov decision model to obtain a resource scheduling strategy. Compared with the prior art, the satellite elastic internet resource scheduling method can comprehensively and effectively optimize problem modeling based on a real application scene with limited resources and the influence of task queuing delay, not only can fully utilize computing resources and storage resources of a satellite and avoid performance influence caused by queuing delay, but also can meet diversified requirements of a user side and improve service quality based on division of task priorities, can also improve computing efficiency of scheduling and distributing mass business resources based on a delay optimization algorithm specifically constructed by a deep reinforcement learning algorithm, further realizes intelligent and efficient satellite elastic internet resource scheduling with priority service, improved delay performance and load balance guarantee, and has strong generalization capability and high practical value.
Drawings
Fig. 1 is a schematic view of an application scenario of a satellite flexible internet resource scheduling method according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating a method for scheduling satellite flexible Internet resources according to an embodiment of the present invention;
fig. 3 is a schematic diagram of a time delay sensitive satellite flexible internet architecture according to an embodiment of the present invention;
FIG. 4 is a pseudo code diagram of a DDRA algorithm designed based on a TD3 architecture of a deep reinforcement learning algorithm in an embodiment of the present invention;
FIG. 5 is a diagram illustrating relevant parameters of a satellite flexible Internet resource scheduling model according to an embodiment of the present invention;
FIG. 6 is a graph of the impact of discount rate on DDRA algorithm convergence in an embodiment of the present invention;
FIG. 7 is a convergence comparison diagram of the DDRA algorithm and the SAC algorithm in the embodiment of the present invention;
FIG. 8 is a graph illustrating the effect of the calculated task amount under different algorithms on the average delay performance under the allocation of computing resources when the task data amount is uniformly distributed in the embodiment of the present invention;
FIG. 9 is a graph comparing the effect of task computation amount under different algorithms on average time-lapse performance under the allocation of computing resources when the task data amount obeys normal distribution in the embodiment of the present invention;
FIG. 10 is a graph illustrating the effect of task computation workload under different algorithms on the average time-lapse performance under the allocation of computing resources when the task computation workload is uniformly distributed in the embodiment of the present invention;
FIG. 11 is a graph illustrating a comparison of the influence of task data under different algorithms on the average delay performance under the allocation of computing resources when the task computation workload obeys normal distribution in the embodiment of the present invention;
FIG. 12 is a graph comparing the influence of the number of users under different algorithms on the average time-lapse performance under the allocation of computing resources when the task data amount and the task calculation amount are subject to normal distribution in the embodiment of the present invention;
FIG. 13 is a schematic structural diagram of a satellite flexible Internet resource scheduling system according to an embodiment of the present invention;
fig. 14 is an internal structural view of a computer device in the embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail below with reference to the accompanying drawings and embodiments, and it is obvious that the embodiments described below are part of the embodiments of the present invention, and are only used for illustrating the present invention, but not for limiting the scope of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The satellite elastic internet resource scheduling method provided by the invention can be applied to a situation that a user side and an MEC server shown in figure 1 meet satellite elastic internet resource scheduling in a many-to-many mode, and can realize intelligent efficient satellite elastic internet resource scheduling with priority service, time delay performance improvement and load balance guarantee; the following embodiments will explain the satellite flexible internet resource scheduling method of the present invention in detail.
In one embodiment, as shown in fig. 2, there is provided a satellite flexible internet resource scheduling method, including the following steps:
s11, establishing a time delay sensitive satellite elastic Internet architecture based on a many-to-many mode of an LEO satellite and a user side; the LEO satellite corresponds to one MEC server, and communication interaction is carried out among the MEC servers to realize load balance; the time delay sensitive satellite elastic internet architecture can be understood as a satellite internet architecture shown in fig. 3 which is established based on an SDN/NFV technology and combined with the basic idea of IEEE 802.1 Qcc in TSN, an MEC server of each LEO satellite provides task unloading service for a ground data node (user end), and the satellite internet architecture is made to be matched with diversified task requirements of the user end as much as possible based on the functions of satellite resource configuration, reason configuration, network configuration and the like of world fusion network real-time management; specifically, the step of establishing the time delay sensitive satellite flexible internet architecture based on the many-to-many mode of the LEO satellite and the user side includes:
and virtualizing computing resources and storage resources of an MEC server corresponding to each LEO satellite based on an SDN/NFV technology, and combining multiple delay related protocols by combining with a TSN (secure storage network) security protocol to establish the delay sensitive satellite elastic Internet architecture on the basis of the principle of minimizing delay optimization target management satellite resource configuration, route forwarding and network configuration.
S12, establishing a satellite elastic Internet resource scheduling model according to the time delay sensitive satellite elastic Internet architecture; the satellite flexible internet resource scheduling model can be understood as a time delay sensitive satellite flexible internet architecture based on the step S11, a scene that a plurality of LEO satellites cover multiple ground data nodes (user terminals) is considered on a data plane, and when the user terminals unload tasks, a scheduling model for determining a resource allocation strategy mainly comprises a communication model, a task model and a calculation model; the task model is established based on the consideration of task priority, and the calculation model can be established based on a preset task dividing principle and mainly comprises a local unloading task calculation model and an MEC unloading task calculation model; specifically, the step of establishing a satellite flexible internet resource scheduling model according to the delay-sensitive satellite flexible internet architecture includes:
acquiring the data transmission rate uploaded to the MEC server by each user side, and constructing the communication model according to the data transmission rate based on a many-to-many mode; the communication model is represented as:
in the formula ,
wherein ,
and
respectively representing a user side set and an MEC server set;
r i,j (t)、I i,j 、h i,j(t) and si,j Respectively representing the user terminal u in the time slot t i Offloading tasks to MEC Server b j Transmission delay, transmission energy consumption, transmission rate, inter-cell interference power, channel gain, and linear distance; w represents the channel bandwidth; sigma 2 Representing a noise power of the user equipment; z is a radical of i (t) indicates the ue u in the time slot t i Generated task Q i (t) size of data volume; c represents the speed of light; p is a radical of i (t) denotes the time gap ttuser u i A transmission power of the transmission signal;
it should be noted that, because the data size after the task is processed is very small, the communication model of the present invention does not always consider the energy consumption and the time delay during downloading; in addition, based on a multi-mode scene with multiple user terminals and multiple LEO nodes, under the support of the existing conditions, the switching time delay of satellite communication can be ignored, and because the user terminals are in an area with a longer distance from the LEO satellite, the distance from the user terminals to the MEC server can be approximately considered to be the same as the distance from the user terminals to the LEO satellite, and the following task model is established under the condition of only paying attention to the 'end-edge' model without considering the influence caused by cloud;
based on a load balancing principle, constructing the task model according to task calculated amount, task data amount and task priority; the task model is represented as:
Q i (t)＝{ω i (t),z i (t),pri i (t)}
wherein ,Qi (t) indicates the ue u in the time slot t i A generated task; omega i (t) represents task Q i (t) the required amount of computation, i.e. the CPU frequency required to complete the task; z is a radical of i (t) represents task Q i (t) data size; pri i (t) represents the task Q i (t) priority, and pri i (t)∈[1,2,…,PN]PN represents the number of priority levels;
dividing each client task into a local unloading task and an MEC unloading task, and respectively constructing a corresponding local unloading task calculation model and an MEC unloading task calculation model; the local offload task computation model may be understood as a computation model of processing delay and energy consumption when the local offload task is processed locally by the ue, and may be represented as:
wherein ,
and
respectively represent the user ends u i To local task Q i L Of (A)Time delay and corresponding energy consumption; f. of i L Representing a user side u i Local CPU frequency of (d); rho i A power coefficient representing the energy consumed per CPU cycle;
the MEC unloading task calculation model can be understood as an MEC server processing delay model established based on the consideration that a plurality of user terminals may unload tasks to the same MEC server and different priorities of the tasks unloaded by different user terminals exist, the task processing delay of the user terminals for unloading the tasks to the MEC server is considered, and the queuing delay of the tasks is considered, and the MEC server processing delay model is expressed as follows:
wherein ,
indicating the ue u in the time slot t i Offloading tasks to MEC Server b j The processing delay of (2);
representing MEC Server b j The CPU frequency of (1);
representing MEC Server b j Allocation of MEC offload task Q within time gap t i E Calculating the resource proportion;
indicating priority pri i MEC off-load task Q i E Average queuing delay of; it should be noted that the energy consumption of the processing task of the MEC server is not considered here;
as described above
The calculation model of (2) is related to the queuing model actually adopted, such as the task queuing model is assumed to be non-preemptiveA limited queuing model (M/M/N queue), wherein tasks with the same priority are processed by a first-come-first-serve principle; and the arrival rate compliance parameter of the task arrival queue with any priority in any time gap is assumed to be lambda i (t) poisson distribution, where the processing time of the MEC server obeys exponential distribution with a parameter μ (t), where the simultaneous storage space of multiple servers is sufficiently large; according to the setting of the task model, the priority has PN, and then the corresponding priority is pri i Task Q of i Average queuing delay of (1):
in the formula ,
the total arrival rate is:
and the constraint conditions include:
wherein ,Oj Representing offload to MEC Server b j A set of computing tasks.
In particular, of the above-mentioned different prioritiesTask Q i The calculation process of the average queuing delay at MEC server j can be understood as follows:
when the priority pri =1 of the task,
when the priority pri =2 of the task,
the different levels of task priority can be analogized to each other, when the priority pri of the task = PN,
and carrying out corresponding task queuing delay calculation.
S13, establishing a minimized time delay optimization model according to the satellite elastic internet resource scheduling model; the minimum time delay optimization model is a time delay optimization model which is established by taking the average processing time delay of all tasks generated in a preset time range corresponding to the minimum time gap set as an optimization target based on the dynamic property generated by the node tasks; specifically, the step of establishing a minimized time delay optimization model according to the satellite flexible internet resource scheduling model includes:
calculating the average time delay of task processing of each time gap within a preset time range according to the satellite elastic internet resource scheduling model; the average time delay of task processing in each time slot can be calculated according to the transmission energy consumption and MEC unloading task calculation model in the communication model, for example, the total time delay generated by all tasks in each time slot t is calculated as follows:
wherein l represents the total number of the user terminals for unloading the tasks to the MEC server in the time gap t;
based on the total time delay obtained above, i.e.Calculating the average time delay of task processing of each user terminal as
Averaging the average time delay of the task processing in each time interval to obtain the average time delay of the task processing in a preset time range, wherein the average time delay of the task processing in the preset time range corresponding to the time interval set T is as follows:
constructing a minimized time delay optimization model by taking the task processing average time delay within the minimized preset time range as an optimization target; the objective function of the minimized time delay optimization model is represented as:
in the formula ,
wherein d (t) represents the total time delay generated by all tasks in the time gap t;
and
respectively represent the user terminal u in the time slot t i Offloading tasks to MEC Server b j Transmission delay and processing delay of (1); l represents the total number of the user terminals; k represents a calculation resource proportion matrix distributed to different user sides by each MEC server;
the constraint condition of the minimized time delay optimization model is expressed as follows:
wherein ,Oj Representing offload to MEC Server b j A set of computing tasks;
indicating the ue u in the time slot t i Offloading tasks to MEC Server b j Transmission energy consumption of (2); t represents the total number of time slots; e i Representing a user side u i Upper limit of transmission energy consumption;
representing MEC Server b j Allocation to user terminal u in time slot t i Task Q of i Calculating the resource proportion;
s14, converting the minimized time delay optimization model into a corresponding Markov decision model; wherein a Markov Decision Process (MDP) model is understood to be a 4-element tuple
And S represents a state space, A represents an action space, R represents a reward function, and χ ∈ [0, 1']Representing a discount coefficient; specifically, the step of converting the minimized latency optimization model into a corresponding markov decision model includes:
constructing a state space of the Markov decision model according to the environment state of each time gap; the environmental state of each time slot is represented as:
wherein s (t) represents the environmental state of the time gap t; ω (t), z: (t) and pri (t) respectively represent the calculated amount, data amount and priority of all tasks in the time gap t;
representing the unloading strategy of the user side;
constructing an action space of the Markov decision model according to the Agent action of the Agent in each time interval; the Agent actions of the agents of each time slot are expressed as:
a(t)＝κ(t)
wherein a (t) represents the Agent action of the Agent in the time gap t; k (t) represents the proportion of computing resources distributed to different user sides by each MEC server under the time gap t;
constructing a reward function of the Markov decision model according to the reward of the Agent of each time interval; the Agent action reward is represented as:
wherein r (t) represents the Agent action reward of the Agent for time gap t;
it should be noted that, for the conventional MDP problem, the cumulative reward function is to be maximized, and the objective function of the minimized delay optimization model according to the present invention is to minimize the average delay, so that, on the basis of the objective function, when the constraint condition is satisfied, the inverse number of the delay is selected as the reward function, and when the constraint condition is not satisfied, the reward function is set to a minimum value.
S15, solving the Markov decision model to obtain a resource scheduling strategy; in order to ensure the high efficiency and accuracy of the solution, the DDRA algorithm is preferably established on the basis of a reinforcement learning algorithm TD3 framework, and a resource scheduling strategy is output through a trained neural network model; specifically, the step of solving the markov decision model to obtain the resource scheduling policy includes:
and constructing a time delay optimization DDRA algorithm based on a reinforcement learning algorithm TD3 framework, and solving the Markov decision model through the time delay optimization DDRA algorithm to obtain the resource scheduling strategy.
When solving the markov decision model based on the reinforcement learning algorithm TD3, 6 networks need to be trained as follows: the method comprises the following steps of an Actor network, a Target Actor network, a first Critic network, a first Target Critic network, a second Critic network and a second Target Critic network, wherein the training process comprises the following steps:
the parameter of the Actor network is phi, the input is the current training environment state s (t), and the output is the current Agent strategy pi (a (t) | s (t);, i.e. the action probability distribution of the current time slot; the corresponding parameter of the Target Actor network is phi ', the input is the current implementation environment state s ' (t), and the output is the current Agent strategy pi (a ' (t) | s ' (t); phi '), namely the action probability distribution of the current time slot;
the parameter of the first Critic network is θ 1 The input is the current Agent strategy pi (a (t) | s (t); phi) and the current training environment state s (t), and the output is the Q function of the action a (t) taken under the current state
Namely, the accumulated expected value of taking specific resource allocation actions under the current satellite communication environment state; the parameter of the corresponding first Target Critic network is θ' 1 The inputs are the current Agent policy pi (a ' (t) | s ' (t); phi ') and the implementation environment state s ' (t), and the output is a Q function of the action a ' (t) taken at the current state
Namely, the accumulated expected value of taking specific resource allocation actions under the current satellite communication environment state;
the parameter of the second Critic network is θ 2 The input is the current Agent strategy pi (a (t) | s (t); phi) and the current training environment state s (t), and the output is the Q function of the action a (t) taken under the current state
Namely, the accumulated expected value of taking specific resource allocation actions under the current satellite communication environment state; the parameter of the corresponding second Target Critic network is θ' 2 The inputs are the current Agent policy pi (a ' (t) | s ' (t); phi ') and the implementation environment state s ' (t), and the output is a Q function of the action a ' (t) taken at the current state
Namely, the accumulated expected value of taking specific resource allocation actions under the current satellite communication environment state;
the pseudo code of the process of training the neural network by adopting the time delay optimization DDRA algorithm constructed based on the reinforcement learning algorithm TD3 algorithm frame is shown in FIG. 4, and comprises the following steps:
1) Interacting with the environment by using an Actor network, and storing a result tuple { s (t), a (t), r (t), s (t + 1) } obtained by each step of interaction into a cache;
2) Randomly taking batches of tuples { s (t), a (t), r (t), s (t + 1) } from the cache to calculate a (t + 1) and Q functions; wherein, taking action a (t + 1) in the current state:
correspondingly, the Q function Q in the current state:
where σ' is the variance, c is the upper limit, and γ is the Q function update rate;
3) Calculating and updating parameter theta of first Critic network 1 And a parameter θ of a second Critic network 2 (ii) a The corresponding updating method is as follows:
parameters of the first critical network
Parameters of the second Critic network
4) Updating parameter phi of the Actor network, parameter phi ' of the Target Actor network and parameter theta ' of the first Target Critic network at preset step interval d ' 1 And parameter θ 'of a second Target Critic network' 2 (ii) a The corresponding parameter updating method comprises the following steps:
parameters of Actor network
Parameter φ ' ← τ φ + (1- τ) φ ' of Target Actor network '
Parameter theta 'of Target Critic1 network' 1 ←τθ 1 +(1-τ)θ′ 1
Parameter theta 'of Target Critic2 network' 2 ←τθ 2 +(1-τ)θ′ 2
wherein ,
eta is the Actor network update rate, and tau is the Target network update rate;
5) Repeating the above steps until the neural network converges.
In order to make a task unloading decision, the Actor network needs to train the first Critic network, the second Critic network, the Target Actor network, the first Target Critic network and the second Target Critic network according to the steps and through multiple times of training and combining experience in a cache; and finally, jointly determining a resource scheduling strategy by adopting the Target Actor network, the first Target Critic network and the second Target Critic network which are obtained through training.
According to the method, the resource scheduling problem among edge servers under satellite network edge computing is considered, the construction of a delay sensitive satellite elastic internet framework is completed based on an SDN/NFV technology and a TSN security protocol, a corresponding delay optimization model is obtained, a method for obtaining a resource scheduling strategy is solved by adopting a delay optimization algorithm constructed based on a deep reinforcement learning algorithm framework, comprehensive and effective optimization problem modeling is carried out based on a real application scene with limited resources and the influence of task queuing delay, not only can computing resources and storage resources of a satellite be fully utilized, performance influence caused by queuing delay is avoided, but also diversified requirements of a user side are met based on division of task priorities, service quality is improved, the delay optimization algorithm constructed based on the deep reinforcement learning algorithm in a targeted mode is also used, the computing efficiency of scheduling and distributing mass service resources is improved, further, intelligent and efficient satellite elastic internet resource scheduling with priority service, delay performance is improved, load balance is guaranteed, application defects of an existing satellite resource scheduling scheme are effectively overcome, and the method has strong generalization capability and high practical value.
In order to verify the application effect and performance of the DDRA scheme, the application also implements a related simulation comparison experiment, and the specific simulation process is as follows:
the simulation platform adopts Python 3.9, 3 LEO satellites with the height of 784km fly over a square area of 1200m × 1200m, 24 (modifiable) user terminals are randomly distributed on the ground by default, and each user terminal can only unload tasks to an MEC server of one LEO satellite or unload the tasks locally. Because the altitude is far greater than the ground area, the distance between each user terminal and the MEC server is approximately considered to be the altitude of the LEO satellite, and since the considered LEO satellites form a galaxy, the loss of channel switching and the influence caused by a communication window are negligible, that is, the user can communicate with the LEO satellite at the moment, and the channel gain can be obtained in advance through a perception technology. In addition, the transmission power of the user terminal is set to 23dBm, the channel bandwidth is 20MHz, and the channel model is selected as a free space fading channel model. For the task parameters, the calculation amount, the data amount, the priority and the possible energy consumption are mainly considered; during calculation, task unloading decision of the last time slot needs to be considered, and the amount of calculation resources which can be provided by each LEO satellite in each time slot is assumed to be a random value within a range; in the DDTO algorithm, through careful adjustment, all the neural networks are divided into 4 layers which are respectively 1 input layer, 2 hidden layers and 1 output layer, wherein hidden layer neurons of an actor network are respectively 2048 and 1024, and hidden layer neurons of a critic network are respectively 1024 and 512; in the training parameters of the model, the learning rate was 0.001 and the discount factor was 0.75. In addition, the remaining parameter settings are detailed in FIG. 5;
simulation A: based on the continuity of the motion space, selecting a SAC algorithm with performance similar to that of the DDRA algorithm, and analyzing and comparing the convergence speed of the DDRA algorithm and other reinforcement learning algorithms; the SAC algorithm is obtained by improving based on the DQN algorithm, the characteristic of preventing over-estimation is achieved on the calculation of the Critic network, and the difference is that the SAC introduces strategy entropy into a reward function, namely, the Agent of the intelligent Agent is encouraged to increase exploration while maximizing reward, so that the Agent is allowed to search a global optimal solution rather than a local optimal solution as much as possible, further more time is required for training, and the training process is not efficient;
simulation B: analyzing and comparing the optimized time delay results of the DDRA algorithm and other resource scheduling algorithms (a SAC algorithm and a local optimization algorithm are selected); the local optimization algorithm (marked as LOA) is based on a model without considering the influence of priority and queuing delay, so that the optimization problem can be proved to be a convex optimization problem, and the Lagrangian equation of the problem is solved through a KKT condition to obtain an optimal solution; meanwhile, in LOA, substituting the optimal solution into the objective function in the minimum time delay optimization model of the invention to obtain the average time delay;
simulation C: based on the fact that the mean value of the data volume is 3.5Mb, the calculated amount respectively obeys the scenes of uniform distribution and normal distribution, and the influence of the calculated amount of the research task on the average time ductility performance under the allocation of the calculation resources is researched;
and (3) simulation D: based on the fact that the mean value of the calculated amount is 5.5Gcycle, the data amount obeys the scenes of uniform distribution and normal distribution respectively, and the influence of the task data amount on the average time-delay performance under the allocation of the calculation resources is researched;
simulation E: the influence of the number of users on the average time-lapse performance under the allocation of computing resources is researched on the basis that the average values of the task computing capacity and the task data capacity are 5 Gcycles and 1Mb respectively and are subjected to the condition of normal distribution.
Based on the above simulation experiment, the following results were obtained:
the results of the analysis shown in FIGS. 6-7 were obtained from simulation A: FIG. 6 compares the impact of discount rate on DDRA algorithm convergence; in deep reinforcement learning, the discount factor is an important hyper-parameter in the markov decision process. By setting the discount factors to be 0.05, 0.75 and 0.95 in sequence, it is found that the influence is caused when the discount factor is too large or too small, and when the discount factors are 0.05 and 0.95, the average delay fluctuates greatly, which indicates that the system is not stable in these 2 cases, and especially when the discount factor is 0.05, a large peak appears around epicode = 50; when the discount factor is 0.75, the convergence of the system is stable and the convergence speed is moderate, so that the method is suitable for serving as a default hyper-parameter of a model when the task calculation amount and the task data amount are analyzed next. Based on this, it can be seen that the discount factor must be chosen reasonably, otherwise it will cause the system to be unstable or converge to an inappropriate value; fig. 7 compares the convergence of different reinforcement learning algorithms under SMTOM, and finds that the DDRA has converged to about 1.8s when epsode =100, while the SAC algorithm converges to almost 2s after epsode = 5000; it can be seen that the convergence speed of the DDRA algorithm is significantly faster than that of the SAC algorithm, and the final average delay obtained by convergence is also lower than that of the SAC algorithm, that is, the DDRA has higher calculation efficiency and better optimization result in the second scenario under SMTOM, which is very important in the satellite communication scenario with scarce resources.
Based on simulation B and simulation C, the analysis results shown in fig. 8-9 were obtained: 8-9 can conclude that the average delay of the system tends to increase when the task computation amount increases, wherein the rising trend of the SAC algorithm is less stable. Meanwhile, the DDRA algorithm and the local optimization algorithm LOA provided by the application have better performance, and the average time delay is lower than that of the SAC algorithm. In addition, when the calculated amount is less than 4.5Gcycle, the performance of the DDRA algorithm is poorer than that of the LOA, but when the calculated amount is greater than 4.5Gcycle, the performance of the DDRA algorithm is better than that of the LOA, and because the LOA does not consider the queuing delay problem, when the calculated amount of the tasks is larger, the processing time of each task is increased, so that the queuing delay is increased, and if the tasks are still locally optimized, the delay performance is reduced; based on the method, the advantages of the DDRA algorithm in the application scene with larger calculation amount are embodied, and the advantages of the SMTMO in the satellite communication scene are further embodied.
When the data amount in fig. 8 is subject to uniform distribution, the average delay of the SAC algorithm is 37.8% higher and the average delay of the DDRA algorithm is 2.7% lower than that of the LOA algorithm. It can be seen in fig. 9 that the difference between the average delay performance of each algorithm when the data amount obeys normal distribution and the average delay performance of each algorithm when the data amount obeys uniform distribution is not large, the average delay of each algorithm is relatively stable when the calculated amount is relatively low, and LOA is superior to dda in delay performance to SAC; after the calculated amount is increased, the SAC algorithm has a result with a large variance, and may not converge to an optimal solution under the limitation of training times, and the DDRA algorithm exceeds LOA when the delay performance is expressed, so that the superiority of the DDRA algorithm under SMTMO under high task calculated amount is reflected.
Based on simulation B and simulation D, the analysis results shown in FIGS. 10-11 were obtained: from fig. 10 to fig. 11, it can be concluded that the task data volume has little influence on the task offloading performance, and as the data volume increases, the average delay of various algorithms approximately shows a smaller increase trend, because the data volume only affects the transmission delay and the transmission delay usually occupies a smaller amount in the whole delay, and under LOA, the optimal value obtained through gradient has no relation with the task data volume. Among the three algorithms, the SAC algorithm has large jitter, and the LOA and the ddr a show a steady rising trend in comparison. In addition, when the calculated amount in fig. 10 is uniformly distributed, compared to LOA, the average delay of SAC algorithm is lower by 1.9%, and the average delay of random algorithm is lower by 11.5%; as can be seen from fig. 11, when the data amount is normally distributed, the average delay of the SAC algorithm is higher by 2.1% and the average delay of the random algorithm is lower by 11.2% compared to the LOA; therefore, under the higher task calculation amount, the SAC algorithm is similar to the LOA average delay performance, and the DDRA provided by the invention is about 11% better than the LOA average delay performance.
Based on simulation B and simulation E, the analysis results shown in fig. 12 were obtained: from fig. 12, it can be seen that the average delay under each algorithm gradually increases with the number of users, and particularly, the average delay increases with a somewhat exponential increase in the local optimization algorithm. Comparing the three algorithms, wherein the average delay performance of the DDRA algorithm is the best on average, and the average delay under different user numbers in the simulation is 6.5s; the SAC algorithm is slightly inferior to the DDRA algorithm in performance, and the average time delay under different user numbers in the simulation is 6.8s; the average time delay of the local optimization algorithm in the simulation is 13.1s under different user numbers, because the influence caused by queuing time delay is neglected, the influence is not large when the user number is small, but if the user number is increased, the average time delay is also increased rapidly, so that the time delay performance under the multi-user task number is greatly reduced, and the user experience is reduced. That is, in general, the average delay of the ddr a algorithm is 4.4% lower than that of the SAC algorithm and 50.4% lower than that of the local optimization algorithm for different numbers of ues.
Based on the results of the simulation experiments, the satellite elastic Internet architecture and the satellite elastic Internet resource scheduling model are established for the resource scheduling problem in the satellite elastic Internet scene, the objective of optimizing the average time delay under the energy limitation condition is provided on the basis, and then the technical scheme that the DDRA algorithm is provided by utilizing the reinforcement learning algorithm TD3 framework to solve the non-convex optimization problem can obviously reduce the average time delay of user side task unloading, and real effective satellite resource scheduling is realized.
It should be noted that, although the steps in the above-described flowcharts are shown in sequence as indicated by arrows, the steps are not necessarily executed in sequence as indicated by the arrows. The steps are not limited to being performed in the exact order illustrated and, unless explicitly stated herein, may be performed in other orders.
In one embodiment, as shown in fig. 13, there is provided a satellite-resilient internet resource scheduling system, the system comprising:
the architecture construction module 1 is used for establishing a time delay sensitive satellite elastic internet architecture based on a many-to-many mode of an LEO satellite and a user side; the LEO satellite corresponds to an MEC server;
the first modeling module 2 is used for establishing a satellite elastic internet resource scheduling model according to the time delay sensitive satellite elastic internet architecture;
the second modeling module 3 is used for establishing a minimized time delay optimization model according to the satellite elastic internet resource scheduling model;
a model conversion module 4, configured to convert the minimized delay optimization model into a corresponding markov decision model;
and the strategy solving module 5 is used for solving the Markov decision model to obtain a resource scheduling strategy.
For specific limitations of a satellite flexible internet resource scheduling system, reference may be made to the above limitations of a satellite flexible internet resource scheduling method, which are not described herein again. All or part of each module in the satellite flexible internet resource scheduling system can be realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
Fig. 14 shows an internal structure diagram of a computer device in one embodiment, and the computer device may be a terminal or a server. As shown in fig. 14, the computer apparatus includes a processor, a memory, a network interface, a display, and an input device, which are connected through a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program to run on the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a satellite-resilient internet resource scheduling method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on a shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.
It will be appreciated by those of ordinary skill in the art that the architecture shown in FIG. 14 is merely a block diagram of a portion of the architecture associated with aspects of the present application and is not intended to limit the computing devices to which aspects of the present application may be applied, and that a particular computing device may include more or fewer components than shown, or may combine certain components, or have a similar arrangement of components.
In one embodiment, a computer device is provided, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the above method when executing the computer program.
In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which computer program, when being executed by a processor, carries out the steps of the above-mentioned method.
In summary, the satellite flexible internet resource scheduling method and system provided by the embodiments of the present invention realize that a delay-sensitive satellite flexible internet architecture is established based on many-to-many modes of LEO satellites and user terminals, and after a satellite flexible internet resource scheduling model is established according to the delay-sensitive satellite flexible internet architecture, a minimized delay optimization model is established according to the satellite flexible internet resource scheduling model, the minimized delay optimization model is converted into a corresponding markov decision model, and the markov decision model is solved to obtain a technical scheme of a resource scheduling policy.
The embodiments in this specification are described in a progressive manner, and all the same or similar parts of the embodiments are directly referred to each other, and each embodiment is described with emphasis on differences from other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment. It should be noted that, the technical features of the embodiments may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several preferred embodiments of the present application, and the description thereof is specific and detailed, but not to be understood as limiting the scope of the invention. It should be noted that, for those skilled in the art, without departing from the technical principle of the present invention, several improvements and substitutions can be made, and these improvements and substitutions should also be regarded as the protection scope of the present application. Therefore, the protection scope of the present patent application shall be subject to the protection scope of the claims.
Claims (10)
1. A satellite flexible Internet resource scheduling method is characterized by comprising the following steps:
establishing a time delay sensitive satellite elastic Internet architecture based on a many-to-many mode of an LEO satellite and a user side; the LEO satellite corresponds to an MEC server;
establishing a satellite elastic internet resource scheduling model according to the time delay sensitive satellite elastic internet architecture;
establishing a minimum time delay optimization model according to the satellite elastic internet resource scheduling model;
converting the minimized time delay optimization model into a corresponding Markov decision model;
and solving the Markov decision model to obtain a resource scheduling strategy.
2. The method for scheduling satellite flexible internet resources according to claim 1, wherein the step of establishing the delay-sensitive satellite flexible internet architecture based on the many-to-many mode of the LEO satellite and the user side comprises:
based on SDN/NFV technology, computing resources and storage resources of an MEC server corresponding to each LEO satellite are virtualized, multiple delay related protocols are combined by combining with a TSN (secure transmission network) security protocol, and the delay sensitive satellite elastic Internet architecture is established according to the principle that the target management satellite resource configuration, route forwarding and network configuration are optimized in a minimized delay mode.
3. The satellite flexible internet resource scheduling method of claim 1, wherein the satellite flexible internet resource scheduling model comprises a communication model, a task model and a computation model; the computation model comprises a local unloading task computation model and an MEC unloading task computation model.
4. The method as claimed in claim 3, wherein the step of building a satellite flexible internet resource scheduling model according to the delay-sensitive satellite flexible internet architecture comprises:
acquiring the data transmission rate uploaded to the MEC server by each user side, and constructing the communication model according to the data transmission rate based on a many-to-many mode; the communication model is represented as:
in the formula ,
wherein ,
and
respectively representing a user side set and an MEC server set;
r i,j (t)、I i,j 、h i,j(t) and si,j Respectively representing the user terminal u in the time slot t i Offloading tasks to MEC Server b j Transmission delay, transmission energy consumption, transmission rate, inter-cell interference power, channel gain, and linear distance; w represents the channel bandwidth; sigma 2 Representing a noise power of the user equipment; z is a radical of i (t) indicates the ue u in the time slot t i Generated task Q i (t) data size; c represents the speed of light; p is a radical of i (t) denotes the time gap ttuser u i A transmission power of the transmission signal;
based on a load balancing principle, according to the task calculated amount, the task data amount and the task priority, constructing the task model; the task model is represented as:
Q i (t)＝{ω i (t),z i (t),pri i (t)}
wherein ,Qi (t) represents the user terminal u in the time slot t i A generated task; omega i (t) represents task Q i (t) the amount of computation required; z is a radical of formula i (t) represents task Q i (t) size of data volume; pri i (t) represents the task Q i (t) priority, and pri i (t)∈[1,2,…,PN]PN represents the number of priority levels;
dividing each client task into a local unloading task and an MEC unloading task, and respectively constructing a corresponding local unloading task calculation model and an MEC unloading task calculation model; the local offload task computation model is represented as:
wherein ,
and
respectively represent the user ends u i For local task Q i L Processing delay and corresponding energy consumption of (1); f. of i L Indicating user u i Local CPU frequency of (d); rho i A power coefficient representing the energy consumed per CPU cycle;
the MEC offload task computation model is represented as:
wherein ,
indicating the ue u in the time slot t i Offloading tasks to MEC Server b j The processing delay of (2);
representing MEC Server b j The CPU frequency of (1);
representing MEC Server b j Allocation of MEC offload task Q within time gap t i E Calculating the resource proportion;
indicates priority of pri i MEC off-load task Q i E Average queuing delay.
5. The satellite flexible internet resource scheduling method of claim 1, wherein the step of building a minimized delay optimization model according to the satellite flexible internet resource scheduling model comprises:
calculating the average time delay of task processing of each time gap within a preset time range according to the satellite elastic internet resource scheduling model;
averaging the task processing average time delay of each time interval to obtain the task processing average time delay within a preset time range;
constructing a minimized time delay optimization model by taking the task processing average time delay within the minimized preset time range as an optimization target; the objective function of the minimized time delay optimization model is represented as:
in the formula ,
wherein d (t) represents the total time delay generated by all tasks in the time gap t;
and
respectively representing the user terminal u in the time slot t i Offloading tasks to MEC Server b j Transmission delay and processing delay; l represents the total number of the user terminals(ii) a K represents a calculation resource proportion matrix distributed to different user sides by each MEC server;
the constraint condition of the minimized time delay optimization model is expressed as follows:
wherein ,Oj Representing offload to MEC Server b j A set of computing tasks;
indicating the ue u in the time slot t i Offloading tasks to MEC Server b j Transmission energy consumption of (2); t represents the total number of time slots; e i Representing a user side u i Upper limit of transmission energy consumption;
representing MEC Server b j To user terminal u in time slot t i Task Q of i The computing resource proportion of (1).
6. The satellite flexible internet resource scheduling method of claim 5, wherein said step of converting said minimized delay optimization model into a corresponding markov decision model comprises:
constructing a state space of the Markov decision model according to the environment state of each time gap; the environmental state of each time slot is represented as:
wherein s: (t) represents the environmental state of the time gap t; ω (t), z (t) and pri (t) respectively represent the calculation amount, data amount and priority of all tasks within the time slot t; gamma ray i j (t) indicates an offloading policy of the user side;
constructing an action space of the Markov decision model according to the Agent action of the Agent in each time interval; the Agent actions of the agents of each time slot are expressed as:
a(t)＝κ(t)
wherein a (t) represents the Agent action of the Agent for the time gap t; k (t) represents the proportion of computing resources distributed to different user sides by each MEC server under the time gap t;
constructing a reward function of the Markov decision model according to the reward of the Agent of each time interval; the Agent action reward is expressed as:
where r (t) represents the Agent action reward for time gap t.
7. The satellite flexible internet resource scheduling method of claim 1, wherein the step of solving the markov decision model to obtain the resource scheduling policy comprises:
and constructing a time delay optimization DDRA algorithm based on a reinforcement learning algorithm TD3 framework, and solving the Markov decision model through the time delay optimization DDRA algorithm to obtain the resource scheduling strategy.
8. A satellite-resilient internet resource scheduling system, the system comprising:
the architecture construction module is used for establishing a time delay sensitive satellite elastic Internet architecture based on a many-to-many mode of an LEO satellite and a user side; the LEO satellite corresponds to an MEC server;
the first modeling module is used for establishing a satellite elastic internet resource scheduling model according to the time delay sensitive satellite elastic internet architecture;
the second modeling module is used for establishing a minimized time delay optimization model according to the satellite elastic internet resource scheduling model;
the model conversion module is used for converting the minimized time delay optimization model into a corresponding Markov decision model;
and the strategy solving module is used for solving the Markov decision model to obtain a resource scheduling strategy.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method of any of claims 1 to 7 are implemented when the computer program is executed by the processor.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211125448.7A CN115514769B (en) | 2022-09-14 | 2022-09-14 | Satellite elastic Internet resource scheduling method, system, computer equipment and medium |
| ZA2023/05873A ZA202305873B (en) | 2022-09-14 | 2023-06-01 | Resource scheduling method and system for satellite elastic internet, computer device, and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211125448.7A CN115514769B (en) | 2022-09-14 | 2022-09-14 | Satellite elastic Internet resource scheduling method, system, computer equipment and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115514769A true CN115514769A (en) | 2022-12-23 |
| CN115514769B CN115514769B (en) | 2023-06-06 |
Family
ID=84504538
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211125448.7A Active CN115514769B (en) | 2022-09-14 | 2022-09-14 | Satellite elastic Internet resource scheduling method, system, computer equipment and medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN115514769B (en) |
| ZA (1) | ZA202305873B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116346206A (en) * | 2023-03-27 | 2023-06-27 | 广州爱浦路网络技术有限公司 | AI/ML model distributed transmission method, device and system based on low orbit satellite and 5GS |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111211830A (en) * | 2020-01-13 | 2020-05-29 | 电子科技大学 | Satellite uplink bandwidth resource allocation method based on Markov prediction |
| US20200249039A1 (en) * | 2019-02-05 | 2020-08-06 | International Business Machines Corporation | Planning vehicle computational unit migration based on mobility prediction |
| CN113346944A (en) * | 2021-06-28 | 2021-09-03 | 上海交通大学 | Time delay minimization calculation task unloading method and system in air-space-ground integrated network |
| CN114362810A (en) * | 2022-01-11 | 2022-04-15 | 重庆邮电大学 | Low-orbit satellite beam hopping optimization method based on migration depth reinforcement learning |
| CN114884949A (en) * | 2022-05-07 | 2022-08-09 | 重庆邮电大学 | Low-orbit satellite Internet of things task unloading method based on MADDPG algorithm |
| CN114900225A (en) * | 2022-04-24 | 2022-08-12 | 南京大学 | Low-orbit giant constellation-based civil aviation Internet service management and access resource allocation method |
| CN114928394A (en) * | 2022-04-06 | 2022-08-19 | 中国科学院上海微系统与信息技术研究所 | Low-orbit satellite edge computing resource allocation method with optimized energy consumption |
| CN114980039A (en) * | 2022-05-24 | 2022-08-30 | 中山大学 | Random task scheduling and resource allocation method in MEC system of D2D cooperative computing |
-
2022
- 2022-09-14 CN CN202211125448.7A patent/CN115514769B/en active Active
-
2023
- 2023-06-01 ZA ZA2023/05873A patent/ZA202305873B/en unknown
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200249039A1 (en) * | 2019-02-05 | 2020-08-06 | International Business Machines Corporation | Planning vehicle computational unit migration based on mobility prediction |
| CN111211830A (en) * | 2020-01-13 | 2020-05-29 | 电子科技大学 | Satellite uplink bandwidth resource allocation method based on Markov prediction |
| CN113346944A (en) * | 2021-06-28 | 2021-09-03 | 上海交通大学 | Time delay minimization calculation task unloading method and system in air-space-ground integrated network |
| CN114362810A (en) * | 2022-01-11 | 2022-04-15 | 重庆邮电大学 | Low-orbit satellite beam hopping optimization method based on migration depth reinforcement learning |
| CN114928394A (en) * | 2022-04-06 | 2022-08-19 | 中国科学院上海微系统与信息技术研究所 | Low-orbit satellite edge computing resource allocation method with optimized energy consumption |
| CN114900225A (en) * | 2022-04-24 | 2022-08-12 | 南京大学 | Low-orbit giant constellation-based civil aviation Internet service management and access resource allocation method |
| CN114884949A (en) * | 2022-05-07 | 2022-08-09 | 重庆邮电大学 | Low-orbit satellite Internet of things task unloading method based on MADDPG algorithm |
| CN114980039A (en) * | 2022-05-24 | 2022-08-30 | 中山大学 | Random task scheduling and resource allocation method in MEC system of D2D cooperative computing |
Non-Patent Citations (1)
| Title |
|---|
| 王世超;吴斌;汪勃;: "天基激光网络在线分布式接入调度算法", 激光与光电子学进展 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116346206A (en) * | 2023-03-27 | 2023-06-27 | 广州爱浦路网络技术有限公司 | AI/ML model distributed transmission method, device and system based on low orbit satellite and 5GS |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115514769B (en) | 2023-06-06 |
| ZA202305873B (en) | 2023-12-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109684075B (en) | Method for unloading computing tasks based on edge computing and cloud computing cooperation | |
| CN111953759B (en) | Collaborative computing task unloading and transferring method and device based on reinforcement learning | |
| Tang et al. | Multi-user computation offloading in mobile edge computing: A behavioral perspective | |
| Guo et al. | An index based task assignment policy for achieving optimal power-delay tradeoff in edge cloud systems | |
| Lakew et al. | Intelligent offloading and resource allocation in heterogeneous aerial access IoT networks | |
| CN109947574B (en) | Fog network-based vehicle big data calculation unloading method | |
| Shu et al. | Dependency-aware and latency-optimal computation offloading for multi-user edge computing networks | |
| CN110069341B (en) | Method for scheduling tasks with dependency relationship configured according to needs by combining functions in edge computing | |
| Yuan et al. | Online dispatching and fair scheduling of edge computing tasks: A learning-based approach | |
| EP4024212B1 (en) | Method for scheduling inference workloads on edge network resources | |
| Liu et al. | VirtualEdge: Multi-domain resource orchestration and virtualization in cellular edge computing | |
| Rahimi et al. | A novel approach to efficient resource allocation in load-balanced cellular networks using hierarchical DRL | |
| CN115022894A (en) | Task unloading and computing resource allocation method and system for low-earth-orbit satellite network | |
| Mishra et al. | A collaborative computation and offloading for compute-intensive and latency-sensitive dependency-aware tasks in dew-enabled vehicular fog computing: A federated deep Q-learning approach | |
| CN115514769A (en) | Satellite elastic internet resource scheduling method, system, computer equipment and medium | |
| Chiang et al. | Deep Q-learning-based dynamic network slicing and task offloading in edge network | |
| Hu et al. | Dynamic task offloading in MEC-enabled IoT networks: A hybrid DDPG-D3QN approach | |
| CN115484205A (en) | Deterministic network routing and queue scheduling method and device | |
| Gholami et al. | Roma: Resource orchestration for microservices-based 5g applications | |
| Yang et al. | Towards 6g joint haps-mec-cloud 3c resource allocation for delay-aware computation offloading | |
| CN115499875B (en) | Satellite internet task unloading method, system and readable storage medium | |
| Zhang et al. | Migration-driven resilient disaster response edge-cloud deployments | |
| Machida et al. | Availability modeling for drone image processing systems with adaptive offloading | |
| Masdari et al. | Energy-aware computation offloading in mobile edge computing using quantum-based arithmetic optimization algorithm | |
| Du et al. | A joint trajectory and computation offloading scheme for UAV-MEC networks via multi-agent deep reinforcement learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |