CN116204319A - Yun Bianduan collaborative unloading method and system based on SAC algorithm and task dependency relationship - Google Patents
Yun Bianduan collaborative unloading method and system based on SAC algorithm and task dependency relationship Download PDFInfo
- Publication number
- CN116204319A CN116204319A CN202310230282.3A CN202310230282A CN116204319A CN 116204319 A CN116204319 A CN 116204319A CN 202310230282 A CN202310230282 A CN 202310230282A CN 116204319 A CN116204319 A CN 116204319A
- Authority
- CN
- China
- Prior art keywords
- task
- execution
- unloading
- computing
- calculation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 49
- 238000004364 calculation method Methods 0.000 claims abstract description 66
- 238000005265 energy consumption Methods 0.000 claims abstract description 54
- 230000002787 reinforcement Effects 0.000 claims abstract description 13
- 230000006870 function Effects 0.000 claims description 26
- 230000009471 action Effects 0.000 claims description 21
- 238000010276 construction Methods 0.000 claims description 9
- 230000008569 process Effects 0.000 claims description 5
- 230000005540 biological transmission Effects 0.000 claims description 4
- 238000005457 optimization Methods 0.000 claims description 4
- 238000012545 processing Methods 0.000 claims description 4
- 238000011156 evaluation Methods 0.000 claims description 3
- 238000012886 linear function Methods 0.000 claims description 3
- 108091006146 Channels Proteins 0.000 description 12
- 239000011159 matrix material Substances 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000003795 chemical substances by application Substances 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 239000013598 vector Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5072—Grid computing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/48—Indexing scheme relating to G06F9/48
- G06F2209/484—Precedence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/5021—Priority
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/509—Offload
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mobile Radio Communication Systems (AREA)
Abstract
The invention discloses a Yun Bianduan collaborative unloading method and a Yun Bianduan collaborative unloading system based on SAC algorithm and task dependency relationship. According to the method, a complete framework for task unloading based on calculation task dependency relationship is constructed under a cloud end scene, a task completion time and calculation energy consumption relation model is built aiming at the unloading conditions of calculation tasks at different levels, then a flexible Actor-Critic deep reinforcement learning algorithm based on maximum entropy is constructed with the aim of minimizing task completion time and calculation energy consumption, more unloading schemes are encouraged to be explored under the constraint of time delay energy consumption, and an optimal calculation unloading strategy is selected. The method provided by the invention can select a proper unloading level according to the task priority and the task dependency relationship generated by the terminal equipment, and meets the requirements of high flexibility, low delay and low energy consumption of calculation and unloading.
Description
Technical Field
The invention belongs to the technical field of mobile communication technology and deep reinforcement learning, in particular relates to a Yun Bianduan collaborative unloading method based on deep reinforcement learning SAC and task dependency, and specifically relates to a Yun Bianduan collaborative unloading method and system based on SAC algorithm and task dependency.
Background
In recent years, with rapid development and iteration of 5G, internet of things and cloud computing technologies, the number of internet applications and intelligent devices has shown a blowout. Mobile cloud computing provides powerful computing storage services for user smart terminals, but faces bottleneck challenges in terms of bandwidth and data processing latency. In order to solve the problems of time delay and energy consumption caused by too far distance between a cloud computing center and a terminal device, mobile edge computing is becoming an important ring of 5G key technology as a delay Shen Jiagou of mobile cloud computing. The mobile edge computing allows a server to be deployed at a geographic position closer to a user, computing capability is provided at a position close to various different types of terminal equipment, meanwhile, the computing capability can be sunk to a brand-new intelligent base station, the requirement of large-batch near-field computing can be met, basic computing processing capability is met, and meanwhile, the transmission delay of large-scale data in a channel can be greatly reduced.
The computing offloading is one of key technologies of mobile edge computing, and can offload service computing tasks of terminal equipment in mobile edge computing to an edge server which is closer to the terminal equipment to run, and the edge server provides functions of service computing, content caching and the like, so that computing pressure of the terminal equipment can be greatly reduced, interaction frequency with a cloud computing centralized data center can be reduced, waiting time in message exchange can be remarkably reduced, and low-delay requirements of mobile application programs can be met.
The research of the current calculation unloading method is mainly focused on the situation of overall unloading of calculation tasks, and the unloading aims at optimizing time delay and system energy consumption. Incomplete offloading of computing tasks can provide greater flexibility in deploying and applying MEC systems than traditional monolithic offloading. In order to more fit with the actual problem of computing and unloading, the invention establishes a dependency relationship and a priority model of computing tasks, distributes subtasks to proper unloading terminals according to the dependency relationship of each computing task, integrally plans local resources, MEC resources and cloud service resources from bottom to top under a cloud side three-layer system architecture, establishes a partial unloading model of computing and unloading for the comprehensive optimization target by using system minimum time delay and energy consumption, and finally adopts a flexible Actor-Critic algorithm in deep reinforcement learning to explore an optimal unloading strategy.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention provides a Yun Bianduan collaborative unloading method and a Yun Bianduan collaborative unloading system based on a SAC algorithm and task dependency relationship, which are used for establishing a calculation task dependency and priority relationship, time delay and energy consumption relationship model in the unloading process on the basis of a cloud side three-layer architecture, and solving the problems of how to unload a calculation task to a place in a three-layer architecture and how to minimize the time delay and energy consumption of the system by adopting the SAC algorithm in deep reinforcement learning.
According to an aspect of the present invention, there is provided a Yun Bianduan collaborative offloading method based on a SAC algorithm and task dependency relationship, the method comprising:
s1: constructing a framework for unloading a computing task in a cloud side scene, and initializing parameters of a central cloud server, an edge server and mobile terminal equipment;
s2: modeling a task model, describing characteristic parameters of a task, and giving definition and a representation method of task dependency relationship;
s3: establishing task completion time and calculation energy consumption relation models under three unloading conditions aiming at the unloading conditions of the calculation tasks at different levels and the dependency relationship of the tasks;
s4: and (3) aiming at minimizing the completion time of the computing task and the computing energy consumption, constructing a SAC algorithm based on the maximum entropy, encouraging more exploration schemes on the premise of constraint of delay energy consumption, and selecting an optimal computing unloading strategy.
As a further technical scheme, the frame for unloading the constructed computing task sequentially comprises a central cloud layer, edge server layers and a local layer from top to bottom, wherein the central cloud layer comprises a central cloud server, each edge server of the edge server layers corresponds to one base station, and the local layer comprises a plurality of mobile terminal devices.
As a further technical solution, step S2 further includes:
it is assumed that corresponding to each mobile terminal device d is a computationally intensive and delay-sensitive application
P= {1, 2..p } means that the application is subdivided into a sequence of P sub-computing tasks, each sub-task being +.>
Representation, C d,n And C p,n Representing the total CPU cycle number required by the application and the CPU cycle number required by the subtask respectively, B d,n And B p,n Representing the total data size of the application and the data size of the subtasks, respectively,/->
And->
Representing the maximum completion times of the application and subtasks, respectively;
representing the precedence dependence relationship among subtasks by using a directed acyclic graph G= (V, E), wherein V is a task set, E is a directed edge set, represents the execution precedence order among tasks, and is a task T p,n Is denoted as Pre (T p,n )；
Assuming that the channel condition of the terminal equipment to the task final execution end is equal to the channel condition of the task final execution end to the terminal equipment, taking the precedence dependence relationship between tasks into consideration, and introducing task waiting time to represent the time spent by waiting for execution of a certain subtask in a queue.
As a further technical scheme, the three execution modes of the calculation task are local execution, edge server unloading execution and cloud server unloading execution respectively, and for each type of unloading mode, the execution time of the calculation task, the execution completion time and the CPU calculation energy consumption are considered.
As a further technical solution, when the computing task p is selected to be executed locally, the execution time, the execution completion time and the CPU computing energy consumption are respectively:
calculating the execution time of the task p in the local execution according to the CPU execution frequency of the local device and the CPU cycle number required by the task on the local device;
calculating the execution completion time of the local execution of the task p according to the task waiting time and the execution time;
according to the super linear function of CPU execution frequency, the CPU locally executing the calculation task p calculates the energy consumption.
As a further technical solution, when the computing task p is offloaded to the edge node for execution, the execution time, the execution completion time and the CPU computing energy consumption are respectively:
calculating the execution time spent of task p unloading through the edge node according to the uploading time of the task from the terminal device to the edge node, the execution time at the edge node and the time for feeding back the calculation result to the terminal device;
calculating the execution completion time of the task unloaded through the edge node according to the task waiting time and the execution time;
according to the energy consumed by the terminal equipment in the process of uploading the task to the edge node by the terminal equipment, the energy consumed by the data transmission in the channel and the energy consumed by the mobile terminal waiting for the calculation result, the calculation task p calculates the energy consumption through the CPU unloaded by the edge node.
As a further technical solution, when the computing task p is offloaded to the central cloud server for execution, the execution time, the execution completion time and the CPU computing energy consumption are respectively:
calculating the execution completion time of the task unloaded by the central cloud server according to the completion time of the task uploaded to the base station and the propagation delay of the core network;
the execution completion time of the task unloaded by the central cloud server is equal to the execution time spent by the task unloaded by the central cloud server;
and calculating CPU (central processing unit) calculation energy consumption of task unloading through the central cloud server according to the transmitting power of the task uploaded to the base station by the equipment, the uplink and downlink power of the core network between the base station and the cloud server, the completion time of the task uploading base station and the propagation delay of the core network.
As a further technical solution, step S4 further includes:
modeling the offload problem as a reinforcement learning triplet (S, a, R), where S is the state set of the entire system, a is the offload action set, and R is the reward function of the system;
constructing an objective function of a state action value network, a loss function of the action value network and a loss function of the state action value network;
using a re-parameterized updating strategy network, taking a deterministic function of a calculation state, strategy parameters and independent noise as a sample adopted by updating, and then using a Gaussian strategy for updating;
the policy evaluation and policy update are repeated until the state value function converges to the optimal offloading policy.
According to an aspect of the present disclosure, a Yun Bianduan collaborative offload system based on SAC algorithm and task dependency is provided, for implementing the method, where the system includes:
the unloading frame construction module is used for constructing a frame for unloading the computing task in a cloud side scene and initializing parameters of the central cloud server, the edge server and the mobile terminal equipment;
the task model construction module is used for modeling a task model, describing characteristic parameters of a task and giving definition and a representation method of task dependency relationship;
the hierarchical construction module is used for establishing task completion time and calculation energy consumption relation models under three unloading conditions aiming at the unloading conditions of the calculation tasks at different hierarchies and the dependency relationship of the tasks;
and the unloading strategy optimization module is used for constructing a SAC algorithm based on maximum entropy with the aim of minimizing the completion time of the computing task and the computing energy consumption, encouraging more exploration schemes on the premise of constraint of delay energy consumption, and selecting an optimal computing unloading strategy.
Compared with the prior art, the invention has the beneficial effects that:
1. according to the invention, a complete framework for unloading the computing task in the cloud side scene is constructed, and a proper unloading level can be selected according to the dependency relationship and the priority of the computing task, so that the requirements of high flexibility and low latency of unloading decisions are met.
2. Aiming at the unloading condition of the computing task at different levels, the invention establishes a task completion time and computing energy consumption relation model under different conditions aiming at the priority of a single task in a task sequence.
3. The method aims at minimizing the completion time of the calculation task and the calculation energy consumption, constructs a flexible Actor-Critic deep reinforcement learning algorithm based on maximum entropy, encourages to explore more unloading schemes on the premise of constraint of time delay energy consumption, and selects an optimal calculation unloading strategy.
Drawings
FIG. 1 is a schematic diagram of a three-layer architecture for cloud-edge co-uninstallation according to an embodiment of the present invention;
FIG. 2 is a flow chart of a method according to an embodiment of the invention;
fig. 3 is a framework diagram of a deep reinforcement learning SAC-based algorithm according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made more apparent and fully hereinafter with reference to the accompanying drawings, in which some, but not all embodiments of the invention are shown. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to fall within the scope of the invention.
The invention provides a Yun Bianduan collaborative unloading method based on SAC algorithm and task dependency relationship, which constructs a complete framework for unloading tasks based on calculation task dependency relationship under cloud end scene, establishes a task completion time and calculation energy consumption relation model aiming at the unloading condition of calculation tasks at different levels, then constructs a flexible Actor-Critic deep reinforcement learning algorithm based on maximum entropy with the aim of minimizing task completion time and calculation energy consumption, encourages exploration of more unloading schemes under the constraint of delay energy consumption, and selects an optimal calculation unloading strategy. The method provided by the invention can select a proper unloading level according to the task priority and the task dependency relationship generated by the terminal equipment, and meets the requirements of high flexibility, low delay and low energy consumption of calculation and unloading.
As shown in fig. 2, the method specifically includes the following steps:
s1, constructing a framework for unloading a computing task in a cloud end scene, and initializing parameters of a central cloud server, a mobile edge server and terminal equipment.
And S2, modeling a task model, describing characteristic parameters of the task, and giving definition and a representation method of task dependency.
And S3, establishing a task completion time and calculation energy consumption relation model under three unloading conditions aiming at the unloading conditions of the calculation tasks at different levels and the dependency relationship of the tasks.
And S4, constructing a flexible Actor-Critic depth reinforcement learning algorithm based on maximum entropy by taking the minimum calculation task completion time and calculation energy consumption as targets, encouraging more exploration schemes on the premise of constraint of delay energy consumption, and selecting an optimal calculation unloading strategy.
In S1, the specific steps of constructing a framework for unloading a computing task in a cloud side scene, and initializing parameters of a central cloud server, a mobile edge server and terminal equipment are as follows: as shown in fig. 1, in a three-layer network system of cloud-edge-end, the first layer from top to bottom is a central cloud layer, a central cloud server is assumed, and computing and communication resources of the central cloud server are infinite; the second layer is a mobile edge server layer, each edge server interfaces with one base station, the number of edge servers and base stations is E, and the set e= {1,2,3, & gt, E }; the third layer is a local layer comprising a number of mobile terminal devices, the set d= {1,2,..n }.
In S2, modeling the task model, describing characteristic parameters of the task, and giving any oneThe defining and representing method of the business dependence relationship comprises the following specific steps: it is assumed that corresponding to each terminal device d is a computationally intensive and delay-sensitive application
P= {1, 2..p } means that the application can be subdivided into a sequence of P sub-computing tasks, each sub-task being +.>
C d,n And C p,n Representing the total number of CPU cycles required by the application and the number of CPU cycles required by the subtask, respectively. B (B) d,n And B p,n Representing the total data size of the application and the data size of the subtasks, respectively,/->
And->
The maximum completion times, i.e. deadlines, of the application and subtasks, respectively, are indicated. As shown in fig. 2, the directed acyclic graph g= (V, E) is used to represent the precedence dependency relationship between subtasks, V is a task set, and E is a set of directed edges, which represents the execution precedence order between tasks. Task T p,n Is denoted as Pre (T p,n )。/>
The computing task has three execution modes, namely local execution, edge server unloading execution and cloud server unloading execution. It is assumed that the channel conditions of the terminal device to the task end execution end are equal to the channel conditions of the task end execution end to terminal device. Because of the fact that the dependency relationship between tasks needs to be considered, before a certain subtask of an application starts to execute, all the relay nodes of the application must complete execution. Introducing a task latency to represent the time it takes a certain subtask to wait in a queue for a given application n's subtask p, task latency TW p,n The earliest execution completion time of all the direct relay nodes Pre (p) denoted as task p. For each type of offloading mode we consider itThe execution of the task takes time, completion time and CPU computing power consumption.
In S3, a task completion time and a calculation energy consumption relation model under three unloading conditions are established for the unloading conditions of the calculation task at different levels and the dependence and priority relation of the task, and the specific steps include:
when the calculation task p is selected to be executed locally, it is assumed that the CPU execution frequency of the local device d is
Hz，C p,n The number of CPU cycles required to execute a task on the local device. The time taken for local execution is determined by the ratio of CPU cycles to CPU execution frequency, by +.>
The representation is:
CPU calculation energy consumption is a super linear function of CPU execution frequency, where k c And delta is an energy parameter related to the CPU chip structure, and delta is more than or equal to 2 and less than or equal to 3, and is used
The expression is as follows:
considering practical situations, there may be multiple tasks that need to be performed locally at the same point in time. Assuming a total of k tasks to be performed on device d, the task sequence
If a certain task can be operated after being scheduled by the CPU, the local start execution time of the task is 0, otherwise, the local start execution time of the task is the execution completion time of the last taskSince the job scheduling time of the CPU is extremely small, it is negligible.
Then the task
The local start execution time of (a) is as follows:
wherein the method comprises the steps of
Representing task->
Is->
Execution completion time of (c).
When the computing task is executed locally, comprehensively considering the computing task sequence of the task on the execution device and the application, and applying the task waiting time of the sub-computing task p of n
The following relationship is satisfied:
wherein the method comprises the steps of
Representing the local start execution time of a certain successor task i of task p,
The local execution time of the task i is represented, and the sum of the local execution time and the local execution time of the task i is the local task execution completion time of the task i. The above indicates that the task p has to be received before its immediate successor task has to be performedIs calculated by the computer. The execution mode of the direct relay node corresponds to three different unloading modes, and the task waiting time of the task p is the local task execution completion time of the relay node and the result return time of unloading the task to the edge end ∈>
And result return time after task offloading to central cloud execution +.>
The maximum of the three. When the task selection is performed locally, the latter two are 0; when a task is offloaded to an edge node, one three items are 0; when a task is offloaded to the central cloud execution, the first two items are 0.
Thus, local tasks
The execution completion time of (c) may be expressed as the sum of the task waiting time of the current task and the execution time spent:
when the calculation task p is offloaded to the edge node for execution, the uploading time of the task from the terminal device to the edge node, the execution time at the edge node and the time for feeding back the calculation result to the terminal device need to be considered.
When transmitting the computing task to the edge node, the user's upload delay
The ratio of the size of the task data volume to the data uploading speed in the channel is expressed as:
Wherein B is p,n Is the size of the data volume of the calculation task,
is the maximum rate at which data is transmitted in the channel. According to shannon's formula, the maximum upload rate of data in the channel is
Where B represents the uplink channel bandwidth,
representing the transmit power of the terminal device when uploading a computational task, v represents the channel gain of the mobile device to the server, n 0 Representing the power density of the noise in the channel.
Set CPU frequency of edge node as
The calculation task is performed at the edge node for the time required +.>
Can be expressed as:
when the task has calculated the result at the edge node, the time to feed back the calculation result to the terminal device is due to the fact that the data size of the result is usually small
Can be regarded as a constant Φ. The total execution time of task p offloaded by the edge node is therefore +.>
The method comprises the following steps:
task offloading to edge node execution energy consumption
Is divided into two parts: energy consumed by the terminal device during the uploading of the task to the edge node by the terminal device, respectively +.>
Energy consumed by the mobile terminal waiting for the calculation result +.>
Wherein->
And->
The power of the terminal equipment waiting for the calculation result to return in the uploading data and idle state respectively. Waiting time of mobile terminal->
I.e. the sum of the time the task is performed at the edge node and the time the result is fed back. The total energy consumption is expressed as follows:
considering the task dependency when a task is offloaded to an edge node, task p can only begin computing when task p is successfully uploaded to the edge server over the uplink and all direct successor tasks of the task complete the computation. Consider the case where there are u tasks on device d that need to be uploaded to an edge server.
The sequence of tasks is
Task->
The start upload time of (c) is as follows:
wherein the method comprises the steps of
Representing task pi in task upload sequence m The task is performed when the uploading of the direct relay node is completed
Uploading completion time->
To start the sum of the upload time and the upload time:
when the computing task is unloaded to the edge node for execution, comprehensively considering the computing task sequence of the task on the execution equipment and the application, and applying the task waiting time of the sub-computing task p of n
The following relationship is satisfied:
wherein the method comprises the steps of
The upload completion time of the upload of the successor task i of task p to the edge server is indicated.
Representing the time it takes for task i to execute locally,/->
Indicating the time for task i to return the result from the edge server,/->
The time for task i to return the result from the cloud server is represented.
Thus, any execution completion time of task p on the edge node is:
when the computing task p is offloaded to the central cloud server for execution, the computing task p needs to be communicated with the cloud server through an intermediate base station, the base station is connected with the cloud server through a core network, and the transmission time of data in the core network is ignored because of abundant bandwidth resources of the core network, and the cloud server is usually deployed at a position far away from the base station, so that the inherent propagation delay gamma of the core network needs to be considered. The computing resources and computing capacity of the central cloud server are rich, and the task can be executed after being uploaded to the cloud server, so that the queuing waiting time of the computing task on the cloud server is ignored. Consider the case where there are multiple tasks to upload to the base station, similar to the edge offload case, the completion time of the task upload to the base station needs to be considered
Thus task p offloads execution completion time to central cloud server +.>
As follows>
The energy consumption for completing the calculation task p terminal and the base station in the cloud server is as follows:
wherein the method comprises the steps of
And->
The device uploads the task to the transmit power of the base station and the uplink and downlink power of the core network between the base station and the cloud server, respectively.
Combining these three offloading modes, the total execution completion time for a given separable computing task p
The method comprises the following steps:
total energy consumption
The method comprises the following steps:
where α, β, λ are the proportions of task execution locally, offload to edge, offload to cloud execution, and α, β, λ e (0, 1), α+β+λ=1, respectively.
In S4, the specific steps of constructing the flexible Actor-Critic deep reinforcement learning algorithm based on the maximum entropy with the goal of minimizing the calculation task completion time and the calculation energy consumption are as follows:
the offloading problem is first modeled as a reinforcement-learned triplet (S, a, R), where S is the state set of the overall system, a is the offloading action set, and R is the rewarding function of the system.
The state space S is defined as S (T) = { G (T), T (T), S l (t),S e (t),S c (T) }, where G (T) and T (T) are the topology dependent order matrix and task data size matrix of the computation task at decision time T. S is S l (t)、S e (t) and S c (t) is a computing resource capability matrix of the local execution device, the edge node, the central cloud server, including the task sequence to be executed and the available CPU computing resources. Action set a (t) = { L (t), M (t), C (t), D n (t) }, where L (t) is the task sequence matrix to be executed locally, M (t) is the task sequence matrix to be offloaded to the edge node, and C (t) is the task sequence matrix to be offloaded to the cloud server. Considering the dependency relationship between tasks, D n (t) is a dispatch level matrix of different subtasks applying n at time t.
The reward function of the system is defined as:
where τ and v are parameters used to control the specific gravity of the delay and energy consumption in the global iteration process.
SAC adopts a reinforcement learning algorithm explored based on a maximum entropy random strategy, and as shown in fig. 3, SAC explores more unloading schemes by introducing entropy parameters H. The algorithm adopts 5 neural networks, namely two action value Q networks, two state value V networks and a strategy network. Establishing a strategy network pi(s) for outputting actions, wherein the parameters are theta; establishing a state value network V(s) which is responsible for outputting the value of the current state, wherein the parameter is ψ; establishing a target state value network V target (s') is responsible for outputting the value of the next state, the parameter being ψ t The method comprises the steps of carrying out a first treatment on the surface of the Establishing two action value networks Q 1 (s,a)、Q 2 (s, a) the value responsible for output action selection, the parameter is θ i ,i∈{1,2}。
The optimal policy expression for SAC is:
wherein ρ is π The representation is represented in the policyUnder the control of a little pi, the intelligent agent encounters a distribution obeyed by the 'state action pair'. Alpha is an entropy coefficient used to trade-off the relationship between policy entropy and expected prize value, r (s t ,a t ) Is a prize value. H (pi (|s) t ))＝-log(π(·|s t ) Is the entropy value of the policy pi. Gamma e (0, 1) is a discount factor, indicating the importance of future rewards.
The invention adopts SAC algorithm based on continuous values and uses a function approximator to express Q value function and strategy function
For avoiding the phenomenon that two Q networks are over fitted in the training process. The objective function of any state action value network is:
wherein,,
for action value network, ++>
For a state value network, D represents an experience playback pool.
The loss function calculation formula of the action value network is as follows:
the loss function calculation formula of the state value network is as follows:
wherein,,
is the motion after noise is addedThe average value was taken.
Representing the policy network after adding noise.
Updating a policy network by using re-parameterization, taking a deterministic function of a calculation state, a policy parameter and independent noise as a sample adopted by updating, and then using a flattened Gaussian policy, wherein a loss function of the policy network is as follows:
wherein the method comprises the steps of
D represents an empirical playback pool, N represents a positive too-distribution, and ζ represents a random action subject to selection of the positive too-distribution.
Finally, the policy evaluation and policy update are repeated continuously, and the state value function converges to the optimal unloading policy.
The invention also provides a Yun Bianduan collaborative unloading system based on the SAC algorithm and the task dependency relationship, which is used for realizing the method, and comprises the following steps:
the unloading frame construction module is used for constructing a frame for unloading the computing task in a cloud side scene and initializing parameters of the central cloud server, the edge server and the mobile terminal equipment;
the task model construction module is used for modeling a task model, describing characteristic parameters of a task and giving definition and a representation method of task dependency relationship;
the hierarchical construction module is used for establishing task completion time and calculation energy consumption relation models under three unloading conditions aiming at the unloading conditions of the calculation tasks at different hierarchies and the dependency relationship of the tasks;
and the unloading strategy optimization module is used for constructing a SAC algorithm based on maximum entropy with the aim of minimizing the completion time of the computing task and the computing energy consumption, encouraging more exploration schemes on the premise of constraint of delay energy consumption, and selecting an optimal computing unloading strategy.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced with equivalents; these modifications or substitutions do not depart from the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.
Claims (9)
1. Yun Bianduan collaborative unloading method based on SAC algorithm and task dependency relationship is characterized by comprising the following steps:
s1: constructing a framework for unloading a computing task in a cloud side scene, and initializing parameters of a central cloud server, an edge server and mobile terminal equipment;
s2: modeling a task model, describing characteristic parameters of a task, and giving definition and a representation method of task dependency relationship;
s3: establishing task completion time and calculation energy consumption relation models under three unloading conditions aiming at the unloading conditions of the calculation tasks at different levels and the dependency relationship of the tasks;
s4: and (3) aiming at minimizing the completion time of the computing task and the computing energy consumption, constructing a SAC algorithm based on the maximum entropy, encouraging more exploration schemes on the premise of constraint of delay energy consumption, and selecting an optimal computing unloading strategy.
2. The Yun Bianduan collaborative offloading method based on the SAC algorithm and the task dependency relationship according to claim 1, wherein the constructed framework for computing task offloading sequentially comprises a central cloud layer, edge server layers and a local layer from top to bottom, wherein the central cloud layer comprises a central cloud server, each edge server of the edge server layers corresponds to a base station, and the local layer comprises a plurality of mobile terminal devices.
3. The SAC algorithm and task dependency based Yun Bianduan collaborative offloading method of claim 1, wherein step S2 further comprises:
it is assumed that corresponding to each mobile terminal device d is a computationally intensive and delay-sensitive application
P= {1, 2..p } means that the application is subdivided into a sequence of P sub-computing tasks, each sub-task being +.>
Representation, C d,n And C p,n Representing the total CPU cycle number required by the application and the CPU cycle number required by the subtask respectively, B d,n And B p,n Representing the total data size of the application and the data size of the subtasks, respectively,/->
And->
Representing the maximum completion times of the application and subtasks, respectively;
representing subtasks with directed acyclic graph g= (V, E)The dependency relationship among the tasks is V, E is a set of directed edges, and represents the execution sequence among the tasks, and T p,n Is denoted as Pre (T p,n )；
Assuming that the channel condition of the terminal equipment to the task final execution end is equal to the channel condition of the task final execution end to the terminal equipment, taking the precedence dependence relationship between tasks into consideration, and introducing task waiting time to represent the time spent by waiting for execution of a certain subtask in a queue.
4. The Yun Bianduan collaborative offload method based on the SAC algorithm and the task dependency relationship according to claim 1, wherein the computing task has three execution modes, namely local execution, edge server offload execution and cloud server offload execution, and for each type of offload mode, the execution time, execution completion time and CPU computing energy consumption of the computing task are considered.
5. The SAC algorithm and task dependency relationship based Yun Bianduan co-offload method according to claim 4, wherein when the computing task p is selected to be executed locally, the execution time, the execution completion time and the CPU computing power consumption are respectively:
calculating the execution time of the task p in the local execution according to the CPU execution frequency of the local device and the CPU cycle number required by the task on the local device;
calculating the execution completion time of the local execution of the task p according to the task waiting time and the execution time;
according to the super linear function of CPU execution frequency, the CPU locally executing the calculation task p calculates the energy consumption.
6. The SAC algorithm and task dependency relationship based Yun Bianduan co-offload method according to claim 4, wherein when the computing task p is offloaded to the edge node for execution, the execution time, the execution completion time and the CPU computing power consumption are respectively:
calculating the execution time spent of task p unloading through the edge node according to the uploading time of the task from the terminal device to the edge node, the execution time at the edge node and the time for feeding back the calculation result to the terminal device;
calculating the execution completion time of the task unloaded through the edge node according to the task waiting time and the execution time;
according to the energy consumed by the terminal equipment in the process of uploading the task to the edge node by the terminal equipment, the energy consumed by the data transmission in the channel and the energy consumed by the mobile terminal waiting for the calculation result, the calculation task p calculates the energy consumption through the CPU unloaded by the edge node.
7. The SAC algorithm and task dependency relationship-based Yun Bianduan collaborative offload method according to claim 4, wherein when the computing task p is offloaded to the central cloud server for execution, the execution time, the execution completion time and the CPU computing energy consumption are respectively:
calculating the execution completion time of the task unloaded by the central cloud server according to the completion time of the task uploaded to the base station and the propagation delay of the core network;
the execution completion time of the task unloaded by the central cloud server is equal to the execution time spent by the task unloaded by the central cloud server;
and calculating CPU (central processing unit) calculation energy consumption of task unloading through the central cloud server according to the transmitting power of the task uploaded to the base station by the equipment, the uplink and downlink power of the core network between the base station and the cloud server, the completion time of the task uploading base station and the propagation delay of the core network.
8. The SAC algorithm and task dependency based Yun Bianduan collaborative offloading method of claim 1, wherein step S4 further comprises:
modeling the offload problem as a reinforcement learning triplet (S, a, R), where S is the state set of the entire system, a is the offload action set, and R is the reward function of the system;
constructing an objective function of a state action value network, a loss function of the action value network and a loss function of the state action value network;
using a re-parameterized updating strategy network, taking a deterministic function of a calculation state, strategy parameters and independent noise as a sample adopted by updating, and then using a Gaussian strategy for updating;
the policy evaluation and policy update are repeated until the state value function converges to the optimal offloading policy.
9. Yun Bianduan collaborative offload system based on SAC algorithm and task dependency, for implementing the method according to any one of claims 1-8, characterized in that it comprises:
the unloading frame construction module is used for constructing a frame for unloading the computing task in a cloud side scene and initializing parameters of the central cloud server, the edge server and the mobile terminal equipment;
the task model construction module is used for modeling a task model, describing characteristic parameters of a task and giving definition and a representation method of task dependency relationship;
the hierarchical construction module is used for establishing task completion time and calculation energy consumption relation models under three unloading conditions aiming at the unloading conditions of the calculation tasks at different hierarchies and the dependency relationship of the tasks;
and the unloading strategy optimization module is used for constructing a SAC algorithm based on maximum entropy with the aim of minimizing the completion time of the computing task and the computing energy consumption, encouraging more exploration schemes on the premise of constraint of delay energy consumption, and selecting an optimal computing unloading strategy.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310230282.3A CN116204319A (en) | 2023-03-10 | 2023-03-10 | Yun Bianduan collaborative unloading method and system based on SAC algorithm and task dependency relationship |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310230282.3A CN116204319A (en) | 2023-03-10 | 2023-03-10 | Yun Bianduan collaborative unloading method and system based on SAC algorithm and task dependency relationship |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116204319A true CN116204319A (en) | 2023-06-02 |
Family
ID=86509362
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310230282.3A Pending CN116204319A (en) | 2023-03-10 | 2023-03-10 | Yun Bianduan collaborative unloading method and system based on SAC algorithm and task dependency relationship |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116204319A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116757095A (en) * | 2023-08-14 | 2023-09-15 | 国网浙江省电力有限公司宁波供电公司 | Electric power system operation method, device and medium based on cloud edge end cooperation |
-
2023
- 2023-03-10 CN CN202310230282.3A patent/CN116204319A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116757095A (en) * | 2023-08-14 | 2023-09-15 | 国网浙江省电力有限公司宁波供电公司 | Electric power system operation method, device and medium based on cloud edge end cooperation |
| CN116757095B (en) * | 2023-08-14 | 2023-11-07 | 国网浙江省电力有限公司宁波供电公司 | Electric power system operation method, device and medium based on cloud edge end cooperation |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113950066B (en) | Single server part calculation unloading method, system and equipment under mobile edge environment | |
| Chen et al. | Energy-efficient task offloading and resource allocation via deep reinforcement learning for augmented reality in mobile edge networks | |
| Nath et al. | Multi-user multi-channel computation offloading and resource allocation for mobile edge computing | |
| CN113543176A (en) | Unloading decision method of mobile edge computing system based on assistance of intelligent reflecting surface | |
| CN113626104B (en) | Multi-objective optimization unloading strategy based on deep reinforcement learning under edge cloud architecture | |
| CN111726854B (en) | Method for reducing calculation unloading energy consumption of Internet of things | |
| WO2024159708A1 (en) | Digital twinning-based end-edge collaborative scheduling method for heterogeneous task and resource | |
| CN113946423B (en) | Multi-task edge computing, scheduling and optimizing method based on graph attention network | |
| CN114205353B (en) | Calculation unloading method based on hybrid action space reinforcement learning algorithm | |
| CN113573363B (en) | MEC calculation unloading and resource allocation method based on deep reinforcement learning | |
| CN116489708B (en) | Meta universe oriented cloud edge end collaborative mobile edge computing task unloading method | |
| CN114661466A (en) | Task unloading method for intelligent workflow application in edge computing environment | |
| CN116009990B (en) | Cloud edge collaborative element reinforcement learning computing unloading method based on wide attention mechanism | |
| CN118250750B (en) | Satellite edge computing task unloading and resource allocation method based on deep reinforcement learning | |
| CN115190033A (en) | Cloud edge fusion network task unloading method based on reinforcement learning | |
| CN116204319A (en) | Yun Bianduan collaborative unloading method and system based on SAC algorithm and task dependency relationship | |
| CN118467127A (en) | Multi-agent cooperation-based mobile edge computing task scheduling and unloading method | |
| CN116185523A (en) | Task unloading and deployment method | |
| CN114980160A (en) | Unmanned aerial vehicle-assisted terahertz communication network joint optimization method and device | |
| CN114698125A (en) | Method, device and system for optimizing computation offload of mobile edge computing network | |
| CN114968402A (en) | Edge calculation task processing method and device and electronic equipment | |
| CN117880122A (en) | Task unloading and resource allocation combined optimization method based on MADFPG | |
| CN117632488A (en) | Multi-user fine-granularity task unloading scheduling method and device based on cloud edge end cooperation | |
| CN117749796A (en) | Cloud edge computing power network system calculation unloading method and system | |
| CN116455903A (en) | Method for optimizing dependency task unloading in Internet of vehicles by deep reinforcement learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |