CN112312299A - Service unloading method, device and system - Google Patents

Service unloading method, device and system Download PDF

Info

Publication number
CN112312299A
CN112312299A CN202011161252.4A CN202011161252A CN112312299A CN 112312299 A CN112312299 A CN 112312299A CN 202011161252 A CN202011161252 A CN 202011161252A CN 112312299 A CN112312299 A CN 112312299A
Authority
CN
China
Prior art keywords
service
network
data
parameter
representing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202011161252.4A
Other languages
Chinese (zh)
Inventor
邹彪
杜伟
王和平
孟小前
武艺
郑思嘉
张嘉琳
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sgcc General Aviation Co ltd
State Grid Corp of China SGCC
Original Assignee
Sgcc General Aviation Co ltd
State Grid Corp of China SGCC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sgcc General Aviation Co ltd, State Grid Corp of China SGCC filed Critical Sgcc General Aviation Co ltd
Priority to CN202011161252.4A priority Critical patent/CN112312299A/en
Publication of CN112312299A publication Critical patent/CN112312299A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

The invention provides a service unloading method, a device and a system, which relate to the technical field of communication, and the method comprises the following steps: acquiring service data to be unloaded, resource data, an initialized policy network and an initialized evaluation network; calculating a time delay parameter of the service data to be unloaded according to the resource data; generating a first service unloading result according to the service data to be unloaded and the initialized policy network; generating a first evaluation result of a first service unloading result according to the time delay parameter and the initialized evaluation network; updating a first gradient parameter of the policy network and a second gradient parameter of the evaluation network by using a depth deterministic policy gradient algorithm according to the first service unloading result and the first evaluation result; and generating a target service unloading result of the service data to be unloaded in the resource data according to the updated strategy network and the updated evaluation network. The invention fully utilizes the resource of fog calculation and provides low-delay response service for the service of the Internet of things.

Description

Service unloading method, device and system
Technical Field
The present invention relates to the field of communications technologies, and in particular, to a method, an apparatus, and a system for offloading services.
Background
Under the background of rapid development of a 5G network, the application range of the Internet of things is wider and wider, and more new services are provided for work and life of people. However, part of the internet of things services are sensitive to the calculation delay, and higher requirements are provided for the performance of the internet of things. The fog computing has the advantage of low time delay, and is gradually used as a supplement of a cloud computing technology, so that the fog computing is more and more widely applied to the Internet of things. Under the background, how to fully utilize the resource of the fog calculation and provide a lower delay response service for the service of the internet of things becomes an important research content.
Disclosure of Invention
The invention provides a service unloading method, a device and a system, which can solve the problem of large delay of network service unloading in a fog computing environment.
In a first aspect, an embodiment of the present invention provides a service offloading method, where the method includes: acquiring service data to be unloaded, resource data, an initialized policy network and an initialized evaluation network; calculating a time delay parameter of the service data to be unloaded according to the resource data; generating a first service unloading result according to the service data to be unloaded and the initialized policy network; generating a first evaluation result of the first service unloading result according to the time delay parameter and the initialized evaluation network; updating a first gradient parameter of the policy network and a second gradient parameter of the evaluation network by using a depth deterministic policy gradient algorithm according to the first service unloading result and the first evaluation result; and generating a target service unloading result of the service data to be unloaded according to the updated strategy network and the updated evaluation network.
In a second aspect, an embodiment of the present invention further provides a service offloading device, where the device includes: the acquisition module is used for acquiring service data to be unloaded, resource data, an initialized policy network and an initialized evaluation network; the time delay module is used for calculating a time delay parameter of the service data to be unloaded according to the resource data; the first generation module is used for generating a first service unloading result according to the service data to be unloaded and the initialized policy network; a second generation module, configured to generate a first evaluation result of the first service offload result according to the delay parameter and the initialized evaluation network; the updating module is used for updating a first gradient parameter of the strategy network and a second gradient parameter of the evaluation network by using a depth certainty strategy gradient algorithm according to the first service unloading result and the first evaluation result; and the result module is used for generating a target service unloading result of the service data to be unloaded according to the updated strategy network and the updated evaluation network.
In a third aspect, an embodiment of the present invention further provides a service offloading system, where the system includes: the system comprises a cloud service main control module, a regional SDN control module, an edge node module, a data module, an equipment module and any one of the service unloading devices; the service unloading device is in communication connection with the cloud service main control module, the regional SDN control module, the node module, the data module and the equipment module respectively.
In a fourth aspect, an embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the service uninstalling method when executing the computer program.
In a fifth aspect, an embodiment of the present invention further provides a computer-readable storage medium, where a computer program for executing the service uninstalling method is stored in the computer-readable storage medium.
The embodiment of the invention has the following beneficial effects: the embodiment of the invention provides a service unloading scheme, which comprises the steps of obtaining service data to be unloaded, resource data, an initialized policy network and an initialized evaluation network; calculating a time delay parameter of the service data to be unloaded according to the resource data; generating a first service unloading result according to the service data to be unloaded and the initialized policy network; generating a first evaluation result of the first service unloading result according to the time delay parameter and the initialized evaluation network; updating a first gradient parameter of the policy network and a second gradient parameter of the evaluation network by using a depth deterministic policy gradient algorithm according to the first service unloading result and the first evaluation result; and generating a target service unloading result of the service data to be unloaded according to the updated strategy network and the updated evaluation network. According to the method and the device, a time delay parameter is obtained through calculation according to resource data, the time delay parameter is used for limiting time delay of different types of resource data, the time delay parameter is used for obtaining a first service unloading result and a first evaluation result, then, a strategy network and an evaluation network are updated through a depth certainty strategy gradient algorithm, a target service unloading result is obtained through the updated network, resources of fog calculation are fully utilized, and low-time-delay response service is provided for the service of the Internet of things.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
In order to make the aforementioned and other objects, features and advantages of the present invention comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a flowchart of a service offloading method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a service offload system architecture according to an embodiment of the present invention;
fig. 3 is a schematic diagram illustrating steps of a service offloading method according to an embodiment of the present invention;
FIG. 4 is a diagram illustrating the effect of the maximum distance between adjacent cells on the average data transmission rate according to an embodiment of the present invention;
FIG. 5 is a schematic diagram illustrating the effect of the number of devices connected to each compute node on the average data transmission rate according to an embodiment of the present invention;
fig. 6 is a block diagram of a service offloading device according to an embodiment of the present invention;
FIG. 7 is a block diagram of another service offload device according to an embodiment of the present invention;
fig. 8 is a block diagram of a computer device according to an embodiment of the present invention.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
At present, in the first existing scheme, the energy consumption of the fog computing nodes is solved, a distributed computing service unloading algorithm is provided by adopting a game theory and an intelligent optimization theory, and the energy consumption of the fog computing nodes is well reduced. In the second conventional scheme, a service unloading algorithm for reducing time delay is provided by aiming at solving the time delay of a plurality of computing tasks and adopting a cooperation mode among the computing tasks, so that the time delay of task execution is well reduced, and the energy consumption of edge nodes is reduced. In the third existing scheme, the energy consumption of the fog computing node is reduced, the task execution delay is delayed, and the task node payment cost is reduced as a comprehensive optimization target, a deep reinforcement learning algorithm is adopted, an optimized service unloading algorithm is provided, and a good effect is achieved in an experiment. In the prior art, the problem is modeled and solved by using an intelligent optimization algorithm with the aim of solving the high-definition video signal processing time delay as an objective and with the aims of minimizing energy consumption and optimizing low time delay as an optimization objective. In the prior art, the network environment density and complexity are taken as research backgrounds, a combined allocation algorithm of service unloading and resource scheduling in the network intensive environment is provided, the resource allocation problem in the network intensive environment is well solved, and the task calculation delay is reduced. In the prior art, the sixth scheme takes reducing the competition duration of user resources as an entry point, provides a service scheduling algorithm, optimizes a user resource allocation mechanism, and reduces the calculation delay of user tasks.
Known from the existing scheme, the existing scheme has achieved better results in the aspects of time delay, energy consumption and the like of service unloading. However, in the existing scheme, in terms of the fog computing environment, factors in aspects of service types, types of edge nodes and the like are less considered, and the service unloading problem in the whole network environment is not considered, particularly, factors such as large network environment range and dynamic change are not considered. Therefore, a new service offloading algorithm is urgently needed, and an algorithm for minimizing service offloading delay under the scenes of comprehensively considering network dynamic change, network scale, service type and the like is available.
Based on the above, the service offloading method, the device and the system provided by the embodiment of the invention provide a layered internet of things architecture in a large-scale fog computing environment, and according to the characteristics of tasks, four service offloading modes are provided, so that a service offloading algorithm based on a depth deterministic policy gradient is designed with the aim of minimizing service offloading delay.
To facilitate understanding of the present embodiment, a detailed description is first given of a service offloading method disclosed in the present embodiment.
An embodiment of the present invention provides a service offloading method, referring to a flow chart of a service offloading method shown in fig. 1, the method including the following steps:
step S102, acquiring service data to be unloaded, resource data, an initialized policy network and an initialized evaluation network.
In the embodiment of the invention, the service data to be unloaded is task data acquired according to the sensors of the equipment layer, and the sensors of the equipment can comprise intelligent factory sensors, intelligent market sensors, intelligent home sensors and the like. The resource data is device information in an internet of things service unloading management architecture, and can be used for determining information such as computing resources, transmission resources and idle resources. The policy network is used for generating unloading results of the service data to be unloaded on each Internet of things service unloading device, and the evaluation network is used for evaluating the unloading results. The initialized policy network and the initialized evaluation network are obtained by initializing the policy network and the evaluation network by using the initial parameters.
And step S104, calculating a time delay parameter of the service data to be unloaded according to the resource data.
In the embodiment of the present invention, the delay parameter is used to determine the execution duration of the service data to be offloaded on the device corresponding to the resource data. According to different types of resource data such as service types, types of edge nodes, network environments and the like, the time delay parameter can be calculated, so that time delay limitation on different types of resource data and different unloading modes is realized by utilizing the time delay parameter.
And step S106, generating a first service unloading result according to the service data to be unloaded and the initialized policy network.
In the embodiment of the invention, the task data to be processed can be determined according to the service data to be unloaded, and the first service unloading result can be obtained by combining the task data to be processed and the initialized policy network, wherein the first service unloading result is a service unloading scheme obtained preliminarily.
Step S108, generating a first evaluation result of the first service unloading result according to the time delay parameter and the initialized evaluation network.
In the embodiment of the present invention, the first evaluation result is used to evaluate the first service offloading result, and the first evaluation result obtained based on the delay parameter is used to evaluate the first service offloading result, so as to screen out a service offloading scheme capable of providing a low-delay response.
Step S110, according to the first service offloading result and the first evaluation result, updating a first Gradient parameter of the Policy network and a second Gradient parameter of the evaluation network by using a Deep Deterministic Policy Gradient algorithm (DDPG).
In the embodiment of the invention, a function required by a calculation process of a depth certainty strategy gradient algorithm is obtained according to a first service unloading result and a first evaluation result, and a first gradient parameter of a strategy network and a second gradient parameter of an evaluation network are updated by using the depth certainty strategy gradient algorithm.
And step S112, generating a target service unloading result of the service data to be unloaded in the resource data according to the updated policy network and the updated evaluation network.
In the embodiment of the invention, after the first gradient parameter of the policy network and the second gradient parameter of the evaluation network are updated, the updated policy network and the updated evaluation network are obtained, the updated policy network and the updated evaluation network are used for obtaining the updated service unloading result and the updated evaluation result, the updated service unloading result is screened according to the updated evaluation result to obtain the target service unloading result, and the target service unloading result is the final unloading result of the service data to be unloaded.
The embodiment of the invention provides a service unloading scheme, which comprises the steps of obtaining service data to be unloaded, resource data, an initialized policy network and an initialized evaluation network; calculating a time delay parameter of the service data to be unloaded according to the resource data; generating a first service unloading result according to the service data to be unloaded and the initialized policy network; generating a first evaluation result of the first service unloading result according to the time delay parameter and the initialized evaluation network; updating a first gradient parameter of the policy network and a second gradient parameter of the evaluation network by using a depth deterministic policy gradient algorithm according to the first service unloading result and the first evaluation result; and generating a target service unloading result of the service data to be unloaded according to the updated strategy network and the updated evaluation network. According to the method and the device, a time delay parameter is obtained through calculation according to resource data, the time delay parameter is used for limiting time delay of different types of resource data, the time delay parameter is used for obtaining a first service unloading result and a first evaluation result, then, a strategy network and an evaluation network are updated through a depth certainty strategy gradient algorithm, a target service unloading result is obtained through the updated network, resources of fog calculation are fully utilized, and low-time-delay response service is provided for the service of the Internet of things.
In order to improve the data processing efficiency, before acquiring the initialized policy network and the initialized evaluation network, the following steps may be further performed:
randomly generating an initial first gradient parameter and an initial second gradient parameter; initializing the strategy network by utilizing the initial first gradient parameter to obtain an initialized strategy network; and initializing the evaluation network by using the initial second gradient parameter to obtain the initialized evaluation network.
In the embodiment of the invention, the first gradient parameter is an initial parameter in the policy network, the second gradient parameter is an initial parameter in the evaluation network, and the policy network and the evaluation network are initialized by respectively using the first gradient parameter and the initial second gradient parameter to obtain the initialized policy network and the initialized evaluation network.
When the DDPG algorithm is used to solve the optimized service offloading policy, an Actor (policy) -Critic (evaluation) architecture is adopted. The Actor refers to a policy network and comprises an Actor _ M network and an Actor _ T network, wherein the main function of the Actor _ T network is to generate a training data set, and the main function of the Actor _ M network is to train parameters of an optimization network. Critic refers to an evaluation network, and comprises a Critic _ M network and a Critic _ T network, wherein the Critic _ T network mainly functions to generate a training data set, and the Critic _ M network mainly functions to train optimization network parameters.
In order to effectively reduce the task delay of service unloading and research the dynamic property of network topology as a variable, the resource data comprises calculation resource data and transmission resource data; calculating the delay parameter of the service data to be unloaded according to the resource data, which can be executed according to the following steps:
acquiring transmission delay data; calculating the execution duration of the service data to be unloaded according to the transmission delay data, the calculation resource data and the transmission resource data; and determining a time delay parameter of the service data to be unloaded according to the execution time length.
In the embodiment of the invention, the task delay consists of two parts, namely calculation delay and transmission delay. And pre-calculating transmission delay data, and then calculating the execution duration of the service data to be unloaded according to the transmission delay data, the calculation resource data and the transmission resource data.
In order to effectively reduce the time delay of service unloading and take the dynamic property of network topology as a variable for research, the embodiment of the invention divides the service unloading into four modes of idle equipment unloading, edge node unloading, cloud unloading and local unloading. The idle device unloading refers to unloading the service to a computing node which meets task requirements in a device layer, the edge node unloading refers to unloading the service to an edge node of the edge node layer, the cloud unloading refers to unloading the service to a corresponding server in a cloud service main control layer, and the local unloading refers to unloading the service to a computing node which meets the task requirements in the local area of the device layer. The idle device unloading and local unloading conditions are more suitable for the environment with small computing task requirements. The unloading of the edge node is more suitable for the environment with large computing task requirement and sensitive time delay. The cloud unloading is suitable for the environment with high computing task requirement and insensitive time delay. For the four service offload modes, only the local service offload mode does not need transmission delay, and the other three service offload modes all need transmission delay to be considered.
In order to obtain more reliable transmission delay data, the following formula may be used to generate the transmission delay data:
Figure BDA0002744383930000071
wherein, TRD,XRepresenting transmission delay data, X representing cloud computing service, sensor terminal or edge computing node, TDD,XIndicating the distance between the node of the task and the node receiving the task, GS indicating the Gaussian white noise power, THD,XRepresenting network messagesThe fading factor of the track, B represents the transmission bandwidth of the network, delta is the loss factor representing the network line,
Figure BDA00027443839300000715
indicating the power of the node accepting the task,
Figure BDA00027443839300000716
indicating the transmission line loss between the mission node and the node receiving the mission.
In order to ensure the calculation efficiency, the execution duration of the service data to be offloaded may be calculated according to the following formula based on the transmitted delay data, the calculation resource data, and the transmission resource data:
Figure BDA0002744383930000072
Figure BDA0002744383930000073
Figure BDA0002744383930000074
Figure BDA0002744383930000075
wherein,
Figure BDA0002744383930000076
indicating the duration of the execution of the task on the idle device,
Figure BDA0002744383930000077
indicating the computing power of the idle device, RD2DRepresenting the transmission delay data from node D to node D,
Figure BDA0002744383930000078
indicating the duration of execution of a task on a local device，
Figure BDA0002744383930000079
Which represents the computing power of the local device,
Figure BDA00027443839300000710
indicating the duration of execution of the task on the edge device,
Figure BDA00027443839300000711
representing the computing power of the edge device, RD2MRepresenting the transmission delay data from node D to the fog computing node M,
Figure BDA00027443839300000712
representing the execution duration of the task on the cloud device, fRRepresenting computing power of cloud-end equipment, RD2RRepresenting the transmission delay data from node D to cloud node R,
Figure BDA00027443839300000713
a set of fog computing node resources is represented,
Figure BDA00027443839300000714
representing a set of sensor nodes, DjRepresenting sensor nodes, QpRepresenting computing resource data, SpRepresenting transmission resource data.
In order to ensure that the time delay parameter is applicable to different unloading mode degrees, the time delay parameter of the service data to be unloaded is determined according to the execution duration by using the following formula:
Figure BDA0002744383930000081
Figure BDA0002744383930000082
wherein,
Figure BDA0002744383930000083
a latency parameter representing the service data to be offloaded,
Figure BDA0002744383930000084
a parameter representing a local offload mode is indicated,
Figure BDA0002744383930000085
indicating a fog computing node offload mode parameter,
Figure BDA0002744383930000086
a cloud offload mode parameter is represented that,
Figure BDA0002744383930000087
Figure BDA0002744383930000088
a parameter indicative of an idle device offload mode,
Figure BDA0002744383930000089
indicating the duration of the execution of the task on the local device,
Figure BDA00027443839300000810
indicating the duration of execution of the task on the edge device,
Figure BDA00027443839300000811
represents the execution time of the task on the cloud device,
Figure BDA00027443839300000812
indicating the duration of execution of the task on the idle device, KpRepresenting the subtasks in the service data to be offloaded.
In order to obtain an optimized service unloading result, the method generates a first service unloading result according to the service data to be unloaded and the initialized policy network by using the following formula:
Figure BDA00027443839300000813
Figure BDA00027443839300000814
Figure BDA00027443839300000815
Figure BDA00027443839300000816
Figure BDA00027443839300000817
wherein,
Figure BDA00027443839300000818
a set of actions is represented that is,
Figure BDA00027443839300000819
representing actions performed at the edge compute node,
Figure BDA00027443839300000820
indicating the action to be performed at the idle device,
Figure BDA00027443839300000829
indicating a fog computing node offload mode parameter,
Figure BDA00027443839300000822
indicating an idle device offload mode parameter, QpRepresenting computing resource data.
In order to evaluate the first service offloading result, a first evaluation result of the first service offloading result is generated according to the time delay parameter and the initialized evaluation network by using the following formula:
Figure BDA00027443839300000823
Figure BDA00027443839300000824
Figure BDA00027443839300000825
Figure BDA00027443839300000826
Figure BDA00027443839300000827
Figure BDA00027443839300000828
Figure BDA0002744383930000091
Figure BDA0002744383930000092
wherein S istThe state space is represented by a representation of,
Figure BDA0002744383930000093
indicating that the edge compute node is used for offloading,
Figure BDA0002744383930000094
indicating that the use of the idle device is off-load,
Figure BDA0002744383930000095
which represents the total delay of the service,
Figure BDA0002744383930000096
to representThe delay parameter of the service data to be offloaded,
Figure BDA0002744383930000097
presentation service KpThe time required for all local executions, r(s)t,at) Representing a reward function, Vπ(s) represents a value function, s represents a state, and π represents a strategy.
In order to improve the efficiency and accuracy of updating the neural network, according to the first service offloading result and the first evaluation result, updating a first gradient parameter of the policy network and a second gradient parameter of the evaluation network by using a depth deterministic policy gradient algorithm, which may be performed according to the following steps:
determining an action value function according to the first service unloading result and the first evaluation result; and updating a first gradient parameter of the policy network and a second gradient parameter of the evaluation network by using a depth deterministic policy gradient algorithm according to the action value function.
In order to improve the calculation efficiency, an action value function is determined according to the first service unloading result and the first evaluation result according to the following formula:
Figure BDA0002744383930000098
Figure BDA0002744383930000099
wherein Q isπ(s, a) represents an action value function, mu represents a strategy generated by the strategy network, gamma represents an attenuation factor, the value range is 0 to 1, s' represents the next state after s takes a, a represents action, and Q represents the next state after s takes aμ(st,at) Representing the action value function obtained according to the bellman equation.
In order to improve the calculation accuracy, a first gradient parameter of the strategy network and a second gradient parameter of the evaluation network are updated by a depth certainty strategy gradient algorithm according to an action value function according to the following formula:
Figure BDA00027443839300000910
Figure BDA00027443839300000911
Figure BDA00027443839300000912
Figure BDA00027443839300000913
wherein m represents the serial number of the sensor node,
Figure BDA00027443839300000914
is the first gradient parameter that is to be measured,
Figure BDA00027443839300000915
is the second gradient parameter that is to be measured,
Figure BDA00027443839300000916
is the updated first gradient parameter and,
Figure BDA00027443839300000917
is the updated second gradient parameter,
Figure BDA00027443839300000918
the status is represented by a number of time slots,
Figure BDA00027443839300000919
represents an action, QiRepresents a motion value function, Q ', obtained from Bellman equation'iRepresents the updated action value function obtained according to the bellman equation,
Figure BDA0002744383930000101
indicating the updated state, gamma indicates the decay factor,the function J is used to measure the behavior of the strategy mu, pβRepresenting the distribution function of the state s.
Referring to the schematic diagram of the implementation steps of the service offloading method shown in fig. 3, the method mainly includes the following steps: designing a layered Internet of things service unloading management architecture in a fog computing environment; constructing a service unloading decision model based on DDPG in a fog computing environment; solving the service unloading algorithm scheme design for minimizing the service unloading delay; a service offload policy is executed. The procedure for carrying out this embodiment is described below in a specific example.
Referring to fig. 2, in the hierarchical internet of things service offloading management architecture in the step-designed fog computing environment, the hierarchical internet of things service offloading management architecture includes, from top to bottom, a cloud service master control layer, a regional SDN control layer, an edge node layer, a data layer, and a device layer.
The function and construction of each layer is described in detail below. (1) The cloud service master control layer mainly comprises two aspects, one is to provide services for various tasks in the environment of the Internet of things, and the other is to perform unified management on SDN controllers in various areas. In order to complete the two functions, the system mainly comprises a web server, an application server, a database server and an SDN main controller. The SDN main controller completes the unified management function of each SDN area control. (2) And the SDN control layer of each area mainly completes resource scheduling and service unloading of computing resources and task requirements in each area. Each zone includes an SDN control that obtains computing resource conditions from edge nodes and computing resource requirements from task requesters. (3) And the edge node layer mainly completes the calculation and storage functions. The computing nodes complete computing functions and are usually located near the task demand nodes; the storage node completes the storage function of accessing data for a high frequency. By reasonably arranging the computing nodes and the storage nodes, the computing and storage access time delay of tasks can be effectively reduced, and the requirement of network bandwidth is remarkably reduced. (4) And the data layer mainly completes the functions of acquiring the task request data of the sensor node of the Internet of things and transmitting the task request data to the edge node. The data layer mainly comprises a domain controller, a wireless access point and a high-reliability industrial switch. The domain controller mainly completes the functions of acquiring the requirements and requesting the resources. (5) The equipment layer mainly refers to the sensor network environment of each application scene, and mainly comprises sensor networks in intelligent environments such as factories, markets, homes and the like.
In the aspect of network environment, the network environment of the Internet of things is divided into N unit areas, and { Cell is used1,Cell2,…CellNRepresents it. The cloud computing service is represented by R. Cell in each Cell areaiIncluding m sensor terminals (using)
Figure BDA0002744383930000102
Representation), n fog computing nodes (usage)
Figure BDA0002744383930000103
Figure BDA0002744383930000104
Representation).
In terms of computing tasks, K is used to represent a service request, each task request comprising p subtasks KpUsing Kp＝(Qp,Sp,Tm) And (4) showing. Wherein Q ispRepresenting the computational resources required by the task, SpIndicating the transmission resources, T, required by the taskmIndicating the task's constraints on latency. The task delay consists of two parts, namely calculation delay and transmission delay. For the four service offload modes, only the local service offload mode does not need transmission delay, and the other three service offload modes all need transmission delay to be considered.
Transmission delay usage formula
Figure BDA0002744383930000111
Performing a calculation wherein TRD,XRepresenting transmission delay data, X representing any one of cloud computing service, sensor terminal or edge computing node, TDD,XIndicating the distance between the node of the task and the node receiving the task, GS indicating the Gaussian white noise power, THD,XTo representThe fading factor of the network channel, B represents the transmission bandwidth of the network, delta is the loss factor representing the network line,
Figure BDA0002744383930000112
indicating the power of the node accepting the task,
Figure BDA0002744383930000113
indicating the transmission line loss between the mission node and the node receiving the mission.
In the step of constructing a DDPG-based service unloading decision model in a fog computing environment, an SDN controller needs to unload services, firstly, a task demand condition and a fog computing resource condition need to be acquired, and then, an intelligent optimization algorithm is adopted to unload the services. In order to effectively reduce the time delay of service unloading and take the dynamic property of network topology as a variable for research, the invention divides the service unloading into four modes of idle equipment unloading, edge node unloading, cloud unloading and local unloading.
Representation of four service offloads: for the computation subtask KpThe four service offloads are respectively expressed as: (1) idle device offload mode usage
Figure BDA0002744383930000114
And (4) showing. (2) Local offload mode usage
Figure BDA0002744383930000115
Represents; (3) fog computing node offload mode usage
Figure BDA0002744383930000116
And (4) showing. (4) Cloud offload mode usage
Figure BDA0002744383930000117
And (4) showing.
Task latency for four service offloads: for the computation subtask KpThe task delays for the four service offloading are described below. Task delay usage formula for idle device offload mode
Figure BDA0002744383930000118
It is shown that, among others,
Figure BDA0002744383930000119
representing the execution time of the task on the idle device,
Figure BDA00027443839300001110
indicating the computing power of the idle device. Local offload mode usage formula
Figure BDA00027443839300001111
It is shown that, among others,
Figure BDA00027443839300001112
representing the execution time of the task on the local device,
Figure BDA00027443839300001113
representing the computing power of the local device. Fog calculation node unloading mode use formula
Figure BDA00027443839300001114
Is shown in which
Figure BDA00027443839300001115
Representing the execution time of the task on the edge device,
Figure BDA00027443839300001116
representing the computing power of the edge device. Cloud offload mode usage formula
Figure BDA00027443839300001117
It is shown that, among others,
Figure BDA00027443839300001118
representing the execution time of a task on a cloud device, fRRepresenting the computing power of the cloud device. In summary, task KpThe total calculated delay can be formulated
Figure BDA0002744383930000121
And (6) performing calculation.
It should be noted that, in the following description,
Figure BDA0002744383930000122
and
Figure BDA0002744383930000123
is 0 or 1, when the value is 0, it indicates that the unloading mode corresponding to the coefficient is not adopted, and when the value is 1, it indicates that the unloading mode corresponding to the coefficient is adopted.
In the step of solving the service unloading algorithm scheme for minimizing the service unloading delay, the solution is carried out based on a depth certainty strategy gradient algorithm in the deep reinforcement learning, so that each intelligent sensor of the Internet of things can obtain the optimized service unloading scheme. Considering that the local offload mode can perform service offload through internal calculation of the sensor, and the embodiment of the present invention mainly solves delay-sensitive service offload, therefore, the embodiment of the present invention mainly studies two offload modes, namely idle device offload and edge node offload.
1) Service unloading decision model based on DDPG
When the DDPG algorithm is used for solving the optimized service unloading strategy, an Actor-criticc framework is adopted. The Actor refers to a policy network and comprises an Actor _ M network and an Actor _ T network, wherein the main function of the Actor _ T network is to generate a training data set, and the main function of the Actor _ M network is to train parameters of an optimization network. Critic refers to an evaluation network, and comprises a Critic _ M network and a Critic _ T network, wherein the Critic _ T network mainly functions to generate a training data set, and the Critic _ M network mainly functions to train optimization network parameters. .
Deep reinforcement learning refers to learning an action strategy by using a deep neural network on the basis of traditional reinforcement learning. In general, the reinforcement learning model includes four main elements, namely environment, state s, strategy pi and action a. And (3) optimizing the four main elements by using the Agent of the Agent in the reinforcement learning model to obtain the reward r as an evaluation standard. For convenience of description, at time t, s is usedtIndicating the state at time t, using st+1Indicating the state at time t + 1. Wherein s ist+1Is that Agent selects action a according to policy πtThe next state entered after execution in the environment. To evaluate the effectiveness of a policy, the return of an Agent using the policy's actions when executed in the environment is defined as the reward rt. The following is a detailed description of each of the key elements.
2) Deep reinforcement learning factor modeling
In the aspect of state space, the states of the fog computing nodes and the idle equipment terminals around the sensor equipment form the state space. For fog computing nodes, use
Figure BDA0002744383930000124
Representing the remaining resource sets of each edge node within the region, wherein,
Figure BDA0002744383930000125
for idle device terminals, use
Figure BDA0002744383930000126
Indicating the set of remaining resources for each free device within the area, wherein,
Figure BDA0002744383930000127
Figure BDA0002744383930000131
based on this, the formula is used
Figure BDA0002744383930000132
A state space is represented in which, among other things,
Figure BDA0002744383930000133
the total time delay of the service is represented by the following calculation method
Figure BDA0002744383930000134
Defining service offload decisions as a formula in terms of action space
Figure BDA0002744383930000135
Wherein,
Figure BDA0002744383930000136
Figure BDA0002744383930000137
in terms of reward functions, the reward functions are defined as formulas
Figure BDA0002744383930000138
Wherein,
Figure BDA0002744383930000139
presentation service KpAll the time required for local execution,
Figure BDA00027443839300001310
indicating that the service task is in state stTaking action oftThe latter execution duration.
In terms of value functions, formulas are used
Figure BDA00027443839300001311
Calculated value Vπ(s) a function representing the expectation of the reward r of the strategy pi at the initial state s. Wherein,
Figure BDA00027443839300001312
representing the desired operation.
Based on this, a formula can be used
Figure BDA00027443839300001313
Calculating a motion value function Qπ(s, a), gamma represents the attenuation factor, the value range is 0 to 1, and s' represents the next state after s takes a. To simplify the computation of the action value function, known from Bellman Equation, according to the public
Figure BDA00027443839300001314
Figure BDA00027443839300001315
The formula Q can be obtainedμ(st,at)＝E[r(st,at)+γQμ(st+1,μ(st+1))]. Where μ denotes a policy generated by the Actor network.
3) Depth-deterministic policy gradients
The strategy gradient is an optimized search algorithm based on gradient, and in the invention, gradient update parameters of a Critic _ M network and an Actor _ M network are as follows
Figure BDA00027443839300001316
And
Figure BDA00027443839300001317
the gradient update parameters of the Critic _ T network and the Actor _ T network are
Figure BDA00027443839300001318
And
Figure BDA00027443839300001319
the calculation process of the Loss function is defined as a formula
Figure BDA00027443839300001320
Wherein,
Figure BDA00027443839300001321
using the formula
Figure BDA00027443839300001322
And (4) calculating.
Based on this, the gradient is calculated as formula
Figure BDA00027443839300001323
Figure BDA00027443839300001324
Wherein the function J uses the formula
Figure BDA00027443839300001325
Calculations to measure the performance of the strategy mu. RhoβRepresenting the distribution function of the state s. Therefore, the formula
Figure BDA00027443839300001326
Is defined as the value of p in terms of s in the state spaceβDistributed computation Qμ(s, μ (s)) as expected.
When the Actor network interacts with the environment, in order to prevent the problem that the neural network cannot be converged due to the correlation of the conversion data sequence in time, the embodiment of the invention stores the conversion data sequence in the experience pool and then performs random sampling based on the mini-batch, thereby realizing the time irrelevance of the data.
4) Service unloading algorithm based on DDPG
A Service Offloading algorithm (Service Offloading algorithm) based on DDPG proposed by the embodiment of the present invention is shown in table 1. The method comprises two processes of model initialization and optimal service unloading strategy solving for each area of the network environment. The model initialization steps include steps 1 to 7 in table 1, and mainly initialize four network models and key parameters in the Actor-critical architecture. The step of solving the optimized service unloading strategy in each area of the network environment comprises the steps from 8 th to 25 th in the table 1, and the optimal service unloading strategy is mainly solved within the iteration number.
Figure BDA0002744383930000141
TABLE 1
In order to verify the performance of the algorithm, a network environment of fog computing is developed by adopting Python language. Based on the zone-dividing idea of the present invention, the network environment is divided into 25 cells. The minimum distance between the individual units is set to 10 meters. In each cell, 2 idle devices and 2 fog computing nodes are set. Where the idle device calculation frequency is 200 and 150. The calculation frequency of the fog calculation node is 650 and 600. The transmission power of the devices is subject to a uniform distribution between (5, 38). In terms of services that need to be offloaded, 100 services that need to be offloaded are randomly generated, with an even distribution of computing resource services (100, 800) required for each service.
In the aspect of parameter setting, the minimum batch in the experience pool is set to be 500, the Actor _ M network learning rate is set to be 0.0001, the criticic _ M network learning rate is set to be 0.001, the gaussian noise power during transmission delay calculation is set to be-114 dbm, the capacity of the memory pool is set to be 50000, the maximum iteration number is set to be 5000, and the exploration number is set to be 10. Considering that the existing Distributed Cooperative Task Offloading (DCTO) based on a plurality of edge nodes better realizes the calculation service Offloading of the intelligent terminal, the sooddppg algorithm of the present invention is compared with the existing DCTO algorithm in the experimental part.
The two algorithms are compared in terms of average data transmission rate, service offloading success rate, and task execution time.
In terms of average data transmission rate, a comparison is made in two dimensions from the maximum distance between adjacent units, the number of devices connected per compute node.
The effect of the maximum distance between adjacent cells on the average data transmission rate is shown in fig. 4. Wherein the X-axis indicates that the maximum distance between adjacent cells varies between 100m and 1000 m. The Y-axis represents the average data transmission rate (bps). It can be seen from the figure that as the maximum distance between adjacent cells increases, the average data transmission rate under both algorithms increases. This is because the network size increases, mutual interference in the network environment decreases, and the data transmission rate increases. Under various circumstances, the average data transfer rate of the inventive algorithm SOoDDPG is greater than the average data transfer rate of the algorithm DCTO. The algorithm can realize the service unloading in a relatively optimized mode, reduces the interference among network equipment and further improves the data transmission rate.
The effect of the number of devices connected per compute node on the average data transfer rate is shown in fig. 5. Where the X-axis represents the number of devices connected per compute node varying between 1 and 8. The Y-axis represents the average data transmission rate (bps). As can be seen from the figure, as the number of devices connected to each node increases, the average data transmission rate under both algorithms decreases rapidly, which means that the number of devices increases, which easily causes the load of the computing node to increase, thereby decreasing the average data transmission rate. However, in the context of different device numbers, the algorithm SOoDDPG of the present invention achieves better results than the algorithm DCTO.
The embodiment of the invention provides a service unloading method, a device and a system, wherein the method is used for solving based on a depth certainty strategy gradient algorithm in deep reinforcement learning, so that each intelligent sensor of the Internet of things can obtain an optimized service unloading scheme, the method has good application effect and performance, and the problem of large service unloading delay in a large-scale network is solved well.
The embodiment of the invention also provides a service unloading device, which is described in the following embodiment. Because the principle of the device for solving the problems is similar to the service unloading method, the implementation of the device can refer to the implementation of the service unloading method, and repeated details are not repeated. Referring to fig. 6, a block diagram of a service offloading device is shown, which includes:
an obtaining module 61, configured to obtain service data to be offloaded, resource data, an initialized policy network, and an initialized evaluation network; the time delay module 62 is configured to calculate a time delay parameter of the service data to be unloaded according to the resource data; a first generating module 63, configured to generate a first service offloading result according to the service data to be offloaded and the initialized policy network; a second generating module 64, configured to generate a first evaluation result of the first service offloading result according to the delay parameter and the initialized evaluation network; an updating module 65, configured to update a first gradient parameter of the policy network and a second gradient parameter of the evaluation network by using a depth deterministic policy gradient algorithm according to the first service offload result and the first evaluation result; and the result module 66 is configured to generate a target service offloading result of the service data to be offloaded according to the updated policy network and the updated evaluation network.
In one embodiment, referring to another structural block diagram of the service offloading device shown in fig. 7, the device further includes an initialization module 67 configured to: randomly generating an initial first gradient parameter and an initial second gradient parameter; initializing the strategy network by utilizing the initial first gradient parameter to obtain an initialized strategy network; initializing the evaluation network by using the initial second gradient parameter to obtain the initialized evaluation network
In one embodiment, the resource data includes computational resource data and transmission resource data; a delay module, specifically configured to: acquiring transmission delay data; calculating the execution duration of the service data to be unloaded according to the transmission delay data, the calculation resource data and the transmission resource data; and determining a time delay parameter of the service data to be unloaded according to the execution time length.
In an embodiment, before the obtaining the transmission delay data, the delay module is further configured to generate the transmission delay data according to the following formula:
Figure BDA0002744383930000161
wherein, TRD,XRepresenting transmission delay data, X representing cloud computing service, sensor terminal or edge computing node, TDD,XIndicating the distance between the node of the task and the node receiving the task, GS indicating the Gaussian white noise power, THD,XRepresenting the fading factor of the network channel, B representing the transmission bandwidth of the network, delta is the loss factor representing the network line,
Figure BDA0002744383930000162
indicating the power of the node accepting the task,
Figure BDA0002744383930000171
indicating the transmission line loss between the mission node and the node receiving the mission.
In an embodiment, the delay module is further configured to calculate an execution duration of the service data to be offloaded according to the transmission delay data, the calculation resource data, and the transmission resource data according to the following formula:
Figure BDA0002744383930000172
Figure BDA0002744383930000173
wherein,
Figure BDA0002744383930000174
indicating the duration of the execution of the task on the idle device,
Figure BDA0002744383930000175
indicating the computing power of the idle device, RD2DRepresenting the transmission delay data from node D to node D,
Figure BDA0002744383930000176
indicating the duration of the execution of the task on the local device,
Figure BDA0002744383930000177
which represents the computing power of the local device,
Figure BDA0002744383930000178
indicating the duration of execution of the task on the edge device,
Figure BDA0002744383930000179
representing the computing power of the edge device, RD2MRepresenting the transmission delay data from node D to the fog computing node M,
Figure BDA00027443839300001710
representing the execution duration of the task on the cloud device, fRRepresenting computing power of cloud-end equipment, RD2RRepresenting the transmission delay data from node D to cloud node R,
Figure BDA00027443839300001711
a set of fog computing node resources is represented,
Figure BDA00027443839300001712
representing a set of sensor nodes, DjRepresenting sensor nodes, QpRepresenting computing resource data, SpRepresenting transmission resource data.
In an embodiment, the delay module is further configured to determine a delay parameter of the service data to be offloaded according to the execution duration by using the following formula:
Figure BDA00027443839300001713
wherein,
Figure BDA00027443839300001714
a latency parameter representing the service data to be offloaded,
Figure BDA00027443839300001715
a parameter representing a local offload mode is indicated,
Figure BDA00027443839300001716
indicating a fog computing node offload mode parameter,
Figure BDA00027443839300001717
a cloud offload mode parameter is represented that,
Figure BDA00027443839300001718
a parameter indicative of an idle device offload mode,
Figure BDA00027443839300001719
indicating the duration of the execution of the task on the local device,
Figure BDA00027443839300001720
indicating the duration of execution of the task on the edge device,
Figure BDA00027443839300001721
represents the execution time of the task on the cloud device,
Figure BDA00027443839300001722
indicating the duration of execution of the task on the idle device, KpRepresenting the subtasks in the service data to be offloaded.
In an embodiment, the first generating module is specifically configured to: according to the service to be offloaded, using the following formulaThe data and the initialized policy network generate a first service offload result:
Figure BDA00027443839300001723
Figure BDA00027443839300001724
wherein,
Figure BDA00027443839300001725
a set of actions is represented that is,
Figure BDA00027443839300001726
representing actions performed at the edge compute node,
Figure BDA00027443839300001727
indicating the action to be performed at the idle device,
Figure BDA00027443839300001728
indicating a fog computing node offload mode parameter,
Figure BDA00027443839300001729
indicating an idle device offload mode parameter, QpRepresenting computing resource data.
In an embodiment, the second generating module is specifically configured to: generating a first evaluation result of the first service offloading result according to the time delay parameter and the initialized evaluation network by using the following formula:
Figure BDA00027443839300001730
Figure BDA00027443839300001731
Figure BDA0002744383930000181
Figure BDA0002744383930000182
wherein S istThe state space is represented by a representation of,
Figure BDA0002744383930000183
indicating that the edge compute node is used for offloading,
Figure BDA0002744383930000184
indicating that the use of the idle device is off-load,
Figure BDA0002744383930000185
which represents the total delay of the service,
Figure BDA0002744383930000186
a latency parameter representing the service data to be offloaded,
Figure BDA0002744383930000187
presentation service KpThe time required for all local executions, r(s)t,at) Representing a reward function, Vπ(s) represents a value function, s represents a state, and π represents a strategy.
In one embodiment, the update module is specifically configured to: according to the first service unloading result and the first evaluation result, updating a first gradient parameter of the policy network and a second gradient parameter of the evaluation network by using a deep deterministic policy gradient algorithm, wherein the method comprises the following steps: determining an action value function according to the first service unloading result and the first evaluation result; and updating a first gradient parameter of the policy network and a second gradient parameter of the evaluation network by using a depth deterministic policy gradient algorithm according to the action value function.
In one embodiment, the update module is specifically configured to: determining an action value function according to the first service uninstalling result and the first evaluation result according to the following formula:
Figure BDA0002744383930000188
Qμ(st,at)＝E[r(st,at)+γQμ(st+1,μ(st+1))]
wherein Q isπ(s, a) represents an action value function, mu represents a strategy generated by the strategy network, gamma represents an attenuation factor, the value range is 0 to 1, s' represents the next state after s takes a, a represents action, and Q represents the next state after s takes aμ(st,at) Representing the action value function obtained according to the bellman equation.
In one embodiment, the update module is specifically configured to: updating a first gradient parameter of the policy network and a second gradient parameter of the evaluation network by using a depth deterministic policy gradient algorithm according to the action value function according to the following formula:
Figure BDA0002744383930000189
Figure BDA00027443839300001810
Figure BDA00027443839300001811
wherein m represents the serial number of the sensor node,
Figure BDA00027443839300001812
is the first gradient parameter that is to be measured,
Figure BDA00027443839300001813
is the second gradient parameter that is to be measured,
Figure BDA00027443839300001814
is the updated first gradient parameter and,
Figure BDA00027443839300001815
is the updated second gradient parameter,
Figure BDA00027443839300001816
the status is represented by a number of time slots,
Figure BDA00027443839300001817
represents an action, QiTo representMotion value function Q 'obtained according to Bellman equation'iRepresents the updated action value function obtained according to the bellman equation,
Figure BDA00027443839300001818
representing the updated state, gamma represents the decay factor, and the function J is used to measure the behavior of the strategy mu, ρβRepresenting the distribution function of the state s.
The embodiment of the invention also provides a service unloading system, which comprises a cloud service main control module, a service unloading module and a service unloading module, wherein the cloud service main control module comprises an SDN main controller, a data server, a Web server and an application server; the regional SDN control module comprises a plurality of SDN controllers; the node module comprises a plurality of computing nodes and cache nodes; the data module comprises a switch, a wireless access unit and a domain controller; the equipment module includes a plurality of sensors.
In the embodiment of the present invention, referring to the schematic diagram of the service offloading system architecture shown in fig. 2, a cloud service master control module may be set as a cloud service master control layer, and the main functions include two aspects, one is to provide services for various tasks in the environment of the internet of things, and the other is to perform unified management on SDN controllers in various areas. In order to complete the two functions, the system mainly comprises a Web server, an application server, a database server and an SDN main controller. The SDN main controller completes the unified management function of controlling each SDN area.
The regional SDN control module can be set as a regional SDN control layer and is mainly used for completing resource scheduling and service unloading of computing resources and task requirements in each region. Each zone includes an SDN control that obtains computing resource conditions from edge nodes and computing resource requirements from task requesters.
The node modules may be configured to perform mainly computing and storage functions for the edge node layer. The computing nodes complete computing functions and are usually located near the task demand nodes; the storage node completes the storage function of accessing data for a high frequency. By reasonably arranging the computing nodes and the storage nodes, the computing and storage access time delay of tasks can be effectively reduced, and the requirement of network bandwidth is remarkably reduced.
The data module can be set as a data layer and mainly achieves the functions of acquiring task request data of the sensor node of the Internet of things and transmitting the task request data to the edge node. The data layer mainly comprises a domain controller, a wireless access point and a high-reliability industrial switch. The domain controller mainly completes the functions of acquiring the requirements and requesting the resources.
The device module can be set as a device layer, mainly refers to the sensor network environment of each application scene, and mainly comprises sensor networks in intelligent environments such as factories, markets, homes and the like.
An embodiment of the present invention further provides a computer device, referring to the schematic block diagram of the structure of the computer device shown in fig. 8, where the computer device includes a memory 81, a processor 82, and a computer program stored in the memory and executable on the processor, and the processor implements any of the steps of the service uninstalling method when executing the computer program.
It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working process of the computer device described above may refer to the corresponding process in the foregoing method embodiment, and is not described herein again.
An embodiment of the present invention further provides a computer-readable storage medium, where a computer program for executing any one of the above-mentioned service offloading methods is stored in the computer-readable storage medium.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Finally, it should be noted that: the above-mentioned embodiments are only specific embodiments of the present invention, which are used for illustrating the technical solutions of the present invention and not for limiting the same, and the protection scope of the present invention is not limited thereto, although the present invention is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present invention, and they should be construed as being included therein. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims (18)

1. A method of service offloading, comprising:
acquiring service data to be unloaded, resource data, an initialized policy network and an initialized evaluation network;
calculating a time delay parameter of the service data to be unloaded according to the resource data;
generating a first service unloading result according to the service data to be unloaded and the initialized policy network;
generating a first evaluation result of the first service unloading result according to the time delay parameter and the initialized evaluation network;
updating a first gradient parameter of the policy network and a second gradient parameter of the evaluation network by using a deep deterministic policy gradient algorithm according to the first service offload result and the first evaluation result;
and generating a target service unloading result of the service data to be unloaded according to the updated strategy network and the updated evaluation network.
2. The method of claim 1, wherein obtaining the initialized policy network and the initialized evaluation network is preceded by:
randomly generating an initial first gradient parameter and an initial second gradient parameter;
initializing a policy network by using the initial first gradient parameter to obtain an initialized policy network;
and initializing the evaluation network by using the initial second gradient parameter to obtain an initialized evaluation network.
3. The method of claim 1, wherein the resource data comprises computational resource data and transmission resource data;
calculating a time delay parameter of the service data to be unloaded according to the resource data, wherein the time delay parameter comprises the following steps:
acquiring transmission delay data;
calculating the execution duration of the service data to be unloaded according to the transmission delay data, the calculation resource data and the transmission resource data;
and determining a time delay parameter of the service data to be unloaded according to the execution time length.
4. The method of claim 3, wherein prior to obtaining the propagation delay data, further comprising generating the propagation delay data according to the following equation:
Figure FDA0002744383920000011
wherein, TRD,XRepresenting transmission delay data, X representing cloud computing service, sensor terminal or edge computing node, TDD,XIndicating the distance between the node of the task and the node receiving the task, GS indicating the Gaussian white noise power, THD,XRepresenting the fading factor of the network channel, B representing the transmission bandwidth of the network, delta is the loss factor representing the network line,
Figure FDA0002744383920000021
indicating the power of the node accepting the task,
Figure FDA0002744383920000022
indicating the transmission line loss between the mission node and the node receiving the mission.
5. The method of claim 4, comprising calculating an execution duration of service data to be offloaded from the transmission delay data, the computational resource data, and the transmission resource data according to the following formula:
Figure FDA0002744383920000023
Figure FDA0002744383920000024
Figure FDA0002744383920000025
Figure FDA0002744383920000026
wherein,
Figure FDA0002744383920000027
indicating the duration of the execution of the task on the idle device,
Figure FDA0002744383920000028
indicating the computing power of the idle device, RD2DRepresenting the transmission delay data from node D to node D,
Figure FDA0002744383920000029
indicating the duration of the execution of the task on the local device,
Figure FDA00027443839200000210
which represents the computing power of the local device,
Figure FDA00027443839200000211
indicating the duration of execution of the task on the edge device,
Figure FDA00027443839200000212
representing the computing power of the edge device, RD2MRepresenting the transmission delay data from node D to the fog computing node M,
Figure FDA00027443839200000213
representing the execution duration of the task on the cloud device, fRRepresenting computing power of cloud-end equipment, RD2RRepresenting the transmission delay data from node D to cloud node R,
Figure FDA00027443839200000214
a set of fog computing node resources is represented,
Figure FDA00027443839200000215
representing a set of sensor nodes, DjRepresenting sensor nodes, QpRepresenting computing resource data, SpRepresenting transmission resource data.
6. The method of claim 5, comprising determining a delay parameter for the service data to be offloaded according to the execution duration using the following formula:
Figure FDA00027443839200000216
wherein,
Figure FDA00027443839200000217
a latency parameter representing the service data to be offloaded,
Figure FDA00027443839200000218
a parameter representing a local offload mode is indicated,
Figure FDA00027443839200000219
indicating a fog computing node offload mode parameter,
Figure FDA00027443839200000220
a cloud offload mode parameter is represented that,
Figure FDA00027443839200000221
Figure FDA00027443839200000222
a parameter indicative of an idle device offload mode,
Figure FDA00027443839200000223
indicating the duration of the execution of the task on the local device,
Figure FDA00027443839200000224
indicating the duration of execution of the task on the edge device,
Figure FDA00027443839200000225
represents the execution time of the task on the cloud device,
Figure FDA00027443839200000226
indicating the duration of execution of the task on the idle device, KpRepresenting the subtasks in the service data to be offloaded.
7. The method of claim 6, comprising: generating a first service unloading result according to the service data to be unloaded and the initialized policy network by using the following formula:
Figure FDA0002744383920000031
Figure FDA0002744383920000032
Figure FDA0002744383920000033
Figure FDA0002744383920000034
Figure FDA0002744383920000035
wherein,
Figure FDA0002744383920000036
a set of actions is represented that is,
Figure FDA0002744383920000037
representing actions performed at the edge compute node,
Figure FDA0002744383920000038
indicating the action to be performed at the idle device,
Figure FDA0002744383920000039
indicating a fog computing node offload mode parameter,
Figure FDA00027443839200000310
indicating an idle device offload mode parameter, QpRepresenting computing resource data.
8. The method of claim 7, comprising: generating a first evaluation result of the first service offloading result according to the delay parameter and the initialized evaluation network by using the following formula:
Figure FDA00027443839200000311
Figure FDA00027443839200000312
Figure FDA00027443839200000313
Figure FDA00027443839200000314
Figure FDA00027443839200000315
Figure FDA00027443839200000316
Figure FDA00027443839200000317
Figure FDA00027443839200000318
wherein S istThe state space is represented by a representation of,
Figure FDA00027443839200000319
indicating that the edge compute node is used for offloading,
Figure FDA00027443839200000320
indicating that the use of the idle device is off-load,
Figure FDA00027443839200000321
which represents the total delay of the service,
Figure FDA00027443839200000322
a latency parameter representing the service data to be offloaded,
Figure FDA00027443839200000323
presentation service KpThe time required for all local executions, r(s)t,at) Representing a reward function, Vπ(s) represents a value function, s represents a state, and π represents a strategy.
9. The method of claim 1, wherein updating a first gradient parameter of the policy network and a second gradient parameter of the evaluation network using a deep deterministic policy gradient algorithm based on the first service offload result and the first evaluation result comprises:
determining an action value function according to the first service unloading result and the first evaluation result;
updating a first gradient parameter of the policy network and a second gradient parameter of the evaluation network using a depth-deterministic policy gradient algorithm according to the action value function.
10. The method of claim 9, comprising determining an action value function based on the first service offload result and the first evaluation result according to the following formula:
Figure FDA0002744383920000041
Figure FDA0002744383920000042
wherein Q isπ(s, a) represents an action value function, mu represents a strategy generated by the strategy network, gamma represents an attenuation factor, the value range is 0 to 1, s' represents the next state after s takes a, a represents action, and Q represents the next state after s takes aμ(st,at) Representing the action value function obtained according to the bellman equation.
11. The method of claim 9, wherein the first gradient parameter of the policy network and the second gradient parameter of the evaluation network are updated using a depth-deterministic policy gradient algorithm according to the action value function according to the following formula:
Figure FDA0002744383920000043
Figure FDA0002744383920000044
Figure FDA0002744383920000045
Figure FDA0002744383920000046
wherein m represents the serial number of the sensor node,
Figure FDA0002744383920000047
is the first gradient parameter that is to be measured,
Figure FDA0002744383920000048
is the second gradient parameter that is to be measured,
Figure FDA0002744383920000049
is the updated first gradient parameter and,
Figure FDA00027443839200000410
is the updated second gradient parameter,
Figure FDA00027443839200000411
the status is represented by a number of time slots,
Figure FDA00027443839200000412
represents an action, QiRepresents a motion value function, Q ', obtained from Bellman equation'iRepresents the updated action value function obtained according to the bellman equation,
Figure FDA00027443839200000413
representing the updated state, gamma represents the decay factor, and the function J is used to measure the behavior of the strategy mu, ρβRepresenting the distribution function of the state s.
12. A service offloading device, comprising:
the acquisition module is used for acquiring service data to be unloaded, resource data, an initialized policy network and an initialized evaluation network;
the time delay module is used for calculating a time delay parameter of the service data to be unloaded according to the resource data;
a first generating module, configured to generate a first service offload result according to the to-be-offloaded service data and the initialized policy network;
a second generation module, configured to generate a first evaluation result of the first service offloading result according to the delay parameter and the initialized evaluation network;
an updating module, configured to update a first gradient parameter of the policy network and a second gradient parameter of the evaluation network by using a depth-deterministic policy gradient algorithm according to the first service offload result and the first evaluation result;
and the result module is used for generating a target service unloading result of the service data to be unloaded according to the updated strategy network and the updated evaluation network.
13. The apparatus of claim 12, further comprising an initialization module to:
randomly generating an initial first gradient parameter and an initial second gradient parameter;
initializing a policy network by using the initial first gradient parameter to obtain an initialized policy network;
and initializing the evaluation network by using the initial second gradient parameter to obtain an initialized evaluation network.
14. The apparatus of claim 12, wherein the resource data comprises computational resource data and transmission resource data; the delay module is specifically configured to:
acquiring transmission delay data;
calculating the execution duration of the service data to be unloaded according to the transmission delay data, the calculation resource data and the transmission resource data;
and determining a time delay parameter of the service data to be unloaded according to the execution time length.
15. The apparatus of claim 12, wherein the update module is specifically configured to:
determining an action value function according to the first service unloading result and the first evaluation result;
updating a first gradient parameter of the policy network and a second gradient parameter of the evaluation network using a depth-deterministic policy gradient algorithm according to the action value function.
16. A service offloading system comprising a cloud service master module, a regional SDN control module, an edge node module, a data module, a device module, and the service offloading device of any of claims 12-15;
the service unloading device is in communication connection with the cloud service main control module, the regional SDN control module, the node module, the data module and the equipment module respectively.
17. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the service offloading method of any of claims 1 to 11 when executing the computer program.
18. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program for executing the service offloading method of any of claims 1 through 11.
CN202011161252.4A 2020-10-27 2020-10-27 Service unloading method, device and system Pending CN112312299A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011161252.4A CN112312299A (en) 2020-10-27 2020-10-27 Service unloading method, device and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011161252.4A CN112312299A (en) 2020-10-27 2020-10-27 Service unloading method, device and system

Publications (1)

Publication Number Publication Date
CN112312299A true CN112312299A (en) 2021-02-02

Family

ID=74330731

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011161252.4A Pending CN112312299A (en) 2020-10-27 2020-10-27 Service unloading method, device and system

Country Status (1)

Country Link
CN (1) CN112312299A (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114363847A (en) * 2021-02-08 2022-04-15 西北工业大学 Task unloading method, device and system
CN114760308A (en) * 2022-04-01 2022-07-15 中国科学技术大学 Edge calculation unloading method and device
CN117176618A (en) * 2023-11-03 2023-12-05 中国西安卫星测控中心 Performance evaluation method for network data exchange software system
CN118295794A (en) * 2024-02-26 2024-07-05 北京中电飞华通信有限公司 Computing resource configuration method and device of power system network and electronic equipment

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109510869A (en) * 2018-11-22 2019-03-22 北京信息科技大学 A kind of Internet of Things service dynamic offloading method and device based on edge calculations
CN109788069A (en) * 2019-02-27 2019-05-21 电子科技大学 Calculating discharging method based on mobile edge calculations in Internet of Things
CN109819046A (en) * 2019-02-26 2019-05-28 重庆邮电大学 A kind of Internet of Things virtual computing resource dispatching method based on edge cooperation
CN111262944A (en) * 2020-01-20 2020-06-09 北京大学 Method and system for hierarchical task offloading in heterogeneous mobile edge computing network
CN111641681A (en) * 2020-05-11 2020-09-08 国家电网有限公司 Internet of things service unloading decision method based on edge calculation and deep reinforcement learning
CN111756812A (en) * 2020-05-29 2020-10-09 华南理工大学 Energy consumption perception edge cloud cooperation dynamic unloading scheduling method

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109510869A (en) * 2018-11-22 2019-03-22 北京信息科技大学 A kind of Internet of Things service dynamic offloading method and device based on edge calculations
CN109819046A (en) * 2019-02-26 2019-05-28 重庆邮电大学 A kind of Internet of Things virtual computing resource dispatching method based on edge cooperation
CN109788069A (en) * 2019-02-27 2019-05-21 电子科技大学 Calculating discharging method based on mobile edge calculations in Internet of Things
CN111262944A (en) * 2020-01-20 2020-06-09 北京大学 Method and system for hierarchical task offloading in heterogeneous mobile edge computing network
CN111641681A (en) * 2020-05-11 2020-09-08 国家电网有限公司 Internet of things service unloading decision method based on edge calculation and deep reinforcement learning
CN111756812A (en) * 2020-05-29 2020-10-09 华南理工大学 Energy consumption perception edge cloud cooperation dynamic unloading scheduling method

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114363847A (en) * 2021-02-08 2022-04-15 西北工业大学 Task unloading method, device and system
CN114760308A (en) * 2022-04-01 2022-07-15 中国科学技术大学 Edge calculation unloading method and device
CN114760308B (en) * 2022-04-01 2023-03-24 中国科学技术大学 Edge calculation unloading method and device
CN117176618A (en) * 2023-11-03 2023-12-05 中国西安卫星测控中心 Performance evaluation method for network data exchange software system
CN117176618B (en) * 2023-11-03 2024-01-09 中国西安卫星测控中心 Performance evaluation method for network data exchange software system
CN118295794A (en) * 2024-02-26 2024-07-05 北京中电飞华通信有限公司 Computing resource configuration method and device of power system network and electronic equipment

Similar Documents

Publication Publication Date Title
CN112312299A (en) Service unloading method, device and system
CN111093203B (en) Service function chain low-cost intelligent deployment method based on environment perception
CN112598150B (en) Method for improving fire detection effect based on federal learning in intelligent power plant
CN111556461A (en) Vehicle-mounted edge network task distribution and unloading method based on deep Q network
CN111641681A (en) Internet of things service unloading decision method based on edge calculation and deep reinforcement learning
CN114340016B (en) Power grid edge calculation unloading distribution method and system
CN113037877B (en) Optimization method for time-space data and resource scheduling under cloud edge architecture
CN111813539B (en) Priority and collaboration-based edge computing resource allocation method
CN111176784A (en) Virtual machine integration method based on extreme learning machine and ant colony system
Hu et al. Dynamic task offloading in MEC-enabled IoT networks: A hybrid DDPG-D3QN approach
CN115408072A (en) Rapid adaptation model construction method based on deep reinforcement learning and related device
CN117707795B (en) Graph-based model partitioning side collaborative reasoning method and system
Li Optimization of task offloading problem based on simulated annealing algorithm in MEC
CN111158893B (en) Task unloading method, system, equipment and medium applied to fog computing network
CN118301002A (en) Edge DPU computing force collaborative optimization unloading method for demand self-adaptive prediction
Kim et al. Partition placement and resource allocation for multiple DNN-based applications in heterogeneous IoT environments
CN117858109A (en) User association, task unloading and resource allocation optimization method based on digital twin
CN117369964A (en) Task processing method and related device of edge computing system
CN116302569A (en) Resource partition intelligent scheduling method based on user request information
CN116684422A (en) Efficient configuration method and system for multidimensional edge resources under 5G power MEC
CN113157344B (en) DRL-based energy consumption perception task unloading method in mobile edge computing environment
CN114385359B (en) Cloud edge task time sequence cooperation method for Internet of things
CN115499876A (en) Computing unloading strategy based on DQN algorithm under MSDE scene
CN111784029A (en) Fog node resource allocation method
Xu et al. Joint Optimization of Task Offloading and Resource Allocation for Edge Video Analytics

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20210202