CN114661466A - Task unloading method for intelligent workflow application in edge computing environment - Google Patents

Abstract

The invention discloses a task unloading method for intelligent workflow application in an edge computing environment, which comprises the steps of firstly constructing an edge environment model aiming at an edge system, and depicting a workflow to be executed by each terminal into a Directed Acyclic Graph (DAG); then constructing a task unloading model for the intelligent workflow task to be executed by the terminal, and serializing DAG according to an average execution time calculation method based on topological sorting; and then defining a maximum user experience (QoE) problem according to the optimization target of the terminal, wherein the user experience is determined by task execution delay and model precision used for executing the intelligent task. And finally, solving the optimization problem of DAG task unloading based on multi-agent reinforcement learning, and taking the output result of the network as an unloading decision of the DAG sub-task. The method and the system consider terminal unloading rational decision and resource competition among multiple terminals, and greatly improve the user experience of the terminal for executing the intelligent workflow task.

Description

Task unloading method for intelligent workflow application in edge computing environment

Technical Field

The invention belongs to the field of edge intelligence and the field of deep learning in computer technology, and particularly relates to a task unloading method for realizing intelligent workflow application in a scene of executing the intelligent workflow application in an edge computing environment.

Background

In recent years, the rapid development of new computing and communication technologies has driven the constant emergence of innovative mobile applications and services, such as augmented reality, virtual reality, face recognition, and autonomous driving. These applications are usually implemented by deep learning, first training a deep neural network offline through a large amount of data, and then using the neural network as the computation of the intelligent application. However, the deep learning method provides effective computation for the intelligent application, and simultaneously gives the intelligent application extremely high computation demand and storage demand, so that it is increasingly difficult for the terminal to meet the resource demand of the intelligent application. In addition, mobile applications are rapidly growing, and while providing better services to people, these applications are becoming more and more complex. Accordingly, workflow techniques are widely used in mobile applications in order to better execute complex smart applications. Workflow (workflow) technology organizes a set of subtasks to complete some complex task flow. With this technique, complex intelligent applications can often be decomposed into several related subtasks, and furthermore, the dependency relationships between the subtasks define their execution order and the requirements of data transfer. More and more intelligent applications improve the execution efficiency in this way, for example, autopilot applications, which provide the possibility of flexible execution of applications by breaking a complex autopilot application into finer-grained modules. Similar examples are face recognition applications, object and recognition applications, gesture recognition applications, etc. developed by OpenCV.

In summary, the workflow of intelligent application has the characteristics of dynamics and complexity, and the intelligent subtasks therein exhibit the characteristics of intensive computation, intensive data and sensitive delay, so that the tasks executed at the terminal with weak computing power cannot be realized. The cloud computing mode selects to transmit the applications to the cloud end for running so as to save the electric quantity of the mobile terminal and meet higher computing requirements. However, with the rapid spread of smart terminals, it is estimated that 850ZB data will be generated on the terminals in 2021, and transmitting these data to the cloud for processing causes huge network pressure and communication delay, which is unacceptable for delay-sensitive applications such as autopilot. Moving Edge Computing (MEC) is gaining increasing attention as a new computational paradigm. The MEC relieves the pressure of the network and introduces lower communication delay to the execution of the application by deploying the server to the edge end closer to the terminal. However, considering economic factors, people usually cannot provide computing power and storage resources of the cloud end at the edge end, so that it is difficult for the edge end to deploy all required services of the user, and the transmission of all applications to the edge end results in too high load at the edge end to provide satisfactory services for the user, and how to perform tasks with high computing resource requirements and low response time requirements is a main problem to be solved by the edge computing.

To solve these problems, task offloading has become a research hotspot with the aim of optimizing the execution of applications to meet the execution of high-demand tasks under limited resource constraints. The advent of edge computing has enabled people to place these applications on edge servers for execution, namely task offloading (task offloading). In a task unloading mode in the edge computing, a user transmits the whole task or a part of the task to an edge server for execution, and after the execution of the edge end is finished, the operation result of the application is sent to the terminal, so that the user obtains lower time delay and saves the energy consumption of the terminal. However, considering the limited factors of the edge end providing services, how to select the position where the task is performed to obtain the highest user experience is also a problem to be solved.

This problem has been extensively studied as edge calculations have evolved due to the importance of task offloading, however, conventional methods still have many deficiencies. From the viewpoint of unloading of the workflow, the traditional method lacks the problem of multi-model selection of the subtasks in the workflow, and the requirements and experience of the user are related to the delay of task execution and the accuracy of the model, so that the traditional method causes the unloaded service to be difficult to match the requirements of the user; secondly, the traditional method ignores the situation that a plurality of users compete for the same resource, and often causes long queuing time caused by service preemption when different users have the same service requirements; finally, the traditional method is difficult to solve the workflow unloading problem of the complex edge environment with multiple edge ends and multiple terminals in the edge computing environment, which often results in high algorithm complexity and difficulty in obtaining the optimal solution.

In summary, the existing task offloading method still has great limitation when applied to the scene of the intelligent workflow in the edge computing environment, and cannot meet the running requirements of low time delay and high precision of the edge intelligent application, and the invention is generated thereby.

Disclosure of Invention

The invention aims to provide a task unloading method for intelligent workflow application in an edge computing environment, which is based on multi-agent reinforcement learning and a workflow task segmentation method and can solve the problems of low user experience, high algorithm complexity and the like of the existing task unloading method applied to the intelligent workflow task in the edge computing environment in the prior art. The method carries out priority sequencing on the workflow so as to reduce the complexity of the workflow, simultaneously describes a multi-model processing method of intelligent subtasks in the workflow so as to meet the user requirements, and finally constructs a workflow task unloading model based on multi-agent reinforcement learning to solve the task unloading problem, thereby improving the user experience brought by application execution.

In order to achieve the purpose, the invention provides the following technical scheme:

the task unloading method for the intelligent workflow application in the edge computing environment comprises the following steps:

step 1, modeling an environment model and a task of an edge system, wherein the system comprises a plurality of edge terminals and a plurality of terminals, a plurality of services are provided for the terminals to select unloading execution on each edge terminal, and each terminal has an intelligent workflow of the terminal and needs to carry out unloading execution; constructing intelligent workflow tasks needing to be unloaded on a terminal into a Directed Acyclic Graph (DAG) model, wherein in the DAG model, points in the graph represent a set of intelligent subtasks forming the workflow, and edges in the graph represent a dependency relationship among the subtasks;

step 2, based on the DAG model provided in the step 1, adopting an algorithm based on topological sorting to obtain the average execution time of each subtask, and serializing the DAG according to the sequence, thereby determining the execution sequence of the subtasks in the workflow;

step 3, modeling a task unloading model of the workflow in the edge environment based on the environment model and the task model constructed in the steps 1 and 2, namely, determining the execution position of each subtask in the DAG, so that an unloading decision is constructed into an integer planning problem; according to an optimization target of a terminal, describing a problem into an optimization problem which maximizes user experience QoE, wherein the QoE comprises model precision used when an intelligent subtask is executed and total DAG execution time;

step 4, designing a limited breadth first search algorithm to solve DAG execution time delay based on the optimization target constructed in the step 3, and finally obtaining the total QoE obtained by the unloading scheme;

and 5, designing a task unloading algorithm based on multi-agent reinforcement learning to solve the optimization problem by combining with the DAG unloading delay solving method in the step 4, obtaining a stable inference model through training a deep neural network, taking the output of the inference model as an unloading strategy executed by a workflow, and applying the unloading strategy to improve the QoE.

Further, the step 1 comprises the following sub-steps:

step 101, constructing a service model included in the edge system, and defining S as a service set provided by the system for w services provided by the edge system, wherein S is_wRepresenting the w-th service, the edge system is executed by using u machine learning models with different precision and time for each intelligent service and using MD_w，uIndicates execution of S_wThe u-th model available;

102, constructing an edge end model contained in an edge system, and aiming at the edge systemE edge terminals included, using ES_eRepresents the e-th edge, each edge providing a certain number of services, using S^eRepresenting a set of services provided by the edge terminal, S_wFor the w-th service, the number of MD is defined^e _wDenotes S_wAt ES_eThe model of the corresponding deployment; finally, use

Representing marginal end

And

the data transmission rate therebetween;

103, modeling a workflow model to be executed by the terminal in the edge system, and using N for i terminals included in the edge system_iRepresenting the ith terminal, each terminal comprises a workflow task needing to be unloaded, describing the workflow task into a DAG model, and using G_i＝{Vi，EG_iDenotes the DAG model on the ith terminal; wherein, V is used_i＝{T_i，1，T_i，2....T_i，jDenotes the set of nodes in the DAG, i.e. the set of subtasks in the workflow, T_i，jDenotes the jth subtask, idx, of the ith terminal_i，jRepresents T_i，jThe corresponding service sequence number.

Further, the step 2 comprises the following sub-steps:

step 201: for each subtask in the DAG, the computation load requirement C executed by the subtask is obtained_i，jTerminal calculation force f_vmEdge-end average calculated force f_ueCalculating the local execution delay of the subtask by using the following formula

And execution latency at the edge end

Step 202: obtaining, for each subtask in the DAG, an upload data volume requirement to be executed by each subtask

Download data volume requirement

Average data transmission rate R of terminal and edge terminal_i，jCalculating the upload delay of the subtask using the following formula

And download latency

Step 203: based on the time summation obtained in step 201 and step 202

And (3) representing the average execution time delay of the task, and applying the following rules to the subtasks in the DAG according to the time to obtain the priority:

wherein T is_i，k∈child(T_i，j) Represents T_i，kIs T_i，jThe predecessor task of (1) needs to precede T_i，jAnd executing, so that the priority of each subtask is obtained by applying topological sorting, and a sorted task sequence is obtained.

Further, the step 3 comprises the following sub-steps:

step 301, the unloading decision model for the workflow is described as follows: for T_i，jI sub-task of the represented i terminal, off^e _i，jIndicating that the task is unloaded to the e-th edge end;

step 302, describing the user experience QoE as follows: the QoE comprehensively considers the task execution delay and the adopted model precision, and the execution delay part comprises the following parts: for task T_i，jUsing t^queue _i，jThe queuing time before the task is executed is represented, and the queuing time is difficult to directly calculate due to the consideration of the resource preemption problem on the edge end; using t^exe _i，jRepresenting the execution delay of the task, determined by the selected task execution model, for two connected nodes in the DAG,

representing data transmission delay generated when two subtasks with dependency relationship are unloaded to different edge terminals; in the model accuracy part adopted, q is used^e _i，jRepresenting model accuracy weighted by user demand;

step 303, depicting the optimization problem of maximizing QoE, and comprehensively considering the effects of delay and model accuracy, where the optimization objective is expressed as:

s.t.z_i，j，k∈S^e

wherein the optimization objective first item represents for workflow G_iThe sum of the execution, queuing and transmission time of the generated tasks, considering that the limited number of services on the edge terminal are occupied by multiple users and multiple loads, the queuing time is difficult to directly calculate, and T is used_i ^finishRepresenting an execution delay; the second item of the optimization target is the sum of precision weighting of the task execution usage model, the sum of the two items is the final QoE, and the target is the maximum QoE; the constraint representing ES only at the edge^eThereon is deployed with T_i，jThe corresponding model can select T_i，jOff-loading to ES^e。

Further, the step 4 comprises the following sub-steps:

step 401, initialize a data structure including a priority queue PQ₁Taking the starting time of the task as the priority for the executable task queue; priority queue PQ₂Taking the task ending time as a task queue for the executing task queue; table map₁Representing a set of tasks that are not executable due to resource constraints, list L₁Representing an edge terminal service resource quantity list; initializing DAG, edge end service resource quantity list and unloading scheme as input, and enqueuing PQ to DAG start node₁；

Step 402, determine executable task queue PQ₁If the value is null, entering step 403 if the value is not null, or entering step 404 if the value is not null;

step 403, dequeuing the tasks in sequence, determining whether there is a resource at the edge, enqueuing if there is a resource, and executing queue PQ₂And the number of corresponding edge resources is reduced, and go to step 403; otherwise, adding the non-executable task list map₁In the process, waiting for the edge terminal to release the resource, and proceeding to step 404;

step 404, enqueue PQ being executed₂Dequeue task, on behalf of execution of task, enqueue PQ the dequeue task's successor in DAG₁Meanwhile, releasing the resources corresponding to the edge terminal, and proceeding to step 405;

step 405, retrieving map when releasing edge resources₁Whether or not to wrap inIncluding the task, if the task is included, the task is located in map₁In-team on-the-fly queue PQ₂Step 406 is entered;

step 406, determining whether all the tasks in the DAG have been executed, and if so, then PQ is executed₂The completion time of the last task dequeued is taken as the total execution delay of the offload scenario and exits, otherwise step 402 is entered.

Further, the step 5 comprises the following sub-steps:

step 501, constructing a state element of reinforcement learning based on the environment information constructed in step 1, namely, the edge server information, the network environment information, the service deployment information in the environment, and the terminal intelligent workflow task information, wherein S ═ S { (S) }₁，s₂…s_nRepresenting the state information received by the ith terminal;

step 502, based on the task unloading decision in step 3, using the unloading decision of the terminal task to form a reinforcement learning action element, wherein a is a set of all terminal unloading decisions,

represents the set of offload decisions for the serialized DAG on the ith terminal, S '═ S'₁，s′₂…s′_nRepresents new environment information that will be formed by executing a under the condition that the environment is S;

step 503, based on step 3, using the optimized QoE to form the reward elements of reinforcement learning, wherein RD ═ RD₁，rd₂，...，rd_nDenotes the reward obtained after performing the action in step 502;

step 504, constructing a multi-agent reinforcement learning model, wherein the multi-agent reinforcement learning model comprises two parts, namely an operator network and a critic network; in the operator network part, for each terminal, initializing its deep learning model pi_iWherein each DNN is represented by s_iAs an input, the initial weight is w_iThe DNN is composed of LSTM to embody the task priority characteristic of the serialized DAG; at the critical network part, for each terminal, DNN is initialized to a value of S,a } as an input;

step 505, initializing an experience pool to record a quadruple { state, action, reward, next state }, updating by using a round updating mode, entering step 506 if the number of rounds is less than a preset maximum training round number, and entering step 510 if the number of rounds is not less than the preset maximum training round number;

step 506, for the operator network on each terminal, using the local information s_iAs input, selecting action by using epsilon-greedy exploration strategy in action selection process to finally obtain action

Performing an action

To obtain new state s'_iAnd earn the reward rd_iRecord the quadruple as

Putting the obtained product into an experience pool, and entering a step 507;

step 507, for the critical network on each terminal, sampling a random minipatch sample from the experience pool, taking global information { S, A } as input, and recording the output of the target critical network as Q_iIndicating the output of the ith terminal critical network, and proceeding to step 508;

step 508, updating the main operator network and iteratively updating the target operator network of each terminal by using the policy gradient method, and proceeding to step 509:

step 509, update the critic network and iteratively update the target critic network according to a manner that minimizes the following loss, and proceed to step 505:

step 510, save the actor model, and then use the output of the actor as an unload decision.

Compared with the prior art, the invention has the following advantages and beneficial effects:

(1) according to the method, the intelligent workflow application is deployed in the edge computing environment, so that the division of the workflow application is realized, and a foundation is provided for a fine-grained task unloading method; the problem is reduced by adopting the mode of carrying out priority serialization on the workflow and multi-agent reinforcement learning, the complexity of the algorithm is effectively reduced, and the practicability is high, so that the method can be suitable for large-scale task environments.

(2) The invention fully considers the accuracy and time requirements of the user on the execution of the subtasks in the workflow, thereby being better suitable for the actual requirements of different intelligent workflow tasks.

(3) Because the invention is based on multi-agent reinforcement learning, the invention can effectively adapt to the dynamic and changeable conditions of the environment in the edge computing environment, effectively utilize the resources of the edge terminal and the terminal, reduce the task execution time and improve the model precision used by the intelligent subtask execution.

(4) The method of the invention considers the rational decision of terminal unloading and the resource competition among multiple terminals, and can greatly improve the user experience of the terminal for executing the intelligent workflow task.

Drawings

FIG. 1 is a schematic diagram of an edge computing execution framework of the intelligent workflow-oriented application of the present invention;

FIG. 2 is a flow diagram illustrating the execution of an intelligent workflow application in an edge computing environment according to the present invention;

FIG. 3 is a diagram of an agent model architecture for multi-agent reinforcement learning in an edge computing environment of the present invention;

FIG. 4 is a flow chart of reinforcement learning training of the present invention.

Detailed Description

The technical solutions provided by the present invention will be described in detail below with reference to specific examples, and it should be understood that the following specific embodiments are only illustrative of the present invention and are not intended to limit the scope of the present invention.

Fig. 1 shows a schematic diagram of an edge computation execution framework for an intelligent workflow application, which includes an edge layer and a terminal layer. In the terminal layer, a plurality of intelligent terminals are included, each terminal can generate an intelligent workflow task according to the requirement, however, the terminal cannot meet the delay requirement of application execution because the terminal is not powerful and the intelligent workflow task often reflects the characteristics of sensitive delay and intensive calculation. Therefore, a plurality of edge servers with certain calculation power are deployed at the edge layer, and machine learning models required by intelligent application execution are deployed in the edge servers for the terminal to select. By adopting the end-edge cooperative architecture, the terminal can unload a part of tasks to the edge layer for execution in a network transmission mode, and the edge end returns the result to the terminal after the execution of the tasks, so that the experience of application execution is improved.

The method comprises the steps of firstly, constructing an edge environment model aiming at an edge system, wherein the edge environment model comprises an intelligent service model provided by the system, an environment resource model composed of an edge terminal and a terminal, and a network environment model, and describing a workflow required to be executed by each terminal into a Directed Acyclic Graph (DAG). And then constructing a task unloading model for the intelligent workflow tasks to be executed by the terminal, serializing the DAG according to an average execution time calculation method based on topological sorting, wherein the unloading problem is that the task execution position is determined for each task in the sequence. Then, a maximum user experience (QoE) problem is defined according to the optimization objective of the terminal, wherein the user experience is determined by task execution delay and model precision used for executing the intelligent task. And finally, solving the optimization problem of DAG task unloading based on multi-agent reinforcement learning, obtaining a stable inference network through training of the deep neural network, and taking the output result of the network as an unloading decision of the DAG sub-task.

As shown in fig. 2, the invention applies a fine-grained workflow execution mode to split a complex intelligent task into a plurality of intelligent subtasks, and combines the task unloading idea to unload a part of subtasks to the edge end for execution, thereby effectively improving the parallelism of the subtasks execution while utilizing the strong computing resources of the edge end and the high-precision machine learning model, thereby reducing the task execution time, improving the model precision adopted by the intelligent task execution, and finally improving the user experience.

The specific steps executed by the intelligent workflow comprise:

step 1: modeling terminal-generated workflows into DAGs

The method comprises the steps of modeling an environment model and workflow tasks of an edge system, wherein the system environment comprises a plurality of edge terminals and a plurality of terminals, each edge terminal provides a plurality of services for the terminals to select to unload and execute, and each terminal has an intelligent workflow of the terminal and needs to unload and execute. And constructing the intelligent workflow tasks needing to be unloaded on the terminal into a Directed Acyclic Graph (DAG) model, wherein in the DAG model, points in the graph represent a set of intelligent subtasks forming the workflow, and edges in the graph represent the dependency relationship among the subtasks.

The specific content of the step is as follows:

step 101, constructing a service model included in the edge system, and defining S as a service set provided by the system for w services provided by the edge system, wherein S is_WRepresenting the w-th service, the edge system can be executed using u precision, time-diverse machine learning models, using MD, for each intelligent service_w，uIndicates execution of S_wThe u-th model available.

102, constructing edge end models contained in the edge system, and using ES for e edge ends contained in the edge system_eIndicating the e-th edge, each edge providing a certain number of services, using S^eRepresenting a set of services provided by the edge terminal, S_wFor the w-th service provision quantity, for this intelligent service, an MD is defined^e _wDenotes S_wAt ES_eThe model of the corresponding deployment. Finally, use

Representing marginal end

And with

The data transmission rate therebetween.

103, modeling a workflow model to be executed by the terminal in the edge system, and using N for i terminals included in the edge system_iRepresenting the ith terminal, each terminal comprises a workflow task needing to be unloaded, describing the workflow task into a DAG model, and using G_i＝{V_i，EG_iDenotes a DAG model at the ith terminal. Wherein V is used_i＝{T_i，1，T_i，2....T_i，jDenotes the set of nodes in the DAG, i.e. the set of subtasks in the workflow, T_i，jDenotes the jth sub-task, idx, of the ith terminal_i，jRepresents T_i，jA corresponding service sequence number; using EG_i＝{eg_i，1，eg_i，2....eg_i，lDenotes the set of edges in the DAG, i.e. the dependencies of the sub-tasks in the workflow, where

Representing two subtasks

There is a dependency relationship between them.

And 2, step: and calculating task execution priority by using a topological sorting method based on average execution time, and sorting the subtasks to obtain a serialized task queue. And (2) on the basis of the DAG model proposed in the step 1, adopting the topological ordering-based algorithm to obtain the average execution time of each subtask for the complex DAG model, and serializing the DAG according to the sequence, thereby determining the execution sequence of the subtasks in the workflow.

The specific content of the step is as follows:

step 201: for each subtask in the DAG, the computation load requirement C executed by the subtask is obtained_i，jThe terminal has large calculation powerSmall f_vmEdge-end average calculated force f_ueUsing the following formula, thereby calculating the execution time delay of the subtask locally

And execution latency at the edge end

Step 202: obtaining, for each subtask in the DAG, an upload data volume requirement to be executed by each subtask

Download data volume requirement

Average data transmission rate R of terminal and edge terminal_i，jThe following formula is used, whereby the upload delay of the subtask is calculated

And download latency

Step 203:based on the time summation obtained in step 201 and step 202

And (3) representing the average execution time delay of the tasks, and applying the following rules to the subtasks in the DAG according to the time to obtain the priority:

wherein T is_i，k∈child(T_i，j) Represents T_i，kIs T_i，jThe predecessor task of (1) needs to precede T_i，jAnd executing, so that the topological sorting is applied to obtain the priority of each subtask, and a sorted task sequence is obtained.

And 3, step 3: based on the environment model and the task model constructed in the step 1 and the step 2, a task unloading model of the workflow in the edge environment is modeled, namely the execution position of each subtask in the DAG needs to be decided, so that an unloading decision is constructed into an integer programming problem. According to the optimization objective of the terminal, the problem is characterized as an optimization problem for maximizing the user experience (QoE), wherein the QoE comprises the model precision used in the execution of the intelligent subtask and the total execution time of DAG.

The specific content of the step is as follows:

step 301, the unloading decision model for the workflow is described as follows: for T_i，jJ sub-tasks of the i terminal indicated, off^e _i，jIndicating that the task is offloaded to the e-th edge.

Step 302, describing the user experience QoE as follows: the QoE comprehensively considers the task execution delay and the adopted model precision, and the execution delay part comprises the following parts: for task T_i，jUsing t^queue _i，jThe queuing time before the task is executed is represented, and the queuing time is difficult to directly calculate due to the consideration of the resource preemption problem on the edge end; using t^exe _i，jRepresenting the execution delay of the task, determined by the selected task execution model, for two connected DAGsThe node(s) of (a) is (are),

representing the data transmission delay generated when two subtasks with dependency relationship are unloaded to different edge terminals. In the model accuracy part adopted, q is used^e _i，jRepresenting the accuracy of the model weighted on the user demand.

Step 303, the optimization problem of maximizing QoE is characterized, and in order to comprehensively consider the effects of delay and model accuracy, the optimization objective is expressed as:

s.t.z_i，j，k∈S^e

wherein the optimization objective first item represents for workflow G_iThe sum of the execution, queuing and transmission time of the generated tasks, considering that the limited number of services on the edge terminal are occupied by multiple users and multiple loads, the queuing time is difficult to directly calculate, and T is used_i ^finishRepresenting an execution delay; the second item of the optimization target is the sum of precision weighting of the task execution usage model, and the sum of the two items is the final QoE, wherein the target is to maximize the QoE. The constraint representing ES only at the edge^eThereon is deployed with T_i，jThe corresponding model can select T_i，jOff-loading to ES^e。

And inputting the task queue into a decision maker to obtain a task unloading decision of each subtask, wherein the task unloading decision can be executed at an edge terminal or a terminal.

And 4, step 4: based on the optimization target constructed in the step 3, DAG execution delay is difficult to directly solve due to the resource preemption problem of multiple terminals of the edge system, a limited breadth first search algorithm is designed to solve the DAG execution delay, and finally the total QoE obtained by the unloading scheme is obtained; the resource-limited breadth-first search DAG task execution delay algorithm comprises the following steps:

step 401, initialize a data structure, whichIncluding a priority queue PQ₁Taking the starting time of the task as the priority for the executable task queue; priority queue PQ₂Taking the task ending time as a task queue for the executing task queue; table map₁Representing a set of tasks that are not executable due to resource constraints, list L₁And representing the list of the number of the edge terminal service resources. Initializing DAG, edge end service resource quantity list and unloading scheme as input, and enqueuing PQ to DAG start node_i。

Step 402, determine executable task queue PQ₁If it is empty, go to step 403 if it is not empty, otherwise go to step 404.

Step 403, dequeuing the tasks in sequence, determining whether there is a resource at the edge, and enqueuing if there is a resource and executing a queue PQ₂And the number of corresponding edge resources is reduced, and go to step 403; otherwise, adding the non-executable task table map₁Wait for the edge to release the resource, and proceed to step 404.

Step 404, enqueue PQ being executed₂Dequeue task, on behalf of execution of task, enqueue PQ the dequeue task's successor in DAG₁Meanwhile, the corresponding resources of the edge terminal are released, and step 405 is proceeded.

Step 405, retrieving map when releasing edge resources₁Whether the task is contained in the table, if so, the task is in the map₁In-team on-the-fly queue PQ₂Step 406 is entered.

Step 406, determining whether all the tasks in the DAG have been executed, and if so, then PQ is executed₂And the completion time of the last task of the intermediate dequeue is used as the total execution delay of the unloading scheme, and the process is exited. Otherwise step 402 is entered.

And 5, designing a task unloading algorithm based on multi-agent reinforcement learning to solve the optimization problem by combining with the DAG unloading delay solving method in the step 4, obtaining a stable inference model through training a deep neural network, taking the output of the inference model as an unloading strategy executed by a workflow, and applying the unloading strategy to improve the QoE.

In the step, machine learning models of the terminal and the edge terminal are applied to execute the intelligent subtasks, intermediate and final results are transmitted by using a network, and finally the terminal receives a workflow execution result.

As shown in FIG. 3, the invention combines the idea of multi-agent reinforcement learning to provide a task unloading method for intelligent workflow application in edge computing environment. Corresponding to the edge computing environment, the multi-agent reinforcement learning architecture comprises a plurality of agents formed by terminals, each agent is composed of two parts of networks, namely an actor network used for selecting output actions, namely making unloading decisions, and the other part of the agent network is a critic network used for evaluating the actions output by the actor network.

Specifically, in the multi-agent reinforcement learning architecture, a centralized training and distributed inference mode is adopted. Wherein, the operator network basic structure of each agent is composed of LSTM to embody the sequential characteristic of the serialized DAG. The actor network uses the edge environment information and local workflow information which needs to be executed locally as input, and finally outputs the unloading decision of the subtask sequence through forward propagation. The criticc network of each agent is composed of 3 layers of full connection layers, global edge server information, workflow task information of all terminals and task unloading decision information of all terminals are used as input, and finally Q values are output to evaluate unloading decisions.

And the centralized global information and the Q value calculated by the critic network are used for training the actor network and the critic network of the intelligent agent by utilizing the reverse transmission of the neural network. The operator network uses a policy gradient training mode, the critic network uses a Q-learning training mode, and finally the two networks converge. After network convergence, each intelligent terminal can use an actor network and local information to carry out distributed inference so as to obtain an unloading decision maximizing user experience, centralized global information is not needed, and the effect of privacy protection is achieved.

As shown in fig. 4, in the multi-agent reinforcement learning method for solving the task offloading problem in the edge computing environment according to the present invention, in the model, the agents are embodied as a plurality of intelligent terminals in the edge computing environment, each terminal needs to select an edge server for task offloading, and a deep neural network obtained through reinforcement learning training is used to obtain an optimal offloading decision. The multi-agent reinforcement learning method specifically comprises the following steps:

step 501, constructing a state element of reinforcement learning based on the environment information constructed in step 1, namely, the edge server information, the network environment information, the service deployment information in the environment, and the terminal intelligent workflow task information, wherein S ═ S { (S) }₁，s₂…s_nAnd represents the status information received by the ith terminal.

Step 502, based on the task offloading decision of step 3, using the terminal task offloading decision to form a reinforcement learning action element, wherein a is a set of all terminal offloading decisions,

represents the set of offload decisions for the serialized DAG on the ith terminal, S '═ S'₁，s′₂…s′_nDenotes execution under the condition that the environment is S

New context information will be formed.

Step 503, based on step 3, using the optimized QoE to form the reward elements of reinforcement learning, wherein RD ═ RD₁，rd₂，...，rd_nDenotes the prize earned after performing the action in step 502.

Step 504, a multi-agent reinforcement learning model is constructed, which comprises two parts, namely an actor network and a critic network. In the operator network part, for each terminal, initializing its deep learning model pi_iWherein each DNN is represented by s_iAs input, 256 neurons are included with an initial weight of w_iThe DNN is comprised of LSTMs to embody the task priority characteristics of the serialized DAG. In the critic network part, for each terminal, DNN is initialized with S, A as input and 256 neuronsAnd (4) forming.

Step 505, initializing an experience pool to record quadruple { state, action, reward, next state }, updating by using a round updating mode, setting the maximum number of training rounds to be 1000, if the number of rounds is less than 1000, entering step 506, otherwise, entering step 510.

Step 506, for the operator network on each terminal, using the local information s_iAs input, selecting an action by using an Ee-greedy exploration strategy in the action selection process to finally obtain the action

Performing an action

To obtain a new state s'_iAnd earn a reward rd_iRecord the quadruple as

Put into the experience pool and go to step 507.

Step 507, for the critical network on each terminal, sampling a random minipatch sample from the experience pool, taking global information { S, A } as input, and recording the output of the target critical network as Q_iAnd indicates the output of the ith terminal critical network, the process proceeds to step 508.

Step 508, updating the main operator network and iteratively updating the target operator network of each terminal by using the policy gradient method, and proceeding to step 509:

step 509, update the critic network and iteratively update the target critic network according to a manner that minimizes the following loss, and proceed to step 505:

step 510, save the actor model, and then use the output of the actor as an unload decision.

And finally, the terminal equipment takes the generated intelligent workflow task as the input of the operator network, obtains the output of the network as an unloading decision, and performs task scheduling and execution on the subtask according to the unloading decision.

The technical means disclosed in the invention scheme are not limited to the technical means disclosed in the above embodiments, but also include the technical scheme formed by any combination of the above technical features. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and such improvements and modifications are also considered to be within the scope of the present invention.

Claims (6)

1. The task unloading method for the intelligent workflow application in the edge computing environment is characterized by comprising the following steps of:

step 1, modeling an environment model and a task of an edge system, wherein the system comprises a plurality of edge terminals and a plurality of terminals, a plurality of services are provided for the terminals to select unloading execution on each edge terminal, and each terminal has an intelligent workflow of the terminal and needs to carry out unloading execution; constructing intelligent workflow tasks needing to be unloaded on a terminal into a Directed Acyclic Graph (DAG) model, wherein in the DAG model, points in the graph represent a set of intelligent subtasks forming the workflow, and edges in the graph represent a dependency relationship among the subtasks;

step 2, based on the DAG model provided in the step 1, adopting an algorithm based on topological sorting to obtain the average execution time of each subtask, and serializing the DAG according to the sequence, thereby determining the execution sequence of the subtasks in the workflow;

step 3, modeling a task unloading model of the workflow in the edge environment based on the environment model and the task model constructed in the steps 1 and 2, namely, determining the execution position of each subtask in the DAG, so that an unloading decision is constructed into an integer planning problem; according to an optimization target of a terminal, describing a problem into an optimization problem which maximizes user experience QoE, wherein the QoE comprises model precision used when an intelligent subtask is executed and total DAG execution time;

step 4, designing a limited breadth first search algorithm to solve DAG execution time delay based on the optimization target constructed in the step 3, and finally obtaining the total QoE obtained by the unloading scheme;

and 5, designing a task unloading algorithm based on multi-agent reinforcement learning to solve the optimization problem by combining with the DAG unloading delay solving method in the step 4, obtaining a stable inference model through training a deep neural network, taking the output of the inference model as an unloading strategy executed by a workflow, and applying the unloading strategy to improve the QoE.

2. The method for task offload for intelligent workflow-oriented applications in edge computing environment according to claim 1, wherein the step 1 comprises the following sub-steps:

step 101, constructing a service model included in the edge system, and defining S as a service set provided by the system for w services provided by the edge system, wherein S is_wRepresenting the w-th service, the edge system is executed by using u machine learning models with different precision and time for each intelligent service and MD_w,uIndicates performing S_wThe u-th model available;

102, constructing edge end models contained in the edge system, and using ES for e edge ends contained in the edge system_eIndicating the e-th edge, each edge providing a certain number of services, using S^eRepresenting a set of services provided by the edge terminal, S_wFor the w-th service, the number of MD is defined^e _wDenotes S_wAt ES_eThe model of the corresponding deployment; finally, use

Indicating marginal end

And

the data transmission rate therebetween;

103, modeling a workflow model to be executed by the terminal in the edge system, and using N for i terminals included in the edge system_iRepresenting the ith terminal, each terminal comprises a workflow task needing to be unloaded, describing the workflow task into a DAG model, and using G_i＝{V_i,EG_iDenotes a DAG model at the ith terminal; wherein V is used_i＝{T_i,1,T_i,2…T_i,jDenotes the set of nodes in the DAG, i.e. the set of subtasks in the workflow, T_i,jDenotes the jth sub-task, idx, of the ith terminal_i,jRepresents T_i,jThe corresponding service sequence number.

3. The method for task offloading of an intelligent workflow-oriented application in an edge computing environment according to claim 1, wherein the step 2 comprises the sub-steps of:

step 201: for each subtask in the DAG, the computation load requirement C executed by the subtask is obtained_i,jCalculating the strength of the terminal f_vmEdge average calculated force f_ueCalculating the local execution delay of the subtask using the following formula

And execution latency at the edge end

Step 202: obtaining, for each subtask in the DAG, an upload data volume requirement to be executed by each subtask

Download data volume requirement

Average data transmission rate R of terminal and edge terminal_i,jCalculating the upload delay of the subtask using the following formula

And download latency

Step 203: based on the time summation obtained in step 201 and step 202

And (3) representing the average execution time delay of the tasks, and applying the following rules to the subtasks in the DAG according to the time to obtain the priority:

wherein T is_i,k∈child(T_i,j) Represents T_i,kIs T_i,jThe predecessor task of (1) needs to precede T_i,jAnd executing, so that the priority of each subtask is obtained by applying topological sorting, and a sorted task sequence is obtained.

4. The method for task offload for intelligent workflow-oriented applications in edge computing environment according to claim 1, wherein the step 3 comprises the sub-steps of:

step 301, the unloading decision model for the workflow is described as follows: for T_i,jJ sub-task of the i terminal indicated, off^e _i,jIndicating that the task is unloaded to the e-th edge end;

step 302, describing the user experience QoE as follows: the QoE comprehensively considers the task execution delay and the adopted model precision, and the execution delay part comprises the following parts: for task T_i,jUsing t^queue _i,jThe queuing time before the task is executed is represented, and the queuing time is difficult to directly calculate due to the consideration of the resource preemption problem on the edge end; using t^exe _i,jRepresenting the execution delay of the task, determined by the selected task execution model, for two connected nodes in the DAG,

representing data transmission delay generated when two subtasks with dependency relationship are unloaded to different edge terminals; in the model accuracy part adopted, q is used^e _i,jRepresenting model accuracy weighted against user demand;

step 303, depicting the optimization problem of maximizing QoE, and comprehensively considering the effects of delay and model accuracy, where the optimization objective is expressed as:

max:

s.t.z_i,j,k∈S^e

wherein the optimization objective first item represents for workflow G_iExecution of, result inThe total time of executing, queuing and transmitting the tasks is taken into consideration that the queuing time is difficult to directly obtain by considering the limited number of services on the edge terminals occupied by multiple users and multiple loads, and T is used_i ^finishRepresenting an execution delay; the second item of the optimization target is the sum of precision weighting of the task execution usage model, the sum of the two items is the final QoE, and the target is the maximum QoE; the constraint representing ES only at the edge^eThereon is deployed with T_i,jThe corresponding model can select T_i,jUnloading to ES^e。

5. The method for task offload for intelligent workflow-oriented applications in edge computing environment according to claim 1, wherein the step 4 comprises the following sub-steps:

step 401, initialize a data structure including a priority queue PQ₁Taking the starting time of the task as the priority for the executable task queue; priority queue PQ₂Taking the task ending time as a task queue for the executing task queue; table map₁Representing a set of tasks that are not executable due to resource constraints, list L₁Representing an edge terminal service resource quantity list; initializing DAG, edge end service resource quantity list and unloading scheme as input, and enqueuing PQ to DAG start node₁；

Step 402, determine executable task queue PQ₁If the value is null, entering step 403 if the value is not null, or entering step 404 if the value is not null;

step 403, dequeuing the tasks in sequence, determining whether there is a resource at the edge, and enqueuing if there is a resource and executing a queue PQ₂And the number of corresponding edge resources is reduced, and go to step 403; otherwise, adding the non-executable task list map₁In the process, waiting for the edge terminal to release the resource, and proceeding to step 404;

step 404, enqueue PQ being executed₂Dequeue task, on behalf of execution of task, enqueue PQ the dequeue task's successor in DAG₁Meanwhile, releasing the resources corresponding to the edge terminal, and proceeding to step 405;

step 405, retrieving map when releasing edge resources₁Whether the task is contained in the list, if so, the task is stored in the map₁Merge queue PQ₂Step 406 is entered;

step 406, determining whether all tasks in the DAG have been executed, and if so, making the PQ₂The completion time of the last task dequeued is taken as the total execution delay of the offload scenario and exits, otherwise step 402 is entered.

6. The method for task offload for intelligent workflow-oriented applications in edge computing environment according to claim 1, wherein the step 5 comprises the sub-steps of:

step 501, constructing a state element of reinforcement learning based on the environment information constructed in step 1, namely, the edge server information, the network environment information, the service deployment information in the environment, and the terminal intelligent workflow task information, wherein S ═ S { (S) }₁,s₂…s_nRepresents the status information received by the ith terminal;

step 502, based on the task unloading decision in step 3, using the unloading decision of the terminal task to form a reinforcement learning action element, wherein a is a set of all terminal unloading decisions,

represents the set of offload decisions for the serialized DAG on the ith terminal, S '═ S'₁,s′₂…s′_nRepresents new environment information that will be formed by executing a under the condition that the environment is S;

step 503, based on step 3, using the optimization objective QoE to construct the reward element of reinforcement learning, where RD ═ is₁,rd₂,…,rd_nDenotes the reward obtained after performing the action in step 502;

step 504, constructing a multi-agent reinforcement learning model, wherein the multi-agent reinforcement learning model comprises two parts, namely an actor network and a critic network; in the actor network part, for each terminal, initializing the deep learning model pi thereof_iWherein each DNN is represented by s_iAs an input, the initial weight is w_iThe DNN is composed of LSTM to embody the task priority characteristic of the serialized DAG; at the critical network part, for each terminal, DNN is initialized with S, a as input;

step 505, initializing an experience pool to record a quadruple { state, action, reward, next state }, updating by using a round updating mode, entering step 506 if the number of rounds is less than a preset maximum training round number, and entering step 510 if the number of rounds is not less than the preset maximum training round number;

step 506, for the operator network on each terminal, using the local information s_iAs input, selecting an action by using an Ee-greedy exploration strategy in the action selection process to finally obtain the action

Performing an action

To obtain a new state s'_iAnd earn a reward rd_iRecord the quadruple as

Putting the obtained product into an experience pool, and entering a step 507;

step 507, for the critical network on each terminal, sampling a random minipatch sample from the experience pool, taking global information { S, A } as input, and recording the output of the target critical network as Q_iIndicating the output of the ith terminal critical network, and proceeding to step 508;

step 508, updating main operator network of each terminal and iteratively updating target operator network by using policy gradient method, and proceeding to step 509:

step 509, update the critic network and iteratively update the target critic network according to a manner that minimizes the following loss, and proceed to step 505:

step 510, save the actor model, and then use the output of the actor as an unload decision.

Images