CN117852621A - Module combination model-free computing offloading method and device in multi-environment MEC - Google Patents

Abstract

The application provides a model-free computing unloading method and device for module combination in a multi-environment MEC, and relates to the field of machine learning. In the method, a policy device acquires environment description information and current state description information of a target edge computing environment; and calling a pre-trained strategy model to process the environment description information and the state description information to obtain a task unloading strategy considering both the target edge computing environment and the state description information. Therefore, compared with the conventional strategy model, the task unloading strategy is generated only according to the current state description information, and the current environment description information is combined, so that the generated task unloading strategy takes the target edge computing environment and the states in the environment into consideration.

Description

Module combination model-free computing offloading method and device in multi-environment MEC

Technical Field

The present application relates to the field of machine learning, and more specifically, to a module-combined model-free computing offloading method and device in a multi-environment MEC.

Background Art

The surge in user equipment (UE) has led to the widespread popularity of mobile applications, such as some computationally intensive and latency-sensitive applications: mobile payments, online games, smart healthcare, and augmented reality. Most UEs are equipped with limited computing resources and energy budgets, which leads to a large gap between the capabilities of UEs and application requirements. Today, with the help of the rapid development of high-speed wireless communication technology, Mobile Edge Computing (MEC) is an effective way to alleviate this problem by offloading the tasks of UEs to nearby base stations (BS) with powerful edge computing resources.

One of the key issues of MEC is task offloading, which makes dynamic decisions (e.g., offloading tasks, transmission power) based on the time-varying MEC state (e.g., task requirements, energy budget, wireless conditions) to improve the computational efficiency of mobile applications. In order to make the best decision for computational offloading, traditional methods are mainly developed based on mathematical programming, which relies heavily on the reliability of the MEC system model. When the system model is unavailable or unreliable, heuristic search-based methods (e.g., genetic algorithms and particle swarm optimization) are methods that achieve near-optimal computational offloading without knowing the system model in advance. However, with the rapid development of wireless network speeds, these methods have gradually become difficult to find efficient offloading solutions in a short period of time. Therefore, when the MEC state and offloading decisions are of high dimensionality, it has become mainstream to use reinforcement learning or more commonly deep reinforcement learning to develop model-free computational offloading methods.

In practice, it is found that currently developed efficient computation offloading methods based on deep reinforcement learning in various scenarios, such as multi-intelligent user equipment MEC, multi-base station MEC, vehicle-assisted MEC, and satellite-integrated MEC, etc.; most of these methods only consider computation offloading in a single MEC environment with constant bandwidth, edge server (ES) capacity, task type, etc. However, these methods cannot adapt to various highly diverse MEC environments in real scenarios.

Summary of the invention

In order to overcome at least one of the deficiencies in the prior art, the present application provides a module combination model-free computing offloading method and device in a multi-environment MEC, specifically including:

In a first aspect, the present application provides a module combination model-free computing offloading method in a multi-environment MEC, the method comprising:

Obtain the environment description information and current status description information of the target edge computing environment;

A pre-trained policy model is called to process the environment description information and the state description information to obtain a task offloading policy that takes into account both the target edge computing environment and the state description information.

In combination with the optional implementation manner of the first aspect, the policy model includes a plurality of policy layers connected in series, each policy layer includes a plurality of sub-models independent of each other, and the calling of the pre-trained policy model processes the environment description information and the state description information to obtain a task offloading strategy that takes into account the target edge computing environment and the state description information, including:

Inputting the state embedding features of the state description information into the multiple strategy layers;

For any adjacent layers in the multiple strategy layers, the features output by each sub-model in the previous strategy layer are screened according to the environment embedding features of the environment description information and the state embedding features to determine the input features of each sub-model in the next strategy layer;

According to the output results of the multiple policy layers, a task offloading strategy that takes into account both the target edge computing environment and the state description information is determined.

In combination with an optional implementation manner of the first aspect, the output features of each sub-model in the previous strategy layer are screened according to the environment embedding features of the environment description information and the state embedding features to determine the input features of each sub-model in the next strategy layer, including:

Generate a weight vector for each sub-model in the next strategy layer according to the environment embedding feature of the environment description information and the state embedding feature;

The output features of each sub-model in the previous strategy layer are weighted according to the weight vector of each sub-model in the next strategy layer to obtain the input features of each sub-model in the next strategy layer.

In combination with an optional implementation manner of the first aspect, the strategy model further includes a plurality of weight layers connected in series, and the plurality of weight layers correspond one-to-one to parts of the plurality of strategy layers; generating a weight vector for each sub-model in the next strategy layer according to the environment embedding feature of the environment description information and the state embedding feature, comprises:

Acquire a fusion feature between the state embedding feature and the environment embedding feature;

Inputting the fused features into the multiple weight layers;

For any adjacent layers among the multiple weight layers, a weight vector output by a previous weight layer is multiplied by the fusion feature and then input into a next weight layer to obtain a weight vector of a strategy layer corresponding to the next weight layer.

In combination with the optional implementation manner of the first aspect, the acquiring a fusion feature between the state embedding feature and the environment embedding feature includes:

The state embedding feature and the environment embedding feature are multiplied element by element to obtain the fusion feature.

In combination with the optional implementation manner of the first aspect, the strategy model further includes a first encoder and a second encoder, and the method further includes:

Processing the state description information by the first encoder to obtain a state embedding feature of the state description information;

The environment description information is processed by the second encoder to obtain environment embedding features of the environment description information.

In combination with the optional implementation manner of the first aspect, the method further includes a training method for the policy model, and the training method includes:

Acquire multiple strategy models to be trained and an evaluation model to be trained for each of the strategy models to be trained, wherein the multiple strategy models to be trained correspond to different edge computing environments respectively;

For each strategy model to be trained, the strategy model to be trained interacts with the corresponding edge computing environment to obtain task offloading experience for the current state of the edge computing environment, and caches it in the experience pool;

After the task offloading experience collected by the experience pool meets a preset condition, sampling experience is sampled from the experience pool, and the multiple strategy models to be trained and their corresponding evaluation models to be trained are updated according to the sampling experience;

If the multiple strategy models to be trained and their corresponding evaluation models to be trained do not meet the convergence conditions, then return to each strategy model to be trained, interact with the corresponding edge computing environment through the strategy model to be trained, and obtain the task offloading experience for the current state of the edge computing environment, until the convergence conditions are met, and use the strategy model to be trained after this iteration as the pre-trained strategy model.

In combination with the optional implementation manner of the first aspect, the multiple strategy models to be trained and their corresponding evaluation models to be trained are updated according to the sampling experience, including:

According to the sampling experience, multiple strategy layers and multiple weight layers in each of the strategy models to be trained are updated alternately, and the evaluation model to be trained corresponding to each of the strategy models to be trained is updated.

In combination with an optional implementation manner of the first aspect, the task offloading experience includes environment description information of the edge computing environment corresponding to the strategy model to be trained.

In a second aspect, the present application also provides a module-combined model-free computing unloading device in a multi-environment MEC, the method comprising:

An information acquisition module is used to obtain the environment description information and current status description information of the target edge computing environment;

The strategy generation module is used to call a pre-trained strategy model to process the environment description information and the state description information to obtain a task offloading strategy that takes into account both the target edge computing environment and the state description information.

In conjunction with the optional implementation manner of the second aspect, the strategy model includes a plurality of strategy layers connected in series, each strategy layer includes a plurality of sub-models independent of each other, and the strategy generation module is further specifically used for:

Inputting the state embedding features of the state description information into the multiple strategy layers;

For any adjacent layers in the multiple strategy layers, the features output by each sub-model in the previous strategy layer are screened according to the environment embedding features of the environment description information and the state embedding features to determine the input features of each sub-model in the next strategy layer;

According to the output results of the multiple policy layers, a task offloading strategy that takes into account both the target edge computing environment and the state description information is determined.

In conjunction with the optional implementation manner of the second aspect, the strategy generation module is further specifically configured to:

Generate a weight vector for each sub-model in the next strategy layer according to the environment embedding feature of the environment description information and the state embedding feature;

The output features of each sub-model in the previous strategy layer are weighted according to the weight vector of each sub-model in the next strategy layer to obtain the input features of each sub-model in the next strategy layer.

In conjunction with the optional implementation manner of the second aspect, the strategy model further includes a plurality of weight layers connected in series, and the plurality of weight layers correspond one-to-one to parts of the plurality of strategy layers; and the strategy generation module is further specifically configured to:

Acquire a fusion feature between the state embedding feature and the environment embedding feature;

Inputting the fused features into the multiple weight layers;

For any adjacent layers among the multiple weight layers, a weight vector output by a previous weight layer is multiplied by the fusion feature and then input into a next weight layer to obtain a weight vector of a strategy layer corresponding to the next weight layer.

In conjunction with the optional implementation manner of the second aspect, the strategy generation module is further specifically configured to:

The state embedding feature and the environment embedding feature are multiplied element by element to obtain the fusion feature.

In conjunction with the optional implementation manner of the second aspect, the strategy model further includes a first encoder and a second encoder, and the strategy generation module is further used to:

Processing the state description information by the first encoder to obtain a state embedding feature of the state description information;

The environment description information is processed by the second encoder to obtain environment embedding features of the environment description information.

In combination with the optional implementation manner of the second aspect, the method further includes a training method for the policy model, and the device further includes:

An experience collection block is used to obtain multiple strategy models to be trained and evaluation models to be trained for each strategy model to be trained, wherein the multiple strategy models to be trained correspond to different edge computing environments respectively;

The experience collection block is also used for interacting with the corresponding edge computing environment through the strategy model to be trained for each strategy model to be trained, to obtain the task offloading experience for the current state of the edge computing environment, and cache it in the experience pool;

A model updating module, configured to sample sampled experience from the experience pool after the task offloading experience collected by the experience pool meets a preset condition, and update the multiple strategy models to be trained and their corresponding evaluation models to be trained according to the sampled experience;

The model update module is also used to return to each strategy model to be trained, interact with the corresponding edge computing environment through the strategy model to be trained, and obtain task offloading experience for the current state of the edge computing environment, if the multiple strategy models to be trained and their corresponding evaluation models to be trained do not meet the convergence conditions, until the convergence conditions are met, and the strategy model to be trained after this iteration is used as the pre-trained strategy model.

In conjunction with the optional implementation manner of the second aspect, the model updating module is further specifically configured to:

According to the sampling experience, multiple strategy layers and multiple weight layers in each of the strategy models to be trained are updated alternately, and the evaluation model to be trained corresponding to each of the strategy models to be trained is updated.

In combination with an optional implementation of the second aspect, the task offloading experience includes environment description information of the edge computing environment corresponding to the strategy model to be trained.

Compared with the prior art, this application has the following beneficial effects:

The present embodiment provides a module-combined model-free computing offloading method and device in a multi-environment MEC. In this method, the policy device obtains the environment description information of the target edge computing environment and the current state description information; calls a pre-trained policy model to process the environment description information and the state description information, and obtains a task offloading strategy that takes into account both the target edge computing environment and the state description information. In this way, compared to the conventional policy model that only generates a task offloading strategy based on the current state description information, the present application also combines the current environment description information, so that the generated task offloading strategy takes into account both the target edge computing environment and the state in the environment.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

FIG1 is a schematic diagram of a scenario provided by an embodiment of the present application;

FIG2 is a schematic diagram of a method flow chart provided in an embodiment of the present application;

FIG3 is a schematic diagram of a model structure provided in an embodiment of the present application;

FIG4 is a schematic diagram of the training principle provided in an embodiment of the present application;

FIG5 is a schematic diagram of the structure of a device provided in an embodiment of the present application;

FIG. 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

Icon: 101 - base station; 102 - edge server; 103 - user equipment; 201 - information acquisition module; 202 - policy generation module; 301 - memory; 302 - processor; 303 - communication unit; 304 - system bus.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

In the description of the present application, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance. In addition, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Based on the above statement, as introduced in the background technology, efficient deep reinforcement learning-based computation offloading methods are currently developed in various scenarios, such as multi-intelligent user device MEC, multi-base station MEC, vehicle-assisted MEC, and satellite-integrated MEC; most of these methods only consider computation offloading in a single MEC environment with constant bandwidth, edge server (ES) capacity, task type, etc. However, these methods cannot adapt to various highly diverse MEC environments in real scenarios. Specifically, there are two problems:

(1) Inefficient empirical exploration and learning, as learning a deep reinforcement learning-based offloading policy even in a single MEC environment may require a large amount of experience.

(2) Experience learning interference in training deep reinforcement learning-based offloading strategies, where the gradients of exploration experiences from different MEC environments may contradict each other, resulting in a significant performance degradation of computation offloading.

Based on the discovery of the above technical problems, the inventors have proposed the following technical solutions to solve or improve the above problems through creative work. It should be noted that the defects in the solutions in the above prior art are the results obtained by the inventors after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed in the embodiments of the present application for the above problems below should all be the contributions made by the inventors to the present application in the process of invention and creation, and should not be understood as technical contents known to those skilled in the art.

Since this embodiment involves reinforcement learning in the field of machine learning, in order to make the solution introduced next easier to understand, the professional concepts that may be involved are explained below.

Reinforcement learning is a machine learning method that aims to enable intelligent systems to interact with the environment to learn how to make optimal decisions. In reinforcement learning, the intelligent system is called an "agent", which observes the state of the environment and the reward signal and takes a series of actions to maximize the future cumulative rewards. It can be understood that the core idea of reinforcement learning is that the agent learns through a trial and error process. That is, the agent knows nothing about the environment at the beginning, and gradually learns which actions can obtain higher rewards by interacting with the environment and observing the results. The agent uses a function called a "policy" to decide which action should be taken in a given state. The strategy can be deterministic or probabilistic. In summary, the core components of reinforcement learning include:

Environment: The external environment with which the agent interacts, which can be a real physical environment or a virtual simulated environment.

State: The current observation of the environment, used to describe the characteristics of the environment.

Action: The action that the agent takes in a given state.

Reward: The signal that the agent gets from the environment based on its actions, used to evaluate how good the actions were.

Policy: The way the agent chooses actions based on the current state, which can be a deterministic mapping or a probabilistic distribution.

Value Function: A function that measures the long-term value of an agent in a state or state-action pair. It is used to evaluate the quality of a policy.

Model: An internal model of the environment that is used to predict the environment's state transitions and reward signals.

The goal of reinforcement learning is to optimize the strategy or value function so that the agent can obtain the maximum cumulative reward in the process of interacting with the environment. The development of reinforcement learning has gone through the value function method, policy gradient method, actor-critic architecture, and deep reinforcement learning. The following is an exemplary introduction to these methods:

(1) Value function method, the most important method in early reinforcement learning. The most classic algorithm is Q-learning, which learns the optimal strategy by iteratively updating the value function of the state-action pair. The value function method focuses on evaluating the value of actions and improving decisions by selecting the action with the highest value.

(2) Policy gradient method: Based on the value function method, technicians in this field began to focus on how to directly optimize the policy. Unlike the value function method, the policy gradient method directly learns the parameterized representation of the policy, rather than indirectly inferring the policy by learning the value function. The core idea of the policy gradient method is to update the parameters of the policy through the gradient descent method so that the policy can produce high-reward behaviors in the environment. Specifically, the policy gradient method is trained through the following steps:

Define the policy: First, choose a parameterized policy function, such as a neural network. The policy function receives the state of the environment as input and outputs a probability distribution over each possible action.

Collect samples: Use the current strategy to interact with the environment to generate a series of states, actions, and reward trajectories. Monte Carlo methods are usually used for sampling, that is, to collect samples by interacting with the environment multiple times.

Compute gradients: For each sample trajectory, calculate its corresponding gradient. The gradient calculation uses importance sampling techniques to adjust the gradient based on the actions selected in the trajectory and the action probabilities in the policy.

Update strategy: Average the gradients of all sample trajectories and use gradient descent to update the parameters of the strategy. The goal is to maximize the expected value of the reward.

Iterative training: Repeat steps 2 to 4 to gradually improve the strategy through interaction with the environment and parameter updates until a predetermined stopping condition is reached (such as convergence or reaching the maximum number of iterations).

(3) Actor-Critic architecture. Based on the value function method and the policy gradient method, technicians in this field have proposed a reinforcement learning method based on the actor-critic architecture, which combines the advantages of the value function method and the policy gradient method. In the actor-critic architecture, the agent is divided into two parts: one is the actor, which is responsible for generating actions, and the other is the critic, which is responsible for evaluating the value of actions. The actor generates actions according to the strategy, and the critic provides feedback signals by estimating the value function. In the actor-critic architecture, the actor uses the gradient ascent method to update the policy parameters to maximize the long-term cumulative reward. The critic uses the value function to evaluate the value of the action and provides gradient signals to the actor by updating the value function. This architecture allows the agent to estimate the value function and optimize the strategy simultaneously during the learning process, thereby making decision improvements more effectively.

(4) Deep reinforcement learning (DL). With the rise of deep learning methods, reinforcement learning has also begun to be combined with deep neural networks to form deep reinforcement learning. Deep reinforcement learning uses deep neural networks to approximate value functions or policy functions, so that it can handle high-dimensional and complex state spaces and action spaces. Typical algorithms include deep Q network (DQN) and deep deterministic policy gradient (DDPG).

Based on the introduction of reinforcement learning in the above embodiment, this embodiment mainly involves improving the existing deep reinforcement learning method. Before specifically introducing the improvement method of this embodiment, the edge computing environment involved in this embodiment is further introduced below. As shown in Figure 1, the edge computing environment includes a base station 101, which is highly associated with an edge server 102 with a computing frequency f ^ES , and is also associated with U user devices 103 (denoted as The calculation frequency f ^UE is highly associated. The user device 103 may be, but is not limited to, a smart phone, a laptop, a smart watch, etc. The user device 103 may offload the computing task to the edge server 102 for computing, or perform computing locally. In this regard, the policy device may formulate a task offloading policy for each user device 103 to be executed locally or offloaded to the edge server 102 for execution based on preset constraints.

Assume that the communication bandwidth of the system is B MHz, the system time is divided into N equal-length time periods, and the length of each time period is s. Each user equipment 103 is represented as u A computing task is generated at the beginning of the nth time period, expressed as:

In the formula, Indicates the computational task The data size, c represents the completion of the calculation task The CPU cycles required, τ represents the computational task The maximum tolerable delay.

In addition, this embodiment also specifies the calculation task is inseparable and provides a binary variable To represent the computational task It is executed locally on the user device u Or offload to edge servers for edge computing (when ). And, the available energy budget of user equipment u in the nth time period is recorded as That is, ^eUE means to perform local calculation or transfer the calculation task to each time period. Energy consumption during wireless transmission. Both the user device u and the edge server deploy task queues to prevent task loss. and They represent the queue sizes at the beginning of the nth time period.

(1) Based on the above edge computing environment, this embodiment provides the following communication model of the edge computing environment:

The block fading model used in this embodiment represents the channel gain from user equipment u to the base station in the nth time period, and the calculation formula is:

In the formula, |·| represents the modular operation of the expression in the symbol, represents a small-scale fading, which can be modeled as a first-order Gaussian Markov process according to the jak edge server model, represents large-scale fading, including path loss and log-normal shadowing. According to the LTE standard, The modeling is:

In the formula, represents the distance between user equipment u and the base station in the nth time period, and z represents the obedience is a lognormal random variable.

Furthermore, in this embodiment, the signal interference plus noise ratio of the offload link from the user equipment u to the base station in the nth time period is expressed as The calculation formula is:

In the formula, Indicates that the wireless transmission power of user equipment u in the nth time period should not exceed the maximum transmission power P ^UE , represents additive white Gaussian noise.

In the nth time period, the transmission rate from user equipment u to the base station is expressed as The calculation formula is:

(2) Based on the above communication model, this embodiment also provides an edge computing model of the edge computing environment:

Assume that the binary variable defined above The calculation task will be offloaded from user device n to the edge server. The delay duration of the task offloaded to the edge server is expressed as The task delay includes the transmission time. Edge server queue time and execution time The respective calculation formulas are:

In the formula, Indicates that the computational task The size of the task previously waiting to be executed by the edge server is calculated as:

The formula includes the queue size starting from the nth time period And computing tasks The size of the task data that has been previously unloaded. For the above expression, if If established, The value of is 1, otherwise, The value of is 0.

Based on the transmission time, queuing time, and execution time offloaded to the edge server, the edge computing task delay The calculation formula is:

Correspondingly, the energy consumption of user equipment u for:

(3) Based on the above communication model and edge computing model, this implementation also provides the following local computing model of the edge computing environment:

assumed The calculation task It will run locally on the user terminal u. Similarly, the task delay of local calculation Including the waiting time and execution time The respective calculation formulas are:

Combined with the above local queue time and execution time, the local calculation task delay calculation formula is expressed as:

Correspondingly, the energy consumption of user equipment u for:

Where ξ depends on the energy efficiency coefficient of the chip architecture of the user device u.

Based on the above calculation model, in the nth time period, the calculation task Task delay The energy consumption of user equipment u The calculation formula is:

In summary, the variables involved in this embodiment can be divided into time-invariant external variables and time-variant internal variables. Among them, the time-invariant external variables include environment description information υ ^env :

υ ^env = {P ^task ,B,f ^UE ,f ^ES }

Where ^Ptask represents the probability distribution of the task input data size, This embodiment will include a collection of V different edge computing environments To represent it, we use subscripts to distinguish different edge computing environments.

The collection of time-varying internal variables is represented by To express, Included Each element is time-dependent and is transmitted within a time period, as follows:

(4) Based on the above communication computing model, edge computing model and local computing model, the task offloading task of the edge computing environment in this embodiment is transformed into the following questions:

Without loss of generality, the goal of the computation offloading problem is defined as minimizing the weighted cost of the average task delay _tn and the energy consumption e _n of the UE over multiple time periods. It should be noted that since this embodiment involves different edge task computing scenarios, the superscripts υ ^env of _tn and e _n are omitted here for ease of description, that is, for each edge computing environment, the average task delay and energy consumption can be expressed as:

The task offloading strategy of user device u in the nth time period is expressed as Taking the task offloading strategy as the optimization variable in this embodiment, for each edge computing environment, the optimization problem of υ ^env can be expressed as:

P1:

C1:

C2:

C3:

C4:

Therefore, for multiple edge computing environments The optimization problem is transformed into:

P2:

For multiple edge computing environments The optimization problem is also subject to the constraints C1 to C4 mentioned above.

It should be noted that the current task offloading strategy Strictly depends on the previous task offloading strategy The internal variables generated (n′<n). Therefore, the above problem P1 can be transformed into an MDP (Markov decision process) problem. In this regard, this embodiment defines a five-tuple <S, A, Pr, R, γ> to represent the MDP. The S in the five-tuple represents the state space, and each time period includes a state s _n ∈S in the state space, which is defined as:

A in the quintuple represents the action space, and each time segment includes an action a _n ∈ A, which is defined as:

Pr in the quintuple represents a non-a priori state transition probability, which is used to determine the transition probability Pr(s _n+1 |s _n ,a _n ) from _sn to sn ₊₁ when executing a _n .

The R in the quintuple represents the timely reward function, which is used to generate the immediate reward R(s _n ,a _n ) in each time period according to the optimization objectives and constraints of a _n in s _n .

The γ in the quintuple represents the discount factor, which ranges from γ∈(0,1] and is used to determine the impact on future rewards.

It should be understood that under the MDP problem, the P1 problem in the above single environment can be transformed into the following P3 problem:

In the P3 problem, the goal is to obtain a strategy π that maximizes the expected long-term return of υ ^env in a single edge computing environment. The P2 problem can be transformed into the following P4 problem:

In the P4 problem, the goal is to obtain a strategy π that makes The expected long-term return in the environment is maximized compared to a single environment.

Based on the above-mentioned problem to be optimized, in order to adapt to different edge computing environments, this embodiment is also based on the Actor-Critic framework of DRL, and proposes a strategy model to be trained that includes multiple levels, multiple sub-models and can be arbitrarily combined, and uses it as the Actor in the Actor-Critic framework. A training framework including multiple strategy models to be trained and their corresponding evaluation models to be trained is constructed to perform reinforcement learning training. In the training process, multiple strategy models to be trained interact with different edge computing environments respectively during training, so as to learn the task offloading experience in different edge computing environments, so that the trained strategy model can adapt to a variety of edge computing environments.

In this embodiment, the result of training the policy model to be trained is called a policy model. Based on the policy model obtained through training, this embodiment provides a module-combined model-free computing unloading method in a multi-environment MEC. In this method, the policy device obtains the environment description information of the target edge computing environment and the current state description information; calls the pre-trained policy model to process the environment description information and the state description information, and obtains a task unloading strategy that takes into account the target edge computing environment and the state description information. In this way, compared to the conventional policy model that only generates a task unloading strategy based on the current state description information, the present application also combines the current environment description information, so that the generated task unloading strategy takes into account the target edge computing environment and the state in the environment.

Among them, the policy device for implementing the module combination model-free computing offloading method in the multi-environment MEC can be, but is not limited to, a mobile terminal, a tablet computer, a laptop computer, a desktop computer, and a server, etc. In some implementations, the server can be a single server or a server group. The server group can be centralized or distributed (for example, the server can be a distributed system). In some embodiments, the server can be local or remote relative to the user terminal. In some embodiments, the server can be implemented on a cloud platform; as an example only, the cloud platform can include a private cloud, a public cloud, a hybrid cloud, a community cloud (Community Cloud), a distributed cloud, an inter-cloud (Inter-Cloud), a multi-cloud (Multi-Cloud), etc., or any combination thereof. In some embodiments, the server can be implemented on an electronic device having one or more components.

To make the solution provided by this embodiment clearer, the various steps of the method are described in detail below in conjunction with Figure 2. However, it should be understood that the operations of the flowchart can be implemented in a non-sequential manner, and steps without logical contextual relationships can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of this application, or remove one or more operations from the flowchart. As shown in Figure 2, the method includes:

SA101, obtain the environment description information and current status description information of the target edge computing environment.

The environment description information includes the probability distribution of the input data size of the tasks corresponding to multiple user devices in the target edge computing environment, the communication bandwidth, the computing frequency of the edge server, and the computing frequency of the user device. The state description information includes the computing tasks set by each user, the bandwidth, the computing frequency of the user device, the computing frequency of the edge server, the current remaining energy, the queue length in the user device, and the queue length in the edge server.

SA102 calls the pre-trained strategy model to process the environment description information and the state description information to obtain a task offloading strategy that takes into account the target edge computing environment and the state description information.

The policy model includes multiple policy layers connected in series, and each policy layer includes multiple independent sub-models. It can be understood that this embodiment arranges and combines multiple sub-models in multiple policy layers in a dynamic management manner to obtain a sub-policy model that can adapt to different edge computing environments. The combination of multiple sub-models in multiple policy layers is described below. In an optional implementation, step SA102 may include:

SA102-1, the state embedding features of the state description information are input into multiple strategy layers.

In an optional implementation, the policy model includes a first encoder and a second encoder, and the policy device processes the state description information through the first encoder to obtain state embedding features of the state description information; and processes the environment description information through the second encoder to obtain environment embedding features of the environment description information.

Exemplarily, in this embodiment, the above-mentioned strategy model is called MC-Actor, and its structure is shown in FIG3 , in which the state description information is represented as s _n , the first encoder is represented as Encoder1, and after s _n is input to Encoder1 for processing, the state embedding feature ^es of s _n is obtained. The environment description information is represented as υ ^env , the second encoder is represented as Encoder2, and after υ ^env is input to Encoder2 for processing, the environment embedding feature ^ev is obtained.

Based on the above introduction of the embedding state characteristics and the embedding environment characteristics, step SA102 further includes:

SA102-2: For any adjacent layers in multiple strategy layers, the features output by each sub-model in the previous strategy layer are screened according to the environment embedding features and state embedding features of the environment description information to determine the input features of each sub-model in the next strategy layer.

It should be understood that, given that the model architecture of the policy model provided in this embodiment is fixed, that is, for any adjacent policy layers, the data transfer relationship between the two sub-models is fixed at the beginning of the model design. In this case, in order to be able to adjust the connection relationship between the sub-models, this embodiment adjusts the data input to each sub-model to achieve the purpose of indirectly adjusting the combination mode between the sub-models. Therefore, the optional implementation of step SA102-2 includes:

SA102-2-1, generates a weight vector for each sub-model in the next strategy layer based on the environment embedding features and state embedding features of the environment description information.

The strategy model further includes a plurality of weight layers connected in series, and the plurality of weight layers correspond one to one to parts of the plurality of strategy layers. The strategy device obtains a fusion feature between the state embedding feature and the environment embedding feature. For example, the state embedding feature and the environment embedding feature are element-wise multiplied to obtain a fusion feature.

Furthermore, the strategy device inputs the fused features into multiple weight layers; for any adjacent layers among the multiple weight layers, the weight vector output by the previous weight layer is multiplied by the fused features and then input into the next weight layer to obtain the weight vector of the strategy layer corresponding to the next weight layer.

SA102-2-2, weight the output features of each sub-model in the previous strategy layer according to the weight vector of each sub-model in the next strategy layer, and obtain the input features of each sub-model in the next strategy layer.

For example, referring to FIG3 , it is assumed that the recording model includes L strategy layers, each strategy layer includes M sub-models. Each sub-model in the lth layer is represented as F _l,m , m∈{1,2,…,M}. The input dimension of each sub-model is represented as The output dimension is expressed as In this example, each policy layer meets the following agreed conditions:

(1) F _l,m ,m∈{1,2,…,M} in the same layer should have the same dimension;

(2) Output dimension of layer l Should be equal to the input dimension of the next layer l+1

Based on the above convention, since the state embedding feature ^es is generated by the first encoder, that is, the first encoder takes the state description information s _n as input and the state embedding feature ^es as output; therefore, at the first policy layer, its input dimension is equal to D, which is consistent with the dimension of the state embedding feature ^es . For the last policy layer, since the action taken by each user device is represented as Including position judgment variables Wireless transmission power They obey Gaussian distribution respectively, and the distribution effect of Gaussian distribution is determined by two parameters μ and σ. Therefore, when there are U user devices in total, the output dimension of the last strategy layer is 4U.

The above example introduces the policy layer in the policy model. Referring to FIG3, the policy model also includes a dynamic management model composed of multiple weight layers. Since multiple weight layers are used to weight the results output by the policy layer, the number of weight layers is less than the number of policy layers. In this example, for L policy layers, L-1 weight layers are provided. For the lth layer in the L-1 weight layers, its output is represented as where i,j∈{1,2,…,M}; the output Responsible for dynamically combining the output of each sub-model F _l,j ,j∈{1,2 _, …,M} in the lth strategy layer as the input of each sub-model F _l+1,i ,i∈{1,2,…,M} in the l+1th strategy layer. For example:

In the formula, represents the kth output of the jth module F _l,j in the lth layer.

Continuing with FIG3 , for the first weight layer W ₁ , the state embedding feature ^es is multiplied element by element with the environment embedding feature ^ev , and the result of the multiplication is used as the input of the first weight layer W ₁ , and generates as output. Input to the second weight layer W ₂ to obtain the D-dimensional output. The result of the element-by-element multiplication between ^es and ^ev is multiplied by the output of W ₂ to obtain the input of the third weight layer W _3. Similarly, the input and output of each weight layer can be obtained. In this way, no matter what kind of environmental description information υ ^env of the target edge computing environment is, the reusable sub-model of the corresponding strategy layer can be extracted through the output weight of the weight layer, so as to combine the appropriate sub-model according to the environmental description information of the target edge computing environment.

It should be noted that each sub-model F _l,m in the l-th policy layer can be implemented using mathematical functions with different equations or neural networks with different hidden architectures, as long as the dimensions of its input and output satisfy the above and The constraints are sufficient.

Based on the introduction of the policy model in the above embodiment, the above step SA102 also includes:

SA102-3, based on the output results of multiple policy layers, obtains a task offloading strategy that takes into account the target edge computing environment and state description information.

In this way, through the policy model provided by this embodiment, the environment description information is used to control the policy layer's processing of the state description information to adapt to the current task computing environment, thereby obtaining the best task offloading strategy.

In addition, this embodiment also provides a training method for the above-mentioned policy model, which can be implemented by the above-mentioned policy device. In some implementations, the training method can also be implemented by other electronic devices that can provide sufficient computing power. The following is a detailed description of the training method in conjunction with a specific implementation method, which specifically includes:

SB101, obtain multiple strategy models to be trained and an evaluation model to be trained for each strategy model to be trained.

Among them, multiple strategy models to be trained correspond to different edge computing environments.

SB102, for each strategy model to be trained, the strategy model to be trained interacts with the corresponding edge computing environment to obtain task offloading experience for the current state of the edge computing environment and caches it in the experience pool.

Among them, the task offloading experience includes the environment description information of the edge computing environment corresponding to the strategy model to be trained.

SB103, after the task offloading experience collected in the experience pool meets the preset conditions, sampled experience is sampled from the experience pool, and multiple strategy models to be trained and their corresponding evaluation models to be trained are updated according to the sampled experience.

SB104, determine whether the multiple strategy models to be trained and their corresponding evaluation models to be trained have not reached the convergence condition. If so, execute step SB105, if not, execute step SB102.

SB105, the strategy model to be trained after this iteration is used as the pre-trained strategy model.

As shown in Figure 4, in a specific implementation, similar to the existing deep reinforcement learning model based on the Actor-Critic architecture, in this embodiment, each strategy model to be trained is used as an Actor to be trained, and each strategy evaluation model to be trained is used as a Critic to be trained. Each Actor to be trained is responsible for interacting with its own edge computing environment, and the Critic to be trained corresponding to the Actor to be trained is used to evaluate the impact of the task offloading strategy generated by the Actor on the long-term reward.

Different from the existing deep reinforcement learning model based on the Actor-Critic architecture, in order to improve the efficiency of experience exploration during the training process, V actors to be trained are activated to interact with their respective edge computing environments at the same time, and the interaction experience (s _n , a _n , r _n , s _n+1 ) and the corresponding environment description information υ ^env are stored in the experience pool, which is then sampled as a batch ε, and the model parameters of the actors to be trained and the critics to be trained are updated according to the preset loss function. After the training is completed, the generated actors can adapt to a variety of edge computing environments to generate the best task offloading strategy.

The above embodiment introduces a model training framework constructed to train a policy model that can adapt to a variety of edge computing environments. On this basis, this embodiment further introduces a loss function used to enable the model to converge.

First, the process of generating the task offloading strategy of the Actor to be trained is regarded as the operation process of a function, and the function is represented by π _φ , and the parameter to be optimized in the function is represented by φ; the evaluation process of the task offloading strategy by the Critic to be trained is regarded as the operation process of a function, and the function is represented by Q _θ (also known as Q value function), and the parameter to be optimized in the function is represented by θ. The result calculated by Q _θ is the long-term reward that can be achieved by performing task offloading according to the task offloading strategy provided by the current π _φ . The mathematical expression of the long-term reward is:

The purpose of training the Actor to be trained is to maximize the value of the long-term task reward after unloading the task according to the task strategy generated by the final strategy model.

It should be noted that the result generated by π _φ satisfies the multidimensional Gaussian distribution N _φ (μ,σ), which means that the mean μ and covariance σ of each element in the task offloading strategy a _n are similar to the result generated by π _φ , that is, π _φ (a _n |s _n ) represents the Gaussian probability distribution of selecting the task offloading strategy a _n under the state represented by s _n . Therefore, the following loss function J _π (φ) is provided for the Actor to be trained;

Provide the following loss function J _π (θ) for the Critic to be trained:

where D _KL represents the DL divergence, Z _θ (s _n ) represents the distribution function of the normalized exp(Q _θ (s _n ,a _n )), represents an independent soft-state value function parameterized by ψ (ψ represents the target network V _ψ for functional stability). ψ can be optimized by minimizing:

Furthermore, since the Actor to be trained includes multiple strategy layers, each strategy layer includes multiple sub-models, in order to be able to learn to dynamically decide whether to reuse a sub-model for different edge computing environments, the Actor to be trained also includes multiple weight layers, and the input of the Actor to be trained includes environment description information and state description information.

Since the Actor to be trained includes multiple policy layers and multiple weight layers, the parameters of both need to be updated during the training process. The Actor network to be trained will output multiple probability distributions, and then sample from this distribution to determine the specific action. However, the sampling operation itself is not differentiable, which means that the backpropagation algorithm cannot be used directly to update the parameters of the Actor network, because the backpropagation algorithm relies on differentiability to calculate the gradient. For this reason, this example uses the non-cumulative parameterization method Gumbel-Softmax to solve The back propagation difficulty problem caused by non-differentiable sampling in the proposed algorithm is solved by introducing additional noise variables to approximate the sampling process, thereby making the sampling from the probability distribution differentiable, so that the model can be trained end-to-end.

For ease of description, φ and denote the network parameters in the strategy layer and the weight layer respectively. The loss function of the unloading strategy π can be rewritten as:

Furthermore, it was found during the research process that some submodules were selected and used multiple times during the training process, while some submodules were hardly selected and trained. Therefore, in order to avoid module degradation, this embodiment also designed a regularization term R and added it to the rewritten loss function of the Actor to be trained:

In the formula, the design of R takes into account the long-term sum of the combined weights of each module j in the lth layer Where T is the time range. In order to prevent some sub-models from being exclusively selected and trained, define The goal is to minimize the two modules j and j′ of any layer l.

In addition, this embodiment also alleviates the problem of overfitting by introducing dropout. In this regard, it should be understood that Dropout is a regularization technology for training neural networks, which can reduce the overfitting problem of neural networks by randomly removing some neurons in the neural network during the training process. Therefore, in this embodiment, dropout is used to randomly discard a sub-model with a certain probability, so that in each training iteration, each sub-model has a certain probability of being selected and closed. In the specific implementation process, a vector is maintained. Each element is a Bernoulli random variable with probability p ^drop equal to 1. Therefore, the kth input of the submodel F _l+1 can be expressed as:

Based on the above loss function, V actors to be trained interact with their respective edge computing environments in parallel to improve the efficiency of exploring experience. The task offloading experience in the exploration process is cached in the experience pool ξ. When the cached task offloading experience reaches the preset condition, it is sampled from the experience pool ξ, and the parameters of the actors to be trained and the critics to be trained are updated with the sampled experience.

It is worth noting that in order to improve the training stability, the strategy layer and weight layer in the training Actor are updated alternately according to the preset period during the training process. The update period of the network parameter φ of the strategy layer is represented as Γ _φ , and the update period of the network parameter φ of the weight layer is represented as Γ φ . The update cycle is expressed as

Based on the same inventive concept as the module-combined model-free computing unloading method in a multi-environment MEC provided in this embodiment, this embodiment also provides a module-combined model-free computing unloading device in a multi-environment MEC. The module-combined model-free computing unloading device in a multi-environment MEC includes at least one software function module that can be stored in the memory 301 in software form or solidified in the policy device. The processor 302 in the policy device is used to execute the executable module stored in the memory 301. For example, the software function modules and computer programs included in the module-combined model-free computing unloading device in the multi-environment MEC. Please refer to Figure 5. From a functional perspective, the module-combined model-free computing unloading device in the multi-environment MEC may include:

The information acquisition module 201 is used to obtain the environment description information and current state description information of the target edge computing environment;

The strategy generation module 202 is used to call a pre-trained strategy model to process the environment description information and the state description information to obtain a task offloading strategy that takes into account the target edge computing environment and the state description information.

In this embodiment, the information acquisition module 201 is used to implement step SA101 in Figure 2, and the strategy generation module 202 is used to implement step SA102 in Figure 2. Therefore, for the details of each of the above modules, please refer to the specific implementation of the corresponding step, and this embodiment will not be repeated.

In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

It should also be understood that if the above implementation is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present application.

Therefore, this embodiment also provides a storage medium, which stores a computer program. When the computer program is executed by the processor, the module combination model-free computing offloading method in the multi-environment MEC provided by this embodiment is implemented. Among them, the storage medium can be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc., which can store program codes.

This embodiment provides a policy device for implementing a module combination model-free computing unloading method in a multi-environment MEC. As shown in FIG6 , the policy device may include a processor 302 and a memory 301. In addition, the memory 301 stores a computer program, and the processor implements the module combination model-free computing unloading method in a multi-environment MEC provided in this embodiment by reading and executing the computer program corresponding to the above implementation method in the memory 301.

6 , the electronic device further includes a communication unit 303. The memory 301, the processor 302 and the communication unit 303 are directly or indirectly electrically connected to each other via a system bus 304 to achieve data transmission or interaction.

The memory 301 may be an information recording device based on any electronic, magnetic, optical or other physical principle, used to record execution instructions, data, etc. In some implementations, the memory 301 may be, but is not limited to, a volatile memory, a non-volatile memory, a storage drive, etc.

In some embodiments, the volatile memory may be a random access memory (RAM); in some embodiments, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), a flash memory, etc.; in some embodiments, the storage drive may be a disk drive, a solid-state drive, any type of storage disk (such as a CD, a DVD, etc.), or a similar storage medium, or a combination thereof, etc.

The communication unit 303 is used to send and receive data through a network. In some embodiments, the network may include a wired network, a wireless network, an optical fiber network, a telecommunication network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a wide area network (WAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network, or a near field communication (NFC) network, or any combination thereof. In some embodiments, the network may include one or more network access points. For example, the network may include a wired or wireless network access point, such as a base station and/or a network switching node, through which one or more components of the service request processing system may be connected to the network to exchange data and/or information.

The processor 302 may be an integrated circuit chip having a signal processing capability, and the processor may include one or more processing cores (e.g., a single-core processor or a multi-core processor). By way of example only, the processor may include a central processing unit (CPU), an application specific integrated circuit (ASIC), an application specific instruction set processor (ASIP), a graphics processing unit (GPU), a physical processing unit (PPU), a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic device (PLD), a controller, a microcontroller unit, a reduced instruction set computer (RISC), or a microprocessor, or any combination thereof.

It is understood that the structure shown in Figure 6 is for illustration only. The policy device may also have more or fewer components than those shown in Figure 6, or may have a different configuration than that shown in Figure 6. Each component shown in Figure 6 may be implemented in hardware, software, or a combination thereof.

It should be understood that the apparatus and method disclosed in the above-mentioned embodiments can also be implemented in other ways. The apparatus embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the apparatus, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and/or the flowchart, and the combination of boxes in the block diagram and/or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims (10)

1. A module combination model-free computing offloading method in a multi-environment MEC, characterized in that the method comprises: Obtain the environment description information and current status description information of the target edge computing environment; A pre-trained policy model is called to process the environment description information and the state description information to obtain a task offloading policy that takes into account both the target edge computing environment and the state description information. 2. According to claim 1, the module combination model-free computing offloading method in a multi-environment MEC is characterized in that the policy model includes a plurality of policy layers connected in series, each policy layer includes a plurality of sub-models independent of each other, and the calling of the pre-trained policy model processes the environment description information and the state description information to obtain a task offloading strategy that takes into account the target edge computing environment and the state description information, including: Inputting the state embedding features of the state description information into the multiple strategy layers; For any adjacent layers in the multiple strategy layers, the features output by each sub-model in the previous strategy layer are screened according to the environment embedding features of the environment description information and the state embedding features to determine the input features of each sub-model in the next strategy layer; According to the output results of the multiple policy layers, a task offloading strategy that takes into account both the target edge computing environment and the state description information is determined. 3. The module combination model-free computing offloading method in multi-environment MEC according to claim 2 is characterized in that the output features of each sub-model in the previous strategy layer are screened according to the environment embedding features of the environment description information and the state embedding features to determine the input features of each sub-model in the next strategy layer, including: Generate a weight vector for each sub-model in the next strategy layer according to the environment embedding feature of the environment description information and the state embedding feature; The output features of each sub-model in the previous strategy layer are weighted according to the weight vector of each sub-model in the next strategy layer to obtain the input features of each sub-model in the next strategy layer. 4. According to the module combination model-free computing offloading method in multi-environment MEC according to claim 3, it is characterized in that the strategy model also includes a plurality of weight layers connected in series, and the plurality of weight layers correspond one-to-one to parts of the plurality of strategy layers; the weight vector of each sub-model in the next strategy layer is generated according to the environment embedding feature of the environment description information and the state embedding feature, comprising: Acquire a fusion feature between the state embedding feature and the environment embedding feature; Inputting the fused features into the multiple weight layers; For any adjacent layers among the multiple weight layers, a weight vector output by a previous weight layer is multiplied by the fusion feature and then input into a next weight layer to obtain a weight vector of a strategy layer corresponding to the next weight layer. 5. The module combination model-free computing offloading method in multi-environment MEC according to claim 4 is characterized in that the step of obtaining the fusion feature between the state embedded feature and the environment embedded feature comprises: The state embedding feature and the environment embedding feature are multiplied element by element to obtain the fusion feature. 6. The module combination model-free computing offloading method in multi-environment MEC according to claim 2, characterized in that the strategy model also includes a first encoder and a second encoder, and the method further includes: Processing the state description information by the first encoder to obtain a state embedding feature of the state description information; The environment description information is processed by the second encoder to obtain environment embedding features of the environment description information. 7. The module combination model-free computing offloading method in multi-environment MEC according to claim 1, characterized in that the method also includes a training method for the policy model, and the training method includes: Acquire multiple strategy models to be trained and an evaluation model to be trained for each of the strategy models to be trained, wherein the multiple strategy models to be trained correspond to different edge computing environments respectively; For each strategy model to be trained, the strategy model to be trained interacts with the corresponding edge computing environment to obtain task offloading experience for the current state of the edge computing environment, and caches it in the experience pool; After the task offloading experience collected by the experience pool meets a preset condition, sampling experience is sampled from the experience pool, and the multiple strategy models to be trained and their corresponding evaluation models to be trained are updated according to the sampling experience; If the multiple strategy models to be trained and their corresponding evaluation models to be trained do not meet the convergence conditions, then return to each strategy model to be trained, interact with the corresponding edge computing environment through the strategy model to be trained, and obtain the task offloading experience for the current state of the edge computing environment, until the convergence conditions are met, and use the strategy model to be trained after this iteration as the pre-trained strategy model. 8. The module combination model-free computing offloading method in multi-environment MEC according to claim 7 is characterized in that the multiple strategy models to be trained and their corresponding evaluation models to be trained are updated according to the sampling experience, including: According to the sampling experience, multiple strategy layers and multiple weight layers in each of the strategy models to be trained are updated alternately, and the evaluation model to be trained corresponding to each of the strategy models to be trained is updated. 9. The module combination model-free computing offloading method in multi-environment MEC according to claim 7 is characterized in that the task offloading experience includes the environmental description information of the edge computing environment corresponding to the strategy model to be trained. 10. A module-combined model-free computing offloading device in a multi-environment MEC, characterized in that the device comprises: An information acquisition module is used to obtain the environment description information and current status description information of the target edge computing environment; The strategy generation module is used to call a pre-trained strategy model to process the environment description information and the state description information to obtain a task offloading strategy that takes into account both the target edge computing environment and the state description information.